SSC DISCORD SERVER AT https://discordapp.com/invite/gpaTCxh ; SCHELLING POINT FOR DISCUSSION IS WED 10 PM EST

Another Followup To “Economists On Education”

Last month I argued that a news article misrepresented the feelings of economists on school vouchers. I got a bit of pushback from people who thought I had just misread it, and that it wasn’t deceptive at all (1, 2, 3), and wrote an addendum post sticking to my guns.

One of my New Years resolutions is to be more empirical, so I thought this would be a good opportunity for an experiment. I offered to bet at 10:1 odds in the other person’s favor that most people shown the article would believe economists were against school vouchers. Noah Smith originally took me up on the bet, but later stopped cooperating with me about it and wouldn’t give a straight answer to the question of whether he was retracting his offer. Vikram Sasi agreed to continue the bet; I’m not sure if he disagreed with my assessment, thought the odds were too good to refuse, or just wanted the experiment to go ahead.

I proposed that we show some random people the article, then ask them a few easy comprehension questions to make sure they were putting in a decent effort. Then we would ask the following:

According to this article:
a) Most economists support privatizing education
b) Most economists oppose privatizing education
c) Unsure / the article doesn’t say

The true answer was (c) – the article doesn’t give enough information to tell you what economists think (which turns out to be mostly unsure, with more supporting the privatization than opposing). But I thought the article heavily (and falsely) implied b, and expected most people to give that answer.

Before Vikram and I could agree on exactly how we were going to do this, we got scooped by two other people who did the experiment without asking us.

Commenter Sripada got 40 people to take the survey on MTurk, of whom 35 answered the easy questions right and qualified for inclusion. Of those 35, 32 of them (91%) answered (b), misinterpreting the article to claim a consensus against privatizing education. Only one of them (3%) gave the correct answer (c), accurately noting that the article doesn’t tell us anything about an economic consensus for or against.

Hanne Watkins, a real research psychologist, also did a 50 person MTurk survey. I don’t have enough access to her data to limit the sample to people who answered the easy questions right, but among all participants 78% believed that most economists opposed privatization, compared to only 10% who correctly noted that the article didn’t say.

Vikram graciously accepted these studies as settling the bet, and since more than half of people misinterpreted the article in both, he paid me my $10.

I’m really happy about this, not just because I won ten dollars but because I feel like it was a rare example of getting to decisively settle a disagreement. Instead of arguing about whether or not the article was misleading, we figured out a way to test it, did the test, and now we know.

I should add that some people might think the article was non-misleading in a different way than I did that can’t be settled empirically (Noah seems to be in this group), and other people might think the particular set of questions the survey asked was unfair or biased. This is why I wish I’d gotten to bet somebody who I knew really disagreed with me about this, so at least that disagreement could have been settled for sure. As a consolation prize, I will take my ten dollars and my increased certainty that the article didn’t accurately convey economists’ thoughts.

Thanks to Vikram, Sripada, and Hanne for their help with this.

This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

203 Responses to Another Followup To “Economists On Education”

  1. TheWackademic says:

    But that isn’t what the survey of economists actually asked. If you re-asked the mTurk question as follows:

    According to this article:
    a) Most economists support privatizing education
    b) Most economists oppose privatizing education or are unsure of its benefits
    c) The article doesn’t say

    I’d be happy to bet that a majority would correctly choose B, and it would like like your initial post was wrong.

    • Scott Alexander says:

      …that’s only because you’re asking us to stick the correct-but-not-said-by-the-article answer in the same bin as the incorrect-but-said-by-the-article answer.

      If an article claimed Chicago was the US capital, and I asked people:

      ACCORDING TO THIS ARTICLE, WHAT IS THE US CAPITAL:

      a) Los Angeles
      b) Chicago or Washington DC
      c) New York City

      …then your survey would find that readers correctly select that Washington is the capital, and so the article was “correct” and “not misleading at all”.

      • TheWackademic says:

        I would suggest instead that the question, as posed by you, resembles the following:

        Which number is larger than -1?
        a) -2
        b) 1
        c) 0 or the article doesn’t say.

        Both B and C are correct, but given the common injunction from high school multiple choice test questions – “choose the most correct answer” – B comes out on top. If you rephrase the possible answers as

        a) -2
        b) 0 or 1
        c) the article doesn’t say

        then I think that our disagreements would be solved.

        Edit: this post was written in response to Scott’s original comment that involved Nazis, but still kinda makes sense after his edit.

        • Scott Alexander says:

          I still think you’re wrong. Your (b) only works because both answers are true. In the real article example, one of the answers is false and the other is true. You can’t just replace a false thing with a true thing in your example and expect it to make sense.

        • batmanaod says:

          I’m not sure what your point is with your first set of answers, but I completely disagree that B is the “best answer” there. Since the article doesn’t say anything about those numbers, C has two correct statements, whereas B has only one. The fact that 1 is further from -1 than 0 is doesn’t make “1 > -1” in any way “more correct” than “1 > 0”.

    • GMHowe says:

      While you are pointing out a potential bias or at least a weirdness in the structure of the mTurk question, your proposed replacement is quite clearly untenable. If B is chosen most Scott would win the bet since it contains the claim he made and you would also win since it contains the claim you made.

  2. Jacob says:

    +100 on promoting futarchy for a brighter future for humanity 🙂

    I think that the Slate Star Codex readership is going to be particularly receptive to doing more of these public prediction bets with each other, and either the blog or the subreddit could be great places to coordinate prediction markets. I do a lot of prediction bets with friends on Facebook, but these work best when a lot of people are involved.

    How about establishing a norm of having a prediction market section on every open thread? It’s probably illegal for Scott to be the clearing counterparty (i.e. hold the money). Instead, we can either find someone who knows both bettors and can vouch for them, or start small and let people build up their reputations of paying their bets honestly.

    • Aninhumer says:

      I never really understood how Futarchy was supposed to work. Predictions markets are a powerful tool, but when it comes to economics decisions, it’s often difficult to know how well different choices would have played out. How would you determine the payout for someone who bets that e.g. Brexit is a good idea, if the predictions market chooses remain?

      • Luke Perrin says:

        For policies that aren’t chosen the bets are just canceled, i.e. the amounts bet are just returned to whoever originally had them.

    • Edward Scizorhands says:

      Once it becomes known how we are making decisions, how do we stop people from brigading the test?

      I’m not saying that happened here, although I’d like to know what the real research psychologist knows that I don’t to make it not happen. I’ve only messed with MTurk a few times and am unfamiliar with its patterns.

      • AnonEEmous says:

        i haven’t used mechanical turk either but I imagine it’s difficult to locate, and any given person is not going to announce the survey beforehand, though they might tip their hand some

        sure, it’s possible to be at the top of your game, anticipate a survey, locate it, and hit it up. that’s a fair concern and you make a good point, that Scott or anyone else should keep the surveys fairly hush-hush until completed. However so long as reasonable efforts are made, we should be all clear

  3. harland0 says:

    Oh, come on. Bashing the NYT for biased journalism? This just plays into the pro-Trump zeitgeist and is something that really needs to be put on the back burner right now. Trump manipulated journalism to serve as icing on the Faux News cake in order to sucker tens of millions of uneducated, flatheaded Amurcans into jumping on his bandwagon… like so many chimpanzees piling onto the fruit cart. Instead maybe we need to talk about the topics the NYT gets right, or maybe a profile of their journalists struggling against the anti-free speech White House that just locked the doors to the press room.

    • Ilya Shpitser says:

      I have a better idea. How about NYT tries harder, instead.

      What you suggest is entirely the wrong lesson to draw from Trump. The race to the bottom approach to dealing with Trump has no winners, in the end.

    • Jacob says:

      How about the NYT work to become more honest, and then our small and politically insignificant group of weirdos who care about honesty will stop criticizing them? By the way, honesty doesn’t mean not making mistakes, it means admitting them when you do.

      For example, there is no doubt that had Scott lost his bet he would have publicly stated that he was wrong about the quality of the article. Similarly, there is little doubt that no amount of readers being actively misinformed by the article will make the NYT publish a correction to it. Also, it would be quite shocking if the NYT made a mistake that misled readers towards a Republican political position, wouldn’t it?

      And finally, the mere fact that Scott is reading the NYT is an endorsement, many other outlets are so much worse that it’s not even interesting to point out when they’re lying.

      • Jacob says:

        An ad for a paid subscription to the New York Times just popped up on my Facebook feed with the slogan: “Truth. It comes at a cost.” I think it’s fair to hold them to slightly higher standards 😉

    • Anonymous Bosch says:

      This comment is triggering my Poe’s Law sense

    • MF says:

      I struggled with deciding on whether to reply to this comment. Political topics are very difficult to discuss without things going very wrong. I’m still not entirely sure it’s serious and not meant as sarcasm. Forgive me if it was sarcastic.

      I think your attitude that we must support the NYT (and presumably focus all efforts on anti-Trump measures) is a major problem.

      Trump is not 100% wholly bad. Those positions of his that are beneficial to society we should support him on.

      The NYT is trying to persuade its viewership, via the discussed article, that education vouchers are a bad idea. But most economists are either unsure or think they would be an alright idea.

      This is important! If education vouchers will indeed save us money without harming our children’s education, we should want to implement education vouchers. The NYT is actively trying to muddy the truth, here. This is a problem.

      I have often seen right-wingers of late bashing left-leaning publications like the NYT for putting out biased stories. Your solution is… to deny or ignore the fact that this criticism is entirely correct. Can you perhaps see a problem here when you give your enemy true reasons to dislike and band against you?

      Both sides do this of course; this is not a failing in particular of the left. But confirmation bias (or whatever variant of it goes in tribalism) means anyone doing it is giving their opponents more ammo.

      This is not a “war” in the typical sense. We do not need to support our “soldiers” when they are in the wrong.

      As well, the NYT admitting they made a mistake and correcting this would probably help with the image of the media more than stubbornly ignoring it and yelling at anyone who brings it up.

      • harland0 says:

        The NYT does not “persuade” anywhere but the editorial page. Reality has a well-known left-wing bias. Can we help it if the NYT reports facts as they are?

        Trump just locked journalists out of the White House in a direct attack on freedom of the press. This needs to be talked about much more than any school vouchers issue!

        • skybrian says:

          Oh, come on. Of course they try, but it is not easy to report facts as they are. Everyone makes mistakes, even good reporters.

          (At the very least, reporters try to persuade you that they did a good job and you should believe them.)

          • cvxxcvcxbxvcbx says:

            I don’t really think this was a mistake on the part of the reporter. It’s too obvious for that. Can you imagine someone looking at that graph and then writing that article without it occurring to them that they were misleading?

        • Deiseach says:

          Reality has a well-known left-wing bias.

          The only bias reality has is that it will kill us all in the end. Entropy increases.

          If somebody tries to tell me a rock has political views, it simply makes me want to hit them over the head with that rock.

          • alwhite says:

            “Pick up one of those rocks, get behind a boulder, in a few minutes the man in black will come running around the bend, the minute his head is in view, HIT IT WITH THE ROCK.”
            -Vizzini

          • 1soru1 says:

            A rock absolutely holds views on the topic of Young Earth creationism, which is borderline relevant to politics. The political views of the atmosphere are rather more significant, though.

          • Tekhno says:

            Excuuuuuuuse me, but you Entropists would say that!

          • komponisto says:

            A rock absolutely holds views on the topic of Young Earth creationism

            By this reasoning, all humans agree that they are descended from non-human apes.

            Also, everything in the universe is in total agreement about absolutely everything.

        • AnonEEmous says:

          AnonEEmous says:

          January 17, 2017 at 1:08 am

          a talent for recognizing internet sarcasm goes a long way it seems

        • cvxxcvcxbxvcbx says:

          We cannot help it the NYT reports facts as they are, nor should we. We can help it if the NYT tells lies and quarter-truths about important policy decisions, and we should.

        • wintermute92 says:

          The NYT does not “persuade” anywhere but the editorial page (…) Can we help it if the NYT reports facts as they are?

          Fine, I’ll bite. In August, the NYT ran this article, explaining how US intervention in Honduras is making one of the most dangerous places in the world safer via things like paying for street lights. The subhead is “Programs funded by the United States are helping transform Honduras. Who says American power is dead?”

          That’s sweet and all, but half a dozen media watchers took them to task for this article. Missing from the story was any discussion of the history of US power in Honduras, which has consisted of repeatedly destabilizing the country to prop up leaders favorable to US government and business interests. From its Banana Republic status in the 40s, to CIA murders in the 80s, to the US-enabled coup as recently as 2009, the worst violence in Honduras has been the legacy of US action.

          Despite its recent murder rate, it was substantially safer than similar countries (e.g. El Salvador) through the 90s under the rule of a legitimately-elected government which reined in the military and attempted to account for the tortures and murders enabled by the US a decade prior. The violent instability which the NYT implies is a longstanding problem has only spiked over the past decade, and despite the article’s cheerleading has not improved under the violent military government the US legitimized in 2009.

          So yes, the NYT reported several ‘facts’ as they are, like dollars spent on streetlights. But the article’s content was aggressively misleading, and its timing was nicely arranged to counterbalance increasing discussion of the Clinton State Department’s role in enabling the 2009 coup. This ran in Opinion, as pretty much all guest articles do, but was certainly not an editorial.

          Reality may have a left-wing bias, but the NYT has a well-known establishment bias, and is happy to engage in some creative analysis to ensure that the “facts as they are” end up looking like they want.

        • Mary says:

          yes, of course, the paper of Walter Duranty is so scrupulous about the facts.

          (Note they have never so much as apologized for publishing him.)

          • Edward Scizorhands says:

            They haven’t said the literal words “We’re sorry” or “We apologize” but they have said “The result was some of the worst reporting to appear in this newspaper” and “Since the 1980′s, the paper has been publicly acknowledging his failures.”

          • Interesting. Do you know if “The Economist” has ever said anything similar about crediting Mao with ending famine in China?

            Since I don’t read either of them on any regular basis I am unlikely to see an apology/retraction and tend to assume, on inadequate grounds, that there hasn’t been one.

          • nydwracu says:

            Has Nick Lemann ever apologized for starting his career in journalism by endorsing the Khmer Rouge?

          • Or, for a more prominent example, has Noam Chomsky ever apologized for writing apologetics for the Khmer Rouge?

      • quarint says:

        Trump is not 100% wholly bad. This is true. He is only 99% bad and both Scott and his readership have a weird obsession with the 1%.

        • AnonEEmous says:

          care to get into a serious argument as regards this percentage ?

          • quarint says:

            Obviously, the percentage was not to be taken literally. My point was merely that Scott spends an awful lot of time lecturing anti-Trump people or media, compared to the time he spends on lecturing Trump supporters, for someone who still finds Trump terrible enough to call his audience to vote for the terrible-but-not-nearly-as-terrible-as-Trump Hillary Clinton.

            I’m not sure how much of this is due to a rationalistic excess of charitableness for the “other” side, to the appeal to his right-wing readership (whose majority doesn’t necessarily endorse Trump, but sure as hell enjoys some good ol’ lefty bashing), or to the potential boringness of criticizing Trump or his supporters, or something else that I didn’t think of.

          • Spookykou says:

            Scott has covered, in detail, why the focus of this blog is as it is. His outgroup is basically just mean/dishonest/uncharitable blue tribe, and he dedicates more of his time to talking about them than his fargroup.

            They will often be the people making uncharitable/dishonest/mean attacks against the red tribe, but it is important to remember that he is motivated by the meta principles above. He is not defending any given red position when he calls out members of the blue, he is calling out members of the blue to try and fix their behavior, period, full stop.

            So when he writes a post saying “Trump is bad, don’t vote Trump” and another post saying “the NYT is dishonest” please don’t be so disingenuous as to imply that Scott is actually trying to defend Trump in the latter, he is trying to fix the NYT, extend even the tiniest bit of good faith or trust and this becomes obvious.

          • Desertopa says:

            It’s also worth keeping in mind that it’s almost impossible to convince people who disagree with you of anything if they don’t trust you to be honest about the evidence relating to your disagreement.

            If Trump supporters can take it for granted that the NYT will oppose Trump on any issue regardless of the evidence, they’ll give no weight to the NYT’s opposition. Scott’s opposition to Trump is more meaningful precisely because he tries only to oppose Trump to the extent that evidence favors doing so.

          • Tarpitz says:

            I would guess that Scott’s social circle contains few if any Trump voters, and an awful lot of anti-Trump people, and that most criticism of Trump consequently seems obvious and unnecessary. I do a very different job to Scott, in a different country, but I speculate that our social circles nevertheless share a median estimate of Trump Good Percentage, namely 0, and have a tendency to look favourably on accurately punctuated but misleading articles which support their existing opinions.

        • The problem is by not separating what’s real, from what’s fake, people lose trust entirely, and then they read Breitbart instead of the NYtimes.

          At the age of 17 I started reading the NYtimes. Ten years later and a graduate degree in Political Economy later I feel confident that I can spot the progressive ideology, and separate it from the clever reporting. Most people can’t do that, because they don’t think staying inside and reading old books, academic papers, and obscure blogs is a good use of their life.

          If anything, Scott’s group is the one saying “I know the NYtimes is filled with bullshit, but we can try to fix it — or at least explain what’s wrong with it — so you don’t instead reject it all and jump ship to the Alt Right (or whatever)”

      • xq says:

        “Actively trying to muddy the truth” seems like a huge leap from what has actually been demonstrated here. People write unintentionally misleading things all the time.

        • MF says:

          On reflection, I agree with you in that I probably worded that too strongly. It certainly hasn’t been proven.

          However…

          The NYT has a clear bias against Trump. Its readership hates him. The point of this article is to continue the Trump hate by bashing his advisors’ positions. And conveniently, the author is unintentionally misinterpreting facts to support this narrative?

          At some point, malice becomes a more likely explanation than stupidity. Susan Dynarski, the author, “is a professor of education, public policy and economics at the University of Michigan”. She’s not an idiot.

          There’s a decent possibility that she read the survey of economists and some sort of insane political bias made her interpret it wholly in favor of her worldview. Perhaps she sat down to write the article, already having her conclusion firm in mind (“Trump’s advisor holds a harmful position on education”) and then searched out evidence to support it as she wrote the article, barely glancing or thinking about what she was doing.

          But then nobody edited it afterwards. Nobody thought to take a look at her source, which trivially would show she was badly misinterpreting it. They put this article on their site. That says a lot.

          • HeelBearCub says:

            It’s essentially an OpEd piece from someone who has a particular viewpoint. If the OpEd was from someone at the Heritage Institute about the promise of school privitization, I think you would understand that it isn’t “The Times” writing this.

          • xq says:

            The point of this article is to continue the Trump hate by bashing his advisors’ positions.

            I see no reason to believe this is true. The author is a professor of education and economics who has been researching this subject for much of her professional life. She didn’t come up with her opinions in order to bash Trump. There’s no reason to believe the NYT solicited the article in order to bash Trump. NYT has written about education policy for a lot longer than Trump has been in politics.

            There’s a decent possibility that she read the survey of economists and some sort of insane political bias made her interpret it wholly in favor of her worldview.

            But the survey does support her worldview, so there’s no evidence she is misinterpreting it. Her point is that “economists are far less optimistic about what an unfettered market can achieve in education” than in the taxi market. This is absolutely correct and this is absolutely demonstrated by the comparison in polls that she points out. It’s fair to say that one sentence is misleading about the views of economists on vouchers, but it is not misleading in a way that is necessary for her argument.

          • Deiseach says:

            Susan Dynarski, the author, “is a professor of education, public policy and economics at the University of Michigan”. She’s not an idiot.

            I’m very tempted to say “One does not preclude the other”, given my (admittedly very limited, thank God!) exposure to trends in education. I wonder if Professor Dynarski’s field lies more in education or in economics? If the former, I wouldn’t be surprised if a little bit of “school vouchers is bad, and look, I got economists what know all about money to tell you so!” is going on.

          • gbdub says:

            “It’s just an op-ed” is a lame defense. Op-ed slots at the NYT are in limited supply and are not assigned by lottery. The selection of authors, topics, the final decision to publish, and often the title all rest with the editorial board, and it’s perfectly fair to hold them to some degree of account if the op-ends they choose to run are misleadingly slanted.

          • Deiseach says:

            Her point is that “economists are far less optimistic about what an unfettered market can achieve in education” than in the taxi market.

            If that were the point being made, I think we’d all agree it’s neutral. But the impression is – and maybe I’m leaning too heavily on the headline here, but again I maintain that the headline is what most people remember – that “a majority of economists think school vouchers are a bad idea”:

            Free Market for Education? Economists Generally Don’t Buy It

            Now, we can argue all day over what the interpretation of “generally” is, but I think it is not going too far to say that this headline implies a negative view of a free market in education, and that it’s not milkmen, bricklayers or chicken farmers who think so but economists, who are reputable and credible when speaking on matters of the economy and spending money. Moreover that more economists than not have this negative (as opposed to “don’t know or don’t want to say one way or the other”) opinion, which is the work “generally” is doing there: ‘economists in general’.

          • xq says:

            I think your interpretation of the article is partly a result of Scott’s own quite misleading posts on this subject, which treated the article as primarily about school vouchers, which it is not. School vouchers are just a couple of paragraphs; only one piece of evidence in support of a larger point. “Generally don’t buy it” does not refer to vouchers, but to “free market in education.” So the headline does not imply that most economists think vouchers are a bad idea.

            But, anyways, let’s say we don’t like the headline. So what? What broader lesson are we supposed to take from that? Misleading headlines are written all the time. I still see no reason to believe it’s a result of deliberate intention to misinform by either the author, who did not write the headline, or whichever editor did.

          • HeelBearCub says:

            @gbdub:

            Op-ed slots at the NYT are in limited supply and are not assigned by lottery.

            This was in “The Upshot” which is a) digital only, and b) definitely not The New York Times Op-Ed page. It’s a digital only sub-brand of the times that was born after Silver took 538 to ESPN.

            The particular opinion piece in question appeared in a regular feature at The Upshot called “Economic View” which is described as “A column that explores life through an economic lens with leading economists and writers.”

            It is not the opinion of The Times editorial board, which is seems to be the implicit argument people are making, nor is it anywhere close to appearing on the main op-ed page of the print edition. But even if it was, that still doesn’t make it the opinion of “The Times”.

            No one is saying that what Ross Douthat or David Brooks writes is the opinion of The Times.

          • gbdub says:

            School vouchers are just a couple of paragraphs; only one piece of evidence in support of a larger point. “Generally don’t buy it” does not refer to vouchers, but to “free market in education.” So the headline does not imply that most economists think vouchers are a bad idea.

            Literally the only evidence in the article about what “economists” “generally” think is the result of a survey question that was specifically about school vouchers. It is the article and headline author(s), not Scott, that make(s) the leap from “economists have mixed responses on vouchers” to “economists don’t buy free markets in education” in general. If anything Scott is being too generous by focusing on the interpretation of the evidence itself, rather than noting that the evidence is too weak to support the claims even if the original interpretation were fair.

          • xq says:

            It is the article and headline author(s), not Scott, that make(s) the leap from “economists have mixed responses on vouchers” to “economists don’t buy free markets in education” in general.

            The article does not make that leap. The author makes a number of additional claims about what economists believe that does not rest on that poll, in support of the general point that “economists don’t buy free markets in education”. These claims are not supported by survey evidence, or any other sort of quantitative data, but that’s not necessary by the general standards of op-eds.

    • Scott Alexander says:

      Part of me agrees with you, but I feel like when the mainstream media really is biased, that makes it much easier for Trump to criticize, and so part of the project of deflating Trump’s criticism of the media is making it less worthy of criticism.

      I’m also not really sure how to talk about topics the NYT gets right. Usually I do it by linking their articles and talking about them. Saying “…and as far as I can tell there wasn’t any bias or lies in this one” seems kind of counterproductive.

      I’ve removed the emphasis on the NYT in the first sentence, since I agree that picking on that paper in particular is becoming annoying.

      • Randy M says:

        I feel like when the mainstream media really is biased, that makes it much easier for Trump to criticize

        I certainly hope this is true!

        Saying “…and as far as I can tell there wasn’t any bias or lies in this one” seems kind of counterproductive.

        It would seem to imply a rather cynical view of the subject. Right or wrongly, tough to say.

      • Edward Scizorhands says:

        “Stop saying the Emperor has no clothes! You are empowering the evil anti-Emperor rebels!”

        All that said, given that this commenter is pushing so many buttons, I’m now confident it’s a troll posing as the most obnoxious liberal it can imagine. In some ways it’s inspiring as a work of art and engineering.

      • rin573 says:

        Assuming the post above is serious, I think that part of harland0’s concern is that you seem to be critiquing bias in the NYT specifically without mentioning misleading news in right wing sources. On one level, this makes sense – presumably you read the NYT, saw this problem, and wanted to point it out. If you assume that your audience is already aware of biased reporting in sources like Breitbart (or to give a less blatant example, Reason), it makes perfect sense. You’re informing them of something they don’t know, challenging what might be a complacent, uncritical mindset. You’re also connecting with people who genuinely want the NYT to do better, and may push it to raise the quality its journalism/opinion pieces.

        But I’m not sure your readers actually do tend to read the NYT more uncritically than Reason. Based on comments to other posts, I’d guess that at least as many active readers do the opposite, reading right-wing sources uncritically while assuming the NYT will be biased. So for a sizeable portion of your readers, the post is an affirmation of their preconceptions rather than a challenge to it. Which can be fine, since your posts are relatively balanced on the whole, but I can see how it rubs people the wrong way.

        • Whatever Happened To Anonymous says:

          Based on comments to other posts, I’d guess that at least as many active readers do the opposite

          I mean, based on comments, I don’t think a lot of people here read Reason at all, let alone uncritically.

          • Protagoras says:

            Really? I read Reason. Obviously, I sometimes disagree with their libertarian slant, and like most libertarians they seem to criticize the Democrats more harshly while being more inclined to be forgiving of Republicans than their respective policies seem to justify, but there are some issues where they are one of the few good sources (e.g. regarding sex work, or drug policy).

        • Cerebral Paul Z. says:

          The day the NYT abandons objectivity as a goal and becomes an out-and-proud advocacy journal like Breitbart and Reason, is the day we’ll start judging it by the same standard as Breitbart and Reason.

          • Cypren says:

            I think this is probably the most salient point in this discussion: Breitbart doesn’t pretend to be objective, and doesn’t enjoy credibility based on the social presumption that it is thorough and even-handed. The NY Times and the Washington Post do, which makes it far more damaging when they selectively distort the facts to mislead their audience.

            It’s as much (or more) about hypocrisy as about comparative standards of journalism.

          • HeelBearCub says:

            Is there anyone definitely being claimed as on the right who rises to the standard of The NY Times or The Washington Post?

            Maybe The Wall St Journal, but they are are mostly right leaning on the editorial pages.

            It seems relevant in that you can then sort of “No True Scotsman” any right wing source, and smear all purportedly objective sources as “left wing”.

            Basically you are saying “we are better because we don’t even try to be objective”…

          • Randy M says:

            Basically you are saying “we are better because we don’t even try to be objective”…

            Of course, that depends on whether you think they are trying. Change “try” to “falsely claim to be” and it is a more defensible statement.

            It’s the difference between making a case for something*, and trying to subtly influence people without their knowing it.

            *objectivity and honesty are of course not synonymous, though it’s hard to determine the source of an error, succumbing to bias is different from hiding the truth ethically, imo, even if not consequentially different.

          • The Nybbler says:

            @HBC
            None of the right-wing sources, except the Wall Street Journal, have been taken seriously as “objective” by the non-right-wing mainstream. If you’re already laughing at Fox News’s claim to be “fair and balanced” (and it is laughable), showing their bias on a given story doesn’t really add much. Basically they never held the territory which is now being denied the NYT and Washington Post.

          • Cerebral Paul Z. says:

            The demographics of mainstream journalism being what they are, I’d be surprised to learn of the existence of a news outlet committed to objective reporting, but with an almost entirely right-wing staff whose preconceptions kept filtering unbidden into their content. I don’t view this as very promising raw material for tribal boo-them-hurrah-us stories, as it arises from a fact about journalists, not a fact about left-wing or right-wing people.

          • James Miller says:

            “Is there anyone definitely being claimed as on the right who rises to the standard of The NY Times or The Washington Post?”

            Fox News’ Special Report with Bret Baier. Right based on topics picked, but otherwise objective.

          • ashlael says:

            I mean, Fox News in general emblazons itself as being “Fair and Balanced” so it’s pretty fair to critique it when it’s not.

            I don’t think Scott watches much Fox News though.

          • TenMinute says:

            The New York Times today published the newspaper’s opinion on the front page. The last time it did that was yesterday.

            –James Taranto

    • AnonEEmous says:

      a talent for recognizing internet sarcasm goes a long way it seems

    • Deiseach says:

      Trump manipulated journalism to serve as icing on the Faux News cake in order to sucker tens of millions of uneducated, flatheaded Amurcans into jumping on his bandwagon… like so many chimpanzees piling onto the fruit cart.

      What a lovely way to describe many of your fellow-humans. Ah, the kind and egalitarian instincts of the persons of inclusiveness! Treating everyone as a human being with innate dignity, working to end hateful speech, rising above invective and stereotypes to create a better, more just and more humane world! Oh, but I thought using slurs about intellectual disability (“flat-headed”) was a no-no in the Nice New World, as this was violating political correctness which means nothing more than speaking of and treating people as they would wish to be treated?

      Now, if that sarcasm is heavy-handed, please point out to me anywhere in any comments where I have compared Clinton voters to snakes, reptiles, cretins or the like. I really hope I have not, but if I have done, I want it shoved in my face. I personally dislike Hillary Clinton which I acknowledge is nothing but irrational bias, as she just rubs me up the wrong way. I can see why people would vote Democrat, I can even see why they would vote for her. I think some (but not all) on the left have particular agendas they are pushing hard with which I disagree and even think are not just wrong, they are bad, and they are using the Democratic Party as their vehicle because they have no other means of affecting mainstream politics.

      But I really, really hope I have never dismissed them as sub-human, idiots, or evil for the sake of being evil. You can think Trump is worse than Hitler and Stalin combined, you can think there was a toxic element to his vote, you can think that those who voted for him were turkeys voting for Christmas. But please refrain from terms such as “flat-headed” and “chimpanzees”.

      Some of those “flatheaded Amurcans” who voted for Trump were African-American and other minorities (8% of the African-American vote, 29% of the Latino vote, 29% of the Asian vote, 37% “other”). Do you really want to go the route of comparing them to chimpanzees? That does not lead to a good result.

      As for the “uneducated”? Yes, he won the greater share of those without a college degree. If not having a BA or BSc makes you uneducated, then I’m uneducated. Interestingly, amongst white voters who have a college degree, Trump beat Clinton to vote share by 49% to 45%. But I suppose that’s down to plain old racism, right? So even the magic letters after your name do not insulate you from being a terrible person?

      In the 2016 election, a wide gap in presidential preferences emerged between those with and without a college degree. College graduates backed Clinton by a 9-point margin (52%-43%), while those without a college degree backed Trump 52%-44%. This is by far the widest gap in support among college graduates and non-college graduates in exit polls dating back to 1980. For example, in 2012, there was hardly any difference between the two groups: College graduates backed Obama over Romney by 50%-48%, and those without a college degree also supported Obama 51%-47%.

      Among whites, Trump won an overwhelming share of those without a college degree; and among white college graduates – a group that many identified as key for a potential Clinton victory – Trump outperformed Clinton by a narrow 4-point margin.

      Trump’s margin among whites without a college degree is the largest among any candidate in exit polls since 1980. Two-thirds (67%) of non-college whites backed Trump, compared with just 28% who supported Clinton, resulting in a 39-point advantage for Trump among this group. In 2012 and 2008, non-college whites also preferred the Republican over the Democratic candidate but by less one-sided margins (61%-36% and 58%-40%, respectively).

      Trump won whites with a college degree 49% to 45%. In 2012, Romney won college whites by a somewhat wider margin in 2012 (56%-42%). Trump’s advantage among this group is the same as John McCain’s margin in 2008 (51%-47%).

      • harland0 says:

        Of everything I wrote that’s what you had a problem with? None of the other commenters even mentioned it. I apologize about the chimpanzees, that was a poor choice of animal. None of which changes the fact that Trump is locking the press out of the White House!

        • Deiseach says:

          Of everything I wrote that’s what you had a problem with?

          Yes.

          Is Trump locking the press out? I don’t know and you know what, I don’t particularly care. Political commentators are too often up the arse of any particular administration because they rely on cosy insider relationships to get the story before their colleagues, and that means they’re a tool for the administration to do some careful leaking when it wants to put spin on something, boost one politician and do down another, damage control, and the rest.

          It’s of as much concern to me as Alex Ferguson refusing to talk to the BBC. In the same way, this little huff will blow over and it’ll be business as usual.

          Dismissing people’s views with “they’re sub-human animals, we don’t need to take them seriously” – that, however, is more important than “are a bunch of journalists upset over not being treated with the gravity they think the Fourth Estate deserves?”

        • Spookykou says:

          None of which changes the fact that Trump is locking the press out of the White House!

          You keep making this claim, but a few moments ago on NPR they were talking about it as a possibility, is there official word from the Trump team that they will be shutting down the White house press core?

          Edit: Also I agree with Deiseach about everything in the above, and I really wish that your condemnation of all Trump voters would make me more likely to believe that you are a troll, but it can’t.

          • gbdub says:

            “Shutting down the White House press corps” is not being considered. They are talking about potentially moving the official presidential press briefing room from its current location in the White House proper to a different spot, possibly in the executive office wing rather than the White House proper.

            The significance is entirely symbolic, just as it was when Nixon put the room in its current spot. Part of the reason the press is grumbling is because the new room would likely be larger – probably making their “White House Reporter” clique a little less exclusive.

          • Deiseach says:

            gbdub, your mention that Nixon had moved the briefing room to its current location made me look it up, and it has a short but very interesting article on Wikipedia. It used to be Franklin D. Roosevelt’s indoor swimming pool!

            I can see why the press corps would grouse – currently it’s located between the press staff offices and press corps offices, so they’d probably have to traipse out of their nice comfortable office (if it is nice and comfortable) all the way to a different part of the building or a different building altogether, which would be inconvenient.

            On the other hand, it has only been there since 1969 so it’s not from time immemorial, it got renovated in 2005 and even then it’s still only one seat bigger than before.

            I imagine their main gripe, if they got shifted, is that they could no longer say “yeah, I work in The West Wing” with the associated aura of cool and glamour that comes with that. Aaron Sorkin has a lot to answer for 🙂

          • Dr Dealgood says:

            This might sound like a cruel joke, but I’m genuinely confused.

            What in the world did FDR need a swimming pool for?

          • The Nybbler says:

            Evidently, despite his paralysis, he could swim:

            http://www.whitehousemuseum.org/west-wing/press-briefing-room.htm

          • Deiseach says:

            What in the world did FDR need a swimming pool for?

            Swimming is low-impact exercise and the water supports you so even if you have limited mobility, it’s much easier and less wear on the body while it does have the benefits of getting heart rate up etc.

          • Dr Dealgood says:

            Ah, ok. I guess that makes sense.

            Carry on then.

        • YehoshuaK says:

          To the question about FDR and swimming, I just want to chime in based on my personal experience. My father, of blessed memory, was paraplegic from the age of five years due to polio. He was also a strong swimmer–I fondly remember going swimming with him on many occasions.

    • James Miller says:

      If you are going to be an anti-Trump consequentialist you really shouldn’t write comments on blog posts that insult Trump voters and so make it less likely that past Trump voters (such as myself) will vote against Trump when he runs for re-election. Seriously, I voted for Trump in the general election but I didn’t donate any money despite the numerous letters I received from his campaign asking for funds. The enormous hatred and ridicule much of the left has heaped on Trump voters since the election has made me emotionally regret not having given his campaign any money.

      • Earthly Knight says:

        As a retributivist, it warms the cockles of my heart to see Trump supporters meet with well-deserved mockery and ridicule, and I encourage people to insult every Trump supporter they encounter at every opportunity. If you have trouble seeing why, look no further than your belief that having your feelings hurt is a good reason to put an emotionally unstable sexual predator in charge of the launch codes.

        • FeepingCreature says:

          Spoken as a retributivist, I hope you suffer.
          Wow this feels weird. I don’t usually want people to suffer.
          I think it’s because you explicitly want people to suffer. Somehow that feels like it makes it justified..?

          • Earthly Knight says:

            Man, I was so close to voting for [your favored candidate] in 2020, but since you were mean to me now I think I’m going to have to vote for Charles Manson instead. It’s your fault, you made me do it!

        • Deiseach says:

          Earthly Knight, I would seriously rather have “an emotionally unstable sexual predator in charge of the launch codes” than Hillary Clinton, because I genuinely think she’s more likely to push the button than Trump. With Libya, she’s shown she’s quite prepared to “bomb the bastards” and with the (to me still mind-boggling) Democrat/Democrat-supporting attitude about getting tough with Russia and standing up to Putin, and her own personal fragility about criticism or dissent with her views, I see no reason she wouldn’t get into a pissing contest over America’s role as the world’s policeman and use the argument of the ultimate sanction – I’ve got the launch codes – as a bluff or threat.

          What is it, by the way, with you Americans and this mania about “the launch codes”? Is that really the first or main thing on your minds – hey, we could start the Third World War and even better make it nuclear? Woo-hoo, go us!

          Do you really wonder why the rest of the world looks at you funny sometimes?

        • Iain says:

          @Earthly Knight: While this may be emotionally satisfying for you, it is counterproductive, and I wish you would cut back. It is entirely fair to point out cases where Trump supporters have surrendered their right to criticize (“Obama is a celebrity president! Hillary Clinton has conflicts of interest!”), but your ridicule is more persuasive in smaller doses. You are correct to say that having your feelings hurt does not justify a Trump vote; it does not logically follow, however, that a Trump vote justifies having your feelings hurt.

          • Dr Dealgood says:

            I agree with you that Earth Knight’s insistence on this is counterproductive, but your characterization of the central question as relating to “hurt feelings” is (more subtly) counterproductive as well.

            Trump voters are adults and, like most people in the world, brush off much more insulting crap every day than this. So called “White Fragility” aside we’re really not that thin-skinned. The reason the dismissive insults are a problem isn’t some nebulous emotional harm but because of what they signal about the intentions of those making them.

            To use a hyperbolic but (hopefully) illustrative example: when Marie Antoinette allegedly said “let them eat cake,” the people who were enraged weren’t mad because they had their feelings hurt. They were mad because they were starving and she, someone with the power and responsibility to help, reacted with callous indifference. The problem wasn’t the insult itself but rather that the insult showed that the royalty was willing to let them die in the streets rather than lift a finger to help.

            The issue isn’t that the white working class are upset at being called rednecks. The issue is that white working class communities are collapsing, have been collapsing for decades, and the response on the part of America’s elites is callous indifference. That’s why the insults matter: because they are that much more evidence that no help will be forthcoming.

          • Earthly Knight says:

            and the response on the part of America’s elites is callous indifference.

            The response of America’s left was to pursue policies that improved the welfare of all, including the white working class. It’s only when these efforts are met with “but Obama is a Muslim Kenyan who’s gonna take away our guns!” that the attitude turns towards callous indifference. I will continue to support policies that will be to the actual benefit of white, working-class communities, but that doesn’t mean I’m going to be nice to people who let fear and racism rule their rational self-interest.

          • Iain says:

            @Dr Dealgood: Sure. I was just borrowing the language that was already in use.

            I agree with the part of Earthly Knight’s response that focuses on the actual concrete things that Democrats have done for the white working class. Millions of previously uninsured Trump voters gained healthcare coverage as a result of the ACA, and stand to lose that coverage if the Republicans have their way. Nobody has a good answer to the collapse of working class communities, but it’s not as if the Democrats haven’t been trying. Obama quietly did a ton of stuff to prop up manufacturing. I will remind you that questions about race were a much better predictor of Trump support than questions about the economy.

            Like, what practical help do you think the white working class can expect from Trump that it could not expect from Clinton?

          • Dr Dealgood says:

            @Earthly Knight,

            Who do you think has a better grasp a man’s self-interest: the man himself, or someone who claims to speak on his behalf?

            You certainly wouldn’t accept that reasoning if it was suggested that, say, blacks in Chicago were voting against their self-interest by electing Democrats who support gun control and oppose policing.

            @Iain,

            Like, what practical help do you think the white working class can expect from Trump that it could not expect from Clinton?

            Enforcement of laws against illegal immigration, including deportation of illegals currently in the country and the construction of physical barriers to impede their return. That and a tightening of visas to counteract the artificially low wages caused by imported workers.

            A trade policy which exploits the US’s global hegemony to establish a more favorable balance of trade with other countries. Including tariffs to eliminate the advantage of manufacturing overseas using cheap foreign labor, as well as incentives for multinational corporations to build new plants in the US.

          • A trade policy which exploits the US’s global hegemony to establish a more favorable balance of trade with other countries.

            A “favorable balance of trade” normally means a balance of payment surplus. A balance of payments surplus means that other countries are getting more stuff from us than we are getting from them.

            Why do you regard that as a good thing?

          • Earthly Knight says:

            @ Dr Dealgood

            Who do you think has a better grasp a man’s self-interest: the man himself, or someone who claims to speak on his behalf?

            We know from surveys that the overwhelming majority of Republicans are profoundly deluded, believing Obama to be a Muslim, pizza joints to be fronts for satanic pedophile rings and so on. If these people had any idea what was in their best interests they would be seeking psychotherapy rather than voting.

            You certainly wouldn’t accept that reasoning if it was suggested that, say, blacks in Chicago were voting against their self-interest by electing Democrats who support gun control and oppose policing.

            Your claim that Chicago Democrats in general “oppose policing” is utter nonsense, so I have nothing to say about that. My view on gun control laws (which in no way depends on what people of any color believe) is that they are warranted only if there’s strong evidence that passing them would cause a significant reduction in violence.

        • Cerebral Paul Z. says:

          I don’t understand. Do launch codes have grabbable pussies?

        • I am curious. Do you disagree with the factual claim James is implying, that the sort of behavior you have just endorsed makes Trump more likely to be reelected?

          • Earthly Knight says:

            I have no idea whether it’s true that insulting Trump supporters makes them more likely to vote for Trump, and I doubt anyone does. It also strikes me as manipulative to threaten to vote for a corrupt, incompetent clown unless people behave the way you want them to. Something like a suicide threat, only this time the blade is at the nation’s throat. The proper way to get other people to be kind to you is to deserve it.

        • nydwracu says:

          I don’t believe that having my feelings hurt is a good reason to put an emotionally unstable sexual predator in charge of the launch codes, but I do believe that eight years of an ex-Communist who used to pal around with unrepentant Communist terrorists and just commuted the sentence of Oscar Lopez Rivera, another unrepentant Communist terrorist (to great fanfare from the mayor of a city that Rivera’s terror cell killed people in), and who was elected by a political faction that believes in riots and purges, fantasizes about mass murder, willingly stocks its quasi-affiliated apparatus of fellow-travelers with no-shit Communists, tries to elevate the sort of scum that comes right out and says that one of their political goals is for there to be no more white people to hero status, and holds the rest of the country in frequently exterminationist contempt is eight years too many.

          It’s important to note that this shit is mostly just an election tactic. They want their base to hate us because they think it’ll drive turnout. Oops! They lost. But they’re probably too institutionally committed to it to stop now. These things can’t turn on a dime.

        • Scott Alexander says:

          Earthly Knight is banned for this and several other comments.

    • Alex Zavoluk says:

      A lot of people are treating this post as literally true, but I’m pretty sure it’s sarcasm.

      • Deiseach says:

        A lot of people are treating this post as literally true, but I’m pretty sure it’s sarcasm.

        I’d hope, but I’m seeing stuff like this elsewhere:

        (a) Re the subject of that BuzzFeed article (and yes, they’re taking BuzzFeed as The One Word of Truth)

        this is going to be long, but trust me, it’s worth reading. i’m not joking, parts of it sounds like it could be taken straight out of a detective novel, but it’s unfortunately very real, and while this is likely just the tip of the iceberg, it’s already shaping up the be the greatest political scandal in US history

        *there follows a list of the main events in the timeline, then it concludes*

        – Steele had nothing to gain by spreading false information
        – the former m16 agent is now currently in hiding, fearing for his life after he was outed as the agent behind the report

        (b) This in all seriousness about the inauguration (different poster from the one who posted the above)

        anyway a fun thing about this week is listening to all my non-white coworkers making each other promise to “stay safe” this week which means “don’t leave your house”

        i mean my white coworkers and i will also not be leaving our houses but not because we would be in IMMEDIATE FUCKING PHYSICAL DANGER from essentially a KLAN RALLY ON THE NATIONAL MALL!!!!!!!!!!

        So kinda hard to tell “comparison of Trump voters with chimpanzees – sarcasm or meant literally?” this weather 🙁

        • Alex Zavoluk says:

          It’s kind of hard to describe why I don’t think it’s real, but the density of jargon and the emphasis on poor-sounding arguments makes it sound more like sarcasm or perhaps someone trying to make the other side sound bad.

        • nydwracu says:

          i mean my white coworkers and i will also not be leaving our houses but not because we would be in IMMEDIATE FUCKING PHYSICAL DANGER from essentially a KLAN RALLY ON THE NATIONAL MALL!!!!!!!!!!

          You know, I really haven’t been able to feel any sympathy for that line of argument ever since, during the height of the BLM shit, a random black guy tried to murder me out of nowhere.

          (Sure, I was out after sunset, but we have a word for trying to kill people for being in the ‘wrong’ neighborhood after sunset, and it’s, um, ‘lynching’. Place was <10% black, even! The fuck?)

      • Urstoff says:

        The charitable reading is sarcasm, anyway.

        • Tekhno says:

          I think a lot of rationalists tend to forget just how many deeply deeply stupid people exist on this planet.

    • Wrong Species says:

      When people talk like this, it usually means they feel defeated but are trying to distract us with non-sequitors. Since someone like harland0 is almost certainly never going to admit they’re wrong, this is about the closest we can get to surrender. Enjoy it!

  4. J says:

    Hooray for empiricism! In honor of you and Vikram, my friends and I have just pooled our funds and donated $200 to Doctors Without Borders.

    • Michael Cohen says:

      Yay philanthropy! I’m curious–how did this experiment inspire your donation?

      • J says:

        Scott and Vikram risked time and money to celebrate one of our group’s shared values. Effective altruism is a related value, and we wanted to share in the celebration.

        • J says:

          I should say too that Hanne Watkins and Sripada deserve just as much credit for doing the hard work it took to run the experiment.

  5. skybrian says:

    This seems like a promising start. It would be cool if someone routinely tested news articles this way. (It’s similar to reading comprehension questions on exams, except that we test the articles for reading comprehension rather than the readers.)

    I wonder, though: what sort of population sample do you get from MTurk? Is it possible they got mostly Democrats? Since this is a hot political issue, how much does motivated reasoning affect reading comprehension?

    Would Google Surveys work better than MTurk for getting a randomish sample?

    • MawBTS says:

      MTurk’s just an MMO with bad graphics. Nobody has the time to put a political “spin” on things, everyone’s frantically grinding thousands of HITs while a few dollars trickle into their accounts.

      If MTurk’s userbase is slanted towards Democrats it would fit NYT’s readership, which is also slanted towards Democrats. A bigger concern would be whether they have lots of Republicans (not that there’s any reason to think that).

      • skybrian says:

        I suspect Republicans would be more likely to be skeptical of the New York Times and pick (c). But who knows, it could be the opposite: they’d expect the New York Times to have an anti-voucher bias and pick (b).

        Or maybe everyone would be more likely to find an anti-voucher argument in the New York Times than in another publication, due to their priors about the New York Times? Which brings up another issue: were the survey takers told where the article was from? What happens if you do the opposite?

        (There are probably all sorts of interesting ways to tinker with the details of survey methodology. I would be surprised if there were a way to reverse the results, but not surprised if there were a way to make it less one-sided.)

      • Randy M says:

        MTurk’s just an MMO with bad graphics. Nobody has the time to put a political “spin” on things, everyone’s frantically grinding thousands of HITs while a few dollars trickle into their accounts.

        This does make it seem like it’s biased towards those with few options or ambitions.

      • Am I the only person here who sees “M’Turk” as “member of Stalky an Co.”?

  6. afeller08@gmail.com says:

    I think you need to retract mistake number 11 now, as the evidence shows that your original reading was fair.

  7. educationrealist says:

    According to this article:
    a) Most economists support privatizing education
    b) A surprisingly large number of economists are uncertain about the benefits of privatizing education
    c) Economists are split on the topic of privatizing education, and so the Times is inexplicably devoting a huge amount of space to something that isn’t unusual or even noteworthy.

    The correct answer is b. You are just having trouble figuring out why it isn’t c.

    • RicardoCruz says:

      Indeed. It would be interesting to test how robust the findings are with regard to verbiage.

      My bet is on them being very robust. Anyone wants to try it?

    • Cliff says:

      There is nothing surprising about the number of uncertain economists at all. If you look at other surveys (not Uber) results are usually very mixed. Vouchers actually enjoy quite a high level of support relative to other things, which is even more clear in the updated survey they performed.

  8. Deiseach says:

    I don’t know Noah Smith and I haven’t a dog in this fight (except that I agree with Scott the article was misleading) so I’m not trying to bash anybody. I’m Irish, I don’t have a particular opinion on school vouchers or is this privatising education (we’re still discussing divestment of Catholic church control over schools) so I’m not for or against any side in this question and I’m not trying to ‘prove’ my side ‘won’.

    Look, I’m your average idiot who skims through the news first thing in the morning to see (a) what’s happening in the world (b) the local death notices (c) anything interesting to read on break.

    I imagine (and this is probably something else that needs empirical testing) that a lot of people get their news not from the traditional dead-tree issue of the paper but online, whether via directly visiting the paper’s website, something like Google News, or an RSS feed. It’s a lot more convenient (I won’t say “easier”) to read on your mobile device while you’re on the train or the bus than struggling with a physical copy of a newspaper, and the same goes for your tea-break at work where you look something up on the PC.

    That means most people will only read the headline. If it’s on something that interests them, they’ll probably then read the full article. But if (a) you need to read the full article (b) you then need to clicky on the linkies to go back to the source data (c) how good are your graph reading skills? let’s find out!, then I feel quite confident in saying, as your average idiot in the street, that’s more effort than most people will put in.

    So even if Noah Smith’s argument is “yes, the headline is misleading but if you read the article and parse the data, you’ll think differently”, it still can happen that most people will read the headline, form the opinion “Huh, economists think school vouchers are a bad idea” and then go on to regurgitate that in discussion (“It’s true, I read it in the Times!”)

    • RicardoCruz says:

      Even if you read the article, it is misleading. Scott in his bet proposed hiding the headline and only showing the text. Too bad nobody took him up on it.

    • nydwracu says:

      Another fun thing about that is that it’s common practice for headlines to be written by newspaper staff — and even if it’s a staff article, the headline may not be written by the author.

      A lot of places are A/B testing headlines to optimize for clicks nowadays, even.

  9. xq says:

    I think it’s important to do a control experiment here. Find an article that makes an argument you think is somewhat nuanced but non-misleading, then poll MTurk users to see if they get the nuance.

    Edit: Better idea. Let’s adjust the article wording to answer Scott’s objections, then poll MTurk users with the same questions Sripada and Watkins asked. I’m willing to pay for the survey if people here will predict how that will change the response.

  10. Freddie deBoer says:

    As is so often the case with your stuff, I find myself saying regarding the original piece “needs more vulgar Marxism.” Not Marxism Marxism, that is, but vulgar Marxism: the application of a simple presumption that most human affairs are influenced by the interest of the moneyed and powerful to maintain and expand that grasp on money and power.

    Incidentally, whatever advantage privatization might appear to have would be the same source of perceived advantage in almost any educational metric: dominantly the product of selection bias.

    • Scott Alexander says:

      As in the article writer, being part of the educational establishment, wants to maintain the power of that establishment by opposing vouchers? Or the economists, being part of the corporate establishment, want to maintain that establishment by supporting vouchers? Or both? Or something else?

      • suntzuanime says:

        You, being part of the new media, want to expand your grasp by kicking the failing and foolish New York Times while it’s down.

        • cassander says:

          If the old media is as bad as is claimed, though, kicking it while it’s down is a virtuous act regardless of self interest.

          • Deiseach says:

            Got the usual begging email about taking up a subscription today (honestly, you register to read one article and they never let it go) from the New York Times.

            I mention it in light of what this discussion is about.

            Subject header: The truth is what we do better.

            Body of email: (in honking big bold font) Truth. It demands your support. (details of subscription offer followed).

            Well, I laughed anyway 🙂

        • Tekhno says:

          Heh. The problem with (even) vulgar Marxism is that it presumes that the rich and powerful have class consciousness and aren’t wildly in disagreement and don’t form into various factions to protect sectoral monied interests, rather than the supposed unified bourgeois interest. If false consciousness can affect the proletariat, it can surely affect the bourgeoisie.

          • Protagoras says:

            The bourgeoisie is smaller, and has more resources for keeping track of one another. Also, while they do betray one another to advance themselves, betraying one another to help the poor and weak is a risky proposition (they usually can’t help you much; not saying it is never a strategy used by some of the wealthy and powerful, as clearly it is, but just that it is risky, and therefore not something that’s constantly happening). On the other hand, the wealthy and powerful can easily reward those poor and weak who betray one another to help the wealthy and powerful, so that sort of thing does happen constantly.

            Honestly, I’m not sure how someone could possibly look at history, or, say, the writings of upper class people, and think that the rich and powerful don’t have class consciousness.

          • cassander says:

            @Protagoras

            >Also, while they do betray one another to advance themselves, betraying one another to help the poor and weak is a risky proposition (they usually can’t help you much; not saying it is never a strategy used by some of the wealthy and powerful, as clearly it is, but just that it is risky, and therefore not something that’s constantly happening

            Individually they don’t help you much, but there are a lot of them. the political coalition “hi-low against the middle” has a long, long, history.

            Honestly, I’m not sure how someone could possibly look at history, or, say, the writings of upper class people, and think that the rich and powerful do have class consciousness.

    • Sluggish says:

      Incidentally, whatever advantage privatization might appear to have would be the same source of perceived advantage in almost any educational metric: dominantly the product of selection bias.

      This seems like an interesting thing for someone in standardised testing to say. Is the implication that testing can only be used to compare students, and cannot be used to compare institutions? I would have expected the comparison to be something like ‘change in test score per year per dollar’, which would (potentially?) offset selection bias.

  11. MartMart says:

    A few days after the original post, I saw this
    http://www.npr.org/sections/health-shots/2017/01/06/508379529/poll-most-americans-say-dont-repeal-obamacare-without-a-replacement

    Which seemed to me to be a very similar sort of misleading headline.

    • andrewflicker says:

      It’s *slightly* less misleading, since it’s grouping two “pro-national-healthcare-policy” positions together with only one “let the free market figure it out” opposite, instead of grouping “uncertain” with “against”. But yeah, still a shitty headline.

      • MartMart says:

        I’m torn between saying that it’s slightly less misleading for the reasons you stated, and saying that it’s even more dishonest, because the conclusion in the headline wasn’t even close to being a question on the poll.

    • phoniel says:

      I don’t think this is dishonest at all. It just makes the presumption that people who don’t want to repeal Obamacare also don’t want to repeal it without a replacement. That seems like a perfectly safe assumption to me.

      This headline, in other words, is on par with taking a survey which reports 33% severe disapproval, 33% disapproval, and 33% approval, and reports it as “66% of Americans disapprove.”

      • MartMart says:

        58% Think its important to repeal the law vs 31% who says it should not be repealed.

        65% believe its important to reduce government role in healthcare vs 20% who think that it should not be reduced.

        Its the “not too important” which are the equivalent of “unsure” here.
        There is not question that asks “Do you favor repealing the law without a replacement?”, instead they are asking “Is it important to repeal the law?” and something similar to “Can you think of something health related that is more important?” and then saying that if you think there is something more important then clearly you don’t think we should repeal it without a replacement.

    • HeelBearCub says:

      You are incorrectly not reading the actual results and assuming the questions about priorities were how they judged the repeal/replace question.

      Here is a link to the graph in the poll results that shows the following:
      * No, should not vote repeal – 47%
      * Wait to vote repeal the law until the details of a replacement plan have been announced – 28%
      * Vote to repeal the law immediately and work out the details of a replacement plan later – 20%
      * Don’t know/Refused – 5%

      • MartMart says:

        Wow, that’s a pretty big mistake on my part. I went by the only graphic in the article, without digging in further. Thanks for setting me straight.

        • HeelBearCub says:

          Thank you for acknowledging this.

          Maybe that sounds conceited? I hope not. I mean it genuinely.

          ETA: and to be fair, the placement of the graphic in the NPR copy is confusing. I think that can be a peril of looking at a transcript of what is meant to be audio. They aren’t really running a stand alone website.

          • MartMart says:

            Its hard to read the tone of a comment, or at least it is to me. You didn’t sound in anyway conceited, you were absolutely correct. I saw a news article in my feed, scrolled thru, saw the graphic, said “hey, that’s not what the headlines said” and proceeded to post it. I assumed I was seing the spread of a new meme (if thats the right word) of using magic words to claim evidence in support of position which the evidence contradicts. You dug a little deeper, and showed me what I should have seen. I’m not sure what the proper etiquette is for acknowledging that I made a mistake, so I thank you for correcting me, and will try not to make that mistake in the future.

  12. Nathan Taylor (praxtime) says:

    If your readers willing to do this kind of MTurk surveys, definitely take them up on it more in 2017. This is a great result.

    Always so incredibly difficult to change people’s minds. And a reader survey is a powerful tool for empirically making the point an article has an incorrect slant. I suspect Noah Smith finds it convincing whether he comments any more or not. To be clear, not picking on Smith. In fact the opposite. He’s exactly the kind of person who would change his mind with this kind of survey data. So keep this in your bag of tricks. For example your Trump article on crying wolf would have been even stronger if you had picked a couple of points on media slant/widespread misunderstanding to do a survey on. In fact you could still do that if a Trump or other related topic comes up in 2017.

  13. phil says:

    Scott, I’m curious as to whether you have any impression as to the intent of the original NYTs article writing (intentionally leaving the name out)

    lets concede that the article was confusing to reader (which doesn’t seem like much of a concession, its seems basically proved at this point)

    Noah’s piece, and it seems like most of the critics of your original piece, are sort of working from a place where they would concede that it was a piece that was likely to confuse readers, but won’t concede mal-intent on the writers part

    which I think I actually buy, and I think leads to more interesting questions

    ————
    ————

    which is mainly, are there systemic things that the NYTs (and the rest of the media does) that make these sorts of reader confusion situations likely

    couple possibilities

    1. The really obvious, is the pervasive practice of making the title as click-baity as possible. Which is a really insidious practice that you see every, for really obvious reasons, and relative to what you see everyday on Slate, the title here doesn’t even seem that terrible an offense

    I don’t know how to make headlines better, other than to hold the publication as a whole entity accountable for them, I suspect that problem isn’t going away though

    2. The what I would categorize as boring observation that NYTs writers and editors probably form their own little groupthink bubble where things that fit their worldview don’t catch their eye as off, but things that don’t fit their worldview do, such that their mistakes tend to all be in the same direction

    3. Sailer has done a few post contrasting the NYTs fairly dry style with the Daily Mail, sample: http://www.unz.com/isteve/the-congos-small-little-mini-war-the-nyts-fear-of-admitting-an-interest-in-human-biodiversity/

    “One of the curious aspects of New York Times articles is that they are often organized in the reverse order of how the same material would be reported in, say, the Daily Mail. NYT articles tend to start off boring and depressing, with only vague hints of why the reporter is interested in the subject, and don’t get to the good stuff until late in the article, by which point, no doubt, most readers have given up.”

    An interesting question is whether this style systemically leads to the sort of reader miscomprehension at issue here

  14. Moon says:

    Has anyone surveyed educators yet to find out what changes they would like to make in the field of economics. Makes just as much sense to do vice versa here. Then have someone write an article on the survey, and ask people whether

    a) Most educators support government control of economists and economics.

    b) Most educators oppose government control of economists and economics.

    c) Unsure / the article doesn’t say

    Then see whether people’s beliefs about the article correspond to what the article says.

    Why anyone should care what economists think about school vouchers. Most of them have no experience with working in school systems, and little understanding of how school vouchers might affect student performance.

    And as if that doesn’t disqualify economists enough from being any kinds of experts on school vouchers, there is another obvious problem. Economics is very politically related. The Kochs have bought whole university economics departments, in return for strings attached which dictate that only Right Wing economics will be taught in those programs.

    Economists’ views on school vouchers, rather than being any expert opinion, are very likely to simply reflect their preferred political ideology. And since there is so much more money to be made from the Kochs and their ilk, by being Right Wing, than by being a Center or Left of Center economist, I would be very surprised if most economists were not Right Wing in their political and economic views. Of course Right Wingers support privatization. Duh…… They probably support privatization of the air and water and National Parks too, just like all crony capitalist welfare queens do.

    • Urstoff says:

      The Kochs have bought whole university economics departments, in return for strings attached which dictate that only Right Wing economics will be taught in those programs.

      *citation needed*

    • Deiseach says:

      They probably support privatization of the air and water and National Parks too, just like all crony capitalist welfare queens do.

      I was under the impression water was already privatised, even in the People’s Democratic Socialist Republic of California? Before you gallop off wildly in all directions, please make sure your saddle girths are fastened first, as falling off your high horse can hurt.

      Moon, you could actually have decent points to argue, if you just took a calming breath and dialled back on ‘the Right Wing control everything and everything to do with the Right Wing is evil’ rhetoric. I think by now we can take it as a given that that is your position.

  15. Spookykou says:

    I think that Scott should continue this line of experiment to his own persuasive essays to make sure that their persuasiveness is properly calibrated to his confidence in his own opinion.

    Similar to the Buzzfeed post recently, I think this method could be used to determine how much hedging is needed such that Scott is not misleading his readership with regard to growth mindset, etc.

    • Randy M says:

      Scott has experimented with such before, recall the ai persuasion surveys. Although that was a case of trying to calibrate his pitch on an opinion piece rather than trying to make sure he communicated facts clearly.

      • Spookykou says:

        So, I started thinking about this after the Buzzfeed essay, where HBC made a comment about how Scott’s Growth Mindset essay was not meaningfully different from the Buzzfeed article in terms of the case it made against Growth Mindset, because a few throw away sentences hedging his bets and saying that this is all confusing, is not enough. Readers of both will ultimately leave with a similar impression that Growth Mindset is bunk.

        Now personally this isn’t the case, I really appreciate the extent to which Scott hedges himself, and I take it into consideration while reading what he writes(and I think is part of why I never like any of the other writers that people here link to, even though, in theory they should fit into my cluster of interests). On the flip side, maybe HBC puts considerable less stock in hedging so the essays look the same to them.

        If more people are like HBC, then the amount of hedging Scott currently does is not enough(maybe no amount ever will be?) and Scott’s essays have similar problems to the problems he points out about the Buzzfeed article, in terms of being too convincing on issues that are not actually very settled.

        • Randy M says:

          You can’t account for everyone’s priors at the same time. In other words, the amount of hedging needed to convince someone with strong pro-fuscia* views that his piece criticizing the fuscia party spokesman isn’t meant to undermine them entirely will probably end up having the opposite effect on pro-taupe** people.

          *It’s hard to pick hypothetical colors that stand for no real world political movement.
          **I think this is a thing? I’ve heard people say it before, really.

          • MartMart says:

            In some of his writings, Scott experimented with naming ideas after lovecraftian monsters. This loads ideas with negative karma and becomes virtually impossible to support.
            I feel the original pro-growth piece pulled a similar trick. Here is a mountain of evidence, it’s all really good, well most of it anyway, but maybe there are alternative explanations, and how about that diabolical devil.
            At first it appears to be the sort of injection of humour that makes scott so fun to read, but at the same time it’s loading everything with negativity, because supporters of growth mindset are satanists and all.

          • Spookykou says:

            I am having a hard time conveying the idea properly, I agree that confirmation bias would be a problem. I guess I am wondering if it would be possible to structure a survey, like the main post here, only for any given persuasive essay.

            I have also heard of taupe.

          • Nornagest says:

            Taupe might stand for the ideology of the Next Generation-era Star Trek shows, since they liked to paint everything in it.

          • Randy M says:

            At first it appears to be the sort of injection of humour that makes scott so fun to read, but at the same time it’s loading everything with negativity, because supporters of growth mindset are satanists and all.

            This is probably a valid point. I’d even go so far as to say Scott, out of charity, would likely avoid such metaphors for, say, Trump supporters.

          • FacelessCraven says:

            “It’s hard to pick hypothetical colors that stand for no real world political movement.”

            The correct term for such hypothetical groups is “skub“.

          • suntzuanime says:

            I gotta say that I did not find the crack about “The Donald Trump Institute Of Research That Is Going To Be Absolutely Yuuuuuuge, which busies itself putting out white papers to a coalition of illiterates” to be particularly charitable, as a member of said coalition.

            To be clear, I don’t have a problem with the sort of injection of humour that makes Scott so fun to read, and I can live with it when it’s my ox being gored. But don’t pretend like it’s only your ox.

          • Fossegrimen says:

            Taupe is RGB(72, 60, 50), kind of a brownish boring colour. Not an easy shade to rally around perhaps, but entirely a thing.

        • HeelBearCub says:

          @spookykou:

          Now personally this isn’t the case, I really appreciate the extent to which Scott hedges himself

          Scott hedges himself quite a bit, sometimes.

          Other times he really does not. He nods in the direction of hedging but then proceeds to show you that he really does not believe that the hedging is meaningful.

          Which is not unique to Scott at all. Most people will write in this manner, and its always a good idea to acknowledge the counter arguments. But I think Scott is actually going past his own opinion frequently, perhaps in service to his contrarian nature.

          He should not then be surprised when he finds himself quoted as having taken a position that he didn’t realize he was taking.

          If he were to turn his “this article is awfully biasing in the impression it leaves” magnifying lens on himself (not sure this kind of thing is really possible for anyone to do) … there are times when I think he would be horrified.

          And the thing is, he frequently does it with statements like “Or did [she] really, honest-to-goodness, make a pact with the Devil in which she offered her eternal soul in exchange for spectacular study results?”

          That is supposed to be a throwaway line that is interesting and makes you laugh. It probably shouldn’t be taken seriously.

          But the effect is to clearly indicate where his intuition lies. It is, if it were taken seriously, a tacit accusation that Dweck is faking her data. He says her studies are so very good, and then says “but nobodies studies are actually that good unless they make a deal with the devil”.

          But you aren’t supposed to take it seriously. Or maybe you are. Who can say?

          • Nornagest says:

            Well, extreme hyperbole is funny when you apply it to stuff that no one in their right mind would care that much about, or in ways that don’t map to ordinary political rhetoric. Hunter S. Thompson is hilarious even if you broadly disagree with his views, because he had a real talent for absurdist caricature; your average political cartoonist on the other hand is about as funny as a cinder block, though a reader might feel a warm little glow of righteous vindication.

            I think Scott’s going for the Hunter style. If you want to communicate revulsion, and you want to be funny about it, and you don’t want your readers to mentally search-and-replace with “yay” or “boo”, then a pretty good approach is to accuse e.g. Richard Nixon of being the king of the lizard people.

          • Spookykou says:

            Which is not unique to Scott at all. Most people will write in this manner, and its always a good idea to acknowledge the counter arguments.

            I mentioned this before, but it seems to me that a lot of the blogs that get linked to here at SSC specifically don’t do this, and hold their vague impressions of humanity up as obvious and unquestioned truths, It is part of the reason that I enjoy SSC so much more than anything else on LW or The Last Psychiatrist, as some examples.

            Other times he really does not. He nods in the direction of hedging but then proceeds to show you that he really does not believe that the hedging is meaningful.

            I have also noticed this, however I think this mostly crops up in a very particular way, in which he raises objections, then also goes into the problems with those objections. When he engages in this behavior he almost always ends on a ‘and this is all very confusing’, or similar.

            To be clear, I actually think your criticism on the Buzzfeed post was pretty valid, I would imagine that for the vast majority of people the difference between ‘this is probably bunk’ and ‘this is bunk’ is not substantive enough. This is why I am interested in this idea of quantifying approximately how much/what kind of hedging is needed such that people will believe that the author actually holds/is trying to promote a hedged position after reading any given essay.

          • HeelBearCub says:

            @spookykou:

            this is probably bunk

            I’m curious. Do you think this is Scott’s actual position on growth-mindset? Is that the impression you have from reading his posts in growth mindset?

          • Spookykou says:

            Based on my reading I would put Scott’s opinion at something like “probably bunk yes, but in theory large effect small interventions could work, and growth mindset is far from proven false, in general we shouldn’t just discount weird things like this because we might miss important stuff.”

          • HeelBearCub says:

            @spookykou:

            Then I think you are still making the exact mistake (which Scott is responsible for) that I am highlighting.

            I believe Scott’s intention is more “extraordinary claims require extraordinary evidence”. Perhaps you think this is a distinction without a difference, but I don’t think so.

            Taking exception to The Buzzfeed article would seem to support the viewpoint.

          • Spookykou says:

            I must confess I am a bit confused here.

            What would extraordinary evidence look like and how does it differ from Dweck’s initial research?

            I thought the whole point of the devil line is that Dweck’s initial research is extraordinary, and yet it is still insufficient to convince Scott.

            Are you saying that Scott actually endorses growth mindset as probably true? If so I will readily agree that I did not get this reading his initial article.

            I do think there is a meaningful difference between your reading and mine, but I would, naively assume that my demand is easier to meet than a demand for extraordinary evidence, so is my reading more accepting of the possibility of Growth Mindset or less?

          • HeelBearCub says:

            I thought the whole point of the devil line is that Dweck’s initial research is extraordinary

            Hence my point about Scott overselling his critique.n I think he thinks the studies are really good.

            I think that what Scott says at the very end of the post in question is his honest belief (except I removed the part where he tried to be funny again, for what I think are obvious reasons at this point):

            There are some really good studies showing that it works in certain lab situations. There’s a lot of excellent research behind it and a lot of brilliant people giving it their support. But there are also other studies showing that it has no long-term real-world effects that we can measure, and others that might (or might not?) contradict its predictions in other ways. I have only the barest of ideas how to square those facts, and I look forward to hearing from anyone who has more.

          • Spookykou says:

            I guess I might be reading that quote differently or we are reading it the same way and I am just adding on something extra? I took it as ‘the evidence for and against this is good so I don’t know what to do with the evidence’. Maybe I did not convey this well in my previous comment but this ‘the evidence is not illuminating’ interpretation is what I also got from Scott’s essay on Growth Mindset.

            I only get to the ‘but it is still probably bunk’ part because, if you have contradictory good evidence for and against something, you are either going to take the outside view and it is the same as no evidence or 50/50 or something, or you are going to lean on your prior assumptions and heuristics.

            In this case, I think Scott has a general heuristic that is hinted at in the essay in question and expressly stated in other essays on education research in particular, that ‘small interventions don’t generate big effects’ , and well, from the essay,

            This is a nothing intervention, the tiniest ghost of an intervention. The experiment had previously involved all sorts of complicated directions and tasks, I get the impression they were in the lab for at least a half hour, and the experimental intervention is changing three short words in the middle of a sentence.

            Edit:

            growth mindset is far from proven false

            was my attempt at ‘the evidence on this is confusing’ it took the negative connotation instead of the positive ‘growth mindest is not totally proven true, because, as I explain I think Scotts heuristic here would be that it would be false. I should probably have gone for a more neutral ‘The evidence for and against growth mindset is good’ or something. I admit that in retrospect this did not convey this idea well and probably looks overly negative on Growth Mindset.

          • HeelBearCub says:

            @spookykou:

            “Unfortunately, I think it will probably will turn out to not actually be true” and “It is probably bunk” are two different things. This is the distinction I was trying to highlight.

            I think we may be on the same page at this point, but I’m not quite sure.

          • Spookykou says:

            Yes I think we are.

  16. HeelBearCub says:

    Do we have a control group that answered the questions before reading the article?

    How do we know that MTurk respondents actually read the article (rather than skimming the article while being asked the questions)?

    I really distrust when you are paying a pittance for piecework that you will get the same kind of quality regarding comprehension that you would get from people self-selecting reading the article?

    • Wrong Species says:

      Ironically enough, I had the exact same concern but I was worried about the other result, people from here being involved in these studies and telling them that it wasn’t misleading. But to the person who only read the original article, it still would be. I’m still not sure how we can control for this.

      • HeelBearCub says:

        Well, that is why (I think) you probably ideally have a group that answered the question before and after reading the articles. Perhaps the before answers can be the control as well (I feel like there ought to be an issue with that, but I can’t immediately think of one).

        If someone had read the various articles beforehand, that is really just going to look like the article not changing their mind. Given that people aren’t coming into this tabula rasa I’m not sure it’s all that interesting to know exactly what they have read.

        On the other hand, the idea that MTurk sample might be in some way biased because they specifically chose to do this task is probably relevant? I’m not exactly sure how one goes about being a Turk.

        • Wrong Species says:

          Imagine that we did this study and ended up being 90% SSC readers and 10% other. The SSCers are either liberal or conservative biased while the others are neither. Regardless of the fact that a control is used, the study is still going to be dominated by whichever way they get split up, swamping the results of the people who came in to this with no preconceptions. A control could help but I still think there are potential problems.

          • HeelBearCub says:

            Agreed, which is why I wondered how MTurk tasks are actually acquired.

            I have a feeling that the sample skews towards “don’t give a shit give me the next question so I can get my next task because baby needs a new pair of shoes” though.

    • Jiro says:

      How do we know that MTurk respondents actually read the article (rather than skimming the article while being asked the questions)?

      How do you know that New York Times readers read the article either? People who are not Mechanical Turk users also read misleading clickbait headlines and get fooled into thinking they represent truth. Having Mechanical Turk users do it too doesn’t make them any different from NYT readers.

      • HeelBearCub says:

        “Headlines on articles should be assumed to be misleading” is not a particularly novel observation, and does not seem to be the core of Scott’s point.

        I’m sure one could assemble a pool of scientific abstracts and get Mechanical Turk users to form misimpressions based on those. It doesn’t tell us very much about whether scientists in that field (the target audience) will come away misinformed.

  17. TenMinute says:

    So in the finest tradition of science bets (and welching on them) from now on Scott can end any Twitter conversation with/about him with “PAY WHAT YOU OWE, NOAH!”

  18. alphago says:

    Scott Alexander, it seems to me that you continue to offer a misleading summary of the underlying survey e.g. “with more supporting the privatization than opposing”.

    The survey didn’t ask if they support privatization, it asks if public school students would receive a higher quality education. It is entirely possible that vouchers would make kids better off in general, but harm the most disadvantage kids, in such a way that most economists would not actually support vouchers, particularly given that uncertainty might bias towards the status quo. Some of the economists who agreed specifically commented about the risks to some kids, esp disadvantaged kids. For instance, Michael Greenstone agreed with the question, but said “Competition is likely beneficial on average. Less clear that all students would benefit leading to tough ?s about social welfare functions”. So does he “support privatization”?

    I would be willing to take a bet with Scott Alexander that most people who read his summary of the survey will be mislead into thinking that the survey asked economists if they “support” privatization.

    • axiomsofdominion says:

      This is absolutely the case. There is a strong faction in education arguing that the weaker students benefit strongly from having much better performing students in their class and its worth it even if it damages the much better students quite a bit.

      • TenMinute says:

        And behind closed doors, I’ve heard the muttered addition: “and it stops the smart ones from getting uppity”.

        • Deiseach says:

          The smart ones will just educate themselves and ignore the teachers who are reduced to baby-sitting a class for weaker and disruptive students.

          • The Nybbler says:

            It’s not the learning of the smart kids which is so hurt; it’s their mental health. And their deportment, which can make them the disruptive kids. Both from having to sit around not being allowed to do anything while the teacher works with the dumb kids.

  19. HeelBearCub says:

    Scott has a bias against school, formal education, what have you. This is well known.

    He has all but said schooling is irrelevant to outcomes.

    Given that Scott is also giving a misleading take on what the survey actually says, I feel like this is relevant.

  20. FollowTheQuest says:

    I’m a lurker coming out of the woodwork, to hopefully make people more aware of what it is actually like to be a participant on MTurk.

    As a former “Turker,” I feel it must be noted, for the unaware, that a core group of participants on MTurk are incentivized to work as quickly as possible, so despite having comprehension questions, it seems unreasonable to expect that the participant has given a full effort to reflect on the content of any particular HIT. Many full-time turkers are attempting to earn a minimum fair wage (typically 10¢/min) by completing HITs, and may even have a browser plugin displaying a timer to let them know if they are working fast enough, so they can abandon HITs that are taking too long or rush through the current one. (Relevant anecdote: For roughly six months, while living with relatives, my spouse and I worked on MTurk to earn food/gas money; we referenced guides available on the MTurk subreddit, forums, etc. to learn how to achieve a minimum wage income solely by completing HITs.)

    I’m not saying this means these MTurk surveys were insufficient to provide a conclusion to the bet, merely pointing out that the participants probably didn’t think very hard about the content of the article because they breezed through the survey, like they breeze through every survey. My actual worry is this: I wonder if people who conduct research on MTurk realize that most of the workers taking their surveys are probably not putting forth their most reflective effort? It just seems to me that this would impact the quality of any serious research.

    • suntzuanime says:

      Speaking as a person who conducts research on MTurk, we do. Even if people aren’t putting deep effort in, it’s still useful for researching snap intuitions, which is good enough for many tasks. Most of human experience takes place at the snap intuition level anyway, things where people sit down and focus and work hard are the exception not the rule.

    • Wrong Species says:

      Like Suntzuanime said, as long as they read it, it doesn’t really matter whether they thought about it too hard. The average NYT reader is not going to be scrutinizing every news story. They shouldn’t have to read between the lines to accurately gauge the truth.

      • HeelBearCub says:

        If the Turks differ in a meaningful way from the typical NYT reader, then you don’t have a test of what the typical NYT reader got out of the article.

      • xq says:

        At some point this just becomes an argument for never writing nuanced articles in the NYT

        • HeelBearCub says:

          or anywhere.

        • Wrong Species says:

          There’s a difference between nuanced and actively misleading. Scott’s articles are usually nuanced but he leaves out his cards on the table. If you strongly imply one thing but use a bunch of weasel words as a future defense, you aren’t doing the world a great service.

          • xq says:

            No one is in favor of “actively misleading.” But you implied that it should never be a requirement to think hard in order to get a NYT article, and that puts a pretty low ceiling on complexity. I don’t think NYT should be writing to the MTurk audience (at least, not always) so I don’t think poor comprehension of MTurk users is in itself sufficient evidence to condemn an article as misleading.

            Like I said above, we really should conduct a control experiment here. If you think Scott’s articles are nuanced but not misleading, let’s subject one of them to a similar test that this NYT piece was subjected to.

  21. dwietzsche says:

    Another related point that I wish people would make more is that the fabled ‘consensus of economists” is always assumed, never demonstrated. On almost no subject where there is any amount of controversy does it even exist in the first place. There’s this widespread assumption that economics is a science that has settled all kinds of economic questions. I can understand why people would assume that, because otherwise why are there all these guys called “economists” whose expert opinion supposedly matters in the first place?

    • On almost no subject where there is any amount of controversy does it even exist in the first place.

      [here “it” is a consensus of economists.]

      There was an AER article quite a long time ago that surveyed a wide range of economists. The pattern as I remember it was three sorts of questions:

      Questions that mainly depended on values: Considerable disagreement. (“Should the income distribution be made more equal.” Not an actual example–I’m working from memory–but the sort of thing in that category.)

      Questions on areas where there was substantial professional disagreement, largely in macro. Considerable disagreement.

      Questions whose answer followed in a straightforward way from conventional theory (“Does a minimum wage law increase unemployment among low skill workers,” “Do tariffs make the country that imposes them richer or poorer,” again not actual examples). Agreement well above 90%.

      Or in other words, there isn’t a consensus in areas where there is substantial professional controversy, such as the use of deficits to reduce unemployment. There is, or at least was at the time, a high degree of consensus on some issues that were politically controversial.

      One feature of the situation that may reduce the public view of consensus is the tension between academia, which is mostly well to the left of center, and economics, some of whose implications are inconsistent with left wing views, as in the case of the minimum wage. Leo Rosten told the story of asking one of the MIT economists about the minimum wage and getting the conventional economic answer. He asked why, if that was the usual view of economists, they didn’t make a point of publicizing it.

      Answer: “I guess we are afraid of sounding as if we agree with Milton Friedman.”

      My impression is that ninety years ago there was a much stronger conservative influence in academia–an impression probably based on Mencken’s writing. If so, I wouldn’t be surprised if the same mechanism reduced the apparent consensus for free trade, which at the time was a position generally supported by Democrats and opposed by Republicans.

      • suntzuanime says:

        If I remember my International Economics class, the theory actually suggests that tariffs can make a large, important country richer if it makes up enough of the market for the tariffed good. It’s just that the gains are more than outweighed by the losses to other countries.

        Of course I may be wrong or misunderstanding something.

        • RicardoCruz says:

          No, you’re wrong. There are arguments based on network effects showing tariffs in your city/state/country might be good for some weird economic activity. But those are very corner cases. EDIT: Actually, I think the arguments are based on path dependence, not network effects. Anyhow, Paul Krugman won the Nobel on this and he himself says David Ricardo is still the holy grail of international trade.

          Another interesting argument is that if all world production is concentrated in a region of competitive industries, then you should apply tariffs on exports in order to capture the surplus value that would go to other countries hehe.
          Again, these are very corner cases which rarely if ever exist in the real world.

          As I understand, the argument today is more whether tariffs can be good in order to protect low skilled labor.

        • You are correct. The same argument implies that export taxes can benefit a country if it has a monopoly or close to it on the goods being exported.

          In both cases, what’s going on is monopoly pricing. The first case is a buying monopoly, a monopsony. Americans are a competitive market for widgets but (we assume) America as a whole is the only consumer of widgets. By putting up a tariff we reduce U.S. demand, which reduces the world price, which benefits us at the cost of the producers abroad. Similarly the other way if we are the only producers of widgets. It’s the same as the benefit to an industry of successfully maintaining a cartel.

          But it isn’t a very likely situation, and we don’t observe that the tariffs that actually get put on fit either that pattern or the infant industry pattern, which is another special case where it is logically possible to benefit from a tariff. So economists generally believe that the actual effect of tariffs in the real world is to make the country that imposes them poorer.

      • RicardoCruz says:

        Bryan Caplan on his “Myth of the Rational Voter” also has interesting things on economists consensus. But nothing as detailed as that article. That was interesting, thank you.

  22. phil says:

    I’m curious as to whether you have any impression as to the intent of the original NYTs article writing (intentionally leaving the name out)

    lets concede that the article was confusing to reader (which doesn’t seem like much of a concession, its seems basically proved at this point)

    Noah’s piece, and it seems like most of the critics of your original piece, are sort of working from a place where they would concede that it was a piece that was likely to confuse readers, but won’t concede mal-intent on the writers part

    which I think I actually buy, and I think leads to more interesting questions

    ————
    ————

    which is mainly, are there systemic things that the NYTs (and the rest of the media does) that make these sorts of reader confusion situations likely

    couple possibilities

    1. The really obvious, is the pervasive practice of making the title as click-baity as possible. Which is a really insidious practice that you see every, for really obvious reasons, and relative to what you see everyday on Slate, the title here doesn’t even seem that terrible an offense

    I don’t know how to make headlines better, other than to hold the publication as a whole entity accountable for them, I suspect that problem isn’t going away though

    2. The what I would categorize as boring observation that NYTs writers and editors probably form their own little groupthink bubble where things that fit their worldview don’t catch their eye as off, but things that don’t fit their worldview do, such that their mistakes tend to all be in the same direction

    3. Sailer has done a few post contrasting the NYTs fairly dry style with the Daily Mail, sample: http://www.unz.com/isteve/the-congos-small-little-mini-war-the-nyts-fear-of-admitting-an-interest-in-human-biodiversity/

    “One of the curious aspects of New York Times articles is that they are often organized in the reverse order of how the same material would be reported in, say, the Daily Mail. NYT articles tend to start off boring and depressing, with only vague hints of why the reporter is interested in the subject, and don’t get to the good stuff until late in the article, by which point, no doubt, most readers have given up.”

    An interesting question is whether this style systemically leads to the sort of reader miscomprehension at issue here

  23. nimim.k.m. says:

    I agree that the article was misleading about the way you Scott described (I didn’t know about this bet going on but I would have bet on you winning this one). But as far as I remember, one of the main disagreements with your bring-it-down criticism was (at least, mine was) that it felt that you span the “mostly unsure, with more supporting the privatization than opposing” as a sign of economists being in favor of privatization.

    I still maintain that the fundamental problems were a) the question asked was slightly different than the presentation of results alluded to (they didn’t ask if economists were favorable, they asked if all kids would be better off or something), b) when there’s distributions with significant number of answers in the “not sure” contingent (or similarly complicated distributions), only faithful way to represent the information is to reproduce the whole distribution, anyway, and c), I believe that economist way of thinking would have bias against taking into account all the social effects a thing like “a local publicly owned school” has when its services are not thought as a tradeable good.

  24. thetitaniumdragon says:

    I think that part of the problem is that the general public is extremely bad at understanding the concept of uncertainty – they think that there is a definitive answer to everything, when in reality, there often is not. As such, an article noting that most people don’t accept some idea leads them to believe that most people oppose some idea, when in reality many of those who are not accepting some idea are uncertain about it.

    To be fair, this is, perhaps, not necessarily a bad way to fail – most ideas are wrong, so if lots of people are uncertain about an idea being wrong, it is probably more likely than not to be wrong. But it does lead people to a greater degree of certainty than is warranted by data.

    On the other hand, this also leads to a bad failure mode where people believe that if 60% support some idea, that it must be true.

  25. paultorek says:

    FYI, if commenter sripada is Chandra Sripada , as I suspect, then he also is a real research psychologist.