A couple of people have challenged my essay yesterday, saying that they didn’t find the article misleading (or that it was only very slightly misleading).
They argue that it pointed out that economists were overwhelmingly in favor of ride-sharing arguments like Uber, but only somewhat in favor of school vouchers. Therefore, it’s fair to say that “economists don’t generally buy school vouchers” in the same sense that they “generally buy” Uber.
This would have been a fine thing to say, but I think the article failed to make this point and instead phrased its argument in a way that made it unlikely to readers to conclude anything other than that economists generally were not in favor of vouchers.
First, I feel like you could write exactly the opposite headline. “Public School: Economists Generally Don’t Buy It”. You would cite the statistic that “only 19% of economists surveyed disagree with the statement that school vouchers would improve education over the existing public system”. Then you could explain some public choice theory about how economists believe government services generally do poorly. This would be exactly as honest or misleading as the existing article. Yet it would produce exactly the opposite impression in readers’ minds: the original article makes you think economists mostly oppose school vouchers, the changed article makes you think they mostly support them. If newspapers are allowed to interpret data so loosely that they can use it to draw two different conclusion precisely opposite each other, what’s the point of having data?
Second, the article uses economists “not buying it” as a segue into a description of why economic theory says school choice could be a bad idea. But it seems like the majority of economists are not convinced by this argument. That is, both the 1/3 of economists who agree and the 1/3 of economists who are uncertain don’t accept the author’s argument that economic theory proves school vouchers can’t work as obviously true (the uncertain ones may be uncertain whether it’s true or not). This gives it a credibility it doesn’t deserve.
Third, really, if this same article was on Breitbart, and it used a survey showing that 40% of economists supported climate change legislation, 40% were uncertain, and 20% opposed it, and it described this as “Economists Generally Don’t Buy Climate Change Solution”, nobody would think anything untoward had happened and they would all agree this was a perfectly fair and unbiased summary of the evidence?
Fourth, you can do this for anything because there’s no clear definition of “uncertain”. How sure does an economist have to be before she “agrees” with a statement rather than being “uncertain” about it. If you are 51% sure school vouchers help, are you pro-voucher or uncertain? What about 60% sure? What about 90% sure? Suppose that all economists believe with 70% probability that vouchers will be good. If your criteria for “certain” is “80% or above”, then as long as you separate them out into support/uncertain/oppose, you can “honestly” declare that “no economists support school vouchers” and convince everyone that it must be an economically absurd plan. Yet I would hope that on any controversial issue more complicated than Uber, most economists are at least a little uncertain about it. We shouldn’t view that as legitimizing us to say that the economic consensus is whatever we want it to be.
Fifth and related, if you look at the economists’ comments, a lot of the people who self-described as “uncertain” thought vouchers would be good on net, but didn’t like the question because they thought it implied that literally 100% of students would be better off. This was such a problem that the IGM redid the study a year later, this time asking whether vouchers would make most students better off. 44% of economists agreed, compared to only 5% who disagreed (again, 34% were uncertain). Weighted by confidence, >50% of the economists agreed that vouchers would improve things, compared to only 6% who thought it wouldn’t improve things – a difference of almost ten times more economists agreeing that vouchers would help compared to disagreeing!
In the face of all of this, the New York Times gives the field’s opinion as “Free Market In Education: Economists Generally Don’t Buy It”.
I’m trying to be more empirical these days, so if you disagree with me, let me make you a bet – and I seriously mean I’ll bet money on this if anyone wants to take me up on it. We find ten random people of ordinary intelligence and economics knowledge and show them this article. Then we ask them a question like “according to this article, what is the economic consensus on vouchers?” We assure them that this isn’t a trick question and they’re just supposed to honestly give the impression they get from the article. If they say something like “they’re generally against vouchers”, I win the bet. If they say something like “probably more economists support vouchers than oppose them, but many are uncertain”, you win the bet. I am willing to alter exact terms if you have a better idea. I am willing to make this bet at 10:1 odds, so if you think there is any chance at all this article is not misleading it should be easy money for you.
[EDIT: I am going to take a version of this bet with Noah Smith. I don’t need to take it with any more people. Offer is now closed.]
[EDIT 2: Noah Smith is apparently no longer willing to bet me on this, although he refuses loudly and at great length to say whether that means he now agrees with me that most people would get a false impression of economists’ position from the article. Is there anyone else who wants to take a bet on these terms or any others?]
It’s also worth noting that the NYT completely ignores the very similar question IGM asked a year later, where the only real difference was a clarification that the question asked about “most” students.
The results seem much more positive. Weighted by confidence 54% strongly agree or agree vs only 6% who disagree.
I don’t think the odds are good for taking the bet, because there is more than 10% chance of a a majority of 10 people being wrong about what they read.
For an example of one such possible misreading, see https://slatestarcodex.com/2016/12/31/addendum-to-economists-on-education.
If something is written in such a fashion that you think a majority of people might misread it in a way that supports the author’s thesis, I think there’s a case to be made that the author is being deceptive.
Scott is offering you 10:1 odds. You would need to think that there’s over a 91% chance that the majority would be wrong about what they read to have the bad side of the bet if you’re right about the substance.
Note that Autor is actually in the “disagree” column, not even the “uncertain” one!
I’m glad you held your ground on this one. Your criticism was measured and accurate. I found it very off-putting the way some were willing to bend over backwards to defend that article using the same technically-true-but-misleading style as the article itself.
While I’m down here… I also want to say I approve of the news-which-must-not-be-named ban. It’s a imprecise garbage term that conveys only one’s disapproval. There are still a billion ways to criticize reporting in a way that isn’t so prone to abuse and equivocation.
Edited.
Well, I was using the term mockingly in order to discredit it. So although a ban prevents people misusing it here, it also prevents mocking it. I actually haven’t seen the term used sincerely by anyone in these circles to refer to anything that wasn’t blatantly made up, and even then they’re usually only calling it that because they’re speaking from the perspective of the mainstream media, quoting them or whatever.
But the quicker this term dies, the better, by whatever means.
Agreed. That wasn’t meant as a shot at you, I actually thought your comment was really funny and spot on. I think the preemptive strike was wise though, as I could see it leading to bad places.
*agreed?
Yeah, thanks, fixed.
You’re underselling it. From the link:
58% agree, 12% disagree. If you break out the ‘strongly’ section, 7% strongly agree and 0% strongly disagree. With the claim that the main drawback to school choice would be that some students wouldn’t use it.
That seems like a legitimate concern. I’m not sure what your point was?
It does not. The article doesn’t even mention vouchers after that paragraph. It never makes any claim nearly so strong as “school vouchers can’t work.”
Framing your contention empirically, in the form of the terms of a hypothetical bet, is an excellent way to lock down exactly what it is you’re claiming.
Some people’s objections are no doubt of the form: “Well a sufficiently intelligent person would not be misled by this”, or the one that I hear far too often “Reasonable people would not be misled by this”, and then, upon showing that the average person is not “reasonable” by this person’s standards, “Well, they’re idiots!”, as if that’s disagreeing with you or somehow means they aren’t relevant.
But actually I think a source of information is best judged by how an average reader would interpret them. I assume the average NYT reader has an above average IQ (my model of actually average people is that they don’t read news at all and only see viral headlines that reach facebook). So I think that’s the standard they should be judged by, and would also bet that they would fail by that standard. But it’s great to put these claims in concrete terms.
This is a pretty good point. I think I’m rather guilty of that – interpreting pieces largely by merit of how I would imagine, for example, the SSC crowd to interpret the article.
Perhaps that is in the end the source of contention – in venting his (completely understandable) frustration of the article here on SSC, his audience was largely educated enough not to fall for the sleight of hand, and many in turn might not realise the extent of the potentially detrimental effect. (e.g. me)
(I do think the article’s disingenuousness was still comparatively harmless, but that’s very much a relative observation. I don’t feel educated enough to make an absolute one. I’d have taken Scott up on his bet, but mostly out of curiosity, not certainty.)
Yeah, I like to think that my function on here is as representative of “the ordinary idiot in the street” and with a headline and article like that, I definitely would come away thinking “Welp, if only one-third of economists are in favour, guess that means two-thirds think it’s a rubbish idea”.
I might personally disagree with the study or think the two-thirds were whistling Dixie if I were strongly in favour myself of school vouchers, but I would certainly think it was a case of “one-third = good idea, two-thirds = bad idea”.
I know I’m always banging on here about “read the source material!” when it comes to historical claims, but most people (including myself) won’t look up the data unless they think “Hang on, something sounds fishy here”. Unless you already have a strong opinion about school vouchers or a strong mathematical sense that starts tingling, you’re not likely to do so.
Well, hopefully asking “the man in the street what he thinks” won’t end up like this 🙂
Just going to copy my comment from the reddit post on this over here, because I think it could yield some more interesting discussion directly in the comments.
The argument here appears to rely on a misunderstanding of how to interpret support for such programs. A person who wants to go away from the status quo and implement vouchers needs to affirmatively show that their policy has support; the burden of proof is on them, not on the opposition. Thus, the group uncertain about the policy should be counted as “not supporting it” because that is the default position. Saying otherwise would presume that the default position is to support a change from the status quo, which is not really done in policy and requires the opposition to prove a negative which can only be done if the policy is implemented on a wide scale. Think if we had the same standards for other policies. If 30% supported banning vaccines due to risks of autism, 50% were unsure of the overall effect, and 20% disagreed, very few would interpret that as any indicator that we should actually change from the status quo and go ahead and ban vaccines. Instead, we would say that most would not suggest changing from the status quo and banning vaccines. (This example is totally unrelated to actual vaccine policy or my views on vaccines) Similarly, scientists would have to conduct experiments where they had to find significant results in favor of the status quo, and every p-value above .05 would indicate their alternative hypothesis was favored, flipping hypothesis testing on its head. Thus, because the burden of proof is on the group supporting vouchers, we actually should say that “economists generally don’t buy it,” and I would find in favor of the results being reported in such a way.
If there is a large group of people saying uncertain, and you don’t report that, you’re being dishonest. There’s really no way around it.
They do not say that researchers say it would not work in an unqualified manner. Yes, the writer could have worded it better, but at this point you are splitting hairs. That could have been as much the result of bad editing as presumed dishonesty. Also, the point is that people are already taking it seriously, given the fact that Betsy DeVos will be running the Dept. of Education. The whole point of the article is to say “No, the consensus is not behind this.” They do not say the consensus is against it, and they do not say the idea is laughably stupid and not worth taking seriously. They simply say economists do not support it, which is true in no uncertain terms, and not even misleading.
1. The title is “Free market for Education? Economists Generally don’t buy it”.
The questions are about a voucher system, not about the free market. The person responsible for the title is either totally ignorant of the definition of “free market” or is being intentionally deceptive.
2.
Privatizing isn’t the free market.
3.
OK, now we can’t blame the editor for the title, this is a complete bait and switch. “economists generally support the free market, but they don’t support vouchers as much as free market initiatives, therefore they don’t support a free market in education”.
4.
Unfettered has a meaning, and it isn’t anywhere in the republicans plan.
4 paragraphs in and the article which is referencing a question about vouchers has repeatedly linked vouchers to an unfettered free market system.
The entire piece is a farce, if not an outright lie.
I agree that there should be a bias towards conservativism, but that bias should be added after we have fairly counted the amount of support and opposition, and not necessarily by rejecting the plan entirely.
For example, ten years ago, 5% of AI scientists were concerned about xrisk, 1% were strongly against it, and 94% didn’t know and didn’t care. This kind of situation requires a very different set of policies compared to 95% being against.
If most economists who have an opinion about vouchers support them, but few have an opinion, that suggests more study and experiment as a high priority; if most simply oppose, that suggests throwing out the idea.
Also, I will be snarky and say something about how Hillary technically won the election, since there were 100 million nonvoters and that is reason enough to avoid the major change of Trump.
I actually agree with you overall, I just think we interpret the meaning of the article differently. I would also like further research into this to get rid of ambiguity in the data. Nowhere in the article does it say that we should not look into these things further, to my understanding.
What the writer does seem to oppose is going full speed ahead and doing this everywhere at all once, which Betsy DeVos seems inclined to do based on prior actions and statements. When you assume that is what the change would consist of, opposition to this plan due to economists’ uncertainty becomes totally valid, in my opinion.
Also, being uncertain is very different from having no opinion, as you state above. There is actually a separate option on the survey for “no opinion” which does not actually weigh into the totals for uncertainty. Uncertain instead means “I have a position on this issue informed by data, and it leads me to believe that there isn’t clear support one way or the other,” which is very different from having no opinion at all like in your example with AI-risk. (I also dont find AI risk to be something we should prioritize, so I probably find that example less compelling than most here, but thats a different discussion altogether)
Lastly, speaking of Trump, I don’t know if you or the rest of the commentariat has seen the IGM poll (http://www.igmchicago.org/surveys/100-day-plan) for Trump’s economic plans, but lets just say that I find it humorous that people put such huge confidence on the IGM poll given the amount of Trump supporters here (I put great stock in IGM for what its worth, but I am also not a Trump supporter).
sure buddy
Come on, condescending comments like “sure buddy” really don’t belong here. Either ignore it or use arguments. I also think there are few actual Trump supporters here (unless you count “Clinton and Trump both bad, Trump possibly slightly better”, which seems to be a position of a few people here, as being a Trump supporter), but comments like yours only manage to make people annoyed, regardless of what they’re replying to.
Condescending comments are exactly the right way to deal with that sort of bullshit. There is an epidemic of unsupported whining about how right-wing these comments are. If we have to respond to every thrown-off ridiculous claim with a long and scholarly essay refuting it, the costs are asymmetric and the whiners get what they want. I apologize if my post has annoyed you, but it’s in the service of defense of the community, so I ask you to bear with me.
Hillary was essentially the status quo candidate.
Do you think that the American system is working well and should not be changed? Vote Hillary
Do you want this set of vaguely hinted at major changes? Vote Trump
I don’t think not voting can be read as supporting either of these positions.
I didn’t vote because I think that the status quo is broken. At the same time, I couldn’t tell what Trump would do, and a lot of what he said was as troubling as what was already happening. Don’t imply that I oppose making major changes. If I did, I would have voted for Hillary.
“Uncertain because not enough information in yet/would prefer to run a small-scale voucher policy and see what happens first/would like to see a concrete example of a policy because there are several ways you could set up a voucher scheme and the outcome would depend on how you did it” is not the same thing as “think this is definitely terribad idea because of reasons”.
So “Economists generally don’t buy it” may have the connotation of “reluctant to change the status quo” without including “think it is a completely bad idea both in principle and in execution”, but most people are not going to read it that way.
So how are you making sure someone didn’t read these articles before hand?
Do you mean Scott’s articles? Presumably you could just ask them?
But they could be lying. If you’re going to put money on something, make sure it’s not trivially easy to cheat.
I’d say, what’s the point of having newspapers.
I’d say, what’s the point of having newspapers.
Tomorrow’s fish wrappers 🙂
Or at least opinion pieces
Scott,
The NYT article was clearly misleading, but your own analysis of the piece was also clearly misleading. Given that I assume your intentions were honest, this suggests that summarizing data like this without explicitly acknowledging the huge amount of ‘uncertain’ responses is more difficult than it might appear.
I think the following statements, from yesterday’s posts and your writing in the comments, clearly present a misleading summary of the data:
Absolutely accurate, but most people would interpret that as about 67% ‘agree’ responses and 33% ‘disagree’ responses, which is a far cry from the actual data.
This suggests that the ‘uncertain’ responses are economists without an opinion on the issue, rather than economists with mixed feelings.
I think ‘strongly support’ suggests that a clear majority of economists support vouchers, when in reality, not even a plurality of them support vouchers.
Of course, if we start looking at the data from the following year, the picture becomes more complicated, but I assume that the phrasing in question was intended as a summary of the statistics explicitly cited in the NYT article.
Eh, I think that it’s okay not to have an opinion on something, but if we’re sounding something out then we should count the people who do have an opinion.
I guess the problem is that “uncertain” can mean either “I have found exactly the same amount of evidence on both sides” or “Look, I run the Federal Reserve, why are you asking me about schools?”
I interpret “mixed feelings” as “not having an opinion” in the relevant sense, and think you are being silly and nitpicky in a way that I was not. I also put my comment in the context of a long post that cited the graph and made it very clear exactly how many people were uncertain.
Either way, they shouldn’t have made it look like a rejection of vouchers.
Scott, there is a “no opinion” option. Nobody took it on the voucher question. Of the two interpretations you gave, “uncertain” essentially only means the first one. The economists saying “uncertain” aren’t saying “I don’t know”, they’re saying “given the evidence we have, it would be misleading to give a yes or no answer to that question”. To make this as obvious as possible, imagine a survey of physicists asking “Is the electron we’ve isolated in our lab spin up or spin down?”, with the same response options. I predict you would get 100% maximally confident “uncertain” responses, corresponding to a very strong consensus among physicists.
(As an aside, the fact that there’s a “no opinion” option makes the reweight-by-confidence graphs look like there is a stronger “no” consensus than their actually is on essentially all questions, because it eliminates the right-most column (which shows “no opinion”.)
All that said, I think you’re right about the actual consensus among economists on vouchers / school choice of some sort: moderate confidence that most students would benefit, with the caveat that a non-negligible minority would not.
The NYT article is a horrible mess in general and misleading on the question of economists’ opinions in the specific, but please please please make sure your criticism is not also misleading. That is a large part of the point of the rationalist community.
(Caveat on that physicist question: if you actually asked the question that way, you’d definitely get some trolls saying “yes”)
It seems plausible that on little known topics, those who have a strong opinion will likely be the idea’s supporters. That “no opinion” is so rarely selected seems plausibly a cultural thing, since unless they’re contacting people who specifically care about vouchers, a lot of them may not be any more familiar with the arguments for or against than our host is.
I do think it’s significant that economists are much more likely to say that they’re uncertain about the merits of school vouchers than about ride-sharing. That doesn’t necessarily reflect that economists think vouchers are bad; rather, I’d guess it’s because there’s not nearly so big a chunk of evidence to point to for “did this work?” as Uber makes for ride-sharing.
While I agree that Scott’s knowledge of the issue is probably comparable to the knowledge of the typical economist on school vouchers*, I don’t think that applies when the question is about the consensus among economists. I view the survey as basically an election of a particular statement as “the consensus among economists”. Based on the fact that “no opinion” sometimes is selected (on other questions), I think we can safely assume that a vote of “uncertain” means that the voter thinks the consensus among economists should be read as “uncertain”, and a vote of “no opinion” should be read as “I am not part of the consensus on [issue]”.
But yeah, I also agree that the fact that “uncertain” carried the day here is very important. (Hence my reply to Scott.)
* Economists are probably the wrong group of people to ask about school vouchers and school choice, though they aren’t the wrongest group imaginable. The right group to ask (among those that usually aren’t) is probably students, but asking students is hard.
@Gazeboist
Thank you! Though I’d speculate that those who are 2-5 years out of the relevant schooling might be better informed than current students.
Imagine that there is a murder, and a suspect is detained. Ten detectives are asked to shift through the evidence and decide whether they have the right person. Four believe the evidence proves the suspect is guilty; two believe the evidence disproves it; four think there is insufficient evidence to come to any conclusion.
In any sane legal system, the suspect would walk free. A blogger summarizing the investigation as “twice as many detectives found the suspect guilty as not guilty” would be rightly charged with dishonest reporting, even though in a very technical sense the statement would be true.
Clearly the crux of the matter is whether “undecided” means “I personally don’t know enough of this issue to have an opinion although other people might” or “I know all there is to be known about this issue and I don’t think that knowledge is enough to be sure either way”. (Or, in the case of the actual IGM panel, it might mean something like “I think most people would be better off but many still would be worse off, and it’s unclear how to map that to the available answers”. The questions of the follow-up panel were clearly designed with that problem in mind and that worked well.) Given that the IGM survey asked how confident the respondents are, and undecideds were on average no less confident than the rest, I don’t think you have a case here about “only a third of economists support vouchers” being misleading. You have a better case about the article as a whole being misleading, in that it uses a source that supports the narrow claim that it is quoted for, but contradicts other parts of the article. (But if that’s your case, and you seem to have goalpost-shifted to it with the way the bet is phrased, then you’ll have a harder time showing that this is an unusually high level of bias for mainstream media.)
4 guilty, 2 innocent, 4 not sure is a case where 6 think “not guilty” – “not guilty” isn’t the same thing as innocent, merely not guilty according to whatever standard is being used.
The question posed to economists was not “should we institute a national policy of school vouchers?”, it was “will vouchers generally help all students?” The former is much closer to the jury-verdict type question you pose. You could say the article author is taking data from the latter question and assuming the answer to the former.
You’re certainly correct to criticize them, and I think I overstated my point. I don’t think that your actual post was misleading when taken in full, but I do think your proposed ‘better summaries’ were misleading.
By your own empirical standard of ‘would laypeople draw the correct conclusion from your summary’, I think a summary like “About twice as many economists believe a voucher system would improve education as believe that it wouldn’t” clearly fails. I’m pretty confident that most people would interpret that as ‘a large majority of economists support vouchers”, which is not supported by the data we’re using.
The issue of how to characterize the ‘uncertain’ responses isn’t as clear cut as your original criticism, but I do think it’s important. If you got responses of 90% uncertain, 8% agree, and 2% disagree, would you still be comfortable summarizing that as “a large majority of economists with an opinion are pro-voucher”? I think other people have made this point more eloquently elsewhere in the comments, but ‘uncertain’ is not at all the same as ‘no opinion’, especially given that the explanations the economists offered were much more along the lines of ‘mixed feelings’ than ‘no opinion’. I think that interpreting those as the same gives too much power to people with extreme viewpoints.
Suggestions for better titles for the same data and opinion:
Economists: No Consensus Favoring School Vouchers
Economists Have Mixed Opinions on Vouchers
Will Vouchers Help Kids? Economists Are Not Sure
Any of those are fine. But “Generally don’t buy it” sounds like “they’ve heard the arguments and are unconvinced”. But the data doesn’t support that – at most, you could say that most economists aren’t convinced that vouchers would help all kids. However, it’s equally true that most economists are unconvinced by the arguments against vouchers!
Let’s be honest here – most of the arguments against Trump’s selection of DeVos (and school vouchers / privatization in general) are not measured critiques of the form “we can’t be sure, so let’s stick with the status quo”. Rather they’ve tended toward the histrionic – “The End of Public Schools!” “DeVos Will Destroy Public Education!” etc. In other words, not merely that vouchers might not help all/most students, but rather that they will significantly hurt a great many students. And I think it is fair to say that there is no economist consensus in favor of thatopinion either.
But “generally don’t buy it” doesn’t refer to vouchers, it refers to “free market for education”. The main claim of the article is that economists see the education market in a distinct way from e.g. the taxi market and that in particular they believe it requires more government intervention than the typical market. The panel poll is merely one point in support of that claim, and it does indeed support it.
If “generally don’t buy it” referred to vouchers I agree the article would be deeply misleading. But it doesn’t. Less than half the article is even about vouchers. It’s really not histrionic either.
But the survey on vouchers is the only actual evidence of economist opinion in the article! The fact that the author extrapolates from that to “free market education” in general makes things worse, not better.
(Also, I’m not arguing that this article in particular is histrionic – it isn’t. Rather, much of the criticism of DeVos has been, so I think NYT readers are already going to be primed to read this article to mean “economists oppose school vouchers”, full stop. That’s not directly the author’s fault, but it’s worth taking into account if we’re asking whether they should reasonably anticipate readers getting an impression the data doesn’t support. )
How so?
You could imagine an economist who believed that vouchers would make students worse off than the status quo, but a completely free market in education would make students better off. But none of the economists on the panel who are uncertain about vouchers seem to be coming from that position, at least among those who offered comment. So “economists uncertain about vouchers” seems reasonable to extrapolate to “economists view the education market as requiring more regulation than the taxi market.” It is at least evidence in support of that claim.
“But “generally don’t buy it” doesn’t refer to vouchers, it refers to “free market for education”. ”
And yet, vouchers are the only “free market for education” policy discussed.
It’s almost like you are admitting the author of the piece is using a straw man here. There is no survey data on whether or not economists support a “free market for education,” as the only question that was asked was about vouchers. If you distinguish between vouchers and a free market, stating that economists generally don’t buy a free market becomes a fabrication made up out of whole cloth.
And yet, vouchers are the only “free market for education” policy discussed.
That’s just not true. Did you even read the article? Most of it is about student loans. Charter schools are also discussed.
I really don’t think many people actually read the original piece.
Which part of the student loan or charter schools part referred to economists’ positions?
Regarding your last paragraph, I think there’s not so much substantive difference between those arguments as you suppose. There really doesn’t seem to be enough evidence yet (or if there is, I and most economists are not aware of it) to conclusively say that vouchers will improve education, so a fairly reasonable position is to experiment first before switching in total to an insufficiently-tested system. It’s reasonable to disagree with this conclusion on the grounds that there actually is more evidence regarding vouchers, or that the status quo is so awful a plunge into the relative unknown must be better, or that vouchers would not be so different that we can’t reasonably predict how they’d do, but those claims would need support.
If DeVos is indeed proposing revolutionary change to the school system, pointing that out is plenty germane. If DeVos really would end public schools or destroy public education, then yeah, I’d oppose her; I think more experimentation is needed to evaluate vouchers a wholesale replacement. Mind you, I don’t actually know what her beliefs are, or what she plans to do. I’ve heard contrasting things on both what she might do and whether that would be good or bad.
I’m not arguing that the argument as you pose it is unreasonable – quite the opposite in fact, I find that quite reasonable.
I’m arguing that many opponents of DeVos have not been so measured as you. The article itself seems to affirmatively argue that a “free market for education” would make things worse, not merely that we lack enough evidence to justify a reform. Your last sentence points to part of the problem – we don’t actually have a proposal to critique, and yet opponents are attacking a fully laissez-faire version of “free market education”. It’s basically straw manning.
Eh, my last sentence is a personal, and perhaps journalistic problem. I don’t know what DeVos has proposed. If people who have been paying more attention to her than I have think she’s proposed a laissez-faire, then they’re not straw-manning to say that would be awful, nor are they overreacting to be rather less measured. (I also don’t know whether on an object level they’d be correct to oppose it so strongly, but that’s a whole other issue.)
Of course, it’s not a given that they really do have enough evidence from her previous words and actions to infer this about her views; these are partisan sources. But my investigation of Myron Ebell found him to be pretty much just as bad as the doomsayers were calling him, so I’m tentatively inclined to think that DeVos would not exactly be my pick for the job.
As a very minor sidenote, Myron Ebell looks exactly like Cook’s Illustrated founder Chris Kimball.
Let’s be honest here – most of the arguments against Trump’s selection of DeVos (and school vouchers / privatization in general) are not measured critiques of the form “we can’t be sure, so let’s stick with the status quo”.
That’s definitely part of the problem: this is not a general “what do economists think of school vouchers?”, it’s “this is probably going to be Trump’s policy since he picked deVos for Education and she is strongly pro-charter schools, and since we know Trump is terrible and his picks are terrible…”
And why the hell are economists giving an opinion on schools, anyway? Do we have economists giving their opinion on “Studies on what bariatric surgery: which of the three methods works best?” or “What colour will fashion say is the new black for 2018?”
Who would you ask about their opinion of whether a different production mechanism for something (which happens to be schooling) works better or worse than the current one? Of course, people who work in education might have interesting insights into the particulars but this is not a question about how to teach. Other than teachers and economists I don’t know whose expertize might be relevant.
Economist here. I think economists are exactly the correct people to ask about school vouchers. Vouchers aren’t an education technique, they are a mechanism for PAYING for education.
This is merely my own anecdote, but I understood the article to imply exactly what Scott said it implies. When the actual data was shown I was surprised and felt like the NYT had not been fully honest.
Scott, is there any chance that you could stop referring to this whole issue as being about the NYT’s reporting? The piece was an opinion piece by an outside expert. The NYT could have handled it better, but that’s a different and arguably rather lesser sin. To judge from the comments, many of your readers are under the impression that the NYT staff itself was doing the number-crunching.
I don’t like naming individuals since people might try to harass them and since shaming is generally bad. I’ve removed all uses of the word “report” from the article.
More power to you, Scott.
Comment for discussing terms of my bet with Noah. I propose a Google form as follows (note that article is slightly edited for length):
(the extra questions are so people don’t overanalyze the economist question in particular and it looks like a real test)
Then I post a link to the Google form on Twitter saying “If you haven’t read my blog in the past week, please take this survey to help me settle a bet.” If you prefer, we can find someone more neutral to post it as long as they have lots of Twitter followers.
We go for five hundred answers or five days, whichever comes first. I win if more than half of people who give the natural answer to the other two questions (1:a, 3:b) answer 2:b, you win if half or fewer do.
If I win, I pay you $100. If you win, you pay me $10.
Does that sound fair to you?
I would worry that calling it a “reading comprehension test” (or even the fact that it will feel like a reading comprehension test even without calling it that) will prime people to parse this more carefully than they would in casual reading. Given that setting the cutoff at “more than half” is probably biasing this to the other side.
And, if I felt like being annoyingly careful, I’d ask if we wouldn’t have a problem with an article where the author knew with certainty that, given her wording, 49% of people with would end up believing the opposite of what the evidence showed?
Yes, and I will gladly take that bet.
What’s the best way for me to do that? I would be willing to send you $10 via paypal or bitcoin at the start of the bet since I am pseudonymous here.
Scott,
I’d just like to say that I agree with your position 100%, but as a degenerate gambler, I feel compelled to state that I think your odds are way too generous. I’m tempted to take the bet, even though I think I’d lose, but 10:1 is great value on a bet that comes down to getting a bunch of laymen to read and summarize an article.
I think you should include people who answer 3:c – the quoted portion of the article is just listing facts, and I think a reasonable reader could conclude that it has “no position”.
If you make that change (or remove the 3:c option altogether), I’ll take the bet.
This sounds unfair. People who follow you on Twitter are likely to want to lie for you. You should use a third-party Twitter unconnected to either of you.
I will take this, with the following adjustments/caveats:
The money is paid not to the participants, but a charity selected by yourself (Malaria nets and MIRI are both acceptable to me, but I doubt I’d be against any charity you select).
This is so that bias towards you/people who have read this and know the answer is somewhat counteracted by bias towards getting more money donated to charity.
It would still probably be best to use a Twitter other than your own.
As for the logistics, I’m comfortable sending you $10 by paypal/google wallet. Upon the outcome of the bet, if you win, you send $10 to the charity. I am out $10, you are out $0, and some amount of life is saved.
If I win, you send me back $10, and send $100 to the charity.
NB: I don’t actually disagree with your assessment, but I don’t mind spending $10 that will go to charity to find out more conclusively.
Regardless if you have any takers on said bet, running such an experiment seems like a fun activity.
Proposal:
split people into 3 groups
1. Reads only NYT article
2. Reads only Chigago economists
3. reads both
Group 1 and 2 will be asked the following question
How much do economists support school voucher programs (negative numbers mean economists are against vouchers posititive numbers mean support -5 means fully against while +5 means full support)
-5 -4-3 -2 -1 0 1 2 3 4 5
Group 3 will also be asked the question as well as a followup
The new york times reporting of this article was
A: honest
B: dishonest
C: Don’t know
D: other (please Explain)
Because of where I live and my personal situation I could only realistically get a convenience and self selected sample of college students who are willing to take part in such a survey,
If Breitbart published an article about economists not supporting climate change, people would just take that as an example of economists being either stupid or malicious. Possibly both.
It happens not too infrequently, while reading pure mathematics, that I misinterpret a precisely stated sentence – sometimes a lemma or a theorem – and then only come to my senses after somehow thinking up a counterexample to my misreading and going through much pain.
I suspect that I am far from the only mathematician who does this, though smarter mathematicians seem in general less likely to do it.
Hence, even in pure mathematics, usually I prefer people to not only say what is true, but often also say what is not true: in other words, work hard and proactively to avoid being misinterpreted. Even text books, let alone journal articles, usually do not try to do that. This may be because people smarter than me run the world, but I would think that people like me are probably in the majority.
Thus, even more so in journalism, given that humans are wired to misinterpret, just as they are to focus more on the subtext than the text, I would favor conventions according to which misinterpretations, while often be the fault of the reader, they are not necessarily so; it should be the responsibility of an author to proactively safeguard against it, and authors who write with minimal chances of misleading should be appreciated for the same (Oh dear, am I sounding like a victim-blamer!)
And this is one of my gripes with many otherwise smart economic and political commentators – they write nuanced pieces without enough safeguards against misinterpretations, sometimes intentionally so and often with a very provocative subtext; they then accuse their readers of not paying enough attention. Such people are usually considered polite though they strike me as in many ways crasser than those who use foul language.
I don’t want to take the original bet, but would be willing to bet at 1:1 odds (stakes of $10-$100) that more than 1/3 of respondents will disqualify themselves by answering the control questions “wrong”.
Any takers?
I think part of the issue, at least for me, was that the original post seemed to claim that the framing was dishonest (coupled with a comments section largely discussing those lying liars at the nyt, and comparing this article disfavorably with pizzagate) when to me it was well within bounds for an advocacy piece. Granted, this it is not how I would characterize the report. But I also wouldn’t have a problem if the NYT or NR had an article with your framing.
To riff on your analogy , if the question was ‘would a solar subsidy decrease climate change,’ and the break down was 35% agree, 45% uncertain, 20% disagree. I’d characterize a National Review headline saying something along the lines of ‘economists on solar subsidy: not buying it’ as legit even though it’s not how I would frame the study.
“when to me it was well within bounds for an advocacy piece. ”
Does the piece claim to be advocacy? Or is it masquerading as a neutral summary of relevant economic consensus?
Opinion may be a better way to phrase it.
I read it before reading Scott’s post and just viewed it as blog post type content, like 538 or the Corner.
I think in terms of phrasing the question for the quiz, you should ensure that “there is no consensus” is a viable answer. This appears to be a reasonable way to interpret the original data, even if the NYT article doesn’t really make that clear.
I would like to just quickly point out that not responding affirmatively to the question posed in the article’s quoted survey is not the same as supporting school vouchers.
If I were posed that question I would probably answer uncertain. Students may get a better education or an equivalent one. There’s a small possibility that they would get a marginally worse one.
However I strongly support school vouchers (at least compared to the current system) because I think they could deliver that roughly equivalent education at a much lower cost.
So even if we get past the way the article reported the statistics, I think the choice of reporting responses to that particular question as reflecting economist’s opinions on school vouchers overall is pretty darn misleading (though possibly unintentionally so).
I wonder why it matters so much whether economists prefer school vouchers. What about educators? Or eligible voters? Or maybe it doesn’t matter so much who prefers them, but whether student performance is better– or something else significant is better, on average, in areas that have school vouchers? And how much it costs to have them vs. to continue the current school system. If it’s more expensive and has no performance results to show for itself, maybe it does not matter so much who prefers them. Because, given that information, all people who prefer not to pay extra taxes for nothing, would then be against them.
Also, I wonder, if you asked economists what their political orientation was, would that predict their preference for school vouchers or lack thereof? If so, you may be just measuring the political orientation of the economists surveyed, on average.
As Scott points out (but with less emphasis than I think it deserves), Ms. Dynarski’s claim is directly contradicted by her source.
IGM poll results:
[emphasis added]
Responses:
Agree or Strongly Agree
44%
Uncertain
34%
Disagree or Strongly Disagree
5%
No Opinion or Did Not Answer
15%
Claim by Ms. Dynarski:
“Only a third of economists on the Chicago panel agreed that students would be better off if they all had access to vouchers to use at any private (or public) school of their choice.” [emphasis added]
To support her claim, Ms. Dynarski links to the IGM poll, but to a different question, wherein only 36% (i.e. roughly “a third”) of the respondents agreed that “students would receive a higher quality of education” with vouchers. [emphasis added]
Ms. Dynarksi’s choice to use economists’ opinions about the effect of vouchers on the “quality of education” as a proxy for their opinions on the effect of vouchers on student welfare, particularly when the same source provides a direct measure of those same economists’ views on the voucher / welfare question, is misleading.
While I agree that it was not the most accurate way to frame the study. I find the harsh criticism of this piece kind of baffling for two reasons.
First, many of the people criticizing it treat it like a front page news story when it should rather be read as a nyt sponsored blog post. Rightly or wrongly this type of content, like newspaper opinion pieces, is a ‘take’ on a issue of public concern. Her take is that economics (both the practitioners and the discipline) show that “Unswerving adherence to free, private markets will not solve the problems faced by our education system.” This take is framed in the context of Trump’s pick for Sec. of Ed., who is gung ho for privatization/vouchers (which, it should be noted, the author praises in piece). In ‘takes’ like this one it is, in my opinion, common practice to read and cite evidence in such a way that is bolsters the point being made. And, it is perfectly acceptable to point out that the evidence could be interpreted a different way, and that this second way is better for xyz reasons, but I don’t think it’s fair to call the author dishonest, or to call it really poor journalistic practice. In sum, criticism is certainly warranted, but Scott’s post presented this as the NYT misreporting a study to promote an ideological view, and I think the framing is very different if it is looked at as a blog presenting a slightly biased view of a study.
Second, THIS ISSUE IS NOT IN ANY WAY CENTRAL TO HER POST! (sorry for yelling) She used the report as a counterpoint to economists views on Uber (90% thought it made consumers better off). It reads almost as well if you substitute in 45% for 33%, because it is compared to the previous 90% on the Uber question, and used to support the notion that economists are “less optimistic about unfettered markets in education.” This point is being made by the author because she says that “you might expect economists to support DeVos'” radical privatization. It’s worth noting that the next time she talks about vouchers she praises them for the impressive results they achieved in Mass., but notes that you need government oversight of the schools to ensure quality, which DeVos opposed. She also discusses college loans by private banks and why moving in that direction is such a bad idea.
All in all, it just seems like a massive kerfuffle over a small issue
That fifth point, about specifying “most” students and getting vastly different results, is incredibly damning. The economists surveyed raised a concern about the survey design, it was addressed in a followup, and the results changed substantially in the face of a better-worded question. Which is to say, the initial data is pretty dubious and the followup data should be given far more weight.
That this was not emphasized, or even mentioned, ought to be a powerful indictment of the article. At best, someone grabbed the first data at hand without looking for newer or better sources. At worst, someone dropped the followup for not conforming to narrative.
Either way, it convinces me that this is not just an ambiguous headline but a serious error of journalism.
Hi Scott,
I like the survey you propose in your 12/31 7:10pm comment. It reminds me of “experimental philosophy,” where you use survey methods to probe people’s intuitive judgments about cases. I do a little Xphi myself, so I ran your survey exactly as you worded it.
I did it on MTurk, and solicited 40 responses.
I used the first question as a comprehension check, retaining 35 subjects who correctly answered “a.”
On the key question:
32 of the 35 answered (b), 2 answered (a), and 1 answered (c).
In short, the data has spoken and it is clear that Dynarski’s text is quite misleading. This is pretty sloppy journalism and the NYT ought to correct the record.
What about the opposite bet: show them your summary of the data from the first article (“About twice as many economists believe a voucher system would improve education as believe that it wouldn’t.”) and ask them if, from that statement, the majority of economists believe that a voucher system would improve education.
I don’t think you’d take that bet against me, and I think you know full-well why – you’d lose.
What makes you think you are being any more honest than the times? The way you are setting this up, you are claiming to be empirically-minded, but you are deliberately setting up a bet that you will win.
But that bet is a strawman, because we all know how people parse data. You’re being righteously indignant, but you’re still not being honest about it with yourself.
The headline – that most economists don’t buy school vouchers – is correct. Most don’t buy it. They don’t feel that the evidence is strong enough to convince them. And given the bias in economic thought towards such systems, this should immediately give you pause when you suggest that the most accurate way to parse the data is that twice as many support it as oppose it, because you’re excluding the largest group.
If an idea is not well-accepted enough that a majority of people in the field accept it, that’s not something I would ever present in a particularly positive light in terms of widespread belief.