A couple of people have challenged my essay yesterday, saying that they didn’t find the article misleading (or that it was only very slightly misleading).
They argue that it pointed out that economists were overwhelmingly in favor of ride-sharing arguments like Uber, but only somewhat in favor of school vouchers. Therefore, it’s fair to say that “economists don’t generally buy school vouchers” in the same sense that they “generally buy” Uber.
This would have been a fine thing to say, but I think the article failed to make this point and instead phrased its argument in a way that made it unlikely to readers to conclude anything other than that economists generally were not in favor of vouchers.
First, I feel like you could write exactly the opposite headline. “Public School: Economists Generally Don’t Buy It”. You would cite the statistic that “only 19% of economists surveyed disagree with the statement that school vouchers would improve education over the existing public system”. Then you could explain some public choice theory about how economists believe government services generally do poorly. This would be exactly as honest or misleading as the existing article. Yet it would produce exactly the opposite impression in readers’ minds: the original article makes you think economists mostly oppose school vouchers, the changed article makes you think they mostly support them. If newspapers are allowed to interpret data so loosely that they can use it to draw two different conclusion precisely opposite each other, what’s the point of having data?
Second, the article uses economists “not buying it” as a segue into a description of why economic theory says school choice could be a bad idea. But it seems like the majority of economists are not convinced by this argument. That is, both the 1/3 of economists who agree and the 1/3 of economists who are uncertain don’t accept the author’s argument that economic theory proves school vouchers can’t work as obviously true (the uncertain ones may be uncertain whether it’s true or not). This gives it a credibility it doesn’t deserve.
Third, really, if this same article was on Breitbart, and it used a survey showing that 40% of economists supported climate change legislation, 40% were uncertain, and 20% opposed it, and it described this as “Economists Generally Don’t Buy Climate Change Solution”, nobody would think anything untoward had happened and they would all agree this was a perfectly fair and unbiased summary of the evidence?
Fourth, you can do this for anything because there’s no clear definition of “uncertain”. How sure does an economist have to be before she “agrees” with a statement rather than being “uncertain” about it. If you are 51% sure school vouchers help, are you pro-voucher or uncertain? What about 60% sure? What about 90% sure? Suppose that all economists believe with 70% probability that vouchers will be good. If your criteria for “certain” is “80% or above”, then as long as you separate them out into support/uncertain/oppose, you can “honestly” declare that “no economists support school vouchers” and convince everyone that it must be an economically absurd plan. Yet I would hope that on any controversial issue more complicated than Uber, most economists are at least a little uncertain about it. We shouldn’t view that as legitimizing us to say that the economic consensus is whatever we want it to be.
Fifth and related, if you look at the economists’ comments, a lot of the people who self-described as “uncertain” thought vouchers would be good on net, but didn’t like the question because they thought it implied that literally 100% of students would be better off. This was such a problem that the IGM redid the study a year later, this time asking whether vouchers would make most students better off. 44% of economists agreed, compared to only 5% who disagreed (again, 34% were uncertain). Weighted by confidence, >50% of the economists agreed that vouchers would improve things, compared to only 6% who thought it wouldn’t improve things – a difference of almost ten times more economists agreeing that vouchers would help compared to disagreeing!
In the face of all of this, the New York Times gives the field’s opinion as “Free Market In Education: Economists Generally Don’t Buy It”.
I’m trying to be more empirical these days, so if you disagree with me, let me make you a bet – and I seriously mean I’ll bet money on this if anyone wants to take me up on it. We find ten random people of ordinary intelligence and economics knowledge and show them this article. Then we ask them a question like “according to this article, what is the economic consensus on vouchers?” We assure them that this isn’t a trick question and they’re just supposed to honestly give the impression they get from the article. If they say something like “they’re generally against vouchers”, I win the bet. If they say something like “probably more economists support vouchers than oppose them, but many are uncertain”, you win the bet. I am willing to alter exact terms if you have a better idea. I am willing to make this bet at 10:1 odds, so if you think there is any chance at all this article is not misleading it should be easy money for you.
[EDIT: I am going to take a version of this bet with Noah Smith. I don’t need to take it with any more people. Offer is now closed.]
[EDIT 2: Noah Smith is apparently no longer willing to bet me on this, although he refuses loudly and at great length to say whether that means he now agrees with me that most people would get a false impression of economists’ position from the article. Is there anyone else who wants to take a bet on these terms or any others?]