Open threads at the Open Thread tab every Sunday and Wednesday

Debunked And Well-Refuted


As usual, I was insufficiently pessimistic.

I infer this from The Federalist‘s article on campus rape:

A new report on sexual assault released today by the U.S. Department of Justice (DOJ) officially puts to bed the bogus statistic that one in five women on college campuses are victims of sexual assault. In fact, non-students are 25 percent more likely to be victims of sexual assault than students, according to the data. And the real number of assault victims is several orders of magnitude lower than one-in-five.

The article compares the older Campus Sexual Assault Survey (which found 14-20% of women were raped since entering college) to the just-released National Crime Victmization Survey (which found that 0.6% of female college students are raped per year). They write “Instead of 1 in 5, the real number is 0.03 in 5.”

So the first thing I will mock The Federalist for doing is directly comparing per year sexual assault rates to per college career sexual assault rates, whereas obviously these are very different things. You can’t quite just divide the latter by four to get the former, but that’s going to work a heck of a lot better than not doing it, so let’s estimate the real discrepancy as more like 0.5% per year versus 5% per year.

But I can’t get too mad at them yet, because that’s still a pretty big discrepancy.

However, faced with this discrepancy a reasonable person might say “Hmm, we have two different studies that say two different things. I wonder what’s going on here and which study we should believe?”

The Federalist staff said “Ha! There’s an old study with findings we didn’t like, but now there’s a new study with different findings we do like. So the old study is debunked!”


My last essay, Beware The Man Of One Study, noted that one thing partisans do to justify their bias is selectively acknowledge studies from only one side of a complicated literature.

The reason it was insufficiently pessimistic is that there are also people like the Federalist staff, who acknowledge the existence of opposing studies, but only with the adjective “debunked” in front of them. By “debunked” they usually mean one of two things:

1. Someone on my side published a study later that found something else
2. Someone on my side accused it of having methodological flaws

Since the Federalist has so amply demonstrated the first failure mode, let me say a little more about the second. Did you know that anyone with a keyboard can just type up any of the following things?

– “That study is a piece of garbage that’s not worth the paper it’s written on.”
– “People in the know dismissed that study years ago.”
– “Nobody in the field takes that study seriously.”
– “That study uses methods that are laughable to anybody who knows statistics.”
– “All the other research that has come out since discredits that study.”

They can say these things whether they are true or not. I’m kind of harping on this point, but it’s because it’s something I didn’t realize until much later than I should have.

There are many “questions” that are pretty much settled – evolution, global warming, homeopathy. But taking these as representative closes your mind and gives you a skewed picture of academia. On many issues, academics are just as divided as anyone else, and their arguments can be just as acrimonious as anyone else’s. The arguments usually take the form of one side publishing a study, the other side ripping the study apart and publishing their own study which they say is better, and the first side ripping the second study apart and arguing that their study was better all along.

Every study has flaws. No study has perfect methodology. If you like a study, you can say that it did the best it could on a difficult research area and has improved upon even-worse predecessor studies. If you don’t like a study, you can say “LOOK AT THESE FLAWS THESE PEOPLE ARE IDIOTS THE CONCLUSION IS COMPLETELY INVALID”. All you need to do is make enough isolated demands for rigor against anything you disagree with.

And so if the first level of confirmation bias is believing every study that supports your views, the second layer of confirmation bias is believing every supposed refutation that supports your views.

There are certainly things that have been “well-refuted” and “debunked”. Andrew Wakefield’s study purporting to prove that vaccines cause autism is a pretty good example. But you will notice that it had multiple failed replications, journals published reports showing he falsified data, the study’s co-authors retracted their support, the journal it was published in retracted it and issued an apology, the General Medical Council convicted Wakefield of sixteen counts of misconduct, and Wakefield was stripped of his medical license and barred from practicing medicine ever again in the UK. The British Medical Journal, one of the best-respected medical journals in the world, published an editorial concluding:

Clear evidence of falsification of data should now close the door on this damaging vaccine scare … Who perpetrated this fraud? There is no doubt that it was Wakefield. Is it possible that he was wrong, but not dishonest: that he was so incompetent that he was unable to fairly describe the project, or to report even one of the 12 children’s cases accurately? No.

Wakefield’s study has been “refuted”. The rape study has been “argued against”.


I saw this same dynamic at work the other day, looking through the minimum wage literature.

The primordial titanomachy of the minimum wage literature goes like this. In 1994, two guys named Card and Krueger published a study showing the minimum wage had if anything positive effects on New Jersey restaurants, convincing many people that minimum wages were good. In 1996, two guys named Neumark and Wascher reanalyzed the New Jersey data using a different source and found that it showed the minimum wage had very bad effects on New Jersey restaurants. In 2000, Card and Krueger responded, saying that their analysis was better than Neumark and Wascher’s re-analysis, and also they had done a re-analysis of their own which confirmed their original position.

Let’s see how conservative sites present this picture:

“The support for this assertion is the oft-cited 1994 study by Card and Krueger showing a positive correlation between an increased minimum wage and employment in New Jersey. Many others have thoroughly debunked this study.” (source)

“I was under the impression that the original study done by Card and Krueger had been thoroughly debunked by Michigan State University economist David Neumark and William Wascher” (source)

“The study … by Card and Krueger has been debunked by several different people several different times. When other researchers re-evaluated the study, they found that data collected using those records ‘lead to the opposite conclusion from that reached by’ Card and Krueger.” (source)

“It was only a short time before the fantastic Card-Krueger findings were challenged and debunked by several subsequent studies…in 1995, economists David Neumark and David Wascher used actual payroll records (instead of survey data used by Card and Krueger) and published their results in an NBER paper with an amazing finding: Demand curves for unskilled labor really do slope downward, confirming 200 years of economic theory and mountains of empirical evidence (source)

And now let’s look at how lefty sites present this picture:

“…a long-debunked paper [by Neumark and Wascher]” (source)

“Note that your Mises heroes, Neumark and Wascher are roundly debunked.” (source)

“Neumark’s living wage and minimum wage research have been found to be seriously flawed…based on faulty methods which when corrected refute his conclusion.” – (source)

“…Neumark and Wascher, a study which Elizabeth Warren debunked in a Senate hearing” (source)

So if you’re conservative, Neumark and Wascher debunked Card and Krueger. But if you’re liberal, Card and Krueger debunked Neumark and Wascher.

Both sides are no doubt very pleased with themselves. They’re not men of one study. They look at all of the research – except of course the studies that have been “debunked” or “well-refuted”. Why would you waste your time with those?


Once again, I’m not preaching radical skepticism.

First of all, some studies are super-debunked. Wakefield is a good example.

Second of all, some studies that don’t quite meet Wakefield-level of awfulness are indeed really bad and need refuting. I don’t think this is beyond the intellectual capacities of most people. I think in many cases it’s easy to understand why a study is wrong, you should try to do that, and once you do it you can safely discount the results of the study.

I’m not against pointing out when you disagree with studies or think they’re flawed. I’d be a giant hypocrite if I was.

But “debunked” and “refuted” aren’t saying you disagree with a study. They’re making arguments from authority. They’re saying “the authority of the scientific community has come together and said this is a piece of crap that doesn’t count”.

And that’s fine if that’s actually happened. But you had better make sure that you’re calling upon an ex cathedra statement by the community itself, and not a single guy with an axe to grind. Or one side of a complicated an interminable debate where both sides have about equal credentials and sway.

If you can’t do that, you say “I think that my side of the academic debate is in the right, and here’s why,” not “your side has been debunked”.

Otherwise you’re going to end up like the minimum wage debaters, where both sides claim to have debunked the other. Or like that woman on Twitter, who calls a common position backed by leading researchers “well-refuted”. Or like the Federalist article that says a study has been “put to bed” as “bogus” just because another study said something different.

I think this is part of my reply to the claim that empiricism is so great that no one needs rationality.

A naive empiricist who swears off critical thinking because they can just “follow the evidence” has no contingency plan for when the evidence gets confusing. Their only recourse is to deny that the evidence is confusing, to assert that one side or the other has been “debunked”. Since they’ve already made a principled decision not to study confirmation bias, chances are it’s going to be whichever side they don’t like that’s “already been debunked”. And by “debunked” they mean “a scientist on my side said it was wrong, so now I am relieved from the burden of thinking about it.”

On the original post, I wrote:

Life is made up of limited, confusing, contradictory, and maliciously doctored facts. Anyone who says otherwise is either sticking to such incredibly easy solved problems that they never encounter anything outside their comfort level, or so closed-minded that they shut out any evidence that challenges their beliefs.

In the absence of any actual debunking more damning than a counterargument, “that’s been debunked” is the way “shuts out any evidence that challenges their beliefs” feels from the inside.


Somebody’s going to want to know what’s up with the original rape studies. The answer is that a small part of the discrepancy is response bias on the CSAS, but most of it is that the two surveys encourage respondents to define “sexual assault” in very different ways. Vox has an excellent article on this which for once I 100% endorse.

In other words, both are valid, both come together to form a more nuanced picture of campus violence, and neither one “debunks” the other. How about that?

This entry was posted in Uncategorized and tagged . Bookmark the permalink.

326 Responses to Debunked And Well-Refuted

  1. Tom says:

    I came to a similar conclusion over the years based on arguments on Facebook (the best place to argue) with a certain someone. He will use any study or blog post completely uncritically if it supports his side, otherwise spend hours refuting it. I know we all do it to some extent (I’ve been working on not doing it quite so much after reading LW and co), but he’s the master. This flow chart complements your article quite nicely.

    • Sally says:

      Interestingly, the foible of credulity you highlight is very much a part of our hosts modus operandi. Above, we see our host present a Wall Street op-ed with self selected signatories as being representative of the mainstream, with absolutely zero critical reflection, while at the same time labouring to refute critics of “The Bell Curve”.

      Historically, “The Bell Curve” marks the rebirth of genetic determinism, and the rebranding of scientific racism. Given how far and how fast science has progressed, invalidating almost all of the methodology of “The Bell Curve”, in today’s era it’s difficult to see how it can be used to illustrate anything more than the depths of entrenched bigotry.

  2. PGD says:

    This Bell Curve is, in fact, quite thoroughly debunked. Not the finding that there are racial differences in IQ test scores (this is well known), but the actual thesis of the book, that racial or genetic differences in IQ scores mean that environmental interventions addressed at differentials in adult success are useless. The book does not prove this as it claims, and the evidence it presents on the matter is so misleading as to raise questions as to whether the authors deliberately cooked the books.

    See for example James Heckman’s review of the book in Reason at the time it came out:

    The gets to an issue in the post above. The issue is not that there are two different sides that argue over whether something has been ‘debunked’ and the outside observer can only conclude that it has been well and truly debunked when an ex cathedra statement by the entire community of reputable scholars has been made. The issue is that there are two different sides, both with university-granted credentials, that argue over whether something has been debunked and *one is right and the other is wrong*.

    That is actually a much bigger problem.

    • Vaniver says:

      This Bell Curve is, in fact, quite thoroughly debunked.

      Whoa whoa whoa. If a few claims out of a set of claims are incorrect, then it makes sense to say “The Bell Curve contains known incorrect arguments: X, Y, and Z.” It does not make sense to use the word “thorough,” because then you will be mistaken for saying that the central claims, A, B, and C, are incorrect. For example, Charles Darwin’s On the Origin of Species contains several arguments we know to be incorrect: to call it “quite thoroughly debunked” would immediately signal that your opinion should not be taken seriously on evolution, since people would think that you meant that, say, natural selection was debunked.

      • Noah Siegel says:

        Origin of Species is a great example.

      • PGD says:

        Nope. The Bell Curve didn’t get attention because it was saying that there were racial differences in scores on IQ tests or that scores on IQ tests had some element of genetic heritability more than zero and less than one. Both of those are well known facts and would not have created much stir. It got attention because it was claiming that differences in *life success* were overwhelmingly genetically determined and could not be impacted by environmental interventions. The arguments and evidence used by Hernnstein and Murray to make this claim have indeed been ‘quite thoroughly debunked’ and are false. Heckman’s review is very polite but he obliterates this claim:

        “throughout much of the book, they equate ability and education and implicitly assume that the economic returns to ability drive the economic returns to education.
        On the empirical grounds chosen by the authors, this implicit assumption is false. Their own evidence (buried in Appendix 6), as well as a vast literature in empirical social science, clearly indicates that controlling for ability lowers but does not eliminate the return to schooling measured in terms of earnings. The evidence on this point is consistent across many studies. Controlling for their measure of ability, the returns to education sometimes fall by as much as 25 percent, but they never go to zero. Ability and education are not the same thing, and both have economic rewards.”

        In other words, the major environmental intervention used to impact cognitive and work performance in our culture — namely education — gets no more than 25 percent and likely less of its impact from pre-existing ability, at least the forms of preexisting ability (scores on IQ-type tests) that H&M are interested in.

        That is a thorough refutation of the fundamental thesis of the Bell Curve You can add to it that education itself has a far from perfect correlation with life success, and that IQ has an even lower correlation with adult metrics of success, certainly the independent impact is far less than 25 percent. The ‘ironclad genetic meritocracy’ argument of the Bell Curve has no real empirical support.

        • Q says:

          “The Bell Curve didn’t get attention because it was saying that there were racial differences in scores on IQ tests or that scores on IQ tests had some element of genetic heritability more than zero and less than one. Both of those are well known facts and would not have created much stir.”

          Try saying it at Amnesty International meeting in my country and you will see, if these are “well known facts and would not have created much stir.”

          • Anonymous says:

            The Overton window has moved since, and probably because of its publication. Also, the conjunction of acceptable ideas is often not acceptable. I’m not sure whether either of those ideas are acceptable today, but the whole is definitely much more than sum of the parts.

            Here’s an example: I think it is acceptable in America today to claim either that there is a racial gap on the SATs or that the SATs are an IQ test, but not both at once.

          • Anonymous says:

            Here’s an example: I think it is acceptable in America today to claim either that there is a racial gap on the SATs or that the SATs are an IQ test, but not both at once.

            Good point.

          • haishan says:

            I’m not sure why you’d need to assert both of those when you could just correctly assert that there’s a racial gap on actual IQ tests. But I guess that may be the point.

          • Anonymous says:

            Right, the brute fact that there is a gap on IQ tests is unacceptable.

          • RCF says:

            Asserting that there is a racial gap in IQ test scores is acceptable if it is done in a manner that conveys that the speaker considers this a shortcoming of IQ tests, rather than of those receiving low scores on IQ tests. It is acceptable to assert that IQ tests are biased, but it is much less acceptable to ask that those asserting that IQ tests are unbiased to actually explain what the word “biased” means in this context.

            It is also acceptable to mention other racial demographic phenomena, such as black people having lower average income, or black people having higher rates of imprisonment, as long as it appears to be a criticism of society, rather than of black people.

        • Jiro says:

          Both of those are well known facts and would not have created much stir.

          You have got to be kidding.

          If anything, they are well-known facts that nobody ever dares say except maybe buried in an academic paper and peppered with disclaimers.

          • Scott Alexander says:

            I kind of agree with PGD that Bell Curve doesn’t get any credit merely for saying IQ is partially genetic and races have different average IQ.

            Most people on the left I know agree that IQ can be part genetic, but that racial differences come from the other, non-genetic part.

            There are probably people who deny even those things, but given how much other stuff Bell Curve said I can’t consider it a focus of the book or of the resulting controversy.

        • Scott Alexander says:

          I don’t identify that position with The Bell Curve and it sounds sort of insane. For example, medical school exists. IQ can be 100% genetic and all success entirely meritocratic, but people who go to medical school will still make more money than people who don’t simply because they’ve been taught profitable skills.

          Are you suggesting M & H are denying that you can improve someone’s earning potential by teaching them a useful skill (including reading, writing, and other primary school skills) or is there something I’m missing here?

          (that’s not even counting the worry that a lot of education increases earning power by giving you access to places that require education – ie a job that demands a college degree, even if it’s in Underwater Basket Weaving.)

          (I realize I might be No True Scotsmanning the thesis until we get rid of the obviously false parts, but I feel like M & H would not have made such a silly mistake and must have had a more serious point, and I even have a feel for what that point is although it’s hard to express it directly)

          (also, see my comment below)

          • Anonymous says:

            The Bell Curve is a long book and there is more to “life success” than money. Mainly the book is about how IQ predicts everything, from car crashes to divorce.

          • DES3264 says:

            “Are you suggesting M & H are denying that you can improve someone’s earning potential by teaching them a useful skill?”

            It’s been over a decade since I read the Bell Curve, and I can’t find a concise quote for you on a reskim, but I think M&H’s claim is that society has already gotten all the benefit which can be gotten this way. The remaining people who haven’t learned the skills to get a middle class job are (according to M&H) people who aren’t intelligent enough to learn them.

          • DES3264 says:

            I see that you have described two of the key points of the Bell Curve below as

            “7. Most ambitious social interventions fail (ie giving people food makes them less hungry, but trying to improve people’s long-term life prospects can’t be done consistently in a cost-effective way despite many attempts).

            8. Seven is true probably because most of the things social interventions are meant to help are more strongly genetically/biologically than socially/environmentally determined, with IQ as the classic example.”

            What I wrote above is your points 7 and 8 applied to the intervention of teaching valuable skills.

        • Have you considered that the returns for education might be substantially down to signalling (consider the evidence here on a slightly different topic: In general there is quite strong evidence that life outcomes are strongly genetic. For example, lifetime income is 50% heritable, 50% non shared, roughly 0% shared enviro.

        • Leonard says:

          The Bell Curve didn’t get attention because it was saying that there were racial differences in scores on IQ tests or that scores on IQ tests had some element of genetic heritability more than zero and less than one. Both of those are well known facts and would not have created much stir.

          “Would not have”? Were you alive then? I assure you that even though race is tangential to the main topic of TBC, its statement of racial differences in intelligence was a — and probably the — major element of contention. Wikipedia has a summary of it. Yes, TBC’s scientific racism was very, very contentious.

        • Steve Sailer says:

          “Heckman’s review is very polite”


          Dr. Heckman is a great statistician but his biggest admirers do not call him polite.

          Heckman’s angry review in Reason can’t be taken at face value because Heckman had been intimately involved behind the scenes in helping Herrnstein and Murray prepare their manuscript, as shown by Heckman being the first individual thanked in The Bell Curve’s acknowledgments.

          In recent years, Heckman has stepped back from his Reason review and taken a more tempered view.

        • Ryan says:

          “The authors also disregard much recent empirical evidence by Richard Murnane and others that indicates the increasing returns to the measures used by Murray and Herrnstein account for only a small portion of the recent increase in the economic return to schooling. While payments for ability have increased somewhat in the past 15 years, there remains a substantial increase in the payment for education unrelated to the authors’ measure of ability.”

          That part is super interesting. As a result of Griggs v. Duke Power Co. industry in the US had to basically outsource aptitude testing to universities. This would seem to indicate the reliance is less than optimal, that people are receiving higher pay and more lucrative jobs because employers perceive their greater education incorrectly as a signifier of ability.

    • Anonymous says:

      Heckman says nothing remotely like that. Just as most reviewers of the book have not read it, most who cite that review have not read it. People who have read the book, or even just Heckman’s review, know what is the topic of the book. But if you want to grandstand to people who have not read the book, such knowledge is only a liability.

      In any event, Heckman has quietly come around to endorse most of the book, which is why his current topic is interventions affecting “non-cognitive skills.”

      • PGD says:

        I read the book and the review, the review, though politely written, is devastating to the core arguments of the book about genetic meritocracy. And of course the review simply serves as a pointer to a bunch of more in-depth empirical research. Heckman has certainly not ‘quietly come around’ to endorse the book, I wonder why you would make this claim.

        • Anonymous says:

          Have you read a word Heckman has written this century? No, don’t answer that — you’ve lied enough about Heckman.

          • Panflutist says:

            Could you back this up? I can’t find much by Heckman on the subject after 1999. There’s an interview at , but it’s not quite in your position’s favor:

            Region: In 1995, you wrote a very strong critique of The Bell Curve, Herrnstein and Murray’s book about IQ, genetics and ability, which argued that nature far outweighs nurture.

            Heckman: My review wasn’t as negative as those of others. I think the book was very important. It broke a taboo by showing that differences in ability existed and predicted a variety of socioeconomic outcomes. So, I thought the book was important in raising that issue, but it failed totally when it focused so much on genetic determination of ability. It had no hard evidence on genetics. The youngest person in the Herrnstein-Murray sample was 14 years of age at the start of the sample. By the time they are 14, people are pretty well formed and environments play a big role. The idea that a test score measured at age 14 was a good measure of genetic determinism is absurd.

          • Emile says:

            Panflutist: I wouldn’t waste much energy arguing with an Anonymous commenter, it’s too easy to sit back and ask for someone to jump through hoops when you run no risk to your reputation if you’re full of it…

          • Lady Catherine Buttington, Phd. says:

            For what it’s worth, Anonymous is correct that Heckman has more of a focus on non-cognitive interventions now. However, this is not an indication that he “endorses The Bell Curve.” One can subscribe to an environmentalist view of the black-white gap and still believe that interventions tend not to be effective in producing lifelong gains in IQ.

          • Anonymous says:

            His current topic is interventions to raise “non-cognitive skills,” ie, conscientiousness. He is not explicit about his beliefs, so I must deduce them his action, in particular his abandonment of his previous program of the effect of interventions on IQ.

            Lady, don’t put words in my mouth. You’re the only one in this thread talking about race.

          • PGD says:

            To be taken seriously, at some point you will have to actually cite to something that really supports your claim rather than just slinging abusive adjectives around.

            Saying that Heckman has somehow changed his mind on the lousy social science of the Bell Curve just because he labels all the non-IQ influences on income and other success metrics as ‘non-cognitive’ is absurd. That is a shorthand term for ‘result on another psychometric test besides an IQ test’, and refers to a broad range of mental abilities, including social skills, motivation, and organizational skills, that are not the particular type of abstract reasoning purportedly measured by IQ tests.

          • Steve Sailer says:

            Heckman had many chances to fix the “lousy social science” of The Bell Curve before it was published, as suggested by his being the first individual thanked in TBC’s acknowledgements.

    • Noah Siegel says:

      Heckman seems to be saying (1) Murray and Herrnstein use seriously flawed methods, (2) their policy suggestions do not necessarily follow from the evidence they present, and (3) there are better studies and policy suggestions that we should listen to instead.
      Which seems to fall into Scott’s category of “books, articles, and papers arguing that The Bell Curve is wrong, often in very strong terms.” Heckman might even be right (I have no idea), but that doesn’t mean he debunked it, it only means that he argued against it by pointing out potential problems.

    • cool cool cool says:

      That is not the book’s thesis. For example, the authors write:

      “If the reader is now convinced that either the genetic or environmental explanation has won out to the exclusion of the other, we have not done a sufficiently good job of presenting one side or the other. It seems highly likely to us that both genes and the environment have something to do with racial differences. What might the mix be? We are resolutely agnostic on that issue; as far as we can determine, the evidence does not yet justify an estimate.”

      The book is mostly about how intelligence affects life-outcomes not the causes of intelligence itself.

      • haishan says:

        Yeah, I’ve been reading over a lot of Bell Curve-related stuff, and while there are some people who mount serious arguments against the thesis that intelligence, as measured by IQ scores, has massive causal effects on all sorts of life outcomes, for every argument of that sort there’s 10 people attacking the claim that racial IQ differences are in part genetic. Those 10 people tar Herrnstein and Murray and their book as racist (not entirely unreasonably), but this means that (a) the fact that different racial groups in America have different mean IQ scores also gets put into the “racist” category, even though it’s, you know, a true fact; and (b) nobody engages the highly-interesting-and-impactful argument that group differences in outcomes are mediated by group differences in intelligence, because that was in The Bell Curve, and that book is, like, super racist.

        (now i’m worried i’m gonna get a reputation as the guy who shoehorns race into everything… I just think it’s interesting! I swear!)

        • PGD says:

          My objection to the book is based on the claim that IQ scores, whatever they measure, have massive causal effects on life outcomes. I think this is the core Bell Curve argument and what makes the thesis of the book so controversial.

          The argument that IQ scores have some component of genetic heritability and that blacks consistently score lower than whites on IQ tests is another matter. There is a massive amount of evidence for that. But the implications of that could range from rather insignificant to all-important.

          • haishan says:

            My objection to the book is based on the claim that IQ scores, whatever they measure, have massive causal effects on life outcomes.

            This is a respectable opinion, and smarter people than myself — SJ Gould, Cosma Shalizi — believe it and can cite evidence to support their position.

            I think this is the core Bell Curve argument and what makes the thesis of the book so controversial.

            I’m pretty sure this is just wrong. Almost certainly the plurality, and maybe the majority, of criticism of The Bell Curve focuses on the question of whether differences in IQ between racial groups are genetic. Maybe this is what Herrnstein and Murray really cared about; it certainly plays a role in their suggested policy prescriptions. But relatively few commentators deal with the thesis that IQ = intelligence and intelligence determines life outcomes to a great degree, because attacking the racist parts accomplishes much the same thing and is way easier.

            (It doesn’t help that H&M don’t [IIRC] ever talk about population-level interventions to increase IQ, like salt iodization, which makes their argument even more like “blacks are just awful, whites are awesome, nothing to be done about it, oh well.”)

          • stubydoo says:

            At first when I started reading this sub-thread it seemed like gloriously classic motte and bailey, but now it really does seem that PGD is just out on their own with their own point of view. I’m not entirely sure why PGD thinks the other attackers of The Bell Curve are with them in considering the IQ->life outcome nexus as the “central” and/or most controversial point of the book – perhaps it comes from a view of human nature more optimistic than mine. I’ve read several dozen attacks against The Bell Curve over the years and don’t recall seeing any evidence for it. Just now I did a quick google search on “Bell Curve debunked” as I was curious – but the first few hits didn’t really resolve the matter either way (the first link was actually from RationalWiki, but I’m ignoring that one as not representative – in any cases it reports a mix).

          • Ryan says:

            Another possible category of the debunked/refuted problem is the following:

            Paper A makes an illogical or otherwise incorrect argument that X is true. It does not follow that X is false.

            So, just because The Bell Curve made an erroneous argument in claiming IQ has a large impact on life outcomes, that does not mean IQ does not have a large impact on life outcomes.

            If you are interested in IQ’s impact on various life outcomes, I’d recommend:


            If you’d like a tl/dr:

            IQ has a wide range of impacts on life outcomes. In some cases it has a small impact (tendency to be law abiding), but in other cases it has a large impact (having a PhD or being an engineer).

          • Anonymous says:

            Is there any advantage of that paper over the book?

          • Ryan says:


            Assuming you were asking me. The paper is a chapter from a psychology textbook. It’s very accessible, designed for undergrads. And it’s a lot shorter.

    • Scott Alexander says:

      I’m not sure I understand entirely what you’re arguing, so let me post a couple of statements I identify with “The Bell Curve” and see whether or not you think they’ve been refuted:

      1. IQ is an excellent measure of what we usually mean by “intelligence” with little bias and likely biological correlates
      2. IQ is 50% to 80% genetic.
      3. IQ strongly affects life outcomes for some reasonable definition of “strongly”
      4. Racial differences in IQ are mostly genetic in origin.
      5. Racial differences in outcomes mirror racial differences in IQ, eg a black person with IQ X does about as well as a white person with IQ X
      6. Five can reasonably be taken to mean that the IQ differences at least partly cause the outcome differences.
      7. Most ambitious social interventions fail (ie giving people food makes them less hungry, but trying to improve people’s long-term life prospects can’t be done consistently in a cost-effective way despite many attempts)
      8. Seven is true probably because most of the things social interventions are meant to help are more strongly genetically/biologically than socially/environmentally determined, with IQ as the classic example

      Once you tell me which of these you do or don’t agree with, I can tell you if I do or don’t agree with you. If you don’t want to admit to believing some of these publicly, you can email me and we can have the discussion there.

      (curious what JayMan and SJW have to say about this as our resident experts from either side)

      • Eric says:

        Scott, I’d be curious which of those things you believe!

        I feel somewhat uncertain, but I think for each one if I had to bet I’d pick true rather than false. Would you?

      • Q says:

        I do believe 1, 2, 3, do not believe 5.
        For the rest, I kind of lean towards believeing it, and would bet my money on that rather than the oposite. But I try not to make these beliefs my identity yet, because I still can theoretically imagine the refutation of 4, 6, 7, 8.

        • Anonymous says:

          It is odd that you single out #5, because that is the item that is easiest to directly measure. Far ahead of the rest.

          • Q says:

            Maybe it is the wording:

            “5. Racial differences in outcomes mirror racial differences in IQ, eg a black person with IQ X does about as well as a white person with IQ X”.

            To me it sounds like “there is no racial discrimination” which is not true. Am I misreading the statement ? I expect, that given the same IQ, another ability or qualification, the white person will be a little better off than the black. Did the Bell curve book give the numbers which oppose me ? My thinking comes from the data from Slovakia and gypsy ethnicum, where e.g. phone-inquiring about the job does not get you an interview invitation, if you identify yourself with a typical gypsy surname. The minorities which have generally bad reputation are less demanded on the job market, and it affects also the intelligent and responsible individuals. That gives them worse negotiating position.

          • Anonymous says:

            Yes, I think you are reading too much into specific phrasing. Does “about the same” mean “exactly the same”? It’s true that it’s phrased differently from the rest of the items, which tend to say “most of the difference,” but doesn’t that context suggest it is supposed to be like the rest, not unlike? Surely 5 is intended to be the correlational version of 6: just as 6 says “most of the difference” is causal, 5 surely means most of the difference is predicted by IQ. Indeed, since 6 is deduced from 5, if the stronger version of 5 were intended, why would it not be followed by the stronger version of 6?

          • Q says:

            ” Does “about the same” mean “exactly the same”? ”

            I am not able to answer such question. I even do not want to bind myself by giving specifications, like one standard deviation or whatever, maybe it is unfair from me ? I expect a consistent disadvantage for the black outcomes, I do not know how big, but, for instance, even the small difference in the *opposite* direction would just absolutely totally surprise me. (Unless it is in the segment of population, where affirmative action has the strongest consequences, for instance in the academia, government jobs etc.)

            O.K., for the sake of the game, let me bet, that given the same intelligence, the difference in outcomes is statistically significant and it shows the disadvantage for the blacks. You made me curious by saying this is the most measurable parameter. Did the Bell curve actually measure that ? Would I lose my bet ?

          • JK says:

            Q, I linked to this paper below. See figures 1 and 2 (note that the first figure is income x IQ, the second job status x IQ, regardless what the captions say). Blacks have higher occupational status than whites throughout the IQ range, and they also have higher income throughout the range except for the three lowest deciles where whites have an advantage.

            The Bell Curve (Chapter 14, see here) found that controlling for IQ eliminates most black-white gaps in labor market outcomes in young adults, but that IQ explains only a part of black-white differences in various family-related outcomes like illegitimacy.

          • Steve Sailer says:

            “To me it sounds like “there is no racial discrimination” which is not true.”

            There’s racial discrimination in both directions, as the careers of the President and First Lady would suggest.

        • Anthony says:

          As I recall – not going to look it up right now – H&M said that blacks have a slightly *higher* income than whites, after controlling for IQ.

          They said that might be a statistical artifact, or a historical one (blacks were more likely to feel compelled to work more), or because there are other factors besides IQ which affect income, and blacks may do better than whites on those factors. (Consider that taller men make more money than shorter men, all else being equal, and that black men are slightly taller than white men on average.)

          • Anonymous says:

            (In America, the average black man is a half inch shorter than the average white man. I think the gap is larger for women.)

          • Troy says:

            “After controlling for IQ, the average black made 98% of the [average] white wage.” (p. 323 of the Bell Curve)

      • PGD says:

        5 is definitely wrong.

        I also disagree with 1 and 3 but the problem is that the subjective terms ‘intelligence’ and ‘strongly’ make it very difficult to reach consensus on what you are arguing about. I think IQ tests measure a particular type of abstract reasoning that is just one component of things the brain does that make human beings good at achieving things. Common sense observation of how people perform on tasks and how they judge each other’s abilities supports this, as does social science research that shows a relatively weak correlation between IQ and adult success *after controlling for other factors* such as education. (Herrnstein and Murray had to more or less doctor their analysis to conceal this consistent finding, which is what Heckman argues in the review). Likewise, I think that evidence shows that the claim in 3 is wrong but it depends entirely on one’s definition of ‘strong’ vs. ‘moderate’ as there is certainly a statistically significant impact of IQ on adult success that is large enough to make IQ worth thinking about.

      • haishan says:

        Six is pretty much wrong on its face, right? It’s saying “these things are strongly correlative, so we can assume a causal relationship.” There are other causal models which would also explain #5; for instance, if your parents’ income determined both your IQ and your life outcomes, we’d still see most of the racial gap in outcomes disappear when controlling for IQ. (There’s likely effects of this sort going on, but this model is vastly oversimplified and doesn’t actually work; it’s meant to be illustrative.)

        • Anonymous says:

          Your model is completely falsified by the book. People go on and on about how the causality in the book is terrible and how there are many models that they didn’t consider, but the one they did consider is the one you mention, and they blew it to smithereens. Which you clearly know, because of your parenthetical. How can you have it both ways, both claiming to give a model that does something and refusing to take responsibility for it doing nothing?

          • haishan says:

            I… don’t think I ever seriously proposed parental income as the main underlying causal factor? I gave it as an example because it’s very salient and very plausible-sounding, which is presumably also the reason that The Bell Curve looked at it. The point is that Herrnstein and Murray don’t actually do any causal inference and so concluding point 6 based on point 5 is irresponsible — maybe IQ and outcomes are both caused by astrological sign, to choose a deliberately silly example.

            For what it’s worth, I do weakly believe that group differences in intelligence cause at least some of the observed group differences in outcomes, but this has a lot to do with the fact that I have a high prior for that hypothesis and very little to do with inferential statistics.

          • Anonymous says:

            Maybe Scott’s phrasing of 6 as purely a consequence of 5 is irresponsible. But that has nothing to do with H&M. What do you mean by “causal inference”? Can you really give a definition broader than randomized interventions that excludes them?

          • haishan says:

            I’m not an expert (and there are experts here who could certainly do better), but I understand there are statistical techniques that can learn causal models from data under certain conditions. Not as well as randomized controlled trials, but much better than linear regression. Clark Glymour discussed the causal problems with The Bell Curve, although with the benefit of sixteen extra years of data it’s kind of hilarious that Glymour states that “compensatory efforts [like affirmative action] come… too late to make a difference to most” while full-throatedly supporting Head Start.

          • Anonymous says:

            If you point to an algorithm, you could define causal inference to be that algorithm and only that algorithm. That would mean that RCTs aren’t causal inference, which is surely a reason to reject the definition. Indeed, when you talk about methods being “better” and “worse” you seem to be conceding that the book does causal inference.

            And, yes, I have read Glymour. He does not justify your slander.

          • haishan says:

            If you point to an algorithm, you could define causal inference to be that algorithm and only that algorithm.

            I guess you could do that, but I’m quite obviously not. Maybe I’m not making myself sufficiently clear, but at this point I’m more inclined to wonder if you’re arguing in good faith.

            Anyway, I’m not remotely interested in lawyering the definition of “causal inference.” The important point is that, as I understand it, H&M’s statistical methods aren’t strong enough to let them conclude that “the IQ differences at least partly cause the outcome differences.” Correlational methods do give some weak evidence for this hypothesis, but they also give weak evidence for hypotheses like “ice cream consumption causes murder.” It’s just not enough.

      • I’m not a good source, because I only read the beginning of _The Bell Curve_, but for what it’s worth, the thesws I got from doing so were not on your list. They were:

        1. Society has become much more meritocratic over time, with the result that social status correlates much more closely that it once did with ability. Harvard graduates are considerably smarter than graduates of Mississippi state not, as was once the case, just richer. This is in some ways a dangerous situation, since it means that the higher status people have good reason to believe they are smarter than the lower status people, reinforcing their natural tendencies to such a belief.

        2. One consequence of a more meritocratic society is more assortive mating by ability, which will tend to widen the ability distribution, causing additional problems.

        Perhaps I should have finished the book—it was a long time ago and I no longer remember what distracted me—but that was what I got out of the beginning of it.

      • Steve Sailer says:

        “5. Racial differences in outcomes mirror racial differences in IQ, eg a black person with IQ X does about as well as a white person with IQ X”

        The Obamas provide a fun anecdotal test of that, especially since Barack reviewed “The Bell Curve” for NPR in 1994:

        Please note, however, that this question has been studied more than anecdotally. It has been subjected to numerous reviews using gigantic longitudinal databases that track nationally representative samples of 10,000 or more individuals for decades. The Bell Curve used the NLSY79 database, which is still continuing, tracking thousands of children of female participants. There is also the NLSY97 database, whose participants are into the primes of their careers by now.

        There’s no shortage of data, there is just a shortage of data debunking The Bell Curve’s conclusions on this question.

      • Steve Sailer says:

        Scott says The Bell Curve implies:

        “4. Racial differences in IQ are mostly genetic in origin.”

        It’s worth quoting Herrnstein and Murray exactly:

        “If the reader is now convinced that either the genetic or environmental explanation has won out to the exclusion of the other, we have not done a sufficiently good job of presenting one side or the other. It seems highly likely to us that both genes and the environment have something to do with racial differences. What might the mix be? We are resolutely agnostic on that issue; as far as we can determine, the evidence does not yet justify an estimate.”

      • Ryan says:

        I’m posting this in several replies because I think it’s a great resource for anyone interested in the issue:

        Social Consequences of Group Differences

        You said “IQ strongly affects life outcomes for some reasonable definition of strongly.”

        Point of fact, IQ affects *some* life outcomes strongly (eg educational achievement), and some outcomes weakly (eg tendency to obey the law). IQ seems to affect every life outcome to some degree, but in many cases the effect is quite small.

    • JK says:

      PGD cites one article by James Heckman which disputes some aspects of The Bell Curve, while ignoring the two consensus statements by dozens of experts cited by Scott that endorsed all the main arguments of the book. Yet s/he thinks that The Bell Curve has been “quite thoroughly debunked.” PGD is a perfect example of the sort of biased thinker Scott bemoans.

      Heckman’s article is not very good and it is full of dubious arguments:

      1) Herrnstein and Murray (H&M) never claimed that IQ is the only cause of any outcome difference, only that it is generally the most important one and more important than childhood socioeconomic status. The fact that much variation is left unexplained after controlling for IQ is not problematic for H&M’s thesis. Also, H&M never said that the effect of environment is zero — in fact, almost all of their analyses use childhood socioeconomic status, which is at least partially an environmental factor, as an explanatory variable.

      2) It is false to infer that because ability differences do not account for all of educational differences that therefore the rest is explained by non-heritable factors. Other heritable factors than IQ, such as conscientiousness, also explain outcome differences.

      3) Regardless of to what extent IQ or some other important variable is genetically or environmentally influenced, if we do not have means to alter the variable, it is effectively immutable. H&M’s view of the efficacy of early childhood interventions is much more pessimistic than Heckman’s for a good reason. More on this below.

      4) Heckman claims that the numerical operations test is the single best predictor of earnings of all the ASVAB test components and that H&M’s emphasis on g is misplaced. I checked this using the NLSY79 data and found that, firstly, numerical operations is not the best predictor (it’s generally among the three best though, depending on year), and that, secondly, it is quite highly g loaded (0.740). More importantly, after the early follow-up rounds of the NLSY79 (when many participants were still finishing their studies), the AFQT composite score, which is an estimate of g, emerges as the best predictor of earnings and accounts for most of the predictive validity of the individual tests, including numerical operations.

      If you’ve followed Heckman’s research over the years, you must have noticed that The Bell Curve fundamentally changed his thinking. In a 2005 interview, he admitted to being “a bigger fan of [the book] than you might think.” Moreover, in recent years Heckman has more or less conceded that H&M were correct in their pessimistic assessment of the efficacy of early interventions when it comes to boosting IQ. Instead, Heckman now promotes the idea that interventions should concentrate on altering non-cognitive traits. Murray and others are pessimistic about this idea, too. I’ll explain why.

      Among typical results of the Perry Preschool Project were such findings as that 77 percent of the treatment group graduated high school compared to 60 percent of the control group, whereas 36 percent of the treatment group had been arrested 5 or more times by age 40, compared to 55 percent of the control group. The Perry study is the absolute gold standard of childhood interventions, heavily promoted by Heckman and others. Yet the actual effect sizes are rather modest and even if they justify the intervention in terms of cost-effectiveness, they in no way refute the pessimistic asssessment by H&M, even if we accept the results at face value. Yet there are many reasons to believe that the results of intervention studies cannot be taken at face value:

      1) The most heavily promotes studies are small-N. Not only this indicates that the reported effect sizes must be taken with a grain of salt, but it also means that it would be difficult to scale them up while maintaining the same level of competence and enthusiasm in the staff.

      2) There are technical problems in the studies. The Perry experiment, for example, suffers from imperfect randomization.

      3) The most heavily promoted studies reporting the largest effect sizes are old and their applicability to today’s society is doubtful, e.g., the Perry study begun 50 years ago the Abecedarian study 40 years ago. Interventions are meant to compensate for deficiencies in the home environments of poor children, but as the general level of prosperity and education rises, interventions are expected to lose their effectiveness, which is exactly what has happened.

      4) Two large new randomized experiments, namely Tennessee’s Voluntary Pre-K Program and the Head Start Impact Study indicate that early childhood intervention programs have essentially no effect.

      I could go on, but I think I have made it clear that it is laughable to argue that The Bell Curve has been “quite thoroughly debunked.”

      • jtgw says:

        Thanks, JK. I’m not even going to involve myself in the specifics of the arguments, and just point out how PGD obviously failed to distinguish between “debunk” and “disagree with”. He may disagree with the Bell Curve, but he doesn’t get to use the word “debunk” unless he can demonstrate that the consensus of experts has turned against the book, in the way that the consensus of medical science has turned against Wakefield’s thesis about vaccines and autism. This he conspicuously failed to do by citing just one expert who disagreed and who didn’t even publish his counter-arguments in a professional social science journal but in a popular news magazine.

      • Hainish says:

        ) The most heavily promoted studies reporting the largest effect sizes are old and their applicability to today’s society is doubtful, e.g., the Perry study begun 50 years ago the Abecedarian study 40 years ago. Interventions are meant to compensate for deficiencies in the home environments of poor children, but as the general level of prosperity and education rises, interventions are expected to lose their effectiveness, which is exactly what has happened.

        Both Perry and Abecedarian are outliers in that they provided explicitly academic instruction. Later pre-school programs, including Head Start, moved away from this, which may account for their lowered effectiveness.

    • Shenpen says:

      Except that is NOT the actual thesis of the book. The actual thesis is that IQ is a better predictor of success today than inherited wealth – that the world is turning intoan IQcracy. Which means helping low-IQ kids works better than redistributing wealth.

      The racial angle was NOT important, introduced only in the last chapters, yet somehow everybody focuses on that.

    • Steve Sailer says:

      Heckman’s angry review in Reason two decades shouldn’t be treated as definitive. Heckman himself has given up trying to refute the Bell Curve and is not trying to outflank it with a focus on character education:

  3. Randy M says:

    “but most of it is that the two surveys encourage respondents to define “sexual assault” in very different ways.”

    This is pretty much the argument against the 1-in-5 statistic from the right–that an expanded definition of rape or sexual assault is used to capture a much larger percentage under a term traditionally, and generally still by the public, held to be violent (in the literal, physical sense) and unambiguous, creating an impression of universities as being uniquely nightmarish, with open assault occurring literally every day, rather than a few rare examples of this amid a miasma of changing boundaries, unclear signals, and regret.
    (That current campus cultures is a bad thing or not is one debate, but whether people should hold the two as interchangeable is another entirely).
    It is the misleadingly use of the older studies that claim the 1-in-5 are raped statistics that I think is usually said to be debunked.

    I will also note that it is quite hard to write about this clearly, due to the wide use of terms such as ‘assault’ and ‘violence’ in the non-literal sense. Perhaps ‘made to feel uncomfortable’ is now a valid use of the word ‘violence’, but it would be useful to be able to have a quick word which separates this from being held down, punched, and etc.

    • ATairov says:

      > Perhaps ‘made to feel uncomfortable’ is now a valid use of the word ‘violence’, but it would be useful to be able to have a quick word which separates this from being held down, punched, and etc.

      That’s *why* people are trying to use violence to mean “made to feel uncomfortable.” They want to use the social and emotional baggage attached to the word for their own purposes. Since that’s what is happening, any replacement word for “violence” will just have its definition muddied again. Better to refuse to accept the redefinition.

  4. primality says:

    According to Exhibit 33 in the study linked in the Vox article, out of women in the general population who’ve experienced drug-facilitated or incapacited rape, only 45% would classify their experience as rape . Does that not mean the legal definition of rape is too broad?

    • Anonymous says:

      Possibly, if we consider having the law match to whatever the majority opinion of English words is as a high priority. However, I am not sure that striking part of the definition of rape from the law and then having to go back and put in a new law about drug-facilitated-or-incapacitated-involuntary-sex-which-we-no-longer-call-rape would be a very productive or useful thing to do compared to leaving the situation as-is – and not having legal recourse available for those women who *do* consider it rape would seem pretty unconscionable to me.

      • Randy M says:

        We invented the word manslaughter, and various subcategories of murder. If the law doesn’t do the same for rape, I’m surprised.

        • Anonymous says:

          Indeed it does: most states have a lesser crime of sexual assault. But note that the difference between murder and manslaughter varies between the states. And neither manslaughter nor sexual assault are part of normal usage.

          • Randy M says:

            I think, should the occasion warrant it, most people could find the words “manslaughter” “negligence” or “justifiable homicide” to describe something they were hesitant to call murder, but I may be granting the average person more savvy than warranted.

          • Anonymous says:

            My guess is that <10% of people would choose to use those words without prompting by others' use, though they understand them when others use them.

        • John Schilling says:

          The law should have established such subcategories for sexual misconduct long ago, but I’d be very surprised if it did so any time in the near future. How does one even propose such a thing without being branded a Male Rape Apologist?

          • William O. B'Livion says:

            In some states (at least) they have.

            Here in Colorado there are at least three different crimes, “sexual assault” (18-3-402) which would commonly be called “rape” (and includes both the notion of “statutory rape”, and a middle ground where it’s a misdemeanor (at least as I read the law) instead of a felony). There is “Unlawful Sexual Contact” (18-3-404) which covers a wide range of things that most people would think *should* be illegal, but wouldn’t really fall into the same bucket as rape (for example coercing a child to engage in sexual acts with a third party for your own sexual gratification). And finally there’s things like indecent exposure (technically a sex crime, and sometimes it is, other times it’s just not being able to find a bathroom in time).

          • mugasofer says:

            >How does one even propose such a thing without being branded a Male Rape Apologist?

            Well, one could always … be a woman? That seems like it should do the trick, at least offline.

            Not the most helpful advice, admittedly …

    • Protagoras says:

      The legal definition of rape should be based on what kind of behavior we think deserves that level of punishment, not the responses of individual victims. And anyway, 45% is a lot of traumatized victims; even if we assume that the 55% who say it wasn’t rape mean that they’re just fine with everything that happened (which seems unlikely), I don’t see it as likely that those who employ the drug-facilitated or incapacited target strategy will ever be particularly motivated (or able, even if they did care) to figure out who the 55% are and make sure to only target them.

      • Cauê says:

        Assuming it’s a “strategy” is part of the confusion. These studies (and/or the reporting) are also picking up a lot of innocent behavior.

        Much of it through taking a much lower level of “intoxication” as a threshold for “unable to consent” than people would normally agree with (sometimes any level at all). Also considering intoxicated women unable to consent, but not equally intoxicated men.

        And much of it through widening the research to include not only violence but also things that are clearly not violent, then packaging it all under some broad term. The aggregate number is then reported by the media as the number of occurences of the central, aggressive behavior (I’ve already talked of this one here, but I do love the table on page 6:

    • grendelkhan says:

      If you dig a bit more, it seems like what’s being captured there is “yeah, they had sex with me when I didn’t want it and I was too out of it to stop them, but they didn’t rape me” on one end, and “yeah, I had sex with them when they didn’t want it and they were too out of it to stop me, but I didn’t rape them”, which sounds like we’re carving reality at the correct joints.

      • fubarobfusco says:

        Let’s imagine a poor-ass junkie has broken into my house and stolen an expensive thing to sell for drug money.

        I could choose not to press charges. I could forgive them. I could rationalize that they needed the money more than I needed that expensive thing, because the world sucks when you’re a poor-ass junkie. I could decide that pressing charges would be more trouble than it’s worth. I could be depressed and think that I deserved to have my expensive thing taken away. Or whatever. The fact that someone robbed me does not entail that I have to give a shit.

        Nonetheless, it’s still robbery. It’s not like that poor-ass junkie asked me if I would let them come in and take the expensive thing. And it’s not like I would have said yes! (Though I might have given them a cheese sandwich or something. In the world where I’m extra super virtuous I might have helped them get in touch with some kind of rehab.)

        But they really did take my expensive thing without my consent.

        And if I were to say on a crime-victimization survey, “No, I have not been robbed,” I would be telling a lie. I have been robbed; I just don’t care.

        • John Schilling says:

          Grendelhan, and probably many of the original study’s respondents, is talking about “want”. Your example, and the traditional definition of rape, are about “consent”. These are not the same thing; it is entirely possible for a person to consent to a thing they do not want. See, e.g., taxation. Or for that matter most of the compromises necessary for human social interaction.

          And I think you also stack the deck by casting the rapist-analog as a “junkie” who one would presumably neither want nor consent to even being present in one’s home. I believe most of the campus incidents being characteried as rapes involve social acquaintances who do want to interact with one another at some level.

          So how’s this: A casual friend, or maybe a friend of a friend, is at a party at my house. A book on my shelf catches her eye; “Do you mind if I borrow this?”

          We both know that unless I object, I’ll probably never see the book again. And I don’t want that. But I don’t object. I don’t say a word, except maybe a noncomittal “Eh”. Possibly because raising an objection might endanger something I value more than the book, the evening’s polite sociability, or the friendship of our mutual acquaintance. Possibly because I’ve had a few drinks and didn’t really take notice.

          Am I the victim of theft? Does it matter if I don’t claim to be or believe myself to be the victim of theft?

    • Anthony says:

      Something not brought up in the discussion following is that some women (and presumably some men) will get drunk to a point where they’re clearly “partially incapacitated” in order to lower their inhibitions to the point where they’re willing to have sex. There’s another form of this where a woman (usually) will get drunk so that they can psychologically absolve themselves from any responsibility for having sex, in the event there are negative consequences.

      So if a woman who has done one or both of these things is asked on a survey “did you have sex when you were too incapacitated to give consent”, she will, if honest, say “yes”. But if the experience wasn’t too negative, she will probably say she wasn’t raped, even if legally, she was.

  5. onyomi says:

    I think one also has to take into account that, even assuming people take the time to carefully read studies and look into the opposition, they are going to have a different standard for what counts as “thoroughly debunked,” depending on their worldview.

    Like, if I see a study defined “had (consentual) sex when both partners were mildly intoxicated” or “had sex when I didn’t really feel like it” as “rape,” then personally I am going to throw out the whole study in my mind, because I see it as disingenuously using an overly-expansive definition of rape (one which doesn’t accord with common usage of the word, given that it results in myself having been “raped” in completely non-traumatic, non-criminal ways several times in my life) in order to arrive at a high number.

    An academic debate may continue to rage on after that, but in my mind, that one flaw rises to the level of “thoroughly debunked.” And this is a big problem, also, because once a particular figure has been “poisoned” in such a way, I must admit I am going to be super skeptical of any future studies showing one in five have been raped, even if they used a more restrictive definition, just because the mind says “oh, I know they can only have arrived at that number by playing with definitions.” This has happened in my mind, to a perhaps less justifiable extent, with studies about the male-female pay gap. (Frequently “debunked” by pointing out that it disappears if you account for women working more part time jobs, etc. without considering why they do this). This last thing, of course, is a bias of my own which I should try to be aware of, but it’s an incredibly difficult mental shortcut to avoid.

    But for some people, intoxicated sex is rape, by definition. They might see this study and say, “what a good job these people are doing in making us aware of how much more prevalent rape is, part of the work of which is showing people how many ‘rape-y’ things happen–even if many wouldn’t call it that, they should.”

    About campus rape–I tend to side with those that say: “if it amounts to a criminal rape, then just call the police; don’t call the guidance counselor,” but this goes against the culture at many, if not most universities, which is of a kind of all-inclusive summer camp/resort for young adults. The school is, for some reason, expected to take care of all of the needs of its students, rather than just educating them and letting them do their own thing, otherwise. Like, if your company provided you housing, and in that housing you died of alcohol poisoning, no one would blame the company. Yet with schools we somehow expect them to manage every facet of students’ lives.

    I like the sense of community at many colleges, but I wonder if there isn’t a happy medium?

    • Protagoras says:

      Are there any studies which define it as rape when two partners have consensual sex when both are mildly intoxicated? I’ve encountered a tiny handful of people who squirm and weasel over the issue of whether mild intoxication always negates consent (they don’t seem to want to say outright that they think it does, but they strongly imply it), but you find all sorts of strange extremists on the internet. I haven’t heard of any academic studies which use the standards of those people as their guide.

      • onyomi says:

        I have read elsewhere that one of the questions on the now (in)famous Campus Sexual Assault Study questionnaire was “Has someone had sexual contact with you when you were unable to provide consent or stop what was happening because you were passed out, drugged, drunk, incapacitated, or asleep?”

        Looking at the study itself,

        I am hard pressed to find where, if anywhere, they list the exact questions asked on the questionnaire, but it seems that “incapacitated sexual assault” is a large part of where they get the “1 in 5” figure from. In the study itself, they define it as “UNWANTED sexual contact while incapacitated by drugs or alcohol,” but if the above question is the one they really used, then it doesn’t seem to state “unwanted.” Also, it seems to imply that it is always a sexual assault if someone is “unable to provide consent” due to drugs or alcohol, regardless of the state of the partner.

        The above question to me doesn’t read as “did anyone ever drug and rape you?” it reads as “have you ever had drunk sex?” It seems to me that people were answering the latter question but being counted as if they answered the former.

        There is also the problem of politicians and media (and, in fairness, the general public) glossing over nuance in the study. The study uses “sexual assault,” which seems to be a much more expansive category than “rape,” for example, but I think when most people hear “sexual assault” their brain just translates that to “rape.”

        • Protagoras says:

          You think a lot of people read a question that said “unable to provide consent or stop what was happening because you were passed out, drugged, drunk…” as asking whether they’d had consensual sex while mildly intoxicated? I admit that my experience as a college professor leads me to have some doubts about undergraduate’s reading abilities, but I really don’t see that.

          • onyomi says:

            The “stop what was happening” part I would say definitely sounds like rape, but the “unable to provide consent” part seems a bit vague to me–like, are you, by definition, unable to provide consent just because you are intoxicated?

            That said, the question and the study are not AS bad as I was led to believe, though I’m sure there are some radicals out there who define all intoxicated sex as “rape.”

            Goes to show I should read the actual study before forming an opinion, though it’s tough to do when the study strongly conflicts with a pre-formed opinion.

            Like, before having heard any statistics, say 12 years ago, when I was a college student myself, if you had asked me, “how many women your age do you think have been raped?” I think I would have guessed somewhere between 1 and 5%. So when you show me a study saying “almost 20%” my immediate reaction is to think “that CAN’T be right.” If I then read an article claiming that that statistic was only arrived at using a super-expansive definition of rape then that will generally be good enough for me to throw it out, though I should probably look into it further.

            But evaluating the methodology of studies on your own, especially when they’re not in your field, is extremely time consuming and difficult, especially given academ-ese, so I’m not sure that’s an entirely satisfactory answer, either.

            It doesn’t SEEM unreasonable to hope that reporters, whose job it is to condense and summarize info for consumption by the public, will actually look into the studies they’re summarizing, but given how partisan most reporting is nowadays, it seems like you really can’t trust them to have done that, or not to preferentially ignore or discount studies that contradict the desired findings.

          • Anatoly says:

            The exact quote at the article is:

            “…when you were drunk, high, drugged, or passed out and unable to consent, how many people ever had vaginal sex with you?”

            This has a very clear Boolean connective problem. That is, it can be read as “drunk, high, drugged, or (passed out and unable to consent)” or “(drunk, high, drugged or passed out) and unable to consent”.

            The latter reading seems to be what the researchers intended, but it seems very plausible to me that many people could read it as the former. In that case they will indeed answer affirmative for all the cases of drunk sex. I don’t see that as outlandish or improbable *at all*; what do you think? (for this specific formulation, not the one you responded to).

            I’ve seen the quote in this form many times, but I haven’t ascertained that it was indeed worded this way on the survey.

          • Liskantope says:

            I can tell you from my experience being in college and continuing to encounter college students who are part of what we might call “drinking culture” that they can be very liberal in their use of terminology describing drunkenness. They can describe having two or three drinks, and probably feeling mildly to moderately tipsy, as getting “drunk”, “shitfaced”, or even “wasted”. So I wouldn’t be surprised if some college students consider their experiences of mild intoxication as sufficient to be considered “drunk”. Of course that doesn’t account for the issue of consenting versus not consenting, which has even more capacity to be ambiguous.

          • youzicha says:


            Hm, that’s a good catch. (Indeed, if anything the punctuation is misleading, it reads as a comma-separated list of phrases, which encourages grouping the “and” with just the last item on the list). Apparently this is a telephone survey, which probably also doesn’t help with careful parsing.

            I guess the source of the quote is page 116 of this pdf
   which has the same phrasing.

            On the other hand, several of the quoted questions there say “remember, we are only asking about things you did not want to happen”. (They don’t quote the full survey; presumably there was some kind of preamble establishing this). So even people who respond yes because they had sex while drunk, rather than sex while unable to consent, describe “unwanted” sex — the numbers should not be inflated by “happy drunk sex”.

            The report was published in 2011. One would hope someone will pick up on the problem with phrasing and try to investigate if it’s an issue. (Edit: according to DES3264’s comment below about Schwartz and Leggett ‘s study, I guess it has been investigated.)

          • llamathatducks says:

            Anatoly: yes, this really bugs me too.

            Which is really a shame, because I am absolutely in the school of thought that holds that studies of rape prevalence must define their terms very clearly precisely because colloquial definitions of “rape” vary widely. So I highly appreciate the effort at precision. But why oh why must they have introduced syntactic ambiguity? Can’t they get a linguist on their IRB board to weed out ambiguity in survey questions?!?!??

          • AJD says:

            I’d be satisfied with getting a linguist on the IRB board so that there’d be a well-informed person reviewing linguistics research.

        • DES3264 says:

          I wouldn’t parse the question that way. I read “unable to provide consent because … you were drunk” as a description of how much drunkenness the survey is asking about: Not just an ordinary buzz, but so much that one couldn’t communicate a clear yes or no. By analogy: “Have you ever missed a day of work due to back pain that prevented you from getting out of bed?” would mean “have you ever had back pain so severe that you couldn’t get out of bed and work?” not “have you ever missed a day of work while you had back pain, and we are asserting that it is impossible to get out of bed while your back hurts?”.

          But I have seen other people who find your reading natural, so it is definitely something to check into. And it has been! Schwartz and Leggett 1999 (paywall) run a similar study with the altered question:

          “Have you engaged in sexual intercourse when you didn’t want to but were so intoxicated under the influence of alcohol or drugs that you could not stop it or object?”

          They got 17% yes answers (N=388) compared to Koss’s 15% (N=6159).

          References found via Ampersand’s Mary Koss tag. Ampersand’s coverage of this subject has always looked extremely thorough and precise to me, although along the theme of these last two posts I should acknowledge that I’ve never looked to see whether there is an equally plausible anti-Ampersand out there.

          • jtroll says:

            “Have you engaged in sexual intercourse when you didn’t want to but were so intoxicated under the influence of alcohol or drugs that you could not stop it or object?”

            This might still suffer some ambiguity. People might feel they couldn’t stop it “mentally” because they were drunk rather than “physically”.

            “Couldn’t stop it” could mean “I couldn’t stop myself”. “Didn’t want to” could mean “wouldn’t have sober”.

          • Randy M says:

            Heck, “didn’t want to” could mean “didn’t feel like it at the time, but was still willing to in order to make my partner happy.”

          • jtroll says:

            Wanted to add, and maybe this is just me, but there is something about this question that makes me inclined to think yes in response to it. Even after reading it many times.

            This may be in part because “have you engaged in sexual intercourse when you didn’t want to” can stand alone and suggest a yes, with tuning out after.

    • Panflutist says:

      Hmm… if consensual intoxicated sex is rape, are all parties involved both victim and perpetrator at the same time?

      • onyomi says:

        Theoretically, yes. But let’s be honest: men are still largely viewed as perpetrators of rape, not as victims, especially if the act involved a man and a woman. The only time men really get raped, according to the popular imagination, is by other men, usually in prison.

        That said, I don’t think most people, or the law, view consensual sex in which both parties were approximately equally intoxicated as rape; the question is, did some of those making the studies under question view it as such?

        • Q says:

          The study with broader (and hidden) definition of rape asks the question, whether someone had a vaginal sex, when you were so intoxicated as to be unable to give consent. What does that mean ? Unable to speak ? Able to speak, but unable to think clearly ? Having a little less boundaries than when sober ? It is probaly open to interpretation of the person, who is filling out the survey.

  6. BenSix says:

    One problem this leads to is that commentators publish response after response as if the last word necessarily entails righteousness. It’s like a Gish Grand National.

  7. anomdebus says:

    Situations are not always symmetrical regarding burden of proof. For an obvious example, in a criminal trial, the prosecution needs to prove beyond a reasonable doubt. The defense just needs to prove reasonable doubt. Note: there are similar rhetorical sins committed after these trials as well.

  8. vV_Vv says:

    Vox has an excellent article on this which for once I 100% endorse.

    From the article: “Previous studies have found that many college women who were victims of sexual assault don’t believe a crime was committed against them — even if what happened met the legal definition of rape. ”

    I think this phrasing is highly misleading: In order to determine that an event meets the legal definition of rape it is necessary that an investigation and a criminal trial with a conviction verdict took place.
    What these studies did, in fact, was to ask several broad and ambiguous questions about events which might be consistent with the legal definition of rape. Anybody who answered positively to any of these question was counted as a rape victim. That’s how they got these 1 in 4 / 1 in 5 results.

    The fact that the findings were so far off from the official criminal victimization statistics, and that most of the “victims” didn’t considered themselves so, and some significant fraction even went on to date their “rapists” after the fact, should be dead giveaways that the methodology of these studies was utter BS: whatever these studies measure it is not something that even approximates the definition of rape used both by common people and law enforcement professionals.
    The self-proclaimed rape experts, instead, explained away these inconsistencies by claiming that people generally don’t know when they are raped. That’s typical of radical ideologues of any political color: they know what’s best for you better than you do.

    • Anonymous says:

      There’s a difference between “meeting the legal definition of rape” and “meeting the legal definition of rape beyond a reasonable doubt.” And many other differences between “meeting the legal definition of rape” and your scenario.

      • vV_Vv says:

        And many other differences between “meeting the legal definition of rape” and your scenario.

        What do you mean by “my scenario”?

      • Caleb says:

        More like “meeting the legal definition of rape” is a null category. If you say a particular set of factual circumstances meets a certain legal definition, you have to include an evidentiary standard to make it a meaningful statement.

    • Liskantope says:

      I have sympathy with some of your points, but you sound way too confident in making the following assertion:

      The fact that the findings were so far off from the official criminal victimization statistics, and that most of the “victims” didn’t considered themselves so, and some significant fraction even went on to date their “rapists” after the fact, should be dead giveaways that the methodology of these studies was utter BS

      I mean, are you denying that many people stay in abusive relationships (abusive even by the more conservative definitions) for years and years while denying that they’re abusive, or repeatedly enter into abusive relationships?

      • vV_Vv says:

        I mean, are you denying that many people stay in abusive relationships (abusive even by the more conservative definitions) for years and years while denying that they’re abusive, or repeatedly enter into abusive relationships?

        No, but how frequent is it? Keep in mind that we are talking about hookups between upper-middle class college kids, not married couples with children from a low-class/high-religiosity background.
        Sure, some abusive relationships exist between college kids as well, but, evidence of the contrary notwithstanding, I expect them to be uncommon.

        • William O. B'Livion says:

          No, but how frequent is it? Keep in mind that we are talking about hookups between upper-middle class college kids, not married couples with children from a low-class/high-religiosity background.

          You may want to re-examine some of your facts and prejudices, it might help firm up your conclusions.

          • vV_Vv says:

            Sure. You just have to provide me with evidence. Otherwise I’m going on with my priors (which are actually posteriors already conditioned on the observation that most victims of violent crime are from poor and uneducated backgrounds).

          • William O. B'Livion says:

            Go read Theodore Dalrymple on abusive relationships and figure out just how much religion plays into it.

            Also re-check your stats on the economic status of those in college. “Upper middle class” is bs. Even 25 years ago when I was in there were–especially at the state schools–a wide range of socio-economic classes represented, and if anything the “upper middle class” was *less* likely to engage in random hookups, although possibly more likely to be exposed to the “greek system” which IME is unfairly stereotyped (not that they don’t do it, it’s just that the sort of behavior they are accused of is much more general).

          • Randy M says:

            What by Dalrymple do you have in mind?

          • vV_Vv says:

            Go read Theodore Dalrymple on abusive relationships and figure out just how much religion plays into it.

            Do you have something easily accessible online based on hard data? I’ve googled “abusive relationships by religion” and I’ve found some sites that claim religiosity has a part in victims staying in abusive relationships. However, these seem to be atheist websites, and they cite no statistics.
            Intuitively the claims certainly sounds reasonable, since high-religiosity sub-cultures tend to enforce very traditional gender roles, and high-religiosity itself correlates with low education and poverty.

            Also re-check your stats on the economic status of those in college. “Upper middle class” is bs.

            According to Wikipedia, the “upper middle class” is defined as the top 15% of the population by wealth. 32% of Americans have a Bachelor’s degree and 12% a Master’s and/or Doctorate and/or professional degree.
            Assuming that most upper middle class kids get a college degree, college should be largely populated by upper middle class, or at least middle class people. Poor/low-education/high-religiosity people are most certainly underrepresented.

    • Jacob Schmidt says:

      … whatever these studies measure it is not something that even approximates the definition of rape used both by common people and law enforcement professionals.

      Even accepting this as true (and, quite frankly, I don’t), why should I be beholden to the common and legal definitions? Both would have excluded marital rape as even possible, in some cases as recently as decades ago. Rape laws and public understanding of what constitutes rape has evolved quite extensively, and is still doing so. To simply declare “that’s not what rape is,” well definitionally you might be right, depending on the jurisdiction and surrounding culture from which one draws one’s definitions, but it doesn’t establish that the “not rape” isn’t a problem that needs to be addressed, and it certainly doesn’t establish that we ought not to change our definitions to include similar behaviour with similar effects on the victims.

      I don’t want to live in a world where one can “not rape” one’s spouse, or “not rape” someone incapacitated, or “not rape” one’s employees, regardless of your definitions.

      • vV_Vv says:

        Even accepting this as true (and, quite frankly, I don’t), why should I be beholden to the common and legal definitions?

        Because if we want to have a honest and rational discussion we should use words to maximize accuracy rather than trying to emotionally manipulate the other parties by using words with a strong emotional and moral connotation with an unusual meaning.

        When the rape culture culture apologists parade with their “1 in 5 woman are raped” signs, or write it in newspaper articles and blog posts, they don’t add the little subtext “according to our non-standard definition of rape”.
        It’s not even a motte-and-bailey, since they wouldn’t even back up to their non-standard definition when challenged. They will just say “studies have shown” and proceed to call you a misogynistic rape apologist if you dare to disagree.

        Both would have excluded marital rape as even possible, in some cases as recently as decades ago. Rape laws and public understanding of what constitutes rape has evolved quite extensively, and is still doing so.

        The definition of the word “planet” changed over time (is Pluto a planet?). Does that mean that I am allowed to call the Moon a planet? Maybe in the future the definition will change again to include the Moon, but this doesn’t imply that I would communicating accurately if I referred to the Moon as a planet today.
        This is obviously much more important when dealing with emotionally and morally charged words.

        I don’t want to live in a world where one can “not rape” one’s spouse, or “not rape” someone incapacitated, or “not rape” one’s employees, regardless of your definitions.

        All of these are already covered by the current common sense and the legal definitions.

        Something that is not always covered by the legal definition, depending on the jurisdiction, is rape of men by women: in some jurisdictions, rape requires that the perpetrator penetrates the victim: if a woman forces a man to penetrate her or perform some other non-consensual sexual act, it’s considered a lesser offense or no offense at all.
        It may be reasonable to call for the legal definition to be updated in this case, on the argument that it lagged behind our common sense definition. The case of the SJWs pushing for their definitions of rape is different since their definitions don’t reflect widespread moral intuitions.

      • Randy M says:

        why should I be beholden to the common and legal definitions?

        Indeed. When I use a word, it means just what I choose it to mean, neither more nor less.

    • Anonymous says:

      If even a victim herself (or himself) doesn’t think that there was a crime, then it is pretty clear that the law shouldn’t consider that particular situation to be a crime. That a victim considers what happened a crime should be a bare minimum that is necessary for an act to be qualified as a crime.

      • vV_Vv says:

        This is not necessarily true as the law punishes behaviors which are deemed socially unacceptable regardless of the opinion of the victim.
        This is most obvious with statutory rape of minors or mentally handicapped people, but it also includes cases where a mentally competent victim doesn’t want to push charges but law-enforcement authorities have ample evidence that an illegal act took place.

        In most cases involving mentally competent victims, however, the victim accusation constitutes a key piece of evidence that would make all the other evidence irrelevant if removed. This is because rape is defined in relation of the mental state of the victim: in order for rape to occur the victim must have been in the mental state of non-consent or unable to consent.

      • grendelkhan says:

        That sounds like a general justification for victimizing the severely disabled, or torturing people into agreeing that yes, they totally deserved and wanted it, or damaging your victims’ minds in some other way that they’re no longer capable of qualifying your actions as a crime.

        • Nornagest says:

          If you can figure out a reliable way of doing that that holds up after you’ve left a position of total immediate control, there are some governments that’d probably be interested in talking to you. Brainwashing is hard, even when you are a prison camp rather than a lone amoral fratboy.

  9. Anatoly says:

    The article you endorse 100% fails to mention that the CSA study: a) is based on just two universities, as opposed to NCVS which is based on a representative nationwide household survey; b) suffers from abysmal response rate, compared to NCVS.

    It also presents the NCVS questions in a misleading way. It makes a big deal of the fact that NCVS questions use “rape” explicitly and the CSA questions don’t, and suggests this as the major difference. But of the two NCVS screener questions that have to do with sexual assault the lengthier one only talks about “(Other than any incidents already mentioned,) have you been forced or coerced to engage in unwanted sexual activity…” – which seems quite broad. The Vox article puts this question right next to the rape one, as if they follow each other, but they do not, they’re quite separate and the rape one is an item in a longer list. I found the full(er) list here:

    And then, if a response to the screener question triggers further questions on this topic, the NCVS methodology says it’s very possible for them to determine rape/sexual assault even if the victim does not refer or think about it that way. So it just isn’t true that if, for example, a student was raped while incapacitated but doesn’t consider that rape, it will not be counted under NCVS. It will not be if they don’t even consider that as “forced or coerced to engage in unwanted sexual activity…”, but if they do, presumably the NCVS researchers look at the fuller details and count it so.

    I’m not saying that the differences in the context of the study (crime vs public health) and the wording of the questions aren’t important; they probably are. But I’m not very impressed with any article that fails to mention the huge differences in the nature of the sample and the response rate. It’s possible that the x10 difference between CSA and NCVS is explained merely by the title of the study and the wording of the questions, but just assuming that to be the case is not a serious response to this huge difference.

    • Randy M says:

      “is based on just two universities”
      I thought I had heard that, but it seemed like an important enough point that Scott would have mentioned it.

      Also, the fact that this is a survey means it is not wholly reliable (for either study). People are self-justifying, so regret may turn to victimization in the remembrance. “I can’t believe I did that” may latter translate into actually not believing that one did it.

      • Protagoras says:

        People rationalize in lots of ways; “I can’t believe they would do that” may turn into actually not believing they did it. You’re right that surveys are unreliable, but you imply that they’d only be unreliable in one direction, when in fact (unfortunately) they’re unreliable in lots of different ways, such that we can’t really predict in which direction the results are likely to ultimately be skewed by all the competing biases.

        • Randy M says:

          Yes, that’s true, although given the zeitgeist, I think a recent study based on recall would tend to exaggerate.

          A more reliable, if not all-encompassing, statistic would be “how many women experience sexual violence severe enough to seek medical attention?”
          I imagine a fairly reliable number could be found, which would not be measuring all bad things that happen but would be reliable for what it is.

          • fubarobfusco says:

            A more reliable, if not all-encompassing, statistic would be “how many women experience sexual violence severe enough to seek medical attention?”

            People who have been sexually assaulted may seek medical attention for STI tests, emergency contraception, or for that matter psychiatric treatment — even if they don’t have the kind of injuries that would be central examples of “severe violence” to a lot of folks.

          • Randy M says:

            I would include those. The key is that they would be verifiable by a third party and recorded after the incident rather than later during a survey.

    • Liskantope says:

      Yes, this is the gist of the criticism of the CSA study that I usually hear. It was only done on two campuses, and the small number of participants suggests that perhaps the few who volunteered to reply to the survey tended to be the ones that felt like they had something of concern to report. I also recall hearing that the study is rather dated, but at the moment can’t find the year in which it was conducted. (One of these days, I need to gain some basic skills at doing online searches and identifications for studies on topics I’m interested in researching.)

      That doesn’t mean that the results should be considered “officially put to bed” in the sense that they don’t shed light on anything worthwhile concerning our use of language in discussing sexual assault, which the NCVS survey neglects to take into account.

      Really I suspect that something more basic is going on here, on top of what Scott is shedding light on in some of his recent posts. When multiple studies are conducted on a controversial issue over a number of years, the one that gains the most attention and is retained in the most minds will probably be the study with the most sensational results. So when the “1 in 5” result became the most quoted statistic, I sort of assumed that it didn’t necessarily come from the most rigorous study, but probably the study whose conclusion simply returned the greatest frequency for sexual assault. Thus, perhaps the most quoted study concerning a controversial social issue should a priori be considered to provide an upper (or lower) bound for the relevant statistic. A secondary effect of this phenomenon is that once the conclusion of the most-quoted study becomes generally accepted “fact”, the next most quoted and widely believed study is likely to be the one whose conclusions are most extreme in the opposite direction… and thus, it may appear to many that the first study has been “debunked”.

    • grendelkhan says:

      the CSA study: a) is based on just two universities, as opposed to NCVS which is based on a representative nationwide household survey; b) suffers from abysmal response rate, compared to NCVS.

      The experimental design that Koss et al. started with (using the Sexual Experience Survey or similar instruments) has been replicated; the ones I’m aware of are the National Violence Against Women Survey, the National Women’s Study, the Sexual Victimization of College Women Study, and the National Intimate Partner and Sexual Violence Survey.

      It could be that the NCVS primes people to think about crimes, whereas the SES-type instruments prime people to think about their sexual history. (The SVCWS suggests that the explanation is the more specific nature of the acts described in the SES.) But there really is something being measured here. You may think that it poorly reflects the underlying reality, but it’s not just an artifact from one poorly designed study.

  10. aesthete says:

    Scott, I think it’s easy to steelman the “debunked” argument:

    1) Study A is used by out-group as evidence for claims 1, 2, and 3.

    2) Study B or Article C shows that Study A does not provide evidence for these claims.

    3) Therefore, it is not justified to adjust one’s priors for claims 1, 2, and/or 3 based on Study A, because either the study itself was flawed or out-group drew erroneous conclusions.

    Generalized arguments of this form are applicable in your example about campus rape:

    1) The “1 in 5” study is used by feminists to as support for their claim that there exists a mass societal “rape culture” — one which is particularly prevalent in colleges. These studies are used as support for feminist priors RE: credulity of any allegations of rape and its prevalence on campus, as in the case of the Rolling Stones’ fraudulent reportage of Jackie’s reported rape narrative.

    2) These studies use definitions of sexual assault which do not correspond to “rape” or even to unsolicited sex, and as there are no questions asking this in a straightforward manner there is no way to tell how much of this population would in fact consider themselves to have been raped or sexually assaulted instead of having had sex they later regretted or that was enjoyed and desired by both parties. This is without noting questions regarding the representative nature of those surveyed. (if you follow one of the linked articles on The Federalist’s OP, you’ll see this case is made: The DoJ study uses a methodology more in line with what you or I would consider to be sexual assault, and reports an incidence rate of 6.1 per thousand — orders of magnitude lower than 1 in 5.

    3) Therefore, one should not update one’s priors to match those of feminists on this issue.

  11. Glen Raphael says:

    To be fair to the “conservative” side of that “primordial titanomachy”, the exchange didn’t stop with Card in 2000. Neumark and Wascher came back with subsequent “no, we were right after all!” papers in 2003 (“The average minimum wage effects we estimate using this sample are consistent with the view that minimum wages cause employment losses among youths”) and 2012. These later studies both argued against the validity of earlier work (“Our evidence points to serious problems with these research
    designs.”) and claimed to find different results (“We conclude that the evidence still shows that minimum wages pose a tradeoff of higher wages for some against job losses for others”).

  12. Josh says:

    For the minimum wage argument, “does raising the minimum wage hurt businesses / employment” is a stupid question. A much better question is “under what conditions does raising the minimum wage hurt businesses / employment?” The former leads to ideological gridlock;the latter leads to learning something about the economy.

    For many of these warring studies situations, I feel like the fundamental problem is not so much that people have a hard time weighing confusing evidence, it’s that the question is mis-framed such that there really is no right answer and therefore the evidence will always be somewhat contradictory and confusing. It seems like a lot of research into complex systems we don’t really understand that well (the economy, the human mind, etc) consists of trying to isolate a variable and make some kind of claim about the role that variable plays, relative to the researchers’ notion of what notmal conditions are. Whereas any given variable tends to not be terribly explanatory on its own — the interesting thing is how it interacts with the rest of the system in what ways. Which leads to much less dramatic headlines, but much more useful results.

    • Alex Godofsky says:

      A much better question is “under what conditions does raising the minimum wage hurt businesses / employment?”

      All of them.

      • Protagoras says:

        I’ve mostly stayed out of this, because it is such a complex issue, and while I’m one of those who believes that the labor market is distorted in such a way as to produce lower than optimal wages, I’m not at all sure minimum wage is the best way to produce it. But I do have to say that the level of dogmatism and endless repetition of econ 101 that everybody knows (and that clearly does not adequately capture the complexity of the labor market) has very negatively affected my opinion of the anti-minimum wage side.

        • Alex Godofsky says:

          and that clearly does not adequately capture the complexity of the labor market

          The entire problem with the exchange, the recurring problem with the exchange, is that this logic is doesn’t sustain the conclusions derived from it.

          We have a general theory that makes strong predictions about the sign of a particular variable in our model.

          We know that this most abstract model doesn’t fully capture every aspect of the system. This is true everywhere.

          We have many bells and whistles that can be added to the model, some of which accurate reflect these aspects and some of which don’t.

          However until we actually distinguish which bells should be added, and which should not, their existence can at best add error bars to our estimate. Error bars in both directions.

          If “the system is complicated” increases your prior that there should be a price floor, it should similarly increase your prior that there should be a price ceiling.

          And yet so far as I know we literally never hear anyone arguing for a price ceiling on low-wage labor. Implementation issues aside, you might think at least someone would mention, as a possibility, the idea that poor people were being paid too much and that for the market to properly clear there should be wage cuts.

          Further poisoning the discussion is the fact that the people with the incentive to bring up specific bells and whistles – the pro-min-wage – all have an incentive to only bring up those going in one direction.

          If you want to empirically study the issue further, you can’t do it with giant studies of existing min-wage implementations (as covered extensively in the previous post, those studies have basically 0 power). You have to look at each proposed feature of the model and try to study it individually to figure out whether it’s true. And you have to do so in a way that suggests you are sampling those features reasonably from the space, rather than only selecting those that would justify the min-wage rather than those that would oppose it.

          Finally, the market for low-wage labor actually is much lesscomplicated than most labor markets, and is much closer to the econ 101 perfect competition model.

          • Protagoras says:

            You keep repeating essentially the same points. I will try not to bring up points that have already been endlessly discussed. The suggestion that the labor market is artificially distorted in a way that adversely affects workers for no benefit is hardly original; Adam Smith firmly believed this to be the case (and cited some evidence, but of course our whole problem here is that every side can find some evidence to point to).

            Now, over the past few decades, wages for the lower income types have stagnated, minimum wages as a fraction of average income have declined, and the incomes of the very wealthy have increased dramatically. Perhaps minimum wage is a negligible part of what’s been going on (that’s one of my reasons for not getting too involved in this debate; I think it’s probably the wrong issue). But something has gone on. Two simple possibilities; perhaps the value of low wage work has declined while the value contributed by those at the top of the scale has increased. Or perhaps the change has only been in relative negotiating positions, with the very wealthy finding ways to divert more resources to themselves at the expense of others.

            An argument for the second theory, that it’s labor’s worsening negotiating position rather than labor becoming less valuable relative to the economic elite, is that economic growth was much greater in the mid 20th century, when labor was getting a larger share of the national income, than it has been recently, with the very wealthy getting a much larger share of the national income. The very wealthy don’t seem to be showing much sign of being hugely more productive.

            Of course you can argue with all of this (again, everybody has their studies), but could you perhaps find something new to say other than endlessly repeating econ 101 and assuming those you are arguing with are uninformed idiots?

          • Alex Godofsky says:

            You keep repeating essentially the same points. I will try not to bring up points that have already been endlessly discussed.

            I believe I’ve written two comments on this particular point, one in each thread, and I saw no one actually address the particular reasoning in the other one. But this is a significantly more nuanced position than the “lol econ 101 rulez” you complained about.

            The suggestion that the labor market is artificially distorted in a way that adversely affects workers for no benefit is hardly original

            This is not nearly the same thing as “the minimum wage is a good idea”.

            In fact, I think this is just totally orthogonal. You can set up a market in a way that has adverse effects for one side, without justifying an artificial price floor for that side. The word “adverse” here is too general.

            Now, over the past few decades, wages for the lower income types have stagnated, minimum wages as a fraction of average income have declined, and the incomes of the very wealthy have increased dramatically. Perhaps minimum wage is a negligible part of what’s been going on (that’s one of my reasons for not getting too involved in this debate; I think it’s probably the wrong issue). But something has gone on. Two simple possibilities; perhaps the value of low wage work has declined while the value contributed by those at the top of the scale has increased. Or perhaps the change has only been in relative negotiating positions, with the very wealthy finding ways to divert more resources to themselves at the expense of others.

            The data behind the measurements of increased inequality is incredibly controversial, and IMO most of it is so bad that, again, the endeavor is just totally sabotaged. I will refer you to this post as just a partial summary of issues with a significant portion of the data.

            As an aside, I also think that CPI inflation is an awful, awful metric for attempting to do comparisons of real value across significant amounts of time, and in fact there are very few good ways to do such comparisons other than manually trying to get people to reveal their preferences for such directly.

            But, as you observe, what does this have to do with the minimum wage?

            If there is some “unnatural” thing causing wages for low-wage workers, that doesn’t mean that a price floor will “make things right again”. It could just do further harm! You have a huge ways to go to show that the two harms counterbalance each other.

            And I think that you recognize this, with your parenthetical. But you still seem to be conflating the idea of “natural prices that would be set in a market without distortions” with “income that people deserve to receive”. There is no connection between market prices and people’s basic moral dues as human beings. But mucking around with prices almost never gets us into a world that better-satisfies the latter. The correct answer is, and consistently has been, a progressive system of tax-transfers layered on top of a market economy.

            This is the standard position of almost every conservative economist except for a small minority who strongly favor natural rights over the crude utilitarianism of most economics.

            The very wealthy don’t seem to be showing much sign of being hugely more productive.

            This sure smuggles in a lot!

          • AJD says:

            And yet so far as I know we literally never hear anyone arguing for a price ceiling on low-wage labor.

            Of course there’s a price ceiling on low-wage labor: it’s whatever the threshold is between a low wage and a medium wage.

            (…In other words, I don’t actually know what you even mean, since the obvious interpretation of that sentence is incoherent.)

            And you certainly do see people arguing for a price ceiling on high-wage labor, which seems like the actual symmetrical proposition.

          • Alex Godofsky says:

            AJD: I was hoping people would be charitable and not force me to elaborate on a philosophical distinction that isn’t very interesting or relevant, viz. that “low-wage labor” in this case refers to “the type of labor which is presently paid a low wage” e.g. being a cashier. Thus my parenthetical immediately following the section you quoted.

            And you certainly do see people arguing for a price ceiling on high-wage labor, which seems like the actual symmetrical proposition.

            Yes, and isn’t it curious that somehow all of their ideas come out resolutely in a particular direction? Surely something in the system ought to militate towards the opposite policy, i.e. put a floor on CEO pay and a ceiling on cashier pay. After all these markets are complicated, are they not?

            Whereas the side that is opposed to the min wage is saying “all of these effects combined are ambiguous and the data is terrible, so we should just not muck around with stuff”.

        • Q says:

          I am not from USA and never heard the term “econ 101” before. Googling it confuses me further. Can somebody explain what is it and, possibly, how the name was created ?

          • Randy M says:

            Collegiate courses are denoted by a three digit number, the first referring to the level the course is offered. A “101” course is an introductory course for first year students; Econ 101 is the most basic and fundamental aspects of the discipline.

          • Jadagul says:

            Most colleges identify courses by the name of the department and then a catalog number (so I can tell people, e.g., that I teach Math 114 to be precise). There are a bunch of different systems used, but almost all of them consider a higher number to be a more advanced course, in general.

            Further, the “traditional” and probably most common involves giving every course a three-digit number, with courses intended for first-years being in the 100s, those intended for second-years in the 200s, and so on. Thus 101 would be the lowest possible course number, and thus the introductory course. Psych 101 is the general survey intro-to-psych course, English 101 is the survey of English literature, etc.

            This is common enough that people will just write “Feminism 101” or “Diets 101” or “Hang-Gliding 101” to mean “an introduction to feminism/dieting/hang-gliding.” But in the case of Econ 101 it’s actually quite literal–Econ 101 is “the information contained in an introductory college econ course for freshmen,” as well as by analogy “basic intro econ principles.” This is extended sometimes, and someone somewhere else on the thread refers to “econ 201,” being “more advanced econ principles if you study it further than just the basics.”

          • Q says:

            @Randy M

            Thanks !

      • Susebron says:

        A terrorist cult has obtained nuclear weapons, and threatens to bomb New York City unless the minimum wage is raised.

        • William O. B'Livion says:

          I believe, in the spirit of the season the appropriate answer would be:

          The economy is frightful
          Your solution so insightful
          Since I’m not there now
          Let it glow, let it glow, let it glow..


      • Mary says:

        There have been times when minimum wage has been effectively been repealed by inflation: market clearing wages even for drudge work were higher.

  13. There’s an on-and-off project I’ve been working on, which is a Chrome plugin for highlighting/tooltipping particular sets of words or linguistic constructs in the browser. Some words or phrases are indicative of certain types of problems, and it would be nice to have them leap out of the page.

    A set of such words/phrases I’ve been thinking about recently are the science-y window-dressing words that appear in politically-charged public discussion on these subjects but rarely appear in reasonable, well-mediated discussion of same. “Debunked” is in that list.

    (It turns out I’m only a little better at coming up with lists of words than finishing personal dev projects.)

    • Anonymous says:

      I would pay for this.

    • Anonymous says:

      Paul Graham has a list which you might find useful as a starting point:

    • mjg235 says:

      Many would be covered by just using a regex, like /[a-zA-Z]+ist/. Although I would prefer the humorous route, replace any match with “bacon”.

      • Luke Somers says:

        baconen, my bacon is hurting from writing the bacon of things in baconory that’d false-positive on, you get the bacon?

      • What I have of it so far is regex-based, but in order to be selective enough to not pepper the document with false bacons, they usually end up being somewhat more sophisticated than you might think. The one you mentioned above, for example, would match with “guitarist”, “unrealistic” and “blistering”.

        In fact, inferring context of a matched regex pattern can make the regex non-performant, especially on a large document. Constructing a robust set of unit and performance tests for any given set of rules is part of my minimum feature set.

  14. Anonymous says:

    “when you were drunk, high, drugged, or passed out and unable to consent, how many people ever had vaginal sex with you?”

    If the CDC considers this a case of rape, then apparently when two drunk people decide to have sex with each other they have both raped the other person; mutual rape has occured. Somehow this doesn’t sound like a reasonable definition to me.

    • Liskantope says:

      …apparently when two drunk people decide to have sex with each other they have both raped the other person; mutual rape has occured.

      I have seen threads in forums where some of the commentators are arguing exactly this.

    • Anatoly says:

      There’s a thorough discussion of this quote in Alas! A Blog, where Ampersand points out (and backs this with a link to the NISVS survey text) that this question is immediately preceded by an introductory paragraph that makes it very clear that consensual drunk sex doesn’t count: “Sometimes sex happens when a person is unable to consent to it or stop it from happening because they were drunk, high, drugged, or passed out from alcohol, drugs, or medications”.

      I still think that the question itself is misleading and is likely to lead to some false positives even when preceded by this explanation, but it isn’t as bad as quoting the question alone makes it seem (and as I argued in a comment above, mea culpa).

      • Anonymous says:

        [the introductory paragraph] makes it very clear that consensual drunk sex doesn’t count…

        Does it? To me, that introductory paragraph sounds like “when you are drunk, you are therefore unable to consent”. Which, if I understand correctly, is actually close to the legal truth in many jurisdictions.

    • John Schilling says:

      The legal definition of rape is generally that intoxication negates consent if and only if it renders the victim incapable of understanding what is happening or incapable of resisting, or if the intoxication itself is involuntary. There’s no law that e.g. having sex with someone with a BAC greater than 0.08 is automatically rape, or anything even vaguely equivalent. Unless there are spiked drinks involved, the victim has to be unconscious or nearly so to have been raped in the legal sense. And it seems unlikely that two unconscious individuals are going to be having sex with each other.

      Of course, there are standards other than purely legal, and by some of them mutual rape may be possible. And, as others have noted, the Krebs “1 in 5” study has some amazingly poor and ambiguous language in that regard. In addition to the criticisms already raised, I will add that listing “drugged” and “drunk” as separate conditions from “incapacitated” and “asleep” strongly implies that it includes intoxication that does not include incapacitation or unconsciousness.

      Krebs looks very much like a study that was designed to encompass as many false positives as possible, and I think “debunked” is probably not too strong a word for it. That still leaves us looking for some better studies to converge on the right number; the DoJ study looks good but is only one study.

      • victoria says:

        Completely unconscious, no, but falling-down drunk to the point of nobody really knows what’s going on? That’s well within the bounds of plausibility to me.

        • jtgw says:

          But you agree it gets fuzzy at this point? How drunk are we talking about? Even mild intoxication lowers inhibitions. I suppose I’d be happy treating it case by case to determine whether someone was really so drunk she couldn’t reasonably be held accountable for her actions. Just having lowered inhibitions shouldn’t count; it should be more like you clearly have little awareness of your surroundings, even if you are technically conscious.

          • victoria says:

            I was talking really damn drunk — not so much “I’m not sure whether I’m good to drive” as “Standing up might not be such a good idea and I may not remember pieces of this evening tomorrow.”

            And yes, I do agree that things can get fuzzy. I think a lot depends on circumstances, too.

            I know more than one happily married-with-kids couple where a kid is nicknamed (privately between the couple or among friends) for the type of alcohol the couple were overindulging in on the night of the kid’s conception. Plenty of couples in functional marriages and LTRs seem to enjoy drunken/otherwise inebriated sex; are there people who view this as marital rape?

            Then consider a couple of cases of people who aren’t in longer-term relationships.

            Person A and Person B meet at a party, are attracted to one another, each get falling-down drunk of their own volition, and have sex. One or both of them realizes after the fact that they would never have done this sober. (I think that’s the textbook “fuzzy” case and personally do not consider this rape, nor do I consider either A or B to have acted immorally towards the other, though I do think they both acted imprudently.)

            Person A asks Person B out. B says “no”; the subject is dropped for a period of weeks or months and A and B become friends, during which time B shows no signs of reconsidering. B has a rough day; A suggests that they get drunk together, secretly hoping that B will initiate sex. A and B get very drunk and do indeed have sex. Is A a rapist? Even if this is not rape, has A done something wrong?

        • John Schilling says:

          If both of the participants are really that drunk, classic PIV rape is unlikely on account of the P is likely to be inconveniently flaccid. It’s not a sure thing; different corners of the nervous system are involved and individual idiosyncracy covers a broad range. So, maybe at the edge of the bounds of plausibility but not likely to greatly affect the statistics under discussion.

          Being drunk enough to know basically what’s going on at the time but not remember when sober, that’s a bit more plausible. Knowing, remembering, and lying about how drunk you were because the truth is embarassing or incriminating, that’s fairly common I think.

  15. stubydoo says:

    Another word used for similar situations like debunked/refuted is “discredited”. I remember running into that word a bunch of times reading 90’s vintage literature on the topic of international development/IMF/World Bank stuff. One great thing about saying “discredited” is that unlike debunking, it doesn’t generally require any actual evidence to come into play – the alleged “discrediting” can take the form of some party who finds someone’s claim inconvenient for some reason just saying they’re against it (which was sometimes the case in the literature I was reading), and the sympathetic half of the audience won’t even hesitate in nodding along. When you say “debunked”, there’s still an expectation of at least some type of scientific study.

    • Anonymous says:

      Oh, so many fancy ways to disagree with your opponents 😀
      I don’t even know which ones should I use 😀

      • stubydoo says:

        Well if your goal is to maximize my annoyance, then making dubious claims about debunkings and discreditings is still actually nowhere near the best/worst you can do. Sometimes I run into people describing the theory they oppose, then saying “…but it turns out that…” then stating a claim in opposition to it – however this “it turns out that” event they’re referring to is not any kind of observation of data, it’s just somebody making an assumption. Disguising your argument from authority as an argument from data – ugh.

  16. Bill Murdock says:

    The Law of Demand has “thoroughly debunked” the Card and Krueger study. The only thing at issue is “how much” raising a price floor on a good will reduce consumption of that good, or cause a switch to substitutes (like capital over labor).

    When the price of something rises, the consumption of that thing falls, ceteris paribus. That goes for things humans buy with money (and by extension, the effort/energy that went into “getting” that money) and for natural processes in the physical universe (like how much heat/energy it will take to melt something).

    If you drop something, it’s going to fall “down.” Now, two swallows may grip it by the husk and pull it up, but that just means you have to control for swallows, not that swallows have debunked gravity. Other than that, I enjoyed the article.

    • Auroch says:

      Except for Giffen goods, and Veblen goods, and potentially labor, whose properties on the demand curve are not well understood.Rules of supply and demand are good heuristics, but only heuristics, not laws. Not remotely.

      And if course even under normal-good assumptions, it still only works for certain under the assumption of a perfectly competitive market, which the labor market most definitely is not. There are far too many massive information asymmetries and power disparities in the hiring process.

    • Charlie says:

      Yep. Economics 101.

      Let me illustrate a scenario from economics 201 for you. Look at this diagram. The diagram was really hard for me to understand the fist time, so I’l try to provide an accompanying explanation:

      The price (S) of something (in this diagram, it’s the price of labor) goes up the more we want to buy. Because the price goes up as we buy more, it’s actually surprisingly expensive to buy 101 of something rather than 100 – we not only have to buy an additional 1, but we also have to pay a higher price on the 100 (In the case of labor, it’s like if you want to attract 101 construction workers, you have to offer a certain wage; even the 100 workers who would have come for ten cents less get that wage). The curve that reflects this surprising expensiveness is the marginal cost (MC). If our benefit from buying these things (MRP) goes down as we have more of them, then at some intermediate point the two lines intersect, at which point buying more of these things is no longer cost-effective and so we stop.

      A minimum price can only raise the price, but it can actually lower the marginal cost – the marginal cost rises quicker than the price because you’re paying the same price for every unit – but if the price is flat, so is the marginal cost. Thus in the diagram, the flattened marginal cost MC’ has a region where it’s lower than MC. If the intersection with MRP is in this region, then a mininimum price will increase consumption.

      Note: Monopsony effects have been shown to not be a very good explanation for real-world reaction to minimum wage laws, this example is largely just to illustrate how economics 101 contains fewer universal truths than physics 101.

      • Bill Murdock says:

        (S), in your own graph, is not the price of labor, it’s the supply of labor. The supply of labor is the relationship between the units of labor offered at different prices/wages.

        You’ve also got causality backwards. “Because the price goes up as we buy more…” No. The argument you’re defending is “Because the price goes up we want to buy more.”

        On a reality-based note, you can buy 100 units of labor at a price, and then after contracts have been signed, buy one more unit at a higher price. Neo-classical PPC models are useful sometimes, but if you use them to describe reality to yourself “you’re gonna have a bad time.”

        More later, but I’ve got something to do. Check back later if you’d like for a more thorough “debunking” of your comment. Instead of appealing to the authority of escalating our levels, we’ll just call it Economics. I promise to reply later today.

        • A monopsony is like a monopoly but the other way around. A monopoly cannot sell all the goods it wants to because at the profit-maximizing price for a monopoly, raised above the competitive price, the quantity demanded for the good in question is not as high as the quantity the monopoly wants to supply at that price. For a monopsony, the reverse is true–the profit-maximizing price is such that the quantity demanded for the good in question (in this case labor) is higher than the quantity laborers are willing to supply at that price.

          So in the case of monopsony a higher price does not make the firm want to buy more labor. What is actually going on is that at the profit-maximizing price of labor (no minimum wage), the firm already wants to hire more labor but cannot because at the profit-maximizing price the quantity supplied of labor is lower than the quantity the firm demands. So the minimum wage, by forcing the firm to raise the price it pays for labor, will cause the quantity supplied of labor to rise and therefore the firm will hire a greater amount (although if the minimum wage is too high this will not be true). So this is a case where the law of demand is perfectly compatible with the claim that a price floor can increase how much of something a firm buys.

          Most wikipedia articles on economics are pretty bad. The comparative advantage one is quite good, but it took significant work from ejmr. They also did the permanent income hypothesis, but unfortunately the bros seem to have stopped working on the project.

        • Charlie says:

          But we agree that “The only thing at issue is “how much” raising a price floor on a good will reduce consumption of that good,” is not necessarily true?

          Okay, great.

    • Intrism says:

      Empirical data cannot be “debunked” by theory.

      • Randy M says:

        Well, no, of course not, but empirical data is rarely as simple as “x=5”. Figuring out what the empirical data means is complex and open to interpretation, at least in most cases of interest. And when there area conflicting data sets and conflicting interpretations of each, having a plausible mechanism can help point to which set is more trustworthy.

      • satanistgoblin says:

        Well, if physics experiment shows results incompatible with speed of light being maximum, laws of conservation, etc, we’d be pretty justified with going with the theory and thinking that experimenters messed something up.

      • Bill Murdock says:

        Data means nothing and everything w/out theory. I expect better from the readers of this blog.

        You cannot interpret data w/out theory. Data just… is. And you never have it all. That is why you can’t do experiments in economics like you can in, say, physics, sometimes.

      • Anonymous says:

        LOL. So I guess we just accepted that blurred image from the camera as the empirical evidence and wouldn’t send someone out to clean the lens.

        Oh, and let’s not forget statistical significance.

        at 95 CI you still have that niggling 5% change of being wrong.

    • Anonymous says:

      Price isn’t value isn’t effort isn’t energy isn’t expectations isn’t desire isn’t demand isn’t price.

      Also gravity varies across the globe and more so in general in regards to orbital mechanics. “Stuff falls down” is only true in a very, very specific sense of “down”.

      Extrapolating to the rest of the solar system give you…
      Almost like extrapolatinh from one assumption of economics. Also, ceteris paribus only holds in theory -where we specifically use it because accounting for reality is *hard*

      • Bill Murdock says:

        So many to reply to, and it will take me a minute, but this – THIS – can’t wait 😉

        I thought about writing something more like “when you let it go it moves towards the greatest mass given …”, but then I was like, “This is a comment, and no one could be so obtuse as to…” I forgot this was the internet, so kudos.

        Also, “ceteris paribus only holds in theory”. There is no face, and no palm, big enough.

        • Anonymous says:

          So do you actually intend to reply to anything or is skirting the line between snark and sass just more fun?

          The entire problem here is the differing variations on theory, projections and evidence made either in favor or against a conclusion derived from said theoretical principles. Me objecting that the statement of all things otherwise being equal is a meaningless interjection when you are actually talking about reality is neither facepalm worthy or particular intransigent, it’s rather the entire point.

          I agree with you entirely that going just by the axioms of demand in a standard economic model, adding a price floor will cause a a change in the model. That’s how the interactions of those things work in that model. It is, exactly, like saying “things falls down”. To belabor the point, the basic interaction of that model based on the construction of the theoretical framework the model illuminates makes that an indisputable fact.

          But the world is not a standard economic model. And the axioms of demand in economic models are not mapable to physical processes except by intuitive kludge and a serious changing of terms. So saying that the law of demand makes statement X and disproves theories Y or Z is not a workable solution. There is no “law” of demand – there is the assumed, rational behavior of logical agents within an exchange system framework. It’s perfectly reasonable to assume that a higher price will decrease the demanded good across all sections of everything, sure. And here’s a few examples of that not happening, so what are we going to do now with our Giffin and our Veblen and our asymmetric labor and our gravity singularities and our positive transfer externalities that help offset increased wages or some glib idiot going on about aggregate macro-economic demand or short-term cyclic prosperity or real wage inflation offsets?

          So in the real world, where people sometimes do hilariously awful things, “all else being equal” is not – and this is kind of the salient point – going to be in the least bit bloody equal.

          It’s nice that you have a model that makes predictions. Those predictions are often useful. Now we’re going to have to sit here and hash things out until you tell me what to do once the model you have no longer fully account for all the observed phenomena we are actually seeing. Some other models try to (or do) account for that, and even if they’re wrong (I do not like the monopsony explanation for labour), at least they’re infinitely more useful than someone going:

          “I’m having some problems account for the limited infinities of aleph and of aleph-plus-one”
          “Well, 2 plus 2 is 4”
          “Err, what?”
          “In addition you add together the disparate sets and arrive at a total sum”
          “Yes, that is true, but I don’t see how that helps me here”
          “The Law of Addition states…”

  17. Tom Scharf says:

    Sigh…I hesitate to even post this.

    “There are many “questions” that are pretty much settled – evolution, global warming, homeopathy”

    Global warming is pretty much settled. If you state this in an Asperger’s literal interpretation mode of thinking, that the globe has warmed over the past 150 years, then you will get little argument from almost everyone, most skeptics included.

    But that is not what people actually hear, is it? The common interpretation of this is that global warming is “dangerous” and requires immediately and costly policy intervention. Pretty much settled?

    Why do I hesitate? Because it is impossible to have a grown up conversation about this subject. This statement by the author is a simplified assertion of what is in reality a technically complex subject with an equally complex set of policy alternatives. There is an entire spectrum of claims about “global warming”. This is a typical debate shutdown statement. Possibly he is only referring to whether CO2 is a greenhouse gas, whether the globe has warmed, and whether man is partially responsible. Fine.

    Environmental science puts out some of the worst propaganda of any science area, with food science (specifically anti-GMO) being a close second. There is of course some good science in both these areas but the media routinely gives the megaphone to outlandish claims more often here than other science areas. 40% species extinction. Miami underwater by 2100. 100M climate refugees. Human conflict. Every extreme weather event. Etc. Settled?

    You can’t just give a pass to this kind of stuff.

    • tom says:

      Indeed. I’d really like Scott approach the AGW subject with the same degree of scrutiny he applies to other areas. Here’s for example an interesting study one could do: take one year of issues of a leading journal in the area from 20 years ago, count the number of falsifiable predictions the papers made about the longer term future and see how many have been validated. The amount of pro-AGW propaganda should make one very cautious about the area.

    • stubydoo says:

      If you describe global warming is even that well settled, then you’re seemingly implying that the many, many, many people who insist that it is a “hoax”, instead of holding firmly to that stance will motte their way back to something along the lines of climate scientists’ attempts to estimate the earth’s dose/response function on CO2 have not yet produced reliable results.

      (I used “motte” as a verb. Neato)

      • I’ve spent quite a lot of time in online AGW arguments. There are a significant number of critics who either deny that global temperatures have gone up—the usual claim is that the data have been adjusted to create that illusion—or concede that it may have gone up but argue that there is no good reason to think that humans are responsible.

        On the other hand, people on the other side of the argument routinely treat every critic of their position as if he was making one of those claims. I had it done to me recently shortly after I had stated that temperatures had trended up and human production of greenhouse gases seemed to me the most plausible explanation—by someone who was explaining why it obvious wasn’t worth the trouble of looking at my arguments. His other explanation was that I was a libertarian, hence obviously too biased for my arguments to be worth looking at.

        The two serious criticisms, which almost never get responded to in the online arguments, are:

        1. The IPCC predictions are not very reliable and haven’t done very well in the past.

        2. Warming will have both positive and negative effects, the size of both is quite uncertain, and there is no good reason to expect the sum to be negative.

        • Harald K says:

          Your claim number one is meaningless unless you give some measure of how wrong IPCC has been.

          As I’ve said, I’d like every post in an AGW discussion to be annotated with the poster’s best guess of the climate sensitivity. If you agree with the IPCC that it’s about 3 degrees C, then you don’t really have much of a case.

          Your claim number two is in dire need for specification too, but it’s in any case pretty ridiculous. There’s no way a two degree increase in temperatures will not be disastrous. Climate change is already having expensive negative effects (ask a reinsurance company), and there’s no reason to think advantages should suddenly appear as it gets worse.

          But utility functions, and especially libertarian utility functions, can be pretty exotic. I suppose I can’t rule out that for the things you care about, it may be a net positive.

          • “Your claim number one is meaningless unless you give some measure of how wrong IPCC has been.”

            For some details see:


            But in any case, my point was not that the criticisms I listed were correct, although I think they are, but that they are more defensible than the claim that AGW isn’t happening, and people who offer them are routinely attacked as if that were their claim.

            “There’s no way a two degree increase in temperatures will not be disastrous.”

            You can find a summary of my discussion of ways at:


            “Climate change is already having expensive negative effects”

            The question isn’t whether there are any negative effects but whether there are net negative effects. One of the negative effects routinely predicted is a decrease in agricultural yields. So far they have been going up.

            So far warming has not produced droughts, or at least the fifth IPCC report retracted the claim from the fourth report that it did. So far it has not increased hurricanes. It has raised sea levels by about eight inches.

          • Harald K says:

            I checked out your blog post. You take the 1990 IPCC first assessment report’s absolute worst case scenario,that climate sensitivity is on the high end of the estimated range (4.5 C), and that there will be no successful attempts to limit emissions and they will continue to increase.

            But the actual emissions were much lower than IPCC’s “business as usual” scenario, and there was also a “best estimate” of climate sensitivity of 2.5 C. With the best estimate climate sensitivity, and emissions scenario matched to the actual emissions (as you should: the IPCC gave those scenarios for a reason, they’re not experts at predicting how much carbon humans will emit) then the model performs well.

            All you’ve proved is that

            1. Judging from the two preceding decades only, it doesn’t look like climate sensitivity is as high as 4.5 C.

            2. We did not emit as much greenhouse gases as we thought we would in 1990.

            Neither claim is controversial. I’m still waiting for your best estimate of climate sensitivity.

          • Jaskologist says:

            If we’re demanding annotations, here’s the two I want:

            1. What do you predict the global temperature will be in 5 years?
            2. What will it be in 10?

          • Harald K says:

            That’s a bad measure Jaskologist. For individual years, there will inevitably be big error bars on the answer. Climate sensitivity is better. But my best guess for the global mean temperature in 2020 is 0.5 C above the average for 1986-2005. It’s unlikely to be more than 0.5 C below that average, and it’s unlikely to be more than 1.5 C. (I got that from eyeballing the IPCC, plus adding very ad-hoc to the error bars since they use 5-year means and you want just one year).

            I note that you too refuse to give an estimate of climate sensitivity, or your own answers to your own question.

            This shows well how climate denial is like evolution denial: it’s always just “wink wink, nudge nudge” toward alternative explanations for the observed facts. As soon as an remotely articulated “climate skeptic” theory is presented (it’s the sun, it’s underseas volcanoes, cloud feedback will save us) it falls apart very quickly.

          • Jaskologist says:

            I am interested in the numbers that can be checked/falsified. Temperature seems like the easiest one, especially since that’s the number we’re actually concerned about in the end. Feel free to modify to “average temperature over the next five years” or whatever.

          • Tom Womack says:

            There’s weather: it’s chaotic, it’s the place that chaos theory came from. There are the seasons: that’s nice and cyclic. There are the oscillations with year- to decade- periods – El Nino’s the biggest of those, but there are probably others. There are quite likely oscillations with century periods; look for them in the histories of Cambodia and the Yucatan, the places where you find big cities with elaborate irrigation systems abandoned in centuries-regrown jungle. And there’s the ultra-long-term of climate, where you count energy in and energy out and claim to be working over a long enough period that the deep ocean is in equilibrium with the atmosphere.

            If you ask for temperature predictions over a period shorter than a generation or so, you’re asking for predictions of El Nino conditions. It’s not properly known what causes El Nino. Whatever causes it may well be as chaotic as the short-term weather. So the question is quite likely to be as unanswerable as ‘will it be raining in a fortnight’.

            (1998 was a super-El-Nino year; which is why the year-average temperatures for a decade after weren’t as high)

            Big volcanoes cool the planet; it’s pretty clear that the eruptions of big volcanoes are chaotic. We haven’t had a medium-to-big volcano since Pinatubo; we haven’t had a big volcano since Krakatoa.

            I would strongly assert that, in 2034, I expect the weather to be warmer than in any year in 1994-2014 with comparable El Nino state and comparable big-volcano activity.

          • Harald K says:

            Jaskologist: Just because climate sensitivity is harder to check, doesn’t mean we can just pretend it doesn’t exist.

            I gave you the number you wanted, now please give me the number I wanted: your best estimate for climate sensitivity. It is in fact a far more important number than what the temperature will be in 5 years.

            Just discussing temperatures without wanting to say a word about the climate system, is a lot like discussing this patient’s pneumonia, lymphoma and opportunistic fungal infections while being completely agnostic to the possibility of immune system problems.

        • (Not sure where this will appear–there doesn’t seem to be a comment option for the lowest level of the comment hierarchy)

          A couple of people ask for estimates. My estimate of sensitivity is between 2 and 3, since that seems to cover most of the range of what people who know more about the subject than I do suggest, and it explains why the IPCC has tended to predict high.

          My estimate of temperature increase over the next ten years is zero. My basis for that is that temperature has been about flat since 2002 and the last such period, in the mid-20th century, lasted about thirty years. I’m not a climate scientist, don’t build my own models, am not inclined to trust the models other people build, so looking at the pattern of the past, while not a very good guide, strikes me as the best one available.

          My main opinion on the issue is not about future temperature but about consequences, which in large part depend on economics, which is my field. Despite much confident talk, I can see no good reason to expect large net negative consequences by 2100 from temperature increase of the scale suggested by the IPCC reports. I think predictions farther than that, and even that far, are worth very little, given the enormous uncertainty produced by technological change, one of the subjects of my Future Imperfect..

    • Deiseach says:

      I think some of the scepticism about climate change (and isn’t that the accepted term nowadays, rather than global warming?) is that these kind of doom-laden prophecies have been heard before.

      As you say, if it were put that humans have effected the climate by industrialisation, I don’t think many would argue against that.

      But if you’re old enough, you’ve heard all these WE ARE IN IMMINENT PERIL OF DYING IN THE NEXT FIVE YEARS/TEN YEARS/SOMETIME REALLY SOON UNLESS SOMETHING IS DONE!!!!! stories in the press and popular-science TV.

      Except back in the day it was all about the new coming Ice Age and how, by the time the 80s rolled round, we would all be freezing and crops would be dead because all the fertile temperate north would be under ice sheets and here was unimpeachable scientific evidence that temperatures were dropping and would continue to drop.

      Well, there have been no polar bears sighted roaming wild in latitude 50, and now the big panic is how we’re all GOING TO ROAST TO DEATH UNFILTERED UV RADIATION WILL KILL US ALL BY SKIN CANCER CROPS WON’T GROW WE’LL ALL STARVE.

      On that level, the popular panic in the press and pop-science TV, people are sceptical because every day we’re hearing this, that and the other is going to kill us.

      And it doesn’t help when you get “we have to cut back on carbon emissions, things like plane journeys produce tons of carbon, people should not be travelling by plane” when the conference organisers are jetting round the world hither, thither and yonder. Physician, heal thyself?

      Trying to explain the science, which needs a sophisticated and flexible understanding of statistics, is going to be hard when even experts in the field quibble; how do you expect the ordinary idiot in the street (like myself) to understand that? So it’s a bit unfair to lump all climate sceptics in as “hypocrites who know it’s really true because SCIENCE!!!! but they’re all evil and choose short-term profit over the suffering and destruction of humanity”.

      • onyomi says:

        I’m going to agree that “global warming” should not be lumped in with evolution and homeopathy.

        It may be settled that there are certain measurable facts about ocean temperatures, etc. but it seems far less settled to what degree humans are causing it and to what degree it is a bad thing (there is more land unusable by humans because it is too cold than because it is too hot, for example).

        See, for example:

        Also, didn’t Scott himself say somewhere that we can’t entirely trust the implications of statistics like “97% of theologians agree God exists”? I hate to say it, but climate science strikes me as having gone somewhat in that direction.

        When I hear “global warming is settled,” I don’t just hear “it’s settled that temperatures have gone up by x,” I hear, “it’s settled that temps have gone up, that humans activity is a primary cause, that the effects of this are going to be really disastrous, that we need big, multilateral govmt action to deal with it, and that anyone who denies any of this is on the level of intelligent design people.” Scott probably didn’t mean to imply all that, but that’s what saying “global warming is settled” generally means in the popular media.

        • Max says:

          “See, for example:

          Let’s look at the claims made here.

          “Hence that is the number of papers that, according to Cook et. al., implied that humans at least contribute to global warming. The number that imply that humans are the primary cause (category 1) is some smaller percentage which Cook et. al. do not report.”

          You can find the information about who falls in Category 1 (the category of explicit endorsement of AGW, in the sense that global warming is happening and humans contribute more than 50%) on the Skeptical Science website. Quoting here:

          “The IPCC position (humans causing most global warming) was represented in our categories 1 and 7, which include papers that explicitly endorse or reject/minimize human-caused global warming, and also quantify the human contribution. Among the relatively few abstracts (75 in total) falling in these two categories, 65 (87%) endorsed the consensus view. Among the larger sample size of author self-rated papers in categories 1 and 7 (237 in total), 228 (96%) endorsed the consensus view that humans are causing most of the current global warming.”

          The self-rated paper set here is larger and so more statistically robust, and 96% is pretty close to 97%. But even if you go with the sample from the paper’s own ratings, there’s still vast consensus that humans are the main cause of global warming.

          Next up:

          “In short, they got their 97 percent by considering only those abstracts that expressed a position on anthropogenic global warming (AGW). I find it interesting that 2/3 of the abstracts did not take a position.”

          This would be less interesting if the author had read the paper itself and not just the abstract, because the authors comment on this issue:

          “Of note is the large proportion of abstracts that state no position on AGW. This result is expected in consensus situations where scientists ‘ . . . generally focus their discussions on questions that are still disputed or unanswered rather than on matters about which everyone agrees’ (Oreskes 2007, p 72). This explanation is also consistent with a description of consensus as a ‘spiral trajectory’ in which ‘initially intense contestation generates rapid settlement and induces a spiral of new questions’ (Shwed and Bearman 2010 ); the fundamental science of AGW is no longer controversial among the publishing science community and the remaining debate in the field has moved to other topics. This is supported by the fact that more than half of the self-rated endorsement papers did not express a position on AGW in their abstracts.”

          Finally, the last main argument:

          “It’s the fact (if it were a fact) that 97% agree with a view that makes it a consensus. Had the authors’ goal not been to bias the discussion, they would have left out the word “consensus” in the sentence quoted above.”

          Another thing that could have been dealt with if this article had examined the paper itself. The authors define the “consensus position” as the position taken by the IPCC reports, not as the position taken by the people in the survey.

          • “But even if you go with the sample from the paper’s own ratings, there’s still vast consensus that humans are the main cause of global warming.”


            Of the papers that expressed an opinion on warming, by their rating, category 1, “principal cause,” contained 1.6%—64 papers (they say 65) out of about 4000 that, according to Cook et. al., expressed an opinion.

            Your figure is comparing the ones that said humans were the main cause with the ones that said humans were not a cause, both tiny numbers. The relevant question is how many of the papers that expressed a view on warming held that humans were the principal cause. That’s what Cook made a claim about in the second paper, and that claim misrepresented 1.6% as 97%, not a small exaggeration.

            So far as the self-rating, if you read the notes to the table in the article you will find that the percentage they report is for categories 1-3. Only category 1 held that humans were the main cause.

            If you are actually curious, I suggest that instead of reading the summary of my argument on another blog you read the argument on my blog, which is at:


            You might also compare what Cook claimed I said with what I actually said—I have the link on my blog. If after all that you can still believe that Cook is an honest man you have my sympathy.

            “You can find the information about who falls in Category 1 (the category of explicit endorsement of AGW, in the sense that global warming is happening and humans contribute more than 50%) on the Skeptical Science website.”

            But you cannot find that information in Cook et. al. 2013. You have to go to the webbed data and count them yourself. You don’t find it in the least suspicious that the paper only reported the sum of categories 1-3 (97%) and didn’t mention that category 1, humans as principal cause, was only 1.6% of that? Pooling 65 papers with 3832 papers and only reporting the sum?

          • Max says:

            “The relevant question is how many of the papers that expressed a view on warming held that humans were the principal cause. That’s what Cook made a claim about in the second paper, and that claim misrepresented 1.6% as 97%, not a small exaggeration.”

            That is a question but it is not the relevant question. The reason it is not relevant is because the vast majority of papers that made a claim about global warming did not make a claim about whether humans were the main cause of the warming. The papers that did not say humans were the main cause cannot be meaningfully concluded to have come out for either side of the debate, since they did not comment on the issue. This is like saying that since only 3% of astrophysics papers take a position on whether general relativity is correct, and of those that do, 95% say it is, that means only 2.85% of astrophysicists really believe that general relativity is correct. You have to compare like against like. A more meaningful point would be that we cannot extrapolate the views of all climate scientists based on the couple percent that do take an explicit stance. But it is clear that of people who do answer the question “do humans play the main role in causing global warming,” the vast majority say “yes.”

            “You don’t find it in the least suspicious that the paper only reported the sum of categories 1-3 (97%) and didn’t mention that category 1, humans as principal cause, was only 1.6% of that? Pooling 65 papers with 3832 papers and only reporting the sum?”

            No, I don’t find it suspicious. The only reason it would be suspicious is if you think that the people who don’t explicitly take a stance on the role humans play in global warming, have a markedly different stance from the ones who do. There’s no good reason to take this stance, since 96% of the authors ratings of their own papers who took a stance on the issue did agree with the IPCC consensus position. The only argument you could even begin to make here is that this sample was too small to extrapolate to the whole community.

          • Max writes:

            “The reason it is not relevant is because the vast majority of papers that made a claim about global warming did not make a claim about whether humans were the main cause of the warming. The papers that did not say humans were the main cause cannot be meaningfully concluded to have come out for either side of the debate, since they did not comment on the issue.”

            Correct. But John Cook, in the later paper that I quoted, wrote:

            “Cook et al. (2013) found that over 97% endorsed the view that the Earth is warming up and human emissions of greenhouse gases are the main cause.”

            1.6% of the papers that expressed an opinion on causation, according to their count, endorsed the view that human emissions are the main cause. About 95% endorsed the view that humans were a cause (the example for category 2 was ‘Emissions of a broad range of greenhouse gases of varying lifetimes contribute to global climate change.’ ) but did not say whether they thought humans were the main cause.

            Hence the claim that 97% endorsed the “main cause” view is a flat lie. I don’t see why you have difficulty seeing that.

            The numbers for categories 1, 2, and 3, taking the SKS figure for the first (my count was one lower) are:

            Level 1 = 65
            Level 2 = 922
            Level 3 = 2910

            And you find nothing suspicious about reporting only the sum in one paper and then describing that sum in the next paper as if all the papers were in category 1?

          • Max says:

            “Hence the claim that 97% endorsed the “main cause” view is a flat lie. I don’t see why you have difficulty seeing that.”

            It would only be a lie if there were some demonstrably false statement being made that the authors were being misleading about. Since the authors explicitly define what they mean by endorsement right in the text of the paper (abstracts that fall in Category 1-3), the only way they could be lying is if the number of people who fall into Category 1-3 is not 97%. Are you arguing that this is the case? If not, then stop saying they are lying and instead say that you disagree with their definition of the word “endorsement.”

            “And you find nothing suspicious about reporting only the sum in one paper and then describing that sum in the next paper as if all the papers were in category 1?”

            I find it slightly odd that they didn’t publish the data on what fraction of papers fell into each category, but I certainly don’t find it suspicious, because I don’t think there was any malicious intent on the part of the authors. Since the authors had the raw data, they would have known that the ratio of Category 1 to Category 7 backs up their assessment, so there was no need to explicitly point that out.

          • Max: I’m not sure if you’ve read my long blog post that we’re arguing about; if not perhaps you should. It sounds from your response as though you may be misunderstanding the situation and my argument

            There are two different papers of which Cook is a coauther. In the second, he claims that the first found 97% support for humans as the main cause of warming. That’s false. The first paper found 97% support for humans as a cause of warming—”contributed to” in the words of the example for Category 2. It found 1.6% support for humans as the principal cause of warming.

            You can’t convert that falsehood into truth by observing that the *first*paper explained that 97% was for Categories 1-3. The second paper claimed that the first showed 97% support for humans as the main cause of warming, which wasn’t what it showed.

            Does that help? My original post is at:


            You might also want to follow links there so whether what Cook claimed I said corresponded to what I actually said.

      • Harald K says:

        and isn’t that the accepted term nowadays, rather than global warming?

        If there was a global warming equivalent of the FAQ, this would have a point.

        Both terms have been used from the start. The first modern paper on the subject was named “Climatic change: are we on the brink of a pronounced global warming?” – look, both terms in the title! (OK, so it’s “climatic change” rather than climate change, and it’s “a global warming” instead of global warming, but you get the picture).

        But wasn’t there something about a political motive for preferring the one term over the other? Yes, there was, but it was the other way around from what most people think: the republican strategist Frank Luntz wrote in this memo that based on his focus group testing, climate change was understood as less scary than global warming, so that term should be preferred.

        Except back in the day it was all about the new coming Ice Age

        Now that I think of it, there is something close to the talk.origin FAQ: Skeptical science. Here’s their page on this common talking point.

        You, and onyomi who says he “hates to say” that climate science has become a cult: Please familiarize yourself with at least the top 20 commonly made and easily refuted arguments (“Scientists predicted an ice age in the 70s” is number 11).

        • Jaskologist says:

          2009 is not “from the start.”

          And “we are entering another Ice Age” was sufficiently consensus that the Smithsonian Museum had an exhibit declaring so as recently as ~5 years ago (though last time I saw it, there was a little sign saying they were going to update it for the new consensus.)

          Edit: As Tom points out, I’m way off on the date.

          • Tom Womack says:

            The directory that the PDF of the paper is in is called 2009/10 because the author of the blog to which the paper was attached attached it on that date. But if you open the PDF you find that the paper is by Wallace Broecker, and published in Science on 8 August 1975.

          • malpollyon says:

            The paper linked isn’t from 2009 it’s from 1975 as you would know if you’d actually followed the link rather than just skimming the url for a “gotcha”.

          • Jaskologist says:

            I got the date wrong. That’s what I get for dashing off a quick response on the way out the door.

            Here’s the more careful response:

            When people talk about the shift from “global warming” to “climate change,” they aren’t saying that nobody ever used the term “climate change” in the past. They are remarking on the relative frequency of the two, which you can see in this naive google trend I just threw together.

            You can see there that “global warming” dominated the discussion back in the day, by a ratio of roughly 3:1. Recently, though, the ratio has shifted so that the two are now equal. This has been very, very noticeable for those who have been watching for more than a few years.

            And it makes us suspicious, since “climate change” is a far more ambiguous claim, and this shift coincides with the news that the vast majority of climate models failed to predict the temperatures of the past 10 years.

          • Harald K says:

            Jaskologist: That a high-prestige museum buys into the narrative that global cooling was the scientific consensus, does not mean it ever was. Museums make mistakes, and they are not experts on the recent history of science.

            Did you even see the skeptical science link? Of the papers they surveyed, 10% predicted cooling, 62% predicted warming.

            Note also that those who predicted warming did so against the trend line. At that point the world had been slightly cooling since the 1940s.

        • onyomi says:

          I’m willing to believe that global warming may be a more serious concern than it seems to be based on what I’ve previously read, but that still doesn’t justify lumping it in with homeopathy and intelligent design. It is much less settled than that.

        • onyomi says:

          Also, I didn’t say it has “become a cult.” I compared climate scientists to academic theologians, whom I would certainly not describe as “a cult.” Rather, they are a self-selecting group with certain incentives that make them more likely than the average educated person to ascribe to a certain view.

        • Tom Womack says:

          I had thought it was the other way round; that if you called the process ‘global warming’, people supported by the fossil-fuel industry dishonestly claimed that cold temperate-zone winters were evidence against its happening, even if modelling predicted cold winters in Northern temperate zones as a consequence of unusually hot weather in the Arctic.

          So the name changed because the previous name was being maliciously abused, and the adversaries then claim that something dubious is happening because the name is changed. This is one that can’t be won without calling out the malicious adversaries as malicious, and the adversaries call that ‘not sticking to the science’

          • onyomi says:

            This was also my impression. It seemed a response to people saying “well, so much for global warming, eh?” every cold winter.

    • beortheold says:

      I suspect the OP is trolling. Global Warming is a classic motte and bailey argument. Predict all sorts of catastrophes requiring drastic state intervention, but when the predictions fail to materialize retreat to the dry boring stuff that doesn’t even predict anything of any consequence to non-scientists.

      • onyomi says:

        One of the other things I really dislike about the global warming debate is it seems to have sucked all the energy out of what I would personally consider more productive uses of the environmental movement’s time, energy, and money. (Pretty much the only other environmental issue I see in the news anymore is trying to prevent energy exploration, which I guess is related in that it’s about a basic antipathy for fossil fuels).

        Like, I could kind of get behind environmentalism when it was “save the whales” (though they used the same hyperbolic tactics with the rainforest, I recall, when I was a kid, with a similar kind of “crying wolf” reaction eventually taking hold in the public), as that was something relatively substantive and achievable. I’ve read about all kinds of ecological disasters, like huge coral reefs just dying due to human activity, for example, all of which could probably be fixed, if not easily, then certainly much more easily than global warming. Like, we could work to preserve the biodiversity of this remote jungle or we could try to get China to curtail its industrial development. One of these sounds more sexy, but one of these sounds more achievable and meaningful.

        If it comes down to “keep the global average temperature one degree lower than it would have otherwise been a century from now” versus “save one thousand species which would have been extinct a century from now,” then I certainly tend to prefer option B (I realize global warming itself could cause some extinctions, but I’m talking all things equal here).

        • Max says:

          “(I realize global warming itself could cause some extinctions, but I’m talking all things equal here).”

          But that very fact means that all things are not equal. Climate change represents serious threats to biodiversity, especially in the oceans due to acidification (carbon emissions are damaging those reefs you mentioned). It is disruptive to many more species than some particular issue in some localized area of the world. Therefore to somehow treat them as equal means you’ve removed a key part of why global warming is serious to deal with. If global warming is a “sexy” cause, it’s because it is fundamentally changing basically all parts of the biosphere.

          “One of the other things I really dislike about the global warming debate is it seems to have sucked all the energy out of what I would personally consider more productive uses of the environmental movement’s time, energy, and money.”

          Most environmentalists dislike that about it too. The debate is a waste of time, in the sense that either collectively deciding to ignore it, or collectively deciding to take it seriously, would have been more efficient routes. But for those of us who take it seriously, it’s not like we enjoy that there’s a debate in the media about it, even though the debate on the central issue has essentially closed in the scientific community.

          • onyomi says:

            What sorts of actions would people be taking if they were “taking it seriously”? I presume reducing CO2 emissions would be a big part of it, but how to achieve that, realistically? And will they still matter if say, the US does them, but China and India do not? And at what cost? I have seen a few studies claiming that somehow the cost of “taking global warming seriously” would be negligible, but this seems rather implausible to me, given the extent to which the world relies on fossil fuels.

            I tend to believe that if a practical solution exists, politicians will not find it, given that the non-productive status quo serves them quite well: continuous fear-mongering as an excuse for more control and more money spent on pet projects, but nothing every really gets done.

            Also, by “all else equal,” I meant, “if we were to spend the same amount of time and money necessary to prevent 1 degree of warming, how many species could we save if we spent an equal amount of time and money pursuing that goal (even if part of achieving that goal meant doing something about warming)?” That is, would focusing 100% on reducing warming really preserve as much biodiversity as actually focusing on biodiversity itself?

          • Cauê says:

            Ocean acidification is the one that gets closest to making me scared.

            There are many things wrong with the debate.

            First, there’s the unmistakable, giant sign saying “THIS IS A POLITICIZED TOPIC” over the whole thing, which throws off my usual calculations of expert reliability in an unknown field.

            Then, there’s the conflation of several factual questions, in addition to several normative questions contingent on the factual ones. This is usually not unpacked, and leads to people debating “yes, the oceans are warmer” vs. “no, China shouldn’t reduce carbon emissions”.

            Then there are the people obviously speaking from moralistic considerations, opposing proposals that can reduce damages but won’t punish the environmental transgressors, or won’t affect the things they *really* want to change, like capitalism.

          • Tom Womack says:

            Onyomi asks ‘What sorts of actions would people be taking if they were “taking it seriously”?’

            The whole point is that there aren’t any positive actions that individuals can take; it’s a planetary-economy-scale problem, it might be possible for a sufficiently well-lobbied US government to be able to move the needle but probably not far enough.

            There are negative actions: not objecting to increased taxes on fossil-fuel-using activities is a good one.

            A counsel of absolute perfection would be to act as if there were a 100% tax on transport and aviation fuel – if you buy a plane ticket or a bus ticket or pay an electricity bill, make a donation to GiveWell for the amount of the ticket or the bill.

          • Randy M says:

            I would say, at a minimum, having a private jet is a pretty big indication of either not taking climate change seriously or taking oneself way too seriously.

          • Tom Womack says:

            Having access to a private jet merely implies that you have money and a particular trade-off of money for comfort and convenience. If you use the capability that the private jet gives you to lobby more effectively for virtuous causes (by being able to turn up well-rested in more places than would be possible), fine.

            It is well worth burning a thousand tons of avtur for a 0.1% chance of convincing the Ruritanians not to burn ten million tons of avtur.

          • Jaskologist says:

            “Every man always has handy a dozen glib little reasons why he is right not to sacrifice himself.” -Solzhenitsyn

            If you’re going to tell other people they can’t drive their SUVs, eat what they want, and heat their homes to 72 degrees, then you better not be keeping your own house hot enough to grow orchids. Even if they don’t logically invalidate the argument, these kinds of hypocrisies do grate on people, since in the real world, it means “people with private jets are trying to reduce my consumption.” And failure to follow stated beliefs in one’s personal life is a decent heuristic to use for whether someone actually believes them.

          • Randy M says:

            “It is well worth burning a thousand tons of avtur for a 0.1% chance of convincing the Ruritanians not to burn ten million tons of avtur.”

            Yes, this predictable and transparent rationalization I already covered under “taking oneself way too seriously.”

            If you really though burning jet fuel endangered the planet, you’d try skype, or at least travel commercial. I will not be convinced a rational cost benefit analysis was performed (especially including the signaling effects) versus deciding that one is virtuous enough to deserve partaking in the earth-imperiling indulgences forbidden to the masses.

        • Harald K says:

          I’m really shocked at the level of climate denialism on display in this thread. Maybe I shouldn’t be, with ideological contagion being what it is (there’s no logical reason why being opposed to abortion should make your favour laissez-faire economics either), but there you are.

          Onyomi: There’s a simple measure about how seriously we should take global warming. That is climate sensitivity, how many degrees (celsius/kelvin) of global temperature increase we get from a doubling of atmospheric CO2.

          In my humble opinion, every post in a discussion on climate change should be annotated with the poster’s best guess at what the climate sensitivity is. Mine’s 3 degrees, because that’s the scientific consensus best guess, and I have no reason to prefer lower or higher estimates.

          From the looks of it, you think that climate sensitivity is about one degree. You also seem to think one degree of warming isn’t such a big deal, and can be had without thousands of species dying. Try finding one “traditional” conservationist save-the-whales type organization which agrees with you on that, and doesn’t think limiting global warming is crucial.

          • onyomi says:

            There is a logical connection between downplaying global warming threats and being anti-government or pro-limited government, and also one between emphasizing global warming threats and being pro-centralization and pro-government.

            This is because global warming seems, at least on the face of it, to be the sort of problem only governments can solve–and not just governments, but many governments working in concert. In short, it approaches that progressive dream (and my nightmare) of world government.

            Not that I think most people emphasizing global warming threats actually want world government anytime soon, but it is the logical endpoint of their generally pro-centralizing views, and one must admit that if reducing global emissions is the goal, nothing could achieve that as well as world government, or, at least, a very much stronger version of the UN, etc.

            Conversely, it makes a lot of sense that decentralist, free-market types like me would want to downplay the threat of warming because it smacks of the kind of hubris we associate with other grand plans that supposedly can only be achieved through massive curtailment of individual freedoms. There’s also the academic thing, and, for better or worse, small government advocates in the US tend to be suspicious of academia, not entirely without reason, considering its strong historical sympathies for Marxism, etc.

            In short, global warming is a “perfect storm” of a political issue, because it ties into so many hopes and anxieties of both sides.

            I understand, of course, that the empirical questions, like how much warming is caused by human activity and how serious is the threat have nothing to do with one ideology or another, yet one can easily see not only how each world view would encourage overestimating or downplaying the severity, but also support different interpretations of raw data and willingness to accept proposals to solve it, depending on priorities.

          • Harald K says:

            I agree on one part. Yes, for many libertarians this is a “huge meteor on crash course with the earth” scenario: you might as well assume it’s not going to hit, regardless of evidence, since the outcome if it hits is too horrible to contemplate, and there’s nothing you could do about it anyway.

            In general, I defend that sort of thinking, existentially motivated optimism. If you need to believe something on the face of it very implausible to keep you going, go ahead. Personally I suck at optimism in all but existential issues (I have been diagnosed with dysthymia), but I see irrational optimists doing a lot of good in this world.

            But for one, I think it’s unwarranted in this case. There are meaningful things we can do about global warming, and life will still be worth living if we do them (it’s not sacrificing our children to Moloch).

            And for another: you’re seriously projecting when you think about how “our side” reacts to the problem. Do you really think ever environmentalist is secretly drooling after a world government?

            But even if we were, it wouldn’t have the same power to precommit people’s minds. Because that would not be a “huge meteor on crash course with earth”: it wouldn’t be based on pessimism, on avoiding an outcome too bad to contemplate, but on optimism. I just don’t think wanting a good has that much power to bias people, compared to avoiding a disaster.

          • onyomi says:

            I clearly stated that not every, or even most progressives/liberals explicitly want world government, but that world government is the logical endpoint of the centralization of authority impulse.

            I also understand that you don’t have to be a progressive or liberal or leftist to think global warming is a serious problem, but I was talking about the former group, not the warming people (though there’s obviously a big overlap).

        • Tom Womack says:

          The huge coral reefs are dying precisely because CO2 emissions are making the oceans more acidic (directly) and warmer (by holding more heat in the atmosphere). Managing the temperature or pH of all the water around a coral reef is the sort of engineering project likely to be significantly harder than convincing India to build more nuclear power plants.

        • Deiseach says:

          we could try to get China to curtail its industrial development

          To which the Chinese would, not unreasonably, reply that why should they? Why should their people not aspire to a Western standard of living? Why should they have to live in picturesque backwardness simply to preserve your lifestyle?

          The same people lecturing them about curbing industry for the sake of the planet are the same people buyig the Apple products manufactured by Chinese industry.

          I do agree that if developing economies approach industrialisation with the same wasteful, polluting attitude that the West did during its Industrial Revolution, the entire planet is going to be wrecked, but people who live in glass houses can’t afford to throw stones.

          When we’re giving up the same fruits of capitalism that we expect them to give up, then we can talk about sacrificing for the greater good. And of course, we’re not: do we really need a new version of the iPhone, for instance? Or something like Google Glass? Sure, shiny and ever more convenient new tech, but then we turn around and say “yeah, we can have this, but you can’t, because it’s too costly to the environment for everyone to have one”.

          I mean, it’s not like we’re making these new toys out of straw and willow withes, so the material has to come from somewhere, e.g. the minerals being mined in Australia by Chinese consortia, because the demand means profit, and demand is constantly being stoked.

          • Tom Womack says:

            Suggesting that China not develop is obviously not going to get anywhere.

            Handing out the patents for free, providing consultancy services for cheap, and in extremis paying the construction-cost difference, so that they build nuclear power plants instead of coal-fired power plants and gas-driven chemical-production infrastructure rather than coal-driven chemical-production infrastructure, is an easier line to push.

            Air pollution is the kind of thing which does start revolutions, and the Standing Committee does not like revolutions, so it has been spending at the sort of scale that a motivated Chinese central government can manage on non-carbon-producing energy sources.

          • onyomi says:

            I actually meant that working to save some particular biome is much easier and more achievable than getting China to curtail its development. Not sure if that was clear.

            And yes, I don’t think it’s reasonable for us to expect China and India to forego the improvement in living standards we have enjoyed thanks to industrialization, though it would be nice if they’d work on the air pollution… I know they have taken some measures, but it’s still horrendous–and even worse than ten years ago–believe me, I’ve been there several times.

            If being unable to breathe in the nation’s capital isn’t starting a revolution yet, how likely are they really to curtail their development, especially considering that what they’re more worried about right now is growing economic inequality, and the only way to lift those billions of rural poor out of poverty is probably more industrialization.

            And in a related question, should US citizens be taxes to build nuclear power plants all over China and India? Uniquely caring about a problem everyone in the world contributes to puts one in the very awkward position of essentially being a hostage to everyone else. Also, the power needs of China and India, especially as they continue to develop, are pretty unimaginable. Remember that China alone has a population 4x that of the US. Are we really going to provide all those people with clean power?

            I have heard of cheaper, more practical suggested solutions, like intentionally increasing ocean cloud cover with ships that release large quantities of water vapor. No idea how feasible that particular proposal is, but I definitely agree with Caue about this:

            “Then there are the people obviously speaking from moralistic considerations, opposing proposals that can reduce damages but won’t punish the environmental transgressors, or won’t affect the things they *really* want to change, like capitalism.”

            Again, no idea about the “cloud machine” idea (though one does wonder about something like Snow Piercer happening in a nightmare scenario), but I’m sure it’s not the only thing anyone could think of–if all ideas other than massive, multilateral emission reduction deals gone over in successive rounds of excruciatingly slow and costly junkets weren’t out of the question.

          • John Schilling says:

            China has little choice but to work on the air polution; Beijing is becoming nigh uninhabitable in the summer (I was there for five days this year, took me two months to recover).

            They will deal with the air polution that is actually troublesome to them, the local particulates and NOX, by first cracking down on burning wood and using coal instead, second by substituting natural gas for coal in new production, and third by implementing modern particulate-and-NOX emissions controls on the coal and gas plants. This is the cheap way to deak with local particulate and NOX problems. This is what Chinal will do.

            Shutting down all the coal plants in favor of solar or nuclear, or even a crash program of gas turbine construction, is the expensive solution. China isn’t going to do that any time soon.

            Going back to pastoralism and not having all those yucky powerplants because peasant farmers are more photogenic, not even in the cards.

            So, notwithstanding both problems being lumped under the “Green” label, China’s response to local air polution is not going to do too much on the global warming front.

          • Daniel Speyer says:

            It sounds like China would change paths if we could get solar cheaper than coal. I’m not sure exactly how to calculate that, since solar’s costs are more front-loaded, but assuming we can figure out China’s depreciation function (maybe just ask them) this is a viable goal. Western governments can subsidize the R&D, and if necessary eminent-domain a few patents. And since everyone benefits from the technologies, it feels less like being taken advantage of.

            There’s no guarantee it will work, but it seems worth a try.

          • nydwracu says:

            I’m assuming there’s a reason we can’t just invent better solar panels and sell them to China so they can stick a bunch of them in the uninhabitable deserts of Xinjiang, but I don’t know it. What is it?

            (Admittedly, uninhabitable deserts are also a good place for nuclear power plants.)

          • Anonymous says:

            The problem with power plants in uninhabitable deserts is transporting electricity. If you were really good at transporting energy, the answer would be windmills.

          • Tom Womack says:

            The Chinese *are* developing better solar panels, and *are* building arrays of them, in the uninhabitable deserts of Xinjiang and elsewhere. They’ve devoted enough capital to building efficient solar panel factories that the Americans have imposed tariffs on solar panels imported from China; the consequence of which will be that the solar panels will be built and will be put up in Xinjiang rather than sent to the US.

            3.3GW of solar was installed in China in 2014H1.

            As for coal, the plan is to ban burning coal in Beijing by 2020. Coal-fired power stations are built with scrubbers; the problem is that they are controlled by local authorities who’d rather turn the scrubbers off. But changing the minds of local authorities whose incentives are incompatible with those of the Central Committee is the kind of thing the Central Committee has had sixty-five years of practice at, often on attitudes less readily tractable than chimneys.

          • Tom Womack says:

            “The problem with power plants in uninhabitable deserts is transporting electricity.”

            Fortunately, because it is monumentally clear that building power plants next to Beijing is a bad idea, and because the big coal fields and the mighty rivers of China are not on the East Coast, transporting electricity is something China’s quite good at. The China Grid pavilion at Expo 2010 had, tucked in the corner, a description of their ‘demonstration project’ building a 6.4GW HVDC line from Xiangjiaba Dam in Sichuan to Shanghai. That appears to have worked well.

            There now appear to be a dozen such lines planned, a similar number of 3GW lines, and one 8GW line.

            The cost of such lines is on the order of one yuan per megawatt-metre, but China has more billions of yuan than can easily be conceived of.

  18. PS says:

    I agree that the term “debunk” is often highly suspect, although I am not sure that debunk/refute are actually appeals to authority. They are in fact deliberately vague, and imply a potential appeal to authority, but also other possibilities. Refutation refers more to exposing logical flaws in an argument, and need not have been done by many studies, or by any set of authorities. Debunk is even more slippery, and my view is that rationalists should view this term as a rhetorical attack (not a credible statement) in all but the least controversial cases.

    Turning now to the object-level example, it’s true that neither of these studies “debunks” the other, because there are legitimate differences between the two – in at least some senses. There are advantages and disadvantages to asking “Have you been raped or sexually assaulted?” outright compared to trying to ask this question more indirectly.

    HOWEVER, I think it’s fair to say that the 1-in-5 figure has been “refuted,” in the sense that its logical flaws have been clearly pointed out. Taking a 5% number and multiplying it by 4 to produce a 4-year figure of 20% is totally dishonest. At the very least, there is certainly going to be overlap between victims from year to year.

    Second, it’s also fair to say that the construction of the CSAS survey (by Krebs et al) is misleading and probably ideological, more so than the other study. (This is, of course, why The Federalist employs its rhetorical arsenal against the so-called “debunked” CSAS study.) By conflating “forced kissing” and vague drunkenness with rape, the study conjures an epidemic when, in fact, rates of sexual assault have been falling considerably. The construction of this study appears to be deliberately misleading.

    So why zero in on The Federalist’s slightly excessive rhetorical flourish when there is so much dishonest scaremongering in this debate from the other side? I know that you are willing to criticize both sides, so it would normally be fine, but then you also endorse (100%?) a Vox article which does not at all address the key flaws of the CSAS study and its common usage among activists (conflation of non-criminal acts with criminality, the 1-in-5 figure). In my view, that is not a tenable position.

    • llamathatducks says:

      Taking a 5% number and multiplying it by 4 to produce a 4-year figure of 20% is totally dishonest.

      My understanding is that that’s not what they did:

      The researchers broke down the results: since the beginning of college, 13.7 percent of women were the victim of a completed sexual assault. In all, 3.4 percent of women had been raped under the threat of force, and 8.5 percent had been raped when they were too incapacitated from drinking or drugs to consent.


      But it’s the 1 in 5 figure that has really stuck — even though it didn’t appear in the original report. For a later article in the Journal of American College Health, published in 2009, the researchers later used their data to estimate the likelihood that women would experience sexual assault by graduation. That likelihood was about 19 percent — one in five.

      Unfortunately there’s no link to that other article, so I’m not sure how they arrived at their estimate, but there’s nothing to suggest they did the naive thing you accuse them of doing.

      It’s still not the best study in the world. But see also this CDC study which found that 19.3% of women had been raped in their lifetime. “1 in 5 women” is a bit less dramatic than “1 in 5 college students”, but still pretty damn dramatic.

      • PS says:

        Here is the motte and bailey in the CDC study, which is a good example of these studies generally:

        Bailey: 17.4% of heterosexual women have been raped (p. 10)

        Motte: “Rape is defined as any completed or attempted unwanted vaginal (for women), oral, or anal penetration through the use of physical force (such as being pinned or held down, or by the use of violence) or threats to physically harm, and includes times when the victim was drunk,
        high, drugged, or passed out and unable to consent.” (p. 9)

        “Unwanted” penetration while drunk or high. Note that unwantedness and drunkenness are both undefined, and that drunkenness is distinguished from being passed out.

        The survey also found that nearly 20% of straight men experienced “sexual violence,” including 5% suffering “sexual coercion” and 5% being “made to penetrate”: men being forced to have sex by women, equivalent to rape of men by women. (p. 11)

        It would be possible for a consistent advocate to argue that there is a culture of impunity surrounding women’s sexual coercion of men, considering female on male rape convictions are well-nigh non-existent, and yet this form of consistency is nowhere to be seen.

        The intent is thus to employ strategic equivocation to advance an ideology of “rape culture” directed against women only, and it’s probable that the ideology is built into the study itself. Why else leave key terms vague and undefined?

        CDC study with page numbers:

      • PS says:

        In fairness, I can still eat crow re: 5% multiplied by four. I had seen this written somewhere and internalized it all too well. In fact, the CSAS study (Krebs et al) took the sample of college seniors (~1,000 at 2 universities) and asked them if they had been raped – or rather, had experienced “unwanted sex,” etc. – and the study arrived at the 19% figure from there. So if you accept their methodology (which you shouldn’t), then the figure does make sense.

  19. “There are many “questions” that are pretty much settled – evolution”

    While “evolution” is pretty settled, whenever there is a perceived intersection with politics it stops being settled: group selection, evolutionary psychology, hereditability of human factors such as IQ are all extremely controversial.

    • Anonymous says:

      What is the politics of group selection?

      • Brian Donohue says:

        I thought this would be pretty obvious. Group selection fits well with a communitarian philosophy.

    • satanistgoblin says:

      Same goes for global warming – I think that only settled part is that it is real, not how dangerous it is or how it should be dealt with in the near term (geo-engineering?). Similarly, an orthodoxy of sorts exists.

      • Charlie says:

        Heck, we could even agree that homeopathy is a placebo, but there’s still serious controversy over how to handle cases when giving the patient a placebo is actually the best course.

        Fractals, man. Fractals.

  20. Scott’s Law: Your ideas are less well supported than you think they are, even after taking into account Scott’s law.

    …and I think Scott has mentioned a while back how the time for metahumor is only when you are Hofstadter, so I pray to be forgiven. Edit: I don’t think this is actually metahumor, so nvm.

  21. Emile says:

    Do you have any suggestions for improving all that?

    I’ve been interested in tools that are supposed to improve discourse, and have been slowly ramping up the skills needed to build such tools, but am somewhat skeptical about how useful such tools would be (the outside view seems to show that so far, a lot of discourse-improving tools have been made, and none has had any significant success).

    • Anonymous says:

      Unfortunately I think the world is adversarial to such tools. As soon as a discourse-improving tool gains any semblance of legitimacy, it attracts efforts to exploit that legitimacy, which undermines the discourse-improving aspects of the tool.

      • Protagoras says:

        If it is possible to generalize about science, it seems to be a collection of such tools. And while some critics have claimed that what you describe has happened with science, and they’re no doubt partly correct, there does seem to be a fairly impressive track record of success and improvement there.

  22. Steve Johnson says:

    I’m shocked that no one here identifies the “rape” issue as classic motte and bailey.

    Bailey – 1 in 4 women on campus are raped!

    Motte – for a definition of rape that includes having sex with your regular lover when you didn’t feel like it that much or when one or both people have had a drink

    Here’s another reasoning rule that I’ve tried to introduce here before – if a statistic is politicized and every single poster example turns out to be basically a fraud then the issue is a lie.

    If 25% of college women were being raped then you could get better examples than this chick who made up a story (and she actually had to be sought out by a reporter who is connected to a whole network of campus “sexual assault” councilors).

    This doesn’t solve every issue – you need other rules of reasoning to figure out the minimum wage debate – but it sure clears up some issues quite nicely.

    • PS says:

      #Win. This is what I mean about the CSAS study being “deliberately misleading.” It’s a classic motte and bailey.

    • grendelkhan says:

      Motte – for a definition of rape that includes having sex with your regular lover when you didn’t feel like it that much

      No studies of this type (see here for a list) count anything that could reasonably be described as “you didn’t feel like it that much” as rape. (I’m a bit confused as to why having sex with “your regular lover” makes some kind of important difference here.)

      or when one or both people have had a drink

      This is apparently a very common truncation of the actual questions asked. (But it is good for getting a “#Win” in the comments. Debunked! Refuted! Bleah.)

      If 25% of college women were being raped then you could get better examples

      Here, there’s a whole book of them, if you’re interested.

      • PS says:

        The first of his examples was excessively flippant, but the issue of vague survey questions is a real problem. Actually, the equivocation found in survey language is the core problem. The link between this issue and motte-and-bailey tactics is thus very robust, not to be dismissed with hand waving.

        Here is the survey question text you linked to: “Sometimes sex happens when a person is unable to consent to it or stop it from happening because they were drunk, high, drugged, or passed out from alcohol, drugs, or medications.”

        This text is far from simple for the average person to parse, and I would argue it is designed to produce false positives.

        Many people feel as though they are “unable to stop” certain things from happening because they are drunk, despite the fact that they are participating in those things. “Unable to stop from happening” is simply not specific enough. By invoking drunkenness (but not specifying incapacitation), the vagueness is then compounded.

        To make matters even worse, terms like “consent” sound like legalese to most people, and for very good reason: “consent” is a formal term, not a colloquial one. So the exact meaning of “consent” is opaque to very many, while “unable to stop from happening” is vivid but highly elastic.

        Far better would be a term like “without your permission,” a phrase that is in wide colloquial usage and has a precise meaning.

        I think it would be possible to have a good indirect sexual assault survey, but I think that existing surveys are engaged in strategic equivocation.

      • Cauê says:

        I can easily imagine situations in which “sex with your regular lover” should be treated significantly differently.

        Suppose my wife wakes me up by performing some sex act. This would very likely be quite pleasing, although there was no consent and we can’t say I “wanted” it. If some other woman did this, however…

        Similarly, suppose I get seriously drunk with my wife at a party, then we go home and have sex. Ok, that’s great. Now, if someone else had taken me home…

        In both (quite realistic) situations, there’s a strong expectation that the sex with the “regular lover” will be wellcome. Meanwhile, we would expect and require a clear sign of consent from people with different levels of relationship.

        Sex is more than just the act, and that’s why it makes a difference. The several emotional/moralistic events that trigger from having sex with someone new (sacredness/purity reactions, jealousy…), as well as social consequences (expectations of commitment, status worries…) are not a factor when a regular lover does something that’s hard to classify as consensual (surprise kiss!), but would be if the same thing happened with anyone else.

        So… “Sometimes sex happens when a person is unable to consent to it or stop it from happening because they were drunk, high, drugged, or passed out from alcohol, drugs, or medications.”

        I’d have to say “yes, this has happened to me”. With a long time girlfriend / wife. Yes, it does makes a large difference.

  23. Pingback: Outside in - Involvements with reality » Blog Archive » Chaos Patch (#40)

  24. Richard Metzler says:

    I’m a bit surprised at the position you take. The “one-in-five” study was a prime case of how a single study that fit into a political narrative was taken as gospel, even though IMO it doesn’t pass basic sanity tests. If it were remotely true that one out of five college students were raped – where are the hundreds of thousands of affected women, their parents, their boyfriends, clamoring to end this epidemic and let the police handle this, instead of the obviously incompetent college administrations? Why did Sabrina Rubin Erdely have to contact several colleges before she found one that offered a rape story juicy enough for her (and that one turned out to be at least partially a hoax)?
    Also, a while ago, various colleges reported numbers on the sexual assaults and rapes that occurred on their campus, and they were nowhere near “one-in-five”. And what was feminists’ response? No, it was not “oh, maybe colleges are not the horrible rapey places we thought they were.” Since they already knew the “correct answer”, their take was that apparently colleges are horrible rapey places with a massive underreporting problem. Hmmmmm…
    Also, the ways how studies come up with numbers like “one-in-five” (very expansive definitions that go out of their way to include every gray-area event in the worst possible category) have been explained dozens of times by various commentators. So IMO, “debunked” is not too strong a word, at least when applied to the interpretation of these studies that is usually thrown around by feminists (where the distinction between different kinds of assault is usually ditched in favor of “one in five is RAPED! RAPED, I tell you!”)

    • Protagoras says:

      You don’t need one in five to get you hundreds of thousands of affected women; one in fifty would get you that. So unless you think the rate is even lower than that, there must be some reason why a lot of those women aren’t clamoring or encouraging others in their lives to clamor. According to the feminists, of course, this is because someone outing herself as a rape victim opens herself up to a lot of abuse and shaming and blaming. Which seems to fit well with what rape victims say, as well as matching what is observed in a lot of the cases where women do come forward. So the feminist theory looks more plausible to me than the campus rape is really rare theory (especially as the number of indicidents involving people I know personally is non-zero, despite the number of people I know well enough to be confident they’d talk to me about it being quite small).

      • onyomi says:

        I am very willing to believe that rape is more prevalent than most people, especially men, generally realize, due to the shame aspect, yet still unwilling to believe it’s as prevalent as 1 in 5, due to the “basic sanity test” Richard mentions.

        Totally anecdotal, but I feel like if it were truly 1 in 5, then I would know at least one person who has been raped. Now, admittedly, maybe I know one or more people who have been raped but never talk about it, but I also have had, in my life, far more than 5 close female friends/relatives–people close enough to me that if they’d been raped, I’d know.

        That said, I may come from a socio-economic background in which rape is rarer than usual (though I’d imagine this varies less in that respect compared to say, theft, or even murder), or underestimate just how strong the shame is that might keep even close female friends from ever mentioning a trauma. Those factors are enough to get me to say 5%, up from the 0% which is my personal experience–definitely worthy of concern, but also not quite an epidemic.

        • Protagoras says:

          I’m curious; how many female friends do you have who have told you about their abortions? The statistics for that are one in three, and I know exactly two women who have told me about their own abortions. I do know slightly more than six women, but women who would share with me on that topic, six doesn’t sound wildly implausible, so my experience doesn’t lead me to be skeptical of the abortion number (and it is also of course based on considerably less difficult and controversial research). In any event, if far less than a third of your close female friends and relatives have told you about their abortions, I think you should seriously consider that they may not be telling you as much as you think they would.

          • onyomi says:

            Well I do know at least one female friend who has had an abortion, and I’m sure abortions are much more common than I realize (and in my subjective impression, much more common than rape), and that my friends and family are also not representative on this score, but 1 in 3 for abortion still sounds really high to me. Do you have any links about that?

          • Protagoras says:

            Here’s the first thing a web search turned up; it says 3 in 10 rather than 1/3.

          • onyomi says:

            Well, I have to admit that that is a lot more prevalent than I would have thought. I still know 100% more people who have had an abortion than who have been raped, though it does drive home the extent to which one cannot assume one has an even vaguely realistic sense of how often things happen when most people consider those things to be very private.

            I am also generally sheltered and naive. I recall finding out after the fact how many people in my high school had been using hard drugs and being completely shocked. It wasn’t that I actively avoided it; rather, I just instinctively avoided hanging out with the types of people and being in the types of situations where such things occurred. And, so, most likely, did the users and dealers realize explicitly or intuitively that I was not their clientele. Reminds me of how nobody knows someone who believes in intelligent design, yet some staggering percentage actually does.

          • anon says:

            1 in 5 sounds about right to me, from hearing about the experiences of female friends and acquaintences. If you add sexual assault into the equation, it feels like the number is closer to 1/3. Perhaps my friends are unusual, but I think it’s more likely that we’re unusually open about these things.

          • onyomi says:

            Consulting with my girlfriend, who probably has more close female friends than myself, and who also has the advantage of, you know, being female, she says she knows two friends who have had abortions and at least two who have been “sexually assaulted,” by which she means aggressive, unwanted sexual contact (one example: at a bar, a man suddenly stuck his hands down a woman’s pants and her fingers into her vagina). She said she does not know of any full-on rapes.

            So the numbers still seem high to her, though again this is just anecdotal, and her circle of friends may not be representative.

            Though if we are talking about “aggressive, unwanted sexual contact” of the kind I just described, then I could easily believe that is 1 in 5, or even higher.

          • Anthony says:

            There are huge social class (and race) variations in abortion rates – even if 30% of women have ever had an abortion, the women you hang out with might have a much lower (or higher) rate.

            I would suspect that the sexual assault rates are less variable, but I could be wrong.

        • Steve Sailer says:

          My impression is that more regrettable things happen to young women than you might think.

        • Steve Sailer says:

          The other thing to keep in mind is that “sex” isn’t just one kind of activity, it can be a variety of activities. A woman might enthusiastically agree to X and Y but be less enthusiastic about engaging in Z. The Harvard coed who drags a mattress with her everywhere decided she was raped many months after engaging in Z.

          Dean Nicole Eramo at UVA was widely denounced for giving an interview in which she explained that some of her job consists of sitting patiently while coeds talk about what boys did to them and how they feel about it. Of course, in the case of Jackie, we now know that Dean Eramo was quite right for not taking her stories to seriously and demanding a drone strike on the frat house.

          But the point is that sex is complicated and it’s not surprising that different research methodologies will come up with different numbers.

    • llamathatducks says:

      1. It’s not exactly one study. The Vox article also linked to this CDC study which found that 19.3% of women were raped in their lifetime. It’s true that this is a different and less dramatic finding than 1 in 5 being raped in college, but it’s still a pretty dramatic “nearly 1 in 5 women are raped in their life”.

      2. The study cited in The Federalist actually does find a massive underreporting problem – about 67% of female nonstudent rape victims did not report the rape, and about 80% of female student rape victims didn’t report. So whichever definition of rape you prefer, the underreporting problem exists.

      • Randy M says:

        That is dramatic, but consider that the difference between 1-in-5 college students and 1-in-5 lifetime is a factor of about 20, although I could think of a few reasons why college would be especially dangerous (and also more protected, so not sure how that shakes out).

        • Army1987 says:

          That is dramatic, but consider that the difference between 1-in-5 college students and 1-in-5 lifetime is a factor of about 20,

          That assumes that 70-year-olds are as likely to be raped as 20-year-olds, which is probably not even in the ballpark of being a reasonable approximation.

  25. Sarah says:

    None of this matters anyway, since women lie about getting raped all the time. I read an article recently where a white woman was raped and blamed a black man who was sent to prison, but when the DNA was tested, it turns out that he didn’t do it and that the DNA belonged to a different man who was already in prison for rape.

    • DES3264 says:

      Your summary sounds far more like a mistaken identification than a lie. Given the facts that a women said she was raped and that DNA from the rape kit belonged to a known rapist, I’d say the evidence that she was raped is pretty good. Considering the problems with eye witness testimony, especially after traumatic events, the facts you’ve described make honest error sound far more likely than a lie.

    • DrBeat says:

      That is like the worst possible objection you could make to rape culture hysteria. People lie about everything all the time because people are liars; this doesn’t mean that we can’t believe anything and nothing is a problem. Conversely, the studies described are situations where lying about being raped does not have much of a productive outcome for the liar, and since people tell lies to benefit themselves, we would not expect an abnormally high rate of lies in response to the survey.

    • Illuminati Initiate says:

      I have no idea how often people lie about rape, but one specific incident is not enough to say “all the time”.

      • Douglas Knight says:

        Sarah’s comment is irrelevant for the reasons others give, but the anecdote is representative: before the advent of DNA testing, 15% of rape convictions jailed the wrong man. (page 6 or 14)

        • Hainish says:

          OK, but (as you can see on pp. 8-9) not because the rape victims are lying liars who lie, but because the innocent men were implicated by co-defendants, falsely identified by eye-witnesses*, or due to “faulty forensic evidence” (whatever that means).

          These seem to be cases where the victim did not know the defendant.

    • Protagoras says:

      I was proceeding on the assumption that this was some variety of sarcasm, though it is always so hard to tell on the internet. If I was in error, I agree with the points made by previous people, and if I was correct, the responses so far seem to be a good indication of how great the risk is of being misinterpreted when you attempt sarcasm in an internet comment thread.

  26. Jos says:

    I would have liked the Vox article better if it had broken down the various categories that the two studies found, so I knew more what both were talking about.

    IMHO, the biggest question about the CSA study is the number of respondents who don’t consider their experiences to be rape. (60% of forcible rape victims and 75% of “incapacited” victims did not classify their experiences as rape.) I think it would be helpful to know more about why the respondents disagreed with CSA.

    Looking at CSA, it looks like a big driver of the difference is “incapacitated assualt” — IIUC, that includes cases in which a person believes themselves to be consenting at the time, but is drunk or high, and regrets it later. That’s a valuable thing to measure, but it’s productive to call it out.

  27. Another example worth mentioning is the Cyril Burt case. Lots of people still believe he was shown to have been a fraud, despite the relevant professional association having withdrawn their condemnation after two different people published books arguing that the accusations, conveniently made after he was dead, were baseless. Also after two of the people who the accusers claimed didn’t exist showed up. Lots of “A debunked B” if you are on one side, “B debunked A” on the other.

    Another good one is Margaret Mead and Samoa, with Derek Freeman as the critic.

    • Steve Sailer says:

      Anthropology controversies tend to be both ideological and have a Heisenbergian element to them in that famous anthropologists tend to have large personalities that may well influence their subjects. For example, I had dinner once with an anthropologist who had been in the jungle with controversial anthropologist N. He was in scientific and ideological agreement with N, but cautioned that N’s distinctive personality may have elicited behaviour from subjects trying to live up to N’s example.

  28. I should probably mention an experience early in my life which taught roughly the same lesson as Scott’s article. I was visiting Yale as a high school senior thinking of applying. They happened to be having a program on the House Unamerican Activity Committee controversy.

    First I saw a move, “Operation Abolition,” which made it perfectly clear that the campaign to abolish the committee was a communist plot.

    Then there was a movie, “Operation Fraud” or some similar title, that made it perfectly clear that the first movie was bogus.

    Then there was written material that made it perfectly clear that the second movie was bogus.

    Then there was … .

    I concluded that reaching a conclusion after hearing only the arguments offered by one side was imprudent.

  29. Steve Sailer says:

    Another reality check is to consider how often women who were “raped” under a broad definition have sex voluntarily with that man later. I vaguely recall a celebrated study from about 25 years ago that was much discussed in the early 90s. The percentage was considerable, perhaps a little under 50%.

    Stuff happens. And it’s often complicated.

    • grendelkhan says:

      Is there really a need for the scare quotes? There’s ample evidence here that victims will frequently avoid defining themselves as victims, that they’ll convince themselves it was “just a bad date” or something like that, because if they stay with a rapist, what kind of person does that make them?

      This is just garden-variety cognitive dissonance, and doesn’t come close to implying that people raped under a “broad definition” (i.e., not by a stranger, without struggle, by a partner) weren’t, in fact, raped.

      • Cauê says:

        Suppose I’m not quite in the mood for sex, but my wife is. I decide to go with it, to please her, and also because I know I’ll enjoy it once it starts.

        This “unwanted sex” is classified in the “internal obligation” subcategory of “sexual coercion” here (I promise I’ll stop linking to this):

        It’s then reported that I, like 43% of young men, am a “victim of sexual coercion”, and, as common, the “aggressor” was an intimate partner.

        All of the scare quotes above are well deserved. But I don’t think we are speaking of the same kinds of situations, and much of the disagreement in this post’s comments is coming from this.

        • Anonymous says:

          That definition is not typical of these surveys. NISVS does not include that in sexual coercion and only finds 13% of women and 6% of men subject to sexual coercion.

          • Cauê says:

            I’ve been a “victim” by five of those eight criteria. I’m not really a victim.

            Neither are many of the people being counted as victims. The cutoff point is somewhere between “I wasn’t quite in the mood but wanted to please my partner” and “if I said I stayed with my rapist what would that make me”. Wherever it is, there is at least one very good point inside “those people don’t consider themselves victims”.

            It’s partly about counting noncentral examples in a way that will create a perception of a greater incidence of the central examples than the evidence warrants.

            But I think it also involves more fundamental confusions and/or disagreements about what exactly we are (or should be) trying to prevent and why.

  30. Tarrou says:

    Definitions are everything, and I suspect that the conflation of a lot of behavior, some of which is criminal, some of which is merely rude, and some of which is perfectly normal under the heading of “RAPEANDSEXUALASSAULTZ!” will eventually lead to a sort of discounting of the moral damage of the actual crimes.

    Think of it this way, I once, when I was young, was horrified at the number of “sex offenders” wandering about on the registry. Then I found out that one can get on the registry for public urination, which means the drunk dude pissing behind a dumpster at 0300 is indistinguishable from a serial rapist. Now when someone mentions someone being on the registry, I don’t immediately assume he’s a bad dude. I can’t. The inclusion of much more innocent behavior discounts the actual bad actors.

    Including things like “rubbing up on someone through their clothes” in the definition of “rape” makes me wonder if those people have ever seen the inside of a club. Or a crowded bus. And it makes me worry that one day we won’t be able to assume a convicted rapist is a bad dude.

    • grendelkhan says:

      Sex offender registries generally list the offense involved, so you can at least distinguish between “forcible rape” and “public indecency”, or whatever you’re interested in.

      Including things like “rubbing up on someone through their clothes” in the definition of “rape” makes me wonder if those people have ever seen the inside of a club.

      I’m not much for clubbing, but I was under the impression that people are generally into that in a club. “Including things like ‘punching someone in the face’ in the definition of ‘aggravated assault’ makes me wonder if those people have ever seen the inside of a boxing ring”, etc. (It’s not the act, it’s the consent.)

      And it makes me worry that one day we won’t be able to assume a convicted rapist is a bad dude.

      Meh. Right now there’s an ongoing problem where women have to sort of assume that maybe one in a dozen of the men they encounter are rapists. Criminal convictions are very rare, and not immediately visible. Figuring out some way to make that first probability lower is a lot more useful than making that second one a stronger signal.

      • Tarrou says:

        Apply the logic at hand to your example mate! Going to a club is consent? I doubt that is going to fly with the SJW crowd.

        The problem is that a great many behaviors are considered “confident” and “attractive” and “romantic” when certain males do them and “creepy” and “rape-like” when other males do them. And this decision is made by the recipient of the behavior, subjectively, and only after the behavior has taken place. Now, you may say that we should, as a society, dispense with these mating rituals and I may agree, but it’s not up to us. It’s up to the gatekeepers of sex, and that’s women, for the most part.

        Read a romance novel, and imagine the male love interest is three hundred pounds and wearing that beloved of internet hate, the neckbeard. All of the sudden, this steamy romance seems pretty dark, eh?

        It is not up to men to change female fantasy and inclination, nor is it up to women to solely police male behavior. But it is up to both groups to have a line we can draw on behavior, and right now, there isn’t much clarity to be had. The ideas of “consent” are important, but they must be translatable to the real world in which we live, not some school skit where everyone talks like they’re in a Dick and Jane book.

  31. Anonymous 2 says:

    To interpret the studies which claim 1 in 5 or 1 in 6 women have been raped (but the subjects don’t call it rape), it’s necessary to read them. As grendelkhan points out about, these studies have been replicated.

    The alcohol related questions have pretty bad wording. Lots of people just say that the study is debunked and call it a day. But what about the other questions in the study?

    For example, let’s look at the original Mary Koss study, which found a 1 in 5 figure. In Table 3 at the end of the PDF, it reports the questions, and the percentage of women who answered “yes.” (Note: the column for women is for victims, and the column for men is aggressors, because feminists typically think of men as aggressors not as victims, but that’s another subject.)

    The question that most sounds like rape is #9: “Have you had sexual intercourse when you didn’t want to because a man threatened or used some degree of physical force (twisting your arm, holding you down, etc) to make you?” 9% of women answered “yes.”

    That’s nearly 1 in 10.

    Unfortunately, we can’t know how many of those women defined themselves as raped. We only know that 27% of the women categorized as raped said they were raped. Let’s move on to that finding. Here are a few explanations (which are not mutually exclusive):

    – They were under-reporting. It’s known that people under-report rape.

    – “Didn’t want to” means something different to some women than “didn’t consent to,” because language is messy. This is the hypothesis raised above, that people can “consent to something that they don’t want.” There is no evidence either for or against this hypothesis that I’m aware of. (There are some feminist studies claiming that miscommunication doesn’t happen, but these studies are crap because they extrapolate from non-sexual communication.)

    – The participants in the study were taking a legalistic definition of rape that required intention on the part of the perpetrator (mens rea). The thought process might be something like “yes, my ex boyfriend pressured me into having sex that I didn’t want, but he didn’t realize that I wasn’t into it and thought I was consenting… he was just drunk and made a mistake, but he’s not a criminal and I’m not a rape victim…” Note: I am not taking a position about whether this is rape or not without knowing the facts of a specific situation; I’m proposing a plausible rationale for the participant’s responses.

    – 17% of women in the Koss study thought that they experienced “a crime other than rape.” We don’t know what that crime is. Perhaps sexual assault? This finding is bad for the hypothesis that these women just had messy or drunk sex which they regretted later. They were sufficiently bothered by their experiences to consider them a crime, even if the crime didn’t match a prototypical rape in their minds.

    In summary, it is quite possible that 1 in 10 women have been raped (i.e. experienced a violation that would widely be called “rape”, not just by shifty feminists). If so, then 1 in 5 is also plausible and hardly debunked (though less so). There are other explanations for these numbers, such as researchers and participants understanding terms in different ways, or ambiguities and miscommunications in the situation. We don’t have good empirical tests of these explanations, and they are speculative enough that they can’t wipe away the entire 1 in 5 figure.

    Hopefully future research will be higher quality. We have definitely learned that people’s reports about sexual violence is very sensitive to the language by which it is framed. Unfortunately, there are many definitions of rape in research, and different understandings in the population.

    • jtroll says:

      The Koss study is from 1985, but the DOJ data has rape and sexual assault trending sharply downward.

    • PS says:

      This is a very fair summary, raising some important issues. But I think that the way we frame the issue assumes in advance the possible existence of epidemic violence against women, without accepting the possibility that reality may be messy and involve ambiguous forms of mild mutual coercion (sexual and nonsexual).

      For example, what about the reverse to all this – what about sexual violence against men, by both women and other men? My understanding is that all surveys turn up significant amounts of sexual violence against men, including a significant amount committed by women. This no doubt involves a lot of false positives, like the data regarding women, but it’s still true that no one has grappled with the widespread existence of sexual coercion against men. I’d like to ask people what they make of it, and what, if anything, should be done about it. As far as I know, very little is being done currently.

    • Hainish says:

      Note: the column for women is for victims, and the column for men is aggressors, because feminists typically think of men as aggressors not as victims, but that’s another subject.

      This isn’t an issue particular to feminism, and I’d go so far as to say that feminists are better than average at distinguishing victim/aggressor from male/female. (Whenever I’ve seen claims that men cannot _ever_ be victims of rape by women, they’ve been from extreme PUA types.)

  32. Anthony says:

    Related – peer review isn’t all that: – fuller explanation – from the actual conference

    The NIPS consistency experiment was an amazing, courageous move by the organizers this year to quantify the randomness in the review process. They split the program committee down the middle, effectively forming two independent program committees. Most submitted papers were assigned to a single side, but 10% of submissions (166) were reviewed by both halves of the committee. This let them observe how consistent the two committees were on which papers to accept. (For fairness, they ultimately accepted any paper that was accepted by either committee.)

    The results were revealed this week: of the 166 papers, the two committees disagreed on the fates of 25.9% of them: 43.

    Other, similar experiments, from the comments in

    The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.

    With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated. – how much review is necessary?

    Incidentally, there are now functions to make your plots look like they came from xkcd in a number of languages, including Mathematica, R, MATLAB, and LaTeX:

  33. Santiago says:

    I make too many isolated demands for rigor, and your essay helped me realize that. I will be more careful about that in the future.

    Thank you.

  34. Eli says:

    I don’t think I understand how 6/1000 becomes 50/1000 through the mechanisms talked about in the Vox article. Wouldn’t you need a much higher unreport rate than 56%? If 50% are unreported, then we’re at 12/1000.

    Wouldn’t we need a lot more than 15% of women claiming it was “unpleasant” but not a crime? A lot more than 32% calling it a crime but not rape? It’s not only not capturing the whole of the difference that needs to be made up, it’s not coming close to half of it.

  35. Pingback: On examining evidences for points of view, etc | The Daily Pochemuchka