Jacques Derrida proposed a form of philosophical literary criticism called deconstruction. I’ll be the first to admit I don’t really understand it, but it seems to have something to do with assuming all texts secretly contradict their stated premise and apparent narrative, then hunting down and exposing the plastered-over areas where the author tries to hide this.

I have no idea whether this works for literature or not, but it’s a useful way to read scientific papers.

Consider a popular field – or, at least, a field where a certain position is popular. For example, we’ve been talking a lot about growth mindset recently. There seem to be a lot of researchers working to prove growth mindset and not a lot working to disprove it. Journals are pretty interested in studies showing growth mindset interventions work, and maybe not so interested in studies showing they don’t. I’ll admit that my strong suspicions of publication bias don’t seem to be borne out by the facts here – see this meta-analysis – but I bet its more sinister cousin “all experimenters believe the same thing and have the same experimenter effects” bias is alive and well.

In a field like that, you’re not going to get the contrarian studies you want, but one way to find the other side of the issue is to look a little more closely at the studies that do get published, the ones that say they’re in support of the thesis, and see if you can find anything incriminating.

Here’s a perfect example: Mindset Interventions Are A Scalable Treatment For Academic Underachievement, by a team of six researchers including Carol Dweck.

The abstract reads:

The efficacy of academic-mind-set interventions has been demonstrated by small-scale, proof-of-concept interventions, generally delivered in person in one school at a time. Whether this approach could be a practical way to raise school achievement on a large scale remains unknown. We therefore delivered brief growth-mind-set and sense-of-purpose interventions through online modules to 1,594 students in 13 geographically diverse high schools. Both interventions were intended to help students persist when they experienced academic difficulty; thus, both were predicted to be most beneficial for poorly performing students. This was the case. Among students at risk of dropping out of high school (one third of the sample), each intervention raised students’ semester grade point averages in core academic courses and increased the rate at which students performed satisfactorily in core courses by 6.4 percentage points. We discuss implications for the pipeline from theory to practice and for education reform.

This sounds really, really impressive! It’s hard to imagine any stronger evidence in growth mindset’s favor.

And then you make the mistake of reading the actual paper.

The paper asked a 1,594 students from a bunch of different high schools to take a 45 minute online course.

A quarter of the students took a placebo course that just presented some science about how different parts of the brain do different stuff.

Another quarter took a course that was supposed to teach growth mindset.

Still another quarter took a course about “sense of purpose” which talked about how schoolwork was meaningful and would help them accomplish lots of goals and they should be happy to do it. This was also classified as a “mindset intervention”, though it seems pretty different.

And the final quarter took both the growth mindset course and the “sense of purpose” course.

Then they let all students continue taking their classes for the rest of the semester and saw what happened, which was this:

[EDIT: I totally bungled these graphs! See discussion of exactly how on the author’s reply above, without which the information below will be misleading at best]

Among ordinary students, the effect on the growth mindset group was completely indistinguishable from zero, and in fact they did nonsignificantly worse than the control group. This was the most basic test they performed, and it should have been the headline of the study. The study should have been titled “Growth Mindset Intervention Totally Fails To Affect GPA In Any Way”.

Instead they went to subgroup analysis. Subgroup analysis can be useful to find more specific patterns in the data, but if it’s done post hoc it can lead to what I previously called the Elderly Hispanic Woman Effect, after medical papers that can’t find their drug has any effect on people at large, so they keep checking different subgroups – young white men…nothing. Old black men…nothing. Middle-aged Asian transgender people…nothing. Newborn Australian aboriginal butch lesbians…nothing. Elderly Hispanic women…p = 0.049…aha! And the study gets billed as “Scientists Find Exciting New Drug That Treats Diabetes In Elderly Hispanic Women.”

As per the abstract, the researchers decided to focus on an “at risk” subgroup because they had principled reasons to believe mindset interventions would work better on them. In their subgroup of 519 students who had a GPA of 2.0 or less last semester, or who failed one or more academic courses last semester:

Growth mindset still doesn’t differ from zero. And growth mindset does nonsignificantly worse than their “sense of purpose” intervention where they tell children to love school. In fact, the students who take both “sense of purpose” and growth mindset actually do (nonsignificantly) worse than sense-of-purpose alone!

But the control group mysteriously started doing much worse in all their classes right after the study started, so growth mindset is significantly better than the control group. Hooray!

Why would the control group’s GPA suddenly decline? The simplest answer would be that by coincidence the class got harder right after the study started, and only the intervention kids were resilient enough to deal with it – but that can’t be right, because this was done at eleven different schools, and they wouldn’t have all had their coursework get harder at the same time.

Another possibility is that sufficiently low-functioning kids are always declining – that is, as time goes on they get more and more behind in their coursework, so their grades at time t+1 are always less than at time t, and maybe growth mindset has arrested this decline. This is plausible and I’d be interested in seeing if other studies have found this.

Perhaps aware that this is not very convincing, the authors go on to do another analysis, this one of percent of students passing their classes.

This is the same group of at-risk students as the last one. It’s graphing what percent of these students pass versus fail their courses. The graph on th left shows that a significantly higher number of students in the intervention conditions pass their courses than in the control condition.

This is better, but one part still concerns me.

Did you catch that phrase “intervention conditions”? The authors of the study write: “Because our primary research question concerned the efficacy of academic mindset interventions in general when delivered via online modules, we then collapsed the intervention conditions into a single intervention dummy code (0 = control, 1 = intervention).

We don’t know whether growth mindset did anything for even these students in this little subgroup, because it was collapsed together with the (more effective) “sense of purpose” intervention before any of these tests were done. I don’t know if this is just for convenience, or if it is to obfuscate that it didn’t work on its own.

[EDIT: Scott McGreal looks further and finds in the supplementary material that growth mindset alone did NOT significantly improve pass rates!]

The abstract of this study tells you none of this. It just says: “Mindset Interventions Are A Scalable Treatment For Academic Overachievement…Among students at risk of dropping out of high school (one third of the sample), each intervention raised students’ semester grade point averages in core academic courses and increased the rate at which students performed satisfactorily in core courses by 6.4 percentage points” From the abstract, this study is a triumph.

But my own summary of these results, as relevant to growth mindset is as follows:

For students with above a 2.0 GPA, a growth mindset intervention did nothing.

For students with below a 2.0 GPA, the growth mindset interventions may not have improved GPA, but may have prevented GPA from falling, which for some reason it was otherwise going to do.

Even in those students, it didn’t do any better than a “sense-of-purpose” intervention where children were told platitudes about how doing well in school will “make their families proud” and “make a positive impact”.

In no group of students did it significantly increase chance of passing any classes.

Haishan writes:

If ye read only the headlines, what reward have ye? Do not even the policymakers the same? And if ye take the abstract at its face, what do ye more than others? Do not even the science journalists so?”

Titles, abstracts, and media presentations are where authors can decide how to report a bunch of different, often contradictory results in a way that makes it look like they have completely proven their point. A careful look at the study may find that their emphasis is misplaced, and give you more than enough ammunition against a theory even where the stated results are glowingly positive.

The only reason we were told these results is that they were in the same place as a “sense of purpose mindset” intervention that looked a little better, so it was possible to publish the study and claim it as a victory for mindsets in general. How many studies that show similar results for growth mindset lack a similar way of spinning the data, and so never get seen at all?

Dodge et al’s Impact Of Early Intervention On Psychopathology, Crime, And Well-Being At Age 25 is published in last month’s American Journal Of Psychiatry. Gratifyingly, it is a randomized controlled trial. Ten thousand kindergarteners in disadvantaged areas were screened for “conduct problems” until they found about 900 who looked like they were at high risk. 445 were randomly selected for the intervention. Another 446 stayed in the control group. The intervention was a bunch of extra classes and ‘enrichment programs’ from elementary school (age 5) all the way through high school (age 16). The study mentions “social skills friendship groups”, “guided parent child interaction sessions”, “tutoring in reading”, “parent-youth groups on topics of adolescent development, alcohol, tobacco, and drugs”, “youth forums on vocational opportunities”, and “Oysterman’s School-To-Job possible selves intervention aimed at examining emerging identity”.

All of these sound so pretentious that I would have loved to be able to report that they had no effect, but in fact the opposite was true. When they caught up with these kids at age 25, the intervention group was found to have an odds ratio of around 0.6 to 0.7 of having developed various psychiatric disorders the study was testing for, including antisocial personality disorder, ADHD, depression, or anxiety. They had odds ratios around 0.7 of developing drug and alcohol abuse problems by various measures. They reported less risky sexual behavior, less domestic abuse, and fewer violent crimes. All of this was significant at the p < 0.05 level, and some of it was significant at much higher levels like p = 0.001 or below. Subgroup analysis found the data were very similar when you restricted the analysis to various subgroups like boys, girls, whites, blacks, highest-risk, lowest-risk, and by study site (it was a multi-site study). As best I can tell there were not an equal number of anaylses they did that came up negative that they covered up.

The apparent conclusion is that intensive interventions can change children’s outcomes and personalities in important ways ten years down the road, even regarding things believed to be highly genetic like antisocial personality disorder.

A few weak attempts to rebut this. First, there were some things that study didn’t do that one might have expected it to. It didn’t change graduation rates or employment rates. The apparent decrease in domestic violence was mediated entirely by the intervention group being less likely to have relationships (!) – the rate of domestic violence among people in relationships was the same. There was no effect on health. There was no effect on self-reported satisfaction with their parents’ parenting. There were (nonsignificantly) higher death rates and incarceration rates in the intervention group than the control group.

So if I wanted to be maximally mean to the study, I could say that whatever it’s doing to violent crime and drug use has to be compatible with a (nonsignificantly) raised incarceration rate, and whatever it’s doing to drug use and risky sexual behavior and criminality has to be compatible with a (nonsignificantly) raised death rate. This suggests the possibility of an attack based on their endpoints being screwy, though I’m not sure what form such an attack could take. One could argue that since many of their outcomes were based on self-report surveys maybe the kids who had been through all of the enrichment programs had grown to like the study people and had a stronger demand effect to say that they were doing great. But a lot of the survey data was backed up by court records confirming fewer drug and violence convictions. So that doesn’t really work.

If you’re less interested in the pure science of individual differences and more interested in policy, one fact that I forgot to mention was that this program cost $60,000 per kid. The paper points out that this is the same cost as a year or two of incarceration, so if it really changes children’s life outcomes and makes tham less antisocial even that hefty price tag might be justified (although again, remember that it didn’t affect employment or incarceration when checked directly).

If you’re looking for an optimistic spin on that number, they freely admit they have no idea which part of their gigantic ten year intervention program produced the positive effects. It could be that all the youth forums and enrichment programs and friendship groups and so on had zero effect, and the entire benefit came from the “Oysterman’s School-To-Job possible selves intervention aimed at examining emerging identity”. And maybe that’s a piece of paper that can be copied on a copy machine for ten cents a sheet. All this suggests is that at least some part of the ten-year, $60,000 intervention did something.

If you’re looking for a pessimistic spin on that number, consider. Every so often I see things that claim to have completely shifted children from the most high-risk of high-risk groups to upstanding successful members of society by giving them a year or preschool, or a couple of after-school lessons, or something like that. And these studies always boast that they did it with only $1000 or $5000 or some number like that, so it’s nice and cost effective. So far, the studies I have seen like this have been wrong. And so far I have not been surprised, because we already spend between $100,000 to $200,000 per child on education and various social programs. If someone ever found a social program that really worked for $1,000, the first thing we would want to do is tar and feather everyone currently in our bureaucracy of social programs, for being so incompetent that changing their $200,000 in spending to $201,000 in spending (with the extra $1000 going to someone besides them) could completely revolutionize life outcomes.

This study seems more in line with everything else. By going from $200,000 to $260,000, we can slightly push a few things in a positive direction a little bit more, maybe. From a scientific view, it’s pretty interesting. From a policy view, it’s nothing to write home about.

Allan Crossman calls parapsychology the control group for science.

That is, in let’s say a drug testing experiment, you give some people the drug and they recover. That doesn’t tell you much until you give some other people who are taking a placebo drug you know doesn’t work – but which they themselves believe in – and see how many of them recover. That number tells you how many people will recover whether the drug works or not. Unless people on your real drug do significantly better than people on the placebo drug, you haven’t found anything.

On the meta-level, you’re studying some phenomenon and you get some positive findings. That doesn’t tell you much until you take some other researchers who are studying a phenomenon you know doesn’t exist – but which they themselves believe in – and see how many of them get positive findings. That number tells you how many studies will discover positive results whether the phenomenon is real or not. Unless studies of the real phenomenon do significantly better than studies of the placebo phenomenon, you haven’t found anything.

Trying to set up placebo science would be a logistical nightmare. You’d have to find a phenomenon that definitely doesn’t exist, somehow convince a whole community of scientists across the world that it does, and fund them to study it for a couple of decades without them figuring out the gig.

Luckily we have a natural experiment in terms of parapsychology – the study of psychic phenomena – which most reasonable people don’t believe exists but which a community of practicing scientists does and publishes papers on all the time.

The results are pretty dismal. Parapsychologists are able to produce experimental evidence for psychic phenomena about as easily as normal scientists are able to produce such evidence for normal, non-psychic phenomena. This suggests the existence of a very large “placebo effect” in science – ie with enough energy focused on a subject, you can always produce “experimental evidence” for it that meets the usual scientific standards. As Eliezer Yudkowsky puts it:

Parapsychologists are constantly protesting that they are playing by all the standard scientific rules, and yet their results are being ignored – that they are unfairly being held to higher standards than everyone else. I’m willing to believe that. It just means that the standard statistical methods of science are so weak and flawed as to permit a field of study to sustain itself in the complete absence of any subject matter.

These sorts of thoughts have become more common lately in different fields. Psychologists admit to a crisis of replication as some of their most interesting findings turn out to be spurious. And in medicine, John Ioannides and others have been criticizing the research for a decade now and telling everyone they need to up their standards.

“Up your standards” has been a complicated demand that cashes out in a lot of technical ways. But there is broad agreement among the most intelligent voices I read (1, 2, 3, 4, 5) about a couple of promising directions we could go:

1. Demand very large sample size.

2. Demand replication, preferably exact replication, most preferably multiple exact replications.

3. Trust systematic reviews and meta-analyses rather than individual studies. Meta-analyses must prove homogeneity of the studies they analyze.

4. Use Bayesian rather than frequentist analysis, or even combine both techniques.

5. Stricter p-value criteria. It is far too easy to massage p-values to get less than 0.05. Also, make meta-analyses look for “p-hacking” by examining the distribution of p-values in the included studies.

6. Require pre-registration of trials.

7. Address publication bias by searching for unpublished trials, displaying funnel plots, and using statistics like “fail-safe N” to investigate the possibility of suppressed research.

8. Do heterogeneity analyses or at least observe and account for differences in the studies you analyze.

9. Demand randomized controlled trials. None of this “correlated even after we adjust for confounders” BS.

10. Stricter effect size criteria. It’s easy to get small effect sizes in anything.

If we follow these ten commandments, then we avoid the problems that allowed parapsychology and probably a whole host of other problems we don’t know about to sneak past the scientific gatekeepers.

Well, what now, motherfuckers?

II.

Bem, Tressoldi, Rabeyron, and Duggan (2014), full text available for download at the top bar of the link above, is parapsychology’s way of saying “thanks but no thanks” to the idea of a more rigorous scientific paradigm making them quietly wither away.

You might remember Bem as the prestigious establishment psychologist who decided to try his hand at parapsychology and to his and everyone else’s surprise got positive results. Everyone had a lot of criticisms, some of which were very very good, and the study failed replication several times. Case closed, right?

Earlier this month Bem came back with a meta-analysis of ninety replications from tens of thousands of participants in thirty three laboratories in fourteen countries confirming his original finding, p < 1.2 * -10¹⁰, Bayes factor 7.4 * 10⁹, funnel plot beautifully symmetrical, p-hacking curve nice and right-skewed, Orwin fail-safe n of 559, et cetera, et cetera, et cetera.

By my count, Bem follows all of the commandments except [6] and [10]. He apologizes for not using pre-registration, but says it’s okay because the studies were exact replications of a previous study that makes it impossible for an unsavory researcher to change the parameters halfway through and does pretty much the same thing. And he apologizes for the small effect size but points out that some effect sizes are legitimately very small, this is no smaller than a lot of other commonly-accepted results, and that a high enough p-value ought to make up for a low effect size.

This is far better than the average meta-analysis. Bem has always been pretty careful and this is no exception.

So – once again – what now, motherfuckers?

III.

In retrospect, that list of ways to fix science above was a little optimistic.

The first nine items (large sample sizes, replications, low p-values, Bayesian statistics, meta-analysis, pre-registration, publication bias, heterogeneity) all try to solve the same problem: accidentally mistaking noise in the data for a signal.

We’ve placed so much emphasis on not mistaking noise for signal that when someone like Bem hands us a beautiful, perfectly clear signal on a silver platter, it briefly stuns us. “Wow, of the three hundred different terrible ways to mistake noise for signal, Bem has proven beyond a shadow of a doubt he hasn’t done any of them.” And we get so stunned we’re likely to forget that this is only part of the battle.

Bem definitely picked up a signal. The only question is whether it’s a signal of psi, or a signal of poor experimental technique.

None of these five techniques even touch poor experimental technique – or confounding, or whatever you want to call it. If an experiment is confounded, if it produces a strong signal even when its experimental hypothesis is true, then using a larger sample size will just make that signal even stronger.

Replicating it will just reproduce the confounded results again.

Low p-values will be easy to get if you perform the confounded experiment on a large enough scale.

Meta-analyses of confounded studies will obey the immortal law of “garbage in, garbage out”.

Pre-registration only assures that your study will not get any worse than it was the first time you thought of it, which may be very bad indeed.

Searching for publication bias only means you will get all of the confounded studies, instead of just some of them.

Heterogeneity just tells you whether all of the studies were confounded about the same amount.

Bayesian statistics, alone among these first eight, ought to be able to help with this problem. After all, a good Bayesian should be able to say “Well, I got some impressive results, but my prior for psi is very low, so this raises my belief in psi slightly, but raises my belief that the experiments were confounded a lot.”

Unfortunately, good Bayesians are hard to come by. People like to mock Less Wrong, saying we’re amateurs getting all starry-eyed about Bayesian statistics even while real hard-headed researchers who have been experts in them for years understand both their uses and their limitations. Well, maybe that’s true of some researchers. But the particular ones I see talking about Bayes here could do with reading the Sequences. Here’s Bem:

An opportunity to calculate an approximate answer to this question emerges from a Bayesian critique of Bem’s (2011) experiments by Wagenmakers, Wetzels, Borsboom, & van der Maas (2011). Although Wagenmakers et al. did not explicitly claim psi to be impossible, they came very close by setting their prior odds at 10^20 against the psi hypothesis. The Bayes Factor for our full database is approximately 10^9 in favor of the psi hypothesis (Table 1), which implies that our meta-analysis should lower their posterior odds against the psi hypothesis to 10^11

Let me shame both participants in this debate.

Bem, you are abusing Bayes factor. If Wagenmakers uses your 10^9 Bayes factor to adjust from his prior of 10^-20 to 10^-11, then what happens the next time you come up with another database of studies supporting your hypothesis? We all know you will, because you’ve amply proven these results weren’t due to chance, so whatever factor produced these results – whether real psi or poor experimental technique – will no doubt keep producing them for the next hundred replication attempts. When those come in, does Wagenmakers have to adjust his probability from 10^-11 to 10^-2? When you get another hundred studies, does he have to go from 10^-2 to 10^7? If so, then by conservation of expected evidence he should just update to 10^+7 right now – or really to infinity, since you can keep coming up with more studies till the cows come home. But in fact he shouldn’t do that, because at some point his thought process becomes “Okay, I already know that studies of this quality can consistently produce positive findings, so either psi is real or studies of this quality aren’t good enough to disprove it”. This point should probably happen well before he increases his probability by a factor of 10^9. See Confidence Levels Inside And Outside An Argument for this argument made in greater detail.

Wagenmakers, you are overconfident. Suppose God came down from Heaven and said in a booming voice “EVERY SINGLE STUDY IN THIS META-ANALYSIS WAS CONDUCTED PERFECTLY WITHOUT FLAWS OR BIAS, AS WAS THE META-ANALYSIS ITSELF.” You would see a p-value of less than 1.2 * 10^-10 and think “I bet that was just coincidence”? And then they could do another study of the same size, also God-certified, returning exactly the same results, and you would say “I bet that was just coincidence too”? YOU ARE NOT THAT CERTAIN OF ANYTHING. Seriously, read the @#!$ing Sequences.

Bayesian statistics, at least the way they are done here, aren’t gong to be of much use to anybody.

That leaves randomized controlled trials and effect sizes.

Randomized controlled trials are great. They eliminate most possible confounders in one fell swoop, and are excellent at keeping experimenters honest. Unfortunately, most of the studies in the Bem meta-analysis were already randomized controlled trials.

High effect sizes are really the only thing the Bem study lacks. And it is very hard to experimental technique so bad that it consistently produces a result with a high effect size.

But as Bem points out, demanding high effect size limits our ability to detect real but low-effect phenomena. Just to give an example, many physics experiments – like the ones that detected the Higgs boson or neutrinos – rely on detecting extremely small perturbations in the natural order, over millions of different trials. Less esoterically, Bem mentions the example of aspirin decreasing heart attack risk, which it definitely does and which is very important, but which has an effect size lower than that of his psi results. If humans have some kind of very weak psionic faculty that under regular conditions operates poorly and inconsistently, but does indeed exist, then excluding it by definition from the realm of things science can discover would be a bad idea.

All of these techniques are about reducing the chance of confusing noise for signal. But when we think of them as the be-all and end-all of scientific legitimacy, we end up in awkward situations where they come out super-confident in a study’s accuracy simply because the issue was one they weren’t geared up to detect. Because a lot of the time the problem is something more than just noise.

IV.

Wiseman & Schlitz’s Experimenter Effects And The Remote Detection Of Staring is my favorite parapsychology paper ever and sends me into fits of nervous laughter every time I read it.

The backstory: there is a classic parapsychological experiment where a subject is placed in a room alone, hooked up to a video link. At random times, an experimenter stares at them menacingly through the video link. The hypothesis is that this causes their galvanic skin response (a physiological measure of subconscious anxiety) to increase, even though there is no non-psychic way the subject could know whether the experimenter was staring or not.

Schiltz is a psi believer whose staring experiments had consistently supported the presence of a psychic phenomenon. Wiseman, in accordance with nominative determinism is a psi skeptic whose staring experiments keep showing nothing and disproving psi. Since they were apparently the only two people in all of parapsychology with a smidgen of curiosity or rationalist virtue, they decided to team up and figure out why they kept getting such different results.

The idea was to plan an experiment together, with both of them agreeing on every single tiny detail. They would then go to a laboratory and set it up, again both keeping close eyes on one another. Finally, they would conduct the experiment in a series of different batches. Half the batches (randomly assigned) would be conducted by Dr. Schlitz, the other half by Dr. Wiseman. Because the two authors had very carefully standardized the setting, apparatus and procedure beforehand, “conducted by” pretty much just meant greeting the participants, giving the experimental instructions, and doing the staring.

The results? Schlitz’s trials found strong evidence of psychic powers, Wiseman’s trials found no evidence whatsoever.

Take a second to reflect on how this makes no sense. Two experimenters in the same laboratory, using the same apparatus, having no contact with the subjects except to introduce themselves and flip a few switches – and whether one or the other was there that day completely altered the result. For a good time, watch the gymnastics they have to do to in the paper to make this sound sufficiently sensical to even get published. This is the only journal article I’ve ever read where, in the part of the Discussion section where you’re supposed to propose possible reasons for your findings, both authors suggest maybe their co-author hacked into the computer and altered the results.

While it’s nice to see people exploring Bem’s findings further, this is the experiment people should be replicating ninety times. I expect something would turn up.

As it is, Kennedy and Taddonio list ten similar studies with similar results. One cannot help wondering about publication bias (if the skeptic and the believer got similar results, who cares?). But the phenomenon is sufficiently well known in parapsychology that it has led to its own host of theories about how skeptics emit negative auras, or the enthusiasm of a proponent is a necessary kindling for psychic powers.

Other fields don’t have this excuse. In psychotherapy, for example, practically the only consistent finding is that whatever kind of psychotherapy the person running the study likes is most effective. Thirty different meta-analyses on the subject have confirmed this with strong effect size (d = 0.54) and good significance (p = .001).

Then there’s Munder (2013), which is a meta-meta-analysis on whether meta-analyses of confounding by researcher allegiance effect were themselves meta-confounded by meta-researcher allegiance effect. He found that indeed, meta-researchers who believed in researcher allegiance effect were more likely to turn up positive results in their studies of researcher allegiance effect (p < .002). It gets worse. There's a famous story about an experiment where a scientist told teachers that his advanced psychometric methods had predicted a couple of kids in their class were about to become geniuses (the students were actually chosen at random). He followed the students for the year and found that their intelligence actually increased. This was supposed to be a Cautionary Tale About How Teachers’ Preconceptions Can Affect Children.

Less famous is that the same guy did the same thing with rats. He sent one laboratory a box of rats saying they were specially bred to be ultra-intelligent, and another lab a box of (identical) rats saying they were specially bred to be slow and dumb. Then he had them do standard rat learning tasks, and sure enough the first lab found very impressive results, the second lab very disappointing ones.

This scientist – let’s give his name, Robert Rosenthal – then investigated three hundred forty five different studies for evidence of the same phenomenon. He found effect sizes of anywhere from 0.15 to 1.7, depending on the type of experiment involved. Note that this could also be phrased as “between twice as strong and twenty times as strong as Bem’s psi effect”. Mysteriously, animal learning experiments displayed the highest effect size, supporting the folk belief that animals are hypersensitive to subtle emotional cues.

Okay, fine. Subtle emotional cues. That’s way more scientific than saying “negative auras”. But the question remains – what went wrong for Schlitz and Wiseman? Even if Schlitz had done everything short of saying “The hypothesis of this experiment is for your skin response to increase when you are being stared at, please increase your skin response at that time,” and subjects had tried to comply, the whole point was that they didn’t know when they were being stared at, because to find that out you’d have to be psychic. And how are these rats figuring out what the experimenters’ subtle emotional cues mean anyway? I can’t figure out people’s subtle emotional cues half the time!

I know that standard practice here is to tell the story of Clever Hans and then say That Is Why We Do Double-Blind Studies. But first of all, I’m pretty sure no one does double-blind studies with rats. Second of all, I think most social psych studies aren’t double blind – I just checked the first one I thought of, Aronson and Steele on stereotype threat, and it certainly wasn’t. Third of all, this effect seems to be just as common in cases where it’s hard to imagine how the researchers’ subtle emotional cues could make a difference. Like Schlitz and Wiseman. Or like the psychotherapy experiments, where most of the subjects were doing therapy with individual psychologists and never even saw whatever prestigious professor was running the study behind the scenes.

I think it’s a combination of subconscious emotional cues, subconscious statistical trickery, perfectly conscious fraud which for all we know happens much more often than detected, and things we haven’t discovered yet which are at least as weird as subconscious emotional cues. But rather than speculate, I prefer to take it as a brute fact. Studies are going to be confounded by the allegiance of the researcher. When researchers who don’t believe something discover it, that’s when it’s worth looking into.

V.

So what exactly happened to Bem?

Although Bem looked hard to find unpublished material, I don’t know if he succeeded. Unpublished material, in this context, has to mean “material published enough for Bem to find it”, which in this case was mostly things presented at conferences. What about results so boring that they were never even mentioned?

And I predict people who believe in parapsychology are more likely to conduct parapsychology experiments than skeptics. Suppose this is true. And further suppose that for some reason, experimenter effect is real and powerful. That means most of the experiments conducted will support Bem’s result. But this is still a weird form of “publication bias” insofar as it ignores the contrary results of hypotheticaly experiments that were never conducted.

And worst of all, maybe Bem really did do an excellent job of finding every little two-bit experiment that no journal would take. How much can we trust these non-peer-reviewed procedures?

I looked through his list of ninety studies for all the ones that were both exact replications and had been peer-reviewed (with one caveat to be mentioned later). I found only seven:

Batthyany, Kranz, and Erber: .268
Ritchie 1: 0.015
Ritchie 2: -0.219
Richie 3: -0.040
Subbotsky 1: 0.279
Subbotsky 2: 0.292
Subbotsky 3: -.399

Three find large positive effects, two find approximate zero effects, and two find large negative effects. Without doing any calculatin’, this seems pretty darned close to chance for me.

Okay, back to that caveat about replications. One of Bem’s strongest points was how many of the studies included were exact replications of his work. This is important because if you do your own novel experiment, it leaves a lot of wiggle room to keep changing the parameters and statistics a bunch of times until you get the effect you want. This is why lots of people want experiments to be preregistered with specific committments about what you’re going to test and how you’re going to do it. These experiments weren’t preregistered, but conforming to a previously done experiment is a pretty good alternative.

Except that I think the criteria for “replication” here were exceptionally loose. For example, Savva et al was listed as an “exact replication” of Bem, but it was performed in 2004 – seven years before Bem’s original study took place. I know Bem believes in precognition, but that’s going too far. As far as I can tell “exact replication” here means “kinda similar psionic-y thing”. Also, Bem classily lists his own experiments as exact replications of themselves, which gives a big boost to the “exact replications return the same results as Bem’s original studies” line. I would want to see much stricter criteria for replication before I relax the “preregister your trials” requirement.

(Richard Wiseman – the same guy who provided the negative aura for the Wiseman and Schiltz experiment – has started a pre-register site for Bem replications. He says he has received five of them. This is very promising. There is also a separate pre-register for parapsychology trials in general. I am both extremely pleased at this victory for good science, and ashamed that my own field is apparently behind parapsychology in the “scientific rigor” department)

That is my best guess at what happened here – a bunch of poor-quality, peer-unreviewed studies that weren’t as exact replications as we would like to believe, all subject to mysterious experimenter effects.

This is not a criticism of Bem or a criticism of parapsychology. It’s something that is inherent to the practice of meta-analysis, and even more, inherent to the practice of science. Other than a few very exceptional large medical trials, there is not a study in the world that would survive the level of criticism I am throwing at Bem right now.

I think Bem is wrong. The level of criticism it would take to prove a wrong study wrong is higher than that almost any existing study can withstand. That is not encouraging for existing studies.

VI.

The motto of the Royal Society – Hooke, Boyle, Newton, some of the people who arguably invented modern science – was nullus in verba, “take no one’s word”.

This was a proper battle cry for seventeenth century scientists. Think about the (admittedly kind of mythologized) history of Science. The scholastics saying that matter was this, or that, and justifying themselves by long treatises about how based on A, B, C, the word of the Bible, Aristotle, self-evident first principles, and the Great Chain of Being all clearly proved their point. Then other scholastics would write different long treatises on how D, E, and F, Plato, St. Augustine, and the proper ordering of angels all indicated that clearly matter was something different. Both groups were pretty sure that the other had make a subtle error of reasoning somewhere, and both groups were perfectly happy to spend centuries debating exactly which one of them it was.

And then Galileo said “Wait a second, instead of debating exactly how objects fall, let’s just drop objects off of something really tall and see what happens”, and after that, Science.

Yes, it’s kind of mythologized. But like all myths, it contains a core of truth. People are terrible. If you let people debate things, they will do it forever, come up with horrible ideas, get them entrenched, play politics with them, and finally reach the point where they’re coming up with theories why people who disagree with them are probably secretly in the pay of the Devil.

Imagine having to conduct the global warming debate, except that you couldn’t appeal to scientific consensus and statistics because scientific consensus and statistics hadn’t been invented yet. In a world without science, everything would be like that.

Heck, just look at philosophy.

This is the principle behind the Pyramid of Scientific Evidence. The lowest level is your personal opinions, no matter how ironclad you think the logic behind them is. Just above that is expert opinion, because no matter how expert someone is they’re still only human. Above that is anecdotal evidence and case studies, because even though you’re finally getting out of people’s heads, it’s still possible for the content of people’s heads to influence which cases they pay attention to. At each level, we distill away more and more of the human element, until presumably at the top the dross of humanity has been purged away entirely and we end up with pure unadulterated reality.

The Pyramid of Scientific Evidence

And for a while this went well. People would drop things off towers, or see how quickly gases expanded, or observe chimpanzees, or whatever.

Then things started getting more complicated. People started investigating more subtle effects, or effects that shifted with the observer. The scientific community became bigger, everyone didn’t know everyone anymore, you needed more journals to find out what other people had done. Statistics became more complicated, allowing the study of noisier data but also bringing more peril. And a lot of science done by smart and honest people ended up being wrong, and we needed to figure out exactly which science that was.

And the result is a lot of essays like this one, where people who think they’re smart take one side of a scientific “controversy” and say which studies you should believe. And then other people take the other side and tell you why you should believe different studies than the first person thought you should believe. And there is much argument and many insults and citing of authorities and interminable debate for, if not centuries, at least a pretty long time.

The highest level of the Pyramid of Scientific Evidence is meta-analysis. But a lot of meta-analyses are crap. This meta-analysis got p < 1.2 * 10^-10 for a conclusion I'm pretty sure is false, and it isn’t even one of the crap ones. Crap meta-analyses look more like this, or even worse.

How do I know it’s crap? Well, I use my personal judgment. How do I know my personal judgment is right? Well, a smart well-credentialed person like James Coyne agrees with me. How do I know James Coyne is smart? I can think of lots of cases where he’s been right before. How do I know those count? Well, John Ioannides has published a lot of studies analyzing the problems with science, and confirmed that cases like the ones Coyne talks about are pretty common. Why can I believe Ioannides’ studies? Well, there have been good meta-analyses of them. But how do I know if those meta-analyses are crap or not? Well…

The Ouroboros of Scientific Evidence

Science! YOU WERE THE CHOSEN ONE! It was said that you would destroy reliance on biased experts, not join them! Bring balance to epistemology, not leave it in darkness!

I LOVED YOU!!!!

Edit: Conspiracy theory by Andrew Gelman

Its latest is Takizawa et al’s Adult Health Outcomes of Childhood Bullying Victimization: Evidence From A Five-Decade Longitudinal British Birth Cohort. It has since been picked up by Fox, the Washington Post, and even Xinhua. I think that’s enough to qualify for “made world headlines”.

The study took some British kids in 1958, sorted them by how much they got bullied, and checked how they did forty years later. In fact, the frequently bullied kids had nearly twice as much psychiatric disease, were twice as likely to attempt suicide, were twice as likely to drop out of high school, and even had double the unemployment rate. Worse physical health, worse cognitive function, less likely to get married, et cetera, et cetera.

Those must be some bullies.

But correlation is not causation. There’s an alternative possibility. Maybe bullies only pick on unpopular disadvantaged kids. And maybe these kinds of things are stable, so that unpopular disadvantaged kids are more likely to grow up to be unpopular disadvantaged adults. The sort of adults who are more likely to have psychiatric disease, drop out of school, be unemployed, et cetera. That sure sounds plausible.

So the researchers “controlled for confounders”. They used a scale called the Bristol Social Adjustment Guide to figure out how socially well-adjusted the kids were, then aded in their social class, their family’s level of contact with child protective services, their IQ, their attractiveness, and even how much their parents loved them (really! check the study!)

They controlled for all these things and found that the bullying-outcomes link was still robust. They concluded that this meant their finding wasn’t just that bullies were bullying kids with problems, it was that bullies were causing the damage themselves.

Do you believe that? It all comes down to one question.

Who is better able to look deep inside you and judge the mettle of your soul? A playground bully? Or the Bristol Social Adjustment Guide?

My money is on the bully. Bullies are like sharks: horrible pinnacles of evolution. Animals have been learning to navigate social dominance hierarchies through violence since pecking orders in chickens, on through wolf packs and chimpanzees, and up into humans – and we are very good at it. The bully is the purest manifestation of the primal instinct, which is why he crops up untaught and unbidden in near-identical form in schoolyards from Los Angeles to London to Lanzhou. And like sharks, a good bully should be able to smell blood in the water and know when an opportunity to attack presents itself.

Most of the findings of this study were in the “frequently bullied” population, and part of the criteria for “frequently” was bullying both at age 7 and age 11. Unless that’s just one really persistent guy, that means the child has gotten independently selected for targeting in two different environments. That could be bad luck but could also be the effect of high inter-bully reliability in what (persistent) qualities make a good victim.

So let’s take another look at those confounders we supposedly controlled for. Where’s height? You think short kids are bullied more often than tall kids? I do. Height is closely related to career success, to attractiveness to the opposite sex, increased happiness and self-esteem, and decreased psychological morbidity. This is something every bully knows intuitively, but which the Takizawa study didn’t think of and therefore couldn’t control for.

But it’s giving them too much credit to be bringing in weird stuff like height-mental-health correlations. What about social skills? Yeah, sure, they did that Bristol Social Adjustment Guide. I’m looking at it right now, and it’s asking the students’ teachers to rate items like “hostility towards adults” and “depression”. I don’t believe that teachers filling numbers into hokey little boxes can capture an assessment of a kid’s social skills as well as a bully trying to decide who can safely be picked on can.

So I will come out and say it: I do not trust the practice of “adjusting for confounders”, at least not the way this study does it. You are adjusting for an imperfect measurement of the confounders you can think of. If you find that there is lingering correlation, then either you your hypothesis is true, or you didn’t adjust for confounders well enough. Given extraordinary results, like being bullied at age seven making you 25% less likely to be married at age fifty, the “you didn’t adjust for confounders well enough” option starts to look really good.

I think the proper way to do this study would have been to do an anti-bullying intervention at a couple schools, leave a couple similar schools as controls, and if the anti-bullying intervention successfully decreases bullying, compare outcomes for children at the two schools. I understand this probably would be logistically impossible, plus you’d have to wait another forty years. But given that you cannot do the study right, I am not sure that doing the study this way adds anything, except of course widely-read articles in every news source in the world.

I would also compare to Reming et al, which attempts much the same study and finds no association after adjusting for their confounders of choice (which, oddly, are much fewer than in the current study). They also find that parent reports about bullying (the method Takizawa et al used) are wildly unreliable, with an inter-rater agreement of just 0.11 with reports by teachers or the children themselves (the statistic goes from perfect agreement being 1.0 to zero information being 0.0). For a completely false measure of bullying to find such spectacular effects is really suspicious, and now we need to consider not only the differences between the types of kids who are and aren’t bullied, but the differences between the types of parents who do and don’t think their kids are being bullied.

Since I insisted on giving this post a silly title, I will now share with you the most interesting perspective on psychology and the “stop hitting yourself” phenomenon I have read all week. This is from Jonathan Haidt on Kohlberg’s moral stages:

During elementary school, most children move on to the two conventional stages, becoming adept at understanding and even manipulating rules and social conventions. This is the age of petty legalism that most of us who grew up with siblings remember well (“I’m not hitting you. I’m using your hand to hit you. Stop hitting yourself!”). Kids at this stage rarely question the legitimacy of authority, but learn to maneuver within and around the constraints that adults impose on them.

I always just thought that was a really dickish joke. I didn’t realize it had a deep philosophical underpinning.

There are two sets of arguments against including wheat in the diet. First, wheat is a carbohydrate, and some people support low carbohydrate diets. Second, something might be especially dangerous about wheat itself.

It was much easier to figure out the state of the evidence on low-carbohydrate diets. They seem to be at least as good and maybe a little better for weight loss than traditional diets, but this might just be because there are lots of carbohydrates that taste very good and when forced to avoid them, people eat less stuff. They may or may not positively affect metabolic parameters and quality of life. (1, 2, 3, 4). They don’t seem to cause either major health benefits or major health risks in the medium term, which is the longest term for which there is good data available – for example, they have no effect on cancer rates. Overall they seem solid but unspectacular. But there’s a long way between “low carbohydrate diet” and “stop eating wheat”.

So I was more interested in figuring out what was going on with wheat in particular.

Wheat contains chemicals [citation needed]. The ones that keep cropping up (no pun intended) in these kinds of discussions are phytates, lectins, gluten, gliadin, and agglutinin, the last three of which for your convenience have been given names that all sound alike.

Various claims have been made about these chemicals’ effects on health. These have some prima facie plausibility. Plants don’t want to be eaten [citation needed] and they sometimes fill their grains with toxins to discourage animals from eating them. Ricin, a lectin in the seeds of the castor oil plant so toxic it gets used in chemical warfare, is a pretty good example. Most toxins are less dramatic, and most animals have enzymes that break down the toxins in their preferred food sources effectively. But if humans are insufficiently good at this, maybe because they didn’t evolve to eat wheat, some of these chemicals could be toxic to humans.

On the other hand, this same argument covers every pretty much every grain and vegetable and a lot of legumes – pretty much every plant-based food source except edible fruits. So we need a lot more evidence to start worrying about wheat.

I found the following claims about negative effects of wheat:

1. Some people without celiac disease are nevertheless sensitive to gluten.
2. Wheat increases intestinal permeability, causing a leaky gut and autoimmune disease.
3. Digestion of wheat produces opiates, which get you addicted to wheat.
4. Wheat something something something autism and schizophrenia.
5. Wheat has been genetically modified recently in ways that make it much worse for you.
6. The lectins in wheat interfere with leptin receptors, making people leptin resistant and therefore obese.

I’ll try to look at each of those and then turn to the positive claims made about wheat to see if they’re strong enough to counteract them.

Some People Without Celiac Disease Are Sensitive To Gluten – Mostly true but of limited significance

Celiac disease is one source of concern. Everybody on all sides of the wheat debate agree about the basic facts of this condition, which affects a little less than 1% of the population. They have severe reactions to the gluten in wheat. Celiac disease is mostly marked by gastroentereological complaints – diarrhea, bloating, abdominal pain – but it is also associated with vitamin deficiencies, anaemia, skin reactions, infertility, and “malaise”. It can be pretty straightforwardly detected by blood tests and gut biopsies and is not subtle.

People start to disagree about the existence of “gluten sensitivity”, which if it existed would be a bad reaction to gluten even in people who don’t test positive for celiac disease. Many people believe they have gastrointestinal (or other) symptoms that go away when they eat gluten-free diets, but science can’t find anything wrong with their intestines that could be causing the problems.

A recent study somewhat vindicated these people. Biesiekierski 2011 describes a double-blind randomized controlled trial: people who said they had “gluten-sensitive” irritable bowel syndrome were put on otherwise gluten-free diets and then randomly given either gluten or a placebo. They found that the patients given gluten reported symptoms (mostly bowel-related and tiredness) much more than those given placebo (p = 0.0001) but did not demonstrate any of the chemical, immunological, or histological markers usually associated with celiac disease. A similar Italian study found the same thing, except that they did find a higher rate of anti-gluten antibodies in their patients. Another study found that non-celiacs with antibodies to gluten had higher rates of mortality. And another study did find a histological change in bowel barrier function on this group of patients with the introduction of gluten. And another study from the same group found that maybe FODMAPs, another component of wheat, are equally or more responsible.

The journal Gastroenterology, which you may not be surprised to learn is the leading journal in the field of gastroenterology, proclaims:

The current working definition of nonceliac gluten sensitivity (NCGS) is the occurrence of irritable bowel syndrome (IBS)-like symptoms after the ingestion of gluten and improvement after gluten withdrawal from the diet after exclusion of celiac disease based on negative celiac serologies and/or normal intestinal architecture and negative immunoglobulin (Ig)E-mediated allergy tests to wheat. Symptoms reported to be consistent with NCGS are both intestinal (diarrhea, abdominal discomfort or pain, bloating, and flatulence) and extra-intestinal (headache, lethargy, poor concentration, ataxia, or recurrent oral ulceration). These criteria strongly and conveniently suggest that NCGS is best understood as a subset of IBS or perhaps a closely related but distinct functional disorder. Although the existence of NCGS has been slowly gaining ground with physicians and scientists, NCGS has enjoyed rapid and widespread adoption by the general public.

But even this isn’t really that interesting. Maybe some people with irritable bowel syndrome or certain positive antibodies should try avoiding gluten to see if it helps their specific and very real symptoms. At most ten percent of people are positive antibody testing, and not all of those even have symptoms. That’s still a far cry from saying no one should eat wheat.

But the anti-wheat crowd says an alternative more sensitive antibody test could raise sensitivity as high as a third of the population. The test seems to have been developed by a well-respected and legitimate doctor, but it hasn’t as far as I can tell been submitted for peer review or been confirmed by any other source. Meh.

That’s boring anyway. The real excitement comes from sweeping declarations that the entire population is sensitive to wheat.

Wheat Increases Intestinal Permeability Causing A Leaky Gut – Probably true, of uncertain significance

There are gluten-induced mucosal changes in subjects without small bowel disease. And gliadin increases intestinal permeability in the test tube, which should be extremely concerning to any test tubes reading this.

But probably the bigger worry here are lectins, which include wheat germ agglutinin. WGA affects the intestinal permeability of rats, which should be extremely concerning to any rats reading this. The same substance has been found to produce pro-inflammatory cytokines and interfere with the growth of various organs including the gut.

So there’s pretty good evidence that chemicals in wheat can increase intestinal permeability. Who cares?

For years, “leaky gut syndrome” was an alternative medicine diagnosis that was soundly mocked by the mainstream medical establishment. Then the mainstream medical establishment confirmed it existed and did that thing where they totally excused their own mocking of it but were ABSOLUTELY OUTRAGED that the alternative medicine community might have in some cases been overenthusiastic about it.

Maybe I’m being too harsh. The alternative medicine community often does take “leaky gut syndrome” way too far.

On the other hand, it’s probably real and Nature Clinical Practice is now publishing papers saying it is “a key ingredient in the pathogenesis of autoimmune diseases” and “offers innovative, unexplored approaches for the treatment of these devastating diseases” and gut health has been deemed “a new objective in medicine”. Preliminary changes to intestinal permeability have been found in asthma, in diabetes, and even in depression.

But it’s not yet clear if this is cause and effect. Maybe the stress of having asthma increases intestinal permeability somehow. Or maybe high intestinal permeability causes asthma somehow. It sure seems like the latter might work – all sorts of weird antigens and stuff from food can make it into the bloodstream and alarm the immune system – but right now this is all speculative.

So what we have is some preliminary evidence that wheat increases intestinal permeability, and some preliminary evidence that increased intestinal permeability is bad for you in a variety of ways.

And I don’t doubt that those two facts are true, but my knowledge of this whole area is so weak that I wonder how much to worry.

What other foods increase intestinal permeability? Do they do it more or less than wheat? Has anyone been investigating this? Are there common things that affect intestinal permeability a thousand times more than wheat does, such that everything done by wheat is totally irrelevant in comparison?

Do people without autoimmune diseases suffer any danger from increased intestinal permeability? How much? Is it enough to offset the many known benefits of eating wheat (to be discussed later?) Fiber seems to decrease intestinal permeability and most people get their fiber from bread; would decreasing bread consumption make leaky gut even worse?

I find this topic really interesting, but in a “I hope they do more research” sort of way, not an “I shall never eat bread ever again” sort of way.

Digestion Of Wheat Produces Opiates, Which Get You Addicted To Wheat – Probably false, but just true enough to be weird

Dr. William Davis, a cardiologist, most famously makes this claim in his book Wheat Belly. He says that gliadin (a component of gluten) gets digested into opiates, chemicals similar to morphine and heroin with a variety of bioactive effects. This makes you addicted to food in general and wheat in particular, the same way you would get addicted to morphine or heroin. This is why people are getting fat nowadays – they’re eating not because they’re hungry, but because they’re addicted. He notes that drugs that block opiates make people want wheat less.

Does Wheat Make Us Fat And Sick, a review published in the Journal of Cereal Science (they have journals for everything nowadays) is a good rebuttal to some of Davis’ claims and a good pro-wheat resource in general.

They say that although gliadin does digest into opiates, those opiates are seven unit peptides and so too big to be absorbed from the gut to the bloodstream.

(note that having opiates in your gut isn’t a great idea either since there are lots of nerves there controlling digestion that can be affected by these drugs)

But I’m not sure this statement about absorption is even true. First, large proteins can sometimes make it into the gut. Second, if all that leaky gut syndrome stuff above is right, maybe the gut is unusually permeable after wheat consumption. Third, there have been sporadically reported cases of gliadin-derived opiates found in the urine, which implied they got absorbed somehow.

There’s a better counterargument on the blog The Curious Coconut. She notes that there’s no evidence these peptides can cross the blood-brain barrier, a precondition for having any psychological effects. And although the opiate-blocker naloxone does decrease appetite, this effect is not preferential for wheat, and probably more related to the fact that opiates are the way the brain reminds itself it’s enjoying itself (so that opiate-blocked people can’t enjoy eating as much).

And then there’s the usual absence of qualifiers. Lots of things are “chemically related” to other chemicals without having the same effect; are gliadin-derived opiates addictive? Are they produced in quantities high enough to be relevant in real life? Corn, spinach, and maybe meat can all get digested into opiates – is there any evidence wheat-derived opiates are worse? This is really sketchy.

The most convincing counterargument is that as far as anyone can tell, wheat makes people eat less, not more:

Prospective studies suggest that weight gain and increases in abdominal adiposity over time are lower in people who consume more whole grains. Analyses of the Physicians’ Health Study (27) and the Nurses’ Health Study (26) showed that those who consumed more whole grain foods consistently weighed less than those who consumed fewer whole grain foods at each follow-up period of the study. Koh-Banerjee et al. (27) estimated that for every 40-g increase in daily whole grain intake, the 8-y weight gain was lower by 1.1 kg.

I’ll discuss this in more detail later, but it does seem like a nail in the coffin for the “people eat too much because they’re addicted to wheat” theory.

Still, who would have thought that wheat being digested into opiates was even a little true?

Wheat Something Something Something Autism And Schizophrenia – Definitely weird

Since gluten-free diets get tried for everything, and everything gets tried for autism, it was overdetermined that people would try gluten-free diets for autism.

All three of the issues mentioned above – immune reactivity to gluten, leaky guts, and gliadin-derived opiates – have been suggested as mechanisms for why gluten free diets might be useful in autism.

Of studies that have investigated, a review found that seven reported positive results, four negative results, and two mixed results – but that all of the studies involved were terrible and the ones that were slightly less terrible seemed to be more negative. The authors described this as evidence against gluten-free diets for autism, although someone with the opposite bias could have equally well looked at the same review and described it as supportive.

However, a very large epidemiological study found (popular article, study abstract) that people with antibodies to gluten had three times the incidence of autism spectrum disease than people without, and that the antibodies preceded the development of the condition.

Also, those wheat-derived opioids from the last section – as well as milk-derived opioids called casomorphins – seem to be detected at much higher rates in autistic people.

Both of these factors may have less to do with wheat in particular and more to do with some general dysregulation of peptide metabolism in autism. If for some reason the gut kept throwing peptides into the body inappropriately, this would disrupt neurodevelopment, lead to more peptides in the urine, and give the immune system more chance to react to gluten.

The most important thing to remember here is that it would be really wrong to say wheat might be “the cause” of autism. Most likely people do not improve on gluten-free diets. While there’s room to argue that people might have picked up a small signal of them improving a little, the idea that this totally removes the condition is right out. If we were doing this same study with celiac disease, we wouldn’t be wasting our time with marginally significant results. Besides, we know autism is multifactorial, and we know it probably begins in utero.

Schizophrenia right now is in a similar place. Schizophrenics are five to seven times more likely to have anti-gliadin antibodies as the general population. We can come up with all sorts of weird confounders – maybe antipsychotic medications increase gut permeability? – but that’s a really strong result. And schizophrenics have frank celiac disease at five to ten times the rate of the general population. Furthermore, a certain subset of schizophrenics sees a dramatic reduction in symptoms when put on a strict gluten-free diet (this is psychiatrically useless, both because we don’t know which subset, and because given how much trouble we have getting schizophrenics to swallow one lousy pill every morning, the chance we can get them to stick to a gluten-free diet is basically nil). And like those with autism, schizophrenics show increased levels of weird peptides in their urine.

But a lot of patients with schizophrenia don’t have reactions to gluten, a lot don’t improve on a gluten free diet, and other studies question the research showing that any of them at all do.

The situation here looks a lot like autism – a complex multifactorial process that probably isn’t caused by gluten but where we see interesting things going on in the vague territory of gluten/celiac/immune response/gut permeability/peptides, with goodness only knows which ones come first and which are causal.

Wheat Has Been Genetically Modified Recently In Ways That Make It Much Worse For You – Probably true, especially if genetically modified means “not genetically modified” and “recently” means “nine thousand years ago”

If you want to blame the “obesity epidemic” or “autism epidemic” or any other epidemic on wheat, at some point you have to deal with people eating wheat for nine thousand years and not getting epidemics of these things. Dr. Davis and other wheat opponents have turned to claims that wheat has been “genetically modified” in ways that improve crop yield but also make it more dangerous. Is this true?

Wheat has not been genetically modified in the classic sense, the one where mad scientists with a god complex inject genes from jellyfish into wheat and all of a sudden your bread has tentacles and every time you try to eat it it stings you. But it has been modified in the same way as all of our livestock, crops, and domestic pets – by selective breeding. Modern agricultural wheat doesn’t look much like its ancient wild ancestors.

The Journal Of Cereal Science folk don’t seem to think this is terribly relevant. They say:

Gliadins are present in all wheat lines and in related wild species. In addition, seeds of certain ancient types of tetraploid wheat have even greater amounts of total gliadin than modern accessions…There is no evidence that selective breeding has resulted in detrimental effects on the nutritional properties or health benefits of the wheat grain, with the exception that the dilution of other components with starch occurs in modern high yielding lines (starch comprising about 80% of the grain dry weight). Selection for high protein content has been carried out for bread making, with modern bread making varieties generally containing about 1–2% more protein (on a grain dry weight basis) than varieties bred for livestock feed when grown under the same conditions. However, this genetically determined difference in protein content is less than can be achieved by application of nitrogen fertilizer. We consider that statements made in the book of Davis, as well as in related interviews, cannot be substantiated based on published scientific studies.

In support of this proposition, in the test tube ancient grains were just as bad for celiac patients’ immune systems as modern ones.

And yet in one double-blind randomized-controlled trial, people with irritable bowel syndrome felt better on a diet of ancient grains than modern ones (p < 0.0001); and in another, people on an ancient grain diet had lower inflammatory markers and generally better nutritional parameters than people on a modern grain one. Isn’t that interesting?

Even though it’s a little bit weird and I don’t think anyone understands the exact nutrients at work, sure, let’s give this one to the ancient grain people.

The Lectins In Wheat Interfere With Leptin Receptors, Making People Leptin Resistant And Therefore Obese – Currently at “mere assertion” level until I hear some evidence

So here’s the argument. Your brain has receptors for the hormone leptin, which tells you when to stop eating. But “lectin” sounds a lot like “leptin”, and this confuses the receptors, so they give up and tell you to just eat as much as you want.

Okay, this probably isn’t the real argument. But even though a lot of wheat opponents cite the heck out of this theory, the only presentation of evidence I can find is Jonsson et al (2005), which points out that there are a lot of diseases of civilization, they seem to revolve around leptin, something common to civilization must be causing them, and maybe that thing could be lectin.

But civilization actually contains more things than a certain class of proteins found in grains! There’s poor evidence of lectin actually interfering with the leptin receptor in humans. The only piece of evidence they provide is a nonsignificant trend toward more cardiovascular disease in people who eat more whole grains in one study, and as we will see, that is wildly contradicted by all other studies.

This one does not impress me much.

Wheat Is Actually Super Good For You And You Should Have It All The Time – Probably more evidence than the other claims on this list

Before I mention any evidence, let me tell you what we’re going to find.

We’re going to find very, very many large studies finding conclusively that whole grains are great in a lot of different ways.

And we’re not going to know whether it’s at all applicable to the current question.

Pretty much all these studies show that people with some high level of “whole grain consumption” are much healthier than people with some lower level of same. That sounds impressive.

But what none of these studies are going to do a good job ruling out is that whole grain is just funging against refined grain which is even worse. Like maybe the people who report low whole grain consumption are eating lots of refined grain, and so more total grain, and the high-whole-grain-consumption people are actually eating less grain total.

They’re also not going to rule out the universal problem that if something is widely known to be healthy (like eating whole grains) then the same health-conscious people who exercise and eat lots of vegetables will start doing it, so when we find that the people doing it are healthier, for all we know it’s just that the people doing it are exercising and eating vegetables.

That having been said, eating lots of whole grain decreases BMI, metabolic risk factors, fasting insulin, and body weight (1, 2, 3, 4,5.)

The American Society For Nutrition Symposium says:

Several mechanisms have been suggested to explain why whole grain intake may play a role in body weight management. Fiber content of whole grain foods may influence food volume and energy density, gastric emptying, and glycemic response. Whole grains has also been proposed to play an important role in promoting satiety; individuals who eat more whole grain foods may eat less because they feel satisfied with less food. Some studies comparing feelings of fullness or actual food intake after ingestion of certain whole grains, such as barley, oats, buckwheat, or quinoa, compared with refined grain controls indicated a trend toward increased satiety with whole grains. These data are in accordance with analyses determining the satiety index of a large number of foods, which showed that the satiety index of traditional white bread was lower than that of whole grain breads. However, in general, these satiety studies have not observed a reduction in energy intake; hence, further research is needed to better understand the satiety effects of whole grains and their impact on weight management.

Whole grains, in some studies, have also been observed to lower the glycemic and insulin responses, affect hunger hormones, and reduce subsequent food intake in adults. Ingestion of specific whole grains has been shown to influence hormones that affect appetite and fullness, such as ghrelin, peptide YY, glucose-dependent insulinotropic polypeptide, glucagon-like peptide 1, and cholecystokinin. Whole grain foods with fiber, such as wheat bran or functional doses of high molecular weight β-glucans, compared with lower fiber or refined counterparts have been observed to alter gastric emptying rates. Although it is likely that whole grains and dietary fiber may have similar effects on satiety, fullness, and energy intake, further research is needed to elucidate how, and to what degree, short-term satiety influences body weight in all age groups.

Differences in particle size of whole grain foods may have an effect on satiety, glycemic response, and other metabolic and biochemical (leptin, insulin, etc.) responses. Additionally, whole grains have been suggested to have prebiotic effects. For example, the presence of oligosaccharides, RS, and other fermentable carbohydrates may increase the number of fecal bifidobacteria and lactobacilli (49), thus potentially increasing the SCFA production and thereby potentially altering the metabolic and physiological responses that affect body weight regulation.

In summary, the current evidence among a predominantly Caucasian population suggests that consuming 3 or more servings of whole grains per day is associated with lower BMI, lower abdominal adiposity, and trends toward lower weight gain over time. However, intervention studies have been inconsistent regarding weight loss

The studies that combined whole and refined grains are notably fewer. But Dietary Intake Of Whole And Refined Grain Breakfast Cereals And Weight Gain In Men finds that among 18,000 male doctors, those who ate breakfast cereal (regardless of whether it was whole and refined) were less likely to become overweight several years later than those who did not (p = 0.01). A book with many international studies report several that find a health benefit of whole grains, several that find a health benefit of all grains (Swedes who ate more grains had lower abdominal obesity; Greeks who ate a grain-rich diet were less likely to become obese; Koreans who ate a “Westernized” bread-and-dairy diet were less likely to have abdominal obesity) and no studies that showed any positive association between grains and obesity, whether whole or refined.

I cannot find good interventional trials on what happens when a population replaces non-grain with grain.

On the other hand, Dr. Davis and his book Wheat Belly claim:

Typically, people who say goodbye to wheat lose a pound a day for the first 10 days. Weight loss then slows to yield 25-30 pounds over the subsequent 3-6 months (differing depending on body size, quality of diet at the start, male vs. female, etc.)

Recall that people who are wheat-free consume, on average, 400 calories less per day and are not driven by the 90-120 minute cycle of hunger that is common to wheat. It means you eat when you are hungry and you eat less. It means a breakfast of 3 eggs with green peppers and sundried tomatoes, olive oil, and mozzarella cheese for breakfast at 7 am and you’re not hungry until 1 pm. That’s an entirely different experience than the shredded wheat cereal in skim milk at 7 am, hungry for a snack at 9 am, hungry again at 11 am, counting the minutes until lunch. Eat lunch at noon, sleepy by 2 pm, etc. All of this goes away by banning wheat from the diet, provided the lost calories are replaced with real healthy foods.”

Needless to say, he has no studies supporting this assertion. But the weird thing is, his message board is full of people who report having exactly this experience, my friends who have gone paleo have reported exactly this experience, and when I experimented with it, I had pretty much exactly this experience. Even the blogger from whom I took some of the strongest evidence criticizing Davis says she had exactly this experience.

The first and most likely explanation is that anecdotal evidence sucks and we should shut the hell up. Are there other, less satisfying explanations?

Maybe completely removing wheat from the diet has a nonlinear effect relative to cutting down on it? For example, in celiac disease there is no such thing as “partially gluten free” – if you have any gluten at all, your disease comes back in full force. This probably wouldn’t explain Dr. Davis’ observation – neither I nor my other wheatless-experimentation friends were as scrupulous as a celiac would have to be. But maybe there’s a nonlinear discrepancy between people who have 75% the wheat of a normal person and 10% the wheat of a normal person?

Maybe there’s an effect where people who like wheat but remove it from the diet are eating things they don’t like, and so eat less of them? But people who don’t like wheat like other stuff, and so eat lots of that?

Maybe wheat in those studies is totally 100% a confounder for whether people are generally healthy and follow their doctor’s advice, and the rest of the doctor’s advice is really good but the wheat itself is terrible?

Maybe cutting out wheat has really positive short-term effects, but neutral to negative long-term effects?

Maybe as usual in these sorts of situations, the simplest explanation is best.

Final Thoughts

Non-celiac gluten sensitivity is clearly a real thing. It seems to produce irritable bowel type symptoms. If you have irritable bowel type symptoms, it might be worth trying a gluten-free diet for a while. But the excellent evidence for its existence doesn’t seem to carry over to the normal population who don’t experience bowel symptoms.

What these people have are vague strands of evidence. Something seems to be going on with autism and schizophrenia – but most people don’t have autism or schizophrenia. The intestinal barrier seems to become more permeable with possible implications for autoimmune diseases – but most people don’t have autoimmune disease. Some bad things seem to happen in rats and test tubes – but most people aren’t rats or test tubes.

You’d have to want to take a position of maximum caution – wheat seems to do all these things, and even though none of them in particular obviously hurt me directly, all of them together make it look like the body just doesn’t do very well with this substance, and probably other ways the body doesn’t do very well with this substance will turn up, and some of them probably affect me.

There’s honor in a position of maximum caution, especially in a field as confusing as nutrition. It would not surprise me if the leaky gut connection turned into something very big that had general implications for, for example, mental health. And then people who ate grain might regret it.

But stack that up against the pro-wheat studies. None of them are great, but they mostly do something the anti-wheat studies don’t: show direct effect on things that are important to you. Most people don’t have autism or schizophrenia, but most people do have to worry about cardiovascular disease. We do have medium-term data that wheat doesn’t cause cancer, or increase obesity, or contribute to diabetes, or any of that stuff, and at this point solely based on the empirical data it seems much more likely to help with those things than hurt.

I hope the role of intestinal permeability in autoimmune disease gets the attention it deserves – and when it does, I might have to change my mind. I hope people stop being jerks about gluten sensitivity, admit it exists, and find better ways to deal with it. And if people find that eliminating bread from their diet makes them feel better or lose weight faster, cool.

But as far as I can tell the best evidence is on the pro-wheat side of things for most people at most times.

[EDIT: An especially good summary of the anti-wheat position is 6 Ways Wheat Can Destroy Your Health. An especially good pro-wheat summary is Does Wheat Make Us Fat And Sick?]

The study surveyed tobacco smokers for various information including whether they smoked e-cigarettes in addition to their tobacco. A year later, they went back and surveyed everyone again and asked them if they were still smoking tobacco. And the people smoking the e-cigarettes were no more likely to have quit than the others.

Let’s transition from reality to Hypothetical World. In Hypothetical World, there are only two kinds of smokers, Short Smokers and Long Smokers. The moment someone smokes their first cigarette, God flips a coin and assigns them into one of the two groups based on the result. Short Smokers are predestined to smoke for exactly one year before quitting; Long Smokers are predestined to smoke for exactly fifty years before quitting.

A scientist in Hypothetical World wants to discovery what percent of first-time-smokers become Short Smokers versus Long Smokers (the real proportion is 50-50 since God’s coin is fair, but she doesn’t know that). So she uses the same methodology as this study. She hangs around a tobacco shop and accosts the first thousand people who come in to buy cigarettes, getting their names and phone numbers. Then a year and a day later, she calls them all up to ask if they are still smoking – since anyone who keeps smoking for a year and a day must be a Long Smoker.

So she finds something close to 2% of people are Short Smokers and a whopping 98% are Long Smokers. She incorrectly concludes that God is rolling a d100 and only assigning Short Smoker status to those who come up 99 or 00.

Don’t see why she would make this mistake? Consider a particular generation of Hypothetical people over their lifetimes. The Short Smokers will only smoke a single year out of their lifetime; the Long Smokers will smoke fifty years. When the scientist does her study in a randomly selected year, she only has a 1/average_lifespan chance of catching any given Short Smoker, but a 50/average_lifespan chance of catching a Long Smoker. So, her original sample will contain fifty times more Long Smokers than Short Smokers, and she will mistakenly conclude that their pattern is fifty times more common.

Now transition back to reality. Suppose there are two types of e-cigarette users – successful and unsuccessful. The successful e-cigarette users try e-cigarettes, immediately decide they are better than regular cigarettes, and switch to using e-cigarettes exclusively within one month. The unsuccessful e-cigarette users try e-cigarettes but just don’t get everything they love about tobacco from them. They sort of futz around with e-cigarettes and regular cigarettes and tell themselves that one of these days, they’re really going to stop the regular ones entirely and transition totally to e-cigarettes. These people continue futzing for let’s say ten years before they either finally quit tobacco, give up on e-cigarettes, or die.

In that case, any sample of tobacco smokers taken at a particular time will include a hundred twenty times as many unsuccessful e-cigarette users as successful ones. We expect unsuccessful e-cigarette users to continue their past pattern of futzing around, so it’s not surprising that this sort of sample finds most e-cigarette users not only can’t easily quit tobacco using e-cigarettes, but actually have a harder time quitting tobacco than normal smokers – they’ve already been preselected as The Group That Even E-Cigarettes Can’t Help; as The Group That Tried Something Billed As An Anti-Smoking Aid But Failed At It. It’s a pretty general rule of medicine that people who failed treatment once are more likely to fail treatment a second time.

This is a very speculative explanation and I haven’t heard anyone respectable at a major institution advance it yet, but it seems to me like the most likely reason for these findings. All I have to go for with the study right now is a preliminary “research letter”, but hopefully we’ll know more when the real thing comes out.

Lest this post be entirely pro-drug, here’s a clip of my addiction-medicine teacher and sometime-boss lecturing people about marijuana on Fox News last weekend. He is a great doctor and it’s neat to see him finally getting some of the celebrity he deserves. Even though his politics are terrible.

No one in the study population had Alzheimers yet, so everyone did okay on the memory test. But subjects who knew they had ApoE4 did significantly worse than subjects who did have ApoE4 but didn’t know it. The subjects who knew they didn’t have ApoE4 didn’t do any better on the memory test than other subjects who didn’t have ApoE4 but had not yet heard the good news, but they did give subjectively better ratings of their memory ability.

The medical community concludes from this that letting people know their genetic risks may be dangerous, and although I hate to admit it, they have a point.

Unfortunately, the study doesn’t give the methodological details I need to really understand the implications.

We know that the researchers waited eight months between giving the genetic test results and doing the memory tests. That’s good.

But was it the same researcher doing both the genes and the memory parts of the study? Was it in the same building? Did they start the memory tests by saying “Hi! I’m Dr. Lineweaver! You may remember me from such medical experiments as the one eight months ago in which you were told that you had a high risk of getting Alzheimer’s disease”?

Or did they sneakily pretend to be a separate study entirely and try to avoid mentioning the A-word throughout?

It would not surprise me if – having been primed with a reminder of their Alzheimer testing results – the subjects then performed worse on a memory test that was given immediately after in an obviously related context.

It would be much more surprising – though still not totally unbelievable – if subjects, having been told they had a high risk of Alzheimers, just went around for eight months having slightly worse memory which was reflected on everything they did including the memory test administered by the researchers.

New England Journal of Medicine compares the finding to “stereotype threat”, the phenomenon in which people can for example sometimes make women perform worse on math tests simply by telling them that it is a “test of their innate mathematical abilities” – something that women are stereotypically bad at.

The memory tests the researchers were giving are equivalent to the “innate mathematical abilities” condition in the stereotype threat research – a test clearly intended to measure how good their memory was in a very scientific way. The activities of daily living that require memory – keeping appointments, paying bills on time, et cetera – are the equivalent of the condition in stereotype threat experiments where researchers just give women a normal math test without introduction and stereotype threat is not seen.

So I see two ways in which we could get results like the ones in this study without any broader implications of ApoE4 testing harming the elderly in general.

First, being called to the same study in which the ApoE4 results were given could have primed their worries about Alzheimers and made them do especially bad on the study’s memory test compared to their usual memory.

Second, the study’s memory test could have been official-looking enough that it activated their stereotype of themselves as having innately poor memory, when, concordant with stereotype threat research, that stereotype doesn’t harm their everyday memory-requiring activities.

In either of these cases, the study would have some very limited implications, which the authors describe in an appropriately circumscribed way: “The patient’s knowledge of his or her genotype and risk of Alzheimer’s disease should be considered when evaluating cognition in the elderly.”

But this would not imply that genetic testing elderly people for ApoE4 is risky and can itself cause them to develop forgetfulness and other Alzheimer’s symptoms.

I worry that the medical community is going to miss this subtlety and start “raising awareness” of the possibility that genetic testing can cause harmful side effects, before finishing the hard task of discovering if that’s actually true.

A very strange but creative methodology with by which to study the notoriously complicated field of diet: scientists find that a gene that makes vegetables taste better also makes you live longer. Weak evidence suggesting that eating more vegetables makes you live longer? Maybe!

A Critical Review of the First Ten Years of Candidate Gene by Environment Interaction Research in Psychiatry. Key phrase: “Ninety-six percent of novel cG×E studies were significant compared with 27% of replication attempts.” Note that gene x environment interaction studies are a very particular kind of study that is especially easy to do bad work on and this might not generalize to other types of genetics research – but that at least to some degree it probably does.

A while ago there was great excitement at the discovery that the drug rapamycin extended lifespan in mice. Although this finding has since been replicated and seems broadly correct, the bad news seems to be that is now clear that the drug just treats some very specific deadly pathologies (like cancer) and does not fight or slow aging. Although if the bad news is that a drug cures cancer, we’re still doing pretty well.

But if you absolutely must have some miracle substance that might cure aging in lab animals to be excited about, you’ll be happy to know that rhodiola extends the lifespans of fruit flies 24% and delays age-related loss in physical performance. Also it might be a nootropic or antidepressant or something.

Speaking of miracles, Skin Abnormality May Prove Biological Basis For Fibromyalgia. I predict this will probably turn out to be nothing, the same way everyone was super excited a few years ago that we’d discovered that the real cause of multiple sclerosis was venous outflow obstruction and then it didn’t replicate, but until then at least fibromyalgia sufferers will get a few good years in of “SEE! I TOLD YOU IT WAS BIOLOGICAL AND YOU DIDN’T BELIEVE ME!”

Not technically a study but a good thing to include here: Eight Toxic Foods and a Little Chemical Education. Describes some of the scare claims the media sometimes makes about chemicals and health risks and dissects them carefully and rigorously.

And if that was too basic for you, here’s the Epic-level version of the same thing: the Last Psychiatrist dissects claims made in a presentation on the drug Geodon. This is old, but I just found it and it terrifies me, in that I thought I knew what to look for and yet this study would have completely passed all the filters I usually have to protect myself from this sort of thing. A good example of how a drug company can run a seemingly rigorous study that stays far away from anything even resembling data falsification or cover-ups – and yet still get exactly the results they want.

Here’s Scientific American giving a good exposition of one of the best current theories about why we sleep. Also, it apparently has evidence behind it now, which it didn’t the last time I heard about it. Still doesn’t really explain why some people can go without sleep completely, but maybe that’s why they brought in the “local sleep” points.

Back when people realized it was easy to get positive results from a drug for spurious reasons, they started adding a control group to the experiment. Now that people have realized it’s easy to get positive results from a controlled trial for spurious reasons, is it time to go one meta-level up and add a control experiment on to the study? One group takes an experiment used to “prove” that SSRIs cause gastric bleeding, compares it to dozens of “control experiments” run with drugs that don’t cause gastric bleeding, and finds that, although the real experiment reported positive results, it in fact does no different than placebo experiments. This is really clever although probably impractical in most cases.

Psychotherapy over the Internet works at least as well and probably better than face-to-face psychotherapy, says a study this month, adding to the small mountain of evidence saying the same. A friend of mine uses online psychotherapy and says it’s easier and more productive because the therapist is less of a Terrifying Authority Figure. Also good for people who want a psychologist who will have severe difficulty calling the cops on them and having them committed. Also good for social phobics who are currently required to leave the house and hang out at a busy medical office if they want to get treatment for their social phobia who the heck came up with this system?

“Adoption study of human obesity” sounds like something you would get from a Things Scott Is Interested In Mad Libs, along with “utilitarian behavioral genetics” or “double-blind placebo-controlled cuddling of cute girls”. But it turns out this is a real field that various people have looked into, and the conclusion of most of the studies, including a very rigorous one in Denmark published in NEJM and a huge UK one by Robert Plomin agree that whether the parents who raise you are obese has zero impact on whether you will become obese, but whether your biological parents whom you may never meet are obese has massive impact on whether you will become obese. This doesn’t completely disprove the idea that the childhood environment affects obesity – it could still be that whether or not parents are good at teaching their children not to be obese just has zero correlation with whether the parents themselves are obese – but it sure casts a lot of doubt on environmental hypotheses and confirms that genetics plays a very big role.

On the other hand, here’s a study from 2011 which shows that people with lower Conscientiousness and higher Impulsivity are much more likely to be obese – “Participants who scored in the top 10% of impulsivity weighed, on average, 11 kg more than those in the bottom 10%”. LWers pointed out that this is not itself incompatible with genetics, since most personality traits are themselves somewhat heritable.

On the mutant third hand, if it’s all just impulsive people who have been poorly trained by their parents, why are wild animals getting fatter?

JAMA Psychiatry: Rapid Improvement of Acute Schizophrenia Symptoms After Intravenous Sodium Nitroprusside. Anything that improves schizophrenia symptoms is good news, but this is especially interesting for two reasons. First, the rapid and dramatic effect is easier to replicate and less corruptible than the usual “take this pill for a month and maybe you will feel better”, and is reminiscent of the very similar effect of ketamine on depression. Second, sodium nitroprusside is a drug used for high blood pressure without any previously known relevance to psychiatry, opening up a whole new direction in research. The small but interesting field of nitric oxide in schizophrenia is about to get a lot more scrutiny.

How The Mid-Victorians Worked, Ate, and Died (h/t Michael Vassar) claims that the mid-Victorian period was a golden age of health during which life expectancy was as high or higher than today and the diseases we consider a “normal part of aging” simply failed to exist. No cancer, no heart disease, just living happily and healthily until you were felled by tuberculosis at age 85. They credit the Victorian diet – in particular its lack of additives and preservatives and its overload of nutrients, the latter made possible by very high calorie diets that compensated for their increased level of physical activity. The authors admit we probably can’t safely replicate such a high calorie diet in today’s sedentary society, but suggest taking lots and lots of vitamins and supplements in order to get the same high nutrient level the Victorians did.

I am really really bad at understanding nutrition (AT LEAST I ADMIT IT!), so I will limit this to attempts to fact-check a few claims, plus some extremely speculative commentary. Let’s start with the thesis:

We argue in this paper, using a range of historical evidence, which Britain and its world-dominating empire were supported by a workforce, an army and a navy comprised of individuals who were healthier, fitter and stronger than we are today. They were almost entirely free of the degenerative diseases which maim and kill so many of us, and although it is commonly stated that this is because they all died young, the reverse is true; public records reveal that they lived as long – or longer – than we do in the 21st century.

And the evidence in support:

The fall in nutritional standards between 1880 and 1900 was so marked that the generations were visibly and progressively shrinking. In 1883 the infantry were forced to lower the minimum height for recruits from 5ft 6 inches to 5ft 3 inches. This was because most new recruits were now coming from an urban background instead of the traditional rural background (the 1881 census showed that over three-quarters of the population now lived in towns and cities). Factors such as a lack of sunlight in urban slums (which led to rickets due to Vitamin D deficiency) had already reduced the height of young male volunteers. Lack of sunlight, however, could not have been the sole critical factor in the next height reduction, a mere 18 years later. By this time, clean air legislation had markedly improved urban sunlight levels; but unfortunately, the supposed ‘improvements’ in dietary intake resulting from imported foods had had time to take effect on the 16–18 year old cohort. It might be expected that the infantry would be able to raise the minimum height requirement back to 5ft. 6 inches. Instead, they were forced to reduce it still further, to a mere 5ft. British officers, who were from the middle and upper classes and not yet exposed to more than the occasional treats of canned produce, were far better fed in terms of their intake of fresh foods and were now on average a full head taller than their malnourished and sickly men.

This is very suspicious. If the British Army started recruiting from different populations, or started lowering their standards generally because they needed more men, these data are useless. I tried to find an alternate source of height data, and the best I could come up with was the extremely thorough Height, Weight, and Body Mass of the British Population Since 1820, who say:

This graph seems to do quite well at picking up real decreases in height like the one starting in the birth cohort of 1825 (possibly due to the Corn Laws?), but it has pretty much nothing to say about any great collapse in the last few years of the 19th century. Assuming the average army recruit was 20 years old. In 1883, they would have been in the 1863 birth cohort; in 1901, in the 1881 birth cohort. But in fact, the 1881 birth cohort is taller on average than the 1863 birth cohort!. The army’s troubles with finding sufficiently tall infantrymen cannot be due to the Amazing Shrinking Englishman – it must be a matter of casting a wider recruiting net or something.

The crude average figures often used to depict the brevity of Victorian lives mislead because they include infant mortality, which was tragically high. If we strip out peri-natal mortality, however, and look at the life expectancy of those who survived the first five years, a very different picture emerges. Victorian contemporary sources reveal that life expectancy for adults in the mid-Victorian period was almost exactly what it is today. At 65, men could expect another ten years of life; and women another eight. This compares surprisingly favourably with today’s figures: life expectancy at birth (reflecting our improved standards of neo-natal care) averages 75.9 years (men) and 81.3 years (women); though recent work has suggested that for working class men and women this is lower, at around 72 for men and 76 for women.

Given that modern pharmaceutical, surgical, anaesthetic, scanning and other diagnostic technologies were self-evidently unavailable to the mid-Victorians, their high life expectancy is very striking, and can only have been due to their health-promoting lifestyle. But the implications of this new understanding of the mid-Victorian period are rather more profound. It shows that medical advances allied to the pharmaceutical industry’s output have done little more than change the manner of our dying. The Victorians died rapidly of infection and/or trauma, whereas we die slowly of degenerative disease. It reveals that with the exception of family planning, the vast edifice of twentieth century healthcare has not enabled us to live longer but has in the main merely supplied methods of suppressing the symptoms of degenerative diseases which have emerged due to our failure to maintain mid-Victorian nutritional standards. Above all, it refutes the Panglossian optimism of the contemporary anti-ageing movement whose protagonists use 1900 – a nadir in health and life expectancy trends – as their starting point to promote the idea of endlessly increasing life span. These are the equivalent of the get-rich-quick share pushers who insisted, during the dot.com boom, that we had at last escaped the constraints of normal economics. Some believed their own message of eternal growth; others used it to sell junk bonds they knew were worthless. The parallels with today’s vitamin pill market are obvious, but this also echoes the way in which Big Pharma trumpets the arrival of each new miracle drug.

I was wondering how long it would take “Big Pharma” to make an appearance.

Anyway, here we turn to Health and Welfare During Industrialization, which includes the following table:

This is confusing, so let me explain.

I’ve taken the mortality tables from Steckel and deleted everything before age 25. Clayton and Rowbotham admit that infant mortality has improved since the Victorian period, so we’re not counting that.

I’ve marked the 1873 to 1877 period – the last period unambiguously before Clayton and Rowbotham’s 1880 “nutritional collapse” – in gold. This is our baseline.

We notice that the subsequent period actually has lower mortality in nearly every category. This is fine. Just because they’ve invented less nutrititious food doesn’t mean that people have started eating it yet, or that it’s had time to affect their health. Instead, I’ve looked for the worst post-1880 period and marked that in red as the bottom of the “nutritional collapse”. As you can see, it’s 1888 – 1892.

Finally, I’ve taken the period including 1900 – what they describe as “a nadir in health and life expectancy trends”, and marked it in green.

As we can see, the “nadir” – 1900, is actually lower mortality in all age groups than the “golden age” of 1873 – 1877. The “collapse”, if it occurred at all, was a tiny statistical blip that was then easily overpowered by the general secular trend of decreasing mortality.

I don’t know exactly how the authors would like me to consider “people born in 1900″ vs. “people dying in 1900″, but luckily I also happen to have life expectancy at age 65 for Britain for the entire period of 1841 to 2001 (God, I love the Internet!):

There’s nothing on there that could remotely be the kind of collapse they’re talking about, unless it be that little blip around 1890 which is more than compensated for five years later. Annnnnd here’s some more data, divided by occupation and showing life expectancy at age 20:

This isn’t especially high resolution, but it’s interesting in that it suggests adult life expectancy in the Victorian period was around 65 and not the more modern 75. Sure enough, the book I took it from estimates life expectancy at age 20 (ie excluding childhood mortality) to be about 62.5. Unfortunately, I can’t access Clayton and Rowbotham’s sources, so I can’t tell whether they make an even better argument that it really was 75. But this matters for claims like “there was much less cancer”, because cancer increases a lot with age and so if you’re cutting off the end of the age distribution of course you’re going to get less cancer (imagine you learned people never went bald in caveman times – this could be either because they all ate nutritious mammoth meat, or because they all died before age 30).

Overall I’m going to say I wish I could access their sources, but the fact that they don’t quote any numbers or show any graphs in support of their “life expectancy decrease from 1880 to 1900″ hypothesis makes me suspect they don’t have any [EDIT: See bottom of post].

Their next point is that the Victorians had much less cancer and heart disease than people today. This I am totally willing to believe. The authors provide ample evidence in another paper of theirs, and it accords well with very similar evidence presented by Gary Taubes in Good Calories, Bad Calories.

The Victorian Age was before the mass-marketing of cigarettes, before most of the carcinogenic chemicals we inhale all the time, and their diet contained ample fruits and vegetables – foods that we know lower cancer and heart disease risk. Many of them were manual laborers, there were much fewer obesogenic foods, and people were generally normal weight. It is not mysterious or revolutionary to propose they had less cancer and heart disease than we did.

But let’s look at Clayton and Rowbotham’s explanation:

Degenerative diseases are not caused by old age (the ‘wear and tear’ hypothesis); but are driven, in the main, by chronic malnutrition. Our low energy lifestyles leave us depleted in anabolic and anti-catabolic co-factors; and this imbalance is compounded by excessive intakes of inflammatory compounds. The current epidemic of degenerative disease is caused by widespread problem of multiple micro- and phyto-nutrient depletion (Type B malnutrition.)
With the exception of family planning and antibiotics, the vast edifice of twentieth century healthcare has generated little more than tools to suppress symptoms of the degenerative diseases which have emerged due to our failure to maintain mid-Victorian nutritional standards. The only way to combat the adverse effects of Type B malnutrition, and to prevent and / or cure degenerative disease, is to enhance the nutrient density of the modern diet.

Whoa whoa whoa whoa whoa.

Gary Taubes does a pretty good job recording all the primitive cultures that also have no degenerative diseases. The Eskimos are one such. They basically just eat meat. You can also find cultures pretty much anywhere, with any diet, who also lack these degenerative diseases. In fact, I think people who are actually malnourished – starving Africans and the like – still have lower rates of these degenerative diseases as long as they’re not eating a “modern” diet.

To me, this suggests that their “phytonutrient depletion” hypothesis needs to contend with another – that it’s not that we’re not getting enough of the right stuff, but rather that we’re getting too much of the wrong stuff. As far as I know, this is what mainstream nutrition science believes as well as most of the more interesting crackpots, although of course everyone differs as to what the wrong stuff is. It is a promising and venerable theory and nothing the authors of this paper have said thus far casts the slightest doubt upon it. Indeed, they admit that the “nutritional collapse” was caused by the introduction of preserved foods, additives, and cheap sugar.

Our levels of physical activity and therefore our food intakes are at an historic low. To make matters worse, when compared to the mid-Victorian diet, the modern diet is rich in processed foods. It has a higher sodium/potassium ratio, and contains far less fruit, vegetables, whole grains and omega 3 fatty acids. It is lower in fibre and phytonutrients, in proportional and absolute terms; and, because of our high intakes of potato products, breakfast cereals, confectionery and refined baked goods, may have a higher glycemic load. Given all this, it follows that we are inevitably more likely to suffer from dysnutrition (multiple micro- and phytonutrient depletion) than our mid-Victorian forebears…

Since it would be unacceptable and impractical to recreate the mid-Victorian working class 4,000 calorie/day diet, this constitutes a persuasive argument for a more widespread use of food fortification and/or properly designed food supplements (most supplements on the market are incredibly badly designed; they are assembled by companies that do not understand the real nutritional issues that confront us today, and sell us pills containing irrational combinations and doses that can do more harm than good.

To insist, as orthodox nutritionists and dieticians do, that only whole fruit and veg contain the magical, health-promoting ingredients represents little more than the last gasp of the discredited and anti-scientific theory of vitalism (‘Vitalism—the insistence that there is some big, mysterious extra ingredient in all living things—turns out to have been not a deep insight but a failure of imagination’, Daniel Dennett) Even the stately FSA concedes that fruit juices count towards your five-a-day, as do freeze-dried powdered extracts of fruits and vegetables. As our knowledge of phytochemistry and phytopharmacology increases, it has become perfectly acceptable to use rational combinations of the key plant constituents in pill or capsule form.

These arguments are developed in ‘Pharmageddon’, a medical textbook which illustrates how micro- and phyto-nutrients can be specifically combined in order to prevent and treat the chronic degenerative diseases that characterise and dominate the 20th and 21st centuries; and how they could be integrated into our food chain in order to reduce the contemporary and excessively high risks of the degenerative diseases to the far lower mid-Victorian levels.

So first things first. I am almost sure I went to medical school, and the sorts of textbooks I read there all had names like “Essentials Of Biochemistry, Third Edition”, and “Introduction To Gastroenterology” and almost never names like “Pharmageddon”.

Second, the authors cite some sources for their claim that all supplements currently on the market are poorly designed and do more harm than good. These sources show that a bunch of different supplements are either ineffective or cause cancer, and are entirely correct. What they do not cite are any sources that show that “correctly designed” supplements do more good than harm. This is because those sources do not exist because no one has ever discovered that.

The reason the medical community isn’t switching wholesale from evil pharmageddon-causing drugs to all-natural Victorian-approved nutrient supplementation isn’t because they’re in the grip of Big Pharma, it’s because no one can find supplements that are consistently proven to work.

(the medical community actually is saying “eat right and exercise”, but NO ONE LISTENS)

And yes, part of the lack of working supplements is the coordination problem of “there’s no money in studying supplements”. I think it would be great if we could figure out some way to coordinate supplement studies more effectively. But it’s kind of hard to make that case when all the supplement studies that have been done have been total duds and the pro-supplement community has just kept replying with “But those supplements were poorly designed!” and then suggesting a perplexing and extremely contradictory trove of other possible designs, all of which themselves are later found not to work.

One point that seriously enlightened me, thought, was the authors’ comments on vitalism. It’s definitely true that eating whole foods has useful properties (like reducing risk of diseases) and also definitely true that taking supplements that supposedly share vitamins with those foods doesn’t. And it’s also true that I was previously taking this as an Unfortunate Fact Of Life. Which is dumb. Of course we should be trying to figure out what magical property of whole foods makes them effective and then trying to deliver that magical property in a pill. I am disappointed at myself for not realizing that was important sooner, and that alone makes me profoundly grateful I read this study.

It’s just that I don’t think we’re quite there yet, and until we are, supplementation isn’t all that useful.

In summary, I can’t confirm this paper’s suggestion of dire health costs from a “nutritional collapse” in the 1880s. However, the notion of a gradual rise in cancer, heart disease, and other degenerative diseases linked to a modern diet seems correct. Their attribution of this to nutrient deficiency is unsupported and probably only a small part of the problem, and their opinions on supplementation seem to possibly veer into crackpottery. However, they are right to note that we need a better science of supplementation and that once we develop this science, supplements really really ought to work.

EDIT: Andrew G points out in comments that the study compared Victorian life expectancy at age 60 to modern life expectancy *at birth* in their attempts to say life expectancy was staying about the same. This is clearly not kosher. Victorian life expectancy at age 20 (a good point to rule out childhood mortality) was about 62.5, and modern life expectancy at age 20 is…something older than life expectancy at birth which is 77 or so. So their “life expectancy has stayed the same” argument is completely wrong.

(this last one brings up a berkeley.edu address, which does not surprise me one bit)

My parents can relax – the searches are because of the comments on a recent post of mine. In response to a claim that pro-lifers should be in favor of contraception since it decreases abortions, I argued that moral philosophy doesn’t always work that way, but let the main point – that contraception decreases abortion – stand. Some people, especially Joe and Gilbert, challenged my assumption, leading to an unpleasant wade through the swamp of contraception-related data.

The Anti-Contraceptive Position

Let’s start with the nay-sayers. In what he claims is a long essay (ye call that long? I’ll be showin’ ye long!) Bad Catholic evaluates correlation between abortion and contraception rates in different countries. He finds – mostly using data from the pro-choice Guttmacher Institute, a huge clearinghouse of abortion data we will be returning to again and again – that:

Contraception has been shown to decrease abortion rates primarily in [ex-Soviet bloc] countries with already high abortion rates. These represent a minority of countries. Contraception has been shown to increase abortion rates primarily in [non ex-Soviet bloc] countries with already low abortion rates. These represent a majority of countries. Contraception has been shown to slightly reduce abortion rates after its initial increase of abortion rates, but has never been shown to reduce abortion rates back to pre-contraception levels. This is my claim. I have no doubt that there’s a lot more to say, given the incredible amount of studies I haven’t seen. But as far as I can tell, this is a claim far closer to the truth than the oft-repeated, always unexamined “Contraception reduces abortion rates”.

This at first sounds bizarre – how could contraception, a technology that decreases unintended pregnancy – increase abortion, a result of unintended pregnancy? Enter the Peltzman effect, aka risk compensation.

I have a deep love for the Peltzman effect. Part of this is that it’s one of the few terms we social scientists have that sounds as nifty as the one physicists and mathematicians bandy around all the time. Another part is that a girl messaged me on OKCupid once explaining the Peltzman effect to me and asking me on a date (I never claimed my life was normal). But the rest of it is that it’s jsut this really elegant and unexpected finding where across a broad set of domains people respond to hard-won advances that make them safer with “Cool! Now I can behave irresponsibly!” It’s been found with anti-lock brakes (drivers drive closer to the car in front of them), with seat belts (people just drive faster), and with childrens’ safety gear (children just behave more recklessly).

The Peltzman effect doesn’t always hold true; sometimes we expect it and can’t find it. And it rarely makes things worse – it usually is cited as keeping things at the same level they were before, and one well-studied exception, the Munich taxi study, only finds a tiny increase in accidents.

But inside view here – how many people here, if they don’t want kids, would be willing to have totally unprotected sex? And how many people would be willing to have sex using condoms? But condoms have a typical failure rate of 15% – meaning that if a couple has sex for a year using only condoms for protection, there’s a 15% chance the woman will get pregnant. With combined oral contraceptive pill, it’s 8%. So if these contraceptive methods make people about ten times more willing to have sex when they don’t want pregnancy – not at all hard to imagine! – they could raise the unintended pregnancy rate and therefore the abortion rate.

This is the context of Joe’s study showing that in Spain from 1997 – 2007, a large rise in contraceptive usage occurred simultaneous with a large rise in abortion.

A few other arguments seem transparently stupid to me – for example, some people like to point out that US states with high contraception rates also have high abortion rates, but that’s mostly a feature of those states being very liberal and so allowing abortion clinics to operate there. So lets move on to…

The Pro-Contraceptive Position

Just in case you thought you were going to escape without any graphs:

Here’s teenage birth rates over the last 75 years. Like nearly all social problems, they have been steadily and somewhat mysteriously declining, but we notice an especially sharp decline around 1960, the year the Pill was introduced. Abortion wasn’t legalized until the 70s and was pretty uncommon before then, so we can leave it out of this analysis and say that it sure looks like the invention of a new form of contraception decreased pregnancies.

Also near an all-time low are abortion rates (I assume they mean “all-time low since abortion was legalized”?). This seems to be due to both fewer unintended pregnancies and less willingness to end unintended pregnancies with abortion. Santelli et al 2002 find the decline to correspond nicely to increasing use of contraceptives. Some pro-lifers claim (data unavailable), that after 2002 abortions continued to drop even though contraception use stayed steady. Unfortunately, all I can find is the CDC saying contraceptive use continued to rise – in the absence of their contrary data, I’m giving this point to the “increased contraception helped reduce abortion” people.

(one way that we could reconcile these two results, if we were feeling very generous, is to say that overall contraception use has remained the same, but users have switched to more effective modern forms of contraception like the implant or IUD, with a failure rate less than 1/100th that of condoms)

But this is actually consistent with Bad Catholic’s claim above – contraception increases the abortion rate when first introduced, then eventually stabilizes and decreases it, but never back to the level before contraception. So let’s evaluate that one. Bad Catholic writes:

An honest look at the data shows that in virtually every country that increased the use of contraception, there was a simultaneous increase in that country’s abortion rate. In England (Rise in contraceptive use: simultaneous rise in abortions), France (Rise in contraceptive use: simultaneous rise in abortions), Australia, (Rise in contraceptive use: simultaneous rise in abortions), Portugal (Whose abortion rate only began to rise after 1999, after oral contraceptive methods were made widely available), Canada (Whose abortion rate only began to rise after the legalization of oral contraceptives in 1969), and, as the Guttmacher Institute shows, Singapore, Cuba, Denmark, the Netherlands, and South Korea, to name a few.

Let’s investigate the countries in order. The claim seems to be only that abortions and contraceptive use rose “simultaneously”, but following his links this turns out to mean “throughout the 20th century”. There is no attempt to prove that the particular shape of the contraception curve matched that of the abortion curve or anything like that, just that there was more contraception in 2000 than in 1950, and, whaddya know, more abortions as well. The same methodology could very easily correlate abortion with global temperature. His statistics on England, France, and Australia all seem to be of this type.

He makes a stronger claim about Canada: that “the abortion rate only began to rise after the legalization of oral contraceptives in 1969″. You know what else was legalized in Canada in 1969? Abortion. I’m going with “probably not a good test case”.

As for Portugal, the claim that oral contraceptive methods were legalized in 1999 seems wrong; his own link says they have been available since 1985 and that only the emergency contraceptive pill was made available in 1999. Further, his claim that “abortion rates only began to rise after 1999″ also seems wrong – his link shows what looks like a pretty linear rise in abortion rates from 1996 to 2006; I don’t think anyone eyeballing those numbers would be tempted to consider 1999 anything remotely like an inflection point. My own guess for an inflection point would be 2007, and sure enough when I Google it that was the year they fully legalized abortion.

The Guttmacher Institute doesn’t link to its sources as diligently as Bad Catholic, so I’m just going to accept their claim that six countries – Singapore, Cuba, Denmark, Netherlands, US, and South Korea – saw simultaneous increases in contraception and abortion – after all, it goes against the direction of their bias so they have no incentive to lie. They give their results the following explanation:

The reason for the confusion stems from the observation that, within particular populations, contraceptive prevalence and the incidence of induced abortion can and, indeed, often do rise in parallel, contrary to what one would expect. The explanation for these counterintuitive trends is clear.2 In societies that have not yet entered the fertility transition, both actual fertility and desired family sizes are high (or, to put it another way, childbearing is not yet considered to be “within the calculus of conscious choice”3). In such societies, couples are at little (or no) risk of unwanted pregnancies. The advent of modern contraception is associated with a destabilization of high (or “fatalistic”) fertility preferences. Thus, as contraceptive prevalence rises and fertility starts to fall, an increasing proportion of couples want no more children (or want an appreciable delay before the next child), and exposure to the risk of unintended pregnancy also increases as a result. In the early and middle phases of fertility transition, adoption and sustained use of effective methods of contraception by couples who wish to postpone or limit childbearing is still far from universal. Hence, the growing need for contraception may outstrip use itself;4 thus, the incidence of unintended and unwanted pregnancies rises, fueling increases in unwanted live births and induced abortion. In this scenario, contraceptive use and induced abortion may rise simultaneously.

As fertility decreases toward replacement level (two births per woman), or even lower, the length of potential exposure to unwanted pregnancies increases further. For instance, in a society in which the average woman is sexually active from ages 20 to 45 and wants two children, approximately 20 of those 25 years will be spent trying to avoid pregnancy. Once use of highly effective contraceptive methods rises to 80%, the potential demand for abortion, and its incidence, will fall. Demand for abortion falls to zero only in the “perfect contraceptive” population, in which women are protected by absolutely effective contraceptive use at all times, except for the relatively short periods when they want to conceive, are pregnant or are protected by lactational amenorrhea.5 Because such a state of perfect protection is never actually achieved, a residual demand for abortion always exists, although its magnitude varies considerably among low-fertility societies, according to levels of contraceptive use and choice of methods.

This seems incredibly reasonable, and we will come back to it later. Let’s abandon all of these time series type studies and see if we can find a halfway-decent controlled experiment.

Well, uh…we can find a controlled experiment. These people in St. Louis gave people free contraceptives and later found that they had a teenage pregnancy rate much lower than the rest of the population. Gilbert gives this study exactly the correct criticism – participants from a very specific population (poor people in St. Louis interested in signing up for a contraceptive study) are being compared to the general population (everyone in the United States). This is inexcusable, especially considering that Real Science has an extremely standard way of avoiding this problem (sign people up for your study, only give the intervention to a randomly selected half, and the other half is an instant control group). Other fatal issues – the study used IUDs, the most effective form of contraception, but most of the worry that contraception might increase abortion comes from less effective means like condoms and the Pill. Finally, if you’re really interested in the way that widespread availability of contraceptives makes a culture more libertine, just giving them to a couple of people within that culture isn’t going to capture that effect. I am maybe a little bit hugely disappointed that most of the media and bloggers reporting on this didn’t mention these sorts of issues.

But it does show one interesting thing, which is that when people get free contraception, they start using more effective contraception methods. Would this also cause risk compensation? I don’t know, but I feel like there has to be some amount of sex beyond which it’s just no longer fun, and some contraceptive methods are so effective that it would be really hard to have so much sex that they’re worse than nothing.

Let’s close this section with a few minor points.

Contraceptive advocates point to the Netherlands, with one of the lowest abortion rates in the world. Given the stereotypes of the Dutch, they probably didn’t get that way through careful abstinence, and indeed their government is unusually generous in providing free contraceptives.

It turns out people can just survey women having abortions and ask them if they used contraceptives or not! 54% of abortion patients were using contraception at the time, which pro-life websites get very excited about: “IT’S MORE THAN HALF!” But putting these numbers in context may diminish their enthusiasm: the four-fifths of American women who use contraception account for 54% of abortions; the fifth of women who don’t use it account for the other 46%. The Guttmacher Institute gets more or less the same numbers, but frames them in a very convincingly pro-contraceptive way:

The two-thirds of U.S. women at risk of unintended pregnancy who use contraception consistently and correctly throughout the course of any given year account for only 5% of all unintended pregnancies. The 19% of women at risk who use contraception but do so inconsistently account for 44% of all unintended pregnancies, while the 16% of women at risk who do not use contraception at all for a month or more during the year account for 52% of all unintended pregnancies.

So it seems clear that the more (and better) you use contraception, the less likely you are to have an abortion.

Summary

I think we can use these results to build a consistent picture.

Contraceptive and abortion rates often rise simultaneously. This rise is not necessarily causal, and is more likely to be due to both being parts of the same philosophy – people want to have lots of sex but not have kids. As this philosophy becomes more widespread, as it has nearly everywhere in the 20th century with the Sexual Revolution and Demographic Transition, both contraception and abortion will rise. As it gains ground, both contraception and abortion will become more legal and available, making them rise even further. It is unclear to what degree the availability of contraception itself causes the rise of this philosophy. I’m intrigued by this claim that penicillin rather than the pill started the Sexual Revolution, but if someone wants to claim that it was all due to contraceptives, I don’t have enough expertise in the area to prove her wrong.

On the other hand, once a society has undergone this transition and settled on “lots of sex, few kids” as being its dominant values, then the local application of more contraception seems to decrease abortion rates. We know this because of the surveys of abortion patients saying they are disproportionately likely not to be contraceptive users. We know this because of the decline in teenage pregnancies with the advent of the Pill. And we also notice the game-changing nature of new, more effective contraceptives with near-zero failure rates replacing older, more fallible ones, and the not-provably-causal but certainly suggestive secular decline in abortion rates that corresponds with that replacement.

Overall my guess would be that a society that legalizes contraceptives would see an increase in abortion rates (which might or might not be causal depending on that society’s situation), but that in a society like our own, where contraceptives are already legal and the demographic transition is pretty much complete, increasing access to contraceptives is probably going to decrease abortion. And increasing access to extremely effective contraceptives like the implant or RISUG, especially when they replace less effective contraceptives like the condom, are very very probably going to decrease abortion.