“Slim by Chocolate!” the headlines blared. A team of German researchers had found that people on a low-carb diet lost weight 10 percent faster if they ate a chocolate bar every day. It made the front page of Bild, Europe’s largest daily newspaper, just beneath their update about the Germanwings crash. From there, it ricocheted around the internet and beyond, making news in more than 20 countries and half a dozen languages. It was discussed on television news shows. It appeared in glossy print, most recently in the June issue of Shape magazine (“Why You Must Eat Chocolate Daily,” page 128). Not only does chocolate accelerate weight loss, the study found, but it leads to healthier cholesterol levels and overall increased well-being. The Bild story quotes the study’s lead author, Johannes Bohannon, Ph.D., research director of the Institute of Diet and Health: “The best part is you can buy chocolate everywhere.”

I am Johannes Bohannon, Ph.D. Well, actually my name is John, and I’m a journalist. I do have a Ph.D., but it’s in the molecular biology of bacteria, not humans. The Institute of Diet and Health? That’s nothing more than a website.

Other than those fibs, the study was 100 percent authentic. My colleagues and I recruited actual human subjects in Germany. We ran an actual clinical trial, with subjects randomly assigned to different diet regimes. And the statistically significant benefits of chocolate that we reported are based on the actual data. It was, in fact, a fairly typical study for the field of diet research. Which is to say: It was terrible science. The results are meaningless, and the health claims that the media blasted out to millions of people around the world are utterly unfounded.

Bohannon goes on to explain that as part of a documentary about “the junk-science diet industry”, he and some collaborators designed a fake study to see if they could convince journalists. They chose to make it about chocolate:

Gunter Frank, a general practitioner in on the prank, ran the clinical trial. Onneken had pulled him in after reading a popular book Frank wrote railing against dietary pseudoscience. Testing bitter chocolate as a dietary supplement was his idea. When I asked him why, Frank said it was a favorite of the “whole food” fanatics. “Bitter chocolate tastes bad, therefore it must be good for you,” he said. “It’s like a religion.”

They recruited 16 (!) participants and divided them into three groups. One group ate their normal diet. Another ate a low-carb diet. And a third ate a low-carb diet plus some chocolate. Both the low-carb group and the low-carb + chocolate group lost weight compared to the control group, but the low-carb + chocolate group lost weight “ten percent faster”, and the difference was “statistically significant”. They also had “better cholesterol readings” and “higher scores on the well-being survey”.

Bohannon admits exactly how he managed this seemingly impressive result – he measured eighteen different parameters (weight, cholesterol, sodium, protein, etc) which virtually guarantees that one will be statistically significant. That one turned out to be weight loss. If it had been sodium, he would have published the study as “Chocolate Lowers Sodium Levels”.

Then he pitched it to various fake for-profit journals until one of them bit. Then he put out a PR release to various media outlets, and they ate it up. They ended up in a bunch of English and German language media including Bild, the Daily Star, Times of India, Cosmopolitan, Irish Examiner, and the Huffington Post.

The people I’ve seen discussing this seem to have drawn five conclusions, four of which are wrong:

Conclusion 1: Haha, I can’t believe people were so gullible that they actually thought chocolate caused weight loss!

Bohannon himself endorses this one, saying bitter chocolate was a favorite of “whole food fanatics” because “Bitter chocolate tastes bad, therefore it must be good for you” and “it’s like a religion.

But actually, there’s lots of previous research supporting health benefits from bitter chocolate, none of which Bohannon seems to be aware of.

A meta-analysis of 42 randomized controlled trials totaling 1297 participants in the American Journal of Clinical Nutrition found that chocolate improved blood pressure, flow-mediated dilatation (a measure of vascular health), and insulin resistance (related to weight gain).

A different meta-analysis of 24 randomized controlled trials totalling 1106 people in the Journal of Nutrition also found that chocolate improved blood pressure, flow-mediated dilatation, and insulin resistance.

A Cochrane Review of 20 randomized controlled trials of 856 people found that chocolate improved blood pressure (it didn’t test for flow-mediated dilatation or insulin resistance)

A study on mice found that mice fed more chocolate flavanols were less likely to gain weight.

An epidemiological study of 1018 people in the United States found an association between frequent chocolate consumption and lower BMI, p < 0.01. A second epidemiological study of 1458 people in Europe found the same thing, again p < 0.01. A cohort study of 470 elderly men found chocolate intake was inversely associated with blood pressure and cardiovascular mortality, p < 0.001, not confounded by the usual suspects. I wouldn't find any of these studies alone very convincing. But together, they compensate for each other's flaws and build a pretty robust structure. So the next flawed conclusion is: Conclusion 2: This proves that nutrition isn’t a real science and we should all just be in a state of radical skepticism about these things

What we would like to do is a perfect study where we get thousands of people, randomize them to eat-lots-of-chocolate or eat-little-chocolate at birth, then follow their weights over their entire lives. That way we could have a large sample size, perfect randomization, life-long followup, and clear applicability to other people. But for practical and ethical reasons, we can’t do that. So we do a bunch of smaller studies that each capture a few of the features of the perfect study.

First we do animal studies, which can have large sample sizes, perfect randomization, and life-long followup, but it’s not clear whether it applies to humans.

Then we do short randomized controlled trials, which can have large sample sizes, perfect randomization, and human applicability, but which only last a couple of months.

Then we do epidemiological studies, which can have large sample sizes, human applicability, and last for many decades, but which aren’t randomized very well and might be subject to confounders.

This is what happened in the chocolate studies above. Mice fed a strict diet plus chocolate for a long time gain less weight than mice fed the strict diet alone. This is suggestive, but we don’t know if it applies to humans. So we find that in randomized controlled trials, chocolate helps with some proxies for weight gain like insulin resistance. This is even more suggestive, but we don’t know if it lasts. So we find that in epidemiological studies, lifetime chocolate consumption is associated with lifetime good health outcomes. This on its own is suggestive but potentially confounded, but when we combine them with all of the others, they become more convincing.

(am I cheating by combining blood pressure and BMI data? Sort of, but the two measures are correlated)

When all of these paint the same picture, then we start thinking that maybe it’s because our hypothesis is true. Yes, maybe the mouse studies could be related to a feature of mice that doesn’t generalize to humans, and the randomized controlled trial results wouldn’t hold up after a couple of years, and the epidemiological studies are confounded. But that would be extraordinarily bad luck. More likely they’re all getting the same result because they’re all tapping into the same underlying reality.

This is the way science usually works, it’s the way nutrition science usually works, and it’s the way the science of whether chocolate causes weight gain usually works. These are not horrible corrupt disciplines made up entirely of shrieking weight-loss-pill peddlers trying to hawk their wares. They only turn into that when the media takes a single terrible study totally out of context and misrepresents the field.

Conclusion 3: Studies Always Need To Have High Sample Sizes

Here’s another good chocolate-related study: Short-term administration of dark chocolate is followed by a significant increase in insulin sensitivity and a decrease in blood pressure in healthy persons.

Bohannon says:

Our study was doomed by the tiny number of subjects, which amplifies the effects of uncontrolled factors…Which is why you need to use a large number of people, and balance age and gender across treatment group

But I say “Short-term administration…” is a good study despite having an n = 15, one less than the Bohannon study. Why? Well, their procedure was pretty involved, and you wouldn’t be able to get a thousand people to go through the whole rigamarole. On the other hand, their insulin resistance measure thing was nearly twice as high in the dark chocolate group as the white chocolate group, and p < 0.001. (Another low sample size study that was nevertheless very good: psychiatrists knew that consuming dietary tyramine when taking a MAOI antidepressant can cause a life-threatening hypertensive crisis, but they didn't know how much tyramine it took. In order to find out, they took a dozen people, put them on MAOIs, and then gradually fed them more and more tyramine with doctors standing by to treat the crisis as soon as it started. They found about how much tyramine it took and declared the experiment a success. If the tyramine levels were about the same in all twelve patients, then adding a thousand more patients wouldn’t help much, and it would definitely increase the risk.)

Sample size is important when you’re trying to detect a small effect in the middle of a large amount of natural variation. When you’re looking for a large effect in the middle of no natural variation, sample size doesn’t matter as much. For example, if there was a medicine that would help amputees grow their hands back, I would accept success with a single patient (if it worked) as proof of effectiveness (I suppose I couldn’t be sure it would always work until more patients had been tried, but a single patient would certainly pique my interest). You’re not going after sample size so much as after p-value.

Conclusion 4: P-Values Are Stupid And We Need To Get Rid Of Them

Bohannon says that:

If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result…the letter p seems to have totemic power, but it’s just a way to gauge the signal-to-noise ratio in the data…scientists are getting wise to these problems. Some journals are trying to phase out p value significance testing altogether to nudge scientists into better habits.

Okay, take the “Short-term administration” study above. I would like to be able to say that since it has p < 0.001, we know it's significant. But suppose we're not allowed to do p-values. All I do is tell you "Yeah, there was a study with fifteen people that found chocolate helped with insulin resistance" and you laugh in my face. Effect size is supposed to help with that. But suppose I tell you "There was a study with fifteen people that found chocolate helped with insulin resistance. The effect size was 0.6." I don't have any intuition at all for whether or not that's consistent with random noise. Do you? Okay, then they say we’re supposed to report confidence intervals. The effect size was 0.6, with 95% confidence interval of [0.2, 1.0]. Okay. So I check the lower bound of the confidence interval, I see it’s different from zero. But now I’m not transcending the p-value. I’m just using the p-value by doing a sort of kludgy calculation of it myself – “95% confidence interval does not include zero” is the same as “p value is less than 0.05″.

(Imagine that, although I know the 95% confidence interval doesn’t include zero, I start wondering if the 99% confidence interval does. If only there were some statistic that would give me this information!)

But wouldn’t getting rid of p-values prevent “p-hacking”? Maybe, but it would just give way to “d-hacking”. You don’t think you could test for twenty different metabolic parameters and only report the one with the highest effect size? The only difference would be that p-hacking is completely transparent – if you do twenty tests and report a p of 0.05, I know you’re an idiot – but d-hacking would be inscrutable. If you do twenty tests and report that one of them got a d = 0.6, is that impressive? No better than chance? I have no idea. I bet there’s some calculation I could do to find out, but I also bet that it would be a lot harder than just multiplying the value by the number of tests and seeing what happens. [EDIT: On reflection not sure this is true; the possibility of p-hacking is inherent to p-values, but the possibility of d-hacking isn’t inherent to effect size. I don’t actually know how much this would matter in the real world.]

But wouldn’t switching from p-values to effect sizes prevent people from making a big deal about tiny effects that are nevertheless statistically significant? Yes, but sometimes we want to make a big deal about tiny effects that are nevertheless statistically significant! Suppose that Coca-Cola is testing a new product additive, and finds in large epidemiological studies that it causes one extra death per hundred thousand people per year. That’s an effect size of approximately zero, but it might still be statistically significant. And since about a billion people worldwide drink Coke each year, that’s a ten thousand deaths. If Coke said “Nope, effect size too small, not worth thinking about”, they would kill almost two milli-Hitlers worth of people.

Yeah, sure, you can never use p-values again, and run into all of these other problems. Or you can do a Bonferroni correction, which is a very simple adjustment to p-values which corrects for p-hacking. Or instead of taking one study at face value LIKE AN IDIOT you can wait to see if other studies replicate the findings. Remember, the whole point of p-hacking is choosing at random form a bunch of different outcomes, so if two trials both try to p-hack, they’ll end up with different outcomes and the game will be up. Seriously, STOP TRYING TO BASE CONCLUSIONS ON ONE STUDY.

Conclusion 5: Trust Science Journalism Less

This is the one that’s correct.

But it’s not totally correct. Bohannon boasts of getting his findings in a couple of daily newspapers and the Huffington Post. That’s not exactly the cream of the crop. The Economist usually has excellent science journalism. Magazines like Scientific American and Discover can be okay, although even they get hyped. Reddit’s r/science is good, assuming you make sure to always check the comments. And there are individual blogs like Mind the Brain run by researchers in the field that can usually be trusted near-absolutely. Cochrane Collaboration will always have among the best analyses on everything.

If you really want to know what’s going on and can’t be bothered to ferret out all of the brilliant specialists, my highest recommendation goes to Wikipedia. It isn’t perfect, but compared to anything you’d find on a major news site, it’s like night and day. Wikipedia’s Health Effects Of Chocolate page is pretty impressive and backs everything it says up with good meta-analyses and studies in the best journals. Its sentence on the cardiovasuclar effects links to this letter, which is very good.

Do you know why you can trust Wikipedia better than news sites? Because Wikipedia doesn’t obsess over the single most recent study. Are you starting to notice a theme?

For me, the takeaway from this affair is that there is no one-size-fits-all solution to make statistics impossible to hack. Getting rid of p-values is appropriate sometimes, but not other times. Demanding large sample sizes is appropriate sometimes, but not other times. Not trusting silly conclusions like “chocolate causes weight loss” works sometimes but not other times. At the end of the day, you have to actually know what you’re doing. Also, try to read more than one study.

I.

I’ve been playing around with data from Internet databases that aggregate patient reviews of medications.

Are these any good? I looked at four of the largest such databases – Drugs.com, WebMD, AskAPatient, and DrugLib – as well as psychiatry-specific site CrazyMeds – and took their data on twenty-three major antidepressants. Then I correlated them with one another to see if the five sites mostly agreed.

Correlations between Drugs.com, AskAPatient, and WebMD were generally large and positive (around 0.7). Correlations between CrazyMeds and DrugLib were generally small or negative. In retrospect this makes sense, because these two sites didn’t allow separation of ratings by condition, so for example Seroquel-for-depression was being mixed with Seroquel-for-schizophrenia.

So I threw out the two offending sites and kept Drugs.com, AskAPatient, and WebMD. I normalized all the data, then took the weighted average of all three sites. From this huge sample (the least-reviewed drug had 35 ratings, the most-reviewed drug 4,797) I obtained a unified opinion of patients’ favorite and least favorite antidepressants.

This doesn’t surprise me at all. Everyone secretly knows Nardil and Parnate (the two commonly-used drugs in the MAOI class) are excellent antidepressants¹. Oh, nobody will prescribe them, because of the dynamic discussed here, but in their hearts they know it’s true.

Likewise, I feel pretty good to see that Serzone, which I recently defended, is number five. I’ve had terrible luck with Viibryd, and it just seems to make people taking it more annoying, which is not a listed side effect but which I swear has happened.

The table also matches the evidence from chemistry – drugs with similar molecular structure get similar ratings, as do drugs with similar function. This is, I think, a good list.

Which is too bad, because it makes the next part that much more terrifying.

II.

There is a sixth major Internet database of drug ratings. It is called RateRx, and it differs from the other five in an important way: it solicits ratings from doctors, not patients. It’s a great idea – if you trust your doctor to tell you which drug is best, why not take advantage of wisdom-of-crowds and trust all the doctors?

The RateRX logo. Spoiler: this is going to seem really ironic in about thirty seconds.

RateRx has a modest but respectable sample size – the drugs on my list got between 32 and 70 doctor reviews. There’s only one problem.

You remember patient reviews on the big three sites correlated about +0.7 with each other, right? So patients pretty much agree on which drugs are good and which are bad?

Doctor reviews on RateRx correlated at -0.21 with patient reviews. The negative relationship is nonsignificant, but that just means that at best, doctor reviews are totally uncorrelated with patient consensus.

This has an obvious but very disturbing corollary. I couldn’t get good numbers on how times each of the antidepressants on my list were prescribed, because the information I’ve seen only gives prescription numbers for a few top-selling drugs, plus we’ve got the same problem of not being able to distinguish depression prescriptions from anxiety prescriptions from psychosis prescriptions. But total number of online reviews makes a pretty good proxy. After all, the more patients are using a drug, the more are likely to review it.

Quick sanity check: the most reviewed drug on my list was Cymbalta. Cymbalta was also the best selling antidepressant of 2014. Although my list doesn’t exactly track the best-sellers, that seems to be a function of how long a drug has been out – a best-seller that came out last year might have only 1/10th the number of reviews as a best-seller that came out ten years ago. So number of reviews seems to be a decent correlate for amount a drug is used.

In that case, amount a drug is used correlates highly (+0.67, p = 0.005) with doctors’ opinion of the drug, which makes perfect sense since doctors are the ones prescribing it. But amount the drug gets used correlates negatively with patient rating of the drug (-0.34, p = ns), which of course is to be expected given the negative correlation between doctor opinion and patient opinion.

So the more patients like a drug, the less likely it is to be prescribed².

III.

There’s one more act in this horror show.

Anyone familiar with these medications reading the table above has probably already noticed this one, but I figured I might as well make it official.

I correlated the average rating of each drug with the year it came on the market. The correlation was -0.71 (p < .001). That is, the newer a drug was, the less patients liked it³.

This pattern absolutely jumps out of the data. First- and second- place winners Nardil and Parnate came out in 1960 and 1961, respectively; I can’t find the exact year third-place winner Anafranil came out, but the first reference to its trade name I can find in the literature is from 1967, so I used that. In contrast, last-place winner Viibryd came out in 2011, second-to-last place winner Abilify got its depression indication in 2007, and third-to-last place winner Brintellix is as recent as 2013.

This result is robust to various different methods of analysis, including declaring MAOIs to be an unfair advantage for Team Old and removing all of them, changing which minor tricylics I do and don’t include in the data, and altering whether Deprenyl, a drug that technically came out in 1970 but received a gritty reboot under the name Emsam in 2006, is counted as older or newer.

So if you want to know what medication will make you happiest, at least according to this analysis your best bet isn’t to ask your doctor, check what’s most popular, or even check any individual online rating database. It’s to look at the approval date on the label and choose the one that came out first.

IV.

What the hell is going on with these data?

I would like to dismiss this as confounded, but I have to admit that any reasonable person would expect the confounders to go the opposite way.

That is: older, less popular drugs are usually brought out only when newer, more popular drugs have failed. MAOIs, the clear winner of this analysis, are very clearly reserved in the guidelines for “treatment-resistant depression”, ie depression you’ve already thrown everything you’ve got at. But these are precisely the depressions that are hardest to treat.

Imagine you are testing the fighting ability of three people via ten boxing matches. You ask Alice to fight a Chihuahua, Bob to fight a Doberman, and Carol to fight Cthulhu. You would expect this test to be biased in favor of Alice and against Carol. But MAOIs and all these other older rarer drugs are practically never brought out except against Cthulhu. Yet they still have the best win-loss record.

Here are the only things I can think of that might be confounding these results.

Perhaps because these drugs are so rare and unpopular, psychiatrists only use them when they have really really good reason. That is, the most popular drug of the year they pretty much cluster-bomb everybody with. But every so often, they see some patient who seems absolutely 100% perfect for clomipramine, a patient who practically screams “clomipramine!” at them, and then they give this patient clomipramine, and she does really well on it.

(but psychiatrists aren’t actually that good at personalizing antidepressant treatments. The only thing even sort of like that is that MAOIs are extra-good for a subtype called atypical depression. But that’s like a third of the depressed population, which doesn’t leave much room for this super-precise-targeting hypothesis.)

Or perhaps once drugs have been on the market longer, patients figure out what they like. Brintellix is so new that the Brintellix patients are the ones whose doctors said “Hey, let’s try you on Brintellix” and they said “Whatever”. MAOIs have been on the market so long that presumably MAOI patients are ones who tried a dozen antidepressants before and stayed on MAOIs because they were the only ones that worked.

(but Prozac has been on the market 25 years now. This should only apply to a couple of very new drugs, not the whole list.)

Or perhaps the older drugs have so many side effects that no one would stay on them unless they’re absolutely perfect, whereas people are happy to stay on the newer drugs even if they’re not doing much because whatever, it’s not like they’re causing any trouble.

(but Seroquel and Abilify, two very new drugs, have awful side effects, yet are down at the bottom along with all the other new drugs)

Or perhaps patients on very rare weird drugs get a special placebo effect, because they feel that their psychiatrist cares enough about them to personalize treatment. Perhaps they identify with the drug – “I am special, I’m one of the only people in the world who’s on nefazodone!” and they become attached to it and want to preach its greatness to the world.

(but drugs that are rare because they are especially new don’t get that benefit. I would expect people to also get excited about being given the latest, flashiest thing. But only drugs that are rare because they are old get the benefit, not drugs that are rare because they are new.)

Or perhaps psychiatrists tend to prescribe the drugs they “imprinted on” in medical school and residency, so older psychiatrists prescribe older drugs and the newest psychiatrists prescribe the newest drugs. But older psychiatrists are probably much more experienced and better at what they do, which could affect patients in other ways – the placebo effect of being with a doctor who radiates competence, or maybe the more experienced psychiatrists are really good at psychotherapy, and that makes the patient better, and they attribute it to the drug.

(but read on…)

V.

Or perhaps we should take this data at face value and assume our antidepressants have been getting worse and worse over the past fifty years.

This is not entirely as outlandish as it sounds. The history of the past fifty years has been a history of moving from drugs with more side effects to drugs with fewer side effects, with what I consider somewhat less than due diligence in making sure the drugs were quite as effective in the applicable population. This is a very complicated and controversial statement which I will be happy to defend in the comments if someone asks.

The big problem is: drugs go off-patent after twenty years. Drug companies want to push new, on-patent medications, and most research is funded by drug companies. So lots and lots of research is aimed at proving that newer medications invented in the past twenty years (which make drug companies money) are better than older medications (which don’t).

I’ll give one example. There is only a single study in the entire literature directly comparing the MAOIs – the very old antidepressants that did best on the patient ratings – to SSRIs, the antidepressants of the modern day⁴. This study found that phenelzine, a typical MAOI, was no better than Prozac, a typical SSRI. Since Prozac had fewer side effects, that made the choice in favor of Prozac easy.

Did you know you can look up the authors of scientific studies on LinkedIn and sometimes get very relevant information? For example, the lead author of this study has a resume that clearly lists him as working for Eli Lilly at the time the study was conducted (spoiler: Eli Lilly is the company that makes Prozac). The second author’s LinkedIn profile shows he is also an operations manager for Eli Lilly. Googling the fifth author’s name links to a news article about Eli Lilly making a $750,000 donation to his clinic. Also there’s a little blurb at the bottom of the paper saying “Supported by a research grant by Eli Lilly and company”, then thanking several Eli Lilly executives by name for their assistance.

This is the sort of study which I kind of wish had gotten replicated before we decided to throw away an entire generation of antidepressants based on the result.

But who will come to phenelzine’s defense? Not Parke-Davis , the company that made it: their patent expired sometime in the seventies, and then they were bought out by Pfizer⁵. And not Pfizer – without a patent they can’t make any money off Nardil, and besides, Nardil is competing with their own on-patent SSRI drug Zoloft, so Pfizer has as much incentive as everyone else to push the “SSRIs are best, better than all the rest” line.

Every twenty years, pharmaceutical companies have an incentive to suddenly declare that all their old antidepressants were awful and you should never use them, but whatever new antidepressant they managed to dredge up is super awesome and you should use it all the time. This sort of does seem like the sort of situation that might lead to older medications being better than newer ones. A couple of people have been pushing this line for years – I was introduced to it by Dr. Ken Gillman from Psychotropical Research, whose recommendation of MAOIs and Anafranil as most effective match the patient data very well, and whose essay Why Most New Antidepressants Are Ineffective is worth a read.

I’m not sure I go as far as he does – even if new antidepressants aren’t worse outright, they might still trade less efficacy for better safety. Even if they handled the tradeoff well, it would look like a net loss on patient rating data. After all, assume Drug A is 10% more effective than Drug B, but also kills 1% of its users per year, while Drug B kills nobody. Here there’s a good case that Drug B is much better and a true advance. But Drug A’s ratings would look better, since dead men tell no tales and don’t get to put their objections into online drug rating sites. Even if victims’ families did give the drug the lowest possible rating, 1% of people giving a very low rating might still not counteract 99% of people giving it a higher rating.

And once again, I’m not sure the tradeoff is handled very well at all.⁶.

VI.

In order to distinguish between all these hypotheses, I decided to get a lot more data.

I grabbed all the popular antipsychotics, antihypertensives, antidiabetics, and anticonvulsants from the three databases, for a total of 55,498 ratings of 74 different drugs. I ran the same analysis on the whole set.

The three databases still correlate with each other at respectable levels of +0.46, +0.54, and +0.53. All of these correlations are highly significant, p < 0.01. The negative correlation between patient rating and doctor rating remains and is now a highly significant -0.344, p < 0.01. This is robust even if antidepressants are removed from the analysis, and is notable in both psychiatric and nonpsychiatric drugs.

The correlation between patient rating and year of release is a no-longer-significant -0.191. This is heterogenous; antidepressants and antipsychotics show a strong bias in favor of older medications, and antidiabetics, antihypertensives, and anticonvulsants show a slight nonsignificant bias in favor of newer medications. So it would seem like the older-is-better effect is purely psychiatric.

I conclude that for some reason, there really is a highly significant effect across all classes of drugs that makes doctors love the drugs patients hate, and vice versa.

I also conclude that older psychiatric drugs seem to be liked much better by patients, and that this is not some kind of simple artifact or bias, since if such an artifact or bias existed we would expect it to repeat in other kinds of drugs, which it doesn’t.

VII.

Please feel free to check my results. Here is a spreadsheet (.xls) containing all of the data I used for this analysis. Drugs are marked by class: 1 is antidepressants, 2 is antidiabetics, 3 is antipsychotics, 4 is antihypertensives, and 5 is anticonvulsants. You should be able to navigate the rest of it pretty easily.

One analysis that needs doing is to separate out drug effectiveness versus side effects. The numbers I used were combined satisfaction ratings, but a few databases – most notably WebMD – give you both separately. Looking more closely at those numbers might help confirm or disconfirm some of the theories above.

If anyone with the necessary credentials is interested in doing the hard work to publish this as a scientific paper, drop me an email and we can talk.

Footnotes

1. Technically, MAOI superiority has only been proven for atypical depression, the type of depression where you can still have changing moods but you are unhappy on net. But I’d speculate that right now most patients diagnosed with depression have atypical depression, far more than the studies would indicate, simply because we’re diagnosing less and less severe cases these days, and less severe cases seem more atypical.

6. Consider: Seminars In General Psychiatry estimates that MAOIs kill one person per 100,000 patient years. A third of all depressions are atypical. MAOIs are 25 percentage points more likely to treat atypical depression than other antidepressants. So for every 100,000 patients you give a MAOI instead of a normal antidepressant, you kill one and cure 8,250 who wouldn’t otherwise be cured. The QALY database says that a year of moderate depression is worth about 0.6 QALYs. So for every 100,000 patients you give MAOIs, you’re losing about 30 QALYs and gaining about 3,300.

I.

Which is worse – ruining ten million people’s sex lives for one year, or making one hundred people’s livers explode?

I admit I sometimes use this blog to speculate about silly moral dilemmas for no reason, but that’s not what’s happening here. This is a real question that I deal with on a daily basis.

SSRIs, the class which includes most currently used antidepressants, are very safe in the traditional sense of “unlikely to kill you”. Suicidal people take massive overdoses of SSRIs all the time, and usually end up with little more than a stomachache for their troubles. On the other hand, there’s increasing awareness of very common side effects which, while not disabling, can be pretty unpleasant. About 50% of users report decreased sexual abilities, sometimes to the point of total loss of libido or anorgasmia. And something like 25% of users experience “emotional blunting” and the loss of ability to feel feelings normally.

Nefazodone (brand name Serzone®, which would also be a good brand name for a BDSM nightclub) is an equally good (and maybe better) antidepressant that does not have these side effects. On the other hand, every year, one in every 300,000 people using nefazodone will go into “fulminant hepatic failure”, which means their liver suddenly and spectacularly stops working and they need a liver transplant or else they die.

There are a lot of drug rating sites, but the biggest is Drugs.com. 467 Drugs.com users have given Celexa, a very typical SSRI, an average rating of 7.8/10. 14 users have given nefazodone an average rating of 9.1/10.

CrazyMeds might not be as dignified as Drugs.com, but they have a big and well-educated user base and they’re psych-specific. Their numbers are 3.3/5 (n = 253) for Celexa and 4.1/5 (n = 47) for nefazodone.

So both sites’ users seem to agree that nefazodone is notably better than Celexa, in terms of a combined measure of effectiveness and side effects.

But nefazodone is practically never used. It’s actually illegal in most countries. In the United States, parent company Bristol-Myers Squibb (which differs from normal Bristol-Myers in that it was born without innate magical ability) withdrew it from the market, and the only way you can find it nowadays is to get it is from an Israeli company that grabbed the molecule after it went off-patent. In several years working in psychiatry, I have never seen a patient on nefazodone, although I’m sure they exist somewhere. I would estimate its prescription numbers are about 1% of Celexa’s, if that.

The problem is the hepatic side effects. Nobody wants to have their liver explode.

But. There are something like thirty million people in the US on antidepressants. If we put them all on nefazodone, that’s about a hundred cooked livers per year. If we put them all on SSRIs, at least ten million of them will get sexual side effects, plus some emotional blunting.

My life vastly improved when I learned there was a searchable database of QALYs for different conditions. It doesn’t have SSRI-induced sexual dysfunction, but it does have sexual dysfunction due to prostate cancer treatment, and I assume that sexual dysfunction is about equally bad regardless of what causes it. Their sexual dysfunction has some QALY weights averaging about 0.85. Hm.

Assume everyone with fulminant liver failure dies. That’s not true; some get liver transplants, maybe some even get a miracle and recover. But assume everyone dies – and further, they die at age 30, cutting their lives short by fifty years.

In that case, putting all depressed people on nefazodone for a year costs 5,000 QALYs, but putting all depressed people on SSRIs for a year costs 1,500,000 QALYs. The liver failures may be flashier, but the 3^^^3 dust specks worth of poor sex lives add up to more disutility in the end.

I don’t want to overemphasize this particular calculation for a couple of reasons. First, SSRIs and nefazodone both have other side effects besides the major ones I’ve focused on here. Second, I don’t know if the level of SSRI-induced sexual dysfunction is as bad as the prostate-surgery-induced sexual dysfunction on the database. Third, there are a whole bunch of antidepressants that are neither SSRIs nor nefazodone and which might be safer than either.

But I do want to emphasize this pattern, because it recurs again and again.

II.

In that spirit, which would you rather have – something like a million people addicted to amphetamines, or something like ten people have their skin eat itself from the inside?

I can’t get good numbers on how many adults abuse Adderall, but a quick glance at the roster for my hospital’s rehab unit suggests “a lot”. Huffington Post calls it the most abused prescription drug in America, which sounds about right to me. Honestly there are worse things to be addicted to than Adderall, but it’s not completely without side effects. The obvious ones are anxiety, irritability, occasionally frank psychosis, and sometimes heart problems – but a lot of the doctors I work with go beyond what the research can really prove and suggest it can produce lasting negative personality change and predispose people to other forms of addictive and impulsive behavior.

If you’ve got to give adults a stimulant, I would much prefer modafinil. It’s not addictive, it lacks most of Adderall’s side effects, and it works pretty well. I’ve known many people on modafinil and they give it pretty universally positive reviews.

On the other hand, modafinil may or may not cause a skin reaction called Stevens Johnson Syndrome/Toxic Epidermal Necrolysis, which like most things with both “toxic” and “necro” in the name is really really bad. The original data suggesting a connection came from kids, who get all sorts of weird drug effects that adults don’t, but since then some people have claimed to have found a connection with adults. Some people get SJS anyway just by bad luck, or because they’re taking other drugs, so it’s really hard to attribute cases specifically to modafinil.

Gwern’s Modafinil FAQ mentions an FDA publication which argues that the background rate of SJS/TEN is 1-2 per million people per year, but the modafinil rate is about 6 per million people per year. However, there are only three known cases of a person above age 18 on modafinil getting SJS/TEN, and this might not be different from background rates after all. Overall the evidence that modafinil increases the rate of SJS/TEN in adults at all is pretty thin, and if it does, it’s as rare as hen’s teeth (in fact, very close to the same rate as liver failure from nefazodone).

(also: consider that like half of Silicon Valley is on modafinil, yet San Francisco Bay is not yet running red with blood.)

(also: ibuprofen is linked to SJS/TEN, with about the same odds ratio as modafinil, but nobody cares, and they are correct not to care.)

I said I’ve never seen a doctor prescribe nefazodone in real life; I can’t say that about modafinil. I have seen one doctor prescribe modafinil. It happened like this: a doctor I was working with was very upset, because she had an elderly patient with very low energy for some reason, I can’t remember, maybe a stroke, and wanted to give him Adderall, but he had a heart arrythmia and Adderall probably wouldn’t be safe for him.

I asked “What about modafinil?”

She said, “Modafinil? Really? But doesn’t that sometimes cause Stevens Johnson Syndrome?”

And then I glared at her until she gave in and prescribed it.

But this is very, very typical. Doctors who give out Adderall like candy have no associations with modafinil except “that thing that sometimes causes Stevens-Johnson Syndrome” and are afraid to give it to people.

III.

Nefazodone and modafinil are far from the only examples of this pattern. MAOIs are like this too. So is clozapine. If I knew more about things other than psychiatry, I bet I could think of examples from other fields of medicine.

And partially this is natural and understandable. Doctors swear an oath to “first do no harm”, and toxic epidermal necrolysis is pretty much the epitome of harm. Thought experiments like torture vs dust specks suggest that most people’s moral intuitions say that no amount of aggregated lesser harms like sexual side effects and amphetamine addictions can equal the importance of avoiding even a tiny chance of some great harm like liver failure or SJS/TEN. Maybe your doctor, if you asked her directly, would endorse a principled stance of “I am happy to give any number of people anxiety and irritability in order to avoid even the smallest chance of one case of toxic epidermal necrolysis.”

And yet.

The same doctors who would never dare give nefazodone, consider Seroquel a perfectly acceptable second-line treatment for depression. Along with other atypical antipsychotics, Seroquel raises the risk of sudden cardiac death by about 50%. The normal risk of cardiac sudden death in young people is about 10 in 100,000 per year, so if my calculations are right, low-dose Seroquel causes an extra cardiac death once per every 20,000 patient-years. That’s ten times as often as nefazodone causes an extra liver death.

Yet nefazodone was taken off of the market by its creators and consigned to the dustbin of pharmacological history, and Seroquel is the sixth-best-selling drug in the United States, commonly given for depression, simple anxiety, and sometimes even to help people sleep.

Why the disconnect? Here’s a theory: sudden cardiac death happens all the time; sometimes God just has it in for you and your heart stops working and you die. Antipsychotics can increase the chances of that happening, but it’s a purely statistical increase, such that we can detect it aggregated over large groups but never be sure that it played a role in any particular case. The average person who dies of Seroquel never knows they died of Seroquel, but the average person who dies from nefazodone is easily identified as a nefazodone-related death. So nefazodone gets these big stories in the media about this young person who died by taking this exotic psychiatric drug, and it becomes a big deal and scares the heck out of everybody. When someone dies of Seroquel, it’s just an “oh, so sad, I guess his time has come.”

But the end result is this. When treatment with an SSRI fails, nefazodone and Seroquel naively seem to be equally good alternatives. Except nefazodone has a death rate of 1/300,000 patient years, and Seroquel 1/20,000 patient years. And yet everyone stays the hell away from the nefazodone because it’s known to be unsafe, and chooses the Seroquel.

I conclude either doctors are terrible at thinking about risk, or else maybe a little too good at thinking about risk.

I bring up the latter option because there’s a principal-agent problem going on here. Doctors want to do what’s best for their patients. But they also want to do what’s best for themselves, which means not getting sued. No one has ever sued their doctor because they got a sexual side effect from SSRIs, but if somebody dies because they’re the lucky 1/300,000 who gets liver failure from nefazodone, you can bet their family’s going to sue. Suddenly it’s not a matter of comparing QALYs, it’s a matter of comparing zero percent chance of lawsuit with non-zero percent chance of lawsuit.

(Fermi calculation: if a doctor has 100 patients at a time on antidepressants, and works for 30 years, then if she uses Serzone as her go-to antidepressant, she’s risking a 1% chance of getting the liver failure side effect once in her career. That’s small, but since a single bad lawsuit can bankrupt a doctor, it’s worth taking seriously.)

And that would be a tough lawsuit to fight. “Yes, Your Honor, I knew when I prescribed this drug that it sometimes makes people’s livers explode, but the alternative often gives people a bad sex life, and according to the theory of utilitarianism as propounded by 18th century philosopher Jeremy Bentham – ” … “Bailiff, club this man”.

And the same facet of nefazodone that makes it exciting for the media makes it exciting for lawsuits. When someone dies of nefazodone toxicity, everyone knows. When someone dies of Seroquel, “oh, so sad, I guess his time has come”.

That makes Seroquel a lot safer than nefazodone. Safer for the doctor, I mean. The important kind of safer.

This is why, as I mentioned before, I hate lawsuits as a de facto regulatory mechanism. Our de jure regulatory mechanism, the FDA, is pretty terrible, but to its credit it hasn’t banned nefazodone. One time it banned clozapine because of a flashy rare side effect, but everyone yelled at them and they apologized and changed their mind. With lawsuits there’s nobody to yell at, so we just end up with people very quietly adjusting their decisions in the shadows and nobody else being any the wiser.

I don’t want to overemphasize this, because I think it’s only one small part of the problem. After all, a lot of countries withdrew nefazodone entirely and didn’t even give lawsuits a chance to enter the picture.

But whatever the cause, the end result is that drugs with rare but spectacular side effects get consistently underprescribed relative to drugs with common but merely annoying side effects, or drugs that have more side effects but manage to hide them better.

I will resist the urge to geek out about its minor medical errors¹ in favor of clarifying something more important.

The impression you’re supposed to get from this piece is a shady looking man handing you a briefcase full of cash and whispering “Hey, here’s $10,000 for you if you prescribe unnecessary medication.” The implication is the doctors who do this are awful and if you were in medicine you would have no trouble resisting this temptation.

In reality, pharma companies have figured out that some people have ethical qualms – “evil cannot possibly understand good” only works in movies – and adjusted their strategies accordingly.

We’ll start with a simple one. Imagine you’re a doctor, and your staff are complaining because the staff at every other doctor’s office has been getting these incredible free lunches every day – the video says drug companies aren’t supposed to give, like, Zagat-rated steakhouse lunches, but there’s still a lot of room between “Zagat-rated” and “Way better than the peanut butter and jelly sandwich you bring from home”. The nurses are grumbling and threatening to revolt and asking if you really appreciate them.

A drug company representative offers to provide your office with free lunches a couple of times a week.

You say “It would be really annoying to actually use the phrase ‘there’s no such thing as a free lunch’ here, so I will just ask what the catch is.”

They say “No catch. We don’t require you to ever prescribe any of our drugs. We don’t require you to listen to our presentation. We don’t even require you to read our promotional literature. Just accept our offer.”

You say “Why are you doing this?”

They say “Because every time you eat one of our lunches, you’ll associate the ice cold taste of Coca-Cola and the sweet warm chewy chocolate chip cookies with our company, and you’ll get positive feelings about it, and maybe those positive feelings will influence your prescription habits.”

You say “I think I’m a good enough doctor not to prescribe a drug solely because I get lunch from their company.”

They say “Look. We all know that most antidepressants are about equally effective. Sure, we split hairs and talk about how one has more anticholinergic side effects so it’s bad for patients with cholinergic sensitivity, and another has more chance of weird visual disturbances, but how often does someone come into your office and announce ‘Hey, I’m depressed, and also I have cholinergic sensitivity, but I LOVE weird visual disturbances!’? Although there are a few cases where one drug’s clearly a better choice than another, most of the time you’re about equally balanced between two or three options, and you just pick one at random. So maybe instead of picking one at random, you’ll pick the one you associate with delicious food. And if you do, so what? Nobody’s harmed. You would have just flipped a coin anyway.”

You say “I’d rather flip a coin than feel like I’m being pressured by what I had for lunch.”

They say “Look, you secretly worry anyway that you sometimes prescribe Effexor because the name makes it sound effective, or Paxil because the name makes it sound peaceful.”

You say “Wait, you can read my thoughts?”

They say “We’re a pharmaceutical company. Of course we can read your thoughts. Look. You already know that the mostly-meaningless choice of which of several equally effective drugs you prescribe is influenced by a bunch of silly marketing factors beyond your control. Why not add one more?”

“But -”

“Come to the Dark Side! We literally have cookies!”

Still not tempting enough for you?

Okay, imagine this. You’re a doctor and one of your patients comes in with incurable chronic pain that’s ruining their life. You try the normal medications on it and nothing works very well. There’s a high-tech next-generation medication available that you think is a good fit for your patient’s disease, but it’s not covered by their insurance and there’s no way the patient can afford it. You have to tell this guy that there’s nothing you can do for him.

Then a drug company representative comes to you bearing a big box of free samples. By “free samples” I mean hundreds of pills, enough to help the patient for the better part of a year – and maybe at the end of that time you’ll get another box of free samples. The drug rep doesn’t want you to sign your life away. She’s giving them for free, no obligation, maybe just listen to a sixty second speech on how to prescribe them safely and effectively (she wouldn’t want to give them to someone who won’t prescribe them effectively!) Are you really so fundamentalist in your approach to medical ethics that you won’t listen to a drug rep for sixty seconds in order to save a patient’s quality of life?

Most doctors – even the ethical ones who would refuse the briefcase full of cash – take the offer. This practice has come under increasing scrutiny recently. Some of the complaints are kind of dumb, but one very valid one is that a lot of the times what happens is you start off by giving the patient 100 days of free sample or something, then the free sample runs out, they’re fixated on that particular medication because it’s the one that worked for them, and they find some costly way to continue the (more expensive new) medication – instead of the two of you working harder to find some older less expensive medication that works equally well. A few drug companies have “fixed” this by giving out cards for “prescription programs” that solve some of the problems with free samples. These are even harder to resist, and they’re also given out by attractive drug reps who just want to tell you a few important facts about the drug before giving it to you.

Still not tempting enough for you?

Fine, then imagine this. You’re a doctor who really believes in a particular drug and is trying to convince the medical community to use more of it. For example, a couple weeks ago I wrote an article on suboxone saying it was one of the best medications for opiate abuse and I wish the medical community would pay more attention and prescribe it more often.

I wrote that article for free as a public service because I think that drug saves lives. But imagine that the company that makes suboxone approached me afterwards and said “Hey, you seem to have an important message to spread. Why don’t we sponsor you to go around the country for a week or two telling it to other doctors at medical conferences? We’ll get you first-class flights, put you up in five-star hotels, and give you a $10,000 stipend.”

I say “Wait a second, that sounds like taking pharmaceutical company money, and taking pharmaceutical company money is evil.”

They say “Look. You were trying to promote suboxone to people already. You were just doing a bad job because you were limited to one little blog. The more suboxone-promoting you do, the more doctors know about this drug – which you yourself have said is life-saving – and so the more lives get saved. If you’re willing to promote suboxone ineffectively for free, why not promote suboxone effectively for $10,000 plus nice hotels?”

I say “I’m still kind of uncomfortable with this.”

They say “Okay, well, it’s not our fault if hundreds of people die of drug overdoses because their doctors didn’t know suboxone was an option.”

You’re probably going to ask if I’ve ever accepted any of these offers. The boring truth is that I haven’t had to consider them because I’m a resident and residents are lower than dirt and the pharmaceutical companies know this and they don’t waste time trying to cozy up to us.

I have tasted the forbidden fruit only once, and it was my attending’s fault. She told us that there was a big dinner being planned for the entire psychiatric community of our city. The goal was to get doctors to meet nurses to meet therapists to meet social workers in one place so we could all get to know each other and talk about changes we could make to the system. It was very important that we attend, or else the nurses and therapists and social workers would think that the doctors were too snooty to interact with them and didn’t care about changing the system. Oh, and by the way the dinner was sponsored by PANEXA (here used in place of the real drug because I don’t want to get in trouble for calling them out) but there wouldn’t be any promotional material or pressure to prescribe PANEXA, honest, no sirree.

This was a tempting offer precisely because it was such a good idea. Everyone in the local psychiatric community deals with each other frequently, but we’d mostly never met before. I know them as the voice on the other side of the phone saying “No, no beds are available in our facility” or as the person who refuses to fax me my patient’s past medical history because the patient is too catatonic to sign a consent form. None of us are ever entirely sure what the others are doing, sometimes there are bad feelings, and it was reasonable to hope that maybe if we all met each other and socialized things would get a little smoother.

So we all meet at this restaurant, and immediately World War III breaks out. It’s like “Hi, I’m Mary, the clerk at Blue Sky Mental Health.” “MARY?! YOU’RE THE ONE WHO DIDN’T FAX ME THOSE RECORDS I NEEDED TWO MONTHS AGO! MY PATIENT WENT A WEEK ON THE WRONG DRUGS BECAUSE OF THAT!”

“Hi, I’m Dr. Alexander, I work at the inpatient unit in Our Lady Of An Undisclosed Location Hospital…” “WE HAD A PATIENT COME FROM THERE TWO WEEKS AGO AND HE ASSAULTED A STAFF MEMBER. IF YOU’RE A REAL HOSPITAL WHY CAN’T YOU DO PROPER VIOLENCE ASSESSMENTS?”

It turned out that the nurses hated the social workers for making them wait on the phone forever in order to get a straight answer. The social workers hated the nurses for always calling them up when they were busy about things and expecting an answer RIGHT NOW. The social workers hated the doctors for giving patients one measly prescription, then handing the case over to them to fix all of the impossible problems in the patient’s life. The doctors hated the social workers, because when we give patients one measly prescription and then hand the case over to the social workers to fix all of the patient’s impossible problems, sometimes the impossible problems don’t get fixed.

Anyway, in the midst of all of this, there was one guy who was staying completely calm, talking nicely to everybody, helping people see each other’s sides of the issue, just a really serene well-adjusted guy. I escaped over to his table and asked him who he was and why he was here.

“Oh,” he said “I’m a paranoid schizophrenic currently on PANEXA.”

Of course he was.

Then we all broke off into our own groups and got some incredible Italian food.²

What I’m saying is, pharmaceutical companies are sneaky.

Footnotes

1: By which I mean “succumb to the urge to geek out about its minor medical errors, but in the footnotes”.

The schizophrenic guy worked for one of the local psychiatric community services groups doing community outreach. I never did figure out whether he was there as a coincidence or whether the pharmaceutical company had arranged to have him there. I suspect the latter but I have no proof.

It’s 2065. Not giving your kids super-enhancement designer baby gene therapy isn’t your “choice”. If you don’t super-enhance your kids, you are a bad parent. It’s that simple.

Harsh? Maybe. But consider the latest survey, which found that about five percent of parents fail to super-enhance their children by the time they enter kindergarten. These aren’t poor people who can’t afford super-enhancement designer baby gene therapy. These are mostly rich, highly educated individuals in places like California and Oregon who say they think it’s more “natural” to leave their children defenseless against various undesirable traits. “I just don’t think it’s right to inject retroviral vectors into my baby’s body to change her from the way God made her,” one Portland woman was quoted by the Times as saying earlier this week. Other parents referred to a 2048 study saying the retroviral injections, usually given in the first year of life, increase the risk of various childhood cancers – a study that has since been soundly discredited.

These parents will inevitably bring up notions of “personal freedom”. But even if we accept the dubious premise that parents have a right to sacrifice their children’s health, refusing super-enhancement designer baby gene therapy isn’t just a personal choice. It’s a public health issue that affects everybody in society.

In 2064 there were almost 200 murders nationwide, up from a low of fewer than 50 in 2060. Why is this killer, long believed to be almost eradicated, making a comeback? Criminologists are unanimous in laying the blame on unenhanced children, who lack the improved impulse-control and anger-management genes included in every modern super-enhancement designer baby gene therapy package.

There were over a dozen fatal car accidents on our nation’s roads last year. The problem is drivers who weren’t enhanced as children and who lack the super-reflexes the rest of us take for granted. This is compounded when they drink before getting on the road, since unenhanced people become impaired by alcohol and their already inferior reflexes deteriorate further. Since the promise of self-driving cars continues to be tied up in regulatory hassles, we can expect many more such needless deaths as long as irresponsible parents continue to consider science “optional”.

And finally, there was a recent outbreak of measles at Disneyland Europa – even though we thought this disease had been eradicated decades ago. Scientists traced the problem to unvaccinated tourists. They further found that all of these unvaccinated individuals were unenhanced. Lacking the cognitive optimization that would help them understand psychoneuroimmunology on an intuitive level, they were easy prey for discredited ideas like “vaccines cause autism”.

So no, super-enhancing your kids isn’t a “personal choice”. It’s your basic duty as a parent and a responsible human being. People in places like India and Neo-Songhai and Venus which suffer from crime and disease make great personal sacrifices to get their children to gene therapy clinics and give them the super-enhancement designer baby gene injection that ensures them a better life. And you start off in a privileged position in America, benefitting from the superenhancement of millions of your fellow citizens, and you think you can just say “No thanks”?

So I don’t want to hear another word from the “but my freedom!” crowd. Unenhanced kids shouldn’t be allowed in school. They shouldn’t be allowed to drive. They shouldn’t be allowed in public places where they can cause problems. And parents who refuse to enhance their children should be put in jail, the same as anyone else whose actions lead to death and suffering. Because not super-enhancing your kids isn’t a “choice”. It’s child abuse.

Mora LeQuivalence is an Assistant Professor of Bioethics at Facebook University. Her latest book, “A Flight Too Far”, argues that the recent Danish experiment with giving children wings is a disgusting offense against the natural order and should be banned worldwide and prosecuted in the International Criminal Court. It is available for 0.02Ƀ on Amazon.com

Related: Transhumanism Is Simplified Humanism, Alicorn’s Alternate Universe Social Justice Series

It’s too bad that most people will probably shy away from reading it, because it gets a lot of stuff really right.

The article’s thesis is also its subtitle: “There’s a treatment for heroin addiction that actually works; why aren’t we using it?” To save you the obligatory introductory human interest story: that treatment is suboxone. Its active ingredient is the drug buprenorphine, which is kind of like a safer version of methadone. Suboxone is slow-acting, gentle, doesn’t really get people high, and is pretty safe as long as you don’t go mixing it with weird stuff. People on suboxone don’t experience opiate withdrawal and have greatly decreased cravings for heroin. I work at a hospital that’s an area leader in suboxone prescription, I’ve gotten to see it in action, and it’s literally a life-saver.

Conventional heroin treatment is abysmal. Rehab centers aren’t licensed or regulated and most have little interest in being evidence-based. Many are associated with churches or weird quasi-religious groups like Alcoholics Anonymous. They don’t necessarily have doctors or psychologists, and some actively mistrust them. All of this I knew. What I didn’t know until reading the article was that – well, it’s not just that some of them try to brainwash addicts. It’s more that some of them try to cargo cult brainwashing, do the sorts of things that sound like brainwashing to them, without really knowing how brainwashing works assuming it’s even a coherent goal to aspire to. Their concept of brainwashing is mostly just creating a really unpleasant environment, yelling at people a lot, enforcing intentionally over-strict rules, and in some cases even having struggle-session-type-things where everyone in the group sits in a circle, scream at the other patients, and tell them they’re terrible and disgusting. There’s a strong culture of accusing anyone who questions or balks at any of it of just being an addict, or “not really wanting to quit”.

I have no problem with “tough love” when it works, but in this case it doesn’t. Rehab programs make every effort to obfuscate their effectiveness statistics – I blogged about this before in Part II here – but the best guesses by outside observers is that for a lot of them about 80% to 90% of their graduates relapse within a couple of years. Even this paints too rosy a picture, because it excludes the people who gave up halfway through.

Suboxone treatment isn’t perfect, and relapse is still a big problem, but it’s a heck of a lot better than most rehabs. Suboxone gives people their dose of opiate and mostly removes the biological half of addiction. There’s still the psychological half of addiction – whatever it was that made people want to get high in the first place – but people have a much easier time dealing with that after the biological imperative to get a new dose is gone. Almost all clinical trials have found treatment with methadone or suboxone to be more effective than traditional rehab. Even Cochrane Review, which is notorious for never giving a straight answer to anything besides “more evidence is needed”, agrees that methadone and suboxone are effective treatments.

Some people stay on suboxone forever and do just fine – it has few side effects and doesn’t interfere with functioning. Other people stay on it until they reach a point in their lives when they feel ready to come off, then taper down slowly under medical supervision, often with good success. It’s a good medication, and the growing suspicion it might help treat depression is just icing on the cake.

There are two big roadblocks to wider use of suboxone, and both are enraging.

The first roadblock is the #@$%ing government. They are worried that suboxone, being an opiate, might be addictive, and so doctors might turn into drug pushers. So suboxone is possibly the most highly regulated drug in the United States. If I want to give out OxyContin like candy, I have no limits but the number of pages on my prescription pad. If I want to prescribe you Walter-White-level quantities of methamphetamine for weight loss, nothing is stopping me but common sense. But if I want to give even a single suboxone prescription to a single patient, I have to take a special course on suboxone prescribing, and even then I am limited to only being able to give it to thirty patients a year (eventually rising to one hundred patients when I get more experience with it). The (generally safe) treatment for addiction is more highly regulated than the (very dangerous) addictive drugs it is supposed to replace. Only 3% of doctors bother to jump through all the regulatory hoops, and their hundred-patient limits get saturated almost immediately. As per the laws of suppy and demand, this makes suboxone prescriptions very expensive, and guess what social class most heroin addicts come from? Also, heroin addicts often don’t have access to good transportation, which means that if the nearest suboxone provider is thirty miles from their house they’re out of luck. The List Of Reasons To End The Patient Limits On Buprenorphine expands upon and clarifies some of these points.

(in case you think maybe the government just honestly believes the drug is dangerous – nope. You’re allowed to prescribe without restriction for any reason except opiate addiction)

The second roadblock is the @#$%ing rehab industry. They hear that suboxone is an opiate, and their religious or quasi-religious fanaticism goes into high gear. “What these people need is Jesus and/or their Nondenominational Higher Power, not more drugs! You’re just pushing a new addiction on them! Once an addict, always an addict until they complete their spiritual struggle and come clean!” And so a lot of programs bar suboxone users from participating.

This doesn’t sound so bad given the quality of a lot of the programs. Problem is, a lot of these are closely integrated with the social services and legal system. So suppose somebody’s doing well on suboxone treatment, and gets in trouble for a drug offense. Could be that they relapsed on heroin one time, could be that they’re using something entirely different like cocaine. Judge says go to a treatment program or go to jail. Treatment program says they can’t use suboxone. So maybe they go in to deal with their cocaine problem, and by the time they come out they have a cocaine problem and a heroin problem.

And…okay, time for a personal story. One of my patients is a homeless man who used to have a heroin problem. He was put on suboxone and it went pretty well. He came back with an alcohol problem, and we wanted to deal with that and his homelessness at the same time. There are these organizations called three-quarters houses – think “halfway houses” after inflation – that take people with drug problems and give them an insurance-sponsored place to live. But the catch is you can’t be using drugs. And they consider suboxone to be a drug. So of about half a dozen three-quarters houses in the local area, none of them would accept this guy. I called up the one he wanted to go to, said that he really needed a place to stay, said that without this care he was in danger of relapsing into his alcoholism, begged them to accept. They said no drugs. I said I was a doctor, and he had my permission to be on suboxone. They said no drugs. I said that seriously, they were telling me that my DRUG ADDICTED patient who was ADDICTED TO DRUGS couldn’t go to their DRUG ADDICTION center because he was on a medication for treating DRUG ADDICTION? They said that was correct. I hung up in disgust.

So I agree with the pessimistic picture painted by the article. I think we’re ignoring our best treatment option for heroin addiction and I don’t see much sign that this is going to change in the future.

But the health care system not being very good at using medications effectively isn’t news. I also thought this article was interesting because it touches on some of the issues we discuss here a lot:

The value of ritual and community. A lot of the most intelligent conservatives I know base their conservativism on the idea that we can only get good outcomes in “tight communities” that are allowed to violate modern liberal social atomization to build stronger bonds. The Army, which essentially hazes people with boot camp, ritualizes every aspect of their life, then demands strict obedience and ideological conformity, is a good example. I do sometimes have a lot of respect for this position. But modern rehab programs seem like a really damning counterexample. If you read the article, you will see that this rehabs are trying their best to create a tightly-integrated religiously-inspired community of exactly that sort, and they have abilities to control their members and force their conformity – sometimes in ways that approach outright abuse – that most institutions can’t even dream of. But their effectiveness is abysmal. The entire thing is for nothing. I’m not sure whether this represents a basic failure in the idea of tight communities, or whether it just means that you can’t force them to exist ex nihilo over a couple of months. But I find it interesting.

My love-hate relationship with libertarianism. Also about the rehabs. They’re minimally regulated. There’s no credentialing process or anything. There are many different kinds, each privately led, and low entry costs to creating a new one. They can be very profitable – pretty much any rehab will cost thousands of dollars, and the big-name ones cost much more. This should be a perfect setup for a hundred different models blooming, experimenting, and then selecting for excellence as consumers drift towards the most effective centers. Instead, we get rampant abuse, charlatanry, and uselessness.

On the other hand, when the government rode in on a white horse to try to fix things, all they did was take the one effective treatment, regulate it practically out of existence, then ride right back out again. So I would be ashamed to be taking either the market’s or the state’s side here. At this point I think our best option is to ask the paraconsistent logic people to figure out something that’s neither government nor not-government, then put that in charge of everything.

Society is fixed, biology is mutable. People have tried everything to fix drug abuse. Being harsh and sending drug users to jail. Being nice and sending them to nice treatment centers that focus on rehabilitation. Old timey religion where fire-and-brimstone preachers talk about how Jesus wants them to stay off drugs. Flaky New Age religion where counselors tell you about how drug abuse is keeping you from your true self. Government programs. University programs. Private programs. Giving people money. Fining people money. Being unusually nice. Being unusually mean. More social support. Less social support. This school of therapy. That school of therapy. What works is just giving people a chemical to saturate the brain receptor directly. We know it works. The studies show it works. And we’re still collectively beating our heads against the wall of finding a social solution.

1. Weight gain does not depend on calories in versus calories out, even in the loosest sense.

2. Weight gain is entirely a function of calories in versus calories out, but calories may move in unexpected ways not linked to the classic “eat” and “exercise” dichotomy. For example, some people may have “fast metabolisms” which burn calories even when they are not exercising. These people may stay very thin even if they eat and exercise as much as much more obese people.

3. Weight gain is entirely a function of calories in versus calories out, and therefore of how much you eat and exercise. However, these are in turn mostly dependent on the set points of a biologically-based drive. For example, some people may have overactive appetites, and feel starving unless they eat an amount of food that will make them fat. Other people will have very strong exercise drives and feel fidgety unless they get enough exercise to keep them very thin. These things can be altered in various ways which cause weight gain or loss, without the subject exerting willpower. For example, sleep may cause weight loss because people who get a good night sleep have decreased appetite and lower levels of appetite-related hormones.

4. Weight gain is entirely a function of calories in versus calories out, and therefore of how much you eat and exercise. That means diet is entirely a function of willpower and any claim that factors other than amount of food eaten and amount of exercise performed can affect weight gain is ipso facto ridiculous. For example, we can dismiss claims that getting a good night’s sleep helps weight loss, because that would violate the laws of thermodynamics.

1 and 4 are kind of dumb. 1 is dumb because…well, to steal an Eddington quote originally supposed apply to the second law of thermodynamics:

If someone points out to you that your pet theory of the universe is in disagreement with Maxwell’s equations — then so much the worse for Maxwell’s equations. If it is found to be contradicted by observation — well, these experimentalists do bungle things sometimes. But if your theory is found to be against…thermodynamics I can give you no hope; there is nothing for it but to collapse in deepest humiliation.

But 4 is also dumb. We have a long list of things that affect weight gain – for example, patients on the powerful psychiatric medication clozapine usually gain a lot of weight – fifteen pounds more on average than people on safer antipsychotics. Other medications are known to increase weight to a lesser degree, and some medications even decrease weight, though you wouldn’t like the side effects of most of them. Certain genetic diseases are also known to cause increased weight – Prader-Willi syndrome, for example.

One could try to rescue 4 by saying that people with rare genetic diseases or taking powerful prescription-only medications are a different story and in normal people it’s entirely controlled by willpower. But first, this is an area where possibility proofs are half the battle, and we have a possibility proof. And second, there are more than enough studies about genetics, microbiome, and, yes, sleep showing that all of these things can have effects in normal people.

So 1 and 4 are out. And although I do sometimes see people pushing them, they mostly seem to do a thriving business as straw men for people who want to accuse their opponents of saying something absurd.

The most interesting debate to be had is between 2 and 3. 3 says that all of the interventions that we know affect weight – certain pills, certain recreational drugs, changes in gut bacteria, whatever – do it by affecting appetite and exercise drive. 2 says that basal metabolism is also involved. 3 seems to at least leave open the possibility of just starving yourself even when your body is telling you really hard to eat. 2 says even that won’t work.

There’s room for a little bit of gradation between 2 and 3. A lot of people suggest that one way “fast metabolism” presents is by people fidgeting a lot, which is sort of the same as “your body increases its exercise drive”.

But in general, I think 2 is an important issue that does cause at least some interpersonal weight differences.

We’ll start with the “possibility proof” again. MRAP2. It’s a gene. Scientists can delete it in mice. These mice will eventually develop excessive appetites. But when they are young, they eat the same amount as any other mouse, but still get fatter.

Likewise, 2,4-dinitrophenol is a cellular uncoupling agent which increases metabolic rate and consistently produces weight loss of 2-3 pounds per week. It would be an excellent solution to all of our obesity-related problems if the papers on it didn’t keep having names like 2,4-Dinitrophenol: A Weight Loss Agent With Significant Acute Toxicity And Risk Of Death.

So what about everyday life?

A study of individual variation in basal metabolic rate found very significant interpersonal differences. A lot of that was just “some people are bigger than others”, but some of it wasn’t – they state that “twenty-six percent of the variance remained unexplained”. The Wikipedia article puts this in context: “One study reported an extreme case where two individuals with the same lean body mass of 43 kg had BMRs of 1075 kcal/day (4.5 MJ/day) and 1790 kcal/day (7.5 MJ/day). This difference of 715 kcal/day (67%) is equivalent to one of the individuals completing a 10 kilometer run every day”

Dr. Claude Bouchard and his team stuck 12 pairs of male identical twins in isolation chambers where their caloric intake and exercise could be carefully controlled, then fed them more calories than their bodies needed. All sets of twins gained weight, and in all twin groups both twins gained about the same amount of weight as each other, but the amount of weight gained varied between twin pairs by a factor of 3 (from 4 to 13 kg).

A lot of the sites that talk about this thing are careful to say that people “can’t blame” genes for their obesity, because obesity levels have been rising for decades and genes can’t change that quickly. I think this is wrong-headed. True, genes are not the source of the modern rise in obesity levels. But it’s entirely possible that a globally rising tide of obesity has disproportionately affected the people with the wrong genes. Just as Bouchard fed the same amount extra to all his study participants but some of them gained more weight than others, so if you put an entire civilization worth of people in an obesogenic environment, some of them might be genetically predisposed to do worse than the rest.

A more practical question – can individual people’s metabolism change?

I am personally predisposed to answer in the affirmative. In my early twenties, I ate a crazy amount every day – two bagels with breakfast, cookies with lunch, a big dinner followed by dessert – and I stayed pretty thin throughout. Now I’m thirty, I eat a very restrained diet, and my weight still hovers at just above the range where I am supposed to be. I know that people are famously bad at understanding how much they’re eating and exercising, but seriously if you try to convince me that I’m eating more now than I was then I’m going to start doubting my own sanity, or at least my autobiographical memory.

But there’s not much evidence to back me up. Metabolic rate is well-known to decline with age, but linearly and predictably. And it changes with muscle mass, but only minimally – and I don’t think I used to be any more muscular.

The sites that talk about drastic and unexpected ways to change metabolism seem mostly crackpottish. This isn’t to say their methods don’t work – green tea, for example, has a statistically significant effect – but it’s all so small as to be pretty meaningless in a real-world context.

So my own story seems to be on shaky ground. But as far as I can tell, the people arguing that they’re trying just as hard as anybody else but still unable to lose weight because of their metabolism are very possibly right.

Standard disclaimer: I’m not a researcher in this field, I’m not board-certified as a full psychiatrist yet, and what I remember of biochemistry is limited to being pretty sure there’s something called a “Krebs cycle” involved somewhere. That having been said:

This is pretty legit.

Start with From inflammation to sickness and depression, Dantzer et al (2008), who note that being sick makes you feel lousy [citation needed]. Drawing upon evolutionary psychology, they theorize this is an adaptive response to make sick people stay in bed (or cave, or wherever) so the body can focus all of its energy on healing. A lot of sickness behavior – being tired, not wanting to do anything, not eating, not wanting to hang around other people – seems kind of like mini-depression.

All of this stuff is regulated by chemicals called cytokines, which are released by immune cells that have noticed an injury or infection or something. They are often compared to a body-wide “red alert” sending the message “sickness detected, everyone to battle stations”. This response is closely linked to the idea of “inflammation”, the classic example of which is the locally infected area that has turned red and puffy. Most inflammatory cytokines handle the immune response directly, but a few of them – especially interleukin-1B and tumor necrosis factor alpha – cause this depression-like sickness behavior. It is noted that:

In general, animals injected with IL-1ß or TNF-a stay in a corner of their home cage in a hunched posture and show little or no interest in their physical and social environment unless they are stimulated. Specifically, they show decreased motor activity, social withdrawal, reduced food and water intake, increased slow-wave sleep and altered cognition

Here are some other suspicious facts about depression and inflammation:

– Exercise, good diet and sleep reduce inflammation; they also help depression.

– Stress increases inflammation and is a known trigger for depression.

– Rates of depression are increasing over time, with the condition seemingly very rare in pre-modern non-Westernized societies. This is commonly attributed to the atomization and hectic pace of modern life. But levels of inflammation are also increasing over time, probably because we have a terrible diet that disrupts the gut microbiota that are supposed to be symbioting with the immune system. Could this be another one of the things we think are social that turn out to be biological?

– SSRI antidepressants, like most medications, have about five zillion effects. One of the effects is to reduce the level of inflammatory cytokines in the body. Is it possible that this is why they work, and all of this stuff about serotonin receptors in the brain is a gigantic red herring?

– It’s always been a very curious piece of trivia that treating depression comorbid with heart disease significantly decreases your chances of dying from the heart disease. People just sort of nod their heads and say “You know, mind-body connection”. But inflammation is known to be implicated in cardiovascular disease. If treating depression is a form of lowering inflammation, this would make perfect sense.

– Rates of depression are much higher in sick people. Cancer patients are especially famous for this. No one gets too surprised here, because having cancer is hella depressing. But it’s always been interesting (to me at least) that as far as we can tell, antidepressants treat cancer-induced depression just as well as any other type. Are antidepressants just that good? Or is the link between cancer being sad and cancer causing depression only part of the story, with the other part being that the body’s immune response to cancer causes inflammatory cytokine release, which antidepressants can help manage?

– Along with cancer, depression is common in many other less immediately emotion-provoking illnesses like rheumatoid arthritis and diabetes. The common thread among these illnesses is inflammation.

– Inflammation changes the activity level of the enzyme indoleamine 2,3 dioxygenase. This enzyme produces kynurenines which interact with the NMDA receptor, a neurotransmitter receptor implicated in depression and various other psychiatric diseases (in case your first question upon learning about this pathway is the same as mine: yes, kynurenines got their name because they were first found in dog urine).

– Sometimes doctors treat diseases like hepatitis by injecting artificial cytokines to make the immune system realize the threat and ramp up into action. Cytokine administration treatments very commonly cause depression as a side effect. This depression can be treated with standard antidepressants.

– Also, it turns out we can just check and people with depression have more cytokines.

There’s also some evidence against the theory. People with depression have more cytokines, but it’s one of those wishy-washy “Well, if you get a large enough sample size, you’ll see a trend” style relationships, rather than “this one weird trick lets you infallibly produce depression”.

But for me the strongest evidence against is a general feeling that it’s very easy to get lots of convincing evidence for a theory in medicine whether or not it’s true.

Twenty years ago, everyone was super-convinced that depression was caused by low serotonin levels. We found that depressed people on average had lower serotonin levels than non-depressed people. We found that giving people drugs that increased serotonin treated depression. We did lots of studies proving serotonin was a vital chemical that regulated mood. We found that genes affecting serotonin-related proteins were linked to depression. We did PET scans that found abnormally high levels of activity in serotonin-related enzymes in the brains of depressed people. It was all very convincing. And right now everyone’s pretty sure it’s wrong.

Ten years ago, everyone was super-convinced that depression was caused by under-secretion of the neuro-hormone BDNF and subsequent decline in hippocampal neurogenesis. It was dutifully found that depressed people had less BDNF than everyone else, and less hippocampal neurogenesis. Exercise, sleep, good diet, and all the other things that help depression were found to also raise levels of BDNF. Chemical pathways were trotted out by which effective antidepressants would probably raise BDNF levels. I think this theory is still very popular, but for the inflammation theory to be right someone will either have to disprove this one or tie it together with some theory of why inflammation decreases BDNF or low BDNF increases inflammation or something else. I do see some evidence that this is true, but to fully integrate the theories is going to take a lot more than that.

And these are just the two most recent and most famous. We have Freud’s psychoanalytic theory of depression, lots of people studying dysregulation of the hypothalamopituitaryadrenocortical axis, some pointers to dysregulation in the second messenger system, et cetera. All of these theories have great evidence.

Point is, now we have another theory that neatly explains how depression starts, how antidepressants work, why diet and exercise are good for you, and all the things all the other theories explained. Maybe the third time’s the charm?

A lot of the things in the body are really complex. Inflammation definitely affects serotonin – the indoleamine 2,3 dioxygenase enzyme acts on serotonin’s immediate precursor. It affects BDNF levels, as above, which in turn affect hippocampal neurogenesis. The hypothalamopituitaryadrenocortical axis releases cortisol, which downregulates the immune system and decreases the action of inflammatory cytokines. All of the anxiety-inducing life events and intrapsychic conflicts and secret desires to marry your mother that Freud thought caused depression produce a lot of stress, which both releases cortisol and reduces normal ability to regulate inflammatory response.

So basically all of these systems are intimately interconnected, and probably before this is done with researchers will find five more systems intimately interconnected with all of these. It might be that inflammation is the master system which causes a cascade of events in all of the others. It might be that one of the others is the master system. It might be that depression is a collection of multiple different diseases, and some are caused by one thing and others by another. It might be that looking for a “master system” is silly and that the true mathematical relationship between all of these things is such a chaotic process that all you can say is that they all stumbled together into the wrong attractor point and things deteriorated from there.

Anyway, all this is for much smarter people than me to figure out. The question I’m most interested in: can we treat depression by giving people anti-inflammatory drugs?

The answer seems to be: it depends how strongly you object to getting a heart attack.

Aspirin is a great anti-inflammatory drug. It’s pretty safe in adults (except for a small risk of GI bleeding) and it decreases risk of cardiovascular disease and cancer as well. If you could treat depression with aspirin, you’d be home free. However, the most convincing review I have seen for aspirin is unimpressed. It points out that some trials have shown negative effects for aspirin, and that long-term use of aspirin can increase intestinal permeability which decreases ability to regulation inflammation which is the opposite of what we want. Right now there isn’t much evidence on this issue, but what there is isn’t promising.

Most researchers have chosen to focus on celecoxib (Celebrex™®©, a high-tech next-generation anti-inflammatory). Here the evidence is actually very strong. Last month’s JAMA Psychiatry contained Effect of anti-inflammatory treatment on depression, depressive symptoms, and adverse effects: a systematic review and meta-analysis of randomized clinical trials, Kohler et al, (2014), which analyzes ten studies with a total of 4000 people taking celecoxib and finds an effect size similar to that of SSRI antidepressants. This is promising and exciting.

They add: “We found no evidence of an increased number of gastrointestinal or cardiovascular events after 6 weeks.”

That’s probably because they didn’t wait long enough. Celecoxib is very closely related to the infamous rofecoxib (Vioxx™®©) which got pulled from the market for quadrupling heart attack risk. Celecoxib is safer; it only increases your risk by some smaller amount depending on dose. Studies conflict, but maybe 33% for a standard regimen?

On the other hand, if ten percent of Americans are on SSRIs right now, and there are 1.5 million heart attacks per year in the US, and celecoxib increases that by 33%, then switching everyone from SSRI to celecoxib would cause…quick Fermi calculation…ignore interactions…50,000 extra heart attacks per year. Ouch.

Celecoxib is a good drug for its indicated uses, which involve treating chronic pain conditions that nothing else can treat safely. But it’s hardly something I’d want to start giving to every depressed patient who walks into a psychiatrist’s office. Maybe as a third line or fourth line drug for desperate people. But then, we already have plenty of good third-line and fourth-line drugs for desperate people. You want strong psychiatric medication and aren’t too concerned about the state of your cardiovascular system? Here, have an antipsychotic!

So in conclusion, I think the inflammatory hypothesis of depression is very likely part of the picture. Whether it’s the main part of the picture or just somewhere in the background remains to be seen, but for now it looks encouraging. Anti-inflammatory drugs do seem to treat depression, which is a point in the theory’s favor, but right now the only one that has strong evidence behind it has side effects that make it undesirable for most people. There’s a lot of room to hope that in the future researchers will learn more about exactly how this cytokine thing works and be able to design antidepressant drugs that target the appropriate cytokines directly. Until then, your best bets are the anti-inflammatory mainstays: good diet, good sleep, plenty of exercise, low stress levels, and all the other things we already know work.

I’ve worked with doctors who think Alcoholics Anonymous is so important for the treatment of alcoholism that anyone who refuses to go at least three times a week is in denial about their problem and can’t benefit from further treatment.

I’ve also worked with doctors who are so against the organization that they describe it as a “cult” and say that a physician who recommends it is no better than one who recommends crystal healing or dianetics.

I finally got so exasperated that I put on my Research Cap and started looking through the evidence base.

My conclusion, after several hours of study, is that now I understand why most people don’t do this.

The studies surrounding Alcoholics Anonymous are some of the most convoluted, hilariously screwed-up research I have ever seen. They go wrong in ways I didn’t even realize research could go wrong before. Just to give some examples:

– In several studies, subjects in the “not attending Alcoholics Anonymous” condition attended Alcoholics Anonymous more than subjects in the “attending Alcoholics Anonymous” condition.

– Almost everyone’s belief about AA’s retention rate is off by a factor of five because one person long ago misread a really confusing graph and everyone else copied them without double-checking.

– The largest study ever in the field, a $30 million effort over 8 years following thousands of patients, had no untreated control group.

Not only are the studies poor, but the people interpreting them are heavily politicized. The entire field of addiction medicine has gotten stuck in the middle of some of the most divisive issues in our culture, like whether addiction is a biological disease or a failure of willpower, whether problems should be solved by community and peer groups or by highly trained professionals, and whether there’s a role for appealing to a higher power in any public organization. AA’s supporters see it as a scruffy grassroots organization of real people willing to get their hands dirty, who can cure addicts failed time and time again by a system of glitzy rehabs run by arrogant doctors who think their medical degrees make them better than people who have personally fought their own battles. Opponents see it as this awful cult that doesn’t provide any real treatment and just tells addicts that they’re terrible people who will never get better unless they sacrifice their identity to the collective.

As a result, the few sparks of light the research kindles are ignored, taken out of context, or misinterpreted.

The entire situation is complicated by a bigger question. We will soon find that AA usually does not work better or worse than various other substance abuse interventions. That leaves the sort of question that all those fancy-shmancy people with control groups in their studies don’t have to worry about – does anything work at all?

I.

We can start by just taking a big survey of people in Alcoholics Anonymous and seeing how they’re doing. On the one hand, we don’t have a control group. On the other hand…well, there really is no other hand, but people keep doing it.

According to AA’s own surveys, one-third of new members drop out by the end of their first month, half by the end of their third month, and three-quarters by the end of their first year. “Drop out” means they don’t go to AA meetings anymore, which could be for any reason including (if we’re feeling optimistic) them being so completely cured they no longer feel they need it.

There is an alternate reference going around that only 5% (rather than 25%) of AA members remain after their first year. This is a mistake caused by misinterpreting a graph showing that only five percent of members in their first year were in their twelfth month of membership, which is obviously completely different. Nevertheless, a large number of AA hate sites (and large rehabs!) cite the incorrect interpretation, for example the Orange Papers and RationalWiki’s page on Alcoholics Anonymous. In fact, just to keep things short, assume RationalWiki’s AA page makes every single mistake I warn against in the rest of this article, then use that to judge them in general. On the other hand, Wikipedia gets it right and I continue to encourage everyone to use it as one of the most reliable sources of medical information available to the public (I wish I was joking).

This retention information isn’t very helpful, since people can remain in AA without successfully quitting drinking, and people may successfully quit drinking without being in AA. However, various different sources suggest that, of people who stay in AA a reasonable amount of time, about half stop being alcoholic. These numbers can change wildly depending on how you define “reasonable amount of time” and “stop being alcoholic”. Here is a table, which I have cited on this blog before and will probably cite again:

Behold. Treatments that look very impressive (80% improved after six months!) turn out to be the same or worse as the control group. And comparing control group to control group, you can find that “no treatment” can appear to give wildly different outcomes (from 20% to 80% “recovery”) depending on what population you’re looking at and how you define “recovery”.

Twenty years ago, it was extremely edgy and taboo for a reputable scientist to claim that alcoholics could recover on their own. This has given way to the current status quo, in which pretty much everyone in the field writes journal articles all the time about how alcoholics can recover on their own, but make sure to harp upon how edgy and taboo they are for doing so. From these sorts of articles, we learn that about 80% of recovered alcoholics have gotten better without treatment, and many of them are currently able to drink moderately without immediately relapsing (something else it used to be extremely taboo to mention). Kate recently shared an good article about this: Most People With Addiction Simply Grow Out Of It: Why Is This Widely Denied?

Anyway, all this stuff about not being able to compare different populations, and the possibility of spontaneous recovery, just mean that we need controlled experiments. The largest number of these take a group of alcoholics, follow them closely, and then evaluate all of them – the AA-attending and the non-AA-attending – according to the same criteria. For example Morgenstern et al (1997), Humphreys et al (1997) and Moos (2006). Emrick et al (1993) is a meta-analyses of a hundred seventy three of these. All of these find that the alcoholics who end up going to AA meetings are much more likely to get better than those who don’t. So that’s good evidence the group is effective, right?

Bzzzt! No! Wrong! Selection bias!

People who want to quit drinking are more likely to go to AA than people who don’t want to quit drinking. People who want to quit drinking are more likely to actually quit drinking than those who don’t want to. This is a serious problem. Imagine if it is common wisdom that AA is the best, maybe the only, way to quit drinking. Then 100% of people who really want to quit would attend compared to 0% of people who didn’t want to quit. And suppose everyone who wants to quit succeeds, because secretly, quitting alcohol is really easy. Then 100% of AA members would quit, compared to 0% of non-members – the most striking result it is mathematically possible to have. And yet AA would not have made a smidgeon of difference.

But it’s worse than this, because attending AA isn’t just about wanting to quit. It’s also about having the resources to make it to AA. That is, wealthier people are more likely to hear about AA (better information networks, more likely to go to doctor or counselor who can recommend) and more likely to be able to attend AA (better access to transportation, more flexible job schedules). But wealthier people are also known to be better at quitting alcohol than poor people – either because the same positive personal qualities that helped them achieve success elsewhere help them in this battle as well, or just because they have fewer other stressors going on in their lives driving them to drink.

Finally, perseverance is a confounder. To go to AA, and to keep going for months and months, means you’ve got the willpower to drag yourself off the couch to do a potentially unpleasant thing. That’s probably the same willpower that helps you stay away from the bar.

And then there’s a confounder going the opposite direction. The worse your alcoholism is, the more likely you are to, as the organization itself puts it, “admit you have a problem”.

These sorts of longitudinal studies are almost useless and the field has mostly moved away from them. Nevertheless, if you look on the pro-AA sites, you will find them in droves, and all of them “prove” the organization’s effectiveness.

III.

It looks like we need randomized controlled trials. And we have them. Sort of.

Brandsma (1980) is the study beloved of the AA hate groups, since it purports to show that people in Alcoholics Anonymous not only don’t get better, but are nine times more likely to binge drink than people who don’t go into AA at all.

There are a number of problems with this conclusion. First of all, if you actually look at the study, this is one of about fifty different findings. The other findings are things like “88% of treated subjects reported a reduction in drinking, compared to 50% of the untreated control group”.

Second of all, the increased binge drinking was significant at the 6 month followup period. It was not significant at the end of treatment, the 3 month followup period, the 9 month followup period, or the 12 month followup period. Remember, taking a single followup result out of the context of the other followup results is a classic piece of Dark Side Statistics and will send you to Science Hell.

Of multiple different endpoints, Alcoholics Anonymous did better than no treatment on almost all of them. It did worse than other treatments on some of them (dropout rates, binge drinking, MMPI scale) and the same as other treatments on others (abstinent days, total abstinence).

If you are pro-AA, you can say “Brandsma study proves AA works!”. If you are anti-AA, you can say “Brandsma study proves AA works worse than other treatments!”, although in practice most of these people prefer to quote extremely selective endpoints out of context.

However, most of the patients in the Brandsma study were people convicted of alcohol-related crimes ordered to attend treatment as part of their sentence. Advocates of AA make a good point that this population might be a bad fit for AA. They may not feel any personal motivation to treatment, which might be okay if you’re going to listen to a psychologist do therapy with you, but fatal for a self-help group. Since the whole point of AA is being in a community of like-minded individuals, if you don’t actually feel any personal connection to the project of quitting alcohol, it will just make you feel uncomfortable and out of place.

Also, uh, this just in, Brandsma didn’t use a real AA group, because the real AA groups make people be anonymous which makes it inconvenient to research stuff. He just sort of started his own non-anonymous group, let’s call it A, with no help from the rest of the fellowship, and had it do Alcoholics Anonymous-like stuff. On the other hand, many members of his control group went out into the community and…attended a real Alcoholics Anonymous, because Brandsma can’t exactly ethically tell them not to. So technically, there were more people in AA in the no-AA group than in the AA group. Without knowing more about Alcoholics Anonymous, I can’t know whether this objection is valid and whether Brandsma’s group did or didn’t capture the essence of the organization. Still, not the sort of thing you want to hear about a study.

Walsh et al (1991) is a similar study with similar confounders and similar results. Workers in an industrial plant who were in trouble for coming in drunk were randomly assigned either to an inpatient treatment program or to Alcoholics Anonymous. After a year of followup, 60% of the inpatient-treated workers had stayed sober, but only 30% of the AA-treated workers had.

The pro-AA side made three objections to this study, of which one is bad and two are good.

The bad objection was that AA is cheaper than hospitalization, so even if hospitalization is good, AA might be more efficient – after all, we can’t afford to hospitalize everyone. It’s a bad objection because the authors of the study did the math and found out that hospitalization was so much better than AA that it decreased the level of further medical treatment needed and saved the health system more money than it cost.

The first good objection: like the Brandsma study, this study uses people under coercion – in this case, workers who would lose their job if they refused. Fine.

The second good objection, and this one is really interesting: a lot of inpatient hospital rehab is AA. That is, when you go to an hospital for inpatient drug treatment, you attend AA groups every day, and when you leave, they make you keep going to the AA groups. In fact, the study says that “at the 12 month and 24 month assessments, the rates of AA affiliation and attendance in the past 6 months did not differ significantly among the groups.” Given that the hospital patients got hospital AA + regular AA, they were actually getting more AA than the AA group!

So all that this study proves is that AA + more AA + other things is better than AA. There was no “no AA” group, which makes it impossible to discuss how well AA does or doesn’t work. Frick.

Timko (2006) is the only study I can hesitantly half-endorse. This one has a sort of clever methodological trick to get around the limitation that doctors can’t ethically refuse to refer alcoholics to treatment. In this study, researchers at a Veterans’ Affairs hospital randomly assigned alcoholic patients to “referral” or “intensive referral”. In “referral”, the staff asked the patients to go to AA. In “intensive referral”, the researchers asked REALLY NICELY for the patients to go to AA, and gave them nice glossy brochures on how great AA was, and wouldn’t shut up about it, and arranged for them to meet people at their first AA meeting so they could have friends in AA, et cetera, et cetera. The hope was that more people in the “intensive referral” group would end out in AA, and that indeed happened scratch that, I just re-read the study and the same number of people in both groups went to AA and the intensive group actually completed a lower number of the 12 Steps on average, have I mentioned I hate all research and this entire field is terrible? But the intensive referral people were more likely to have “had a spiritual awakening” and “have a sponsor”, so it was decided the study wasn’t a complete loss and when it was found the intensive referral condition had slightly less alcohol use the authors decided to declare victory.

So, whereas before we found that AA + More AA was better than AA, and that proved AA didn’t work, in this study we find that AA + More AA was better than AA, and that proves AA does work. You know, did I say I hesitantly half-endorsed this study? Scratch that. I hate this study too.

IV.

All right, @#%^ this $@!&*. We need a real study, everything all lined up in a row, none of this garbage. Let’s just hire half the substance abuse scientists in the country, throw a gigantic wad of money at them, give them as many patients as they need, let them take as long as they want, but barricade the doors of their office and not let them out until they’ve proven something important beyond a shadow of a doubt.

This was about how the scientific community felt in 1989, when they launched Project MATCH. This eight-year, $30 million dollar, multi-thousand patient trial was supposed to solve everything.

The people going into Project MATCH might have been a little overconfident. Maybe “not even Zeus could prevent this study from determining the optimal treatment for alcohol addiction” overconfident. This might have been a mistake.

The study was designed with three arms, one for each of the popular alcoholism treatments of the day. The first arm would be “twelve step facilitation”, a form of therapy based off of Alcoholics Anonymous. The second arm would be cognitive behavioral therapy, the most bog-standard psychotherapy in the world and one which by ancient tradition must be included in any kind of study like this. The third arm would be motivational enhancement therapy, which is a very short intervention where your doctor tells you all the reasons you should quit alcohol and tries to get you to convince yourself.

There wasn’t a “no treatment” arm. This is where the overconfidence might have come in. Everyone knew alcohol treatment worked. Surely you couldn’t dispute that. They just wanted to see which treatment worked best for which people. So you would enroll a bunch of different people – rich, poor, black, white, married, single, chronic alcoholic, new alcoholic, highly motivated, unmotivated – and see which of these people did best in which therapy. The result would be an algorithm for deciding where to send each of your patients. Rich black single chronic unmotivated alcoholic? We’ve found with p < 0.00001 that the best place for someone like that is in motivational enhancement therapy. Such was the dream. So, eight years and thirty million dollars and the careers of several prestigious researchers later, the results come in, and - yeah, everyone does exactly the same on every kind of therapy (with one minor, possibly coincidental exception). Awkward. “Everybody has won and all must have prizes!”. If you’re an optimist, you can say all treatments work and everyone can keep doing whatever they like best. If you’re a pessimist, you might start wondering whether anything works at all.

By my understanding this is also the confusing conclusion of Ferri, Amato & Davoli (2006), the Cochrane Collaboration’s attempt to get in on the AA action. Like all Cochrane Collaboration studies since the beginning of time, they find there is insufficient evidence to demonstrate the effectiveness of the intervention being investigated. This has been oft-quoted in the anti-AA literature. But by my reading, they had no control groups and were comparing AA to different types of treatment:

Three studies compared AA combined with other interventions against other treatments and found few differences in the amount of drinks and percentage of drinking days. Severity of addiction and drinking consequence did not seem to be differentially influenced by TSF versus comparison treatment interventions, and no conclusive differences in treatment drop out rates were reported.

So the two best sources we have – Project MATCH and Cochrane – don’t find any significant differences between AA and other types of therapy. Now, to be fair, the inpatient treatment mentioned in Walsh et al wasn’t included, and inpatient treatment might be the gold standard here. But sticking to various forms of outpatient intervention, they all seem to be about the same.

So, the $64,000 question: do all of them work well, or do all of them work poorly?

V.

Alcoholism studies avoid control groups like they are on fire, presumably because it’s unethical not to give alcoholics treatment or something. However, there is one class of studies that doesn’t have that problem. These are the ones on “brief opportunistic intervention”, which is much like a turbocharged even shorter version of “motivational enhancement therapy”. Your doctor tells you ‘HELLO HAVE YOU CONSIDERED QUITTING ALCOHOL??!!’ and sees what happens.

Brief opportunistic intervention is the most trollish medical intervention ever, because here are all these brilliant psychologists and counselors trying to unravel the deepest mysteries of the human psyche in order to convince people to stop drinking, and then someone comes along and asks “Hey, have you tried just asking them politely?”. And it works.

Not consistently. But it works for about one in eight people. And the theory is that since it only takes a minute or two of a doctor’s time, it scales a lot faster than some sort of hideously complex hospital-based program that takes thousands of dollars and dozens of hours from everyone involved. If doctors would just spend five minutes with each alcoholic patient reminding them that no, really, alcoholism is really bad, we could cut the alcoholism rate by 1/8.

(this also works for smoking, by the way. I do this with every single one of my outpatients who smoke, and most of the time they roll their eyes, because their doctor is giving them that speech, but every so often one of them tells me that yeah, I’m right, they know they really should quit smoking and they’ll give it another try. I have never saved anyone’s life by dramatically removing their appendix at the last possible moment, but I have gotten enough patients to promise me they’ll try quitting smoking that I think I’ve saved at least one life just by obsessively doing brief interventions every chance I get. This is probably the most effective life-saving thing you can do as a doctor, enough so that if you understand it you may be licensed to ignore 80,000 Hours’ arguments on doctor replaceability)

Anyway, for some reason, it’s okay to do these studies with control groups. And they are so fast and easy to study that everyone studies them all the time. A meta-analysis of 19 studies is unequivocal that they definitely work.

Why do these work? My guess is that they do two things. First, they hit people who honestly didn’t realize they had a problem, and inform them that they do. Second, the doctor usually says they’ll “follow up on how they’re doing” the next appointment. This means that a respected authority figure is suddenly monitoring their drinking and will glare at them if they stay they’re still alcoholic. As someone who has gone into a panic because he has a dentist’s appointment in a week and he hasn’t been flossing enough – and then flossed until his teeth were bloody so the dentist wouldn’t be disappointed – I can sympathize with this.

But for our purposes, the brief opportunistic intervention sets a lower bound. It says “Here’s a really minimal thing that seems to work. Do other things work better than this?”

The “brief treatment” is the next step up from brief intervention. It’s an hour-or-so-long session (or sometimes a couple such sessions) with a doctor or counselor where they tell you some tips for staying off alcohol. I bring it up here because the brief treatment research community spends its time doing studies that show that brief treatments are just as good as much more intense treatments. This might be most comparable to the “motivational enhancement therapy” in the MATCH study.

Chapman and Huygens (1988) find that a single interview with a health professional is just as good as six weeks of inpatient treatment (I don’t know about their hospital in New Zealand, but for reference six weeks of inpatient treatment in my hospital costs about $40,000.)

Edwards (1977) finds that in a trial comparing “conventional inpatient or outpatient treatment complete with the full panoply of services available at a leading psychiatric institution and lasting several months” versus an hour with a doc, both groups do the same at one and two year followup.

And so on.

All of this is starting to make my head hurt, but it’s a familiar sort of hurt. It’s the way my head hurts when Scott Aaronson talks about complexity classes. We have all of these different categories of things, and some of them are the same as others and others are bigger than others but we’re not sure exactly where all of them stand.

We have classes “no treatment”, “brief opportunistic intervention”, “brief treatment”, “Alcoholics Anonymous”, “psychotherapy”, and “inpatient”.

We can prove that BOI > NT, and that AA = PT. Also that BT = IP = PT. We also have that IP > AA, which unfortunately we can use to prove a contradiction, so let’s throw it out for now.

So the hierarchy of classes seems to be (NT) < (BOI) ? (BT, IP, AA, PT) - in other words, no treatment is the worst, brief opportunistic intervention is better, and then somewhere in there we have this class of everything else that is the same.

Can we prove that BOI = BT?

We have some good evidence for this, once again from our Handbook. A study in Edinburgh finds that five minutes of psychiatrist advice (brief opportunistic intervention) does the same as sixty minutes of advice plus motivational interviewing (brief treatment).

So if we take all this seriously, then it looks like every psychosocial treatment (including brief opportunistic intervention) is the same, and all are better than no treatment. This is a common finding in psychiatry and psychology – for example, all common antidepressants are better than no treatment but work about equally well; all psychotherapies are better than no treatment but work about equally well, et cetera. It’s still an open question what this says about our science and our medicine.

The strongest counterexample to this is Walsh et al which finds the inpatient hospital stay works better than the AA referral, but this study looks kind of lonely compared to the evidence on the other side. And even the authors admit they were surprised by the effectiveness of the hospital there.

And let’s go back to Project MATCH. There wasn’t a control group. But there were the people who dropped out of the study, who said they’d go to AA or psychotherapy but never got around to it. Cutter and Fishbain (2005) take a look at what happened to these folks. They find that the dropouts did 75% as well as the people in any of the therapy groups, and that most of the effect of the therapy groups occurred in the first week (ie people dropped out after one week did about 95% as well as people who stayed in).

To me this suggests two things. First, therapy is only a little helpful over most people quitting on their own. Second, insofar as therapy is helpful, the tiniest brush with therapy is enough to make someone think “Okay, I’ve had some therapy, I’ll be better now”. Just like with the brief opportunistic interventions, five minutes of almost anything is enough.

This is a weird conclusion, but I think it’s the one supported by the data.

VI.

I should include a brief word about this giant table.

I see it everywhere. It looks very authoritative and impressive and, of course, giant. I believe the source is Miller’s Handbook of Alcoholism Treatment Approaches: Effective Alternatives, 3rd Edition, the author of which is known as a very careful scholar whom I cannot help but respect.

And the table does a good thing in discussing medications like acamprosate and naltrexone, which are very important and effective interventions but which will not otherwise be showing up in this post.

However, the therapy part of the table looks really wrong to me.

First of all, I notice acupuncture is ranked 17 out of 48, putting in a much, much better showing than treatments like psychotherapy, counseling, or education. Seems fishy.

Second of all, I notice that motivational enhancement (#2), cognitive therapy (#13), and twelve-step (#37) are all about as far apart as could be, but the largest and most powerful trial ever, Project MATCH, found all three to be about equal in effectiveness.

Third of all, I notice that cognitive therapy is at #13, but psychotherapy is at #46. But cognitive therapy is a kind of psychotherapy.

Fourth of all, I notice that brief interventions, motivational enhancement, confrontational counseling, psychotherapy, general alcoholism counseling, and education are all over. But a lot of these are hard to differentiate from one another.

The table seems messed up to me. Part of it is because it is about evidence base rather than effectiveness (consider that handguns have a stronger evidence base than the atomic bomb, since they have been used many more times in much better controlled conditions, but the atomic bomb is more effective) and therefore acupuncture, which is poorly studied, can rank quite high compared to things which have even one negative study.

But part of it just seems wrong. I haven’t read the full book, but I blame the tendency to conflate studies showing “X does not work better than anything else” with “X does not work”.

Remember, whenever there are meta-analyses that contradict single very large well-run studies, go with the single very large well-run study, especially when the meta-analysis is as weird as this one. Project MATCH is the single very large well-run study, and it says this is balderdash. I’m guessing it’s trying to use some weird algorithmic methodology to automatically rate and judge each study, but that’s no substitute for careful human review.

VII.

In conclusion, as best I can tell – and it is not very well, because the studies that could really prove anything robustly haven’t been done – most alcoholics get better on their own. All treatments for alcoholism, including Alcoholics Anonymous, psychotherapy, and just a few minutes with a doctor explaining why she thinks you need to quit, increase this already-high chance of recovery a small but nonzero amount. Furthermore, they are equally effective after only a tiny dose: your first couple of meetings, your first therapy session. Some studies suggest that inpatient treatment with outpatient followup may be better than outpatient treatment alone, but other studies contradict this and I am not confident in the assumption.

So does Alcoholics Anonymous work? Though I cannot say anything authoritatively, my impression is: Yes, but only a tiny bit, and for many people five minutes with a doctor may work just as well as years completing the twelve steps. As such, individual alcoholics may want to consider attending if they don’t have easier options; doctors might be better off just talking to their patients themselves.

If this is true – and right now I don’t have much confidence that it is, it’s just a direction that weak and contradictory data are pointing – it would be really awkward for the multibazillion-dollar treatment industry.

More worrying, I am afraid of what it would do to the War On Drugs. Right now one of the rallying cries for the anti-Drug-War movement is “treatment, not prison”. And although I haven’t looked seriously at the data for any drug besides alcohol. I think some data there are similar. There’s very good medication for drugs – for example methadone and suboxone for opiate abuse – but in terms of psychotherapy it’s mostly the same stuff you get for alcohol. Rehabs, whether they work or not, seem to serve an important sort of ritual function, where if you can send a drug abuser to a rehab you at least feel like something has been done. Deny people that ritual, and it might make prison the only politically acceptable option.

In terms of things to actually treat alcoholism, I remain enamoured of the Sinclair Method, which has done crazy outrageous stuff like conduct an experiment with an actual control group. But I haven’t investigated enough to know whether my early excitement about them looks likely to pan out or not.

I would not recommend quitting any form of alcohol treatment that works for you, or refusing to try a form of treatment your doctor recommends, based on any of this information.

In one of them, I was informed that America is medicalizing normal childhood mischief and loading anyone who gets worse than a B+ up with Ritalin or amphetamines as part of the pathologization of everyday life.

In another, I was informed that ADHD is shamefully underdiagnosed and most of the children who need stimulants most are going without them and failing school unnecessarily, so we need better screening programs and more efforts to seek out potential sufferers of the condition.

So I asked one of my attendings, Dr. L, which one it was. Are we overdosing ADHD? Or underdiagnosing it?

He answered that we are both overdiagnosing and underdiagnosing ADHD, the same as every other psychiatric disease, and then explained this so it made perfect sense and I was embarassed for not realizing it before.

Suppose that 3% of the population has ADHD.

Suppose that of people with ADHD, 50% of them realize they have ADHD like symptoms and go to a psychiatrist to get checked out.

Suppose that of people without ADHD, 10% of them falsely believe they have ADHD and also go to a psychiatrist to get checked out.

The Conners Continuous Performance Test is a commonly used test that evaluates children for ADHD. It is found to have a sensitivity of 75% and a specificity of 73%. In theory our system is based on faith that a trained psychiatrist can do better than a neuropsychological test; in practice they probably do much worse. Let’s give them the benefit of the doubt and say this is an excellent psychiatrist who outperforms the test handily and has both a sensitivity and specificity of 85%.

We can see that of every 100 people, 3 will have ADHD and 97 won’t. 1.5 true patients and 9.7 false patients will show up for psychiatric evaluation. The psychiatrist will diagnose 1.275 true patients and 1.455 false patients with the condition, and prescribes stimulants according to the diagnosis.

So we have three things that, surprisingly, all happen at once:

1. We have an excellent psychiatrist who outperforms the tests and is right 85% of the time.
2. The majority of people who are on Ritalin, shouldn’t be.
3. The majority of people who should be on Ritalin, aren’t.

Number two sounds a lot like what we mean by “overdiagnosis”, and number three sounds a lot like what we mean by “underdiagnosis”. So even with a pretty good psychiatrist acting honestly, we expect ADHD to be both overdiagnosed and underdiagnosed at the same time.

Even in conditions that do not quite satisfy the “majority” part of (2) and (3), we might still expect it to be true at the same time that a sizeable chunk of people diagnosed with the disease don’t have it and a sizeable chunk of people with the disease aren’t diagnosed.

If this seems counterintuitive, it is just another example of the annoying world of medical sensitivity and specificity statistics, which are constantly tripping up even the most experienced doctors. See also the infamous mammogram problem.

Once I understood this joint-overdiagnosis-and-underdiagnosis problem, several other candidate situations immediately leapt to mind. Antidepressants are almost certainly both overprescribed and underprescribed. So are opiate pain medications.

Not all the relevant examples are medical. I was reminded of Athrelon’s recent attempts to explain to me his version of the far-right concept of anarcho-tyranny. At first this didn’t make sense to me – how could there be anarchy and tyranny at the same time? Athrelon was able to walk me through the logic, which it turns out is the exact same as above. Imagine the government as trying to “diagnose” the situations where it needs to use force, and over- and under- diagnosing them at the same time. He will make this into a blog post soon, and I will link you to it.

Athrelon is a doctor. This may or may not be a coincidence. Sensitivity and specificity statistics are weird.