I.

Which is worse – ruining ten million people’s sex lives for one year, or making one hundred people’s livers explode?

I admit I sometimes use this blog to speculate about silly moral dilemmas for no reason, but that’s not what’s happening here. This is a real question that I deal with on a daily basis.

SSRIs, the class which includes most currently used antidepressants, are very safe in the traditional sense of “unlikely to kill you”. Suicidal people take massive overdoses of SSRIs all the time, and usually end up with little more than a stomachache for their troubles. On the other hand, there’s increasing awareness of very common side effects which, while not disabling, can be pretty unpleasant. About 50% of users report decreased sexual abilities, sometimes to the point of total loss of libido or anorgasmia. And something like 25% of users experience “emotional blunting” and the loss of ability to feel feelings normally.

Nefazodone (brand name Serzone®, which would also be a good brand name for a BDSM nightclub) is an equally good (and maybe better) antidepressant that does not have these side effects. On the other hand, every year, one in every 300,000 people using nefazodone will go into “fulminant hepatic failure”, which means their liver suddenly and spectacularly stops working and they need a liver transplant or else they die.

There are a lot of drug rating sites, but the biggest is Drugs.com. 467 Drugs.com users have given Celexa, a very typical SSRI, an average rating of 7.8/10. 14 users have given nefazodone an average rating of 9.1/10.

CrazyMeds might not be as dignified as Drugs.com, but they have a big and well-educated user base and they’re psych-specific. Their numbers are 3.3/5 (n = 253) for Celexa and 4.1/5 (n = 47) for nefazodone.

So both sites’ users seem to agree that nefazodone is notably better than Celexa, in terms of a combined measure of effectiveness and side effects.

But nefazodone is practically never used. It’s actually illegal in most countries. In the United States, parent company Bristol-Myers Squibb (which differs from normal Bristol-Myers in that it was born without innate magical ability) withdrew it from the market, and the only way you can find it nowadays is to get it is from an Israeli company that grabbed the molecule after it went off-patent. In several years working in psychiatry, I have never seen a patient on nefazodone, although I’m sure they exist somewhere. I would estimate its prescription numbers are about 1% of Celexa’s, if that.

The problem is the hepatic side effects. Nobody wants to have their liver explode.

But. There are something like thirty million people in the US on antidepressants. If we put them all on nefazodone, that’s about a hundred cooked livers per year. If we put them all on SSRIs, at least ten million of them will get sexual side effects, plus some emotional blunting.

My life vastly improved when I learned there was a searchable database of QALYs for different conditions. It doesn’t have SSRI-induced sexual dysfunction, but it does have sexual dysfunction due to prostate cancer treatment, and I assume that sexual dysfunction is about equally bad regardless of what causes it. Their sexual dysfunction has some QALY weights averaging about 0.85. Hm.

Assume everyone with fulminant liver failure dies. That’s not true; some get liver transplants, maybe some even get a miracle and recover. But assume everyone dies – and further, they die at age 30, cutting their lives short by fifty years.

In that case, putting all depressed people on nefazodone for a year costs 5,000 QALYs, but putting all depressed people on SSRIs for a year costs 1,500,000 QALYs. The liver failures may be flashier, but the 3^^^3 dust specks worth of poor sex lives add up to more disutility in the end.

I don’t want to overemphasize this particular calculation for a couple of reasons. First, SSRIs and nefazodone both have other side effects besides the major ones I’ve focused on here. Second, I don’t know if the level of SSRI-induced sexual dysfunction is as bad as the prostate-surgery-induced sexual dysfunction on the database. Third, there are a whole bunch of antidepressants that are neither SSRIs nor nefazodone and which might be safer than either.

But I do want to emphasize this pattern, because it recurs again and again.

II.

In that spirit, which would you rather have – something like a million people addicted to amphetamines, or something like ten people have their skin eat itself from the inside?

I can’t get good numbers on how many adults abuse Adderall, but a quick glance at the roster for my hospital’s rehab unit suggests “a lot”. Huffington Post calls it the most abused prescription drug in America, which sounds about right to me. Honestly there are worse things to be addicted to than Adderall, but it’s not completely without side effects. The obvious ones are anxiety, irritability, occasionally frank psychosis, and sometimes heart problems – but a lot of the doctors I work with go beyond what the research can really prove and suggest it can produce lasting negative personality change and predispose people to other forms of addictive and impulsive behavior.

If you’ve got to give adults a stimulant, I would much prefer modafinil. It’s not addictive, it lacks most of Adderall’s side effects, and it works pretty well. I’ve known many people on modafinil and they give it pretty universally positive reviews.

On the other hand, modafinil may or may not cause a skin reaction called Stevens Johnson Syndrome/Toxic Epidermal Necrolysis, which like most things with both “toxic” and “necro” in the name is really really bad. The original data suggesting a connection came from kids, who get all sorts of weird drug effects that adults don’t, but since then some people have claimed to have found a connection with adults. Some people get SJS anyway just by bad luck, or because they’re taking other drugs, so it’s really hard to attribute cases specifically to modafinil.

Gwern’s Modafinil FAQ mentions an FDA publication which argues that the background rate of SJS/TEN is 1-2 per million people per year, but the modafinil rate is about 6 per million people per year. However, there are only three known cases of a person above age 18 on modafinil getting SJS/TEN, and this might not be different from background rates after all. Overall the evidence that modafinil increases the rate of SJS/TEN in adults at all is pretty thin, and if it does, it’s as rare as hen’s teeth (in fact, very close to the same rate as liver failure from nefazodone).

(also: consider that like half of Silicon Valley is on modafinil, yet San Francisco Bay is not yet running red with blood.)

(also: ibuprofen is linked to SJS/TEN, with about the same odds ratio as modafinil, but nobody cares, and they are correct not to care.)

I said I’ve never seen a doctor prescribe nefazodone in real life; I can’t say that about modafinil. I have seen one doctor prescribe modafinil. It happened like this: a doctor I was working with was very upset, because she had an elderly patient with very low energy for some reason, I can’t remember, maybe a stroke, and wanted to give him Adderall, but he had a heart arrythmia and Adderall probably wouldn’t be safe for him.

I asked “What about modafinil?”

She said, “Modafinil? Really? But doesn’t that sometimes cause Stevens Johnson Syndrome?”

And then I glared at her until she gave in and prescribed it.

But this is very, very typical. Doctors who give out Adderall like candy have no associations with modafinil except “that thing that sometimes causes Stevens-Johnson Syndrome” and are afraid to give it to people.

III.

Nefazodone and modafinil are far from the only examples of this pattern. MAOIs are like this too. So is clozapine. If I knew more about things other than psychiatry, I bet I could think of examples from other fields of medicine.

And partially this is natural and understandable. Doctors swear an oath to “first do no harm”, and toxic epidermal necrolysis is pretty much the epitome of harm. Thought experiments like torture vs dust specks suggest that most people’s moral intuitions say that no amount of aggregated lesser harms like sexual side effects and amphetamine addictions can equal the importance of avoiding even a tiny chance of some great harm like liver failure or SJS/TEN. Maybe your doctor, if you asked her directly, would endorse a principled stance of “I am happy to give any number of people anxiety and irritability in order to avoid even the smallest chance of one case of toxic epidermal necrolysis.”

And yet.

The same doctors who would never dare give nefazodone, consider Seroquel a perfectly acceptable second-line treatment for depression. Along with other atypical antipsychotics, Seroquel raises the risk of sudden cardiac death by about 50%. The normal risk of cardiac sudden death in young people is about 10 in 100,000 per year, so if my calculations are right, low-dose Seroquel causes an extra cardiac death once per every 20,000 patient-years. That’s ten times as often as nefazodone causes an extra liver death.

Yet nefazodone was taken off of the market by its creators and consigned to the dustbin of pharmacological history, and Seroquel is the sixth-best-selling drug in the United States, commonly given for depression, simple anxiety, and sometimes even to help people sleep.

Why the disconnect? Here’s a theory: sudden cardiac death happens all the time; sometimes God just has it in for you and your heart stops working and you die. Antipsychotics can increase the chances of that happening, but it’s a purely statistical increase, such that we can detect it aggregated over large groups but never be sure that it played a role in any particular case. The average person who dies of Seroquel never knows they died of Seroquel, but the average person who dies from nefazodone is easily identified as a nefazodone-related death. So nefazodone gets these big stories in the media about this young person who died by taking this exotic psychiatric drug, and it becomes a big deal and scares the heck out of everybody. When someone dies of Seroquel, it’s just an “oh, so sad, I guess his time has come.”

But the end result is this. When treatment with an SSRI fails, nefazodone and Seroquel naively seem to be equally good alternatives. Except nefazodone has a death rate of 1/300,000 patient years, and Seroquel 1/20,000 patient years. And yet everyone stays the hell away from the nefazodone because it’s known to be unsafe, and chooses the Seroquel.

I conclude either doctors are terrible at thinking about risk, or else maybe a little too good at thinking about risk.

I bring up the latter option because there’s a principal-agent problem going on here. Doctors want to do what’s best for their patients. But they also want to do what’s best for themselves, which means not getting sued. No one has ever sued their doctor because they got a sexual side effect from SSRIs, but if somebody dies because they’re the lucky 1/300,000 who gets liver failure from nefazodone, you can bet their family’s going to sue. Suddenly it’s not a matter of comparing QALYs, it’s a matter of comparing zero percent chance of lawsuit with non-zero percent chance of lawsuit.

(Fermi calculation: if a doctor has 100 patients at a time on antidepressants, and works for 30 years, then if she uses Serzone as her go-to antidepressant, she’s risking a 1% chance of getting the liver failure side effect once in her career. That’s small, but since a single bad lawsuit can bankrupt a doctor, it’s worth taking seriously.)

And that would be a tough lawsuit to fight. “Yes, Your Honor, I knew when I prescribed this drug that it sometimes makes people’s livers explode, but the alternative often gives people a bad sex life, and according to the theory of utilitarianism as propounded by 18th century philosopher Jeremy Bentham – ” … “Bailiff, club this man”.

And the same facet of nefazodone that makes it exciting for the media makes it exciting for lawsuits. When someone dies of nefazodone toxicity, everyone knows. When someone dies of Seroquel, “oh, so sad, I guess his time has come”.

That makes Seroquel a lot safer than nefazodone. Safer for the doctor, I mean. The important kind of safer.

This is why, as I mentioned before, I hate lawsuits as a de facto regulatory mechanism. Our de jure regulatory mechanism, the FDA, is pretty terrible, but to its credit it hasn’t banned nefazodone. One time it banned clozapine because of a flashy rare side effect, but everyone yelled at them and they apologized and changed their mind. With lawsuits there’s nobody to yell at, so we just end up with people very quietly adjusting their decisions in the shadows and nobody else being any the wiser.

I don’t want to overemphasize this, because I think it’s only one small part of the problem. After all, a lot of countries withdrew nefazodone entirely and didn’t even give lawsuits a chance to enter the picture.

But whatever the cause, the end result is that drugs with rare but spectacular side effects get consistently underprescribed relative to drugs with common but merely annoying side effects, or drugs that have more side effects but manage to hide them better.

The two doctors leading the team are both very experienced and have kind of seen it all, so it’s interesting to get a perspective on drug issues from people on the front line. In particular, one of my attendings is an Obama-loving long-haired hippie who nevertheless vehemently opposes medical marijuana or any relaxation on marijuana’s status at all. He says that “just because I’m a Democrat doesn’t mean I have to support stupid policies I know are wrong” and he’s able to back up his opinion with an impressive variety of studies.

To be honest, I had kind of forgotten that the Universe was allowed to contain negative consequences for legalizing drugs. What with all the mental energy it took protesting the the Drug War and getting outraged at police brutality and celebrating Colorado’s recently permitting recreational cannabis use and so on, it had completely slipped my mind that the legalization of marijuana might have negative consequences and that I couldn’t reject it out of hand until I had done some research.

So I’ve been doing the research. Not to try to convince my attending of anything – as the old saying goes, do not meddle in the affairs of attendings, because you are crunchy and taste good with ketchup – but just to figure out where exactly things stand.

I. Would Relaxation Of Penalties On Marijuana Increase Marijuana Use?

Starting in the 1970s, several states decriminalized possession of marijuana – that is, possession could not be penalized by jail time. It could still be penalized by fines and other smaller penalties, and manufacture and sale could still be punished by jail time.

Starting in the 1990s, several states legalized medical marijuana. People with medical marijuana cards, which in many cases were laughably easy to get with or without good evidence of disease, were allowed to grow and use marijuana, despite concerns that some of this would end up on the illegal market.

Starting last week, Colorado legalized recreational use of marijuana, as well as cultivation and sale (subject to heavy regulations). Washington will follow later this year, and other states will be placing measures on their ballots to do the same.

One should be able to evaluate to what degree marijuana use rose after these policy changes, and indeed, many people have tried – with greater or lesser levels of statistical sophistication.

The worst arguments in favor of this proposition are those like this CADCA paper, which note that states with more liberal marijuana laws have higher rates of marijuana use among teenagers than states that do not. The proper counterspell to such nonsense is Reverse Causal Arrows – could it not be that states with more marijuana users are more likely to pass proposals liberalizing marijuana laws? Yes it could. Even more likely, some third variable – let’s call it “hippie attitudes” – could be behind both high rates of marijuana use and support for liberal marijuana regimes. The states involved are places like Colorado, California, Washington, and Oregon. I think that speaks for itself. In case it doesn’t, someone went through the statistics and found that these states had the highest rates of marijuana use among teens since well before they relaxed drug-related punishments. Argument successfully debunked.

A slightly more sophisticated version – used by the DEA here – takes the teenage marijuana use in a state one year before legalization of medical marijuana and compares it to the teenage marijuana use in a state one (or several years) after such legalization. They often find that it has increased, and blame the increase on the new laws. For example, 28% of Californians used marijuana before it was decriminalized in the 70s, compared to 35% a few years after. This falls victim to a different confounder – marijuana use has undergone some very large swings nationwide, so the rate of increase in medical marijuana states may be the same as the rate anywhere else. Indeed, this is what was going on in California – its marijuana use actually rose slightly less than the national average.

What we want is a study that compares the average marijuana use in a set of states before liberalization to the average marijuana use in the country as a whole, and then does the same after liberalization to see if the ratio has increased. There are several studies that purport to try this, of which by far the best is Johnston, O’Malley & Bachman 1981, which monitored the effect of the decriminalization campaigns of the 70s. They survey thousand of high school seniors on marijuana use in seven states that decriminalize marijuana both before and for five years after the decriminalization, and find absolutely no sign of increased marijuana use (in fact, there is a negative trend). Several other studies (eg Thies & Register 1993) confirm this finding.

There is only a hint of some different results. Saffer and Chaloukpa 1999 and Chaloupka, Grossman & Tauras 1999 try to use complicated econometric simulations to estimate the way marijuana demand will respond to different variables. They simulate (as opposed to detecting in real evidence) that marijuana decriminalization should raise past-year use by about 5 – 8%, but have no effect on more frequent use (ie a few more people try it but do not become regular users). More impressively, Model 1993 (a source of some exasperation for me earlier) finds that after decriminalization, marijuana-related emergency room visits went up (trying to interpret their tables, I think they went up by a whopping 90%, but I’m not sure of this). This is sufficiently different from every other study that I don’t give it much weight, although we’ll return to it later.

Overall I think the evidence is pretty strong that decriminalization probably led to no increase in marijuana use among teens, and may at most have led to a small single-digit increase.

Proponents of stricter marijuana penalties say the experiment isn’t fair. In practice, decriminalization does not affect the average user very much – even in states without decriminalization, marijuana possession very rarely leads to jail time. The only hard number I have is from Australia, where in “non-decriminalized” Australian states only 0.3% of marijuana arrests lead to jail time, but a quick back-of-the-envelope calculation suggests US numbers are very similar. And even in supposedly decriminalized states, it’s not hard for a cop who wants to get a pot user in jail to find a way (possession of even small amounts can be “possession with intent to sell” if someone doesn’t like you). So the overall real difference between decriminalized and not decriminalized is small and it’s not surprising the results are small as well. I mostly agree with them; decriminalization is fine as far as it goes, but it’s a bigger psychological step than an actual one.

The next major milestone in cannabis history was the legalization of medical marijuana. Anderson, Hansen & Rees (2012) did the same kind of study we have seen above, and despite trying multiple different measures of youth marijuana use found pretty much no evidence that medical marijuana legalization caused it to increase. Other studies find pretty much the same.

This could potentially suffer from the same problems as decriminalization studies – the laws don’t always change the facts on the ground. Indeed, for about ten years after medical marijuana legalization, the federal government kept on prosecuting marijuana users even when their use accorded with state laws, and many states had so few dispensaries that in reality not a whole lot of medical marijuana was being given out. I haven’t found any great studies that purport to overcome these problems.

When we examined decriminalization, we found that the studies based on surveys of teens looked pretty good, but that the one study that examined outcomes – marijuana-related ER visits – was a lot less encouraging. We find the same pattern here, and the rain on our parade is Chu 2013, who finds that medical marijuana laws increased marijuana-related arrests by 15-20% and marijuana-related drug rehab admissions by 10-15%.

So what’s going on here? I have two theories. First, maybe medical marijuana use (and decriminalization) increase use among adults only. This could be because the system is working – giving adults access to medical marijuana while keeping it out of the hands of children – or because kids are dumb and don’t understand consequences but adults are more responsive to incentives and punishments. Second, we know that medical marijuana has twice as much THC as street marijuana. Maybe everyone keeps using the same amount of marijuana, but when medical marijuana inevitably gets diverted to the street, addicts can’t handle it and end up behaving much worse than they expected.

Or the studies are wrong. Studies being wrong is always a pretty good bet.

I can’t close this section without mentioning the Colorado expulsion controversy. Nearly everyone who teaches in Colorado says there has been an explosion of marijuana-related problems since medical marijuana was legalized. Meanwhile, the actual surveys of Colorado high school students say that marijuana use, if anything, is going down. A Colorado drug warrior has some strong objections to the survey results, but they center around not really being able to prove that there is a real downward trend (which is an entirely correct complaint) without denying that in fact they show no evidence at all of going up.

The consensus on medical marijuana seems to be that it does not increase teen marijuana use either, although there is some murky and suggestive evidence that it might increase illicit or dangerous marijuana use among adults.

There is less information on the effects of full legalization of marijuana, which has never been tried before in the United States. To make even wild guesses we will have to look at a few foreign countries plus some econometric simulations.

No one will be surprised to hear that the first foreign country involved is the Netherlands, which was famously permissive of cannabis up until a crackdown a few years ago. Despite popular belief they never fully legalized the drug and they were still pretty harsh on production and manufacture; distribution, on the other hand, could occur semi-openly in coffee shops. This is another case where we have to be careful to distinguish legal regimes from actual effects, but during the period when there were actually a lot of pot-serving coffee shops, the Netherlands did experience an otherwise-inexplicable 35% rise in marijuana consumption relative to the rest of Europe. This is true even among teenagers, and covers both heavy use as well as occasional experimentation. Some scientists studying the Netherlands’ example expect Colorado to see a similar rise; others think it will be even larger because the legalization is complete rather than partial.

The second foreign country involved is Portugal, which was maybe more of a decriminalization than a legalization case but which is forever linked with the idea of lax drug regimes in the minds of most Americans. They decriminalized all drugs (including heroin and cocaine) in 2001, choosing to replace punishment with increased treatment opportunities, and as we all have been told, no one in Portugal ever used drugs ever again, or even remembers that drugs exist. Except it turns out it’s more complicated; for example, the percent of Portuguese who admit to lifetime use of drugs has doubled since the law took effect. Two very patient scientists have sifted through all the conflicting claims and found that in reality, the number of people who briefly experiment with drugs has gone way up, but the number of addicts hasn’t, nor has the number of bad outcomes like overdose-related deaths. There are many more people receiving drug treatment, but that might just be because Portugal upped its drug treatment game in a separate law at the same time they decriminalized drugs. Overall they seem to have been a modest success – neither really raising nor decreasing the number of addicts – but they seem more related to decriminalization (which we’ve already determined doesn’t have much effect) than to legalization per se.

Returning to America, what if you just ask people whether they would use more marijuana if it’s legal? Coloradans were asked if they plan to smoke marijuana once it becomes legal; comparing survey results to current usage numbers suggests 40% more users above the age of 18; it is unclear what the effect will be on younger teens and children.

Finally, we let the economists have their say. They crunch all the data and predict an increase of 50 – 100% based solely on the likely price drop (even with taxes factored in). And if there’s one group we can trust to make infallible predictions about the future, it’s economists.

Overall I find the Dutch evidence most convincing, and predict a 25 – 50% increase in adult marijuana use with legalization. I would expect a lower increase – 15 – 30% – among youth, but the data are also perfectly consistent with no increase at all.

Conclusion for this section: that decriminalization and legalization of medical marijuana do not increase youth marijuana use rates, although there is some shaky and indirect evidence they do increase adult use and bad behavior. There is no good data yet on full legalization, but there’s good reason to think it would substantially increase adult use and it might also increase youth use somewhat.

II. Is Marijuana Bad For You?

About 9% of marijuana users eventually become addicted to the drug, exposing them to various potential side effects.

Marijuana smoke contains a lot of the same chemicals in tobacco smoke and so it would not be at all surprising if it had some of the same ill effects, like cardiovascular disease and lung cancer. But when people look for these effects, they can’t find any increase in mortality among marijuana smokers. I predict that larger studies will one day pick something up, but for now let’s take this at face value.

Much more concerning are the attempts to link marijuana to cognitive and psychiatric side effects. Meier et al (2012) analyzed a study of a thousand people in New Zealand and found that heavy marijuana use was linked to an IQ decline of 8 points. Rogeberg 2012 developed an alternative explanation – poor people saw their IQs drop in their 20s more than rich people because their IQs had been artificially inflated by schooling; what Meier et al had thought to be an effect of cannabis was really an effect of poor people having an apparent IQ drop and using cannabis more often. Meier et al pointed out that actually, poor people didn’t use cannabis any more often than anyone else and effects remained when controlled for class. Other studies, like Fried et al (2002) find the same effect, and there is a plausible biological mechanism (cannabinoids something something neurotransmitters something brain maturation). As far as I can tell the finding still seems legit, and marijuana use does decrease IQ. It is still unclear whether this only applies in teenagers (who are undergoing a “sensitive period of brain development”) or full stop.

More serious still is the link with psychosis. A number of studies have found that marijuana use is heavily correlated with development of schizophrenia and related psychotic disorders later in life. Some of them find relative risks as high as 2 – heavy marijuana use doubles your chance of getting schizophrenia, which is already a moderately high 1%. But of course correlation is not causation, and many people have come up with alternative theories. For example, maybe people who are already kind of psychotic use marijuana to self-medicate, or just make poor life choices like starting drugs. Maybe people of low socioeconomic status who come from broken homes are more likely to both use marijuana and get schizophrenia. Maybe some gene both makes marijuana really pleasant and increases schizophrenia risk.

I know of three good studies attempting to tease out causation. Arseneault et al (2004) checks to see which came first – the marijuana use or the psychotic symptoms – and finds it was the marijuana use, thus supporting an increase in risk from the drug. Griffith-Lendering et al (2012) try the same, and find bidirectional causation – previous marijuana use seems to predict future psychosis, but previous psychosis seems to predict future marijuana use. A very new study from last month boxes clever and checks whether your marijuana use can predict schizophrenia in your relatives, and find that it does – presumably suggesting that genetic tendencies towards schizophrenia cause marijuana use and not vice versa (although Ozy points out to meet that the relatives of marijuana users are more likely to use marijuana themselves; the plot thickens). When a meta-analysis tries to control for all of these factors, they get a relative risk of 1.4 (they call it an odds ratio, but from their discussion section I think they mean relative risk).

Is this true, or just the confounders they failed to pick up? One argument for the latter is that marijuana use has increased very much over the past 50 years. If marijuana use caused schizophrenia, we would expect to see much more schizophrenia, but in fact as far as anyone can tell (which is not very far) schizophrenia incidence is decreasing. The decrease might be due (maybe! if it even exists at all!) to obstetric advances which prevent fetal brain damage which could later lead to the disease. The effect of this variable is insufficiently known to pretend we can tease out some supposed contrary effect of increased marijuana use. Also, some people say that schizophrenia is increasing in young people, so who knows?

The exact nature of the marijuana-psychosis link is still very controversial. Some people say that marijuana causes psychosis. Other people say it “activates latent psychosis”, a term without a very good meaning but which might mean that it pushes people on the borderline of psychosis – eg those with a strong family history but who might otherwise have escaped – over the edge. Still others say all it does is get people who would have developed psychosis eventually to develop it a few years earlier. You can read a comparison of all the different hypotheses here.

I’ve saved the most annoying for last: is marijuana a “gateway drug”? Would legalizing it make it more or less of a “gateway drug”? This claim seems tailor-made to torture statisticians. We know that marijuana users are definitely more likely to use other drugs later – for example, marijuana users are 85x more likely than non-marijuana users to use cocaine. but that could be either because marijuana affects them in some way (implying that legalizing marijuana would increase other drug use), because they have factors like genetics or stressful life situation that makes them more likely to use all drugs (implying that legalizing marijuana would not affect other drug use), or because using illegal marijuana without ill effect connects them to the illegal drug market and convinces them illegal drugs are okay (implying that legalizing marijuana would decrease other drug use). RAND comes very close to investigating this properly by saying that when the Dutch pseudo-legalized marijuana, use of harder drugs stayed stable or went down, but all their study actually shows is that the ratio of marijuana users : hard drug users went down. This is to be expected when you make marijuana much easier to get, but it’s still consistent with the absolute number of hard drug users going way up. The best that can be said is that there is no direct causal evidence for the gateway theory and some good alternative explanations for the effect. Let us accept their word for it and never speak of this matter again.

Conclusion for this section: Marijuana does not have a detectable effect on mortality and there is surprisingly scarce evidence of tobacco-like side effects. It probably does decrease IQ if used early and often, possibly by as many as 8 IQ points. It may increase risk of psychosis by as much as 40%, but it’s not clear who is at risk or whether the risk is even real. The gateway drug hypothesis is too complicated to evaluate effectively but there is no clear casual evidence in its support.

III. What Are The Costs Of The Drug War?

There are not really that many people in jail for using marijuana.

I learned this from Who’s Really In Prison For Marijuana?, a publication of the National Office Of Drug Control Policy, which was clearly written by someone with the same ability to take personal offense at bad statistics that inspires my posts about Facebook. The whole thing seethes with indignation and makes me want to hug the drug czar and tell him everything will be okay.

Only 1.6% of state prisoners are serving time for marijuana, only 0.7% are serving for marijuana possession, and only 0.3% are first time offenders. Some of those are “possession” in the sense of “possessing a warehouse full of marijuana bales”, and others are people who committed much more dangerous crimes but were nailed for marijuana, in the same sense that Al Capone was nailed for tax evasion. The percent of normal law-abiding people who just had a gram or two of marijuana and were thrown in jail is a rounding error, and the stories of such you read in the news are extremely dishonest (read the document for examples).

Federal numbers are even lower; in the entire federal prison system, they could only find 63 people imprisoned with marijuana possession as the sole crime, and those people were possessing a median of one hundred fifteen pounds of marijuana (enough to make over 100,000 joints).

In total, federal + state prison and counting all the kingpins, dealers, manufacturers, et cetera, there are probably about 16,000 people in prison solely for marijuana-related offenses, serving average actual sentence lengths of three year. But it’s anybody’s guess whether those people would be free today if marijuana were legal, or whether their drug cartels would just switch to something else.

Looking at the other side’s statistics, I don’t see much difference. NORML claims that there are 40,000 people in prison for marijuana use, but they admit that half of those people were arrested for using harder drugs and marijuana was a tack-on charge, so they seem to agree with the Feds about around 20,000 pure marijuana prisoners. SAM agrees that only 0.5% of the prison population is in there for marijuana possession alone. I see no reason to doubt any of these numbers.

A much more serious problem is marijuana-related arrests, of which there are 700,000 a year. 90% of them are for simple possession, and the vast majority do not end in prison terms; they do however result in criminal records, community service, a couple days of jail time until a judge is available to hear the case, heavy fines, high cost of legal representation, and moderate costs to the state for funding the whole thing. Fines can be up to $1500, and legal representation can cost up to $5000 (though I am suspicious of this paper and think it may be exaggerating for effect). These costs are often borne by poor people who will have to give up all their savings for years to pay them back.

Costs paid by the government, which cover everything from police officers to trials to prison time, are estimated at about $2 billion by multiple sources. This is only 3% of the total law enforcement budget, so legalizing marijuana wouldn’t create some kind of sudden revolution in policing, but as the saying goes, a billion here, a billion there, and eventually it adds up to real money. And a Harvard economist claims that the total monetary benefits from legalization, including potential tax revenues, could reach $14 billion.

Some people worry that legalizing marijuana would cause an increase in car accidents by “stoned drivers”, who, like drunk drivers, have impaired reflexes and poor judgment, and indeed there is a small but real problem of marijuana-induced car accidents. But Chaloukpa and Laixuthai (1994) crunch the numbers and find that decreased price/increased availability of marijuana is actually associated with decreased car accidents, probably because marijuana is substituting for alcohol in the “have impairing substances and then go driving” population. This finding – that marijuana and alcohol substitute for each other – has been spotted again and again. Anderson & Rees (2013) find that states that legalize medical marijuana see a 5% drop in beer sales. There are however a few dissenting opinions: Cameron & Williams (2001), in complex econometric simulations that may or may not resemble the real world in any respect, find that increasing the price of alcohol increases marijuana use, but increasing the price of marijuana does not affect alcohol use, and the same researcher finds that banning alcohol on a college campus also decreases marijuana use. Also, possibly marijuana use increases smoking? This whole area is confusing, but I am most sympathetic to to the Andersen and Rees statistics which say that medical marijuana states are associated with 13% fewer traffic fatalities.

Overall conclusion for this section: full legalization of marijuana would free about 20,000 people from jail (although most of them would not be exactly fine upstanding citizens), prevent 700,000 arrests not resulting in jail time per year, save between 2 and 14 billion dollars, and possibly reduce traffic fatalities a few percent (or, for all we know, increase them).

IV. An Irresponsible Utilitarian Analysis

Decriminalization and legalization of medical marijuana seem, if we are to trust the statistics in (I) saying they do not increase use among youth, like almost unalloyed good things. Although there are some nagging hints of doubt, they are not especially quantifiable and therefore not amenable to analysis. Without a very strong predisposition to try as hard as possible to fit the evidence into a pessimistic picture, I don’t think there’s a great argument against either of these two propositions. Let’s concentrate on legalization, which would mean something like “People can grow and sell as much marijuana as they want and it’s totally legal for people over 21, with the same level of penalties as today for people under 21″.

Section (I) concludes that legalization could lead to an increase in adult marijuana use up to 50%. There’s not a lot of evidence on what it could do to teen marijuana use, but since it seems teen marijuana use is less responsive to legal changes, I made up a number and said 20%. Lest you think I am being unfair, note that this is well below the percent increase predicted by the survey that asked 18 year olds if they would start using marijuana if it were legal.

Right now about 1.5 million teenagers use marijuana “heavily”. Most of the detrimental effects of marijuana seem concentrated in teens and people in their early twenties; I’m going to artificially round that up to 2 million to catch the early 20 year olds. If this 2 million number increased 20%, 400,000 extra teens would start heavily using marijuana.

Those 400,000 teens would lose 8 IQ points each. IQ increases your yearly earnings by about $500 per point, so these people would lose about $4,000 a year. Making very strong assumptions about salary being a measure of value to society, society would lose about $1.6 billion a year directly, plus various intangibles from potential artists and scientists losing the ability to create masterpieces and inventions, plus various really intangibles like a slightly dumber electorate.

We need to use a different number to calculate psychosis risk, since the studies were done on “people who had used marijuana at least once”. The appropriate number turns out to be 8 million teenagers; of those, 1%, or 80,000, would naturally develop schizophrenia. If the 1.4 relative risk number is correct, marijuana use will increase that to 112,000, for a total increase of 32,000 people. Schizophrenia pretty much always presents in the 15 – 25 age window, so we’ll say we get 3,200 extra cases per year.

There were 35000 road traffic accident fatalities in the US last year. If greater availability of marijuana decreases those fatalities by 13% (note that I am using the number from medical marijuana legalization and not for marijuana legalization per se, solely because it is a number I actually have), that will cause 4500 fewer road traffic deaths per year. There may be additional positive effects of alcohol substitution from, for example, less liver disease. But there may also be additional negative effects from increasing use of tobacco, so let’s just pretend those cancel out.

So here is my guess at the yearly results of marijuana legalization:

– 20,000 fewer prisoners (but they might switch to other criminal enterprises)
– 700,000 fewer arrests
– $2 billion less in law enforcement costs
– Some amount of positive gain (let’s say $5 billion) in taxes
– 4500 fewer road traffic deaths (if you believe the preliminary alcohol substitution numbers)

– 400,000 people with lower IQ
– $2 billion in social costs from above dumber people
– 3,200 more cases of schizophrenia a year

We’ll proceed to calculate the nonmonetary burden of each of these in QALYs, then add the monetary burden in dollars, then convert.

The searchable public database of utility weights for all diseases (God I love the 21st century) tells me that schizophrenia has a QALY weight of 0.73. It generally starts around 20 and lasts a lifetime, so each case of schizophrenia costs us 0.27 * 50 or 13.5 QALYs. Therefore, the total burden of the 3,200 added schizophrenia cases is 43 kiloQALYs.

There’s no good way to calculate the QALY weight of having 4-8 fewer IQ points, and unfortunately this is going to end up being among the most important numbers in our results. If we say the lifetime cost of this problem is 3 QALYs, and divide the number by eight to represent eight years worth of teenagers in our sample population, we end up with 400,000/8 * 3 = 150 kiloQALYs.

My own survey tells me that being in prison has a QALY weight around 0.5. Marijuana sentences generally last an average of three years, which suggests that 1/3 of these marijuana prisoners are arrested every year, so the total burden of the ~6000ish marijuana imprisonments each year is 3 * ~6000 * 0.5 = 10 kiloQALYs.

Assume the average road traffic death occurs at age 30, costing 40 years of potential future life. The total cost of 4500 road traffic deaths is 40 * 4500 = 180 kiloQALYs.

The arrests are going to require even more fudging than normal. Average jail time for a marijuana arrest (when awaiting trial) is “one to five days” – let’s round that off to two and then use our prison number to say that the jail from each arrest is 2/365 * 0.5 = three-thousandths of a QALY. I am going to arbitrarily round this up to one one-hundredth of a QALY to account for emotional trauma and the burden of fines, then even more arbitrarily round this up to a tenth of a QALY to account for possibility of getting a criminal record. This sets the burden of 700,000 arrests at 70 kiloQALYs.

Now our accounting is:

Costs from legalization compared to current system: 200 kQALYs and $2 billion
Benefits from legalization compared to current system: 260 kQALYs and $7 billion

Although it’s not going to be necessary, we can interconvert QALYs and dollars at the going health-care rate of about $100,000/QALY ($100 million/kQALY):

Costs from legalization compared to current system: 220 kQALYs
Benefits from legalization compared to current system: 330 kQALYs

And get:

Net benefits from legalization: +110 kQALYs

Except that this is extremely speculative and irresponsible. By far the largest component of the benefits of legalization turned out to be the effect on road traffic accidents, which is based on only two studies and which may on further research turn out to be a cost. And by far the largest component of the costs of legalization turned out to be the effect on IQ, and we had to totally-wild-guess the QALY cost of an IQ point loss. The wiggle room in my ignorance and assumptions is more than large enough to cover the small gap between the two policies in the results.

So my actual conclusion is:

There is not a sufficiently obvious order-of-magnitude difference between the costs and benefits of marijuana legalization for a evidence-based utilitarian analysis of costs and benefits to inform the debate. You may return to your regularly scheduled wild speculation and shrill accusations.

But I wouldn’t say this exercise is useless. For example, it suggests that whether marijuana legalization is positive or negative on net depends almost entirely on small changes in the road traffic accident rate. This is something I’ve never heard anyone else mention, but which in retrospect should be obvious; the few debatable health effects and the couple of people given short jail sentences absolutely can’t compare to the potential for thousands more (or fewer) traffic accidents which leave people permanently dead.

So my actual actual conclusion is:

We should probably stop caring about health effects of marijuana and about imprisonment for marijuana-related offenses, and concentrate all of our research and political energy on how marijuana affects driving.

This cements my previous intuitions on irresponsible use of statistics – it’s unlikely to unilaterally solve the problem, but it can be very good at pointing out where you’re being irrational and suggesting new ways of looking at a question.

EDIT: People in the comments have pointed out several important factors left out, including:
– Some people enjoy smoking marijuana
– The opening of a permanent criminal record may mean arrests are worse than I estimate. I can’t find good statistics on how often this happens, but do note that decriminalization prevents a record from being opened.
– Loss of 8 IQ points may have wider social effects than I estimate, since IQ affects for example crime rate.
– Legalizing marijuana might remove a source of funding for organized crime

Lately a bunch of my Facebook friends have been sharing the Slate article Military Strikes Are An Extremely Expensive Way To Help Foreigners, which is too bad because it’s super super wrong. It is wrong both in design and in execution, and it consigns itself to the special hell for people who make a numerical argument without checking any actual numbers.

Here is its argument: the United States is currently considering whether or not to intervene in Syria. Some people say there are humanitarian grounds for this – after all, Syrians are currently under the heel of a repressive regime and trapped in a deadly war. However, if we were true humanitarians, we could just take the cost of intervening in Syria – maybe a few billion dollars – and donate that money to GiveWell, which would save even more lives.

What I mean by “wrong in design” is that saying “this is less effective than GiveWell’s number one top rated efficient charity” is damning by really really faint praise. Tautologically, if you believe GiveWell then everything is less effective than GiveWell’s number one top rated efficient charity. This includes policies that you – and Matt Yglesias – support. For example, he says nice things about publicly subsidized preschools, but spending money on these preschools is less effective than GiveWell’s top-rated charity. The same is true of other nice things you presumably like such as public libraries, food stamps, PBS, et cetera.

So imagine that in the debate over whether to de-fund PBS a few years ago, someone had brought up that PBS is less cost-effective than GiveWell’s top charities, probably even less effective than its top US charities. And then we all patted ourselves on the back for noticing and decided not to renew funding for PBS. What do you think are the chances that all that funding would have ended up in the hands of poor African villagers?

If someone wants to propose a bill that our government can only spend money on the most effective things, starting with the most effective and only moving on to the second most effective once number one has been saturated, I’d be totally in support of that (and I’ve even tried imagining what such a government could look like). In the absence of such a policy, “let’s stop funding PBS since it’s less effective than GiveWell’s top-rated charity” is a terrible idea, and so is abandoning Syria to its fate for the same reason.

A better policy might be “defund things that produce less utility than tax cuts or the typical government policy”, since we can expect the money saved to either be refunded in taxes or to be distributed among other government policies. And this is where it becomes important that the Slate article was also wrong in its execution.

The article focuses on Libya as an example of an intervention which, although “successful”, was apparently “too expensive”. Let’s calculate how expensive it really was.

The US spent about $1 billion on military intervention in Libya (our allies contributed some more, but we can assume that US participation “bought” allied participation).

There’s no good way of calculating how many lives US intervention saved, but the war up to that point had caused 25,000 casualties, and everyone expected the rebels’ final defeat to be something of a bloodbath. Let’s say intervention prevented another 25,000 casualties.

Suppose the average Libyan currently alive can be expected to live 25 more years. We multiply by 0.7 (for reasons to be explained later) and so those 25,000 lives saved = a gain of 437,500 QALYs.

However, intervention did more than save potential casualties. It also freed everyone from the government of Moammar Gaddafi, a ridiculously evil guy who squandered the country’s wealth and raped his populace both figuratively and literally.

How much should this count for? When I tried to elicit conditional utility weights from people, I didn’t have anything that exactly corresponded to Libya, but it seems reasonable to say it was better than North Korea but worse than China, so maybe around 0.6? And that though post-Gaddafi Libya is still poor and conflict-ridden, it’s just a little bit better, so perhaps 0.7?

So if you improve the lives of 6 million people by 0.1 QALYs/year x 25 years, that’s another 15 million QALYs gained, for a total of about 16 million.

If you purchase 15.5 million QALYs for $1 billion, you find you spent about $65 per QALY.

Compare this to the Against Malaria Foundation. The Slate article boasts that “according to The Life You Can Save, handing out these bed nets saves about one life for every $1,865 spent.”

So here $1,865 buys one life = 50 QALYs (we said 25 before, but let’s be generous since most deaths from malaria are children and so will have more lifespan left to save), but in this case we multiply by 0.5 (the weight the survey gave to life in Ethiopia, which is a proxy for the sorts of countries where AMF likely operates) and get about $75.

So if you accept the main premise of the above – that life is ~10% better in Libya after Gaddafi was overthrown – military intervention in Libya was a bit more effective towards humanitarian goals than donations to AMF, buying QALYs for $65 versus $75.

Let’s put these numbers in a broader context than “one is bigger than the other”. In health care, anything that costs less than about $25,000 per QALY is generally considered excellent value. People wax poetic about the amazing cost-effectiveness of preventative health care because it usually comes out to “only” $1000 – $5000 per QALY. These numbers are both mind-bogglingly low.

And in the case of Libya, this may an underestimate, since it doesn’t take into account shortening the war, or spurring foreign investment, or the fact that Gaddafi probably would have become more repressive after the rebellion, or less tangible effects like deterrence of future dictators.

Didn’t I just say that donating to GiveWell’s top charity must tautologically be the best use of money if you trust GiveWell? Yes, but that’s only true for the average person with a marginal dollar to spare. If you happen to be a superpower with an air force of 5,500 planes and several billions of dollars to burn, you have options that GiveWell probably will not have evaluated.

My guess is a perfectly utilitarian US Air Force (now there’s a plot hook!) would probably require its own GiveWell, at least for a while, rather than selling its planes for scrap and donating to the civilian version.

Am I saying that military intervention is always more cost-effective than civilian charity? Certainly not. Military interventions sometimes cause large numbers of deaths, don’t always keep the costs down as effectively as the Libyan operation did, don’t always succeed, and when they do succeed they risk unintended consequences (like replacing a brutal-but-effective dictator with an anarchic power vacuum).

But if we still had InTrade, we could come up with a probability that each of those things would happen, factor them into our calculations, and who knows, it might still be better than giving to Against Malaria Foundation. Or at least better than giving money to Egypt to buy weapons to point at Israel to counteract the money we gave to Israel to buy weapons to point at Egypt – which, let’s face it, is a much more realistic description of where the typical foreign aid dollar goes than Against Malaria Foundation is.

(note that I do not intend this article to assert that intervening in Syria at present is a good idea. I am currently neutral-ish on this point, since the possibility of making the situation much better seems counterbalanced by a good chance of making it much worse. But this should not be taken as a critique of interventions more broadly, let alone a Fully General one.)

Yglesias’ article ends by saying:

Now, before the kill-and-maim-for-the-sake-of-humanity crowd shoots a Tomahawk missile at me, it’s worth conceding up front that none of this amounts to a logically airtight case against blowing up some Syrian infrastructure and killing various Syrian bad guys. It is very possible for a given undertaking to be worth doing without being the optimal policy. But I do think it’s worth interrogating the larger political and ideological construct that says that spending a few billions dollars to help foreigners is a thinkable undertaking if and only if the means of providing assistance is to kill some people and blow some stuff up. The explosives-heavy approach to humanitarianism has a lot of unpredictable side effects, sometimes backfires massively, and offers an extremely poor value proposition. So whatever you think about killing some Syrians this summer, please consider throwing a few dollars in the direction of a cost-effective charity of some kind.

A big “amen!” to all of that except the “extremely poor value proposition” part.

Consider the possibility that it is gravity. In statements like “Sentencing guidelines should take into account the gravity of the offense”, the words “gravity” and “immorality” are used interchangeably. Gravitational language informs our moral discourse in other ways too: immoral people are described as “fallen”, sin is a “weight” upon the soul, and we worry about society undergoing moral “collapse”. So the argument from common usage (is best argument! is never wrong!) makes a strong case for an unexpected identity between morality and gravity similar to that between (for example) electricity and magnetism.

We can confirm this to the case by investigating inverse square laws. If morality is indeed an unusual form of gravitation, it will vary with the square of the distance between two objects.

Imagine a village of a hundred people somewhere in the Congo. Ninety-nine of these people are malnourished, half-dead of poverty and starvation, oozing from a hundred infected sores easily attributable to the lack of soap and clean water. One of those people is well-off, living in a lovely two-story house with three cars, two laptops, and a wide-screen plasma TV. He refuses to give any money whatsoever to his ninety-nine neighbors, claiming that they’re not his problem. At a distance of a ten meters – the distance of his house to the nearest of their hovels – this is monstrous and abominable.

Now imagine that same hundredth person living in New York City, some ten thousand kilometers away. It is no longer monstrous and abominable that he does not help the ninety-nine villagers left in the Congo. Indeed, it is entirely normal; any New Yorker who spared too much thought for the Congo would be thought a bit strange, a bit with-their-head-in-the-clouds, maybe told to stop worrying about nameless Congolese and to start caring more about their friends and family.

This is, of course, completely rational. New York City, at ten thousand kilometers, is one million times further away from the suffering villagers as the original well-off man’s ten meters. Since moral force decreases with the square of the distance, the moral force of the Congolese on the New Yorker is diminished by a factor of one million squared – that is, one trillion.

At that distance, all one billion Africans matter only 1/1000th as much as would a person at zero distance. There is, in fact, a person at zero distance from the average New Yorker – that New Yorker herself. So we find that our theory predicts that our obligations to the Congo are only one tenth of one percent as important as our obligations to ourselves.

We can confirm this experimentally. This article from 2005 lists private US overseas charitable contributions at $10.7 billion a year. The 2000 US Census gave a population of 281,421,906, meaning that the average American gave $38.02 in overseas charity. This is 0.107% of the average 2005 per capita income of $35,242, compared to a predicted .0100; that is, a margin of error of only about twenty four cents.

(This is why I love physics. You’d never get results that match up to predictions that precisely in the so-called “social sciences”.)

This methodology can be used to answer a seemingly very different problem that many of us face every day: just how far away from a beggar do you need to walk before you don’t have to feel bad about not giving her money?

Suppose the marginal value of an extra dollar to a beggar is ten times its value to a well-off person such as yourself. We start with the money in your pocket, about a meter away from your brain. If you pass right by the beggar then the money may be a meter away from the beggar as well. Distance to both people is equal, so here the moral force exerted by the beggar is ten times stronger than your own moral force: you are clearly obligated to give her the money.

As you double your distance from the beggar to two meters, the moral force of her need decreases by a factor of four; however, she still has a 2.5x greater claim to the money than you do. Even three meters is not sufficient; her claim will be 1.1x as strong as your own.

However, four meters ought to do it. At this distance, the importance of the beggar’s poverty has decreased by a factor of sixteen, while your own moral force has stayed constant. It’s now 1.6x better for you to keep the money for yourself – a comfortable margin of safety.

There has been some discussion on whether it is acceptable to just hang to the far outside of the sidewalk in order to avoid a beggar, or whether this is unethical and it necessary to cross to the entire opposite side of the street. We now have the tools necessary to solve this problem. If you are on a commercial throughway, downtown residential, or other sidewalk listed on this table as having a minimum width of 4m or greater, it is borderline acceptable (ignoring air resistance) simply to move to the other side of the walkway. However, on the smaller neighborhood residential sidewalks, industrial sidewalks and alleyways – not to mention anywhere the beggar is in the middle of the walkway – it is unfortunately necessary to cross all the way to the other side of the street.

Once again, the results of even a back-of-the-envelope calculation like this one mesh admirably with most people’s native intuitions. Just as even a young child who throws a ball will have a “gut feeling” about how long it will stay up in the air, so even people unaware that morality is a variant of gravitation can correctly apply these same “gut feelings” to moral dilemmas.

In summary, morality is a form of gravitation, albeit an unusual one. Calculations performed based on inverse square law assumptions correctly predict most people’s moral actions. Indeed, the majority of human moral behavior make no sense except under these assumptions, and without them our everyday moral reasoning would be ridiculous indeed.

A few weeks ago I got to go to a free CFAR tutorial (you can hear about these kinds of things by signing up for their newsletter). During this particular tutorial, Julia tried to explain Bayes’ Theorem to some, er, rationality virgins. I record a heavily-edited-to-avoid-recognizable-details memory of the conversation below:

Julia: So let’s try an example. Suppose there’s a five percent chance per month your computer breaks down. In that case…
Student: Whoa. Hold on here. That’s not the chance my computer will break down.
Julia: No? Well, what do you think the chance is?
Student: Who knows? It might happen, or it might not.
Julia: Right, but can you turn that into a number?
Student: No. I have no idea whether my computer will break. I’d be making the number up.
Julia: Well, in a sense, yes. But you’d be communicating some information. A 1% chance your computer will break down is very different from a 99% chance.
Student: I don’t know the future. Why do you want to me to pretend I do?
Julia: (who is heroically nice and patient) Okay, let’s back up. Suppose you buy a sandwich. Is the sandwich probably poisoned, or probably not poisoned?
Student: Exactly which sandwich are we talking about here?

In the context of a lesson on probability, this is a problem I think most people would be able to avoid. But the student’s attitude, the one that rejects hokey quantification of things we don’t actually know how to quantify, is a pretty common one. And it informs a lot of the objections to utilitarianism – the problem of quantifying exactly how bad North Korea shares some of the pitfalls of quantifying exactly how likely your computer is to break (for example, “we are kind of making this number up” is a pitfall).

The explanation that Julia and I tried to give the other student was that imperfect information still beats zero information. Even if the number “five percent” was made up (suppose that this is a new kind of computer being used in a new way that cannot be easily compared to longevity data for previous computers) it encodes our knowledge that computers are unlikely to break in any given month. Even if we are wrong by a very large amount (let’s say we’re off by a factor of four and the real number is 20%), if the insight we encoded into the number is sane we’re still doing better than giving no information at all (maybe model this as a random number generator which chooses anything from 0 – 100?)

This is part of why I respect utilitarianism. Sure, the actual badness of North Korea may not be exactly 37%. But it’s probably not twice as good as living in the First World. Or even 90% as good. But it’s probably not two hundred times worse than death either. There is definitely nonzero information transfer going on here.

But the typical opponents of utilitarianism have a much stronger point than the guy at the CFAR class. They’re not arguing that utilitarianism fails to outperform zero information, they’re arguing that it fails to outperform our natural intuitive ways of looking at things, the one where you just think “North Korea? Sounds awful. The people there deserve our sympathy.”

Remember the Bayes mammogram problem? The correct answer is 7.8%; most doctors (and others) intuitively feel like the answer should be about 80%. So doctors – who are specifically trained in having good intuitive judgment about diseases – are wrong by an order of magnitude. And it “only” being one order of magnitude is not to the doctors’ credit: by changing the numbers in the problem we can make doctors’ answers as wrong as we want.

So the doctors probably would be better off explicitly doing the Bayesian calculation. But suppose some doctor’s internet is down (you have NO IDEA how much doctors secretly rely on the Internet) and she can’t remember the prevalence of breast cancer. If the doctor thinks her guess will be off by less than an order of magnitude, then making up a number and plugging it into Bayes will be more accurate than just using a gut feeling about how likely the test is to work. Even making up numbers based on basic knowledge like “Most women do not have breast cancer at any given time” might be enough to make Bayes Theorem outperform intuitive decision-making in many cases.

And a lot of intuitive decisions are off by way more than the make-up-numbers ability is likely to be off by. Remember that scope insensitivity experiment where people were willing to spend about the same amount of money to save 2,000 birds as 200,000 birds? And the experiment where people are willing to work harder to save one impoverished child than fifty impoverished children? And the one where judges give criminals several times more severe punishments on average just before they eat lunch than just after they eat lunch?

And it’s not just neutral biases. We’ve all seen people who approve wars under Republican presidents but are horrified by the injustice and atrocity of wars under Democratic presidents, even if it’s just the same war that carried over to a different administration. If we forced them to stick a number on the amount of suffering caused by war before they knew what the question was going to be, that’s a bit harder.

Thus is it written: “It’s easy to lie with statistics, but it’s easier to lie without them.”

Some things work okay on System 1 reasoning. Other things work badly. Really really badly. Factor of a hundred badly, if you count the bird experiment.

It’s hard to make a mistake in calculating the utility of living in North Korea that’s off by a factor of a hundred. It’s hard to come up with values that make a war suddenly become okay/abominable when the President changes parties.

Even if your data is completely made up, the way the 5% chance of breaking your computer was made up, the fact that you can apply normal non-made-up arithmetic to these made-up numbers will mean that you will very often still be less wrong than if you had used your considered and thoughtful and phronetic opinion.

On the other hand, it’s pretty easy to accidentally Pascal’s Mug yourself into giving everything you own to a crazy cult, which System 1 is good at avoiding. So it’s nice to have data from both systems.

In cases where we really don’t know what we’re doing, like utilitarianism, one can still make System 1 decisions, but making them with the System 2 data in front of you can change your mind. Like “Yes, do whatever you want here, just be aware that X causes two thousand people to die and Y causes twenty people an amount of pain which, in experiments, was rated about as bad as a stubbed toe”.

And cases where we don’t really know what we’re doing have a wonderful habit of developing into cases where we do know what we’re doing. Like in medicine, people started out with “doctors’ clinical judgment obviously trumps everything, but just in case some doctors forgot to order clinical judgment, let’s make some toy algorithms”. And then people got better and better at crunching numbers and now there are cases where doctors should never use their clinical judgment under any circumstances. I can’t find the article right now, but there are even cases where doctors armed with clinical algorithms consistently do worse than clinical algorithms without doctors. So it looks like at some point the diagnostic algorithm people figured out what they were doing.

I generally support applying made-up models to pretty much any problem possible, just to notice where our intuitions are going wrong and to get a second opinion from a process that has no common sense but is also lacks systematic bias (or else has unpredictable, different systematic bias).

This is why I’m disappointed that no one has ever tried expanding the QALY concept to things outside health care before. It’s not that I think it will work. It’s that I think it will fail to work in a different way than our naive opinions fail to work, and we might learn something from it.

EDIT: Edited to include some examples from the comments. I also really like ciphergoth’s quote: “Sometimes pulling numbers out of your arse and using them to make a decision is better than pulling a decision out of your arse.”

By far the most interesting result from my utility and QALY survey earlier was the striking last name imbalance in readers of this blog. 93 people took the survey intended for people A-M, versus 34 for the N-Zs.

American last names begin with A-M 62% of the time (most of the imbalance is in M; I blame the Scots and Irish). StatTrek informs me that a 62% balance giving me results at least as lopsided as 93-34 should happen by chance only 0.5% of the time. Perhaps people just took the first survey no matter what their name was? Maybe they were turned off by the somewhat more complicated instructions on the second survey? It seems to be a mystery.

(as, indeed, is why about 85% of respondents came from Juneau, Alaska. Must be one of those coincidences :P)

I would like to think the survey showed that most people are pretty anti-death. The survey with death as the floor didn’t give great evidence for that, since I specifically told them to act as if death was a good floor in order to judge other things properly. But in the version with prison as the floor people consistently rated death significantly worse than states like prison, being in a starving African country, or being in North Korea. The survey was not really aimed at discussing death or suicide, so I would like to think it was less about signaling than some direct reports might be.

The next most interesting result was in the Relationship question. Single people consistently said a relationship would be about 1.54x better than their current single life, but people in a relationship estimated singlehood would only be about .85x as good as their current life; these numbers are obviously not reciprocals of each other. I blame myself; in the wording of the question, I asked single people to imagine a relationship with their “ideal” partner. For anyone with a remotely good imagination, this is a very desirable state. People in real relationships, even very happy relationships, probably can’t live up to that “ideal partner” standard, so it may be rational for these numbers to be different and not a sign of bias at all. On a similar note, men expected that women have significantly lower quality of life, but women did not expect men to have significantly higher quality of life; this failure of matched reciprocals seems much less excusable.

But what about the utility calculations? This was a mix of good and bad news.

The three methods of determining utility correlated extremely haphazardly with one another. Sometimes they would correlate as high as .7 or .8, other times not at all. There seemed to be no pattern to on what questions they would correlate, nor to which of the three would correlate best with which other of the three methods. The best I can say is that correlations between the three methods on the same question always seemed higher than correlations between different questions, which is good, I guess.

But once they were averaged out, they came up with remarkably consistent results. The lists of averages on each method correlated with the lists of averages of both other methods at > 0.9 level.

There was a very wide range of answers on each question; even something as uncontroversially bad as blindness would get utility weights of anywhere from 0.1 to 0.9. The standard deviations, once each question was standardized to be between 0 and 1, were pretty consistently around 0.2.

But most people had very similar preference orderings, and most people agreed on which things were only mildly bad (only tiny deviations from 1) versus worse. I feel like although the data were very noisy, there was also a very real signal in there.

Another encouraging fact is that even with different floors, the people on Survey 1 and the people on Survey 2 came up with almost the same results once they were adjusted onto compatible scales:

Last point. I put “blindness” on there because many people in health research have studied the negative utility of blindness and I wanted to see if this test, amateurish and haphazard as it was, would get the same results they did (which might in turn legitimize some of its other numbers). The CHUM-T gets a utility weight for blindness of .63. Tufts’ database of utility weights lists numbers between .4 and .69 (also other numbers for various technical terms that mean blindness, but I think the exact wording is important here).

So I guess if I can conclude anything from this, it’s that utility measurement is very hard and produces confusing results, but that this test doesn’t seem to be any more wrong than others.

You can download the raw data in .csv format here. Tell me if you find anything cool.