What To Make Of New Positive NSI-189 Results?

I.

I wanted NSI-189 to be real so badly.

Pharma companies used to love antidepressants. Millions of people are depressed. Millions of people who aren’t depressed think they are. Sell them all a pill per day for their entire lifetime, and you’re looking at a lot of money. So they poured money into antidepressant research, culminating in 80s and 90s with the discovery of selective serotonin reuptake inhibitors (SSRIs) like Prozac. Since then, research has moved into exciting new areas, like “more SSRIs”, “even more SSRIs”, “drugs that claim to be SNRIs but on closer inspection are mostly just SSRIs”, and “drugs that claim to be complicated serotonin modulators but realistically just work as SSRIs”. Some companies still go through the pantomime of inventing new supposedly-not-SSRI drugs, and some psychiatrists still go through the pantomime of pretending to be excited about them, but nobody’s heart is really in it anymore.

How did it come to this? Apparently discovering new antidepressants is really hard. Part of it is that depression has such a high placebo response rate (realistically probably mostly regression to the mean) that it’s hard for even a good medication to separate much from placebo. Another part is that psychopharmacology is just a really difficult field even at the best of times. Pharma companies tried, tried some more, and gave up. All the new no-really-not-SSRIs are the fig leaf to cover their failure. Now people are gradually giving up on even pretending. There are still lots of exciting possibilities coming from the worlds of academia and irresponsible self-experimentation, but the Very Serious People have left the field. This is a disaster, insofar as they’re the only people who can get things through the FDA and into the mass market where anyone besides fringe enthusiasts will use them.

Enter NSI-189. A tiny pharma company called Neuralstem announced that they had a new antidepressant that worked on directly on neurogenesis – a totally new mechanism! nothing at all like SSRIs! – and seemed to be getting miraculous results. Lots of people (including me) suspect neurogenesis is pretty fundamental to depression in a way serotonin isn’t, so the narrative really worked – we’ve finally figured out a way to hit the root cause of depression instead of fiddling around with knobs ten steps away from the actual problem. Irresponsible self-experimenters managed to synthesize and try some of it, and reported miraculous stories of treatment-resistant depressions vanishing overnight. Someone had finally done the thing!

There are many theories about what place our world holds in God’s creation. Here’s one with as much evidence as any other: Earth was created as a Hell for bad psychiatrists. For one thing, it would explain why there are so many of them here. For another, it would explain why – after getting all of our hopes so high – NSI-189 totally flopped in FDA trials.

I don’t think the data have been published anywhere (more evidence for the theory!), but we can read off the important parts of the story from Neuralstem’s press release. In Stage 1, they put 44 patients on 40 mg NSI-189 daily, another 44 patients on 80 mg daily, and 132 patients on placebo for six weeks. In Stage 2, they took the people from the placebo group who hadn’t gotten better in Stage 1 and put half of them on NSI-189, leaving the other half on placebo – I think this was a clever trick to get a group of people pre-selected for not responding to placebo and so avoid the problem where everyone does well on placebo and so it’s a washout. But all of this was for nothing. On the primary endpoint – a depression rating instrument called MADRS – the NSI-189 group failed to significantly outperform placebo during either stage.

Neuralstem’s stock fell 61% on news of the study. Financial blog Seeking Alpha advised readers that Neuralstem Is Doomed. Investors tripped over themselves to withdraw support from a corporation that apparently was unable to handle the absolute bread-and-butter most basic job of a pharma company – fudging clinical trial results so that nobody figures out they were negative until half the US population is on their drug.

From last month’s New York Times:

The first thing you feel when a [drug] trial fails is a sense of shame. You’ve let your patients down. You know, of course, that experimental drugs have a poor track record – but even so, this drug had seemed so promising (you cannot erase the image of the cancer cells dying under the microscope). You feel as if you’ve shortchanged the Hippocratic Oath […]

There’s also a more existential shame. In an era when Big Pharma might have macerated the last drips of wonder out of us, it’s worth reiterating the fact: Medicines are notoriously hard to discover. The cosmos yields human drugs rarely and begrudgingly – and when a promising candidate fails to work, it is as if yet another chemical morsel of the universe has been thrown into the dumpster. The meniscus of disappointment rises inside you: That domain of human biology that the medicine hoped to target may never be breached therapeutically.

And so the rest of us gave a heavy sigh, shed a single tear, and went back to telling ourselves that maybe vortioxetine wasn’t exactly an SSRI, in ways.

II.

But the reason I’m writing about all of this now is that Neuralstem has just put out a new press release saying that actually, good news! NSI-189 works after all! Their stock rose 67%! Investment blogs are writing that Neuralstem Is A Big Winner and boasting about how much Neuralstem stock they were savvy enough to hold on to!

What are these new results? Can we believe them?

I’m still trying to figure out exactly what’s going on; the results themselves were presented at a conference and aren’t directly available. But from what I can gather from the press release, this isn’t a new trial. It’s new secondary endpoints from the first trial, that Neuralstem thinks cast a new light on the results.

What are secondary endpoints? Often during a drug trial, people want to measure whether the drug works in multiple different ways. For depression, these are usually rating scales that ask about depressive symptoms – things like “On a scale of 1 to 5, how sad are you?” or “How many times in the past month have you considered suicide?”. You could give the MADRS, a scale that focuses on emotional symptoms. Or you could give the HAM-D, a scale that focuses more on psychosomatic symptoms. Or since depression makes people think less clearly, you could give them a cognitive battery. Depending on what you want to do, all of these are potentially good choices.

But once you let people start giving a lot of tests, there’s a risk that they’ll just keep giving more and more tests until they find one that gives results they like. Remember, one out of every twenty statistical analyses you do will be positive at the 0.05 level by pure coincidence. So if you give people ten tests, you’ve got a pretty good chance of getting one positive result – at which point, you trumpet that one to the world.

Statisticians try to solve this loophole by demanding researchers pre-identify a primary endpoint. That is, you have to say beforehand which test you want to count. You can do however many tests you want, but the other ones (“secondary endpoints”) are for your own amusement and edification. The primary endpoint is the one that the magical “p = 0.05 means it works” criteria gets applied to.

Neuralstem chose the MADRS scale as their primary endpoint and got a null result. This is what they released in July that had everybody so disappointed. The recently-released data are a bunch of secondary endpoints, some of which are positive. This is the new result that has everybody so excited.

You might be asking “Wait, I thought the whole point of having primary versus secondary endpoints was so people wouldn’t do that?” Well…yes. I’m trying to figure out if there’s any angle here besides “Company does thing that you’re not supposed to do because it can always give you positive results, gets positive results, publishes a press release”. I am not an expert here. But I can’t find one.

The pattern of positive results shows pretty much the random pattern you would expect from spurious findings. They’re divided evenly among a bunch of scales, with occasional positive results on one scale followed by negative results on a very similar scale measuring the same thing. Most of them are only the tiniest iota below p = 0.05. Many of them only work at 40 mg, and disappear in the 80 mg condition; there are occasional complicated reasons why drugs can work better at lower doses, but Occam’s razor says that’s not what’s happening here. One of the results only appeared in Stage 2 of the trial, and disappeared in Stage 1 and the pooled analysis. This doesn’t look exactly like they just multiplied six instruments by two doses by three ways of grouping the stages, got 36 different cells, and rolled a die in each. But it’s not too much better than that. Who knows, maybe the drug does something? But it sure doesn’t seem to be a particularly effective antidepressant, even by our very low standards for such. Right now I am very unimpressed.

III.

Except…why did their stock jump 67%? We just got done talking about the efficient market hypothesis and the theory that the stock market is never wrong in a way detectable by ordinary humans.

First of all, maybe that’s wrong. My dad is a doctor, and he swears that he keeps making a lot of money from medical investments. He just sees some new medical product, says “Yeah, that sounds like the sort of thing that will work and become pretty popular”, and buys it. I keep telling him this cannot possibly work, and he keeps coming to me a year later telling me he made a killing and now has a new car. Maybe all financial theory is a total lie, and if you get a lucky feeling when looking at a company’s logo you should invest in them right away and you will always make a fortune.

Or maybe the it’s that it’s not investors’ job to answer “Does this drug work?” but rather “Will investing in this stock make me money?”. Neuralstem has mentioned that they’ll be bringing these new results in front of the FDA, presumably in the hopes of getting a Phase III trial. FDA standards seem to have gotten looser lately, and maybe a fig leaf of positive results is all they need to give the go ahead for a bigger trial anyway – after all, they wouldn’t be approving the drug, just saying more research is appropriate. Then maybe that trial would come out better. Or it would be big enough that they would discover some alternate use (remember, Viagra was originally developed to lower blood pressure, and only got switched to erectile dysfunction after Phase 1 trials). Or maybe Neuralstem will join the 21st century and hire a competent Obfuscation Department.

I don’t know. I’m beyond caring. The sign of a really deep depression is abandoning hope, and I’ve abandoned hope in NSI-189…

…which just leaves me even more time to be excited about SAGE-217, the novel GABA-A positive allosteric modulator that just passed Phase 2 trials! This one is going to be great!

[EDIT: Wait, is SAGE-217 just a weird attempt to rebrand benzodiazepines? Surely it’s got to be more than that, right?]

This entry was posted in Uncategorized and tagged . Bookmark the permalink.

171 Responses to What To Make Of New Positive NSI-189 Results?

  1. Scott H. says:

    I’m sure most people can do the math, but just to make sure…

    When a stock drops 61% and then rises by 67% it is still down ~35% from its original price.
    That might have some relevance for the EMH discussion.

    • Toby Bartels says:

      Yep. Put NASDAQ CUR into Google, select a 1-year graph, and it becomes clear. Not only did the stock fall much more than it subsequently rose, it even fell about halfway back down again the next day. (And all of this is way down from what it was in 2014 and 2015.)

  2. Walter says:

    I agree with your father, for what it is worth. Efficient Market Hypothesis is gibberish.

  3. Mr Mind says:

    Primary and secondary endpoints fight.
    So much data wasted under bad statistics.
    My Bayesian heart bleeds.

    • aphyer says:

      Could have been haiku
      Now just boring net comment
      My poet’s heart bleeds.

      • Toby Bartels says:

        Who says it's not haiku? English-language haiku doesn't need to follow a strict 5-7-5 syllable structure. For that matter, contemporary Japanese haiku doesn't usually count syllables. (And strictly speaking, Japanese haiku never counts syllables, but moras, which are often shorter. By my count, your haiku's lines have at least 9-12-9 moras, depending on how one counts them.)

        Emphasizing the number of syllables or moras over kireji and kigo turns haiku into doggerel. (Your haiku has a nice kireji (‘now’), even though it’s at the beginning of the line instead of at the end, but no kigo.) In contrast, check out the examples of English haiku at Wikipedia.

        (I apologize that my comment is not in the form of a haiku. I'm in a hurry.)

        • aphyer says:

          An Internet smartass with no wit
          Thought to measure himself as a poet
          He wrote a haiku
          But people like you
          Are better at that and they show it.

          • Toby Bartels says:

            I had to read that three times before I put the correct stress on ‘no wit’. (But it scans perfectly, so it's my fault, I guess.)

            But to be fair, I have not shown that I'm better. I haven't written any haiku here!

  4. MostlyCredibleHulk says:

    > the theory that the stock market is never wrong in a way detectable by ordinary humans.

    I don’t think efficient market hypothesis implies the market could never be wrong in any particular point at time, and any price is always justified at any particular moment. Otherwise bubbles wouldn’t exist and they clearly did, many times.

  5. Bugmaster says:

    This may be a stupid question, but why don’t people use something like the Bonferroni Correction for repeated trials ? Basically, discount the P-value of each trial, to still get the original P-value in aggregate. Wouldn’t this help avoid P-hacking ?

    • Murphy says:

      If you’re actually doing the analysis you can adjust for multiple comparisons.

      if you’re not the one doing the analysis you can’t know how many comparisons were discarded entirely and never mentioned in the paper and you may not be able to recreate the numbers without the raw data which may be hard to access if it contains patient level data.

  6. Telomerase says:

    Instead of trying to treat “the vapors”, or “depression”, or other medieval terms that don’t refer to actual cellular events… let’s try treating our epigenetics. Europeans already do this with S-adenosyl methionine, which drives DNA methylation.

    The next steps would be telomerase activation, metabolic therapy (e.g. nicotinamide riboside), and other general “anti-aging” treatments. You don’t see Bowhead Whales lying around depressed, even at 230 years old.

  7. Aapje says:

    So…I’m getting some desire to purchase put options for this company.

    Although, I’m a little worried that part or all of the initial stock price decrease and the later increase is because the investors are judging the ability of the company to bamboozle. If the FDA has low standards, which are easily met with the statistical creativity shown by Neuralstem, then the observed investor behavior is fully rational.

    When Neuralstem published honest, bad results, the investors may have simply concluded that the company was run by ethical and thus incompetent management. Then Neuralstem later published dishonest, good results, the investors may have concluded that the company wised up and now is run by unethical and thus competent management.

    • Murphy says:

      This possibility has been niggling at me. The really depressing possibility would be if there were no correlation between statistical fudging of results and compounds later failing phase 3 trials or being withdrawn while still in patent as that would imply an entire structure being hollowed out to uselessness.

      Side note: are an unusually large number of compounds withdrawn shortly after their patents expire?

  8. Sniffnoy says:

    The main thing I’m getting out of this is that “GC-161” is in fact a realistic name for an experimental drug. 😛

  9. RKN says:

    The Stage 2 trial is too under powered to draw any meaningful conclusions about real efficacy, even if significance (p<0.05) had been achieved. That's what Stage 3 trials are for. Stage 1 trials aren't designed to measure effect size, but instead to assess toxicity and pharmacokinetics. In any case, I think the current estimate of experimental small molecule drugs failing Stage 2 is ~70%, so not terribly surprising.

  10. Calecute says:

    Do note that if it’s expensive enough to get medical knowledge, and the ineficiencies are small enough, a medical doctor getting higher than market returns on medical investiments seens consistent with EMH.

  11. anonymousskimmer says:

    My dad is a doctor

    This is the most problematic part of the post for me.

    Do you follow in your father’s footsteps because you wanted to, because it was easy, because that’s where your interests lay, because you had no other idea of what you wanted to do, because he sold it to you?

    I know there are children who genuinely, innately want to pursue the same career category as their parent. And I must be happy with this desire of theirs to pursue their dreams along the same direction their parents first trod. But it’s hard to do as based on my personal experience there seems to be so much implicit parental compulsion in the process.

    • The Nybbler says:

      But it’s hard to do as based on my personal experience there seems to be so much implicit parental compulsion in the process.

      There are reasons besides compulsion. Children tend to take after their parents for genetic reasons. Children are likely to be more familiar with the career fields of their parents than others, which goes to both interest and in some cases having a leg up in getting into the field.

      • anonymousskimmer says:

        Oh yeah, I know this. These are by far lesser compulsions (and any genetic trait is basically no compulsion at all).

        The “leg up” I do see as a form of compulsion. An ideal good parent should go to the lengths necessary to find other mentors for their child in the areas the child is interested in that the parent has no experience in.

    • Murphy says:

      Myself and my brother ended up in computing while my dad got into the field in the days of punchcards.

      I never felt pressured into it. Sure, there were always computers around the house and bookshelves full of computing related material but nobody ever so much as hinted that I should.

      I was however mathematics was always easy-mode for me from around the time I could talk and 3/4 of my siblings were similar. There’s also going to be some element of familiarity since any field close family members end up in is going to be talked about more and you have a better idea of what everyday life working in the field is like.

      • MostlyCredibleHulk says:

        Same here. While my parents emphasized the need to study, get education, etc. I don’t think I was pressured to choose a specific field. I just happened to like it. Maybe if my father did something different I ended up knowing it better and liking it too, but life is full of this kind of random stuff. But I don’t think “pressure” has much to do with it.

    • In my case, I chose an interesting field (physics) other than my father’s (economics), in part because I didn’t want to spend my life being identified primarily as my father’s son. I eventually concluded that I was a better economist than physicist and switched fields, staying out of the particular subfield where my father had made his largest contributions.

      I ended up in a subfield pioneered by, among others, my uncle, but he was a lot less prominent and had a different last name so it wasn’t really a problem.

  12. chernavsky says:

    The best treatment I’ve seen about antidepressants (and the serious problems with them) has been in the work of Robert Whitaker, particularly in his book, Anatomy of an Epidemic: Magic Bullets, Psychiatric Drugs, and the Astonishing Rise of Mental Illness in America. See also, “Now Antidepressant-Induced Chronic Depression Has a Name: Tardive Dysphoria“.

  13. jchrieture says:

    Quoted from the Centers for Disease Control and Prevention (CDC)

    What is an adjuvant
    and why are adjuvants added to vaccines?

    An adjuvant is an ingredient of a vaccine that helps create a stronger immune response in the patient’s body.

    In other words, adjuvants help vaccines work better.

    Some vaccines made from weakened or dead germs contain naturally occurring adjuvants and help the body produce a strong protective immune response. However, most vaccines developed today include just small components of germs, such as their proteins, rather than the entire virus or bacteria.

    These vaccines often must be made with adjuvants to ensure the body produces an immune response strong enough to protect the patient from the germ he or she is being vaccinated against.

    A parallel extension leads us to imagine, a decade from now:

    What is a psychoadjuvant
    and why are psychoadjuvants included in psychotherapy?

    A psychoadjuvant is a medicinal component of a psychotherapy program that helps create a stronger connectomic response in the patient’s brain.

    In other words, psychoadjuvants help psychotherapy work better.

    Exercise and diet are natural psychoadjuvants that strengthen neurogenetic remodeling (repetitive cognitive behavioral exercises too are psychoadjuvants).

    However, most psychotherapeutic treatments today include pharmaceutic psychoadjuvants that directly stimulate neurogenetic remodeling (i.e., the plasticized remodeling of the synaptic, connectomic, and cellular structures of the brain).

    Effective psychotherapy commonly requires psychoadjuvants to ensure the brain produces an remodeling response strong enough to provide the patient with the capacities that he or she seeks from therapy.

    What is the probability that neurogenetic medications will find their most prevalent and effective use as psychotherapeutic adjuvants? Reasons exist to regard this outcome as pretty much inevitable in the long-term … inevitable scientifically, medically, economically, sociologically, morally, and politically.

    Inevitable scientifically: because functional/connectomic MRI will foreseeably provide (eventually) inarguable objective evidence for the physiological efficacy of psychoadjuvants.

    Inevitable medically: because neither therapy nor psychopharmacy work well individually — and neither does good scientific reason exist to believe that therapy and psychopharmacy should work well individually. Isn’t this why at least some Very Serious People (in Scott’s useful phrase) turn their attention to conjoint psychopharmacy and psychotherapy?

    Inevitable sociologically: because extended families prevalently have members who suffer from conditions that psychoadjuvant therapy can remedy (potentially).

    Inevitable morally: because medical ethics requires the medical relief of patients who suffer, in all cases in which that medical relief can be provided.

    Inevitable politically: because in the long run, political reality is grounded in public morality.

  14. Murphy says:

    Re: efficient market hypothesis.

    Best guess I can come up with is that sometimes you’re in a more niche group than you think.

    Just look at the results from CompareTrials

    http://compare-trials.org/

    The idea is simple, read the pre-reg for trials, read the published paper. for each outcome was it actually reported in the final paper, yes/no. if no was the change noted.

    They then sent a letter to the journal for each paper with changed outcomes.

    Again, looks trivial. But this took a team of people quite a lot of effort and more than half of the letters were rejected by the editors.

    This is amongst the 0.001% of the population who should be best about this kind of crap.

    Wanker-banker city trader types are not medics and few are statisticians. I know some academic statisticians who do occasional contract work for banks who are typically of the view that the work the banks want done is laughably trivial.

    Despite billions of investments riding on this kind of data there’s apparently no 3rd party company providing this data on clinical trial papers as a service. We could try setting up a company to do exactly this but 99.99% of the populations eyes glaze over when the problem is explained and there’s nothing to keep others from doing the same thing if it turns out to actually be a service anyone wants. Good luck selling it.

    Most doctors struggle with actually paying attention to this problem. It’s not high level stats but it’s also quite conceivable that for a small-cap stock enough people are crap at this kind of stats to distort the price.

    Even if there’s lots of better informed people there’s enough poorly informed people to distort the price.

    You could try shorting the stock but what’s the old saying “the market can stay irrational longer than you can stay solvent.”

    On average over long enough time periods markets are efficient. Sometimes though they’re insane and inefficient but most of the people who can see that don’t have the depth of capital they can risk to take advantage of it.

    • Murphy says:

      I find myself thinking about this more and more…. does anyone actually know of any firm already exists that privately assesses clinical trial research in this manner as a service to stock traders: comparing the pre-reg to the final paper and noting changes.

      It might also be possible to do a historical analysis to quantify the correlation to drugs failing later stage trials or being withdrawn to that kind of fudging (if such a correlation exists) and estimate more exactly the predictive power of such observations.

      If a drug is fudged through phase 1 and 2 trials by changing endpoints might it be possible to quantify it’s chances of making it through phase 3 trials compared to drugs which were assessed on the straight and narrow and thus better inform traders about whether shorting the stock will pay off….

      • BillyZoom says:

        There are a number of services that provide health care analysis, newsletters, etc. There are also firms in the expert network space that focus on biotech. The big EN player is GLG, or Gerson Lehrman Group, but there are smaller ENs focused on specific market verticals, including bio, pharma, medical devices, etc.

        But really, if you’re good at this sort of thing, you can make a ton more money running your own fund than selling research, e.g., Oracle Partners.

        I’ve not worked @ Oracle, nor had interaction w/ them for 8-10 years, but back in the day when they were a client, they had quite a number of docs and phd chemists working there.

    • BillyZoom says:

      Just want to clarify a couple things here. It comes across as a bit of an apologia for bankers, but since I guess I am kind of one, well, whatever…

      First banker != trader, so “wanker banker city trader types” is not a thing. Second, neither bankers nor traders (especially) have a background in statistics or medicine. Third, both bankers and traders are in the moving business, not the storage business, so typically have no opinion whatsoever on the assets they trade (for traders) or underwrite (for bankers).

      I don’t know what bank would (or ever has) hired out statistical analysis, but if they did it would be some small thing they didn’t have anyone around to do.

      Banks used (pre Dodd-Frank/Volcker) have more quants that they do now, but I’m guessing most of the phd physicists holding those positions would be capable of more than laughingly trivial statistical analysis.

      They majority of this type of thing happens in companies in the storage business, not the moving business. Think hedge funds and other asset managers vs. banks.

      • Murphy says:

        The reference was more to the urbandictionary version of “wanker banker” which covers bankers, traders, and/or financial types.

        They’re a very visible *type* you encounter regularly if you live near any financial hub.

        You’re probably correct that analysis they hire external contractors for may just happen to be lacking experts for.

        But then we come back to the question of why doing statistics badly can cause the stock price of the company scott mentioned to bounce back without the analysts catching it as obvious.

  15. newsroot says:

    Wait I am very curious about Sage 217 though. Those Phase II results look super impressive! Am I missing something that makes them less so?

  16. Will Fithian says:

    Enjoyed the piece. One statistical quibble (different from the ones litigated above):

    So if you give people ten tests, you’ve got a pretty good chance of getting one positive result – at which point, you trumpet that one to the world.

    It sounds like you may be implicitly assuming independence here (in which case the chance would be almost 50% under the null). But tests for different measures of depression on the same group of patients will likely be positively correlated, and that can make a big difference (as correlation —> 1 the null probability of a false positive goes back down to 0.05).

    Not sure what a “realistic” correlation is for what you’re talking about but it generally matters.

    (Edit: Maybe this is what you were getting at with “This doesn’t look exactly like they just… got 36 different cells, and rolled a die in each.”)

  17. DanielB says:

    Remember, one out of every twenty statistical analyses you do will be positive at the 0.05 level by pure coincidence.

    That’s not correct, the chance depends on the chance that the hypothesis itself is true and the tests power. A probability that we can not know but can make some guesses about. Power is the probabilty of finding an effect given that it is true(i.e. P(significan result|null hypothesis false)).

    This means that in practice the chance for a false positive finding givena positive finding often is much higher than 1 in 20. Even with 100% power if the hypothesis is rare then without adjusting the p-value threshold is very likely that it is a false positive.

    A good article on this subject is John Ioannidis Why most published research findings are false

    • John Schilling says:

      “This means that in practice the chance for a false positive finding given a positive finding often is much higher than 1 in 20″

      But that’s not what Scott was saying.

      The chance of a false finding given a statistical analysis is 1 in 20. Independent of the chance that the hypothesis itself is true, and without a requirement that any particular finding be positive. Do a large number of analyses, and one in twenty will be positive by pure coincidence. Some other number will additionally be true for reasons other than coincidence, but depending on your reason for all the testing that may not matter. If your goal is to sell stock and/or drugs to the gullible, maybe it doesn’t matter.

      • DanielB says:

        No that is not correct, it is 1 in 20 given true null hypothesisis. It’s the definition of p-value. Technically it’s also assuming that other assumptions, the model etc are true. You’re also only talking about false positives and not false findings in total, which include false negatives. The number of false negatives comes from the power of the test.

        The overall number of false findings depend on the null hypthesis being true or not, i.e. it’s probabilty. If it’s always true the number is the alpha level(i.e. usually 1 in 20). If it’s always false it’s 1-power. Anywhere in between depends on both power and alpha as well as the probabilty of the hypothesis.

        But yeah to sell the drug for statistical illiterate people then it doesn’t really matter. But then you might as well just p-hack the data anyway.

  18. billybob mcbob says:

    On the efficient markets point…

    There’s a big difference between stock prices reflecting true fundamentals, and also reflecting sentiment. When sentiment doesn’t depend only on real fundamentals or actual data (ie a reasonable chunk of the time) this breaks the basic, purist EMH theory which assumes pure rationality based on public or private data

    Unfortunately in the short/med term at least, the market seems to me to be driven by sentiment most of all, which is often not linked to fundamentals in any “rational” way (though is often predictably related to “news” in an “irrational” manner like in the example in the post)

    For an investor, a reasonable strategy is to try to predict this sort of “irrationality” in the rest of the market and try to beat it. It requires you to assume the EMH is flawed since people are not purely rational agents

    Quotes used above around words used a bit loosely/colloquially – hope the intent is clear

    tl;dr – I’m with your dad, since his attitude tends to dominate – people are not only influenced by rational interpretation of data

  19. deciusbrutus says:

    I think that the efficient market hypothesis has premises that are false in this instance, and that people are performing stock price manipulation by choosing the type and timing of study releases.

    Surely it’s not illegal for a scientist to think that the market is overconfident about a company, and short it? Then think that the market is underconfident and buy it while it’s down? He’s literally doing research about the company’s main product, and publishing it as soon as it’s ready- anybody else that was willing to jump through the same hoops could also be doing that research!

  20. Cameron Mahoney says:

    “We just got done talking about the efficient market hypothesis and the theory that the stock market is never wrong in a way detectable by ordinary humans.”

    Oh dear… what’s really going on is, this thing is a complete {starts with s and rhymes with tram}.

    Scott has talked about his tendency to be completely trusting of anyone’s intentions. This may be an example of that. Or you can look at it as an issue of not focusing on the relevant variables – which in this case are ‘how does a large portion the stock market really work?’ If you don’t want to mainline industrial quantities of short-selling techniques, stop reading now. This isn’t just one rabbit hole – it’s a whole warren.

    In pre-revenue biotech, you can do pretty good analysis of a company and completely ignore the science. That’s what I’ll do until the very end here. Marc Cohodes, a renowned short-seller who uncovered the biggest dot-com fraud – Lernout and Hauspie – uses the phrase “Bet the horse, not the jockey.” John Hempton, who sniffed out the fake forest company Sino-Forest likes to say “Once a scumbag, always a scumbag.”

    One document will inform our other research, and that is S-3 Registration Filing. This was the most complete prospectus, filed in 2008, to prepare the company to offer its common stock to investors. The first jockey we come to is the companies…

    [Lawyer]
    —————————————————————————
    Their lawyer is a fellow whose name spelled backwards is ertsevliS luaR. I don’t want this easily parsed by search engines so I’ll refer to the fellow as [lawyer]. They appears in SEC filings as the lawyer (or pre-public investor in one case) for several other small biotech, energy, and Chinese stocks:
    Research Solutions aka Inspyr Therapeutics
    Social Reality Inc
    China Integrated Energy
    Signallife
    Lilis Energy

    China Integrated Energy
    —————————————————————————
    China Integrated Energy was one of the Chinese reverse mergers (a subject on which more later?) that was revealed to be a fraud in 2011. [Lawyer] helped bring them to the states in 2007. Four years later, their auditor, KPMG, resigned and declared they repudiated their clean audit of the company and resigned as auditors. In 2014, the Nasdaq revoked their registration, and this year, the company paid 2.1 million dollars to shareholders.

    A short-seller that started the whole ball rolling with the company sent private investigators to one of their plants that was supposedly processing 100,000 tons of biofuel a year. The thing was actually dormant.

    Signallife
    —————————————————————————
    A fellow attorney for the company, Mitchell Stein was sentenced to 17 years in jail for his part in manipulating Signallife stock. From court proceedings on the matter we find [lawyer] saying he beleived he’d taken part in manipluating the stock by buying it to support the price: “Mitch [Stein] made us by stock today” and “I had to be in the market buying today.”

    Research Solutions aka Inpsyr Therapeutics
    —————————————————————————
    Another firm where [lawyer] is listed as council in SEC stock registration filings. The company has paid a known ‘research analyst’ / stock promoter – one ebuarT nodlehS – to write a positive piece on it.

    This analyst has been sanctioned by the SEC on for writing untoward reports promoting stocks, like Ferrofluidics.

    Oh, did I mention the analyst’s mob connections? Though they’ve tried to hide it on their Bloomberg/Linkedin/etc profiles, a FINRA search of the analyst shows they were employed from 1996 to 2001 at First Liberty Investment Group, a financial services firm which had links to the Bonanno and Colombo crime families, and was brought down by the FBI in 2000.

    Social Reality Inc
    —————————————————————————
    Per FINRA Broker Check: The firm’s CEO worked for not one, but two companies expelled by FINRA from the securities industry for stock fraud:
    Vision Investment Group – see their State of Washington charge heres
    – Meyers Robbins Pollock – more connections to the NY Mob. Brokers were bribed to sell shares in pump-and-dumps.

    Also they just announced an Initial Coin Offering. I’ll say no more.

    Lilis Energy
    —————————————————————————
    The [lawyer] owned shares – prior to it’s public filing – in Lilis Energy, a firm whose CEO is currently being sued by the SEC for pump-and-dump manipulation. You only get to be a pre-public shareholder in a pump-and-dump promote by working and swimming in that world, with those kinds of people. Regular everyday folks don’t just accidentally end up shareholders in promote after promote after promote.

    Last Thoughts on the Lawyer
    —————————————————————————
    To some of you, this may all seem hazy, circumstantial, and loose, but this number of unsavory connections is nothing less than a quilt made entirely of red flags. Once you learn to recognize these patterns, you can gauge the chance a gold mine never dredges up any significant ore, or the likelihood a phase III trial is a complete bust, or the odds that a small bank made terrible loans… all with minimal domain expertise, because the most important variable here isn’t the business, it’s who is running it and working with it.

    At this point we can say that [lawyer] has worked alongside some VERY shady characters. On the strength of these association alone, there’s greater than 50% odds that this company is a whee bit of a {rhymes with tram}. But I haven’t even made it past the first page of their S-3 filing. I’ll share more reasons to be, uh… less than optimistic about the stock, shall we say… in later posts, when I have time.

    • BillyZoom says:

      Great analysis.

      Just one small quibble, Sino-Forest was Muddy Waters.

      • Cameron Mahoney says:

        True true, Block got the publicity on it, but Hempton was short as well – I’d would wager they traded some e-mails on it.

        Follow-up question, for you Mr Zoom, since no one else is paying attention: Why in the world is everyone arguing about p-values/placebo effect/study design on trial ran by a shady promotional small-cap no revenue biotech affiliated with scoundrels? The odds NSI-89 works aren’t poor, low, or minimal. In circumstances when a company has this kind of lawyer (and the kind of auditor, and the kind of broker they have) the odds their drug works ARE ZERO. You don’t even need to read the clinical data.

        Is this the kind of thing you only grok after you’ve had your eyes taped open and forced to read dozens of SEC filings / FINRA sanctions / securities-fraud court transcripts?

        The question’s mostly rhetorical I suppose…

        Personally, I only know to look for these things because I went down the rabbit hole on a Boca Raton biotech some family members bought stock in. This company was brought public by the same people that used to run pump-and-dumps for the Philly mob. I don’t ever sell anything short, but I now maintain a recreational interest in stock fraud…

        …which is the kind of thing liable to make you post long rants on blogs, or at least that’s what I’ve heard. 😀

        • BillyZoom says:

          I don’t have a great answer, but I think that asset/market/whatever analysis is done with a stronger focus on scaleability and repeatability rather than deep analyses of specific companies or assets.

          As examples, the number of people who read a given bond prospectus is around 0, and is, often exactly 0. The number of people who read an 8-k probably tops out a 50, with maybe 10 for a mid-cap stock.

          Like Moneyball, it’s much easier to imagine that you can get an answer analytically (especially if it can fit in row in a spreadsheet), which can then be applied to the universe of whatever your are looking at, and then be able to force rank the results.

          So for bonds, it’s the coupon + rating + capital structure + maybe industry + maybe country. For equities, it’s EPS (or EBITDA or FCF depending on industry) + industry metrics. Equities also has the “narrative” element.

          But the specifics of the company, the shareholder rights, the bond prospectus, the history of counsel, etc. are not things that scale very well.

          Most companies are not helmed, audited by, or represented by felons, or people who have been sanctioned by FINRA. Most bond prospectuses are anodyne.

          So, people talk about the things that are common, repeatable, and scalable. Because that’s the way to make real money, even if you leave some on the table for the random felon-led pharma with bonds callable @ 10% par if Mercury aligns with Mars.

    • Ketil says:

      And if you try to Google the people (un-)mentioned above, you get some bland pages and sports stars with somewhat similar names, and the ever more frequent warning that “Some results may have been removed under scammer^H^H^H^H^H^H^Hdata protection law in Europe“. Thanks a lot, Brussels.

      • jonmarcus says:

        Ketil, are you in Europe? When I googled [Analyst] from the US, it auto-completed his name before I finished typing. And showed entries in line with what Cameron said. But nothing untoward about [Lawyer].

      • cameronmahoney says:

        A few tools:

        Edgar Raw Text Search – I’m not sure how far back SEC text search goes – IIRC in early 2016 it only went back to 2013 or mid-2012, but the SEC may have expanded it’s capabilities.

        FINRA Broker Check

        You’d be surprised at how deep some stuff gets buried under noise in search engines. I find searching for “name litigation”, “name sanction”, or “name fraud” bring alarming things up to page 1 that would be on page 15 or 16 of a Google search. Leaves me feeling some combination of amused, sad and like this episode of Its Always Sunny In Philadelphia.

        As far the latter part of jonmarcus’ comment, I will direct you to the 17 year stock fraud conviction of the chief lawyer for Signallife, and page 20 of the court case where his fellow lawyer/counsel stated he felt he was manipulating the stock price of the company.

        Both were linked above, but not with nearly enough exposition – I wrote the summary rather too rapidly to make the prose sparkle. My apologies.

  21. Michael Arc says:

    It doesn’t help the EMH for investors to not be judging drug quality here.

  22. Baeraad says:

    My dad is a doctor, and he swears that he keeps making a lot of money from medical investments. He just sees some new medical product, says “Yeah, that sounds like the sort of thing that will work and become pretty popular”, and buys it. I keep telling him this cannot possibly work, and he keeps coming to me a year later telling me he made a killing and now has a new car. Maybe all financial theory is a total lie, and if you get a lucky feeling when looking at a company’s logo you should invest in them right away and you will always make a fortune.

    Er, why is that so mysterious? It sounds like your dad is following one of the most common tips for investment: “invest in what you know.” He may not know anything about the stock market, but he knows plenty about medicine – a lot more than the average person, including the average market analyst. Why wouldn’t that make him more capable of telling a promising medical investment from a dud than the stock market at large?

    • Deiseach says:

      The advantage there may not be “knows more about medical stocks than a stockbroker”, it’s “knows how patients act and how doctors think, and what they are likely to prescribe to patient who comes in with Thing complaining that all the old pills didn’t cure Thing” and so his judgement “yeah, this is gonna be the Hot New Drug of the month” is better than an average amateur investor.

  23. Deiseach says:

    the absolute bread-and-butter most basic job of a pharma company – fudging clinical trial results so that nobody figures out they were negative until half the US population is on their drug

    I should probably just say the entire post is fantastic, but I loved this particular bit extra-hard 😀

  24. Deiseach says:

    Earth was created as a Hell for bad psychiatrists.

    There, there. Biology is complicated 🙂 Which always made me laugh at the optimistic 50s SF where the shiny future societies would have worked out the One Weird Trick to make them function perfectly, mostly based on “Real Scientific Psychiatry is figuring out basic human drives and how to manipulate them, and once we know how to turn off the bad bits of human nature and turn on the good bits, everyone will be peaceful, hard-working, extraverted model citizens! Also physicists will be the new God-Emperors overseeing the rule of the Solar Empire and mathematically-derived systems of everything will solve poverty, illness, hunger, scarcity and death!” (Hmmm, is that last sounding a bit familiar still in modern terms? Never mind!)

    I think it’s human nature to want one simple answer to a problem, be that a sociologist’s dream model of how to create a better culture or one magic pill that will cure depression or any other ailment, but we’re not likely to get one single simple answer to anything.

    In today’s “Feck’s sake, weight gain/loss is bloody complicated” news, there is (yet another) miracle new ‘this one for sure will curb hunger and make you lose weight and keep it off’ molecule discovery. Ghrelin was the last Latest Hope but ghrelin was a bust, but now this time for sure! It is tentatively looking like how bariatric surgery works is not so much “we cut away half your digestive system so you can’t eat as much, you blob of lard” and more:

    The scientists operated on obese mice, performing a type of bariatric surgery called vertical sleeve gastrectomy that involves removing most of the stomach. They then examined which genes became more or less active after the procedure. As they report online today in Cell Metabolism, the rodents’ downsized stomachs produced 52 times more of a protein named LEAP2 than normal.

    Typically, LEAP2 comes from the liver and small intestine, not the stomach, but what it does isn’t clear. To investigate its function, the researchers tested whether the hormone stimulates or blocks an assortment of human receptor proteins, which allow our cells to respond to different molecular signals. They discovered that LEAP2 inhibits the receptor for ghrelin by fastening to it and possibly preventing ghrelin from latching on.

    So why do we need ghrelin anyway?

    Next, the researchers wanted to determine whether LEAP2 also foils ghrelin in the body. One of ghrelin’s important roles appears to be preventing blood sugar from plummeting when food is scarce. Ghrelin exerts this protective effect by spurring release of growth hormone, which in turn triggers the synthesis of glucose. Injecting food-deprived mice with ghrelin sparks a surge in growth hormone, but the scientists found that giving the mice a shot of LEAP2 as well prevented that increase.

    The team further delved into LEAP2’s role by genetically modifying mice to produce three times the normal amount of the protein. After a week on an extremely low calorie diet, the altered mice showed dramatically reduced blood glucose levels and became extremely weak. “They don’t have the strength to jump on the food and eat it,” says co-author Xuecai Ge, now a cell biologist at the University of California, Merced.

    Okay, that makes sense: no good in having food if you’re too weak to even eat it, so you could starve even when food supply is back to normal.

    Oh, and the reason people gain back weight after coming off a diet?

    People who lose weight often gain it back because their appetite revs up, possibly because of a corresponding rise in ghrelin levels. Using LEAP2 or a similar molecule, perhaps “we can prevent that almost inevitable hunger that follows weight loss,” he says.

    “Almost inevitable”? Ghrelin makes you hungry? So you eat more to make up for the past time of scarcity (the diet) you were on? I’m not going to blame medical science for not being able to see into the future and realise the finer details of metabolic control programmed into the organism, but it sure would have been nice to get something other than “the only reason you get back the weight is lack of will-power and laziness and greed; no you’re not hungrier, you only imagine that because you’re greedy and lazy and have no will-power” kind of ‘advice’ to the fat in years gone by.

    Ah, well. Maybe this will be the One Simple Cure for being fat! (Note: no it won’t, because Biology Is Complicated).

  25. chrism says:

    “Remember, one out of every twenty statistical analyses you do will be positive at the 0.05 level by pure coincidence.”

    You know better than that. It would be a true statement if it read ‘Remember, one out of every twenty statistical analyses that show significance at the 0.05 level, will be positive by pure coincidence’.

    It’s the difference between saying 1 in 20 of all analyses will be a false positive, and saying that 1 in 20 of positive analyses will be falsely so. They are very different things.

    Otherwise – good piece, as always.

    • Rachael says:

      I think he means that if you do twenty (and the null hypothesis is true), the expectation of how many will come out with p<0.05 is 1.

    • Rachael says:

      See his post on the elderly Chinese woman effect.

    • Dacyn says:

      Yes, they’re definitely different things, but Scott’s version is the correct one. For example, if the null hypothesis is true, then 5% of all analyses will show statistical significance, but out of those that show significance, 100% will be coincidences.

    • He is correct if you interpret his statement as “if all that is going on is pure coincidence, one out of twenty will be … .” Incorrect if you read it literally.

      But your statement is also incorrect. Suppose all the studies that are being done happen to be of things where the relationship looked for is real, hence none are positive by coincidence.

    • Freddie deBoer says:

      This thread of comments is why I have a strict rule of never explaining p-value to anyone.

      • The problem is that most people who think they understand p values have it backwards. The p value is about the probability of getting your result conditional on your theory being false (i.e. on the null hypothesis being true). They think it’s about the probability of your theory being false conditional on getting your result.

  26. tentor says:

    Regarding the efficient market, I just want to point to this story of last year:
    https://www.theguardian.com/technology/2016/jul/25/pokemon-go-nintendo-shares-tokyo-stock-exchange-niantic
    Nintendo’s stock dropped after they pointed out they are not the ones making Pokemon Go, an information that was always available to everyone.

    The EMH may say that “the stock market is never wrong in a way detectable by ordinary humans”, but it does not say the market is never wrong. There are three ways you can beat the market.
    1) Know something that everyone else doesn’t (Maybe like your dad)
    2) Do not believe something wrong that everyone else does (like the Nintendo-case, or like trusting the US mortgage system)
    3) Have different investment goals (short vs long term, or ability to carry risk)

    You also don’t need to be smarter than everyone else, only smarter than the majority (weighted by money). Of course, if you go for strategy 1 and 2 you should have a very good answer why you think this is the case.

    • A Definite Beta Guy says:

      Of course, at that point in time, Nintendo’s stock value was 28,000 Yen, and now it’s 44,000 Yen. So if you had tried to leverage your superior knowledge about Pokémon Go and its actual creators, in the form of naked short-selling, you could’ve possibly gone bankrupt, and at the very least missed out on a huge appreciation in value.

  27. Rachael says:

    “Millions of people are depressed. Millions of people who aren’t depressed think they are.”

    Slight tangent, but how common is the opposite – being depressed and thinking you’re not? I’ve had a couple of GPs and counsellors say they think I’m depressed, but I don’t think I am. Trying to figure out how likely I am to be mistaken.

  28. Peter Shenkin says:

    Hi,

    You refer to “irresponsible self-experimenters.” However, I’d like to point out that there was a time when medicinal chemists developing neurologically active agents were expected to try their products. À propos of this, I’d like to recommend the book Mentored by a Madman, by A. J. Lees, a well-known Parkinson’s neurologist and researcher in the UK. The madman is William Burroughs, and one of the messages of the book is how great the contribution of these “irresponsible self-experimenters” (as well as irresponsible acts by otherwise respectable people) has been to neurological research.

    • Scott Alexander says:

      I make fun of them because I love them

      • The Ilerminaty says:

        Whenever I, against my better judgement, visit the longecity forums again, I eventually leave with the impression that their members are mostly utter fucking idiots lacking even the basics of scientific literacy, yet cheerfully dunning-krugering on. Overall, it is heuristically unwise to wade through that ocean of crap hoping to find the occasional gem.

        • jamesbarney says:

          Everytime I read a “I don’t think I have enough glutamate, how can I increase it?”

          I die a little on the inside.

          • The Obsolete Man says:

            I used to visit Psychobabble every day and read a lot of posts like that back in the late 90s and the 00s. After taking multi-drug cocktails for ‘soft bipolar’, seasonal depression, and the anxiety disorder of the week, I got well by stopping drinking, vigorous exercise, and taking a SAM-e tablet in the morning. It would be nice to have all the money back I spent on prescriptions and 15 minute med checks.

          • Bugmaster says:

            I die a little on the inside

            You sound a little depressed. Have you checked your glutamate levels lately ? >:-)

    • markk116 says:

      But just because it turned out really really well doesn’t mean it isn’t still irresponsible and dangerous…

    • Deiseach says:

      how great the contribution of these “irresponsible self-experimenters” (as well as irresponsible acts by otherwise respectable people) has been to neurological research

      Often in a “yikes! well, now we know not to do that” way? It may be ethical to experiment on yourself, when your example encourages others to start lashing the compound into themselves in an enthusiastic, non-supervised, non-experimental manner, it may not be so ethical.

  29. poignardazur says:

    Maybe this is just me failing economics, but I thought it was always possible to beat the efficient market as long as you had a comparative advantage?

    So your father can make a lot of money by investing in pharma companies, but an outsider who saw on the TV “this company has developed a new exciting drug” wouldn’t be able to.

    • You’re misusing the term “comparative advantage”. To beat the efficient market you need an absolute advantage in some area.

      Imagine you can produce 10 wheat per hour or 15 wool per hour, while your friend can produce 30 wheat per hour or 20 wool per hour. Your friend can produce every commodity more quickly than you can. Still, according to the theory of comparative advantage, both you and your friend will want to trade with you specializing in producing wool and your friend specializing in producing wheat, because you have a comparative advantage in producing wool.

      In contrast, imagine hedge funds hire experts with 40 points in picking software stocks and experts with 25 points in picking medical product stocks, where “points” is a made-up unit of measure for domain-specific expertise. You have 5 points in picking software stocks and 15 points in picking medical product stocks. You have a comparative advantage in picking medical product stocks over picking software stocks, but you have an absolute disadvantage in both areas. Then you can’t make money picking medical product stocks, since the stock prices reflect the opinions of the hedge fund experts that already have a better idea than you do of which stocks will succeed.

      • poignardazur says:

        I was kind of under the impression that the same law applied to investment, in the sense that in your example hedge funds experts who know more than me about medical product stocks might be too busy looking up software stocks to apply their full intelligence to medical stocks…

        But yeah, now that I think about it, it does seem a little silly. Information isn’t scarce the way commodities are.

      • since the stock prices reflect the opinions of the hedge fund experts that already have a better idea than you do of which stocks will succeed.

        It’s tricky, and either of you might be right.

        Very good stock pickers are expensive. Suppose there are only a few of them. They are all busy picking software stocks and driving the gains from doing that down to their (very high) wage. None of them switch to medical stocks until the deviation from true value in those is enough to give them the same return, at which point the return to you from picking medical stocks is higher than the return to you from picking software stocks, and possibly higher than your (much lower) wage in alternative occupations.

  30. corticalcircuitry says:

    I think a better thing to be hopeful for is sticking electrodes in people’s brains. And zapping them into happiness.

    Paper:
    Deep Brain Stimulation for Treatment-Resistant Depression

    Program:
    Systems-Based Neurotechnology for Emerging Therapies (SUBNETS)

    Newspaper article:
    BRAIN-ALTERING SCIENCE AND THE SEARCH FOR A NEW NORMAL

  31. Le Maistre Chat says:

    Scott, i just wanted to point out that SSRIs are exciting for reasons other than depression. I was first prescribed the SSRI Celexa/Citalopram at age 16 for an anxiety disorder and still take it today. I don’t need an antidepressant qua antidepressant; in fact I find an over the counter depressant/heart therapy drug called red wine helpful for managing anxiety. :p

    • Scott Alexander says:

      SSRIs are great even for depression, but once you have one of them the marginal gain from there being twenty more is relatively low.

  32. jamienyc says:

    Efficient markets: it seems that you are confused about how ‘efficient’ markets operate. Stock market is not efficient in and of itself. It is efforts of people like, presumably, your father, to ferret out useful information and bid up the stock of successful companies (or short the stock of companies that are duds) that bring the markets closer and closer to the efficient frontier. So, practically no market is perfectly efficient, they are just closer or further away from the efficiency boundary.

    • Lambert says:

      > people like, presumably, your father

      I think the presumption is that the kind of people pulling the market closer to efficiency by beating it are all bankers in big cities with BMWs and silly watches. That domain-specific knowledge is of less use than devoting much of your life to learning how to beat the market.

  33. Toby Bartels says:

    [EDIT: Wait, is SAGE-217 just a weird attempt to rebrand benzodiazepines? Surely it’s got to be more than that, right?]

    You can’t fool me with your rhetorical tricks! That ‘edit’ was in the original that I got by email.

  34. Inty says:

    I wonder if a good way to beat the market is to buy stock in companies whose drugs recently received non-significant trial results. Given the confusion people have over interpreting p-values they’re likely to be making a lot of type-II errors and the drugs could end up undervalued. Problems I can see with this:
    -This has already been noticed-but-not-known by other investors and it’ll be factored into the price anyway.
    -Other variables matter way more in determining the price and this one doesn’t generate enough signal to profit from.

    • Nornagest says:

      I bet you anything there are a lot more type-I than type-II errors being published in the pharmaceutical testing space.

  35. edd91 says:

    Hi Scott,

    I particularly enjoy your posts on depression.

    In one of your previous posts on depression and neurogenesis, you pointed to the rapid reversal symptoms in bipolar disorder as a possible argument against neurogenesis being a cause of depression, I find this objection convincing and how do you deal with it? One possible response is that depression is not linked to neurogenesis per se, but rather the presence of BDNF.

    • Scott Alexander says:

      My guess is that depression is what happens when you modify some particular computational property of the brain in some direction. Neurogenesis modifies it in that direction, and bipolar/ketamine/etc are different things that for different reasons modify it very quickly in the opposite direction.

      Example: a car going slowly might be due to years of wear on the tires, but if you strap a jet engine to the back it will go very fast anyway.

      • nameless1 says:

        What about old literature, like, Hamlet, where depression is a sign of some inner dilemma?

      • nameless1 says:

        Shouldn’t modifying computation lead to more synaptogenesis, not less? Anecdotally, depressed people tend to ruminate about old hurts, never letting them go, and tend to be less aware of their surroundings. Even relatively happier depressed people I know tend to live in their memories, how cool things where when they were young and not too open to new experiences. Less synaptogenesis, effectively, less change in the brain, the brain being more “hardwired”, fixed, less would intuitively make sense as a cause of depression but it doesn ot match with changing computation.

        I have long suspected that what I consider “getting old” is mostly getting depressed, basically, tired of life, – that conspciously not-depressed people even at 50 tend to resemble young people in always wanting to try new experiences and think good things can still happen to them, while the normal i.e. depressed 50 year old lives in their memories of youth. Think Al Bundy in Married With Children as my kind of normal. This would mesh with the synaptogeneis theory, but changing computation?

  36. Bugmaster says:

    Does this mean that we should update our beliefs a little in the direction of FDA being a good thing overall ? It seems like stopping placebos from being sold as drugs is a net positive.

    • thevoiceofthevoid says:

      Possibly, but I would say it’s a positive in this particular case rather than a net positive. If the FDA stops placebos from being sold as drugs, but also stops useful drugs from being sold as drugs, it’s less clear whether they’re a net positive or a net negative.

  37. ncovington89 says:

    “Lots of people (including me) suspect neurogenesis is pretty fundamental to depression in a way serotonin isn’t…”

    I just read a book and reviewed a book “The Neurogenesis Diet and Lifestyle” by Brant Cortright. Interestingly, on of the things he recommends for neurogenesis is getting your Omega 3/6 ratio 1:1, interestingly enough something that has antidepressant effects (as Stephen Ilardi, author of “The Depression Cure” notes).
    https://www.skepticink.com/humesapprentice/2017/12/05/review-neurogenesis-diet-lifestyle/

    • The Ilerminaty says:

      Unfortunately, as I began to watch Dr. Cortright’s talk at Google, he repeated the common myth that the brain consisted mostly of fat (~ 67 %, in his case), when it is, in fact, ~ 77 % water and only ~ 12 % fat.
      While not necessarily invalidating his overall advice, this does make me question his scientific literacy. I have encountered this myth, possibly stemming from confusing wet and dry mass, so unusually often from supposed experts that it has become one of my litmus tests.

      Edit: Watched more of the talk and the questionable or grossly oversimplified claims kept rolling in. He basically repeated many paleotard claims and while his advocated diet may be healthful overall, he is not a trustworthy source. Thanks for wasting my time.

      • Creutzer says:

        Could it be, charitably, that he meant the dry mass of the brain? 12 isn’t quite 67% of 23, and I don’t know what the variance on such estimates is, but maybe he was just referring to a different estimate of the dry mass percentage.

      • Careless says:

        Something can be mostly fat and mostly water at the same time. It can even be mostly Oxygen, too. [edit: or maybe carbon outweighs it, I don’t know, but you get the idea)

  38. rahien.din says:

    [EDIT: Wait, is SAGE-217 just a weird attempt to rebrand benzodiazepines? Surely it’s got to be more than that, right?]

    It’s probably more than that!

    There are non-benzo GABA-A positive allosteric modulators on the market. The most common example is zolpidem.

    Sage has also been testing the neurosteroid allopregnanolone for the treatment of super refractory status epilepticus, on the strength of its positive allosteric GABAergic activity.

  39. I think this was a clever trick to get a group of people pre-selected for not responding to placebo and so avoid the problem where everyone does well on placebo and so it’s a washout.

    It reduces the problem but doesn’t avoid it.

    Assume placebo effect is just regression to the mean. When you select people on the placebo who didn’t respond, you are selecting the ones who haven’t yet regressed to the mean. They are still above their average level of depression, so if you keep feeding them the placebo they will, on average, get less depressed.

  40. jnp says:

    Re: neurogenesis

    What do well-informed people think of things like Lion’s Mane mushrooms (Hericium erinaceus), and/or any other supplements/interventions that are supposed to increase neurogenesis?

    I’m subject to depressive episodes, and my mother is entering into early onset Alzheimer’s dementia, so I am particularly interested in neurogenesis as it relates to those conditions. I can share some links to research I’ve found so far if anyone is interested.

    • Scott Alexander says:

      In my nootropics survey, Lion’s Mane was rated one of the least effective by users, which confirms the anecdotes I’ve heard. I agree this is pharmacologically confusing, but no more so than everything else.

      • jnp says:

        Thank you.

        I would note that this survey was of nootropics users, so presumably relatively healthy people in general who were pursuing cognitive enhancement, rather than impaired people seeking remediation, which might possibly make a difference and definitely does make it difficult to incorporate this bit of evidence into my particular focus.

        I don’t mean to sound ungrateful. I appreciate you responding and linking that. Just wish we knew more, more certainly.

  41. It’s also true that health care companies have delivered superior returns in general over the past 10 years (~6% v ~11%)

    Aren’t the questions not whether your dad beat the market historically, but whether
    (1) your dad has been reliably beating generic healthcare return (the 11% number)
    (2) whether these healthcare trends will predictably continue

    • Matt M says:

      Even putting all that aside, it’s entirely possible his dad is just lucky. Statistically speaking, some people will have above-average returns. That doesn’t mean the average isn’t negative.

      “My dad makes speculative investments and seems to do well therefore the EMH is bunk” is about the equivalent of “My dad smoked a pack a day and lived to be 100 therefore smoking isn’t unhealthy”

  42. pocketjacks2 says:

    Maybe all financial theory is a total lie, and if you get a lucky feeling when looking at a company’s logo you should invest in them right away and you will always make a fortune.

    Or by seeing the Virgin Mary patterns in random lines and calling it “technical analysis”.

    To be fair, though, technical analysis seems to work about as well as anything else we have, and there’s a reason why traders and analysts still use it, even though theory tells us it shouldn’t work.

    Or maybe the it’s that it’s not investors’ job to answer “Does this drug work?” but rather “Will investing in this stock make me money?”.

    This seems to make the EMH unfalsifiable. If it ever looks like it’s producing irrational results, that’s just because rational people are correctly betting on everyone else’s irrationality, and so what’s irrational is actually rational!

    But if “everyone else” is irrational, that sounds like a pretty severe knock against the EMH.

    • kaleberg says:

      That’s exactly right. The EMH is unfalsifiable because it is axiomatic. The price it produces is defined to be the correct price. Unfortunately, knowing this isn’t particularly useful.

      • That is not correct.

        Suppose you can show that stocks fall sharply on Friday the 13th, recover over the next week. You have just produced strong evidence against the efficient market hypothesis.

        • Chalid says:

          A real-world example is post earnings announcement drift. “For firms that report good news in quarterly earnings, their abnormal security returns tend to drift upwards for at least 60 days following their earnings announcement. Similarly, firms that report bad news in earnings tend to have their abnormal security returns drift downwards for a similar period.”

        • Lambert says:

          Then you become fantastically rich.

        • TrivialGravitas says:

          You’re ignoring risk adjustment. EMH is specific to the risk adjusted market returns, not unadjusted returns. You’ve only shown the EMH to be wrong once you can prove that the method will work reliably. And once you prove it enough people will copy you that it no longer actually works.

  43. Joe says:

    We just got done talking about the Efficient Market Hypothesis and the theory that the stock market is never wrong in a way detectable by ordinary humans.

    That’s true at the margin, not on average. The reason you can’t beat the market is because prices are already at their equilibrium levels – but the way they got there is by people making a profit from pushing them in the right direction.

    • pocketjacks2 says:

      People do beat the market all the time – that’s how prices maintain accuracy in the first place. It’s not like prices are starting from a zero point, nor become static once they reach the “correct” level.

      There’s a dynamic interplay between “you probably won’t beat the market because all material factors have been priced in already” and “the way we know prices are efficient and all material factors have been priced in, is because people have (very recently) beaten the market in order to get them here”. Personally, I’m a conservative investor, so if I wanted exposure to equity markets, I’d go with index funds all the way. But the only reason a strategy like mine can be effective is because of the efforts of market-beaters.

      • Joe says:

        Yep. The phrasing “the way they got there” wasn’t meant to imply that prices reach an equilibrium level and then remain static. They will keep changing as information changes – each piece of new info potentially means the stock is more or less valuable than previously thought, and so represents a new opportunity to profit from correcting its price.

  44. TheThirteenthSide says:

    Thanks for the post!

    First of all, maybe that’s wrong. My dad is a doctor, and he swears that he keeps making a lot of money from medical investments. He just sees some new medical product, says “Yeah, that sounds like the sort of thing that will work and become pretty popular”, and buys it. I keep telling him this cannot possibly work, and he keeps coming to me a year later telling me he made 300% returns and now has a new car.

    Great, now some smart guy who reads SSC and works at a hedge fund is going to implement this strategy and ruin it for your father!

    • pontifex says:

      I mean, if the hedgies are buying and holding the same stocks Scott’s dad is buying, wouldn’t that help Scott’s dad? “Buying and then just holding” doesn’t sound like an advanced strategy (unless you’re a bitcoin investor). 🙂

      • Yosarian2 says:

        The problem is, the hedges are faster; if they can figure out how to invest based on what Scott’s dad knows, they’ll get there first and snap up the underpriced stock. By the time Scott’s dad gets there he won’t be able to get the great deal anymore.

    • benquo says:

      No, they cannot, unless their sense of “that sounds like the sort of thing that will work and become pretty popular” is as good as Scott’s dad’s, which seems unlikely given that he is specifically trained in medicine and all his professional experience is in medicine, unlike the hedge fund guy.

  45. Nornagest says:

    I’d like this to be true as much as the next guy, but it smells of p-hacking to me.

    As to the Efficient Markets Hypothesis… who knows? But Neuralstem has a market cap of $30M, which is tiny by publicly traded company standards, so maybe it just hasn’t attracted the attention of the big institutional investors that can pay for a broad range of medical expertise. Your dad does enjoy some privileged information when it comes to medicine, and maybe that’s enough to arbitrage into the occasional new car if he can find a promising medical company without many eyes on it. Although if you take that at face value it implies that I should be investing in tech startups, and I have this nasty feeling that I’d lose my shirt if I tried that.

    • gbear605 says:

      Far more investors are interested in tech startups than in medical startups, or at least that what it seems like to me.

      • pocketjacks2 says:

        Everyone wants to be in on the next unicorn. You can create a tech startup for under $10,000 without any physical assets whatsoever other than your laptops. Hell, you can get 5% of their company by just giving them office space at an incubator, not even a proper seed investment. And then the hope is that one of these guys reaches $100M market cap from an initial investment of practically nothing.

        Medical startups need to start from a larger asset base and can fall back on other institutional partners than VC’s.

    • toastengineer says:

      I don’t think there’s anything about the efficient markets hypothesis that precludes people with decades of experience and training in medicine from making superior decisions in fields that are heavily influenced by the reality of medicine.

      I mean then again, I don’t see any reason why investment people wouldn’t just consult doctors…

      • Toby Bartels says:

        Some of them do, I’m sure.

        But even if they didn’t, the efficient market would still emerge, as long as somebody with expertise gets paid to tell it which new products are likely to do well. And if nobody else, that somebody is Scott’s dad; the market is paying him, and he is providing information with his buying and selling.

    • John Schilling says:

      The Efficient Market Hypothesis guarantees that a talented, knowledgeable, and diligent investor can beat the market. By a small percentage if they are trading in large-cap stocks, by a rather larger percentage if dealing in small-cap but in that case the capitalization limits the amount of money you can effectively invest, and in the end you’ll find that if you are diligent enough to spend 40+ hours a week on the task you will net roughly $200K/year (plus or minus roughly $300K/year).

      If it were not possible to meet this expectation, then that would mean the market had inefficiently sucked an excess of valuable labor into the task of smoothing out valuation errors well past the point of diminishing returns. In an efficient market, price signals instead send the excess analysts off to do other things more worthy of a talented, knowledgeable, diligent analyst’s time at about the $150K/year (but less variance) level.

    • RicardoCruz says:

      As to the Efficient Markets Hypothesis… who knows?

      Actually, Scott explained why such small companies are likely to be overvalued in a recent post:

      (…) Eliezer and his friends sometimes joke about how really stupid Uber-for-puppies style startups are overvalued. The people investing in these startups are making a mistake big enough for ordinary people like Eliezer to notice. But it’s not exploitable – there’s no way to short startups, so neither Eliezer nor anyone else can make money by correcting that error. So it’s not surprising that the error persists. All you need is one stupid investor who thinks Uber-for-puppies is going to be the next big thing, and the startup will get overfunded. All the smart investors in the world can’t fix that one person’s mistake.

      • rlms says:

        But the companies Scott’s dad invests in are presumably undervalued (at the time he invests), if he profits from an increase in their stock prices.

    • Yosarian2 says:

      A lot of economists favor the “weak” version of the efficient market hypothesis, which isn’t “nobody can beat the stock market”, it’s “investment firms usually won’t be able to beat the stock market by a wide enough margin to do more then cover the costs of paying for their stock traders.”

      It’s a subtle difference, but it’s a difference. So maybe a stock company could beat the market in medical stocks a little bit by hiring a bunch of medical experts, but the cost of hiring those experts would cost as much as they’d make in stock returns, so it’s not quite worth it to do much. But that does mean a medical expert who doesn’t charge himself for his own time and spends time and effort stock hunting and expertise might be able to beat the market a little.

      • My favorite explanation is to look at the economics of checkout lines in a supermarket. To first approximation they are all the same length, because if one is shorter the next person looking for a line goes to it. But if nobody makes any effort to check out all the lines, in particular look not only at how many people are in line but how much stuff is in their carts, then length in time won’t be quite even and there will be some payoff in time saved to checking out all the lines.

        How much payoff? If everyone calculates correctly, just enough payoff to make it worth the effort of just enough people checking lines to hold the differences in line time down to the difference that gives that payoff.

        For a longer explanation, see chapter 1 of Price Theory. Subtitle “Supermarket Lines”.

        • anonymousskimmer says:

          Interestingly enough, a couple decades ago when I was in a position to notice, I noticed that at least some of the US military Exchanges/Commissaries socialized this process: There is one line feeding all of the check-out counters, with the next person in line waved to the next available counter.

          Off-hand I recall Fry’s, Staples, Office Depot, and many convenience stores and restaurants also following this model, and wish more stores would. I imagine that enough other customers feel the same way that there is a net gain in customer satisfaction for those which do.

          I wonder why the Targets and Walmarts of the world don’t?

          • Protagoras says:

            Works better if you have 3 or 4 checkout counters than if you have 20 or 30.

          • anonymousskimmer says:

            The commissary generally had closer to 8 manned checkout counters.

            Fry’s uses lights over the checkout stations to indicate an open station. I’ve only ever seen 3 or 4 manned at the times I shopped, but there are at least a dozen stations down the line.

            You just need a properly gated line.

          • A Definite Beta Guy says:

            Bed Bath and Beyond does exactly this by me, with about 16 or 17.

            I believe the closest Whole Foods in West Loop Chicago also does this, with 6 or 7. Also, I want to get some brioche now….

  46. apollocarmb says:

    So much blue. It hurts my eyes to read.

  47. ndwignall says:

    “…depression has such a high placebo response rate that it’s hard for even a good medication to separate much from placebo.”

    Don’t take this the wrong way, but I feel like this sentence is more fascinating than the rest of the post. Specifically, what does this say about the nature of depression? What does it mean for how we conceptualize and treat it?

    Are you familiar with Eiko Fried’s work (https://ppw.kuleuven.be/okp/_pdf/Fried2014DINAC.pdf) suggesting that depression isn’t really one consistent syndrome or disease at all?

    Interesting stuff…

    • sandoratthezoo says:

      My father was a general practitioner and this is very much an anecdote, but he felt like from his experience with referring depressed people to psychiatrists and seeing them come back on SSRIs was that depressed people are terrible at rating their own symptoms.

      He’d see people who said, “I don’t feel like these drugs are helping at all,” and their family members were saying, “He’s a new man since he started taking the drugs!” And vice versa.

      If that’s true (big if), then it might not be so much that depression is actually that responsive to placebos, so much as that people are suggestible and if they have a lot of difficulty self-assessing their depression, the suggestion overwhelms the results. (I’m not suggesting this is generally how placebo works, just specifically to depression).

  48. baconbacon says:

    Part of it is that depression has such a high placebo response rate that it’s hard for even a good medication to separate much from placebo.

    Wait, why aren’t we just prescribing placebos to start with then?

    • Markus Ramikin says:

      You could say we kind of are?

      • johan_larson says:

        I expect there are some medications out there that aren’t very effective, but are cheap and have minimal side effects, that doctors prescribe mostly for the placebo effect. Also, the get-out-of-my-office effect.

      • bbartlog says:

        It’s called homeopathy!

    • Scott Alexander says:

      My impression is that the “placebo effect” for depression is primarily due to depression being a relapsing-remitting disease. It very often goes away on its own. There’s probably still some room for placebo to be helpful, but every treatment for depression – from antidepressants to therapy to bright light to exercise – gives you a bit of placebo, so it doesn’t seem that helpful making it a separate drug (even apart from the ethical issues).

      • gbear605 says:

        How much more effective is a placebo than doing nothing?

        • jeff daniels says:

          Scott has tackled this before (for example, this article links to plenty of nice meta-analyses), but the short answer is “usually and mysteriously, nearly as well as whatever anti-depressant it’s being tested against, however well that may be.”

          EDIT: A useful sentence (and attached caveat) here:

          According to McAllister and Williams, who are working off of slightly different data and so get slightly different numbers, the effect size of placebo is 0.92 and the effect size of antidepressants is 1.24, which means antidepressants have a 0.32 SD benefit over placebo

          Effect size is a hard statistic to work with (albeit extremely fun)

          Here’s the link to the study they’re talking about

      • baconbacon says:

        so it doesn’t seem that helpful making it a separate drug

        Unless depressed people have a hard time keeping doctor’s appointments, or those other treatments have side effects.

        • klfwip says:

          The doctor’s appointment itself is part of the effectiveness of a good placebo though. As is paying money and going to a pharmacy regularly.

          And besides this, there are some real benefits for severely depressed people. The very fact that you are able to do it at all shows you’re not a total wreck, having a routine makes you feel a bit more in control of the problem, the minimal social contact makes you a bit less isolated, the exercise has some small benefits.

          I would expect a placebo which was delivered with 1 year’s dose at a time without requiring an appointment or any other action on the patient’s part would be far less effective than placebos which closely mimic the actions one takes while using a supposedly effective drug.

          • Standing in the Shadows says:

            That is probably how various cultural, social, and religious practices for ministering to the sick worked, in cultures before they were invaded and infected by 21C “universal culture”.

      • nameless1 says:

        Article request: How can you tell you probably DON’T have depression?

        While you were hinting at depression being overdiagnosed, I suspect a strong cultural element in it. Maybe in California everybody expects to be super happy they go to the doc every time their life is less than perfectly happy. But imagine the average British small town where everybody is morose, constantly complaining etc. is it possible a lot of people are depressed there without knowing it? Parallel: people tend to think an alcoholic is someone who drinks more than the social norm. But that is of course not the medical definition, like, Moscow’s cultural norms are pretty permissive about this. Similarly there are cultural norms about how happy you can expect your life be. If you are able to hold a job and raise a family but there is no joy and no smile in it, just toughing it out, are you depressed? Effectively we need something like the opposite of the usual depression checklists, that lists what kind of positive things non-depressed people do. We sort of need some info what would be the normal life. For example, do non-depressed people have hobbies at 40 years old? If nobody above 30 you know has hobbies or interests does this suggest you live in a culture where depression is the norm?

        • danohu says:

          ” people tend to think an alcoholic is someone who drinks more than the social norm. But that is of course not the medical definition”

          Except..it kind of is, at least indirectly. DSM criteria for alcohol abuse are mostly about “is your drinking out of line with social expectations”:

          Recurrent alcohol use resulting in failure to fulfil major role obligations at work, school, or home (e.g., repeated absences or poor work performance related to substance use; substance-related absences, suspensions or expulsions from school; or neglect of children or household).
          Recurrent alcohol use in situations in which it is physically hazardous (e.g., driving an automobile or operating a machine).
          Recurrent alcohol-related legal problems (e.g., arrests for alcohol-related disorderly conduct).
          Continued alcohol use despite persistent or recurrent social or interpersonal problems caused or exacerbated by the effects of the alcohol (e.g., arguments with spouse about consequences of intoxication or physical fights).

          • Toby Bartels says:

            But none of that sounds like drinking more than the social norm to me. It sounds like drinking so much that it causes you problems, regardless of whether that much would cause problems for other people or whether other people drink that much.

    • Hyzenthlay says:

      Because if doctors started prescribing placebos, word would eventually get out that they were doing this, and everyone who went to the doctor for medicine would wonder whether they were getting the real deal or a placebo. This would make the placebos less effective, probably, and would also make people more reluctant to go to the doctor in general, because they might be paying $200 a month for a sugar pill.

      • baconbacon says:

        Why are doctors needed here?

        New off label use for FDA approved*, available with no prescription, demonstrated to reduce depression by X% in studies Q, R and S. Scientists have long been baffled by how this miracle pill works, and its yours for only 0.10 a pill!

        * sugar pill is approved for trial use, so….

      • Aapje says:

        @Hyzenthlay

        You just declare that it is a type of alternative medicine, calling it something like ‘homeopathy’ and make up some scientifically sounding, but nonsensical explanation how the midi-chlorians in the sugar pills heal people.

        The more difficult part is ensuring that people only turn to these pills when they have depression, but not when they have cancer.

        • Hyzenthlay says:

          True, in that sense it’s already happening all over the place, those things just aren’t being given to patients by actual doctors. (Unless you count SSRIs as placebos.)

          And yeah, that is a bit trickier. I guess regulatory bodies could take the approach of cracking down on any snake oil that claims to cure cancer, but turning a blind eye when the condition it claims to treat is one actually affected by placebos.

          • Aapje says:

            There are Dutch General Practitioners that do both homeopathy and real medicine. That may be preferable to having pure snake oil doctors, since those doctors have a license that may be withdrawn if they do silly stuff.

            Now that I think of it, photon therapy is also a form of alternative medicine. It’s putting people in front of a lamp. Seems like a good placebo therapy for depressed people 🙂

      • kronbergweb says:

        Real Coca Cola with sugar cane is $2. Immediate results.
        Original Coca Cola with cocaine might help; if you can get any — or make your own if you’re on that side of the street.

    • Garrett says:

      Imagine for a moment that your mood was rated somewhere between -10 (I’m too depressed to breathe) and 10 (I’m optimistic I can solve human flight after I jump off this bridge) with “normal” around a 0.

      Now imagine that an anti-depressant made it somehow that your mood could never drop below, say, a -5. That would be what a typical person would call “terrible” and a person with Major Depressive Disorder would call “morning”. So a typical person would almost never get to the point that the drug would have any beneficial effect. And the person with Major Depressive Disorder would just feel continually awful.

      But! The person with Major Depressive Disorder also won’t get worse, either. So they’ll still hate life, be bitter, sarcastic, and post on SSC all day. And at the same time, they’ll still be functional and won’t have episodes of “can’t be bothered to get out of bed” and hopefully won’t drop into suicide-is-a-good-idea territory, either.

      • Deiseach says:

        So they’ll still hate life, be bitter, sarcastic, and post on SSC all day.

        I came out to have a good time and I’m honestly feeling so attacked right now 🙂

        Now imagine that an anti-depressant made it somehow that your mood could never drop below, say, a -5. That would be what a typical person would call “terrible” and a person with Major Depressive Disorder would call “morning”.

        To be a bit more serious, I think Scott’s “depression probably regresses to the mean” idea would work as follows: if you’re scoring “On a scale of 1-10, how awful/good do you feel today?” and the depressed person scores over, say, a month an average 4-5, an ordinary person would say “that’s terrible, I have no joy in life any more, I can’t do anything because I have no energy” while the depressed person on a day that’s a 4 or 5 would say “Well, I’m not feeling like throwing myself off a bridge any more and I managed to make myself leave the house to do the grocery shopping, so today was a pretty good day”.

        If you’re a doctor treating the depressed person who is telling you “yeah, today was pretty good” then that sounds like “okay, they’re not depressed anymore, so I guess depression is a remitting disorder” and in one way, yes, that’s true. On the other hand, by normal standards, that’s not a great level to be operating at, so they’re still depressed. Average 4 out of 10 is not “happy happy joy joy” by any means, but neither is it “today was a 2 and I seriously am having thoughts about that bridge on my walk home from work”.

        So for a depressed person “this drug keeps me operating at an average 4-5” means it is effective, while for a normal person “this drug keeps me at 4-5 on a 1-10 happiness scale” makes it a placebo – surely if it worked, it would get you up to 6 or 7 at least!

      • baconbacon says:

        Imagine for a moment that your mood was rated somewhere between -10 (I’m too depressed to breathe) and 10 (I’m optimistic I can solve human flight after I jump off this bridge) with “normal” around a 0.

        Now imagine that an anti-depressant made it somehow that your mood could never drop below, say, a -5. That would be what a typical person would call “terrible” and a person with Major Depressive Disorder would call “morning”. So a typical person would almost never get to the point that the drug would have any beneficial effect. And the person with Major Depressive Disorder would just feel continually awful.

        If you are running a clinical trial against placebo this should show up, right? If the treatment prevents depressed people from going from -5 to -10, and placebo doesn’t then some of the depressed people getting placebo should fall from -5 to -10, enhancing the score of the drug vs placebo.

        • Protagoras says:

          If they can get significant numbers of people with extremely severe depression for the trial. Mild depression is a lot more common, and there’s also a general ethical tendency to prefer to give people with very severe conditions whatever the best treatment presently available is, rather than sticking them in clinical trials where they have a good chance of getting placebo and quite possibly dying (probably suicide in this case, or of whatever their condition is in other cases).

          • baconbacon says:

            You are still in this model presuming that minor depression doesn’t grow into major depression, is this supported? Are they distinct and non casual?

          • Garrett says:

            In most trials, the alternative isn’t placebo, it’s the standard of care. So for the first drugs to treat a particular condition, the alternative is placebo. But after other drugs are accepted, you usually have to show that your drug is better than those.

            There are some exceptions, like if you pick a narrow category of patients. Like “those who are allergic to the standard of care”. But that also limits what a drug will be approved for by the FDA.

    • baconbacon says:

      I was about to start a reply with the phrase “Placebos don’t have side effects”, but then I realized what a terrible assumption that is. So, do placebos have side effects?

    • temujin9 says:

      One of the best explanations I have heard for the placebo effect (for which I cannot remember source, sadly) is:

      Primitive human life was a dangerous situation, and many of the body’s healing mechanisms can cause or increase danger (e.g. fever sapping strength and energy that could be necessary for escape from a tiger). Being cared for is a useful pack animal signal that “this is a safe enough situation that my mates have extra energy and attention; do the dangerous healing thing”. Any medical practice (whether or not its independently effective) should have a positive effect on healing, over doing nothing. Probably the same could also be expected for folk remedies: hot soup and lots of blankets, and so on.

      To further engage in wild speculations:

      It would also seem to be a partial explanation for why the biggest effect size observed in psychiatry is that the most effective therapy modality is the one the therapist prefers. If the key there is “convincing the mind it’s safe to heal”, therapist confidence is probably a huge component.

      Which might also provide some stronger understanding of the “healing power of religion” that so many believers appreciate innately, and so many non-believers struggle to accept. Rituals in general seem to be very good at providing a sense of stability and safety for their adherents . . . if the ritual fits their world-view.

    • Dan Fitch says:

      https://www.smithsonianmag.com/science-nature/why-i-take-fake-pills-180962765/

      Yes, you can get prescribed placebos. Yes, how your doctor does it theoretically matters. Yes, you know they are a placebo. No, I don’t know how well tested any of this is. But it’s pretty hilarious!

      • A Definite Beta Guy says:

        I was going to say, I was pretty damn sure we do prescribe placebos. I have several family members that have worked in pharmacies, and they have all mentioned stories about placebo pills in the past.