Not a social justice blog! Stop only reading my posts about social justice!

Nootropics Survey Results And Analysis

In case you are just joining us: nootropics are substances which purportedly improve mental functioning with relatively few side effects when used responsibility.

Caffeine is a good example. It improves wakefulness and energy, and when used responsibly is so safe it can be sold “over the counter” in coffee and other drinks.

Over the past few years, various substances from experimental drugs to common supplements have been claimed to satisfy these criteria. If they really work, obviously we want to know about it.

Double-blind randomized controlled trials have confirmed exciting properties for some purported nootropics. A meta-analysis has confirmed that the neuroprotective drug piracetam improves cognition in demented patients, and smaller studies suggest it does the same in healthy individuals. A meta-analysis of traditional Ayurvedic medicine bacopa monnieri finds it improves memory and reaction time in healthy subjects.

The online nootropics community consists of mostly healthy individuals looking to use these substances to gain motivation or clearer thinking. They tend to “outpace” the scientific literature, using some substances that haven’t yet been shown to work, or haven’t been shown to benefit healthy individuals. Information travels mostly through word of mouth, and people combine a bewildering variety of chemicals without being sure which ones work better than others or even whether any of them work at all.

This survey does not intend to rectify this state of affairs, but it can at least shed some light on it.

I asked people in the two largest online nootropics communities (that I know of), Reddit’s r/nootropics and Longecity’s Brain Health forum, to take a survey describing their use of 31 different substances. I got 162 responses. My respondents were 92% male and 8% female, with an average age of 25. Most of them had tried only a few different chemicals, but a few had sampled nearly two dozen. Most were taking “stacks” of several different chemicals daily.

Three substances – DMAE, centrophenoxine, and a combination stack called TULIP – got fewer than ten responses, so I removed them from the survey because there wasn’t enough of a sample size to do anything with.

Below I describe the results I got on the remaining 28 substances and some hypotheses that I tested:

Head-to-Head Comparison Of Nootropics

Respondents were asked to rate their subjective experiences of different nootropics on a scale of zero (completely useless, did nothing) to ten (life-changing). Comments on Reddit suggest several people misunderstood the scale, but not much I can do about that at this point.

I asked people to reserve numbers five or above for effects that, in their subjective opinion, couldn’t possibly be placebo – but the whole point of the placebo effect is that people can’t consistently make this judgment. I would have liked to have some placebo nootropics to use as a comparison, but since I wasn’t running a trial, I didn’t have that option.

I did, however, try to assess the placebo effect in a couple of sneaky ways. One of the nootropics measured, choline bitartrate, has carefully regulated ability to cross the blood-brain barrier beyond a certain level, and so it would be surprising if it had direct and immediate cognitive effects. Nevertheless, 27 respondents had used this substance and they gave it an average rating of 3.7 out of ten. This suggests that 3.7 out of ten is a good first guess for the placebo effect.

Another of the nootropics measured, bacopa monnieri, has been found in multiple studies not to exert any effect before a month of consistent use. 6 respondents had used this substance for 20 days or less, and they gave it an average rating of 4 – which is very close to 3.7 and so confirms our previous guess.

These methods aren’t perfect – there are some mechanisms by which choline can relieve deficiency even in the absence of direct cross-BBB transport, and some people say that at least some anxiolytic effects of bacopa are immediate – but they’re the best we’ve got. So let’s say anything that comes up 4 or less has a good chance of being placebo.

This does not imply that numbers above 4 are therefore necessarily real effects; we know that more promising drugs exert stronger placebo effects. Thus, in the early days of antidepressants when no one believed they would work, they had a very weak placebo effect, whereas after the culture was saturated with messages of antidepressant efficacy, they started having a much stronger placebo effect. So a much-vaunted nootropic might well be able to muster a placebo effect somewhat greater than 4.

Below, I give tables ranking the nootropics studied along several axes.

The mean and median ratings for each substance

The percent of respondents who rated each substance as >=5 (ie they could not believe it was placebo), and = 10 (ie life-changing)

The number of respondents (out of 162) who used each substance, and the mean number of times they have used it.

I did not include standard deviations in these tables, so they conceal the true amount of variation in each substance. In general, nearly every substance had at least one zero and at least one nine or ten. Very few showed an obvious consensus on whether they worked or not. I reproduce below a graph of piracetam – one of the most-used nootropics – as a very typical example:

The results themselves were not surprising. The most effective nootropics were caffeine – which pretty much everyone in the world knows works – and modafinil (along with its fellow -afinils) which has been proven to work by countless studies and is a prescription drug much used by the medical establishment.

Below these two, the most effective “exotic” nootropics were phenylpiracetam and coluracetam. I suspect that despite the nootropic community’s enthusiasm about phenylpiracetam it mostly just works as a mild stimulant not too different from other stimulants like amphetamines. Coluracetam, a very new and pretty unique substance currently being investigated as an Alzheimer’s treatment, is more exciting.

Of the next two, Noopept is a Russian prescription drug mostly unknown in the United States; Russian studies suggest it improves memory and learning. Theanine is one of the active ingredients in tea and is widely considered calming. Their success should not surprise anybody paying attention.

Creatine is a chemical related to amino acids. Although it seemed to do well, most people who ranked it highly mentioned in the comments that they liked it as an exercise supplement but didn’t get any cognitive benefits from it. It should probably be disqualified.

Ayurvedic medicines had a decidedly blah performance. A lot of people swear by rhodiola, ashwagandha, and bacopa, and they all seem to have a moderate amount of experimental support. But they were firmly in the middle of the pack here, outperforming our 4 threshold for placebo but only barely. Much like piracetam above, these were all over the place: here is the graph of rhodiola, the best-performing of the bunch:

An optimist might say that everyone has a unique biochemistry. A pessimist might say that everyone has a unique ability to extract patterns from random noise. I leave the choice between these two hypotheses to the reader.

Two other traditional Eastern medications – ginseng and gingko – fell completely flat, with both doing (nonsignificantly) worse than our suspected placebo level.

Dosing Issues

A secondary goal of this survey was to inspect the dose-response relationship of these substances and see whether the recommended doses were really set at the most effective level. My original plan was to report the dose at which a substance was most effective, but this proved beyond my technical abilities. Instead I report the median dose given by all users of a substance:

In milligrams, except where it isn’t. Vitamin D is IUs.

I investigated dosing of two substances in more detail – piracetam and vitamin D – because I had good data on both of them and there has been some controversy on correct dosing.

The median dose of piracetam was 1600 mg. I ran a t-test to see whether people who took high doses (>1600) had stronger effects than those who took low doses (< 1600) - the means looked different (5.6 vs. 4.4) but the effect was only marginally significant (p = .09).

I tried to see if I could break the significance barrier with a larger dichotomy and so compared people taking below the 25th percentile (1200) to those taking above the 75th percentile (3000). This actually had less difference – means of 4.42 vs. 4.88 – so I gave up. An attempt to correlate dose to response with Pearson’s r also returned uninspiring results of r = -.04, p = .75.

The whole analysis doesn’t really give any good reason to use anything other than the median dose of 1600 mg.

Vitamin D was much more interesting. I compared people taking more than the Vitamin D mean dose of 3000 IU with those taking less – the means seemed to differ (5.0 vs. 3.9), but the significance wasn’t there (p = .17).

Once again I tried to gun for significance by switching to quartiles – in this case, below 1625 IU vitamin D vs. above 5000. The means widened to 5.3 vs. 3.4, and the p value reached 0.8, within a hair-breadth of significance.

I decided to make a last desperate effort and restrict the analysis to the true extremes. I compared the people taking OVER NINE THOUSAND units to those taking 1000 or less. Here the means were 7.4 vs. 3.1, and p was 0.02. I am not entirely confident in this analysis since there were some small subgroups – n = 12 in the very low group and n = 5 in the very high.

Also, it should be kept in mind that these correlations might not be causal in the expected direction – just to raise one possibility, maybe people who really like vitamin D take more of it.

I investigated a bunch of other substances for dose-response relationships using bivariate correlations and couldn’t find anything. Overall my attempts to investigate dosing were a disappointment.

Correlations

My first several attempts to correlate different nootropics with each other were failures. Everything correlated with everything else due to a general “low standards factor” – some people were just very effusive and willing to give high ratings to everything, while other people had a much higher threshold before getting excited.

I did something very sketchy to adjust for this, and divided every person’s ratings for each substance by their average rating for all substances, so that each substance’s “rating” was transformed into the degree for which it was below or above average for that particular rater. This introduced a whole host of complicated biases, but it at least got me to the point where I could run correlations between substances without everything lighting up simultaneously.

By correlating 28 substances with one another I ran 378 separate correlations, so with a significance threshold of p = .005 I would expect about two hits by chance. Instead I got about a dozen.

Some of these were expected – for example, the -afinils correlated with each other. Others were bizarre and nonsensical. I have removed the inter-afinil correlations as too obvious to be worth noting, and am left with the following:

ALCAR -> Modafinil -.733, .000
Vitamin D -> Theanine +.591, .000
Creatine -> Noopept -.629, .001
Vitamin D -> Rhodiola -.634, .002
Piracetam -> Sulbutiamine +.683 .002
Piracetam -> ALCAR +.548 .003
Piracetam -> Modafinil -.505 .003
Phenylpiracetam -> Choline -.681 .004
Huperzine -> Creatine -.810, .005
Gingko -> Piracetam +.602 .005

Bizarrely, none of the racetams correlated with each other!

I have very low confidence that these represent real effects rather than artifacts of the gymnastics I had to do to make the data correlatable at all, and I report them only to encourage other people to do more sober analyses.

Odds and Ends

Choline is a commonly discussed nootropic used both on its own and as an adjunct to other substances. There are four main kinds of choline – lecithin, choline bitartrate, CDP-choline, and alpha-GPC choline. The first two do not effectively cross the blood-brain barrier beyond a certain level and so should theoretically be pretty ineffective for nootropic purposes. The second two do directly cross the blood-brain barrier and may have an effect. When I asked people to rate their choline, I got the following means:

Choline bitartrate: 3.7 (n = 27)
Lecithin: 3.8 (n = 5)
Alpha-GPC choline: 5.07 (n = 15)
CDP-choline: 5.37 (n = 19)

I ran a t-test comparing the two presumably ineffective forms of choline to the two potentially neuroactive ones and found a significant result (p = .045). This suggests that the forms of choline that could theoretically work are more effective than than ones that don’t – or that people are aware of the chemical properties of their forms of choline and adjusting their placebo effects accordingly.

Choline is sometimes used to improve the effects of -racetams, so I investigated whether people who took active forms of choline had better experiences with racetams than those who took inactive forms of choline. Out of six racetams, none of them achieved significance (although five out of the six had trends in the expected direction). This stayed pretty much the same when I restricted the analysis to people who told me specifically that they used choline with their racetams.

Choline is also sometimes used to prevent headaches with racetams, so I asked people about that. 40% said they found choline to help prevent headaches with racetams, 26% said they found it didn’t, and 34% just took choline because everyone else told them they should and didn’t know one way or the other.

The other category of drugs I looked at in more detail were the afinils. There are three common afinils. Modafinil is a well-known prescription medicine used by doctors as a wakefulness-promoting agent to treat narcolepsy, shift-work sleep disorder, and a couple other diseases. Adrafinil is an over-the-counter substance that gets metabolized to modafinil and should have about the same effecst. Armodafinil is an isomer of modafinil that should do about the same thing but require a lower dose.

I asked respondents if they had any preferences among these three similar substances. Of those willing to compare adrafinil and modafinil, 50% said modafinil was better, only 8% thought adrafinil was better, and 42% didn’t want to pick a favorite.

Of those willing to compare modafinil and armodafinil, 62% thought armodafinil was better, 23% thought modafinil was better, and 15% didn’t want to pick a favorite.

There is some theoretical concern about liver-related side effects from adrafinil, but some pharmacologists say these are overblown and that its liver profile is similar to Tylenol – ie don’t take a massive overdose on it or use it every day for years and you’ll be fine. None of my respondents reported ever having any liver problems with adrafinil – not even asymptomatic elevation of liver enzymes – but n = only 17 so the results don’t rule out even moderately common adverse reactions.

Last of all, there are some worries about tolerance to modafinil. Although the medical community is happy to give it to patients for several years straight and expects its results to remain just as strong as when it was first prescribed, the nootropics community has a lot of anecdotal evidence of tolerance developing if it is used too much.

I compared the modafinil experience ratings of people who used the substance occasionally (n = 52) to people who used it every day (n = 7). This corresponded, as expected, to a wide gap in how many doses they had taken on average – the occasional users had taken it 39 times, compared to the everyday users’ 184. However, the everyday users did not report decreased satisfaction with modafinil – both groups gave it an average satisfaction rating of 7.4. This suggests that at least this small subgroup of 7 people are not experiencing tolerance. We should, however, be careful to worry that they might be self-selected for exactly that quality – maybe a lot people try to use it everyday, but all but a rare group of non-tolerance-developers later back off.

Conclusions and Future Research

Overall I was not too impressed with the results of this survey. No unexpected substances jumped out as promising, there was a very wide distribution from “useless” to “life-changing” in nearly everything except the -afinils, and it was very hard to find much of a dose-response relationship anywhere. There is no “smoking gun” for most these substances being definitely-exciting, aside from the ones like modafinil that we were already aware of.

I repeat the other survey I run yearly and I’d like to establish a similar tradition here. Next year I hope this survey will have earned me enough cred in the nootropics community to be able to harass more people into taking it – whether that means getting it stickied on top of appropriate forums, getting nootropics distributors to advertise on their sites, or just getting more word of mouth. I know there are several thousand people in the community and I feel like I could do a lot more if I had two or three times as many responses.

Other goals for the future are to drag people kicking and screaming into reading directions so I know everyone’s rating off the same scale, to ask more specifically about tolerance development, and perhaps to separate main effects from side effects (although that brings up a lot of complications I will have to sort through).

Below are the raw data available for download, minus about a dozen people who asked for their entries to stay private. I am making it available in two forms. First are the raw raw data, which include people’s comments, people’s stacks, and a whole bunch of incorrectly entered machine-unreadable responses that I had to go through by hand and correct ಠ_ಠ . The sanitized raw data are clear, pure numbers that you can get to work on right away, if you don’t mind the fact that I deleted a lot of ambiguous responses and removed some stuff entirely just because keeping it seemed like a lot of work. I put a very brief explanation of how I coded some bits of the sanitized data up here; otherwise you’re on your own or can ask me questions if it’s too confusing.

I know Gwern is hoping to analyze these data further and I look forward to seeing what other people can make of them.

Raw data: (.xls) (.csv)
Sanitized data: (.xls) (.csv)

This entry was posted in Uncategorized and tagged , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

38 Responses to Nootropics Survey Results And Analysis

  1. Anonymous says:

    What supplements and nootropics do you Mr. Scott Alexander use yourself?

    Report comment

  2. Mordanicus says:

    Interesting study. But what we really need is the following thing: a nootropic which can be given to pregnant women, and which would raise the IQ levels of the children born after this treatment.

    Report comment

  3. Tenoke says:

    > Next year I hope this survey will have earned me enough cred in the nootropics community to be able to harass more people into taking it – whether that means getting it stickied on top of appropriate forums, getting nootropics distributors to advertise on their sites, or just getting more word of mouth.

    If you care about the number of participants then why did you only let it run for a couple of days? Were the respondents really dying out that quickly?

    Also I found the choice of nootropics questionable and the instructions confusing enough to mess results up to some extent, but I believe people have pointed that out to you already.

    Sorry if I sound negative, I just expected a bit more.

    Report comment

    • Scott Alexander says:

      I only got two responses in the last 24 hours, and the thread about it dropped off the front page of r/nootropics. I figured the chance of getting more than about one more response per day was minimal.

      I asked people for advice on choice of nootropics, waited the better part of a week for answers, and got nothing. I can only consider suggestions people give me.

      Report comment

      • zc says:

        We can sticky it next time you want to run it. The only issue is technical, reddit only allows text posts to be stickied. If you link to the survey in a self-post we could sticky it for at least a week or so.

        Report comment

  4. Multiheaded says:

    I use phenylpiracetam semi-regularly, as it’s like $10/gram over the counter here; I’ve found that following a 200mg dose up with an energy drink after 4-5 hours, as the unpleasant comedown effects begin to settle in, mostly negates them (as does chewing nicotine gum) – actually, if I take a week’s break, phenylpiracetam + lots of caffeine gives me an incredible speed high for hours! I was just amazed the first time it happened. A shame that more frequent usage degrades this effect.

    Report comment

  5. DefectiveAlgorithm says:

    I know of at least one fan of nootropics who claims that while the racetams are beneficial in most contexts, they’re actually detrimental when it comes to the sorts of tasks which require large amounts of creativity and drawing on concepts from multiple fields (ie undertakings that push the boundaries of our current knowledge). Is there any data on anything like this?

    Report comment

    • gwern says:

      Creativity is, understandably, even harder to study than stuff like working memory… I don’t know of anything for the racetams other than random speculation about brain hemispheres and blood flow. For modafinil, there’s one or two inconsistent studies on benefits.

      Report comment

  6. Have you looked into animal fats as nootropics?

    http://blog.sethroberts.net/2013/06/29/mental-effects-of-butter-my-video-gaming-is-better/#comments

    Seth Roberts has done experimentation (testing the ability to do simple arithmetic quickly) and found improvements from various animal fats, especially butter.

    Report comment

  7. suntzuanime says:

    In that second chart, do you mean percent or fraction? It says percent, but in that case it seems like no drug has more than 1% supporters, so fraction seems more likely.

    Report comment

  8. A. says:

    For your next survey also add Tyrosine (amino acid). I have found that it works, but reaches tollerance very fast. So it’s good for special situations, and not for everyday use.

    Report comment

  9. d says:

    F@!# nootropics! end the drug war so human kind can have a responsible relationship with the PLANTS that started all medicine.

    Report comment

  10. Luke Stanley says:

    Nice work doing this. Looking forward to the next study!

    Report comment

  11. Chuck says:

    I’m a late middle-aged male. I started taking Piracetam in 2002, and kept taking it, 3 x 800 mg, until the beginning of 2009, as I felt it helped me reduce my ritalin intake (I was diagnosed with ADD way before it was common). When taking it, I felt I experienced a noticeable sharpening of intellect and, interestingly, higher levels of motivation / focus. When I was off it for more than several days, I experienced a noticeable drop in mental acuity.

    After a year off due to an unrelated health problem, I started back in during mid 2010, and have found that steady use doesn’t yield the same results I remember earlier.

    If I only take it maybe once or twice a week, I sometimes see an effect, however it’s hit or miss these days. Maybe it’s age, maybe it’s neurological changes brought on by the drug, who knows. I miss the benefits I used to enjoy using the nootropic.

    Report comment

  12. Ryan Carey says:

    excellent survey and discussion.
    1 – “and the p value reached 0.8, within a hair-breadth of significance.” – you mean 0.08?
    2 – Some senior medical doctors have also described modafinil tolerance to me with increasing dose requirements over 5 years or 10 years. The best anecdotes are going to come from sleep specialists, and I haven’t talked to any of them in a while.

    Report comment

  13. Kulogg says:

    I think suggestions were necessary to be asked for, but you probably should have checked the scientific world as you missed some key nootropics, like nicotine for example. There’s more research on the effectiveness of that drug than any other.

    Report comment

  14. Sol Orwell says:

    Interesting stuff mate. Shared it, hope it gets more exposure.

    Report comment

  15. IamMarktoo says:

    For creativity, getting in the zone while painting, I’ve found a small amount of cannabis to be a great aid. It seems from my reading that great art, literature, etc… is often produced by those with various addictions. I’ve also found alcohol in mild quantities helps with creative thinking. Of course this is all very anecdotal.

    Report comment

  16. Aaron Black says:

    Thank you for this excellent survey. I definitely would paint this an accurate measurement of my own personal experiences with these nootropics. Great stuff, its nice to see what other people are rating. The only disagreement I would have is the high rating of l-theanine, didn’t do much for me maybe I should try a higher dosage

    Report comment

  17. Great information. May we repost this on our blog?

    Report comment

  18. Andrés Gómez Emilsson says:

    I strongly applaud this effort! Thank you!

    If I have time I will look at the data and perform other statistical analysis that may be fitting for this dataset.

    But something does stand out: You mentioned that there was this general factor of correlation between all nootropics… obviously caused by people’s different standards for how to answer. You rightly pointed out that it was sketchy to divide by the average rating of the user. This is better than nothing, but not the right way to do it. You can instead use random and fixed effects in generalized linear models. This amazing and super useful… as it allows you to control for individual differences in response patterns (even account for non-linear scale differences – e.g. if someone interprets the scale to be logarithmic rather than linear).

    Check out the lmer function in the lme4 package for R, and then follow some examples. E.g. : http://anythingbutrbitrary.blogspot.com/2012/10/hierarchical-linear-models-and-lmer.html#ref

    For future surveys, make sure to add a line such as: “Choose the single best nootropic you’ve tried and give it a rating of 10, and choose the worst and give it a rating of 0.” That way you attenuate the problem.

    Report comment

    • gwern says:

      This amazing and super useful… as it allows you to control for individual differences in response patterns (even account for non-linear scale differences – e.g. if someone interprets the scale to be logarithmic rather than linear).

      I’ve used lme4 before a few times*, but always in a relatively simple nested design, rather than the ‘wide’ format of this survey…

      I spent the last 2 days working on it, and I think I’ve finally got it right. (I took a long detour into using the eRm library to try to apply item-response models directly instead of googling to see how someone would use lme4 for a Rasch model; turns out the models don’t like being fed sparse data with lots of NAs and unrepresented levels, and would have preferred the most obscure nootropics like hordenine to be omitted entirely.) An hour or two of struggling with reshape later… It looks like the subject-specific effects don’t change the nootropics estimates very much, even though subjects did vary a lot in their default rating level. (Nootropic random effects, subject) That said, I don’t understand how I would use lme4 to correct for scale differences as you imply: all the coefficients are the usual intercept+beta stuff, no?

      Anyway, my analysis (better formatted copy of the R code):


      library(reshape)
      library(lme4)
      nootropics <- read.csv("http://dl.dropboxusercontent.com/u/182368464/2014-ssc-nootropicssurvey.csv")

      # not going to mess with doses or schedules: just want the ratings of the most-popular nootropics
      nootropicsResponses <- with(nootropics, data.frame (AdrafinilExperience=AdrafinilExperience, AniracetamExperience=AniracetamExperience, ArmodafinilExperience=ArmodafinilExperience, AshwagandhaExperience=AshwagandhaExperience, BacopaExperience=BacopaExperience, CILTEPExperience=CILTEPExperience, CaffeineExperience=CaffeineExperience, CentrophenoxineExperience=CentrophenoxineExperience, CholineExperience=CholineExperience, ColuracetamExperience=ColuracetamExperience, CreatineExperience=CreatineExperience, DMAEExperience=DMAEExperience, GingkoExperience=GingkoExperience, GinsengExperience=GinsengExperience, HordenineExperience=HordenineExperience, HuperzineExperience=HuperzineExperience, LionapossManeExperience=LionapossManeExperience, MCTOilExperience=MCTOilExperience, ModafinilExperience=ModafinilExperience, NoopeptExperience=NoopeptExperience, OxiracetamExperience=OxiracetamExperience, PhenylpiracetamExperience=PhenylpiracetamExperience, PiracetamExperience=PiracetamExperience, PramiracetamExperience=PramiracetamExperience, RhodiolaExperience=RhodiolaExperience, SulbutiamineExperience=SulbutiamineExperience, TULIPExperience=TULIPExperience, TheanineExperience=TheanineExperience, VinpocetineExperience=VinpocetineExperience, VitaminDExperience=VitaminDExperience ))
      # convert each of the '*Experience' columns into a single level of a generic 'Nootropic' factor
      long <- reshape(nootropicsResponses, idvar = "Subject", v.names="Response", timevar="Nootropic", times=names(nootropicsResponses), varying=list(names(nootropicsResponses)), direction = "long")
      summary(long)
      Nootropic Response Subject
      Length:4620 Min. : 0 Min. : 1.0
      Class :character 1st Qu.: 3 1st Qu.: 39.0
      Mode :character Median : 6 Median : 77.5
      Mean : 5 Mean : 77.5
      3rd Qu.: 8 3rd Qu.:116.0
      Max. :10 Max. :154.0
      NA's :3561

      # non-nested design: we're examining a crossover design of Subject x Nootropics, not a nested design like Subject(Nootropics).
      # Nootropics is a random effect: each level/nootropics gets its own estimate; and likewise, each Subject gets their own estimate
      # see http://www.jstatsoft.org/v20/i02/paper "Estimating the Multilevel Rasch Model: With the lme4 Package" for another example
      lmr1 <- lmer(Response ~ (1|Subject) + (1|Nootropic), data=long); summary(lmr1)

      ...REML criterion at convergence: 4875

      Random effects:
      Groups Name Variance Std.Dev.
      Subject (Intercept) 2.98 1.73
      Nootropic (Intercept) 1.24 1.11
      Residual 4.42 2.10
      Number of obs: 1059, groups: Subject, 150; Nootropic, 30

      Fixed effects:
      Estimate Std. Error t value
      (Intercept) 4.995 0.271 18.4

      # do Subjects really vary enough to justify 154 random effects rather than 1 fixed effects?
      lmr2 Chisq)
      lmr1 4 4882 4902 -2437 4874
      lmr2 4 5162 5182 -2577 5154 0 0 1

      # extract the random effects for both Subjects & Nootropics
      rr <- ranef(lmr1, postVar=TRUE)

      # visualize the two sets of random-effects
      qq <- qqmath(rr)
      qq$Nootropic
      # http://i.imgur.com/PmrHwqA.png
      qq$Subject
      # http://i.imgur.com/a0nJiOk.png

      # view nootropics sorted by coefficient size
      noots <- rr$Nootropic
      noots[order(noots$'(Intercept)'), , drop = FALSE]
      (Intercept)
      GingkoExperience -1.85454
      DMAEExperience -1.84701
      GinsengExperience -1.13313
      CholineExperience -1.08120
      HordenineExperience -0.90128
      VinpocetineExperience -0.87606
      HuperzineExperience -0.72931
      LionapossManeExperience -0.72782
      VitaminDExperience -0.51186
      MCTOilExperience -0.46230
      BacopaExperience -0.41610
      CentrophenoxineExperience -0.33918
      CILTEPExperience -0.26260
      PiracetamExperience -0.10523
      SulbutiamineExperience -0.02941
      OxiracetamExperience 0.08505
      AshwagandhaExperience 0.09323
      AniracetamExperience 0.10354
      NoopeptExperience 0.16937
      CreatineExperience 0.29738
      PramiracetamExperience 0.36177
      RhodiolaExperience 0.44556
      PhenylpiracetamExperience 0.50122
      AdrafinilExperience 0.59758
      TheanineExperience 0.63526
      ColuracetamExperience 0.69391
      TULIPExperience 0.80629
      CaffeineExperience 1.58431
      ModafinilExperience 2.39321
      ArmodafinilExperience 2.50935

      # what does the alternate version look like?
      rr2 <- ranef(lmr2, postVar=TRUE)
      noots2 <- rr2$Nootropic
      noots2[order(noots2$'(Intercept)'), , drop = FALSE]
      (Intercept)
      GingkoExperience -1.399810
      GinsengExperience -1.188626
      CholineExperience -0.978048
      DMAEExperience -0.954214
      VitaminDExperience -0.901817
      LionapossManeExperience -0.662886
      VinpocetineExperience -0.611086
      MCTOilExperience -0.457433
      HuperzineExperience -0.437104
      BacopaExperience -0.320166
      HordenineExperience -0.241617
      CentrophenoxineExperience -0.233822
      PiracetamExperience -0.148546
      AshwagandhaExperience -0.135738
      SulbutiamineExperience -0.015350
      CILTEPExperience 0.003327
      CreatineExperience 0.012964
      TULIPExperience 0.144545
      RhodiolaExperience 0.207083
      AniracetamExperience 0.326791
      PramiracetamExperience 0.354544
      OxiracetamExperience 0.374706
      NoopeptExperience 0.407759
      TheanineExperience 0.515546
      PhenylpiracetamExperience 0.568206
      AdrafinilExperience 0.594847
      ColuracetamExperience 0.778954
      CaffeineExperience 1.234636
      ArmodafinilExperience 1.456277
      ModafinilExperience 1.706078

      # What did correcting for subject-specific ratings buy us?
      # How much did nootropic rankings or estimates change from
      # the alternative?
      subjectDelta <- rr$Nootropic - rr2$Nootropic
      sum(subjectDelta$'(Intercept)')
      [1] 1.767e-12
      subjectDelta[order(subjectDelta$'(Intercept)'), , drop = FALSE]
      (Intercept)
      DMAEExperience -0.892796
      HordenineExperience -0.659664
      GingkoExperience -0.454732
      HuperzineExperience -0.292204
      OxiracetamExperience -0.289652
      CILTEPExperience -0.265925
      VinpocetineExperience -0.264976
      NoopeptExperience -0.238387
      AniracetamExperience -0.223253
      CentrophenoxineExperience -0.105358
      CholineExperience -0.103155
      BacopaExperience -0.095930
      ColuracetamExperience -0.085046
      PhenylpiracetamExperience -0.066984
      LionapossManeExperience -0.064930
      SulbutiamineExperience -0.014057
      MCTOilExperience -0.004862
      AdrafinilExperience 0.002729
      PramiracetamExperience 0.007223
      PiracetamExperience 0.043313
      GinsengExperience 0.055499
      TheanineExperience 0.119715
      AshwagandhaExperience 0.228967
      RhodiolaExperience 0.238472
      CreatineExperience 0.284411
      CaffeineExperience 0.349678
      VitaminDExperience 0.389960
      TULIPExperience 0.661741
      ModafinilExperience 0.687129
      ArmodafinilExperience 1.053072
      # so the biggest difference was DMAE falls by +1 rating
      # and Armodafinil looks better by +1 (and Modafinil improves too)

      Sort of an odd setup, interpretation-wise: a negative coefficient here means simply that something was below the average (the intercept of 4.995 or 5), not that it was useless or harmful.

      * in http://www.gwern.net/Google%20Alerts / http://www.gwern.net/Lewis%20meditation / http://www.gwern.net/LSD%20microdosing / http://www.gwern.net/Lunar%20sleep / http://www.gwern.net/Weather

      Report comment

      • gwern says:

        Since I had that analysis of just the ratings done, a thought kept nagging at me: ‘what about the subjects’ age, dose, and history? can’t we throw those in as covariates as well? it can’t be that big a change to the reshape code…’

        Well, as usual it wasn’t as easy as I hoped, but I think I got it right (source):


        library(reshape)
        library(lme4)
        nootropics <- read.csv("http://dl.dropboxusercontent.com/u/182368464/2014-ssc-nootropicssurvey.csv")
        nootropics$Timestamp <- NULL; nootropics$Directions <- NULL # unnecessary
        nootropics$CholineHistory <- NA # omitted from original but absence breaks symmetry of the triplet of fields for each nootropic

        # it's simpler to grep over the column names than list each and every of the 31 columns by hand
        # and we need to sort because having to add 'CholineHistory' screws up the otherwise-sorted-hardwired-columns
        experiences <- sort(Filter(function (x) {grepl("Experience", x)}, names(nootropics)))
        doses <- sort(Filter(function (x) {grepl("Dose", x)}, names(nootropics)))
        histories <- sort(Filter(function (x) {grepl("History", x)}, names(nootropics)))
        long <- reshape(nootropics, idvar = "Subject", timevar="Nootropic", v.names=c("Response", "Dose", "History"), times=experiences, varying=list(experiences, doses, histories), direction = "long")

        lmr1 <- lmer(Response ~ (1|Subject) + (1 |Nootropic), data=long)
        lmr2 <- lmer(Response ~ (1|Subject) + ( Dose |Nootropic), data=long)
        lmr3 <- lmer(Response ~ (1|Subject) + ( History|Nootropic), data=long)
        lmr4 <- lmer(Response ~ (1|Subject) + ( Dose+History|Nootropic), data=long)
        lmr5 <- lmer(Response ~ Age + (1|Subject) + ( Dose+History|Nootropic), data=long)
        lmr6 Chisq)
        lmr1 4 5036 5056 -2514 5028
        lmr2 6 3555 3583 -1772 3543 1484 2 <2e-16
        lmr3 6 3854 3883 -1921 3842 0 0 1
        lmr4 9 2893 2933 -1437 2875 967 3 <2e-16
        lmr5 10 3006 3050 -1493 2986 0 1 1
        lmr6 13 3064 3121 -1519 3038 0 3 1

        summary(lmr4)
        ...
        Random effects:
        Groups Name Variance Std.Dev. Corr
        Subject (Intercept) 6.29e+00 2.507269
        Nootropic (Intercept) 4.29e+00 2.070161
        Dose 6.97e-08 0.000264 -1.000
        History 4.39e-06 0.002096 -0.797 0.797
        Residual 3.18e+00 1.783544
        Number of obs: 639, groups: Subject, 121; Nootropic, 27

        Fixed effects:
        Estimate Std. Error t value
        (Intercept) 5.47 0.34 16.1

        rr1 <- ranef(lmr1, postVar=TRUE)
        rr4 <- ranef(lmr4, postVar=TRUE)
        noots4 <- rr4$Nootropic
        noots4[order(noots4$'(Intercept)'), , drop = FALSE]
        (Intercept) Dose History
        GingkoExperience -3.11497 3.973e-04 3.884e-03
        DMAEExperience -2.32433 2.964e-04 1.666e-03
        VitaminDExperience -2.15718 2.751e-04 4.044e-04
        ALCARExperience -1.44127 1.838e-04 1.150e-03
        PiracetamExperience -1.36419 1.740e-04 1.917e-03
        VinpocetineExperience -1.25651 1.603e-04 1.197e-03
        BacopaExperience -1.18497 1.511e-04 1.837e-03
        LionapossManeExperience -1.15578 1.474e-04 9.508e-04
        HordenineExperience -1.15456 1.472e-04 9.246e-04
        SulbutiamineExperience -1.04894 1.338e-04 1.335e-03
        HuperzineExperience -0.76957 9.815e-05 7.428e-04
        GinsengExperience -0.42476 5.417e-05 6.865e-04
        AniracetamExperience -0.30357 3.872e-05 1.129e-03
        NoopeptExperience -0.29668 3.784e-05 7.789e-04
        CentrophenoxineExperience -0.19564 2.495e-05 1.389e-04
        OxiracetamExperience -0.14437 1.841e-05 4.861e-04
        CreatineExperience -0.12480 1.592e-05 -6.631e-04
        AshwagandhaExperience -0.12103 1.544e-05 -5.619e-05
        PramiracetamExperience 0.03029 -3.863e-06 3.963e-04
        RhodiolaExperience 0.21512 -2.744e-05 1.100e-04
        PhenylpiracetamExperience 0.21959 -2.801e-05 1.032e-04
        ColuracetamExperience 0.35140 -4.482e-05 -9.800e-05
        AdrafinilExperience 0.48123 -6.137e-05 -2.582e-04
        TheanineExperience 0.49004 -6.250e-05 -1.659e-05
        CaffeineExperience 1.04602 -1.334e-04 1.169e-04
        ModafinilExperience 2.47240 -3.153e-04 -5.729e-04
        ArmodafinilExperience 2.83790 -3.619e-04 -2.132e-03

        qq4 <- qqmath(rr4)
        # http://i.imgur.com/aQu8Igv.png
        qq4$Nootropic

        # Like before, compare simple nootropics analysis with dose+history+nootropics:
        # what changed the most in absolute terms (least to greatest)?
        tmp <- merge(rr1$Nootropic, rr4$Nootropic[1], by="row.names", all=T)
        tmpDelta <- abs(tmp$"(Intercept).x" - tmp$"(Intercept).y")
        tmp[order(tmpDelta), , drop = FALSE]
        Row.names (Intercept).x (Intercept).y
        20 ModafinilExperience 2.40961 2.47240
        17 HuperzineExperience -0.70487 -0.76957
        8 CentrophenoxineExperience -0.30221 -0.19564
        1 AdrafinilExperience 0.59810 0.48123
        28 TheanineExperience 0.64818 0.49004
        5 AshwagandhaExperience 0.09079 -0.12103
        26 RhodiolaExperience 0.46314 0.21512
        22 OxiracetamExperience 0.11794 -0.14437
        16 HordenineExperience -0.88016 -1.15456
        4 ArmodafinilExperience 2.53205 2.83790
        23 PhenylpiracetamExperience 0.52651 0.21959
        25 PramiracetamExperience 0.39316 0.03029
        11 ColuracetamExperience 0.71994 0.35140
        30 VinpocetineExperience -0.87629 -1.25651
        3 AniracetamExperience 0.12498 -0.30357
        12 CreatineExperience 0.31701 -0.12480
        18 LionapossManeExperience -0.70730 -1.15578
        21 NoopeptExperience 0.19493 -0.29668
        13 DMAEExperience -1.81606 -2.32433
        7 CaffeineExperience 1.59909 1.04602
        15 GinsengExperience -1.12405 -0.42476
        6 BacopaExperience -0.40310 -1.18497
        2 ALCARExperience -0.59846 -1.44127
        27 SulbutiamineExperience -0.02826 -1.04894
        14 GingkoExperience -1.84778 -3.11497
        24 PiracetamExperience -0.09529 -1.36419
        31 VitaminDExperience -0.47647 -2.15718

        Interpretation gets harder as you throw in more variables, but oh well. I’m also a little dubious about the Dose variable: when you plot the random effects, there seems to be much less variation than I’d expect. I wonder if the reshaping to a long format somehow put them all on the same scale or something?

        Report comment

  19. brian says:

    it seems like your phrasing of “a scale of zero (completely useless, did nothing) to ten (life-changing)” renders this part of the study completely useless, since it doesn’t account for negative side effects (meaning it did worse than nothing) and life-changing is a vague term. kudos for doing this research because we need stuff like this, but an ounce of prevention..

    Report comment

  20. St. Rev says:

    Choline citrate and choline chloride are also used, though the latter is rare nowadays.

    Report comment

  21. Dave says:

    I am surprised none of the amphetamines seem to be included here. I assume you decided to leave them out of the survey because nootropic enthusiasts are not posting about them. But that seems odd to me.

    I have read that 20%+ of students at some schools have used prescription stimulants as performance enhancers. That figure is probably unrepresentative, but even a much more conservative guess would make these drugs the most widely used nootropics outside of coffee and nicotine. Are people not discussing them because they are too hard to get, or too illegal? Drug policy does make it very inconvenient to fill your Adderall prescriptions.

    And while we’re at it, what about nicotine?

    Report comment

  22. Rachael says:

    Enjoying all the interesting posts lately!

    I thought about filling in the nootropics survey but then didn’t because I thought I wasn’t its target audience, having only tried one substance other than caffeine. I tried modafinil because I wanted to be less tired and need less sleep, but it had no effect on me. My husband also tried it and it had no effect on him either. It looks from your results as though we’re actually quite unusual in this.

    I’ve considered writing a dystopia in which use of drugs like modafinil becomes ubiquitous and most people become more productive and need less sleep, and the few people who are resistant to it become an underclass who can’t keep up professionally or socially.

    Report comment

  23. Patrick says:

    I agree with your comments about nootropic users tending to ‘outpace’ the research.

    We definitely need more research on nootropics. While not a RCT your survey and analysis is a good start.

    Awesome in-depth article!
    Thanks

    Report comment

  24. Michael Mouse says:

    I vote this Genuine On-Topic Reasonable-quality Post Most Resembling Comment Spam Of The Year, and consider it a hot contender for ‘Of The Decade’ too.

    Report comment

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>