Here’s a vignette from cognitive-behavioral therapy book When Panic Attacks, heavily edited for length:

A chronically anxious medical school professor named Nate suffered from low-self-esteem and feelings of inadequacy. One day, Nate brought me a copy of his CV. I was blown away. He’d listed over sixty pages of research publications, prestigious awards, and keynote addresses he’d given at major conferences around the world. I asked Nate how he reconciled his low self-esteem with all of his accomplishments. He said that every time he looked at his CV, he felt discouraged and told himself that his colleagues’ research studies were far more rigorous and important than his own. He said his paper seemed “soft” and consisted primarily of theoretical work, rather than hard-core laboratory research with real tissue. He said “Dr. Burns, no matter how much I accomplish, it never seems good enough.”

Perfectionism was clearly one of Nate’s self-defeating beliefs. I suggested that Nate use the Pleasure/Perfection Balance Worksheet to test this belief. I told him to write “If I can’t do something perfectly, it’s not worth doing at all” on the top of the sheet, and asked him to list several activities in the left-hand column. I told him to predict how satisfying and rewarding each activity would be, to record how satisfying and rewarding it was afterwards, and to rate how perfectly he did each activity. That way he could find out of it was true that he only enjoyed the things he did perfectly.

The next week, Nate had some interesting results to share with me. One of his activities was giving the welcoming lecture ot the incoming class of medical students. Nate gave this lecture every year because he was considered to be the most charismatic speaker at the medical school. Nate predicted this lecture would be 70% satisfying, but his actual satisfaction as only 20%. This was surprising, since he’d received a thirty-second standing obation, and he’d rated his perfection level for the talk at 90%.

I asked Nate why his satisfaction rating was so low. He explained that he always got standing ovations, so he routinely timed them. The previous year, the medical students had stood and cheered for more than a minute at the end of his talk. This year, the only stood and cheered for half a minute. Nate felt disappointed and started worrying that he was over the hill.

The second entry on Nate’s Pleasure/Perfection Balance Worksheet was that [he fixed a broken pipe in his bathroom]. He had to make several trips to the hardware story to buy tools and parts and to get tips on how to do it, so he didn’t get the pipe fixed until 10 PM. How explained that any plumber could have fixed the pipe in five minutes, so he rated his perfection as 5%. But his satisfaction level for this activity was 100%. In fact, he felt exhilarated. Nate said it was the most satisfying thing he’d done in years.

The result of Nate’s experiment was not consistent with his belief that things weren’t worth doing unless he did them perfectly. It dawned on him that there were many sources of satisfaction in his life that he’d overlooked, such as taking a walk through the woods with his wife, even though neither of them were world-class hikers, playing squash with his son, even though neither of them were champions, or just going out with his family for ice cream cones on a warm summer evening.

This experiment had a significant impact on Nate’s feelings of self-esteem and on his career. He told me that his feelings of anxiety and inferiority decreased, and his productivity actually increased because he was no longer so worried about having to do everything so perfectly.

At first I assumed this story was made up, but the book claims these are based on real patients, and even mentions how the writer showed videos of some of these therapy sessions to his classes. Interesting. How about another?

Several years ago, I did a three-day intensive workshop for a small group of psychotherapists in Florida. A marriage and family therapist named Walter explained that he’d been struggling with anxiety and depression for several months because Paul, the man he’d lived with for eight years, had found a new lover and left him. He put his hand on his chest and said: “It feels real heavy, right here. There’s just a sense of loneliness and emptiness about the whole experience. It feels so universal and final. I feel like this pain is going to go on forever, until the end of time.”

I asked Walter how he was thinking and feeling about the breakup with Paul. What was he telling himself? He saidL “I feel incredibly guilty and ashamed, and it seems like it must have been my fault. Maybe I wasn’t skillful enough, attractive enough, or dynamic enough. Maybe I wasn’t there for him emotionally. I feel like I must have screwed up. Sometimes I feel like a total fraud. Here I am, a marriage and family therapist, and my own relationship didn’t even work out. I feel like a loser. A really, really big loser.”

Walter recorded these five negative thoughts on his daily mood log:

1. I’ll never be in a loving relationship again
2. I must be impossible to live with and impossible to be in a relationship with
3. There must be something wrong with me
4. I totally screwed up and flushed my life down the toilet
5. I’ll end up as an old, fat, gray-haired, lonely gay man

He believed all of these thoughts very strongly.

You can see that most of Walter’s suffering results from the illogical way he’s thinking about the rejection. You could even say that Walter is treating himself far more harshly than Paul did. I thought the Double Standard Technique might help because Walter seemed to be a warm and compassionate individual. I asked wehat he’d say to a dear friend who’d been rejected by someone he’d been living with for eight years. I said “Would you tell him that there’s something wrong with him, that he screwed up his life and flushed it down the toilet for good?”

Walter looked shocked and said he’d never say something like that to a friend. I suggested we try a role-playing exercise so that he could tell me what he would say to a friend who was in the same predicament […]

Therapist (role-playing patient’s friend): Walter, there’s another angle I haven’t told you about. What you don’t understand is that I’m impossible to live with and be in a relationship with. That’s the real reason I feel so bad, and that’s why I’ll be alone for the rest of my life.

Patient (role-playing as if therapist is his friend who just had a bad breakup): Gosh, I’m surprised to hear you say that, because I’ve known you for a long time and never felt that way about you. In fact, you’ve always been warm and open, and a loyal friend. How in the world did you come to the conclusion that you were impossible to be in a relationship with?

Therapist (continuing role-play): Well, my relationship with [my boyfriend] fell apart. Doesn’t that prove I’m impossible to be in a relationship with?

Patient (continuing role-play): In all honesty, what your’e saying doesn’t make a lot of sense. In the first place, your boyfriend was also involved in the relationship. It takes two to tango. And in the second place, you were involved in a reasonably successful relationship with him for eight years. So how can you claim that you’re impossible to live with?

Therapist (continuing role-play:) Let me make sure I’ve got this right. You’re saying that I was in a reasonably successful relationship for eight years, so it doesn’t make much sense to say that I’m impossible to live with or impossible to be in a relationship with?

Patient (continuing-role-play:) You’ve got it. Crystal clear.

At that point, Walter’s face lit up, as if a lightbulb had suddenly turned on in his brain, and we both started laughing. His negative thoughts suddenly seemed absurd to him, and there was an immediate shift in his mood…after Walter put the lie to his negative thoughts, I asked him to rate how he was feeling again. His feeling of sadness fell all the way fromj 80% to 20%. His felings of guilt, shame, and anxiety fell all the way to 10%, and his feelings of hopelessness dropped to 5%. The feelings of loneliness, embarassment, frustration, and anger disappeared completely.

The book is quite long, and it’s full of stories like this. The author, who’s one of the top cognitive-behavioral psychiatrists in the world, describes his experience with the therapy as:

[When I first learned about this therapy, I thought] depression and anxiety seemed far too serious and severe for such a simplistic approach. But when I tried these methods with some of my more difficult patients, my perceptions changed. Patients who’d felt hopeless, worthless, and desperate began to recover. At first, it was hard to believe that the techniques were working, but I could not deny the fact that when my patients learned to put the lie to their negative thoughts, they began to improve. Sometimes they recovered right before my eyes during sessions. Patients who’d felt demoralized and hopeless for years suddenly turned the corner on their problems. I can still recall an elderly French woman who’d been bitterly depressed for more than fifty years, with three nearly-successful suicide attempts, who started shouting “Joie de vivre! Joie de vivre!” (“joy of living”) one day in my office. These experiences made such a strong impact on me that I decided my calling was in clinical work rather than brain research. After considerable soul-searching, I decided to give up my research career and become a full-time clinician. Over the years, I’ve had more than 35,000 psychotherapy sessions with depressed and anxious patients, and I’m every bit as enthusiastic about CBT as when I first began learning about it.

Okay. I am not one of the top cognitive-behavioral therapists in the world. I’ve been studying formal cognitive-behavioral therapy for about a week now, and been doing untrained ad hoc therapy on inpatients for a couple years. But I’ve also gotten to observe a lot of other people doing therapy, and talked to people who have had therapy, and treated patients who were simultaneously undergoing therapy, and the impression I got was very different.

Dr. Burns asks patients to question whether their anxiety and their negative thoughts are rational, and their faces light up and all of their psychiatric problems suddenly melt away.

The therapists I’ve seen ask patients to question whether their anxiety and their negative thoughts are rational, ever so tactfully, and the patients say “No shit, Sherlock, of course they aren’t, but just knowing that doesn’t help or make them go away, and I’ve been through this same spiel with like thirty people already. Now shut up and give me my Xanax.”

In my last post, someone asked what to do if they found cognitive-behavioral therapy hokey and patronizing. I said, only half joking, that “if you don’t like hokey patronizing things, CBT may not be for you.” I know it’s mean, and pessimistic, but everyone I’ve talked to has had pretty much the same experience. I used to attribute this to my friends being pretty smart, and maybe CBT was aimed as less intelligent people, but Nate The Genius Medical School Professor seems pretty smart. So does Walter The Therapist. Burns’ book includes a bunch of other vignettes about high-powered lawyers, graduate students, et cetera. They all find his suggestions of “Well, have you considered that your irrational negative thoughts might not be rational?” super life-changing.

You might have read the study this graph comes from: The Effects of Cognitive-Behavioral Therapy As An Anti-Depressive Treatment Is Falling: A Meta-Analysis. As you can see, the Hedges’ g declined from about 2.5 in 1980 to around 1 today. The latest embarrassing set of results now show CBT doing no better than its old nemesis psychoanalysis. Why?

There are a lot of possible explanations. The smart money is always on “it never worked very well, but we’re finally doing studies that aren’t hopelessly biased”, but the analysis doesn’t find a clear difference in study quality. Other suggestions are that therapists have gotten less committed over time, or that the patient populations has changed. All of these sound reasonable. But let me mention one more possibility.

Every so often, psychiatrists joke about how so many people are depressed we might as well put Prozac in the water supply. Sometimes we say the same thing about lithium, although in that case we’re not joking.

Nobody’s ever talked about putting cognitive-behavioral therapy in the water supply, but insofar as that’s meaningful at all I would say we’ve kind of done it. Cognitive-behavioral ideas, like perfectionism, excessive self-blame, conditional versus unconditional self-respect, deep breathing, goal-setting, et cetera have become basic parts of popular culture. The whole self-esteem movement isn’t exactly cognitive-behavioral, but it’s certainly allied, and it certainly represents a shift to a style of thinking about the self and about psychology in a way that’s much more fertile for cognitive-behavioral ideas. Inside Out was kind of “Cognitive Behavioral Therapy: The Movie”.

Although the particular book I’m reading is from 2006, Burns himself was one of Aaron Beck’s original students and one of the first cognitive-behavioral therapists ever. I wonder how many of these patients who seem absolutely shocked to realize that maybe their anxiety isn’t rational come from that very early period.

It’s very hard to track changes in people’s basic beliefs about psychology. I was flabbergasted to learn that until Dr. Benjamin Spock’s landmark 1940s book on child care, parents were told not to hug, kiss, or show affection to babies, because that would coddle them and make them weak, pampered adults. Before that, parents interacted with their kids much less, and it was assumed that siblings and nannies and friends would raise them, or they would raise themselves. It’s easy to read books about ancient Greece and not notice that they have a completely different view of the role of the self/individual than we do. So it wouldn’t surprise me if a lot of the psychology we consider “obvious” is CBT that has seeped out into the water supply over the past thirty years.

If that were true, it would explain why CBT is no longer as effective – it’s just telling people things they already know.

It could be fairly asked: then why isn’t everybody already better? Depression seems to be increasing, though there’s a lot of argument about exactly how much; that doesn’t sound like what would happen if everyone were automatically getting a background level of therapy.

Here’s a theory, though it’s on even shakier ground than the other one. The meta-analysis proposes that CBT may have lost some placebo effect over time because patients no longer think of it as The Exciting New Thing. I’m not sure I can go along with that – my own analysis of psychotropic medications suggests patients very much prefer the old ones for some reason. But a big part of psychotherapy is placebo effect, so they might be on to something.

What part of psychotherapy provides the placebo? Is it going to the clinic? Talking to the therapist? Hearing fancy words like “self-estimation”? Doing worksheets?

One thing a lot of therapies have in common is that they provide the feeling of insights. For example, psychoanalysts are very good at coming up with surprising-but-plausible ways that your current problems are linked to things that happened to you as a child; the usual result is a patient feeling enlightened, like “You’re right, the leg pain that’s been bothering me is in the same part of my leg that accidentally brushed up against my mother’s breast one time when I was seven, that’s pretty interesting.”

Suppose that in the old days, CBT was an insight a minute and you were constantly hearing surprising things you’d never thought about before. And nowadays, you’re kind of absorbing a lot of those things by osmosis without it seeming too insightful, and then the therapy itself is anticlimactic. Could that lessen the placebo effect enough to account for the data?

I don’t know. Maybe after I’ve been training in formal CBT for more than a week, I’ll have more data and can report back to you.

[EDIT: Sarah writes: “In a way, seeing CBT stuff in pop culture inoculates people, I think. People will get as far as noticing “this negative thought is an anxiety symptom”, but not as far as *actually reversing it*. When people hadn’t heard of CBT, they first got the “this negative thought is irrational” message in a context when they were actively working on their problems, so they followed through with the ‘hard’ step of actually reversing the thought. Now, people run into the revelation that the ‘inner critic’ is wrong just by browsing facebook, when they’re *not* actively trying to fight their anxiety problems, so the revelation loses its force.”]

[EDIT 2: Paul Crowley points out a very similar theory in The Guardian]

Last month I criticized a recent paper, Paunesku et al’s Mindset Interventions Are A Scalable Treatment For Academic Underachievement, saying that it spun a generally pessimistic set of findings about growth mindset into a generally optimistic headline.

Earlier today, lead author Dr. Paunesku was kind enough to write a very thorough reply, which I reproduce below:

I.

Hi Scott,

Thanks for your provocative blog post about my work (I’m the first author of the paper you wrote about). I’d like to take a few moments to respond to your critiques, but first I’d like to frame my response and tell you a little bit about my own motivation and that of the team I am a member of (PERTS).

Good criticism is what makes science work. We are critical of our own work, but we are happy to have help. Often critics are not thoughtful or specific. So I very much appreciate the intent of your blog (to be thoughtful and specific).

What is our motivation? We are trying to improve our education system so that all students can thrive. If growth mindset is effective, we want it in every classroom possible. If it is ineffective, we want to know about it so we don’t waste people’s time. If it is effective for some students in some classrooms, we want to know where and for whom so that we can help those students.

What is our history and where are we now? PERTS approached social psychological interventions with a fair amount of skepticism at first. In many ways, they seemed too good to be true. But, we thought, “if this is true, we should do everything we can to spread it”. Our work over the last 5 years has been devoted to trying to see if the results that emerged from initial, small experiments (like Aronson et al., 2002 and Blackwell et al., 2007) would continue to be effective when scaled. The paper you are critiquing is a step in that process — not the end of the process. We are continuing research to see where, for whom, and at what scale social psychological approaches to improving education outcomes can be effective.

How do I intend to respond to your criticisms? In some cases, your facts or interpretations are simply incorrect, and I will try to explain why. I also invite you to contact me for follow up. In others cases, we simply have different opinions about what’s important, and we’ll have to agree to disagree. Regardless, I appreciate your willingness to be bold and specific in your criticism. I think that’s brave, and I think such bravery makes science stronger.

First, what is growth mindset?

This quote is from one of your other blog posts (not your critique of my paper), from your post:

If you’re not familiar with it, growth mindset is the belief that people who believe ability doesn’t matter and only effort determines success are more resilient, skillful, hard-working, perseverant in the face of failure, and better-in-a-bunch-of-other-ways than people who emphasize the importance of ability. Therefore, we can make everyone better off by telling them ability doesn’t matter and only hard work does.

If you think that’s what growth mindset is, I can certainly see why you’d find it irritating — and even destructive. I’d like to assure you that the people doing growth mindset research do not ascribe to the interpretation of growth mindset you described. Nor is that interpretation of growth mindset something we aim to communicate through our interventions. So what is growth mindset?

Growth mindset is not the belief that “ability doesn’t matter and only effort determines success.” Growth mindset is the belief that individuals can improve their abilities — usually through effort and by learning more effective strategies. For example, imagine a third grader struggling to learn long division for the first time. Should he interpret his struggle as a sign that he’s bad at math — as a sign that he should give up on math for good? Or would it be more adaptive if he realized that he could probably get a lot better at math if he sought out help from his peers or teachers? The student who thinks he should give up would probably do pretty badly while the student who thinks that he can improve his abilities — and tries to do so by learning new study strategies and practicing them — would do comparatively better.

That’s the core of growth mindset. It’s nothing crazy like thinking ability doesn’t matter. It’s keeping in mind that you can improve and that — to do so — you need to work hard and seek out and practice new, effective strategies.

As someone who has worked closely with Carol Dweck and with her students and colleagues for seven years now, I can personally attest that I have never heard anyone in that extended group of people express the belief that ability does not matter or that only hard work matters. In fact, a growth mindset wouldn’t make any sense if ability didn’t matter because a growth mindset is all about improving ability.

One of the active goals of the group I co-founded (PERTS) is to try to dispel misinterpretations of growth mindset because they can be harmful. I take it as a failure of our group that someone like you — someone who clearly cares about research and about scientific integrity — could walk away from our work with that interpretation of growth mindset. I hope that PERTS, and other groups promoting growth mindset, can get better and better at refining the way we talk about growth mindset so that people can walk away from our work understanding it more clearly. For that perspective, I hope you can continue to engage with us to improve that message so that people don’t continue to misinterpret it.

Anyway, here are my responses to specific points you made in your blog about my paper:

Was the control group a mindset intervention?

You wrote:

“A quarter of the students took a placebo course that just presented some science about how different parts of the brain do different stuff. This was also classified as a “mindset intervention”, though it seems pretty different.”

What makes you think it was classified as a mindset intervention? We called that the control group, and no one on our team ever thought of that as a mindset intervention.

The Elderly Hispanic Woman Effect

You wrote:

Subgroup analysis can be useful to find more specific patterns in the data, but if it’s done post hoc it can lead to what I previously called the Elderly Hispanic Woman Effect…

First, I just want to note that I love calling this the “elderly Hispanic woman effect.” It really brings out the intrinsic ridiculousness of the subgroup analyses researchers sometimes go through in search of an effect with a p<.05. It is indeed unlikely that "elderly Hispanic women" would be a meaningful subgroup for analyzing the effects of a medicine (although it might be a fun thought exercise to try to think of examples of a medicine whose effects would be likely to be moderated by being an elderly Hispanic woman). In bringing up the elderly Hispanic woman effect, you're suggesting that we didn't have an a priori reason to think that underperforming students would benefit from these mindset interventions and that we just looked through a bunch of moderators until we found one with p<.05. Well that's not what we did, and I hope I can convince you that our choice of moderator was perfectly reasonable given prior research and theory. There's a lot of research (and common sense too) to suggest that mindset -- and motivation in general -- matters much more when something is hard than when it is easy. Underachieving students presumably find school more difficult, so it makes sense that we'd want to focus on them. I don't think our choice of subgroup is a controversial or surprising prediction. I think anyone who knows mindset research well would predict stronger effects for students who are struggling. In other words, this is obviously not a case of the elderly Hispanic woman effect because it is totally consistent with prior theory and predictions. What ultimately matters more than any rhetorical argument, however, is whether the effect is robust -- whether it replicates. On that front, I hope you'll be pleased to learn that we just ran a successful replication of this study (in fall 2014) in which we again found that growth mindset improves achievement specifically among at-risk high school students (currently under review). We're also planning yet another large scale replication study this fall with a nationally representative sample of schools so that we can be more confident that the interventions are effective in various types of contexts before giving them away for free to any school that wants them. Is the sense of purpose intervention just a bunch of platitudes?

You wrote:

Still another quarter took a course about “sense of purpose” which talked about how schoolwork was meaningful and would help them accomplish lots of goals and they should be happy to do it.

[Later you say that those “children were told platitudes about how doing well in school will “make their families proud” and “make a positive impact”.]

I wouldn’t say those are platitudes. I think you’re under-appreciating the importance of finding meaning in one’s work. It’s a pretty basic observation about human nature that people are more likely to try hard when it seems like there’s a good reason to try hard. I also think it’s a pretty basic observation about our education system that many students don’t have good reasons for trying hard in school — reasons that resonate with them emotionally and help them find the motivation to do their best in the classroom. In our purpose intervention, we don’t just tell students what to think. We try to scaffold them to think of their own reasons for working hard in school, with a focus on reasons that are more likely to have emotional resonance for students. This type of self-persuasion technique has been used for decades in attitudes research.

We’ve written in more depth about these ideas and explored them through a series of studies. I’d encourage you to read this article if you’re interested.

Our paper title and abstract are misleading

You wrote:

Among ordinary students, the effect on the growth mindset group was completely indistinguishable from zero, and in fact they did nonsignificantly worse than the control group. This was the most basic test they performed, and it should have been the headline of the study. The study should have been titled “Growth Mindset Intervention Totally Fails To Affect GPA In Any Way”.

I think the title you suggest would have been misleading. How?

First, we did find evidence that mindset interventions help underachieving students — and those students are very important from a policy standpoint. As we describe in the paper, those students are more likely to drop out, to end up underemployed, or to end up in prison. So if something can help those students at scale and at a low cost, it’s important for people to know that. That’s why the word “underachievement” is in the title of the paper — because we’re accurately claiming that these interventions can help the important (and large) group of students who are underachieving.

Second, the interventions influenced the way all students think about school in ways that are associated with achievement. Although the higher performing students didn’t show any effects on grades in the semester following the study, their mindsets did change. And, as per the arguments I presented above about the link between mindset and difficulty, it’s quite feasible that those higher-performing students will benefit from this change in mindset down the line. For example, they may choose to take harder classes (e.g., Romero et al., 2014) or they may be more persistent and successful in future classes that are very challenging for them.

A misinterpretation of the y-axis in this graph.

You wrote:

Growth mindset still doesn’t differ from zero [among at-risk students].

This just seems to be a simple misreading of the graph. Either you missed the y-axis of the graph that you reproduced on your blog or you don’t know what a residual standardized score is. Either way, I’ll explain because this is pretty esoteric stuff.

The zero point of the y-axis on that graph is, by definition, the grand mean of the 4 conditions. In other words, the treatment conditions are all hovering around zero because zero is the average, and the average is made up mostly of treatment group students. If we had only had 2 conditions (each with 50% of the students), the y-axis “zero” would have been exactly halfway in between them. So the lack of difference from zero does not mean that the treatment was not different from control. The relevant comparison is between the error bars in the control condition and in the treatment conditions.

You might ask, “why are you showing such a graph?” We’re doing so to focus on the treatment contrast at the heart of our paper — the contrast between the control and treatment groups. The residual standardized graph makes it easy to see the size of that treatment contrast.

We’re combining intervention conditions

You wrote:

Did you catch that phrase “intervention conditions”? The authors of the study write: “Because our primary research question concerned the efficacy of academic mindset interventions in general when delivered via online modules, we then collapsed the intervention conditions into a single intervention dummy code (0 = control, 1 = intervention).

[This line of argument goes on for a long time to suggest that we’re unethical and that there’s actually no evidence for the effects of growth mindset on achievement.]

We collapsed the intervention conditions together for this analysis because we were interested in the overall effect of these interventions on achievement. We wanted to see if it is possible to use scalable, social-psychological approaches to improve the achievement of underperforming students. I’m not sure why you think that’s not a valid hypothesis to test, but we certainly think it is. Maybe this is just a matter of opinion about what’s a meaningful hypothesis to test, but I assure you that this hypothesis (contrast all treatments to control) is consistent with the goal of our group to develop treatments that make an impact on student achievement. As I described before, we have a whole center devoted to trying to improve academic achievement with these types of techniques (see perts.net); so it’s pretty natural that we’d want to see whether our social-psychological interventions improve outcomes for the students who need them most (at-risk students).

You’re correct that the growth mindset intervention did not have a statistically significant impact on course passing rates by itself (at a p<.05 level). However, the effect was in the expected direction with p=0.13 (or a 1-tailed p=.07 -- I hope you'll grant that a 1-tailed test is appropriate here given that we obviously predicted the treatment would improve rather than reduce performance). So the lack of a p<.05 should not be interpreted -- as you seem to interpret it -- as some sort of positive evidence that growth mindset "actually didn't work." Anyway, I would say it warrants further research to replicate this effect (work we are currently engaging in). To summarize, we did not find direct evidence that the growth mindset intervention increased course passing rates on its own at a p<.05 level. We did find that growth mindset increased course passing rates at a trend level -- and found a significant effect on GPA. More importantly for me (though perhaps less relevant to your interest specifically in growth mindset), we did provide evidence that social-psychological interventions, like growth mindset and sense of purpose, can improve academic outcomes for at-risk students. We're excited to be replicating this work now and giving it away in the hopes of improving outcomes for students around the world. Summary

I hope I addressed your concerns about this paper, and I welcome further discussion with you. I’d really appreciate it if you’d revise your blog post in whatever way you think is appropriate in light of my response. I’d hate for people to get the wrong impression of our work, and you don’t strike me as someone who would want to mislead people about scientific findings either.

Finally, you’re welcome to post my response. I may post it to my own web page because I’m sure many other people have similar questions about my work. Just let me know how you’d like to proceed with this dialog.

Thanks for reading,

Dave

II.

First of all, the obvious: this is extremely kind and extremely well-argued and a lot of it is correct and makes me feel awful for being so snarky on my last post.

Things in particular which I want to endorse as absolutely right about the critique:

I wrote “A quarter of the students took a placebo course that just presented some science about how different parts of the brain do different stuff. This was also classified as a “mindset intervention”, though it seems pretty different.” Dr. Paunesku says this is wrong. He’s right. It was an editing error on my part. I meant to add the last sentence to the part on the “sense of purpose” intervention, which was classified as a mindset intervention and which I do think seems pretty different. The placebo intervention was never classified as a mindset intervention and I completely screwed up by inserting that piece of text there rather than two sentences down where I meant it to be. It has since been corrected and I apologize for the error.

If another successful replication found that growth mindset continues to only help the lowest-performing students, I withdraw the complaint that this is sketchy subgroup mining, though I think that in general worrying about this is the correct thing to do.

I did misunderstand the residual standardized graph. I suggested that the control group must have severely declined, and got confused about why. In fact, the graph was not about difference between pre-study scores and post-study scores, but difference between group scores and the average score for all four groups. So when the control group is strongly negative, that means it was much worse than the average of all groups. When growth mindset is not-different-from-zero, it means growth mindset was not different from the average of all four groups, which consists of three treatment groups and one control group. So my interpretation – that growth mindset failed to change children’s grades – is not supported by the data.

(In my defense, I can only plead that in the two hundred fifty comments I received, many by professional psychologists and statisticians, only one person picked up on this point (admittedly, after being primed by my own misinterpretation). And the sort of data I expected to be seeing – difference between students’ pre-intervention and post-intervention scores – does not seem to be available. Nevertheless, this was a huge and unforgiveable screw-up, and I apologize.)

III.

But there are also a few places where I will stick to my guns.

I don’t think my interpretation of growth mindset was that far off the mark. I explain this a little further in this post on differing possible definitions of growth mindset, and I will continue to cite this strongly worded paper by Dweck as defense of my views. It’s not just an obvious and innocuous belief about about always believing you should be able to improve, it’s a belief about very counterintuitive effects of believing that success depends on ability versus effort. It is possible that all sophisticated researchers in the field have a very sophisticated and unobjectionable definition of growth mindset, but that’s not the way it’s presented to the public, even in articles by those same researchers.

Although I’m sure that to researchers in the field statements like “Doing well at school will help me achieve my goal” don’t sound like platitudes, it seems important to me in the context of discussions about growth mindset. Some people have billed growth mindset as a very exciting window into what makes learning tick, and how we should divide everyone into groups based on their mindset, and how it’s the Secret To Success, and so on. Learning that a drop-dead simple intervention – telling students to care about school more – actually does as well or better than growth mindset seems to me like a damning result. I realize it would be kind of insulting to call sense-of-purpose an “active placebo” in the medical sense, but that’s kind of how I can’t help thinking of it.

I’m certainly not suggesting the authors of the papers are unethical for combining growth mindset intervention with sense of purpose intervention. But I think the technique is dangerous, and this is an example. They got a result that was significant at p = 0.13. Dr. Paunesku suggests in his email to me that this should be one-tailed (which makes it p = 0.07) and that this obviously trends towards significance. This is a reasonable argument. But this wasn’t the reasonable argument made in the paper. Instead, they make it look like it achieved classical p < 0.05 significance, or at least make it very hard to notice that it didn't. Even if in this case it was - I can't even say white lie, maybe a white spin - I find the technique very worrying. Suppose I want to prove homeopathy cures cancer. I make a trial with one placebo condition and two intervention conditions - chemotherapy and homeopathy. I find that the chemotherapy condition very significantly outperforms placebo, but the homeopathy condition doesn't. So I combine the two interventions into a single bin and say "Therapeutic interventions such as chemotherapy or homeopathy significantly outperform placebo." Then someone else cites it as "As per a study, homeopathy outperforms placebo." This would obviously be bad. I am just not convinced that growth mindset and sense of purpose are similar enough that you can group them together effectively. This is what I was trying to get at in my bungled sentence about how they're both "mindset" interventions but seem pretty different. Yes, they're both things you tell children in forty-five minute sessions that seem related to how they think about school achievement. But that's a really broad category.

But doesn’t it mean something that growth-mindset was obviously trending toward significance?

First of all, I would have had no problem with saying “trending toward significance” and letting readers draw their own conclusions.

Second of all, I’m not totally sure I buy the justification for a one-tailed test here; after all, it seems like we should use a one-tailed test for homeopathy as well, since as astounding as it would be if homeopathy helped, it would be even more astounding if homeopathy somehow made cancer worse. Further, educational interventions often have the opposite of their desired effect – see eg this campaign to increase tolerance of the disabled which made students like disabled people less than a control intervention. In fact, there’s no need to look further than this very study, which found (counterintuitively) that among students already exposed to sense-of-purpose interventions, adding on an extra growth-mindset intervention seemed to make them do (nonsignificantly) worse. I am not a statistician, but my understanding is you ought to have a super good reason to use a one-tailed test, beyond just “Intuitively my hypothesis is way more likely than the exact opposite of my hypothesis”.

Third of all, if we accept p < 0.13 as "trending towards significance", we have basically tripled the range of acceptable study results, even though everyone agrees our current range of acceptable study results is already way too big and some high percent of all medical studies are wrong and only 39% of psych studies replicate and so on.

(I agree that all of this could be solved by something better than p-values, but p-values are what we’ve got)

I realize I’m being a jerk by insisting on the arbitrary 0.05 criterion, but in my defense, the time when only 39% of studies using a criterion replicate is a bad time to loosen that criterion.

IV.

Here’s what I still believe and what I’ve changed my mind on based on Dr. Paunesku’s response.

1. I totally bungled my sentence on the placebo group being a mindset intervention by mistake. I ashamedly apologize, and have corrected the original post.

2. I totally bungled reading the residual standard score graph. I ashamedly apologize, and have corrected the original post, and put a link in bold text to this post on the top.

3. I don’t know whether the thing I thought the graph showed (no significant preintervention vs. postintervention GPA improvement for growth mindset, or no difference in change from controls) is true. It may be hidden in the supplement somewhere, which I will check later. Possible apology pending further investigation.

4. Growth mindset still had no effect (in fact nonsignificantly negative) for students at large (as opposed to underachievers). I regret nothing.

5. Growth mindset still failed to reach traditional significance criteria for changing pass rates. I regret nothing.

Jacques Derrida proposed a form of philosophical literary criticism called deconstruction. I’ll be the first to admit I don’t really understand it, but it seems to have something to do with assuming all texts secretly contradict their stated premise and apparent narrative, then hunting down and exposing the plastered-over areas where the author tries to hide this.

I have no idea whether this works for literature or not, but it’s a useful way to read scientific papers.

Consider a popular field – or, at least, a field where a certain position is popular. For example, we’ve been talking a lot about growth mindset recently. There seem to be a lot of researchers working to prove growth mindset and not a lot working to disprove it. Journals are pretty interested in studies showing growth mindset interventions work, and maybe not so interested in studies showing they don’t. I’ll admit that my strong suspicions of publication bias don’t seem to be borne out by the facts here – see this meta-analysis – but I bet its more sinister cousin “all experimenters believe the same thing and have the same experimenter effects” bias is alive and well.

In a field like that, you’re not going to get the contrarian studies you want, but one way to find the other side of the issue is to look a little more closely at the studies that do get published, the ones that say they’re in support of the thesis, and see if you can find anything incriminating.

Here’s a perfect example: Mindset Interventions Are A Scalable Treatment For Academic Underachievement, by a team of six researchers including Carol Dweck.

The abstract reads:

The efficacy of academic-mind-set interventions has been demonstrated by small-scale, proof-of-concept interventions, generally delivered in person in one school at a time. Whether this approach could be a practical way to raise school achievement on a large scale remains unknown. We therefore delivered brief growth-mind-set and sense-of-purpose interventions through online modules to 1,594 students in 13 geographically diverse high schools. Both interventions were intended to help students persist when they experienced academic difficulty; thus, both were predicted to be most beneficial for poorly performing students. This was the case. Among students at risk of dropping out of high school (one third of the sample), each intervention raised students’ semester grade point averages in core academic courses and increased the rate at which students performed satisfactorily in core courses by 6.4 percentage points. We discuss implications for the pipeline from theory to practice and for education reform.

This sounds really, really impressive! It’s hard to imagine any stronger evidence in growth mindset’s favor.

And then you make the mistake of reading the actual paper.

The paper asked a 1,594 students from a bunch of different high schools to take a 45 minute online course.

A quarter of the students took a placebo course that just presented some science about how different parts of the brain do different stuff.

Another quarter took a course that was supposed to teach growth mindset.

Still another quarter took a course about “sense of purpose” which talked about how schoolwork was meaningful and would help them accomplish lots of goals and they should be happy to do it. This was also classified as a “mindset intervention”, though it seems pretty different.

And the final quarter took both the growth mindset course and the “sense of purpose” course.

Then they let all students continue taking their classes for the rest of the semester and saw what happened, which was this:

[EDIT: I totally bungled these graphs! See discussion of exactly how on the author’s reply above, without which the information below will be misleading at best]

Among ordinary students, the effect on the growth mindset group was completely indistinguishable from zero, and in fact they did nonsignificantly worse than the control group. This was the most basic test they performed, and it should have been the headline of the study. The study should have been titled “Growth Mindset Intervention Totally Fails To Affect GPA In Any Way”.

Instead they went to subgroup analysis. Subgroup analysis can be useful to find more specific patterns in the data, but if it’s done post hoc it can lead to what I previously called the Elderly Hispanic Woman Effect, after medical papers that can’t find their drug has any effect on people at large, so they keep checking different subgroups – young white men…nothing. Old black men…nothing. Middle-aged Asian transgender people…nothing. Newborn Australian aboriginal butch lesbians…nothing. Elderly Hispanic women…p = 0.049…aha! And the study gets billed as “Scientists Find Exciting New Drug That Treats Diabetes In Elderly Hispanic Women.”

As per the abstract, the researchers decided to focus on an “at risk” subgroup because they had principled reasons to believe mindset interventions would work better on them. In their subgroup of 519 students who had a GPA of 2.0 or less last semester, or who failed one or more academic courses last semester:

Growth mindset still doesn’t differ from zero. And growth mindset does nonsignificantly worse than their “sense of purpose” intervention where they tell children to love school. In fact, the students who take both “sense of purpose” and growth mindset actually do (nonsignificantly) worse than sense-of-purpose alone!

But the control group mysteriously started doing much worse in all their classes right after the study started, so growth mindset is significantly better than the control group. Hooray!

Why would the control group’s GPA suddenly decline? The simplest answer would be that by coincidence the class got harder right after the study started, and only the intervention kids were resilient enough to deal with it – but that can’t be right, because this was done at eleven different schools, and they wouldn’t have all had their coursework get harder at the same time.

Another possibility is that sufficiently low-functioning kids are always declining – that is, as time goes on they get more and more behind in their coursework, so their grades at time t+1 are always less than at time t, and maybe growth mindset has arrested this decline. This is plausible and I’d be interested in seeing if other studies have found this.

Perhaps aware that this is not very convincing, the authors go on to do another analysis, this one of percent of students passing their classes.

This is the same group of at-risk students as the last one. It’s graphing what percent of these students pass versus fail their courses. The graph on th left shows that a significantly higher number of students in the intervention conditions pass their courses than in the control condition.

This is better, but one part still concerns me.

Did you catch that phrase “intervention conditions”? The authors of the study write: “Because our primary research question concerned the efficacy of academic mindset interventions in general when delivered via online modules, we then collapsed the intervention conditions into a single intervention dummy code (0 = control, 1 = intervention).

We don’t know whether growth mindset did anything for even these students in this little subgroup, because it was collapsed together with the (more effective) “sense of purpose” intervention before any of these tests were done. I don’t know if this is just for convenience, or if it is to obfuscate that it didn’t work on its own.

[EDIT: Scott McGreal looks further and finds in the supplementary material that growth mindset alone did NOT significantly improve pass rates!]

The abstract of this study tells you none of this. It just says: “Mindset Interventions Are A Scalable Treatment For Academic Overachievement…Among students at risk of dropping out of high school (one third of the sample), each intervention raised students’ semester grade point averages in core academic courses and increased the rate at which students performed satisfactorily in core courses by 6.4 percentage points” From the abstract, this study is a triumph.

But my own summary of these results, as relevant to growth mindset is as follows:

For students with above a 2.0 GPA, a growth mindset intervention did nothing.

For students with below a 2.0 GPA, the growth mindset interventions may not have improved GPA, but may have prevented GPA from falling, which for some reason it was otherwise going to do.

Even in those students, it didn’t do any better than a “sense-of-purpose” intervention where children were told platitudes about how doing well in school will “make their families proud” and “make a positive impact”.

In no group of students did it significantly increase chance of passing any classes.

Haishan writes:

If ye read only the headlines, what reward have ye? Do not even the policymakers the same? And if ye take the abstract at its face, what do ye more than others? Do not even the science journalists so?”

Titles, abstracts, and media presentations are where authors can decide how to report a bunch of different, often contradictory results in a way that makes it look like they have completely proven their point. A careful look at the study may find that their emphasis is misplaced, and give you more than enough ammunition against a theory even where the stated results are glowingly positive.

The only reason we were told these results is that they were in the same place as a “sense of purpose mindset” intervention that looked a little better, so it was possible to publish the study and claim it as a victory for mindsets in general. How many studies that show similar results for growth mindset lack a similar way of spinning the data, and so never get seen at all?

But a while ago I wrote Beware The Man Of One Study, in which I wrote that there is such a variety of studies finding such a variety of contradictory things that anybody can isolate one of them, hold it up as the answer, and then claim that their side is right and the other side are ‘science denialists’. The only way to be sure you’re getting anything close to the truth is to examine the literature of an entire field as a gestalt.

And here’s something no one ever said: “Man, I’m so glad I examined the literature of that entire field as a gestalt, things make much more sense now.”

Two years ago Moss-Racusin et al released Science Faculty’s Subtle Gender Biases Favor Male Students, showing a strong bias in favor of men in STEM hiring. The methodology was almost identical to this current study, but it returned the opposite result.

Now everyone gets to cite whichever study accords with their pre-existing beliefs. So Scientific American writes Study Shows Gender Bias In Science Is Real, and any doubt has been deemed unacceptable by blog posts like Breaking: Some Dudes On The Internet Refuse To Believe Sexism Is A Thing. But the new study, for its part, is already producing headlines like The Myth About Women In Science and blog posts saying that it is “enough for everyone who is reasonable to agree that the feminists are spectacular liars and/or unhinged cranks”.

So probably we’re going to have to do that @#$%ing gestalt thing.

Why did these two similar studies get such different results? Williams and Ceci do something wonderful that I’ve never seen anyone else do before – they include in their study a supplement admitting that past research has contradicted theirs and speculating about why that might be:

1. W&C investigate hiring tenure-track faculty; MR&a investigate hiring a “lab manager”. This is a big difference, but as far as I can tell, W&C don’t give a good explanation for why there should be a pro-male bias for lab managers but a pro-female bias for faculty. The best explanation I can think of is that there have been a lot of recent anti-discrimination campaigns focusing on the shortage of female faculty, so that particular decision might activate a cultural script where people think “Oh, this is one of those things that those feminists are always going on about, I should make sure to be nice to women here,” in a way that just hiring a lab manager doesn’t.

Likewise, hiring a professor is an important and symbolic step that…probably doesn’t matter super-much to other professors. Hiring a lab manager is a step without any symbolism at all, but professors often work with them on a daily basis and depend on their competency. That might make the first decision Far Mode and the second Near Mode. Think of the Obama Effect – mildly prejudiced people who might be wary at the thought of having a black roommate were very happy to elect a black President and bask in a symbolic dispay of tolerance that made no difference whatsoever to their everyday lives.

Or it could be something simpler. Maybe lab work, which is very dirty and hands-on, feels more “male” to people, and professorial work, which is about interacting with people and being well-educated, feels more “female”. In any case, W&C say their study is more relevant, because almost nobody in academic science gets their start as a lab manager (they polled 83 scientists and found only one who had).

2. Both W&C and MR&a ensured that the male and female resumes in their study were equally good. But W&C made them all excellent, and MR&a made them all so-so. Once again, it’s not really clear why this should change the direction of bias. But here’s a hare-brained theory: suppose you hire using the following algorithm: it’s very important that you hire someone at least marginally competent. And it’s somewhat important that you hire a woman so you look virtuous. But you secretly believe that men are more competent than women. So given two so-so resumes, you’ll hire the man to make sure you get someone competent enough to work with. But given two excellent resumes, you know neither candidate will accidentally program the cyclotron to explode, so you pick the woman and feel good about yourself.

And here are some other possibilities that they didn’t include in their supplement, but which might also have made a difference.

3. W&C asked “which candidate would you hire?”. MR&a said “rate each candidate on the following metrics” (including hireability). Does this make a difference? I could sort of see someone who believed in affirmative action saying something like “the man is more hireable, but I would prefer to hire the woman”. Other contexts prove that even small differences in the phrasing of a question can lead to major incongruities. For example, as of 2010, only 34% of people polled strongly supported letting homosexuals serve in the military, but half again as many – a full 51% – expressed that level of support for letting “gays and lesbians” serve in the military. Ever since reading that I’ve worried about how many important decisions are being made by the 17% of people who support gays and lesbians but not homosexuals.

For all we know maybe this is the guy in charge of hiring for STEM faculty positions

4. Williams and Ceci asked participants to choose between “Dr. X” (who was described using the pronouns “he” and “him”) and “Dr. Y” (who was described using the pronouns “she” and “her”). Moss-Racusin et al asked participants to choose between “John” and “Jennifer”. They said they checked to make sure that the names were rated equal for “likeability” (whatever that means), but what if there are other important characteristics that likeability doesn’t capture? We know that names have big effects on our preconceptions of people. For example, people with short first names earn more money – an average of $3600 less per letter. If we trust this study (which may not be wise), John already has a $14,400 advantage on Jennifer, which goes a lot of the way to explaining why the participants offered John higher pay without bringing gender into it at all!

Likewise, independently of a person’s gender they are more likely to succeed in a traditionally male field if they have a male-sounding name. That means that one of the…call it a “prime” that activates sexism…might have been missed by comparing Dr. X to Dr. Y, but captured by pitting the masculine-sounding John against the feminine-sounding Jennifer. We can’t claim that W&C’s subjects were rendered gender-blind by the lack of gender-coded names – they noticed the female candidates enough to pick them twice as often as the men – but it might be that not getting the name activated the idea of gender from a different direction than hearing the candidates’ names would have.

5. Commenter Lee points out that MR&a tried to make their hokey hypothetical hiring seem a little more real than W&C did. MR&a suggest that these are real candidates being hired…somewhere…and the respondents have to help decide whom to hire (although they still use the word “imagine”). W&C clearly say that this is a hypothetical situation and ask the respondents to imagine that it is true. Some people in the comments are arguing that this makes W&C a better signaling opportunity whereas MR&a stays in near mode. But why would people not signal on a hiring question being put to them by people they don’t know about a carefully-obscured situation in some far-off university? Are sexists, out of the goodness of their hearts, urging MR&a to hire the man out of some compassionate desire to ensure they get a qualified candidate, but when W&C send them a hypothetical situation, they switch back into signaling mode?

6. Commenter Will points out that MR&a send actual resumes to their reviewers, but W&C send only a narrative that sums up some aspects of the candidates’ achievements and personalities (this is also the concern of Feminist Philosophers). This is somewhat necessitated by the complexities of tenure-track hiring – it’s hard to make up an entire fake academic when you can find every published paper in Google Scholar – but it does take them a step away from realism. They claim that they validated this methodology against real resumes, but it was a comparatively small validation – only 35 people. On the other hand, even this small validation was highly significant for pro-female bias. Maybe for some reason getting summaries instead of resumes heavily biases people in favor of women?

Or maybe none of those things mattered at all. Maybe all of this is missing the forest for the trees.

I love stories about how scientists set out to prove some position they consider obvious, but unexpectedly end up changing their minds when the results come in. But this isn’t one of those stories. Williams and Ceci have been vocal proponents of the position that science isn’t sexist for years now – for example, their article in the New York Times last year, Academic Science Isn’t Sexist. In 2010 they wrote Understanding Current Causes Of Women’s Underrepresentation In Science, which states:

The ongoing focus on sex discrimination in reviewing, interviewing, and hiring represents costly, misplaced effort: Society is engaged in the present in solving problems of the past, rather than in addressing meaningful limitations deterring women’s participation in science, technology, engineering, and mathematics careers today. Addressing today’s causes of underrepresentation requires focusing on education and policy changes that will make institutions responsive to differing biological realities of the sexes.

So they can hardly claim to be going into this with perfect neutrality.

But the lead author of the study that did find strong evidence of sexism, Corinne Moss-Racusin (whose name is an anagram of “accuser on minor sins”) also has a long history of pushing the position she coincidentally later found to be the correct one. A look at her resume shows that she has a bunch of papers with titles like “Defending the gender hierarchy motivates prejudice against female leaders”, “‘But that doesn’t apply to me:’ teaching college students to think about gender”, and “Engaging white men in workplace diversity: can training be effective?”. Her symposia have titles like “Taking a stand: the predictors and importance of confronting discrimination”. This does not sound like the resume of a woman whose studies ever find that oh, cool, it looks like sexism isn’t a big problem here after all.

So what conclusion should we draw from the people who obviously wanted to find a lack of sexism finding a lack of sexism, but the people who obviously wanted to find lots of sexism finding lots of sexism?

This is a hard question. It doesn’t necessarily imply the sinister type of bias – it may be that Drs. Williams and Ceci are passionate believers in a scientific meritocracy simply because that’s what all their studies always show, and Dr. Moss-Racusin is a passionate believer in discrimination because that’s what her studies find. On the other hand, it’s still suspicious that two teams spend lots of time doing lots of experiments, and one always gets one result, and the other always gets the other. What are they doing differently?

Problem is, I don’t know. Neither study here has any egregious howlers. In my own field of psychiatry, when a drug company rigs a study to put their drug on top, usually before long someone figures out how they did it. In these two studies I’m not seeing anything.

And this casts doubt upon those four possible sources of differences listed above. None of them look like the telltale sign of an experimenter effect. If MR&a were trying to fix their study to show lots of sexism, it would have taken exceptional brilliance to do it by using the names “John” versus “Jennifer”. If W&C were trying to fix their study to disguise sexism, it would have taken equal genius to realize they could do it by asking people “who would you hire?” rather than “who is most hireable?”.

(the only exception here is the lab manager. It’s just within the realm of probability that MR&a might have somehow realized they’d get a stronger signal asking about lab managers instead of faculty. The choice to ask about lab managers instead of faculty is surprising and does demand an explanation. And it’s probably the best candidate for the big difference between their results. But for them to realize that they needed to pull this deception suggests an impressive ability to avoid drinking their own Kool-Aid.)

Other than that, the differences I’ve been considering in these studies are the sort that would be very hard to purposefully bias. But the fact that both groups got the result they wanted suggests that the studies were purposefully biased somehow. This reinforces my belief that experimenter effects are best modeled as some sort of mystical curse incomprehensible to human understanding.

(now would be an excellent time to re-read the the horror stories in Part IV of “The Control Group Is Out Of Control”)

Speaking of horror stories. Sexism in STEM is, to put it mildly, a hot topic right now. Huge fortunes in grant money are being doled out to investigate it (Dr. Moss-Racusin alone received nearly a million dollars in grants to study STEM gender bias) and thousands of pages are written about it every year. And yet somehow the entire assembled armies of Science, when directed toward the problem, can’t figure out whether college professors are more or less likely to hire women than men.

This is not like studying the atmosphere of Neptune, where we need to send hundred-million dollar spacecraft on a perilous mission before we can even begin to look into the problem. This is not like studying dangerous medications, where ethical problems prevent us from doing the experiments we really need. This is not like studying genetics, where you have to gather large samples of identical twins separated at birth, or like climatology, where you hang out at the North Pole and might get eaten by bears. This is a survey of college professors. You know who it is studying this? College professors. The people they want to study are in the same building as them. The climatologists are getting eaten by bears, and the social psychologists can’t even settle a question that requires them to walk down the hallway.

It’s not even like we’re trying to detect a subtle effect here. Both sides agree that the signal is very large. They just disagree what direction it’s very large in!

A recent theme of this blog has been that Pyramid Of Scientific Evidence be damned, our randomized controlled trials suck so hard that a lot of the time we’ll get more trustworthy information from just looking at the ecological picture. Williams and Ceci have done this (see Part V, Section b of their supplement, “Do These Results Differ From Actual Hiring Data”) and report that studies of real-world hiring data confirm women have an advantage over men in STEM faculty hiring (although far fewer of them apply). It also matches the anecdotal evidence I hear from people in the field. I’m not necessarily saying I’m ambivalent between the two studies’ conclusions. Just that it bothers me that we have to go to tiebreakers after doing two good randomized controlled trials.

At this point, I think the most responsible thing would be to have a joint study by both teams, where they all agree on a fair protocol beforehand and see what happens. Outside of parapsychology I’ve never heard of people taking such a drastic step – who would get to be first author?! – but at this point it’s hard to deny that it’s necessary.

In conclusion, I believe the Moss-Racusin et al study more, but I think the Williams and Ceci study is more believable. And the best way to fight sexism in science is to remind people that it would be hard for women to make things any more screwed up than they already are.

A lot of the comments I’ve gotten about Tuesday’s post on growth mindset have been pretty similar. They’ve argued that yes, innate ability might matter, but that even the most innate abilityed person needs effort to fulfill her potential. If someone were to believe that success were 100% due to fixed innate ability and had nothing to do with practice, then they wouldn’t bother practicing, and they would fall behind. Even if their innate ability kept them from falling behind morons, at the very least they would fall behind their equally innate abilityed peers who did practice.

I will call this the Bloody Obvious Position, since it’s hard to believe it isn’t true. I once tried to imagine a world without it as a thought experiment, but it was pretty weird and I wasn’t serious.

The Bloody Obvious Position was what I was trying to get at with my post on basketball, and with my terrible ad hoc graph:

Nevertheless, some people thought I was denying the Bloody Obvious Position. Other people thought I was accusing Carol Dweck of denying the Bloody Obvious Position (see eg here). This despite my making sure to say:

I want to end by correcting a very important mistake about growth mindset that Dweck mostly avoids but which her partisans constantly commit egregiously.

I believe the Bloody Obvious Position. Dweck believes the Bloody Obvious Position. I acknowledge that Dweck believes the Bloody Obvious Position. There are a lot of growth mindset partisans online who don’t believe the Bloody Obvious Position, and I satisfied my urge to yell at them, but now they’ve been yelled at, and the more important issues debated by reasonable people still remain.

So where do I disagree with Dweck? I interpret Dweck as making the following statement:

The more important you believe innate ability to be compared to effort, the more likely you are to stop trying, to avoid challenges, to lie and cheat, to hate learning, and to be obsessed with how you appear before others

Call it the Controversial Position. This is not the same thing as the Bloody Obvious Position. In the Bloody Obvious Position, someone can believe success is 90% innate ability and 10% effort. They might also be an Olympian who realizes that at her level, pretty much everyone is at a innate ability ceiling, and a 10% difference is the difference between a gold medal and a last-place finish. So she practices very hard and does just as well as anyone else.

According to the Controversial Position, this athlete will still do worse than someone who believes success is 80% ability and 20% effort, who will in turn do worse than someone who believes success is 70% ability and 30% effort, all the way down to the person who believes success is 0% ability and 100% effort, who will do best of all and take the gold medal.

And this is why I deny that I’m secretly agreeing with Dweck, or strawmanning Dweck, or whatever. I don’t believe the Controversial Position, but I think Dweck does. For example, here she writes: “The more a player believed athletic ability was a result of effort and practice rather than just natural ability, the better that player performed”.

There is nothing in there about “the more a player realizes that, no matter how important innate ability is, effort matters too.” Her statement says that it’s entirely about what degree a player attributes success to effort versus innate ability. The natural conclusion there is that the player who believes success is 0% innate ability and 100% effort will do the best.

Her studies reflect this as well. The most common design uses the IAR, a test where children are asked to attribute different things to effort versus ability. Those who attribute too many things to ability are classified as “helpless” and “fixed mindset”. There’s no question about “Okay, some things are due to ability, but if you work hard that still helps, right?” Nor have I ever seen any of the literature claim “it’s important to believe effort matters a little, but after a certain point more effort-attribution doesn’t help”, or “Maybe there’s an L-shaped relationship between belief-in-importance-of-ability and success.”

I’d like to be able to teach my children that success is X% innate ability and Y% practice, for non-zero values of both X and Y. I think growth mindset theory claims that if some other parent teaches their kids the same thing for a lower value of X and higher value of Y, their children will be more honest, harder-working, and more successful. And the parent who says it’s 0% innate ability and 100% practice will do best of all. If growth mindset people don’t believe that, I can only confess I have never been able to infer that lack of belief from their writings.

II.

Worse, we can distinguish between a Sorta Controversial Position and a Very Controversial Position:

SCP: The more children believe effort matters, and the less they believe innate ability matters, the more successful they will be. This is because every iota of belief they have in effort gives them more incentive to practice. A child who believes innate ability and effort both explain part of the story might think “Well, if I practice I’ll become a little better, but I’ll never be as good as Mozart. So I’ll practice a little but not get my hopes up.” A child who believes only effort matters, and innate ability doesn’t matter at all, might think “If I practice enough, I can become exactly as good as Mozart.” Then she will practice a truly ridiculous amount to try to achieve fame and fortune. This is why growth mindset works.

VCP: Belief in the importance of ability directly saps a child’s good qualities in some complicated psychological way. It is worse than merely believing that success is based on luck, or success is based on skin color, or that success is based on whatever other thing that isn’t effort. It shifts children into a mode where they must protect their claim to genius at all costs, whether that requires lying, cheating, self-sabotaging, or just avoiding intellectual effort entirely. When a fixed mindset child doesn’t practice as much, it’s not because they’ve made a rational calculation about the utility of practice towards achieving success, it’s because they’ve partly or entirely abandoned success as a goal in favor of the goal of trying to convince other people that they’re Smart.

Carol Dweck unambiguously believes the Very Controversial Position. In a quotation which I admit I am mangling and ellipsis-ing heavily to remove extra verbiage, but which I think preserves the meaning of her claim:

[People with fixed mindsets] are so concerned with being and looking talented that they never realize their full potential. In a fixed mindset, the cardinal rule is to look talented at all costs. The second rule is don’t work too hard or practive too much…having to work casts doubt on your ability. The third rule is, when faced with setbacks, run away. They say things like ‘I would try to cheat on the next test’. They make excuses, they blame others, they make themselves feel better by looking down on those who have done worse.”

Can we all agree this is a much stronger claim than “ability matters, but effort also matters?”

III.

I was not intending to “debunk” growth mindset, or even present a pure polemic against growth mindset. I admitted that many of the studies around it were very good, and that I don’t have good answers to them. My bias is against the theory, but I tried not to just follow my bias. I tried to treat it on the level of “there’s a lot of good evidence for growth mindset, now what’s the best evidence we can find against it?”

So I guess I should probably come out and say what I believe about each position.

I believe the Bloody Obvious Position is bloody obvious.

I believe the Somewhat Controversial Position is probably not a good way to parse things. Part of it is that we might be confusing explicit versus implicit beliefs. Maybe a particular geneticist is very aware of research showing how important genetics is to success, and would give a very high estimate if asked, but in her own life, when she fails, lack of effort is still the first explanation to immediately leap to mind. Or maybe some teacher is very on board with growth mindset and things IQ is a racist construct, but is convinced that she can’t do physics because she’s just “not a math kind of person”. The research I’ve seen hasn’t really distinguished between explicit and implicit beliefs. The priming experiments sure seem more likely to affect what immediately comes to mind than your stable, well-reasoned beliefs about how the world works (even though a few priming experiments have checked stable well-reasoned beliefs to see if the intervention worked!)

If you put a gun to my head, I’ll say it certainly works in the lab, and give you about 50-50 odds that it matters in real life. The studies don’t show any real-life correlation between growth mindset and any measures of success. Many people have pointed out that this could be confounded – dumb people might preferentially believe ability doesn’t matter to make themselves feel better about not having it, and smart people might preferentially believe effort doesn’t matter because they rarely have to use it. But if you accept that, some of the rest of it starts to look confounded. If fixed mindset = smart people, than might the reason they react poorly to challenges and failure be that they have no experience with them? Might it be that the more challenges and failures you’ve encountered before, the better you are at dealing with them? Certainly that is how I interpret this Dweck paper, even though she thinks it is purely a mindset effect. Another way of explaining the ecological results without bringing in that particular confounder would be if growth mindset helped in some situations, but fixed mindset helped in others. For example, a person with fixed mindset risks not trying hard enough because they think there’s no point. But a person with growth mindset risks the opportunity costs of prolonging their inevitable failure instead of (in the Silicon Valley term) “failing fast” and pivoting towards a higher-payoff activity.

The Very Controversial Position is also well-supported, also contradicted by ecological data, and really really doesn’t match my experience. The people I know who are most interested in issues of innate ability don’t behave at all like Dweck’s subjects. In fact, I wonder if a lot of the “life-hacking” movement might be ability-mindset people trying to figure out how to succeed more by improving ability – certainly the people who practice dual-n-back every day because they think it increases IQ fall into this category, but so do nootropics users, people who follow special diets to increase energy, and “try this one weird trick to improve your motivation”. And these same people seem interested in things like spaced repetition software, which might be thought of as sort of prosthetic ability-enhancers. On the other hand, I’ve also met people who say “I could succeed if only I put in some effort, but I have some mental block / depression / ADHD / low conscientiousness score that makes it impossible for me to work that hard, so better go eat worms”, then sabotage themselves at every opportunity.

And yes, it’s a sin to privilege your own experience and priors over the results of good studies, but sometimes it’s necessary. And it’s another sin to prefer the results of broad ecological studies to controlled experimental trials, but sometimes that’s necessary too.

I deny the claim that I don’t disagree with Dweck on anything of substance. I don’t absolutely disagree with her on anything, but there are a lot of things I doubt, or that I expect to capture true insights without being the best way to express them. Growth mindset makes some surprising and genuinely controversial claims, and I’m not yet at the point where I can feel sure about them either way.

Admitting a bias is the first step to overcoming it, so I’ll admit it: I have a huge bias against growth mindset.

(if you’re not familiar with it, growth mindset is the belief that people who believe ability doesn’t matter and only effort determines success are more resilient, skillful, hard-working, perseverant in the face of failure, and better-in-a-bunch-of-other-ways than people who emphasize the importance of ability. Therefore, we can make everyone better off by telling them ability doesn’t matter and only hard work does. More on Wikipedia here).

See, I can sometimes be contrarian, and growth mindset is pretty much the only idea from social psychology that is universally beloved. If I try to search for criticism of growth mindset, I am buried in the Google-shadow of people raving about how wonderful a discovery it is and how we all need more of it. Google “growth mindset debunked” and you just get a bunch of articles talking about how growth mindset debunked all the other inferior ideas before it was discovered. Google “growth mindset publication bias”, and you just get a bunch of articles on how we need to keep a growth mindset about fighting publication bias.

It’s unnatural, is what it is. A popular psychological finding that doesn’t have gruff people dismissing it as a fad? That doesn’t have politicians condemning it as a feel-good justification for everything wrong with society? That doesn’t have a host of smarmy researchers saying that what, you still believe that, didn’t you know it failed to replicate and has since been entirely superseded by a new study out of Belarus? I’m not saying Carol Dweck has definitely made a pact with the Devil, I’m just saying I don’t have a good alternative explanation.

Which brings me to the second reason I’m biased against it. Good research shows that inborn ability (including but not limited to IQ) matters a lot, and that the popular prejudice that people who fail just weren’t trying hard enough is both wrong and harmful. Social psychology has been, um, very enthusiastic about denying that result. If all growth mindset did was continue to deny it, then it would be unexceptional.

But growth mindset goes further. It’s not (just?) that ability doesn’t matter. It’s that belief that ability might matter is precisely what makes people fail. People who believe ability matters will refuse to work hard, will avoid challenges, will become “helpless” in the face of pressure, will hate learning as a matter of principle, will refuse to work hard, will become blustery and defensive about their “brilliance”, will lie to people and hide their failures, and will drop out of school and turn to drugs (really)! People who believe that anyone can succeed if they try hard enough will be successful, well-adjusted, and treat life as a series of challenging adventures. It all strikes a curmudgeon like me as just about the thickest morality tale since Pilgrim’s Progress, and as just about the most convenient explanatory coup since “the reason psychic powers don’t work on you is because you’re a skeptic!”

Which brings me to the third reason I’m biased against it. It is right smack in the middle of a bunch of fields that have all started seeming a little dubious recently. Most of the growth mindset experiments have used priming to get people in an effort-focused or an ability-focused state of mind, but recent priming experiments have famously failed to replicate and cast doubt on the entire field. And growth mindset has an obvious relationship to stereotype threat, which has also started seeming very shaky recently.

So I have every reason to be both suspicious of and negatively disposed toward growth mindset. Which makes it appalling that the studies are so damn good.

Consider Dweck and Mueller 1998, one of the key studies in the area. 128 fifth-graders were asked to do various puzzles. First they did some easy ones and universally succeeded. The researchers praised them as follows:

All children were told that they had performed well on this problem set: “Wow, you did very well on these problems. You got [number of problems] right. That’s a really high score!” No matter what their actual score, all children were told that they had solved at least 80% of the problems that they answered.

Some children were praised for their ability after the initial positive feedback: “You must be smart at these problems.” Some children were praised for their effort after the initial positive feedback: “You must have worked hard at these problems.” The remaining children were in the control condition and received no additional feedback.

This is a nothing intervention, the tiniest ghost of an intervention. The experiment had previously involved all sorts of complicated directions and tasks, I get the impression they were in the lab for at least a half hour, and the experimental intervention is changing three short words in the middle of a sentence.

And what happened? The children in the intelligence praise condition were much more likely to say at the end of the experiment that they thought intelligence was more important than effort (p < 0.001) than the children in the effort condition. When given the choice, 67% of the effort-condition children chose to set challenging learning-oriented goals, compared to only 8% (!) of the intelligence-condition. After a further trial in which the children were rigged to fail, children in the effort condition were much more likely to attribute their failure to not trying hard enough, and those in the intelligence condition to not being smart enough (p < 0.001). Children in the intelligence condition were much less likely to persevere on a difficult task than children in the effort condition (3.2 vs. 4.5 minutes, p < 0.001), enjoyed the activity less (p < 0.001) and did worse on future non-impossible problem sets (p...you get the picture). This was repeated in a bunch of subsequent studies by the same team among white students, black students, Hispanic students...you probably still get the picture. Or take An Analysis Of Learned Helplessness. Dweck has used a test called the IAR to separate children out into those who think effort is more important (“mastery-oriented”) and those who think ability is more important (“helpless”). Then she gave all of them impossible problems and watched them squirm – or, more fomally, tested how long the two groups continued working on them effectively. She found extremely strong results – of the 30 subjects in each group, 11 of the mastery-oriented tried harder after failure, compared to 0 helpless. 21 of the helpless children stopped trying hard after failure, compared to only 4 mastery-oriented. She described the mastery-oriented children as saying things like “I love a challenge,” and the helpless children begging to be allowed to stop.

This study is really weird. Everything is like 100% in one group versus 0% in another group. Either something is really wrong here, or this one little test that separates mastery-oriented from helpless children constantly produces the strongest effects in all of psychology and is never wrong. None of the children whose test responses indicated that they thought ability was important to success ever monitored their own progress – not one – while over 95% of the children who said they thought effort was more important did. None of them ever expressed a positive statement about their own progress, while over two-thirds of the children who thought effort was more important did.

Normally I would assume these results are falsified, but I have looked for all of the usual ways of falsifying results and I can’t find any. Also, the boldest falsifier in the world wouldn’t have the courage to put down numbers like these. And a meta-analysis of all growth mindset studies finds more modest, but still consistent, effects, and only a little bit of publication bias.

So – is growth mindset the one concept in psychology which throws up gigantic effect sizes and always works? Or did Carol Dweck really, honest-to-goodness, make a pact with the Devil in which she offered her eternal soul in exchange for spectacular study results?

I don’t know. But here are a few things that predispose me towards the latter explanation. A warning – I am way out of my league here and post this only hoping it will spark further discussion.

II.

The first thing that bothers me is the history.

I’ve been trying really hard to trace its origin story, but it is pretty convoluted. It seems to have grown out of a couple of studies Carol Dweck and a few collaborators did in the seventies. But these studies generally found that a belief in innate ability was a positive factor alongside belief in growth mindset, with the problem children being the ones who attributed their success or failure to bad luck, or to external factors like the tests being rigged (which, by the way, they always were).

A good example of this genre is Learned Helplessness And Reinforcement Responsibility In Children. Its abstract describes the finding as: “Subjects who showed the largest performance decrements were those who took less personal responsibility for the outcomes of their actions…and who, when they did accept responsibility, attributed success and failure to presence or absence of ability rather than to expenditure of effort.”

But that seems like a somewhat loaded way of interpreting this table:

As you can see, the “persistent” children (the ones who kept going in the face of failure) had stronger belief in the role of ability in their successes (I+a) and failures (I-a) than the “helpless” children (the ones who gave up in the face of failure)! These don’t achieve statistical significance in this n = 10 study, but they do repeat across all four combinations of success x gender tested. The real finding of the study was that children who attributed their success or failure to any stable factor, be it effort or ability, did better than those who did not.

Likewise, in The Role Of Expectations And Attributions, Dweck describes her findings as “persistent and helpless children do not differ in the degree to which they attribute success to ability”. When you actually look at the paper, this is another case of the persistent children actually having a higher belief in the importance of ability, which fails to achieve statistical significance because the study is on a grand total of twelve children.

(I should say something else about this study. Dweck compared two interventions to make children less helpless and better at dealing with failure. In the first, she gave them a lot of easy problems which they inevitably succeeded on and felt smart about. In the second, she gave them difficult problems they were bound to fail, then told them it was because they weren’t working hard enough. Finally, both groups were challenged with the difficult bound-to-fail problems to see how hard they tried on them. The children who had been given the impossible problems before did better than the ones who felt smart because they’d only gotten easy problems. Dweck interpreted this to prove that telling children to work hard made them less helpless. To me the obvious conclusion is that children who are used to failing get less flustered when presented with impossible material than children who have artificially been made to succeed every moment until now.)

Then there’s there’s this, a preliminary to the second study I mentioned in Part I. Does it show the mastery-oriented children outperforming the helpless children on every measure. Yeah. But listen to this part from the discussion section:

The results revealed striking differences both in the pattern of performance and in the nature of the verbalizations made by helpless and mastery-oriented children following failure. It was particularly noteworthy that while the helpless children made the expected attributions to uncontrollable factors, the mastery-oriented children did not offer explanations for their failures

.
But if you look at the data, this doesn’t seem right.

Mastery-oriented children were about six times more likely to attribute their failures to the most uncontrollable factor of all – bad luck. They were also about six times more likely to attribute their failures to the task “not being fair”. This contradicts every previous study, including Dweck’s own. The whole field of attribution theory, which is intensely studied and which Dweck cites approvingly, says that attributing things to luck is a bad idea and attributing them to ability is, even if not as good as effort, pretty good. But Dweck finds that the kids who used ability attributions universally crashed and bomb, and the kids who attribute things to luck or the world being unfair do great.

It might not be fair for me to pick on these couple of small studies in particular when there’s so much out there, but the fact is that these are the first, and a lot of the reviews cite only these and a few theses which as far as I know were never published. So this is what I’ve got. And from what I’ve got, I find that until about 1980, every study including Dweck’s found that belief in ability was a protective factor. Suddenly this disappeared and was replaced with it being a toxic plague. What happened? I don’t know.

III.

The second thing that bothers me is the longitudinal view.

So you have your helpless, fixed-mindset, believe-in-innate-ability children. According to Dweck, they “…are so concerned with being and looking talented that they never realize their full potential. In a fixed mindset, the cardinal rule is to look talented at all costs. The second rule is don’t work too hard or practive too much…having to work casts doubt on your ability. The third rule is, when faced with setbacks, run away. They say things like ‘I would try to cheat on the next test’. They make excuses, they blame others, they make themselves feel better by looking down on those who have done worse.”

These people sound like total losers, and it’s clear Dweck endorses this reading:

“Almost every great athlete – Michael Jordan, Jackie Joyner-Kersee, Tiger Woods…has had a growth mindset. Not one of these athletes rested on their talent…research has repeatedly shown that a growth mindset fosters a healthier attitude toward practice and learning, a hunger for feedback, a greater ability to deal setbacks, and significantly better performance over time…over time those with a growth mindset appear to gain the advantage and begin to outperform their peers with a fixed mindset.”

Man, it sure would be awkward if fixed mindset students generally did better than growth-mindset ones, wouldn’t it?

Aronson, Fried, and Good (2001) looks at first like just another stunning growth mindset study. They do a half-hour intervention to teach college students growth mindset and find they are still getting higher grades a couple of months later (an effect so shocking I wrote about it here). But one thing they do kind of as an afterthought is measure the students’ general level of growth mindset, as well as some measures of academic performance before the intervention.

People with high growth mindset had lower GPA (decent effect size but not statistically significant) and lower SAT scores (which was statistically significant).

The authors are obviously uncomfortable with this, but they propose that people who get low SAT scores just tell themselves ability doesn’t matter/exist in order to protect their self-esteem since they don’t seem to have much of it.

And okay, that’s probably true (a commenter makes the equally good point that smart people may coast on their native intelligence without ever applying effort, and so accurately describe their experience as ability mattering but effort not doing so).

But if Dweck is to be believed, people with growth mindset are amazing ubermenschen and people with fixed mindset are disgusting failures at everything who hate learning and give up immediately and try to cheat. In the real world, however big the effect is, it is totally swamped by this proposed “people with low SAT scores protect their self-esteem or whatever” effect.

The same study also notes the awkward result that blacks are more likely to believe intelligence is flexible and growth-mindset-y than whites, even though blacks do worse in school and even though half the reason people are pushing growth mindset is to try to explain minority underperformance.

This is not an isolated finding. For example, Furnham (2003) finds in a sample of students at University College London that mindset is not related to academic performance. I’ve been told there’s a study from Pennsylvania that shows the same thing, though I can’t find it.

If you look hard enough, you can even notice this in Dweck’s studies themselves. One little-remarked-upon feature of Dweck’s work is that the helpless children and the mastery-oriented children always start out performing at the same level. It’s only after Dweck stresses them out with a failure that the mastery-oriented children recover gracefully and the helpless children go into free-fall.

But these are fifth-graders! For the two groups of children to do equally well on the first set of problems means that from first through fourth grade, their “helpless” “fixed-mindset” work-hating nature hasn’t impaired their ability to learn the material to a fifth-grade level one bit! (In this study, the fixed mindset children actually start out doing better; I can’t find any studies where the growth mindset children do).

When it’s convenient for her argument, Dweck herself admits that:

Some of the brightest, most skilled individuals exhibit the maladaptive pattern. Thus, it cannot be said that it is simply those with weak skills or histories of failure who (appropriately) avoid difficult tasks or whose skills prove fragile in the face of difficulty.

But I don’t think she follows the full implication of this statement, that despite being doomed to failure by their fixed mindset, these people have become “the brightest and most skilled individuals”.

Indeed, there has recently been some growth mindset studies done on gifted students, at elite colleges, or in high-level athletics. All of these dutifully show that people with fixed mindset respond much worse to whatever random contrived situation the experimenters produce. But thus far nobody has pointed out that there seem to be about as many of these people at, say, Stanford as there are anywhere else. If growth mindset was so great, you would expect fixed mindset people at Stanford to be as rare as, say, people with < 100 IQ are at Stanford. Given that you will search in vain for the latter but have no trouble finding a bunch of the former for your study on how great growth mindset is, it sure looks like IQ is useful but growth mindset isn't. When people are in a psychology study, the fixed mindset individuals universally crash and bomb and display themselves to be totally incapable of learning or working hard. At every other moment, they seem to be doing equally well or better than their growth mindset peers. What's going on? I have no idea. IV.

The third thing that bothers me is Performance Deficits Following Failure, a study which manages to be quite interesting despite coming from a university in a city that very possibly doesn’t exist.

They use a procedure much like Dweck’s. They make children do some problems. Then they give them some impossible problems. Then they give them more problems, to see if they’ve developed “learned helplessness” from their failure on the impossible ones. Dweck’s theory predicts that the fixed-mindset children would and the growth mindset children wouldn’t. The Bielefeld team wasn’t testing growth mindset, but they indeed found that a bunch of children got flustered and stopped trying and did poorly from then on.

Then they repeated the experiment, but this time they made it look like no one would know how the children did. They told the kids they would be on teams, and the scores of everyone on their team would be combined before anybody saw it. The kids could fail as much as they wanted, and it would never reflect on them.

After that, children did exactly as well after failure as they had before. There was no sign of any decrease, or any “fixed mindset” group that suddenly gave up in order to protect their ego.

This doesn’t strike me as fully consistent with mindset theory. In mindset theory, people are acting based on their own deep-seated beliefs. Once a fixed mindset child fails, that’s it, she knows she’s Not Intelligent, there’s no helping it, all she can do is sabotage herself on the problems in order to protest a spiteful world that has failed to recognize her genius blah blah blah. Instead, there seems to be a very social role to these failures. The Bielefeld team describes it as “self-esteem protection”, but that doesn’t make much sense to me, since if they were worried about their self-esteem they could still be worried about it when no one else knew their performance.

To me it seems like some kind of interaction between self-esteem and other-esteem. Fixed mindset people get flustered when they have to fail publicly in front of scientists. This doesn’t seem like an unreasonable problem to have. A more interesting question is why it’s correlated with belief in innate ability.

Suppose that the difference in “people who talk up innate ability” and “people who talk up hard work” maps onto a bigger distinction. Some people really want to succeed at a task; other people just care about about clocking in, going through the motions, and saying “I did what I could”.

Put the first group in front of an authoritative-looking scientist, tell them to solve a problem, and make sure they can’t. They’re going to view this as a major humiliation – they were supposed to get a result, and couldn’t. They’ll get very anxious, and of course anxiety impedes performance.

Put the second group in front of an authoritative-looking scientist, and they’ll notice that if they write some stuff that looks vaguely relevant for a few minutes until the scientist calls time, then whatever, they can say they tried and no one can bother them about it. They do exactly this, then demand an ‘A’ for effort. At no point do they experience any anxiety, so their performance isn’t impeded.

Put both groups on their own in private, and neither feels any humiliation, and they both do about equally well.

Now put them in real life. The success-oriented group will investigate how to study most effectively; the busywork-oriented group will try to figure out how many hours of studying they have to put in before other people won’t blame them if they fail, then put in exactly that amount. You’ll find the success-oriented group doing a bit better in school, even though they fail miserably in Dweck-style experiments.

And if an experimenter praises children for working hard, it will make them believe that all the experimenter cares about is their effort. Next problem, when the experimenter poses an impossible question, the child will beat their head against it for no reason, since that’s apparently what the experimenter wants. But if the experimenter praises a child’s ability, then the child will feel like the experimenter really wants them to correctly solve the questions. When the next question proves unsolveable, the child will admit it and expect the experimenter to be disappointed.

I doubt that this is the real phenomenon behind growth mindset, simply because it flatters my own prejudices in much the same way mindset theory flatters everyone else’s. But I think it shows there are a lot of different narratives we could put in this space, all of which would be able to explain some of the experimental results.

V.

I want to end by correcting a very important mistake about growth mindset that Dweck mostly avoids but which her partisans constantly commit egregiously. Take this article, Why A Growth Mindset Is The Only Way To Learn:

[Some people think] you’ll always have a set IQ. You’re only qualified for the career you majored in. You’ll never be any better at playing soccer or dating or taking risks. Your life and character are as certain as a map. The problem is, this mindset will make you complacent, rob your self-esteem and bring meaningful education to a halt.

In short, it’s an intellectual disease and patently untrue.

The article goes on to show how growth mindset proves talent is “a myth”, a claim repeated by growth mindset cheerleader articles like Debunking The Genius Myth and The Learning Myth: Why I’ll Never Tell My Son He’s Smart and this woman who says we need to debunk the idea of innate talent.

Suppose everything I said in parts I – IV was wrong, and growth mindset is 100% true exactly as written.

This still would not provide an iota of evidence against the idea that innate talent / IQ / whatever is by far the most important factor determining success.

Consider. We know from countless studies that strong religious belief increases your life expectancy, makes you happier, reduces your risk of depression and reduces crime. Clearly believing in, say, Christianity has lots of useful benefits. But no one would dare argue that proves Christianity true. It doesn’t even imply it.

Likewise, mindset theory suggests that believing intelligence to be mostly malleable has lots of useful benefits. That doesn’t mean intelligence really is mostly malleable. Consider, if you will, my horrible graph:

Suppose this is one of Dweck’s experiments on three children. Each has a different level of innate talent, represented by point 1. After they get a growth mindset and have the right attitude and practice a lot, they make it to point 2.

Two things are simultaneously true of this model. First, all of Dweck’s experiments will come out exactly as they did in the real world. Children who adopt a growth mindset and try hard and practice will do better than children who don’t. If many of them are aggregated into groups, the growth mindset group will on average do better than the ability-focused group. Intelligence is flexible, and if you don’t bother practicing than you fail to realize your full potential.

Second, the vast majority of difference between individuals is due to different levels of innate talent. Alice, no matter how hard she practices, will never be as good as Bob. Bob, if he practices very hard, will become better than Carol was at the start, but never as good as Carol if she practices as hard as Bob does. The difference between Alice and Carol is a vast, unbridgeable gap which growth mindset has nothing whatsoever to say about.

Here is a graph which is less terrible because it was not made by me. I have taken it from one of the two other sources I have found on the entire Internet that don’t like growth mindset:

We can argue all day about whether poor students do worse because they have bad health, because they have bad genes, because they have bad upbringings, or because society is fixed against them. We have argued about that all day before here, and it’s been pretty interesting.

But in this case it doesn’t matter. If the only thing that affects success is how much effort you put in, poor kids seem to be putting in a heck of a lot less effort in a surprisingly linear way. But the smart money’s not on that theory.

A rare point of agreement between hard biodeterminists and hard socialists is that telling kids that they’re failing because they just don’t have the right work ethic is a crappy thing to do. It’s usually false and it will make them feel terrible. Behavioral genetics studies show pretty clearly that at least 50% of success at academics and sports is genetic; various sociologists have put a lot of work into proving that your position in a biased society covers a pretty big portion of the remainder. If somebody who was born with the dice stacked against them works very hard, then they might find themselves at A2 above. To deny this in favor of a “everything is about how hard you work” is to offend the sensibilities of sensible people on the left and right alike.

Go back to that 1975 paper above on “Role Of Expectations And Attributions” and look more closely at the proposed intervention to help these poor fixed mindset students:

Twelve extremely helpless children were identified [and tested on how many math problems they could solve in a certain amount of time]…the criterion number was set one above the number he was generally able to complete within the time limit. On these trials, he was stopped one or two problems short of criterion, his performance was compared to the criterion number required, and experimenter verbally attributed the failure to insufficient effort.

So basically, you take the most vulnerable people, set them tasks you know they’ll fail at, then lecture them about how they only failed because of insufficient effort.

Imagine a boot stamping on a human face forever, saying “YOUR PROBLEM IS THAT YOU’RE JUST NOT TRYING NOT TO BE STAMPED ON HARD ENOUGH”.

And maybe this is worth it, if it builds a growth mindset that allows the child to be more successful in school, sports, and in the rest of her life. But you’re not “debunking the myth of genius”. Genius remains super-important, just like conscientiousness and wealth and health and privilege and everything else. No, you’re telling a Noble Lie to the children because you think it’s useful. You can make it palatable by saying “Well, we’re not denying reality, we’re just selectively emphasizing certain parts of reality, but in the end that’s what you’re doing. If you can square that with your moral system, go ahead.

But I remain agnostic. There are some really good – diabolically good? – studies showing that it works in certain lab situations. There’s a lot of excellent research behind it and a lot of brilliant people giving it their support. But there are also other studies showing that it has no long-term real-world effects that we can measure, and others that might (or might not?) contradict its predictions in other ways. I have only the barest of ideas how to square those facts, and I look forward to hearing from anyone who has more.

Willpower: Rediscovering the Greatest Human Strength by Roy Baumeister and John Tierney attracted me with the following pitch: there are only two quantities in psychology that have been robustly linked to a broad range of important life outcomes. One is IQ and isn’t changeable. The other is willpower and is easily changeable. Therefore, study willlpower.

I expected the book to center around Baumeister’s groundbreaking experiments in “ego depletion” – where people who are forced to expend willpower on one task have less willpower left over for future tasks – and “ego repletion”, where people who have been depleted of willpower get it back after taking some glucose. I’d been left kind of confused by competing claims about those studies and I hoped this book would fill out my knowledge of them and settle my confusion.

Instead, it spent a couple of chapters mentioning their existence and praising them as revolutionary, and then got deep enough into Pop Science Self=Help Book Mode to be almost a self-parody. Chapter three is called “The To-Do List From God To Drew Carey” and illustrates why to-do lists are a great idea with anecdotes from Drew Carey’s life and the Bible. Chapter four is the same but with Eliot Spitzer. Chapter six is David Blaine, Chapter seven is H.M. Stanley, Chapter eight is Eric Clapton, and Chapter ten is Oprah. All of these people apparently have important lessons about willpower to teach us, of which the interesting ones are:

– Willpower is a limited resource that is depleted by use and restored by glucose

– Having to make too many small decisions in a day causes “decision fatigue”, leading you to be exhausted and make bad decisions.

– Using willpower a lot strengthens your willpower, allowing you to be more effective.

– People with more willpower do much better in life; for example, in the famous “marshmallow test”, children who were able to resist eating a marshmallow for a few minutes in order to get two marshmallows had better life outcomes twenty years later.

– Careful quantification of goals, setting precommitments, and publicizing your success or failure to other people helps you stick to resolutions. (If you are wondering exactly how to do this, this might be a good time to mention there’s a new ad on the sidebar for Beeminder, a company that manages this for you with some neat evidence-based tricks.)

– Religious people have more willpower than non-religious people for some reason. But you have to believe if you want to get this benefit – you can’t just hang out at church and go through the motions in order to reap the willpower gains.

= Chinese-Americans do better than white Americans on willpower tests from toddlerhood toward adulthood. This is the most likely reason Chinese people outperform whites in the real world, and in fact although average white and Chinese IQ are pretty similar, Chinese people can break into “elite” professions at a lower IQ threshold than whites because their increased self-control compensates. The book admits this may be partly genetic, but also attributes some of it to Chinese parents teaching their children discipline and setting hard goals, which they contrast with white parents who tell their kids to “have fun” and “have high self-esteem” and “be self-directed”. Obviously in order to believe this result you’d have to believe parenting styles can affect children, which this book takes on faith but which is on shaky ground.

– Amount of self-control does not affect body weight (!) or success at dieting (!!) very much.

Overall there were some interesting findings in here, even if I found the pop sci tone a little bit over-the-top after a while.

II.

But I was disappointed. The reason I got this book is that there’s a big debate going on over “willpower”, “ego depletion”, and their younger cousins “growth mindset” and “grit”. All of these are slightly different constructs, but they’re all measures of stick-to-it-ness, and all of their proponents make grand claims about how small interventions to make people have them will bring those people success at school, work, and life. On the other hand, there are a lot of other people who think this whole area is a load of bunk.

For example, Mischel’s marshmallow test started all this off by “proving” that children who were able to delay gratification longest had higher SAT scores, higher parent-rated competence, and better coping skills. But now the test is under fire as people question whether it just shows that people from good environments learn to be more trusting and so more likely to believe the researcher’s promise of extra rewards for delayed gratification. Other people ask if it just shows that kids who are smart enough to think of good strategies to distract themselves are also smart enough to get good SAT scores and all of the other positive correlates of high IQ. Still other people point out the very low sample size, the reverse correlation in other subgroups, and an apparent failure to replicate. Mischel fires back in an Atlantic article where he says of course he took these things into account, that the test was done on a homogenous upper-class population.

This book just says the marshmallow test proves willpower is important, and leaves it at that.

Or how about the idea that glucose is the limited resource that willpower depletes? Robert Kurzban very correctly points out that the metabolic math doesn’t come close to adding up – we know how much glucose things in the body use, and these short little willpower tasks aren’t really going to affect blood glucose levels at all. Also, turns out that if you rinse your mouth out with a tasty glucose solution, you get just the same amount of ego replenishment even though none of the glucose actually entered your body. And for that matter, how come I can’t get infinite willpower just by snacking while I work? How come M&Ms don’t work as a poor man’s Adderall?

Kurzban goes further and says he doesn’t believe in willpower as a limited resource at all. He notes that even when you’ve stopped studying because you’re too ego-depleted and exhausted to make yourself go on any further, if I offer you a million dollars to study another hour then you’ll do it. Guess that resource wasn’t so depleted after all. He proposes a different model of willpower, where it’s your brain’s way of nagging you about the opportunity cost of your actions – “you’ve been studying three whole hours, don’t you think you could use that lobe of your brain for something else now?” But this strikes me as ridiculous – my brain is very concerned that it has better things to do than study, but is perfectly happy with me playing Civilization IV: Fall From Heaven forever or simply lying in bed doing nothing?

Finally, Carol Dweck finds that willpower is only depletable if you think it is, which sounds like exactly the sort of thing Carol Dweck would find. If Carol Dweck ever became an oncologist, we would have to revise all the medical textbooks to say that people only get cancer if they think they will.

There is a meta-analysis of about a hundred studies said Baumeister was basically right about everything. On the other hand, Baumeister’s theory failed what sounds like a formal replication. So it’s complicated.

I really want to know what willpower is. It seems like one of the big challenges of my life; I never have enough willpower for everything I want to do, and I’d at least like to have a theory of what I’m up against. This book did not give me enough information to navigate the controversy. Instead, it totally denied there was any controversy and spent chapter after chapter on cute pieces of trivia interspersed with stories about Drew Carey and Oprah.

III.

So let me end this with some thoughts that a good explanation of willpower should take into account.

First, mental willpower seems a lot like physical willpower – by which I mean our ability to push through exhaustion to keep exercising. Kurzban complains that willpower can’t be a limited resource, because even after it’s all depleted you can still force yourself to keep going for a big enough reward. But the same is true of exercise. I’ll get exhausted and stop running after a certain number of miles, but if you offer me $1 million to run another one I’ll probably make it. And the studies showing that rinsing your mouth out with glucose gives you the same willpower boost as actually drinking is identical to the results of similar studies measuring exercise duration.

So it’s probably worth asking what exactly causes exercise fatigue. This is less firmly known than I expected, but it seems to be a combination of decreasing levels of inputs (especially glycogen stores in the muscles), buildup of toxic metabolic waste (including heat, lactic acid, etc), and cellular damage. Exercise long enough and your muscles need time to replenish their stores, clear away waste, and repair themselves.

Second, mental willpower seems to do something a lot like budgeting. Money, despite being pretty much the classic example of a limited resource, shares some of the features of willpower that Kurzban mocks. A few days ago a pipe burst, my house flooded, and I may have to spend several thousand dollars unflooding it. This means I will have no money for the next forever and then some, and if you ask me to meet you at a fancy restaurant I’ll probably refuse on financial grounds. But if my mother is on her deathbed and her final wish is to see me one last time, I will find the money to get an expensive flight to California. How’s that any different from me not having enough willpower to keep studying until you offer me a million dollars to do so?

Likewise, if tomorrow my boss offers me a $5000 bonus to be handed out next month, I’ll probably relax my budgetary constraints and start buying myself nice things even before I get the paycheck. How’s that any different from the body reacting to sugar when it’s in the mouth but not the bloodstream?

Finally, the question that I as a psychiatrist find most interesting – how come drugs can change willpower so dramatically? The guy who can’t concentrate on a project for more than five minutes straight will pull a whole week of all-nighters when he’s on Adderall or modafinil. Those certainly aren’t increasing blood glucose, so what’s up?

The model that makes the most sense to me is of a stupid default system running on short-term reinforcement learning, plus an evolutionarily novel (and therefore poorly implemented) executive system that can overrule the default. The executive system’s overrule isn’t a simple veto, but a constant action, the same way holding your hand high in the air for a long period requires constant action by your muscles. This effort is metabolically costly in the same way that using muscles is metabolically costly, and so your body runs a general-purpose budgeting function on it that convinces you to turn it off before it overheats. Given enough incentive, you can let it overheat, but then it’s going to be damaged and need to repair itself for a few days before you can use it effectively again. This seems to fit all the evidence except the drugs, which I interpret as acting on the default system so that you don’t need to bring in the executive planner. I freely admit this is sort of cheating.

Overall I recommend Willpower if you want a quick and fun survey of a bunch of loosely connected psychology topics, but not if you want a deep and balanced exploration into the literature of anything.

Here are three interesting psychological studies:

1. Kirschenbaum, Malett, and Humphrey gave students a three month course in how to make monthly plans, then followed them up a year later to see how well their grades were doing. The students who made the plans got an average GPA of 3.3 compared to the students who didn’t getting 2.5. They concluded that plan-making skills are academically important, and that benefits persist at least one year after completion of the plan-making course.

2. Aronson asked Stanford students to write a letter to a middle school “pen pal” urging them to adopt a “growth mindset”. Since this was a psych study, it was all lies and there was no pen pal; the study examined whether writing a letter urging growth mindset made the students themselves have a growth mindset and whether this improved grades. Three months later, the students who wrote the letter had higher GPAs.

3. Oaten and Chang helped undergraduates set up an 8-week time management program involving schedules and diaries. The students who participated not only had better time-management, they also studied more, smoked less, drank less alcohol, exercised more, ate a better diet, spent less money, rated their emotions as better, missed fewer appointments, and were less likely to leave dishes in the sink (really!).

So, remember a couple of weeks ago when I wrote about some psychiatrists conducting a really big (n =~ 1000) study about an early intervention program for troubled youth? A program that cost $58,000 per person and lasted ten years?

And remember how, although it was deemed a success, it was deemed a success because it had modest effects on a couple of outcomes, without improving the really big ones like school retention, employment, or incarceration?

So on the one hand, having a short discussion about making monthly plans will boost your GPA almost a whole point a year later. On the other hand, ten years of private tutoring and pretty much every social service known to mankind will do next to nothing.

This suggests a dilemma: either psychological research sucks or everything else sucks.

I mean, you tell your social engineer “Here’s ten thousand dollars and a thousand hours of class time per pupil per year, go teach our kids stuff,” and they try their hardest.

And then some researcher comes along, performs a quick experimental manipulation (the pen pal one probably took 30 minutes and 30 cents) and dramatically improves outcomes over what the social engineer was able to do on her own.

Then one gets the impression that the social engineer was not using their $10,000 and 1K hours very wisely.

And if it were just the one example, then we could say Carol Dweck or Roy Baumeister or whoever is a genius, the rest of us couldn’t have been expected to come up with that, now that we know we’ll reform the system. But these results have been coming in several times a year for decades. If we’ve been adopting all of them, why aren’t people much better in every way? If we haven’t been adopting them, why not?

My money is on the other branch of the dilemma. The reason the $58,000 study got so much less impressive results is that it was run by medical professionals to medical standards, meaning it only showed the effects that were really there. The reason psychology gets such impressive results is…

Okay. I have only skimmed these three studies, so I don’t want to make it sound like I’m definitively crushing them. But here are some worrying things I notice.

The first study results are actually limited to a small subgroup with I think a single-digit number of students per cell.

The second study results work only on a complex statistical manipulation and disappear when you do basic correlation; further, although the manipulation is supposed to work by increasing trait growth mindset, the correlation between trait growth mindset and academic achievement when correlated directly is actually negative across all variables and in some cases significantly so (!)

The third study is really about stress-related behaviors during an examination period, meaning that all they showed was that people who as part of their time management course were forced to study in a carefully scheduled way for a term show less stress-related behavior during the term exam period, which makes more sense as they probably studied more earlier, had less studying left to do during the exam period, had more free time, and were less stressed. This context was dropped by the popular science press, turning the study into proof that good time management in general always produced all of these effects.

And although there are several other studies I have not been able to find equally worrying flaws with, if they report massive long-term gains from seemingly minor interventions, I expect they’re there and I just missed them.

Basically, you remember this chart?

Source: xkcd

In “Crazy Phenomenon”, add “any large and persistent effect from social psychology”. In “If it worked…” add “education, rehab, and mental health”. In “Are They?”, add “not nearly as much as I would expect”.

I.

I sometimes blog about research into IQ and human intelligence. I think most readers of this blog already know IQ is 50% to 80% heritable, and that it’s so important for intellectual pursuits that eminent scientists in some fields have average IQs around 150 to 160. Since IQ this high only appears in 1/10,000 people or so, it beggars coincidence to believe this represents anything but a very strong filter for IQ (or something correlated with it) in reaching that level. If you saw a group of dozens of people who were 7’0 tall on average, you’d assume it was a basketball team or some other group selected for height, not a bunch of botanists who were all very tall by coincidence.

A lot of people find this pretty depressing. Some worry that taking it seriously might damage the “growth mindset” people need to fully actualize their potential. This is important and I want to discuss it eventually, but not now. What I want to discuss now is people who feel personally depressed. For example, a comment from last week:

I’m sorry to leave self a self absorbed comment, but reading this really upset me and I just need to get this off my chest…How is a person supposed to stay sane in a culture that prizes intelligence above everything else – especially if, as Scott suggests, Human Intelligence Really Is the Key to the Future – when they themselves are not particularly intelligent and, apparently, have no potential to ever become intelligent? Right now I basically feel like pond scum.

I hear these kinds of responses every so often, so I should probably learn to expect them. I never do. They seem to me precisely backwards. There’s a moral gulf here, and I want to throw stories and intuitions at it until enough of them pile up at the bottom to make a passable bridge. But first, a comparison:

Some people think body weight is biologically/genetically determined. Other people think it’s based purely on willpower – how strictly you diet, how much you can bring yourself to exercise. These people get into some pretty acrimonious debates.

Overweight people, and especially people who feel unfairly stigmatized for being overweight, tend to cluster on the biologically determined side. And although not all believers in complete voluntary control of weight are mean to fat people, the people who are mean to fat people pretty much all insist that weight is voluntary and easily changeable.

Although there’s a lot of debate over the science here, there seems to be broad agreement on both sides that the more compassionate, sympathetic, progressive position, the position promoted by the kind of people who are really worried about stigma and self-esteem, is that weight is biologically determined.

And the same is true of mental illness. Sometimes I see depressed patients whose families really don’t get it. They say “Sure, my daughter feels down, but she needs to realize that’s no excuse for shirking her responsibilities. She needs to just pick herself up and get on with her life.” On the other hand, most depressed people say that their depression is more fundamental than that, not a thing that can be overcome by willpower, certainly not a thing you can just ‘shake off’.

Once again, the compassionate/sympathetic/progressive side of the debate is that depression is something like biological, and cannot easily be overcome with willpower and hard work.

One more example of this pattern. There are frequent political debates in which conservatives (or straw conservatives) argue that financial success is the result of hard work, so poor people are just too lazy to get out of poverty. Then a liberal (or straw liberal) protests that hard work has nothing to do with it, success is determined by accidents of birth like who your parents are and what your skin color is et cetera, so the poor are blameless in their own predicament.

I’m oversimplifying things, but again the compassionate/sympathetic/progressive side of the debate – and the side endorsed by many of the poor themselves – is supposed to be that success is due to accidents of birth, and the less compassionate side is that success depends on hard work and perseverance and grit and willpower.

The obvious pattern is that attributing outcomes to things like genes, biology, and accidents of birth is kind and sympathetic. Attributing them to who works harder and who’s “really trying” can stigmatize people who end up with bad outcomes and is generally viewed as Not A Nice Thing To Do.

And the weird thing, the thing I’ve never understood, is that intellectual achievement is the one domain that breaks this pattern.

Here it’s would-be hard-headed conservatives arguing that intellectual greatness comes from genetics and the accidents of birth and demanding we “accept” this “unpleasant truth”.

And it’s would-be compassionate progressives who are insisting that no, it depends on who works harder, claiming anybody can be brilliant if they really try, warning us not to “stigmatize” the less intelligent as “genetically inferior”.

I can come up with a few explanations for the sudden switch, but none of them are very principled and none of them, to me, seem to break the fundamental symmetry of the situation. I choose to maintain consistency by preserving the belief that overweight people, depressed people, and poor people aren’t fully to blame for their situation – and neither are unintelligent people. It’s accidents of birth all the way down. Intelligence is mostly genetic and determined at birth – and we’ve already determined in every other sphere that “mostly genetic and determined at birth” means you don’t have to feel bad if you got the short end of the stick.

Consider for a moment Srinivasa Ramanujan, one of the greatest mathematicians of all time. He grew up in poverty in a one-room house in small-town India. He taught himself mathematics by borrowing books from local college students and working through the problems on his own until he reached the end of the solveable ones and had nowhere else to go but inventing ways to solve the unsolveable ones.

There are a lot of poor people in the United States today whose life circumstances prevented their parents from reading books to them as a child, prevented them from getting into the best schools, prevented them from attending college, et cetera. And pretty much all of those people still got more educational opportunities than Ramanujan did.

And from there we can go in one of two directions. First, we can say that a lot of intelligence is innate, that Ramanujan was a genius, and that we mortals cannot be expected to replicate his accomplishments.

Or second, we can say those poor people are just not trying hard enough.

Take “innate ability” out of the picture, and if you meet a poor person on the street begging for food, saying he never had a chance, your reply must be “Well, if you’d just borrowed a couple of math textbooks from the local library at age 12, you would have been a Fields Medalist by now. I hear that pays pretty well.”

The best reason not to say that is that we view Ramanujan as intellectually gifted. But the very phrase tells us where we should classify that belief. Ramanujan’s genius is a “gift” in much the same way your parents giving you a trust fund on your eighteenth birthday is a “gift”, and it should be weighted accordingly in the moral calculus.

II.

I shouldn’t pretend I’m worried about this for the sake of the poor. I’m worried for me.

My last IQ-ish test was my SATs in high school. I got a perfect score in Verbal, and a good-but-not-great score in Math.

And in high school English, I got A++s in all my classes, Principal’s Gold Medals, 100%s on tests, first prize in various state-wide essay contests, etc. In Math, I just barely by the skin of my teeth scraped together a pass in Calculus with a C-.

Every time I won some kind of prize in English my parents would praise me and say I was good and should feel good. My teachers would hold me up as an example and say other kids should try to be more like me. Meanwhile, when I would bring home a report card with a C- in math, my parents would have concerned faces and tell me they were disappointed and I wasn’t living up to my potential and I needed to work harder et cetera.

And I don’t know which part bothered me more.

Every time I was held up as an example in English class, I wanted to crawl under a rock and die. I didn’t do it! I didn’t study at all, half the time I did the homework in the car on the way to school, those essays for the statewide competition were thrown together on a lark without a trace of real effort. To praise me for any of it seemed and still seems utterly unjust.

On the other hand, to this day I believe I deserve a fricking statue for getting a C- in Calculus I. It should be in the center of the schoolyard, and have a plaque saying something like “Scott Alexander, who by making a herculean effort managed to pass Calculus I, even though they kept throwing random things after the little curly S sign and pretending it made sense.”

And without some notion of innate ability, I don’t know what to do with this experience. I don’t want to have to accept the blame for being a lazy person who just didn’t try hard enough in Math. But I really don’t want to have to accept the credit for being a virtuous and studious English student who worked harder than his peers. I know there were people who worked harder than I did in English, who poured their heart and soul into that course – and who still got Cs and Ds. To deny innate ability is to devalue their efforts and sacrifice, while simultaneously giving me credit I don’t deserve.

Meanwhile, there were some students who did better than I did in Math with seemingly zero effort. I didn’t begrudge those students. But if they’d started trying to say they had exactly the same level of innate ability as I did, and the only difference was they were trying while I was slacking off, then I sure as hell would have begrudged them. Especially if I knew they were lazing around on the beach while I was poring over a textbook.

I tend to think of social norms as contracts bargained between different groups. In the case of attitudes towards intelligence, those two groups are smart people and dumb people. Since I was both at once, I got to make the bargain with myself, which simplified the bargaining process immensely. The deal I came up with was that I wasn’t going to beat myself up over the areas I was bad at, but I also didn’t get to become too cocky about the areas I was good at. It was all genetic luck of the draw either way. In the meantime, I would try to press as hard as I could to exploit my strengths and cover up my deficiencies. So far I’ve found this to be a really healthy way of treating myself, and it’s the way I try to treat others as well.

III.

The theme continues to be “Scott Relives His Childhood Inadequacies”. So:

When I was 6 and my brother was 4, our mom decided that as an Overachieving Jewish Mother she was contractually obligated to make both of us learn to play piano. She enrolled me in a Yamaha introductory piano class, and my younger brother in a Yamaha ‘cute little kids bang on the keyboard’ class.

A little while later, I noticed that my brother was now with me in my Introductory Piano class.

A little while later, I noticed that my brother was now by far the best student in my Introductory Piano Class, even though he had just started and was two or three years younger than anyone else there.

A little while later, Yamaha USA flew him to Japan to show him off before the Yamaha corporate honchos there.

Well, one thing led to another, and right now if you Google my brother’s name you get a bunch of articles like this one:

The evidence that Jeremy [Alexander] is among the top jazz pianists of his generation is quickly becoming overwhelming: at age 26, Alexander is the winner of the Nottingham International Jazz Piano Competition, a second-place finisher in the Montreux Jazz Festival Solo Piano Competition, a two-time finalist for the American Pianist Association’s Cole Porter Fellowship, and a two-time second-place finisher at the Phillips Jazz Competition. Alexander, who was recently named a Professor of Piano at Western Michigan University’s School of Music, made a sold-out solo debut at Carnegie Hall in 2012, performing Debussy’s Etudes in the first half and jazz improvisations in the second half.

Meanwhile, I was always a mediocre student at Yamaha. When the time came to try an instrument in elementary school, I went with the violin to see if maybe I’d find it more to my tastes than the piano. I was quickly sorted into the remedial class because I couldn’t figure out how to make my instrument stop sounding like a wounded cat. After a year or so of this, I decided to switch to fulfilling my music requirement through a choir, and everyone who’d had to listen to me breathed a sigh of relief.

Every so often I wonder if somewhere deep inside me there is the potential to be “among the top musicians of my generation.” I try to recollect whether my brother practiced harder than I did. My memories are hazy, but I don’t think he practiced much harder until well after his career as a child prodigy had taken off. The cycle seemed to be that every time he practiced, things came fluidly to him and he would produce beautiful music and everyone would be amazed. And this must have felt great, and incentivized him to practice more, and that made him even better, so that the beautiful music came even more fluidly, and the praise became more effusive, until eventually he chose a full-time career in music and became amazing. Meanwhile, when I started practicing it always sounded like wounded cats, and I would get very cautious praise like “Good job, Scott, it sounded like that cat was hurt a little less badly than usual,” and it made me frustrated, and want to practice less, which made me even worse, until eventually I quit in disgust.

On the other hand, I know people who want to get good at writing, and make a mighty resolution to write two hundred words a day every day, and then after the first week they find it’s too annoying and give up. These people think I’m amazing, and why shouldn’t they? I’ve written a few hundred to a few thousand words pretty much every day for the past ten years.

But as I’ve said before, this has taken exactly zero willpower. It’s more that I can’t stop even if I want to. Part of that is probably that when I write, I feel really good about having expressed exactly what it was I meant to say. Lots of people read it, they comment, they praise me, I feel good, I’m encouraged to keep writing, and it’s exactly the same virtuous cycle as my brother got from his piano practice.

And so I think it would be too easy to say something like “There’s no innate component at all. Your brother practiced piano really hard but almost never writes. You write all the time, but wimped out of practicing piano. So what do you expect? You both got what you deserved.”

I tried to practice piano as hard as he did. I really tried. But every moment was a struggle. I could keep it up for a while, and then we’d go on vacation, and there’d be no piano easily available, and I would be breathing a sigh of relief at having a ready-made excuse, and he’d be heading off to look for a piano somewhere to practice on. Meanwhile, I am writing this post in short breaks between running around hospital corridors responding to psychiatric emergencies, and there’s probably someone very impressed with that, someone saying “But you had such a great excuse to get out of your writing practice!”

I dunno. But I don’t think of myself as working hard at any of the things I am good at, in the sense of “exerting vast willpower to force myself kicking and screaming to do them”. It’s possible I do work hard, and that an outside observer would accuse me of eliding how hard I work, but it’s not a conscious elision and I don’t feel that way from the inside.

Ramanujan worked very hard at math. But I don’t think he thought of it as work. He obtained a scholarship to the local college, but dropped out almost immediately because he couldn’t make himself study any subject other than math. Then he got accepted to another college, and dropped out again because they made him study non-mathematical subjects and he failed a physiology class. Then he nearly starved to death because he had no money and no scholarship. To me, this doesn’t sound like a person who just happens to be very hard-working; if he had the ability to study other subjects he would have, for no reason other than that it would have allowed him to stay in college so he could keep studying math. It seems to me that in some sense Ramanujan was incapable of putting hard work into non-math subjects.

I really wanted to learn math and failed, but I did graduate with honors from medical school. Ramanujan really wanted to learn physiology and failed, but he did become one of history’s great mathematicians. So which one of us was the hard worker?

People used to ask me for writing advice. And I, in all earnestness, would say “Just transcribe your thoughts onto paper exactly like they sound in your head.” It turns out that doesn’t work for other people. Maybe it doesn’t work for me either, and it just feels like it does.

But you know what? When asked about one of his discoveries, a method of simplifying a very difficult problem to a continued fraction, Ramanujan described his thought process as: “It is simple. The minute I heard the problem, I knew that the answer was a continued fraction. ‘Which continued fraction?’ I asked myself. Then the answer came to my mind”.

And again, maybe that’s just how it feels to him, and the real answer is “study math so hard that you flunk out of college twice, and eventually you develop so much intuition that you can solve problems without thinking about them.”

(or maybe the real answer is “have dreams where obscure Hindu gods appear to you as drops of blood and reveal mathematical formulae”. Ramanujan was weird).

But I still feel like there’s something going on here where the solution to me being bad at math and piano isn’t just “sweat blood and push through your brain’s aversion to these subjects until you make it stick”. When I read biographies of Ramanujan and other famous mathematicians, there’s no sense that they ever had to do that with math. When I talk to my brother, I never get a sense that he had to do that with piano. And if I am good enough at writing to qualify to have an opinion on being good at things, then I don’t feel like I ever went through that process myself.

So this too is part of my deal with myself. I’ll try to do my best at things, but if there’s something I really hate, something where I have to go uphill every step of the way, then it’s okay to admit mediocrity. I won’t beat myself up for not forcing myself kicking and screaming to practice piano. And in return I won’t become too cocky about practicing writing a lot. It’s probably some kind of luck of the draw either way.

IV.

I said before that this wasn’t just about poor people, it was about me being selfishly worried for my own sake. I think I might have given the mistaken impression that I merely need to justify to myself why I can’t get an A in math or play the piano. But it’s much worse than that.

The rationalist community tends to get a lot of high-scrupulosity people, people who tend to beat themselves up for not doing more than they are. It’s why I push giving 10% to charity, not as some kind of amazing stretch goal that we need to guilt people into doing, but as a crutch, a sort of “don’t worry, you’re still okay if you only give ten percent”. It’s why there’s so much emphasis on “heroic responsibility” and how you, yes you, have to solve all the world’s problems personally. It’s why I see red when anyone accuses us of entitlement, since it goes about as well as calling an anorexic person fat.

And we really aren’t doing ourselves any favors. For example, Nick Bostrom writes:

Searching for a cure for aging is not just a nice thing that we should perhaps one day get around to. It is an urgent, screaming moral imperative. The sooner we start a focused research program, the sooner we will get results. It matters if we get the cure in 25 years rather than in 24 years: a population greater than that of Canada would die as a result.

If that bothers you, you definitely shouldn’t read Astronomical Waste.

Yet here I am, not doing anti-aging research. Why not?

Because I tried doing biology research a few times and it was really hard and made me miserable. You know how in every science class, when the teacher says “Okay, pour the white chemical into the grey chemical, and notice how it turns green and begins to bubble,” there’s always one student who pours the white chemical into the grey chemical, and it just forms a greyish-white mixture and sits there? That was me. I hated it, I didn’t have the dexterity or the precision of mind to do it well, and when I finally finished my required experimental science classes I was happy never to think about it again. Even the abstract intellectual part of it – the one where you go through data about genes and ligands and receptors in supercentenarians and shake it until data comes out – requires exactly the kind of math skills that I don’t have.

Insofar as this is a matter of innate aptitude – some people are cut out for biology research and I’m not one of them – all is well, and my decision to get a job I’m good at instead is entirely justified.

But insofar as there’s no such thing as innate aptitude, just hard work and grit – then by not being gritty enough, I’m a monster who’s complicit in the death of a population greater than that of Canada.

Insofar as there’s no such thing as innate aptitude, I have no excuse for not being Aubrey de Grey. Or if Aubrey de Grey doesn’t impress you much, Norman Borlaug. Or if you don’t know who either of those two people are, Elon Musk.

I once heard a friend, upon his first use of modafinil, wonder aloud if the way they felt on that stimulant was the way Elon Musk felt all the time. That tied a lot of things together for me, gave me an intuitive understanding of what it might “feel like from the inside” to be Elon Musk. And it gave me a good tool to discuss biological variation with. Most of us agree that people on stimulants can perform in ways it’s difficult for people off stimulants to match. Most of us agree that there’s nothing magical about stimulants, just changes to the levels of dopamine, histamine, norepinephrine et cetera in the brain. And most of us agree there’s a lot of natural variation in these chemicals anyone. So “me on stimulants is that guy’s normal” seems like a good way of cutting through some of the philosophical difficulties around this issue.

…which is all kind of a big tangent. The point I want to make is that for me, what’s at stake in talking about natural variations in ability isn’t just whether I have to feel like a failure for not getting an A in high school calculus, or not being as good at music as my brother. It’s whether I’m a failure for not being Elon Musk. Specifically, it’s whether I can say “No, I’m really not cut out to be Elon Musk” and go do something else I’m better at without worrying that I’m killing everyone in Canada.

V.

The proverb says: “Everyone has somebody better off than they are and somebody worse off than they are, with two exceptions.” When we accept that we’re all in the “not Elon Musk” boat together (with one exception) a lot of the status games around innate ability start to seem less important.

Every so often an overly kind commenter here praises my intelligence and says they feel intellectually inadequate compared to me, that they wish they could be at my level. But at my level, I spend my time feeling intellectually inadequate compared to Scott Aaronson. Scott Aaronson describes feeling “in awe” of Terence Tao and frequently struggling to understand him. Terence Tao – well, I don’t know if he’s religious, but maybe he feels intellectually inadequate compared to God. And God feels intellectually inadequate compared to Johann von Neumann.

So there’s not much point in me feeling inadequate compared to my brother, because even if I was as good at music as my brother, I’d probably just feel inadequate for not being Mozart.

And asking “Well what if you just worked harder?” can elide small distinctions, but not bigger ones. If my only goal is short-term preservation of my self-esteem, I can imagine that if only things had gone a little differently I could have practiced more and ended up as talented as my brother. It’s a lot harder for me to imagine the course of events where I do something different and become Mozart. Only one in a billion people reach a Mozart level of achievement; why would it be me?

If I loved music for its own sake and wanted to be a talented musician so I could express the melodies dancing within my heart, then none of this matters. But insofar as I want to be good at music because I feel bad that other people are better than me at music, that’s a road without an end.

This is also how I feel of when some people on this blog complain they feel dumb for not being as smart as some of the other commenters on this blog.

I happen to have all of your IQ scores in a spreadsheet right here (remember that survey you took?). Not a single person is below the population average. The first percentile for IQ here – the one such that 1% of respondents are lower and 99% of respondents are higher – is – corresponds to the 85th percentile of the general population. So even if you’re in the first percentile here, you’re still pretty high up in the broader scheme of things.

At that point we’re back on the road without end. I am pretty sure we can raise your IQ as much as you want and you will still feel like pond scum. If we raise it twenty points, you’ll try reading Quantum Computing since Democritus and feel like pond scum. If we raise it forty, you’ll just go to Terence Tao’s blog and feel like pond scum there. Maybe if you were literally the highest-IQ person in the entire world you would feel good about yourself, but any system where only one person in the world is allowed to feel good about themselves at a time is a bad system.

People say we should stop talking about ability differences so that stupid people don’t feel bad. I say that there’s more than enough room for everybody to feel bad, smart and stupid alike, and not talking about it won’t help. What will help is fundamentally uncoupling perception of intelligence from perception of self-worth.

I work with psychiatric patients who tend to have cognitive difficulties. Starting out in the Detroit ghetto doesn’t do them any favors, and then they get conditions like bipolar disorder and schizophrenia that actively lower IQ for poorly understood neurological reasons.

The standard psychiatric evaluation includes an assessment of cognitive ability; the one I use is a quick test with three questions. The questions are – “What is 100 minus 7?”, “What do an apple and an orange have in common?”, and “Remember these three words for one minute, then repeat them back to me: house, blue, and tulip”.

There are a lot of people – and I don’t mean floridly psychotic people who don’t know their own name, I mean ordinary reasonable people just like you and me – who can’t answer these questions. And we know why they can’t answer these questions, and it is pretty darned biological.

And if our answer to “I feel dumb and worthless because my IQ isn’t high enough” is “don’t worry, you’re not worthless, I’m sure you can be a great scientist if you just try hard enough”, then we are implicitly throwing under the bus all of these people who are definitely not going to be great scientists no matter how hard they try. Talking about trying harder can obfuscate the little differences, but once we’re talking about the homeless schizophrenic guy from Detroit who can’t tell me 100 minus 7 to save his life, you can’t just magic the problem away with a wave of your hand and say “I’m sure he can be the next Ramanujan if he keeps a positive attitude!” You either need to condemn him as worthless or else stop fricking tying worth to innate intellectual ability.

This is getting pretty close to what I was talking about in my post on burdens. When I get a suicidal patient who thinks they’re a burden on society, it’s nice to be able to point out ten important things they’ve done for society recently and prove them wrong. But sometimes it’s not that easy, and the only thing you can say is “f#@k that s#!t”. Yes, society has organized itself in a way that excludes and impoverishes a bunch of people who could have been perfectly happy in the state of nature picking berries and hunting aurochs. It’s not your fault, and if they’re going to give you compensation you take it. And we had better make this perfectly clear now, so that when everything becomes automated and run by robots and we’re all behind the curve, everybody agrees that us continuing to exist is still okay.

Likewise with intellectual ability. When someone feels sad because they can’t be a great scientist, it is nice to be able to point out all of their intellectual strengths and tell them “Yes you can, if only you put your mind to it!” But this is often not true. At that point you have to say “f@#k it” and tell them to stop tying their self-worth to being a great scientist. And we had better establish that now, before transhumanists succeed in creating superintelligence and we all have to come to terms with our intellectual inferiority.

VI.

But I think the situation can also be somewhat rosier than that.

Ozy once told me that the law of comparative advantage was one of the most inspirational things they had ever read. This was sufficiently strange that I demanded an explanation.

Ozy said that it proves everyone can contribute. Even if you are worse than everyone else at everything, you can still participate in global trade and other people will pay you money. It may not be very much money, but it will be some, and it will be a measure of how your actions are making other people better off and they are grateful for your existence.

(in real life this doesn’t work for a couple of reasons, but who cares about real life when we have a theory?)

After some thought, I was also inspired by this.

I’m never going to be a great mathematician or Elon Musk. But if I pursue my comparative advantage, which right now is medicine, I can still make money. And if I feel like it, I can donate it to mathematics research. Or anti-aging research. Or the same people Elon Musk donates his money to. They will use it to hire smart people with important talents that I lack, and I will be at least partially responsible for those people’s successes.

If I had an IQ of 70, I think I would still want to pursue my comparative advantage – even if that was ditch-digging, or whatever, and donate that money to important causes. It might not be very much money, but it would be some.

Our modern word “talent” comes from the Greek word talenton, a certain amount of precious metal sometimes used as a denomination of money. The etymology passes through a parable of Jesus’. A master calls three servants to him and gives the first five talents, the second two talents, and the third one talent. The first two servants invest the money and double it. The third literally buries it in a hole. The master comes back later and praises the first two servants, but sends the third servant to Hell (metaphor? what metaphor?).

Various people have come up with various interpretations, but the most popular says that God gives all of us different amounts of resources, and He will judge us based on how well we use these resources rather than on how many He gave us. It would be stupid to give your first servant five loads of silver, then your second servant two loads of silver, then immediately start chewing out the second servant for having less silver than the first one. And if both servants invested their silver wisely, it would be silly to chew out the second one for ending up with less profit when he started with less seed capital. The moral seems to be that if you take what God gives you and use it wisely, you’re fine.

The modern word “talent” comes from this parable. It implies “a thing God has given you which you can invest and give back”.

So if I were a ditch-digger, I think I would dig ditches, donate a portion of the small amount I made, and trust that I had done what I could with the talents I was given.

VII.

The Jews also talk about how God judges you for your gifts. Rabbi Zusya once said that when he died, he wasn’t worried that God would ask him “Why weren’t you Moses?” or “Why weren’t you Solomon?” But he did worry that God might ask “Why weren’t you Rabbi Zusya?”

And this is part of why it’s important for me to believe in innate ability, and especially differences in innate ability. If everything comes down to hard work and positive attitude, then God has every right to ask me “Why weren’t you Srinivasa Ramanujan?” or “Why weren’t you Elon Musk?”

If everyone is legitimately a different person with a different brain and different talents and abilities, then all God gets to ask me is whether or not I was Scott Alexander.

This seems like a gratifyingly low bar.

[more to come on this subject later]

The argument was that there are lots of studies showing that parenting has important effects – for example, if parents yell at their kids, their kids will turn out angry and violent, or something. But these studies neglect possible genetic contributions – angry violent parents are more likely to yell at their kids, so maybe the kids are just inheriting genes for anger and violence. A lot of parenting studies are subject to these kinds of confounds. And one of the best tools we have for disentangling them – behavioral genetics twin studies – very consistently show that most important outcomes are 50% genetically determined, 50% determined by “non-shared environment”, and almost completely unrelated to the “shared environment” of parenting. Therefore, we should conclude that pretty much all of the effect supposedly due to parenting is in fact due to genetics, and it doesn’t matter much what kind of “parenting style” you use unless it can somehow change your child’s DNA.

One of the stories I most remember from the book – and I’m sorry, I don’t have a copy with me, so I’m going from memory – was about the large literature of studies showing that children of divorce raised by single mothers have worse outcomes than children of intact two-parent families. This seems like a convincing argument that children need both parents to develop properly, which if true would be a shared environmental effect and an example of why good stable parenting is necessary.

But other studies found that children who lost a father in (for example) a car accident had outcomes that looked more like those of children from stable two-parent families than like those of children of divorce. So maybe the divorce effect doesn’t reflect the stabilizing influence of two parents in a kid’s life. Maybe it reflects that the sort of genes that make parents unable to hold a marriage together have some bad effects on their kids as well.

(damn you, Rs7632287! This is all your fault!)

It’s compelling, it’s believable, and I believed it. Unfortunately, I recently had the time to double-check, and it doesn’t seem to be true at all.

The best introduction to divorce research I could find was Amato & Keith’s meta-analysis Parental Divorce And The Well-Being Of Children. It looks through 92 studies that compare children of divorced and non-divorced families and finds that “children of divorce scored lower than children in intact families across a variety of outcomes, with the median effect size being 0.14 of a standard deviation,” this last clause of which is almost New Cuyaman in its agglomerativeness.

This is a small effect size, and indeed most of the studies they’re looking at aren’t even significant. But once agglomerated together they become very significant, and the analysis tries to determine the cause. The most popular proposed causes are “children in divorced families lose the benefits of having two parents”, “children in divorced families are in economic trouble”, and “children in divorced families have to deal with stressful family conflict.”

Although there’s a little bit of evidence for all three, in general the evidence lines up for the last one of these – the family conflict hypothesis.

If the problem is not enough parents or not enough money, then having the custodial parent (usually a single mom) remarry ought to help a lot, especially if she marries somebody wealthy. But usually this doesn’t help very much at all.

If the problem is not enough money, then children of divorce should do no worse than children of poor two-parent families. But in fact they do, and children of divorce still do worse when controlled for income.

If the problem is not enough parents or not enough money, then these ought to persist over time if the custodial parent doesn’t remarry or get richer. But if the problem is stressful conflict, then it ought to get better over time, since the stress and conflict of the divorce gradually becomes more and more remote. Although there are some dueling studies here, the best studies seem to find the latter pattern – bad outcomes of divorce gradually decrease over time.

If the problem is stressful conflict, then children of divorce ought to do no worse than children in families full of stressful conflict who are nevertheless staying together. Indeed, controlling for the amount of stressful conflict within a family gets rid of most of the negative effect of divorce.

Therefore, although there was some evidence for all three hypotheses, the stressful conflict hypothesis was best-supported. But the stressful conflict hypothesis could also explain the pattern where kids whose fathers died in car accidents don’t show the same pattern of problems as children of divorce. Having a parent die in an accident is no doubt traumatic, but it’s a very different kind of trauma from constant familial yelling and bickering.

More to the point, the genetic explanation of divorce has been investigated specifically in at least four studies that I know of, using different methodology each time.

Brodzinsky, Hitt, and Smith studied the effect of divorce on biological versus adopted children. They were unable to find any differences in the level of disruption and poor outcomes.

O’Connor, Caspi, DeFries, and Plomin (yes, that Plomin) also studied biological versus adopted children. They found that biological children showed a stronger effect on academic achievement and social adjustment (consistent with genetic explanations), but adopted children showed an equal effect on behavioral problems and substance use (consistent with environmental explanations).

Burt, Barnes, McGue, and Iacono use a different methodology and compare children whose parents divorced when they were alive with children whose parents divorced before they were born. Presumably, only the former group get any environmental stress from the divorce, but both groups suffer from any genetic issues that caused their parents to split. They find that the negative effects of divorce are mostly limited to the group whose parents got divorced when they were alive, consistent with an environmental explanation.

Finally, a bunch of people including Eric Turkheimer get the requisite twin study in and compare the children of pairs of identical twins where one of them got divorced and the other didn’t (where do they find these people?) Somehow they scraped together a sample size of 2,554 people, and they found that even among children of identical twins, the children of the divorced twin did worse than the children of the non-divorced twin to a degree consistent with the negative effects not being genetic. They tried to adjust for characteristics of the twins’ spouses, but that’s the obvious confound here. I look forward to seeing if future researchers can get a sample of pairs of identical male twins who married pairs of identical female twins, one couple among whom got divorced.

So I owe mainstream psychology an apology here. I was pretty sure they had just completely dropped the ball on this one and were foolishly assuming everything had to be social and nothing could be genetic. In fact, they were only doing that up until about ten or twenty years ago, after which point they figured it out and performed a lot of studies, all of which supported their idea of the stress of divorce having significant (though small!) non-gene-related effects.

And although I haven’t had time to look through them properly yet, here’s a study claiming that the association between fathers’ and childrens’ emotional and behavioral problems is “largely shared environmental in origin”. And here’s a study claiming that “analyses revealed that [shared environment] accounted for 10%-19% of the variance within conduct disorder, oppositional defiant disorder, anxiety, depression, and broad internalizing and externalizing disorders, regardless of their operationalization. When age, informant, and sex effects were considered, [shared environment] generally ranged from 10%-30% of the variance.”

So the shared environment folks haven’t completely dropped the ball, some of them seem to be fighting back, and it will be interesting to see where this goes and whether anybody is able to reconcile the different evidence.

One likely talking point: shared environment and childhood situation obviously impacts things during childhood. For example, if you have parents who are mean and abusive, this can make you stressed and you don’t get enough sleep and then maybe you do really badly at school. But once you get out of that environment, your academic abilities will revert to whatever your genes say they should be. The Nurture Assumption never denies this and is absolutely willing to admit that shared environment can affect outcomes during childhood, although even there less than one might expect. This also seems to be the tack Plomin is taking when he discusses the Burt study.

But studies have found that the negative effects of divorce can last well into adulthood. On the other hand, none of those studies have been the ones that compare genetic and environmental effects, and I get the feeling their quality is kind of weak. So it’s not completely ruled out by the data that the short-term effects of divorce are robust and environmental, but the long-term effects of divorce are spurious and/or genetic. But this seems kind of like fighting a rearguard action against the evidence.

Finally, a sanity check. Suppose your parents get divorced when you’re 16. Your high school grades drop and your behavior gets worse. Maybe you fail a couple of classes and start using drugs. The couple of classes failed mean you’re going to a second-tier instead of a first-tier college, and the drug use means you’re addicted. How does that not affect your life outcomes, even if five years later you’ve forgotten all about whatever psychological stresses you once had?

Overall I am less confident than before that shared environment is harmless.

And while I’m bashing Nurture Assumption, I don’t remember the exact arguments used against birth order effects, but we found such impressive numbers on the last Less Wrong survey that I’m not very impressed with the claims that they don’t exist.