Should Buzzfeed Publish Claims Which Are Explosive If True But Not Yet Proven?

Buzzfeed, January 14: A Mindset Revolution Sweeping Britain’s Classrooms May Be Built On Shaky Science.

Somebody needed to write this article. It’s written very well. I’ve talked to the writer, Tom Chivers, and he was very careful and seems like a great person. The article even quotes me, although I think if I had gotten to choose a quote of mine for thousands of people to see, it wouldn’t have been the one speculating about Carol Dweck making a pact with the Devil.

But I’m not entirely on board with it.

Growth mindset has been really hyped and Carol Dweck has said it can do implausibly exciting things, okay. A lot of smart people are very suspicious of growth mindset and think there has to be some trick, sure. There’s a high prior that something is up, definitely.

But one thing that needs to be at the core of any article like this is that, if there’s a trick, we haven’t found it.

I tried to be really clear about this in my own (mostly pessimistic) article on the subject:

It is right smack in the middle of a bunch of fields that have all started seeming a little dubious recently. Most of the growth mindset experiments have used priming to get people in an effort-focused or an ability-focused state of mind, but recent priming experiments have famously failed to replicate and cast doubt on the entire field. And growth mindset has an obvious relationship to stereotype threat, which has also started seeming very shaky recently. So I have every reason to be both suspicious of and negatively disposed toward growth mindset.

Which makes it appalling that the studies are so damn good.

This is the context of my speculation that Carol Dweck has made a pact with the Devil. I haven’t accused (for example) the stereotype threat people of making a pact with the Devil. They did some crappy studies and exaggerated the results. That doesn’t require any diabolic help. Any social scientist can do that, and most of them do. What’s interesting about the growth mindset research is that it looks just like the sort of thing that should fall apart with a tiny gust of wind, but it actually hangs together pretty well.

BuzzFeed doesn’t really challenge that. The article spends most of its time snarking about how overhyped growth mindset is – and no objections there, given that its advocates claim that it can eg help defuse the Israel-Palestine conflict and bring peace to the Middle East. It spends a bit more time talking about how many people are doubtful – no objections there either, I’m doubtful too.

But in terms of the evidence against it, it’s kind of thin. I only see three real points:

First, it uses a technique called GRIM (granularity-related inconsistency of means). I like its explanation so I’m just going to quote it verbatim:

t works like this: Imagine you have three children, and want to find how many siblings they have, on average. Finding an average, or mean, will always involve adding up the total number of siblings and dividing by the number of children – three. So the answer will always either be a whole number, or will end in .33 (a third) or .67 (two thirds). If there was a study that looked at three children and found they had, on average, 1.25 siblings, it would be wrong – because you can’t get that answer from the mean of three whole numbers.

But Dweck says that she “took ambiguous answers as half scores” – maybe if the child was halfway between growth mindset and fixed mindset it was counted as a 0.5. It’s bad practice to do this kind of thing without mentioning it. But everyone does some bad practices sometime. And I don’t see anybody claiming it affected the results, which were very strong and not likely to stand or fall based on these sorts of things. Nobody is claiming fraud, and Dweck released her original data which looks pretty much like she was generally honest but had some bad reporting practice. Neither the statistician involved nor BuzzFeed claims this affects Dweck’s work very much.

Second, it mentions Stuart Ritchie’s criticism of a couple of recent Dweck papers which show “marginally significant” results. These results are so weak that they’re probably coincidence, but the paper hypes them up. There are a couple of studies like this, but they’re all in very tangential areas of mindsetology, like how children inherit their parents’ mindsets. The original studies, again, show very strong results that don’t need this kind of pleading. For example, the one I cited in my original post got seven different results at the p < 0.001 level. And there are a lot of studies like this.

Third, it mentions a psychologist Timothy Bates who has tried to replicate Dweck’s experiments (at least) twice, and failed. This is the strongest evidence the article presents. But I don’t think any of Bates’ failed replications have been published – or at least I couldn’t find them. Yet hundreds of studies that successfully demonstrate growth mindset have been published. Just as a million studies of a fake phenomenon will produce a few positive results, so a million replications of a real phenomenon will produce a few negative results. We have to look at the entire field and see the balance of negative and positive results. The last time I tried to do this, the only thing I could find was this meta-analysis of 113 studies which found a positive effect for growth mindset and relatively little publication bias in the field.

My intuition tells me not to believe this meta-analysis. But I think it’s really important to emphasize that I’m going off intuition. There’s no shame in defying the data when you think that’s justified, but you had better be really aware that’s what you’re doing.

I guess my concern is this: the Buzzfeed article sounds really convincing. But I could write an equally convincing article, with exactly the same structure, refuting eg global warming science. I would start by talking about how global warming is really hyped in the media (true!), that people are making various ridiculous claims about it (true!), interview a few scientists who doubt it (98% of climatologists believing it means 2% don’t), and cite two or three studies that fail to find it (98% of studies supporting it means 2% don’t). Then I would point out slight statistical irregularities in some of the key global warming papers, because every paper has slight statistical irregularities. Then I would talk about the replication crisis a lot.

I could do this with pretty much any theory I wanted. Any technique strong enough to disprove anything disproves nothing.

(and this is especially important in light of recent really strange negative results that eg fail to find a sunk cost effect, something I would hate to enshrine as “well, guess this has been debunked, no such thing as sunk cost now”)

Again, this isn’t to say I believe in growth mindset. I recently talked to a totally different professor who said he’d tried and failed to replicate some of the original growth mindset work (again, not yet published). But we should do this the right way and not let our intuitions leap ahead of the facts.

I worry that one day there’s going to be some weird effect that actually is a bizarre miracle. Studies will confirm it again and again. And if we’re not careful, we’ll just say “Yeah, but replication crisis, also I heard a rumor that somebody failed to confirm it,” and then forget about it. And then we’ll miss our chance to bring peace to the Middle East just by doing a simple experimental manipulation on the Prime Minister of Israel.

I think it’s good that people are starting to question growth mindset. But at this point questioning it isn’t enough. In my essay I tried to find problems that might have caused spurious effects in Dweck’s studies, and patterns inconsistent with growth mindset being powerful. I think we need to do more of that, plus look for specific statistical and experimental flaws in the papers supporting growth mindset, plus start collecting real published papers that fail to replicate growth mindset. Instead of talking about how sketchy it is, we need to actually disprove it.

We owe it to ourselves, to Carol Dweck, and to her infernal masters.

This entry was posted in Uncategorized and tagged . Bookmark the permalink.

64 Responses to Should Buzzfeed Publish Claims Which Are Explosive If True But Not Yet Proven?

  1. TheWackademic says:

    Dweck’s theory of mind never really made sense to me. My own theory of self seems to operate one meta-level above the fixed vs. growth mindset, and I wonder if anyone else feels the same way?

    In my self perception, I have the ability to work really hard, learn new things, and acquire new skills: e.g., I sort of have a growth mindset. However, one level higher, I think that hard work and mental flexibility is a skill-set that I was born with and that hasn’t really changed: e.g., fixed mindset. I can’t increase (or decrease) the amount of effort I can put into a task or the number of new things I can learn. I was just fortunate that that number happens to be high for me.

    Maybe there are people who have two meta-levels of growth mindset: they can learn new things, and they can learn skills to increase the number of new things that they can learn.

    However, the one-level “growth” vs. “fixed” mindset seems woefully inadequate. Also, I think it contradicts the abundance of data suggesting that intelligence and achievement are strongly, strongly genetic. Any thoughts on this?

    • paranoidfunk says:

      I can’t increase (or decrease) the amount of effort I can put into a task or the number of new things I can learn.

      You can’t? Even for tasks you are especially motivated to advance in or complete, and with caffeine / nootropics involved? I suspect that may be a generalization, and I say that to say surely the level of motivation is a factor – I know for me it is.

      It seems inevitable that the mind projection fallacy will hamper discussions about meta-cognitive learning skills, but I can’t help but think that self-fulfilling prophecies and socialized mindsets extremely impact learning. Despite being rather great at ye ol’ multiplication tables and all that, I was put in the lowest math class upon entering middle school — I can’t remember why — and ever since, I’ve been terrible at math; or at least that’s what I hear myself saying whenever I find myself struggling. I’ve never taken a stats or calculus class because of this. But maybe that’s My Problem, and n=1.

      To get out of the anecdote: I really do think a) motivation / a want to learn and b) perception that you are capable are huge factors in learning, regardless of intelligence. Self-fulfilling prophecies are the mind-killer. :p

    • Deiseach says:

      There’s natural talent or innate capacities. Growth mindset means that Susan can improve her grades/skills/performance in an area, but she’ll never be quite as good as Jane who has a talent for this thing.

      On the other hand, if Jane skates by on “I don’t need to study/work/practice, I am already good at this”, eventually that will come back to haunt her. Hard-working Susan may be more successful than talented Jane because Jane screws up as many times as her talent enables her to pull a miracle out of thin air, while Susan has a steady level of achievement.

      Telling kids “you can’t do this because you don’t have the talent for it, so it’s useless to even try” will put them off. They may never be a star at it, but they might enjoy it or get some benefit out of it, and the discipline of working hard, studying, finding ways to get around problems and so forth will stand to them.

      Telling kids “you can all be pop stars and sports stars if you just work hard enough!” is also bad in the long run, because not everybody can, and even with talent and hard work, there’s also an element of luck – of being in the right place at the right time.

      So, a bit from column A and a bit from column B.

      • Tarpitz says:

        One important question’s how much from column A and how much from column B, though, right?

        I’m guessing Jerry Rice had/has a growth mindset and Randy Moss had/has a fixed mindset. And that may very well be the reason that Jerry Rice was a better wide receiver than Randy Moss. On the other hand, Randy Moss was a better wide receiver than literally every human being who ever lived not named Jerry Rice or Randy Moss. When it comes to pro football, it seems pretty clear that talent is vastly more important than mindset. That may or may not translate to other fields, but it’s a possibility to be aware of.

        Also a crucial consideration: growth mindset being valuable is not enough. It also has to be learnable. Dweck knows this, and her experiments do indeed appear to show that growth mindset is not only important but also possible to actually instill in people who did not previously have it. But the two questions are separate and both really matter.

    • wintermute92 says:

      I’ve had a lasting suspicion that Dweck’s results are clustered – that they might work on some self-images powerfully, and others barely at all. This would explain what we see, of good, convincing studies and fervent advocates being opposed by people going “that sounds insane!”

      This sort of thing would fit nicely.

      On the other hand… It’s bad practice to diagnose something as healthy for everyone else. (“I’m fine, but other people need religion to hold communities together!”) And two is a number that needs justification. It would be odd if some people were affected by growth mindset, but others had more (or less?) meta-awareness and weren’t. I would expect a spectrum, not a split. And finally, the effects are so damn strong. If there’s a decent population that was ‘immune’, I sure haven’t seen it crop up in Dweck’s data.

      An extension of the theory, perhaps: brain growth keeps going into the 20s, and growth mindset helps better in kids? I could spin a story where, like toddlers checking whether they’re hurt with a parent, children are more susceptible to working harder because they think they can. (“The growth was inside you all along!”) Adults are either less flexible, or more honest about their limits, and so more of them are ‘immune’.

      …but of course, I don’t have a damn bit of data for that.

  2. metacelsus says:

    “Growth mindset” was used by my high school administrators to try to justify getting rid of all the Honors classes, since they “promoted the belief that you’re either an Honors student or not, and that it’s impossible to change.” Fortunately, the meddling administrators couldn’t actually get rid of the classes.

    Unsurprisingly, I think growth mindset is BS. Whether or not the studies show real effects, the way the concept gets used is inexcusable.

    • Anthony says:

      I suspect that if Carol Dweck had published studies showing almost the exact opposite of the “growth mindset”, and it had gotten hyped the same way, your school’s administrators would have come up with some sort of excuse to use that thinly-based research to try to get rid of honors classes.

      • albertborrow says:

        Well, we all know high school is actually about college readiness and not learning – just like middle school was for high school readiness and how college is simply preparing you for your career. It seems silly to make classes that are actually difficult. How else would our students learn to cope with the struggles of modern life? Certainly not by burning them out with reality before they even got started. And the teachers that demand more from their students – that make tests harder than they need to be and the class more difficult than the common core? They are discouraging learning by alienating the worst preforming kids in the class and not giving them a chance to shine.

        It seems clear to me that no matter what class the children are taking, the most important thing is that it should be absolutely no challenge at all. The longer they are in school, the better – after all, I only get payed when there are students to teach, and the union only gets their dues if there are teachers.

    • wintermute92 says:

      It succeeded at my school. Well, with a secondary spin of “and the best students will motivate and inform the others!”

      As you can imagine, the best students were less motivational once they were getting stabbed with pencils every few minutes.

      • nydwracu says:

        My high school never tried to get rid of honors classes, but some of the people in my honors English class could barely read.

  3. “So, how to we instill emotional propaganda in the human child’s mind in the hopes of getting them to study more instead of accurately teaching them about ability thresholds ,genetic probabilities, and hours of practice?…or idk, forcing them to study and practice more?”

    Kids will obviously reject a full-blown work-ethic propaganda model, since its obvious that both jimmy and johnny paid attention in cursive but johnny had the best handwriting no matter what jimmy did.

    There will probably be a rejection of a full blown innate talent model too, since people get better at some things they practice a lot. Its a revolutionary thought, really.

    Ever since a young age, it seems everyone understood most talents were some blend of the two.

    I don’t believe this is a new thing sweeping the classroom, just a reworded hodgepodge of other ideas. Blend the self-esteem movement(was that actually a thing?) with the aphorisms “Try, try again” “If you quit, you will never improve and never know” and it sounds like you get this.

  4. Jack Lecter says:

    So… is there some special significance to the title, or is it just one of those random ‘not-a-coincidence-because-nothing-ever-is’ thingies?

    I mean, I dig the whole Kabbalistic Parallels art-form just for its own sake, but a lot of the time, when you do this, there’s a broader point you’re trying to make, and even I’ve heard about the whole political kerfuffle going on about Buzzfeed right now.

    • Elijah says:

      I worry that one day there’s going to be some weird effect that actually is a bizarre miracle. Studies will confirm it again and again. And if we’re not careful, we’ll just say “Yeah, but replication crisis, also I heard a rumor that somebody failed to confirm it,” and then forget about it. And then we’ll miss our chance to bring peace to the Middle East just by doing a simple experimental manipulation on the Prime Minister of Israel.

      Replace “Prime Minister of Israel” with “President of Russia”, and I think the parallel to Buzzfeed’s “Trump is a Russian plant” scandal holds up.

      And nothing is ever a coincidence.

    • Deiseach says:

      If Scott is trying to avoid politics, I think he’s right. I’m about sick to the back teeth of the whole thing right now, and I don’t want to get dragged into “Is Pissgate accurate or just propaganda?”

      I’m much more willing to discuss “is clickbait a monstrous carbuncle on the face of journalism?” by staying away from examples involving R___a, T____p, C__A, H____y, F__I, etc.

    • Subb4k says:

      I think it’s a case of Scott baiting people into commenting mostly about the title, so then he can claim that when he makes an off-hand political joke in a long post that is absolutely not about politics, people only talk about politics. And of course he’ll deny any baiting.

    • Obviously the title was designed to look as though it was about the recent Trump/Buzzfeed controversy. I attribute that to Scott’s sense of humor, not to any profound philosophical point.

      • Rob K says:

        Yeah, I took it as a little “got you to read this post about standards of proof and disproof in social science with a headline that appeared to be about today’s outrage” joke.

      • Careless says:

        +1. Serendipitous timing allows for the joke title.

    • suntzuanime says:

      He’s trolling. I approve heartily.

  5. AlphaCeph says:

    > SHOULD BUZZFEED PUBLISH CLAIMS WHICH ARE EXPLOSIVE IF TRUE BUT NOT YET PROVEN?

    Dude, such clickbait!

    You know what everyone was expecting to hear about

  6. Ninmesara says:

    What I remember from your earlier blog posts is that Growth Mindset works in the lab but not in the real world. Is anyone trying hard enough to look for Growth Mindset effects outside of the lab? Even if they’re using ecological studies?

  7. Deiseach says:

    This is the kind of quality content that keeps me returning to this site.

    Is there actually a discipline called “mindsetology” or is that just a term you invented? See, right there that is what is wrong with modern science: the good old-fashioned “cobble together Greek and Latin roots to create an impressive, and more importantly, official Sounds Like Real Science name for your newly invented field” is too much work for the whipper-snappers of these days! How do they expect to get good results if it sounds like any wandering idiot without a PhD could understand what they’re doing? 🙂

    Imagine you have three children, and want to find how many siblings they have, on average.

    I was going to say if you have three kids and don’t know how many siblings they have, you either need to brush up on your maths or stop getting blind drunk on Friday nights and going home with nubile young things, but then I remembered my time in social housing provision and yeah, this could be a real question: I had three kids with Joe, but now we’re split up and both with new partners. As well, I had two kids by two different fathers (who also had kids by different girlfriends before we hooked up) before I met Joe, and Joe also had a few babymommas before he met me. Now I’m pregnant again and his new girlfriend is also pregnant. Adding up full, half- and step-siblings, how many siblings do each of my kids have?

    (There’s also, to my knowledge, at least one instance of “If my father is also the half-sibling of my half-sibling, does that mean my father is also my step-brother?”)

    How can you be in-between growth and fixed mindset? You’re fixed in some areas but open to growth on others? All the beating, encouragement, praise, threats and gold stars in the world won’t make you able to carry a tune because you’re tone-deaf so unfortunately no room for growth mindset in “If I work hard enough, I can become an opera star at La Scala”?

    What I’m taking away from this is that growth mindset probably works but not the way they think it works, is that correct?

    As for BuzzFeed, I generally think sensationalist articles should not be printed, no matter what or who they are about or in what organ. But modern media is in such a state that they’re competing for clicks (the dead-tree versions of newspapers are in a slough) and to get hits and page views and all the rest of the things their ad departments are selling to prospective customers (and the revenue from ads is what keeps the whole operation afloat), they need scandal! shock, horror! you won’t believe this one weird trick! clickbait.

    • Careless says:

      Are you joking with the confusion on the “three kids” thing? 3 unrelated kids.

      • Deiseach says:

        Are you joking with the confusion on the “three kids” thing? 3 unrelated kids.

        So then the question is not “I have three children, how many siblings does each child have?”, it is “There are three children; how many siblings do they have?”

        • Spookykou says:

          I think that was the question, it was explaining the GRIM system, about possible averages, so however many siblings they have must be divided by 3 so only answers ending in .34 or .67 or whole numbers are possible.

          If you say the mean number of siblings of these three kids is .5 then you did something funny in your math.

  8. Peter Gerdes says:

    The problem with all this growth mindset research is that the research only evaluates local and near term effects of imposing growth mindset while the effects are extrapolated to global long term effects.

    It seems to me the following model of ‘growth mindset’ is plausible:

    Interventions that teach growth mindset in effect amount to strong persuasion that investing additional effort in the area (e.g. math class or whatever) where the intervention occurred will pay off despite evidence to the contrary.

    If so this would result in very strong seeming empirical evidence about the effectiveness of growth mindset. Many subjects who would otherwise invest their time and energy in other activities instead invest them in the area in which the intervention occurred improving their results. However, at the same time their other goals probably suffer in ways which aren’t noticed by the experimental setup. Moreover, it would suggest that trying to use a growth mindset in every aspect of life would accomplish nothing besides encouraging poor investments of time.

    • MawBTS says:

      This would concord with IQ-boosting interventions. The usual picture is short term gains that fade quickly.

  9. dianelritter says:

    I went over to mindsetworks.com/science (the first thing that came up on google search) and it said:
    “When students believe they can get smarter, they understand that effort makes them stronger. Therefore they put in extra time and effort, and that leads to higher achievement.

    Telling people that they will do better if they work harder seems like a sure shot recipe for success.

    • Deiseach says:

      When students believe they can get smarter

      If that’s what they’re telling them, then it’s fakery. You won’t get “smarter” (i.e. increase your native IQ or talents for sports, art, etc.) but you will get “better” (i.e. develop what potential you have, learn and retain more facts, learn skills, learn how to use your time more effectively, etc.)

      It’s about doing more with what you already have, rather than getting more (be that brains, capacity to put in effort or whatever).

      Maybe this is where the “growth mindset” confusion as it is applied happens – it’s being sold as “make yourself smarter by hard work and application” and that’s not what it is.

      • moridinamael says:

        I know some really stupid people with high IQs. I don’t think it’s fakery at all to tell kids that they can get smarter by studying. It shouldn’t be assumed that “smarts” refers to the immutable G score, especially when kids don’t know that IQ or G are things.

        I would have no trouble at all telling my kids that working hard at learning will make them smarter. It’s basically true. Taking an advanced math course enables you to do things you couldn’t do before you took the course, and to learn further things that would have been inaccessible; the layman might understandably infer that the student had become smarter.

        I haven’t looked at the growth mindset papers, but I wonder how much of the effect size might be simply due to remediating really remarkably bad default mindsets in some subset of kids. I remember from being a kid that some kids would respond to every challenge or difficult with a variation of “I can’t do it” or “I’m just not good at this”. Maybe most of what the intervention does is just preemptively stop those kids from forming that default reaction.

      • nydwracu says:

        I know a fairly respected scientist in a fairly prestigious field who claims to have an IQ somewhere around 115 and just makes up for it with an insane work ethic.

  10. spork says:

    I’m curious to see what people see as the potential harm in us being wrong about this “mindset” effect.

    This is totally orthogonal to the question of whether it’s real, and I’m not dismissing that question. I just suspect that the biggest driver behind the impact of Dweck’s work is that her results give educators a great excuse for doing what they would have done anyway, which is trying to talk kids into doing homework and studying. They just re-describe what they did all along in Dweck’s jargon, and everybody feels better.

    I’m trying to picture how much about pedagogy would change if Dweck’s effect received a successful debunking. I suppose we could get all Calvinist about it, telling a kid that the amount of studying she’s capable of is handed out by a genetic lottery, so if she finds she’s able to study hard, it indicates that she’s been blessed in this regard, and should expect a successful future. One thing not up for debate is that exerting effort (studying, practice) helps performance and outcomes. The debate is about whether “mindset” moves the needle on effort, and if it does, whether nearly costless environmental manipulations like Dweck’s move the needle on mindset. But that returns to the question of whether the effect is real. I’m asking instead whether Dweck’s work actually changed anything except the jargon in today’s pep talks for teachers and students.

    • Deiseach says:

      There has to be something about the “mindset” idea that is different from generations of teachers telling students “you have to pay attention in class, you have to do your homework, you have to study or else”.

      Maybe that’s what the message has been reduced down to – everyone wants a quick fix and if government implementation of “growth mindset” reduces it to “we need something that is not going to cost extra”, then probably it is “do your homework, study for the test” dressed up in new language.

      So that leaves the question: (1) is there something there when we talk about “growth mindset”? That seems to be what the replication problem is about (2) if there is, what is it that is different from “stick to it”, “studiousness”, “put in the effort” and previous advice of that nature?

    • Douglas Knight says:

      If we care about effectiveness of teaching methods and some methods work better than others, promoting bad methods is bad. Promoting as great methods that are merely as good as the status quo is also bad. It is bad because it distorts incentives among researchers.

      We have had decades of people claiming to have found a silver bullet, so even though every one has been false, every new researcher claims to have found a silver bullet. There is no reward for finding a small positive effect. Not only do we fail to harness small improvements in education today, but worse, we fail to take them as the starting steps for new research. If education research is achieve large effects, it is through the accumulation of small effects.

    • tomchivers says:

      Hi: author of the BuzzFeed piece here. Other people have asked this – what’s the harm in promoting mindset stuff, even if the science isn’t “true” per se, if it’s just telling people that they can improve if they work hard?

      The two earlier replies to this comment cover most of the things I’d say, but I’d also like to add something a teacher raised with me, which is that it could be pretty harsh on pupils. If I was struggling with some school subject, being told that it was because I had what amounts to a bad attitude – you just don’t believe in the importance of hard work! – might be hard to take. More importantly it might stop teachers from helping kids with actual special educational needs. Funnily enough Scott’s written pretty much that exact point before.

      But yes, also, this is billed as a major breakthrough in how teachers can help children, and if it’s just “telling them that working hard is important”, then we’re wasting a lot of teachers’ time and public money.

      FTR I thought Scott’s piece made some very reasonable points.

      • Scott H. says:

        The “could be harsh on pupils” fear is somewhat like saying “We can’t use robots in manufacturing. Someone might mess up the programming.” In each case the fear is orders of magnitude less important than the good of the proposed solution.

        Children typically don’t understand our messages that take experience, but they do understand the messages that take no experience. Therefore, children are very quick on the “special needs” explanation for skills deficiencies, and are classically short on the “if I keep at it, it will come” explanation.

        Even “hard work is important” often fails with children. In the children’s eyes the carrier of such a message is more likely to be viewed a sadist or simply a stickler for the rules than someone really providing a tool to improve their future. I mean, that was my children’s view anyway.

        (I guess I’m assuming that growth mindset teaching would develop the concept beyond simple cliches.)

        • Spookykou says:

          In each case the fear is orders of magnitude less important than the good of the proposed solution.

          This assumes a level of understanding of both the potential costs (which I am not sure how you can know) and the gains which are, obviously, on shaky grounds with the person you are replying to. You seem to be opening with the assumption that they are wrong.

          Therefore, children are very quick on the “special needs” explanation for skills deficiencies, and are classically short on the “if I keep at it, it will come” explanation.

          I have a vague memory(maybe I made it up?) that this phenomenon is cultural, something about how students in Asian countries tend to be much more likely to speak of success and failure in terms of effort.

          Which seems to call into question the idea that this is some sort of universal constant for children and only the specific interventions of growth mindset(which as I understand it are little more than “Hard work is important”) can fix it.

          • Scott H. says:

            This assumes a level of understanding of both the potential costs (which I am not sure how you can know) and the gains which are, obviously, on shaky grounds with the person you are replying to.

            While I don’t KNOW the potential costs, I do feel I understand them, much in the same way of my analogy. I don’t know the potential gains of robots versus robot software glitches, but I understand the magnitude of their effects on the decision.

            I have a vague memory(maybe I made it up?) that this phenomenon is cultural, something about how students in Asian countries tend to be much more likely to speak of success and failure in terms of effort.

            It wouldn’t be difficult for me to believe that Asian culture is more predisposed than ours to teach a growth mindset. Heck, I was commenting here somewhere that I was teaching growth mindset to my kids without having ever heard of the science or even the discipline. There’s nothing new under the sun.

            Which seems to call into question the idea that this is some sort of universal constant for children and only the specific interventions of growth mindset(which as I understand it are little more than “Hard work is important”) can fix it.

            Yes, I would imagine compounding growth mindset training or indoctrination (or whatever we’ll call it) on a population already steeped in the discipline will not be so useful. I address a growth mindset concept education versus cliches about hard work in some other posts here. There isn’t THAT much more to growth mindset really, but my experience is that the kids don’t get it without deeper explanations.

      • Scott H. says:

        One last nit to pick… I didn’t think your example of Michael Jordan taking the game winning shots was a good example of growth mindset. It would be different had Jordan gotten better over time at hitting the winning shot, but that wasn’t addressed. Jordan seems to be saying simply that in order to take the winning shot you’ve got to be willing to take the losing shot as well. That’s all fine, and perhaps a good lesson, but it’s not growth mindset.

        • tomchivers says:

          I probably agree, but Jordan is one of Dweck’s examples of growth mindset in sportspeople, not mine. I may have misrepresented her evidence for that, although I hope not.

  11. Mazirian says:

    The last time I tried to do this, the only thing I could find was this meta-analysis of 113 studies which found a positive effect for growth mindset and relatively little publication bias in the field.

    It seems that about half of those studies actually used differences in mindsets to predict achievement; the rest had other predictor or outcome variables. It also appears that the vast majority of the studies included are observational/correlational rather than experimental a la Bates.

    The fact that there’s a (small) correlation between “naturally existing” (i.e., not experimentally induced) growth mindsets and achievement seems to be due to genetic factors that they share with other character traits like conscientiousness. In correlational studies growth mindset is therefore just a relabeling of traditional personality measures, so it’s not surprising that there’s a small correlation between it and achievement that is not due to publication bias.

    Evidence for growth mindset’s relevance beyond the fact that it is a (weak) measure of genetic temperament would have to come from experimental studies where mindsets are manipulated and the subsequent outcomes (e.g. grade-point averages) of the treatment and control groups are compared. I couldn’t find information on how many experiments were included in the meta-analysis, but my hunch is that there aren’t a whole lot of them. They say that they didn’t find evidence of publication bias, but I think they lumped the more numerous correlational studies together with the few experimental ones when analyzing bias, so there’s no way to tell if the experimental studies suffer from publication bias.

  12. The Nybbler says:

    I don’t think there’s much danger we’ll miss something huge because jaded skeptics (including myself) refuse to believe it. There’s always lots of suck…err, people who aren’t jaded skeptics (one might even say there’s one born every minute), and if something big and real does come by, one of those people will use it to bring peace to the Middle East, another will use it to create fusion power, yet another will build the first AGI, and pretty soon we’ll be on our way to fusion-powered paperclip nirvana.

  13. Jiro says:

    I worry that one day there’s going to be some weird effect that actually is a bizarre miracle.

    That could potentially happen not just for the type of questioning in this essay, but for any type of questioning whatsoever. Substitute in something else for “not properly proven”. What if there’s some weird effect, which is real, but the only evidence for it is fraudulent? Should you worry that we’re missing out on real effects by not believing in ideas for which the only evidence is fraudulent? If the only evidence for something consisted of reading tea leaves, would you worry that we are missing out on real effects for which the only evidence is tea leaves?

  14. suntzuanime says:

    It found that a mindset intervention increased the time teenagers spent trying to solve a maths question. But Ritchie said that one of the central findings didn’t meet the usual scientific definition of “significance”. This is measured by a value called “p” that is usually required to be less than 0.05 for a finding to be considered significant.

    “They report the interaction as ‘p [is less than] 0.1’ – so, er, not statistically significant then,” Ritchie said. “But then they just interpret it as if it was significant. I think there’s good reason to be concerned that this isn’t super-solid science.”

    Yeah, the reason to be concerned is that it reports a p value at all, instead of a Bayes factor.

    Seriously though, I think you’re right that they needed to go a little more meta and provide context as to how much other science was equally “shaky”. Your example of climate science is a good one, where the orthodox position is that the Science is Settled and anyone who disagrees about any point is just trying to destroy the world and quite possibly even a Republican. But figuring out the truth is really hard, most scientists are imperfect, and once you get beyond particle physics, things are chaotic and messy all around. So if people think that science generally works in an ideal fashion, they’ll see climate scientists conspiring to exaggerate their results, or unreported ambiguous scores in growth mindset data, and conclude that it’s all a fraud. When in reality, either effect may be real, whatever the professional indiscretions of the people studying it.

    If you truly Fucking Love Science, you need to Fucking Understand Science As A Whole Person, rather than building up in your mind a Fucking Idealized Version Of Science That Can Do No Wrong.

  15. Moon says:

    Certainly if you believe that studying/practicing is likely to make a positive difference in your performance in something you want to perform well in, then you are likely to study or practice more, and there is a high probability that your performance will improve.

    And there are a certain percentage of people who don’t operate that way– who think that people are born winners or losers.

    And, of course, there are people between the black/white extremes– who believe in the influence of both genetics and practice/study.

    It’s reasonable to give a try to the belief that studying/practicing is likely to make a positive difference in your performance in something you want to perform well in– if you don’t have that belief already.

    The only harm in it, as with most generally constructive ideas, is if you make it too rigid, or too black/white. E.g. you believe that practice totally determines performance, without any effect of natural ability. Then you are likely going to be frustrated, and maybe depressed.

    And if you pressure kids hard to study/practice, to the extent that you seldom or never let them play, that can drain the joy from their lives. An Asian American woman in my neighborhood had a “Tiger Mom.”

    https://en.wikipedia.org/wiki/Battle_Hymn_of_the_Tiger_Mother

    She has a great career and is a brilliant physician. But her voice often trembles for no apparent reason when talking about everyday things, and she is a highly anxious person. Perhaps this is a better life than if she’d not studied at all in school. But the ideal way would be somewhere in the middle, I think, where she wouldn’t end up feeling the constant sense of pressure in her life to perform, and the fear that she won’t be able to measure up to her mother’s perfectionist standards. Which, of course, she won’t. But she was pressured from such a young age, that it is not easy to let go of her Tiger Mother’s standards.

    I do agree with someone up above in the thread here, who said that the growth mindset may be attractive to schools because it doesn’t cost anything.

    I do think it’s a bit different from just telling kids to study hard. It has to do with inspiring the kid’s belief in themselves to be able to put in effort and to get results from that effort. Which is good, as long as you don’t go overboard about it.

    There are certainly numerous factors that influence kids’ learning. I kept advising my sister for years to get a psychological evaluation done on her son, and finally, when he was 17, she went ahead and did it. It turned out that he was very bright but had a learning disability. He had to tape record lectures and books, and listen to them. And then his grades shot way up higher than they had been.

    So another appeal of the growth mindset is that it is a simple solution and people love simple solutions– although many problems require complex solutions instead. The biggest complex and expensive but good solution may be to expand the social safety and social service net, so that, if something traumatic is going on at home, or just chronic child neglect, that assistance can be given to the child to help solve that problems so that the child can survive and even thrive.

    And another issue is that you want schools to be run in a way that is conducive to students being physically and emotionally safe, benefiting from their natural curiosity, and getting intellectual stimulation rather than being bored to tears.

    I read Dweck’s book. And that’s great that growth mindset works so well for some teachers and their students. But some of these teachers were very emotionally mature, highly skilled communicators and almost psychotherapists– in the sense of being highly aware of what kids were going through emotionally and being constructively responsive to that. The average teacher is certainly not that way. Perhaps there needs to be a more extended program for teachers to train them not just in growth mindset, but also in responsiveness to students’ emotional needs.

    Another factor is that the politics of the school system has a huge effect on teachers, students, and learning. In some schools with Common Core, for example, the pressure may be very high on teachers, to get the students to score high on standardized tests, even though many of them may be disadvantaged inner city kids, some of whom are traumatized at home, some for whom English is not their first language etc. And the system may assign the most difficult kids to the least experienced teachers, who don’t have the seniority to demand a class of “easy” students.

    In those least experienced teachers’ classes at that school, “growth mindset” or the lack of it, is the least of the problems for the student– whose teacher either has to get that student to perform on tests up to an unrealistic standard– or else the teacher gets let go at the end of the year, because her students’ test scores are not up to the Common Core standard.

    • Scott H. says:

      “Fixed mindset” is the predominant mindset for children. That’s why teaching “growth mindset” is so important and so fruitful. Anyone who has kids knows that they constantly dwell on one question with regards to ANY task — Am I good or am I bad?

  16. Moon says:

    Suntzu

    “If you truly Fucking Love Science, you need to Fucking Understand Science As A Whole Person, rather than building up in your mind a Fucking Idealized Version Of Science That Can Do No Wrong.”

    Good point. But there is a problem with homo sapiens. He/she typically does not love science. He/she typically can’t handle the truth. Sure, there are some scientists and some scientific types who search for the truth. But we are the exception, not the rule. And even many who think they are rational, are in reality simply highly skilled pickers– nit pickers, cherry pickers, and lemon pickers.

    That’s one reason why conspiracy theories and lies in the news are so popular. People love a good story, especially a lie that confirms their pre-existing biases and fits seamlessly into their world view. The truth– not so much.

    • webnaut says:

      People love a good story so much it makes one suspect this is an attribute of ourselves that aids survival.

      There must be some deep reason why we love stories so much. The drawbacks are clear but it is less obvious what the advantages are.

    • caethan says:

      It is true and depressing that lies are so much more popular than hard truths. This is one of my favorite poems:

      Magna est Veritas

      Here in this little bay,
      Full of tumultuous life and great repose,
      Where twice a day,
      The purposeless glad ocean comes and goes,
      Under high cliffs, and far from the huge town,
      I sit me down.
      For want of me the world’s course will not fail;
      When all its work is done, the lie shall rot;
      The truth is great and shall prevail,
      When none cares whether it prevail or not.

  17. HeelBearCub says:

    @Scott Alexander:
    Consider the possibility that you have been writing the buzzfeed article you describe over and over.

    Let’s look at your quotes from the article:

    “either something is really wrong here, or [the growth mindset intervention] produces the strongest effects in all of psychology”.
    He asks: “Is growth mindset the one concept in psychology which throws up gigantic effect sizes … Or did Carol Dweck really, honest-to-goodness, make a pact with the Devil in which she offered her eternal soul in exchange for spectacular study results?”

    That doesn’t look to me like you are being cautious.

    Especially when you follow it with:

    But here are a few things that predispose me towards the latter explanation.

    … and precede to write a book on all of the ways that you think Dweck has to be wrong.

    Do you think saying, at the beginning, that you are biased, and, at the end, that you are (officially) agnostic somehow means people are supposed to ignore what you wrote in between? Could the Buzzfeed article be absolved by simply including some more sentences as caveats at the beginning and end?

    • Spookykou says:

      What amount of hedging do you think would be appropriate?

      I guess he could try and hold himself to the standard he set a few posts ago.

      After each article he writes down a hidden confidence statement in what he is saying in the article, then performs a survey on people who have read the article and see what they predict his confidence in the subject is?

      He could then calibrate to make sure that on average people leave his essays with similar confidence in what he has said to his own feelings on the subject?

  18. Aleph Garden says:

    For growth mindset stuff, I’ve always had a rough model where outcomes were effort*talent, or max(effort, talent), or something. And in particular, the more talent you had, the more results you would get out of putting in effort. And as a result, the more talented and successful people would be more likely to have growth mindset, as a result of this.

    I realize that a lot of Dweck’s papers involve changing people’s mindsets, so this doesn’t really apply. But I think it’s part of it.

  19. brentdax says:

    Typo thread:

    he first letter of the blockquote about GRIM has been omitted.

  20. Scott H. says:

    I’m not growth mindset expert, but after reading Tom Chiver’s article — my first exposure — the theory strikes me as obviously true. Isn’t this a “Captain Obvious” type of thing?

    Coincidently, I am a father of two teenage daughters and have independently been pushing this growth mindset on them most of their lives.

  21. WorstMirari says:

    I worry that one day there’s going to be some weird effect that actually is a bizarre miracle. Studies will confirm it again and again. And if we’re not careful, we’ll just say “Yeah, but replication crisis, also I heard a rumor that somebody failed to confirm it,” and then forget about it.

    This sounds a lot like the history of the EMDrive? Though apparently NASA is putting it on a satellite, and that should be conclusive enough, one way or the other.

  22. thetitaniumdragon says:

    I think the results are actually quite easily explained, and aren’t even very surprising.

    So, why does growth mindset work?

    The answer is pretty simple: while IQ is extremely important, so is conscientiousness. In fact, it is probably the most important psychological trait after IQ.

    Believing that hard work pays off is likely to strongly correlate with conscientiousness – people who believe that hard work pays off are more likely to work hard and thus have it pay off. If you think that you can get better at something, that your abilities aren’t constrained by your intrinsic abilities, you’re more likely to try and do things.

    And in the end, your success at a task is a function of your ability, which is a combination of both raw ability and your trained ability.

    Given that a lot of improving at a task is repetition at it and getting feedback (especially with an eye towards improvement!), this would make people who believe that hard work and practice pay off will be more successful than those who don’t. And thus, people will end up more skilled if they practice at it more, and will practice at it more if they believe they can improve.

    There’s a powerful feedback mechanism at work here!

    More competent people believe hard work pays off (because, after all, it did for them). And if you are someone who has high ability AND high conscientiousness, you’ve maximized the two largest traits over your success.

    Likewise, per the Dunning-Kruger Effect, people who are incompetent often do not recognize their own incompetence. In the face of repeated failures, they will conclude that the system is rigged against them and there’s no point in working hard. Moreover, their inability to recognize that their failure comes from within cripples their ability to learn – they don’t recognize what they’re doing wrong, and ergo, will not change their behavior.

    Thus, we have both positive AND negative reinforcement – successful people will percieve that hard work pays off, and thus continue to improve, while failures cannot recognize the source of their failure, will attribute it to external forces, and will stop trying to get better.

    Incidentally, this idea is related to the scrub mentality, as detailed by David Sirlin in his book “Playing to Win”:

    http://www.sirlin.net/ptw-book/introducingthe-scrub

    I think you can see echos of this negative feedback loop in the scrub mentality.

    TL; DR; growth mentality is not surprising, it is simply a reflection of conscientiousness. It doesn’t mean ability doesn’t matter – it does. But people who give up because they don’t believe they’re able to succeed are pretty much certain to fail.

    EDIT: Incidentally, the idea that ability in many areas cannot be improved is far from obvious. You cannot make your eyesight better by practicing looking at things (we think, anyway), but you can make yourself stronger by lifting weights, and improve your endurance by running or swimming or hiking or engaging in other such activities on a regular basis. Vaccines can protect us against bacteria and viruses, and it is in fact possible to build up a resistance to certain toxins, while other poisons (like heavy metal) cannot have resistance built up to them at all.

    Just because some mental attributes, like intelligence, aren’t very malleable doesn’t necessarily mean that others are not. There’s no particular reason to believe that conscientiousness is not something which can be improved.

    • David Didau says:

      I don’t think there’s much evidence to suggest ‘conscientiousness’ is a stable construct. It’s more likely to be context dependent – we’re all conscientious about some stuff and we all slack off sometimes. One of the (many) problems with Dweck’s claims is that she says that giving students a fairly brief growth mindset intervention (which focuses on explaining the neuroscience involved in very basic layman’s terms e.g. “the brain is like a muscle [it isn’t]) improves students’ academic performance over the long term. I’m not sure how this could be said to correlate with even situational conscientiousness.

  23. David Didau says:

    Here is a link to the Bates & Li paper you refer to above: https://www.dropbox.com/s/yr2x8w0d23adth8/Li%20and%20Bates%20%282017%29%20SocArXiv%20Dweck%20Replication.pdf?dl=0

    Also, have you come across the ‘false growth mindset’? Dweck uses this to explain why her research isn’t always replicated. I suggested this is dangerously close to making her theory unfalsifiable here: http://www.learningspy.co.uk/research/is-growth-mindset-pseudoscience/

    Keep up the good work!