[Partial Retraction] Age Gaps and Birth Order Effects

On Less Wrong, Bucky tries to replicate my results on birth order and age gaps.

Backing up: two years ago, I looked at SSC survey data and found that firstborn children were very overrepresented. That result was replicated a few times, both in the SSC sample and in other samples of high-opennness STEM types. Last year, I expanded those results to look at how age gaps affected birth order effects. Curiously, age gaps less than seven years did not seem to attenuate birth order, but age gaps of more than seven years attenuated it almost completely.

Bucky analyzed the same data and found that I bungled one and a half of my results. Left graph in each pair is mine, right is Bucky’s.

In the first analysis, Bucky replicates my results: in people with exactly one sibling show a sudden cliff in birth order effects after seven years.

In the second and third analysis, Bucky finds that I screwed up. I mislabeled the second analysis (people with more than one sibling) as the third (full sample), and my third analysis was just wrong (I double-counted people with one sibling).

When Bucky corrects these errors, they find that the weird cliff at seven years is present only in the sample of people with exactly one sibling. This makes it more likely to be a weird coincidence about that sample, and less likely to be a weird phenomenon. They also identify a potential confounder (there may be longer gaps between later-born children than between first-borns and second-borns) which also slightly affects the results, although does not dramatically change the conclusions.

Bucky then does their own analysis of the correct results, and finds that most likely the sudden drop at seven years is a coincidence. They conclude that:

The SSC 2019 survey data support a constant, high, birth order effect (~2.4 oldest siblings for every 1 second oldest sibling) for age gaps less than 4-8 years. This is followed by a decline to a lower birth order effect at an undetermined rate. The decline does not necessarily completely remove any birth order effect although this may be the case for very large age gaps.

The data provide some evidence that:

– The reduction may not be the same (or might disappear) for larger families (4+ children)
– Birth order effect may be lower at 1 year age gap vs 2-7 year age gap

However the evidence for both of these points is relatively slim.

As those of you who have put up with my constant typos realize, I am not a very careful person. I try to double-check any result I present on my blog, and in some cases (including this one) ask other people to double-check them for me as well. Sometimes mistakes still slip through and I’m sorry. I am partially retracting the previous results (“partially” because some of the analyses were correct and the conclusion is still basically the same).

In response to this, I have added a note in bold to the top of the original age gaps post directing readers to the failed replication and reanalysis. I’ve also added a paragraph about this to my Mistakes page to help people calibrate how much to believe my future results. I am also writing this post to make sure the replication gets at least as much prominence as the original results.

I continue to make the raw survey data available for everybody to double-check my work, except for parts that could seriously compromise people’s privacy. Please, if you have any doubt in my findings at all, do your own analysis and let me know what you get.

This entry was posted in Uncategorized and tagged , , . Bookmark the permalink.

21 Responses to [Partial Retraction] Age Gaps and Birth Order Effects

  1. Randy M says:

    As those of you who have put up with my constant typos realize, I am not a very careful person

    I think even if a writer prints out and reads aloud their work there will still be a typo or two remaining.
    Given the length of some posts, I can’t fault you for anything but the the occasional repeated word.

    • niohiki says:

      We can only wish other blogs/newspapers had the same standards of “not being careful”…

    • Protagoras says:

      Yeah, proofreading one’s own work is hard. I very reliably see what I intended to say rather than what’s actually on the page, and so typos can survive multiple rereadings.

    • beleester says:

      I think I’ve been on this site too long, because the the instant you mentioned repeated words I knew exactly what you were up to.

    • Tom Chivers says:

      I wrote a book; I read it through three times; it was edited, line-edited and proof-read, at least four people in total. And then when it came time for me to do the audiobook, I still found dozens of typos.

      Edit: ah ha. I should have been looking out for the the trick you were playing.

  2. hnau says:

    Even after the retraction this investigation is still very cool.

    Age gap is reported WRT the next-oldest or next-youngest sibling, right? I can’t help but wonder if the “lower birth order effect at 1-year age gap” phenomenon disappears for families of 3+ siblings because the sample includes people with, say, a 1-year-older *and* a 4-year-older sibling.

    • Emma_B says:

      Indeed. It does not change much this very suprising result! I am really wondering what could cause the observed pattern. As a first step toward an explanation, it would be interesting to see if the same thing is observed among the readers of other blogs with similar focus.

      And kudos for making the whole dataset publicly available, and for publishing the correction!

    • Bucky says:

      In the reanalysis only oldest and second oldest children are included so anyone with more than 1 older sibling is excluded.

      I wandered about the same pattern but the sample sizes are small enough (120-150 in each of the three groupings) that the difference between 1&2 siblings vs 3+ siblings isn’t significant (p>0.05).

  3. Statismagician says:

    I just want to give you extra props for a) having thought of this in the first place, and b) having published such a forthright account of the new results. Statistical programming is very fiddly; anybody can make a mistake, but only a few would be willing to publicize the correction. Much praise to you, Scott.

    • eric23 says:

      A lot of people are willing to admit their mistakes when it won’t affect their careers.

      Scott, whose career is in individual therapy not online social research, fits that category well.

      • Taleuntum says:

        I don’t know. It might just be me, but I find that I often get overly attached to things I say during a casual conversation with my friends. I can only imagine what it could feel like to admit a mistake to the whole readership of a popular blog whose reputation was built with a decade of hard work.

    • discountdoublecheck says:

      Fully agree. This level of disclosure is laudable. Even for something which mostly didn’t change much.

      Scott — your diligence in posting data is what makes this possible. It would be easier for people to check (and expand on) your work if you posted your analysis code/spreadsheets as well. Even if you do everything perfectly, for someone to replicate your initial data work and then expand on something you do can be difficult without guidance.

    • Bucky says:

      Plus publishing the data in the first place – this is pretty rare and I couldn’t have done a thing without it

  4. baconbits9 says:

    I have been going around trying to make myself sound smart repeating these results. Now what?

  5. fwiffo says:

    I suspected this result was wrong in the initial post (proof: https://slatestarcodex.com/2019/05/14/age-gaps-and-birth-order-effects/#comment-752355) just because it was too weird — why should something be special about 7 years and not 6 years? It’s pretty rare for natural social phenomena to have cliff effects like this. But dramatic effects like these often appear in data due to chance or coding errors.

    I was reluctant at the times to make the claim on the basis of “this effect is just too weird to be plausible” even though that was my instinct as someone who has been analyzing social science data for years. I was reluctant b/c it seems like on a random internet forum, an argument based on one’s personal priors can’t carry all that much weight.

    Kudos to Bucky for actually hitting the numbers.

    • baconbits9 says:

      I don’t think it would be surprising if there was a cliff effect, with universal schooling there would be an age where if there was an X year gap the first year of your life (for a high % of kids) would be spent at home with another sibling at home and X+1 would be spent at home with the sibling in school for 6 hours a day.

  6. meltedcheesefondue says:

    Thanks so much for having published this negative result. More people should follow your example.

  7. keaswaran says:

    Watch the y-axis in that second graph on the right! All the other charts are 50-80, while that one is 30-90! I only noticed this because I was trying to figure out how the third chart could have values lower than both the first and second chart – and then noticed the discrepancy in y-axis so that it’s actually in between them!

    Anyway, it looks like the truth is that there’s a birth order effect that is strongest for two-child families with a small number of years of separation, but dies off as the family size increases and the age separation increases, together with a weird outlier in the relatively small sample of 2-child 8-year separation families (if there had been six more first-child 2-child 8-year families and six fewer second-child 2-child 8-year families, then it would have looked much more like a linear decrease, and a bigger effect for the one column that bins together multiple age gaps).

  8. stopandgo says:

    The SSC 2019 survey data support a constant, high, birth order effect (~2.4 oldest siblings for every 1 second oldest sibling) for age gaps less than 4-8 years. This is followed by a decline to a lower birth order effect at an undetermined rate.

    In short, still a very large effect due to some major effects on IQ (typically estimated at 3 points, which has enormous effects at the small area of the right tail represented by readers here) or potentially some other personality trait like intellectual openness or seeking out rational explanations and theory rather than watching and learning from a similar-age sibling carrying forward into life generally.

    Given the enormous amount of evidence for the IQ spread, and a very reasonable causal mechanism (most unshared effects on IQ are negative, and there is a lot of evidence antibodies against the first fetus affect the second–whether increased odds of homosexuality, or the dangers of Rh mismatch without treatment), it’s probably mostly this. And either the antibodies from the first baby fade with time or more and more of those pregnancies involve a second wife or lover.