On Less Wrong, Bucky tries to replicate my results on birth order and age gaps.
Backing up: two years ago, I looked at SSC survey data and found that firstborn children were very overrepresented. That result was replicated a few times, both in the SSC sample and in other samples of high-opennness STEM types. Last year, I expanded those results to look at how age gaps affected birth order effects. Curiously, age gaps less than seven years did not seem to attenuate birth order, but age gaps of more than seven years attenuated it almost completely.
Bucky analyzed the same data and found that I bungled one and a half of my results. Left graph in each pair is mine, right is Bucky’s.
In the first analysis, Bucky replicates my results: in people with exactly one sibling show a sudden cliff in birth order effects after seven years.
In the second and third analysis, Bucky finds that I screwed up. I mislabeled the second analysis (people with more than one sibling) as the third (full sample), and my third analysis was just wrong (I double-counted people with one sibling).
When Bucky corrects these errors, they find that the weird cliff at seven years is present only in the sample of people with exactly one sibling. This makes it more likely to be a weird coincidence about that sample, and less likely to be a weird phenomenon. They also identify a potential confounder (there may be longer gaps between later-born children than between first-borns and second-borns) which also slightly affects the results, although does not dramatically change the conclusions.
Bucky then does their own analysis of the correct results, and finds that most likely the sudden drop at seven years is a coincidence. They conclude that:
The SSC 2019 survey data support a constant, high, birth order effect (~2.4 oldest siblings for every 1 second oldest sibling) for age gaps less than 4-8 years. This is followed by a decline to a lower birth order effect at an undetermined rate. The decline does not necessarily completely remove any birth order effect although this may be the case for very large age gaps.
The data provide some evidence that:
– The reduction may not be the same (or might disappear) for larger families (4+ children)
– Birth order effect may be lower at 1 year age gap vs 2-7 year age gap
However the evidence for both of these points is relatively slim.
As those of you who have put up with my constant typos realize, I am not a very careful person. I try to double-check any result I present on my blog, and in some cases (including this one) ask other people to double-check them for me as well. Sometimes mistakes still slip through and I’m sorry. I am partially retracting the previous results (“partially” because some of the analyses were correct and the conclusion is still basically the same).
In response to this, I have added a note in bold to the top of the original age gaps post directing readers to the failed replication and reanalysis. I’ve also added a paragraph about this to my Mistakes page to help people calibrate how much to believe my future results. I am also writing this post to make sure the replication gets at least as much prominence as the original results.
I continue to make the raw survey data available for everybody to double-check my work, except for parts that could seriously compromise people’s privacy. Please, if you have any doubt in my findings at all, do your own analysis and let me know what you get.