Whenever someone sneers “Evidence-based medicine? You wouldn’t demand a double-blind placebo-controlled clinical trial of PARACHUTES, would you?” I feel a strong urge to use them as the control group in my double-blind parachute experiment.
Of course, deep down inside I know that this would be morally wrong. Groups need to be determined by random assignment.
Also, we already have fairly good natural experiments in what happens when people with the ‘falling’ condition don’t get the inteventon.
Can’t resist posting a link to this:
Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials
I remember that paper, sent a link to a friend who was trying skydiving for the first time so she’d know not to put her faith in insufficiently researched protective methods. But she went skydiving anyway, and fortunately was one of the lucky ones who survived (and for all we know, the parachute might have helped; we just don’t have the data yet to know for sure).
The ethical thing to do is to test against current best practice, not against a placebo. So the control group should remain on the plane, not jump without a parachute.
(This point was already made by Timothy Church in his response to the original paper.)
Yes, it’s a classic joke and widely-referenced. Problem is, when people invoke it to defend not testing their favorite intervention, the intervention is never quite on the same footing as parachutes.
My first stats textbook required we design exactly this, then asked us to explain why achieving a double-blind test could be tricky.
Of course, it’s also the textbook with this style of footnotes.
Why would a double-blind trial be tricky? Just give backpacks that, when the cord is pulled, either release a parachute or release cords not attached to anything, then blindfold the participants. The skydiving instructors hand out backpacks without knowing which is which.
Because people will know when they pull the cord whether something activates and comes out. You’d have to blind them, numb their entire bodies so they couldn’t feel the chute pull (or not pull), and probably deafen them with earplugs for good measure.
… You could also use crash test dummies as a proxy for injury outcomes 😛
Yeah, if you wanted to study the effects of parachutes on crash test dummies.
A nice story for you:
One day when I was a junior medical student, a very important Boston surgeon visited the school and delivered a great treatise on a large number of patients who had undergone successful operations for vascular reconstruction.
At the end of the lecture, a young student at the back of the room timidly asked, “Do you have any controls?” Well, the great surgeon drew himself up to his full height, hit the desk, and said, “Do you mean did I not operate on half the patients?” The hall grew very quiet then. The voice at the back of the room very hesitantly replied, “Yes, that’s what I had in mind.” Then the visitor’s fist really came down as he thundered, “Of course not. That would have doomed half of them to their death.”
God, it was quiet then, and one could scarcely hear the small voice ask, “Which half?”
Source: Dr. E. E. Peacock, Jr., quoted in Medical World News (September 1, 1972), p. 45, as quoted in Tufte’s 1974 book Data Analysis for Politics and Policy.
Parachutes *are* evidence-based medicine, there are other sorts of evidence than the double-blind RCT. The double-blind RCT is the *best* form of evidence (short of the actual Word of God Spoken From On High), but, as in the parachute case, sometimes it’s prohibitively expensive to run one.
A very important point, and one that is often lost when this stuff becomes popularized. The cargo cult of the RCT with p > 0.05 is initially a step up from the cargo cult of argument from authority, but with rampant gaming, that advantage has been eroding for some time.
For related reasons, I have serious doubts that the spread of “evidence-based medicine” is currently improving clinical decision-making on the margin; there are many instances when poorly understood EBM, despite using the buzzwords of rationality, ends up being actively anti-empirical.
First-rate truth finding ultimately requires first-rate reasoners; there’s no stable institutional shortcut.
What kinds of things do you have in mind for “actively anti-empirical”?
Focus on the “poorly-understood” part. If you read a study on some drug and assume it to be fully true and accurate, you might not ever hear about the rest of the literature that disproves that study or shows it’s results only apply in some limited circumstances.
This happens constantly in real life, and is fairly scary; doctors learn something at some point in their career (often medical school) and then don’t keep up on the progress of their field, and end up giving decades-old medical advice that would be grounds for disciplining a medical student of today.
A double blind study is a statistical method that we use because we have no underlying model to engineer a solution. We do not have a model how different chemicals work in the human body, but we do have a model how a parachute works.