Comments on: 90% of all claims about the problems with medical studies are wrong http://slatestarcodex.com/2013/02/17/90-of-all-claims-about-the-problems-with-medical-studies-are-wrong/ In a mad world, all blogging is psychiatry blogging Fri, 24 Jul 2015 03:41:17 +0000 hourly 1 http://wordpress.org/?v=4.2.3 By: Q http://slatestarcodex.com/2013/02/17/90-of-all-claims-about-the-problems-with-medical-studies-are-wrong/#comment-65945 Mon, 28 Apr 2014 20:10:43 +0000 http://slatestarcodex.com/?p=59#comment-65945 Here is a 90% number: “He (Ioannidis) charges that as much as 90 percent of the published medical information that doctors rely on is flawed.”
http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/308269/

Report comment

]]> By: The Problem With Connection Theory | The Rationalist Conspiracy http://slatestarcodex.com/2013/02/17/90-of-all-claims-about-the-problems-with-medical-studies-are-wrong/#comment-60454 Wed, 23 Apr 2014 03:48:08 +0000 http://slatestarcodex.com/?p=59#comment-60454 […] when conducted in university laboratories with large grants, 80% of non-randomized medical studies turn out to be false, and even 25% of “gold-standard randomized trials” turn out to be false. And this is in […]

Report comment

]]> By: DanielLC http://slatestarcodex.com/2013/02/17/90-of-all-claims-about-the-problems-with-medical-studies-are-wrong/#comment-59387 Tue, 22 Apr 2014 04:49:08 +0000 http://slatestarcodex.com/?p=59#comment-59387 > Of these, 4 were negative results (ie “X doesn’t work”) so he threw them out. This is the first part I think is kind of unfair.

I disagree. Negative results don’t show that an effect isn’t there. They just show that it’s too small to see with that sample size. A negative result being later disproven does not show a flaw in the original study.

If you show that the effect is there and is large enough that the first study shouldn’t have missed it, that’s a problem, but it makes this way more complicated so it’s easier just to ignore those studies.

Thinking about this more, I guess they’d have to do something like this either way, to show that the study didn’t just fail to replicate because the second study had a false negative. Either they’d have to look at ones where the study where it fails is much more powerful, or they’d have to use a two-tailed T-test and show that the two studies shouldn’t result from the same effect.

Report comment

]]> By: MetaMed launch day | Slate Star Codex http://slatestarcodex.com/2013/02/17/90-of-all-claims-about-the-problems-with-medical-studies-are-wrong/#comment-429 Wed, 27 Feb 2013 21:50:46 +0000 http://slatestarcodex.com/?p=59#comment-429 […] so much. It kind of fizzled out before we came to any strong conclusions, but in a world where some hard-to-determine number which is not ninety percent of medical studies are wrong, it’s exactly the sort of thing that should be done […]

Report comment

]]> By: Future tense | Slate Star Codex http://slatestarcodex.com/2013/02/17/90-of-all-claims-about-the-problems-with-medical-studies-are-wrong/#comment-196 Wed, 20 Feb 2013 15:23:04 +0000 http://slatestarcodex.com/?p=59#comment-196 […] to survive or ever have a career on them sticking around. Besides, just in case they get tired of me accidentally misrepresenting their positions or projecting general cluelessness at them, I should try not to have all my eggs in one basket. […]

Report comment

]]> By: Sniffnoy http://slatestarcodex.com/2013/02/17/90-of-all-claims-about-the-problems-with-medical-studies-are-wrong/#comment-133 Tue, 19 Feb 2013 13:58:25 +0000 http://slatestarcodex.com/?p=59#comment-133 Yes, but you say that the empirical 5% matches what you’d expect from a p-value of 5%, and I don’t think that’s correct. Unless you just mean “hey look these numbers are the same!” which doesn’t really mean anything by itself.

I mean, you talk about just taking the p-value into account rather than the base rate, but it’s not at all clear to me that the way you do so is meaningful. Just considering the equation P(H|E)=P(E|H)*P(H)/P(E), you’re suggesting that we “don’t take into account base rate” by assuming P(H)/P(E) is about 1? I really don’t see what makes such an assumption reasonable.

Now if you want to say, “Let’s not worry about what P(H) is, and so just assume P(H)/P(E) is some constant”, that might make more sense. But then you can’t get any particular number out of it.

Report comment

]]> By: Nancy Lebovitz http://slatestarcodex.com/2013/02/17/90-of-all-claims-about-the-problems-with-medical-studies-are-wrong/#comment-99 Tue, 19 Feb 2013 03:06:36 +0000 http://slatestarcodex.com/?p=59#comment-99 I thought a lot of the point of MetaMed was to find sound but neglected research– at least as much that as debunking bad research.

Report comment

]]> By: Scott Alexander http://slatestarcodex.com/2013/02/17/90-of-all-claims-about-the-problems-with-medical-studies-are-wrong/#comment-82 Mon, 18 Feb 2013 21:08:35 +0000 http://slatestarcodex.com/?p=59#comment-82 Do feel like I’ve heard the 90% number somewhere, possibly somewhere nonofficial in private conversation with someone, but unless I can track down a source I’ll edit it out with apologies. I still think “80% of non-experimental studies” is a pretty big caveat compared to “80% of research” but I have no idea how this was phrased and for all I know you said it that way. Sorry about that.

Report comment

]]> By: Scott Alexander http://slatestarcodex.com/2013/02/17/90-of-all-claims-about-the-problems-with-medical-studies-are-wrong/#comment-81 Mon, 18 Feb 2013 21:03:01 +0000 http://slatestarcodex.com/?p=59#comment-81 The 5% number comes not from p-values but from the empirical observation that of 40 studies analyzed, 2 were wrong. This matches the number of studies that would be wrong merely by chance if we only took the p-value into account rather than the base rate.

Report comment

]]> By: Alyssa Vance http://slatestarcodex.com/2013/02/17/90-of-all-claims-about-the-problems-with-medical-studies-are-wrong/#comment-80 Mon, 18 Feb 2013 19:31:42 +0000 http://slatestarcodex.com/?p=59#comment-80 For what it’s worth, I don’t know of anyone at MetaMed who has ever claimed that 90% of studies are false, so I think your first sentence might be straw manning. Me and several others have claimed that 80% are false, but that’s much more in line with his actual results, as you note.

Report comment

]]>