The Marginal Revolution comment section is one of the the worst this side of a PUA blog’s though. I think the only reason I still read them is out of misplaced sense of nostalgia.

Report comment

Bayes’ theorem is how we deal with conditional probabilities. You cannot interpret a p-value without using it. The probability of obtaining a false positive depends on the probability that the null hypothesis is false. Without an estimate of that probability you have no way of interpreting a p-value.

– In the example I used (with a prior of 10% false H0s) a p-value of 0.05 gave a probability of a false positive of 0.36.

– If the prior was that we are equally likely to be testing a false H0 as a true one then it would be 9%.

– The two probabilities only become equal when we are 100% certain that the null hypothesis is false. In which case we would need to do the research in the first place.

This is really basic stuff and very few researchers understand it. This is a huge failing of statistics.

Report comment

Report comment

Say 10% of null hypotheses can be disproven for a given study design (power, size of effect etc) and the rest are true within the limitations of our sample size.

Say we get p<0.05 and the study had 80% power. What is the probability that the null hypothesis is false?

Out of 1000 trials 100 will have a false null hypothesis.

We will get p<0.05 for 80 of these (because we have 80% power).

900 will have a true null hypothesis.

We will get p<0.05 for 45 of these.

In total we have 125 'positive' results and 45 of them are false positives.

So we can only be ~65% certain that the null hypothesis is false when p<0.05 in this situation.

If you repeat the test for all the positive trials (65% of which have a false null hypothesis this time around) you get down to a p-value of around 0.03. A Bayesian argument for confirmatory trials.

A randomised controlled trial is just a diagnostic test for treatment differences. The p-value is the specificity and the power is the sensitivity. Calculate PPV and NPV as normal.

[This was first pointed out to me by MKB Parmar and has been written a lot about by John Ioannides.]

Report comment

Report comment

Report comment

However, maybe publish the likelihood ratio, an assumed prior, and the implied posterior probability.

Report comment

No, ob/gyns won’t be evaluating genetic tests (except BRCA). But neither will geneticists. The relevant comparison is genetic counselors, a job created in 1979 to counsel pregnant women about the morality of aborting Downs fetuses.

Report comment

Report comment

Report comment