Pre-emptive plagiarism is the worst. I was all set to write about how I thought the problems I brought up in The Control Group Is Out Of Control could be addressed.
Then Josh Haas wrote A Modest Proposal To Fix Science, which took the words right out of my mouth. Separate out exploratory and confirmatory research, have the latter done by different people with no stake in the matter.
So if I want to wring a blog post out of this I’m going to have to go way further than that, come up with something really outlandish.
So here is how science works in the utopian culture of Raikoth.
Anyone can do exploratory research. It can be experiments published in special exploratory research journals. Or it can be a collection of anecdotes supporting a theory published in a magazine. Or it can be a list of arguments on a website. The point is to get an idea out there, build interest.
Remember the Angel of Evidence? The centralized nationwide prediction market? Anyone with a theory can list it there. The goal of exploratory research is to get people interested enough in the idea to bet about it on the Angel.
Suppose you become convinced that eating grapes cures cancer. So you submit a listing to the Angel: “Eating grapes cures cancer”. Probably most people doubt this proposition and the odds are around zero. So you do some exploratory research. You conduct a small poorly controlled study of a dozen cancer patients who sign up, and feed them a lot of grapes. You report that all of them seemed to have their cancer go away. You talk about how chemicals in grapes are known tumor inhibitors. Gradually a couple of people start thinking there’s something to what you’re saying. They make bets on the prediction market – maybe saying there’s only a 10% chance that you’re right, but it’s enough. The skeptics, and there are many, gladly bet against them, hoping to part gullible fools from their money. Business on the bet starts to go up.
These research prediction markets are slightly negative-sum. Maybe the loser loses $10, but the winner only gets $9. When enough people have bet on the market, the value of this “missing money” becomes considerable. This is the money that funds a confirmatory experiment.
What this means is that it is interest in – and disagreement about – the question at hand that makes an experiment possible. When no one believes grapes cure cancer, everyone’s probability is around zero and so no one bets against anyone else and there is no money available for the experiment. When you do your exploratory research and come up with good arguments why grapes should work, then if you really make your case some people should be willing to bet on it – not at even odds, maybe, but at least at 10:1 odds favoring them or whatever.
Suppose the experiment returns positive results (what qualifies as “positive results” is predefined – maybe the bet specifies “effect size > 0.4, p < 0.05"). Now one of two things happens. Either everyone is entirely convinced that grapes cure cancer, and people stop doing science and start eating more grapes. Or the controversy continues. If the controversy continues, a bet can be placed on the prediction market for the success or failure of a replication. No doubt people will want to take or short this bet at very different odds than they did the last one. Maybe you could get 10:1 odds against grapes curing cancer the first time, when you were going on a tiny exploratory study, but now you can only get even odds. No problem. The pro-grape side bets in favor, the anti-grape side is still willing to bet against, and the replication takes place. Rinse and repeat. At every step, one of three things is true. First, there is still controversy, in which case the controversy funds more experiments, and the odds at which people will bet on those experiments is the degree of credence we should have in the scientific prediction involved. Second, there isn't enough money in favor of the proposition to get a market going, in which case the proposition has been soundly disproven. Third, there isn't enough money against the proposition to get a market going, in which case the proposition is universally accepted scientific fact. In practice things are not this easy. The system is excellent at resolving controversies, and you can easily get as much money as you need to study whether guns decrease crime or whatever. But science includes not just controversies but basic research. Things like particle physics might suffer - who is going to bet for or against the proposition that the Higgs boson has a mass greater than 140 GeV? Only a couple of physicists even understand the question, and physicists as a group don't command large sums of spare capital. So what happens is that scientific bodies - the Raikothin equivalent of our National Science Foundation - subsidize the prediction markets. This is very important. Instead of donating $1 million to CERN to do boson research, they donate $1 million to the Angel of Evidence to make the prediction market more lucrative. Suddenly the market is positive-sum; maybe you lose $10 if you're wrong, but gain $11 if you're right. The lure of free money is very attractive. Some ordinary people jump in, not really sure what a boson is but knowing that the odds are in their favor. But more important, so do "science hedge funds" that hire consultant physicists to maximize their likely return. Just as hedge fundies in the US might do lots of research into copper mining even though they don't care about copper at all in order to figure out which mining company is the best buy, so these "science hedge funds" would try to figure out what mass the Higgs boson is likely to have, knowing they will win big if they're right. Although the National Science Fund type organization funds the experiments indirectly, it is the money of these investors that directly goes to CERN to buy boson-weighing machinery.
So much for funding. How are the actual experiments conducted?
They are conducted by consultant scientists. The number one rule of being a consultant scientist is you do not care about the hypothesis.
The Raikolin would have a lot of reasons to react in horror if someone pointed them to Earth, but one of the bigger ones is that the person who invented a hypothesis is responsible for testing it. Or at least someone in the same field, who has been debating it for years and whose entire career depends upon it. This makes no more sense than asking criminals to judge their own trials, or having a candidate count the votes in their own election.
Having any strong opinion on the issue at hand is immediate disqualification for a consultant scientist to perform a confirmatory experiment.
The consultant scientist is selected by the investors in the prediction market. Corporate governance type laws are used to select a representative from both sides (those who will profit if the theory is debunked, and those who will profit if it is confirmed). Then they will meet together and agree on a consultant. If they cannot agree, sometimes they will each hire their own consultant scientist and perform two independent experiments, with the caveat that a result only counts if the two experiments return the same verdict.
As the consultant plans the experiment, she receives input from both the pro- and the con- investors. Finally, she decides upon an experimental draft and publishes it in a journal.
This publication is a form of pre-registration, but it’s also more than that. It is the exact published paper that will appear in the journal when the experiment is over, except that all numbers in the results section have been replaced by a question mark, ie “We compared three different levels of grape-eating and found that the highest level had ? percent less cancer than the lowest, p < ?". The only difference between this draft and the real paper is that the real one fills in the numbers and adds a Discussion section. This gives zero degrees of freedom in what tests are done and in how the results are presented.
Two things happen after the draft is published.
First, investors get one final chance to sell their bets or bow out of the experiment without losses. Perhaps some investors thought that grapes cured cancer, but now that they see the experimental protocol, they don’t believe it is good enough to detect this true fact. They bow out. Yes, this decreases the amount of money available for the experiment. That comes out of the consultant scientist’s salary, giving her an incentive to make as few people bow out as possible.
Second, everyone in the field is asked to give a statement (and make a token bet) on the results. This is the most important part. It means that if you believe grapes cause cancer, and the experiment shows that grapes have no effect, you can’t come back and say “Well, OBVIOUSLY this experiment didn’t detect it, they used overly ripe grapes, that completely negates the anti-tumor effect, this study was totally useless and doesn’t discredit my theory at all”. No. When the draft is published, if you think there are flaws in the protocol, you speak then or forever hold your peace. If you are virtuous, you even say something like “Well, right now I think grapes cure cancer with 90% probability, but if this experiment returns a null result, I guess I’ll have to lower that to 10%.”
These statements are made publicly and recorded publicly. If you say an experiment will prove something, and it doesn’t, and this happens again and again, then people will start noticing you don’t actually know any science.
(If you’re always right, you immediately get hired by a science hedge fund at an obscenely high salary.)
Finally, the consultant scientist does her experiment. Some result is obtained. The question marks in the draft are filled in, and it is resubmitted as a published paper. The appropriate people make or lose money. The appropriate scientific experts gain or lose prestige as people who are or aren’t able to predict natural processes. The appropriate consultant scientists gain or lose prestige as people whose results were or weren’t replicated. The exploratory scientist who proposed the hypothesis in the first place gains or loses prestige that make people more or less likely to bet money on the next idea she comes up with.
There are no homeopaths in Raikoth.
I mean, there were, ages ago. They proposed experiments that could be done to prove homeopathy, and put their money where their mouth was. They lost lots of money when it turned out not to work. They added epicycles, came up with extra conditions that had to be in place before homeopathy would have an effect. Their critics were more than happy to bet the money it took to test those conditions as well, and the critics ended up rich. Eventually, the homeopaths were either broke, or sufficiently mugged by reality that they stopped believing homeopathy worked.
But homeopathy is boring. The real jewel of this system is to be able to go online, access the Angel of Evidence, and see a list of every scientific hypothesis that anyone considers worth testing, along with the probability estimate that each is true. To watch as the ones in the middle gradually, after two or three experiments, end up getting so close to zero or one hundred as makes no difference, and dropping off the “active” list. To hear the gnashing of the teeth of people whose predictions have been disconfirmed and who no longer have a leg to stand on.
Also, if you can predict the masses of bosons consistently enough, you get crazy rich.