This week I’ll be presenting entries from the adversarial collaboration contest.
Remember, an adversarial collaboration is where two people with opposite views on a controversial issue work together to present a unified summary of the evidence and its implications. In theory it’s a good way to make sure you hear the strongest arguments and counterarguments for both sides – like hearing a debate between experts, except all the debate and rhetoric and disagreement have already been done by the time you start reading, so you’re just left with the end result.
A few months ago, I asked readers to write adversarial collaborations and submit them to me. After the inevitable flakeouts and disappearances, I got four entries:
1. Does the current US education system adequately serve advanced students? (by Michael Pershan and TracingWoodgrains)
2. Is Islam compatible with liberal democracy? (by John Buridan and Christian Flanery)
3. Should childhood vaccination be mandatory? (by Mark Davis and Mark Webb)
4. Should children who identify as transgender start transitioning? (by a_reader and flame7926)
I’m going to post one of these per day. Over the weekend, I’ll post a link to a poll where readers can vote for their favorite. I’m also going to vote for my favorite, and my vote will be worth 5% of the total number of reader votes. Whoever gets the most votes wins. The prize is $1000; thanks to everyone who donates to the Patreon for making this possible.
Please put any comments about the contest itself here, not on the individual entries.
Should our votes be based solely on which participants did the best collaborative work, on which collaboration we found most interesting, or some combination of the two? For example, I’m less interested in the topic of transgender children than I am in mandatory vaccinations, but I don’t want to bias my vote against that pair of collaborators if they do good research but theirs ends up not being my favorite purely for topic reasons.
I’m going to deliberately avoid having a position on this. Vote in whatever way seems most natural to you.
So vote for the highest-status authors? Got it. 😛
To be serious for a moment, thanks for organizing this.
The vote is not for which one is best, or most insightful, or about anything other than “Which one of these entries should be paid $1k”.
In that case, we should vote for the entry that we want future collaborations to be most like.
And also comment on which elements of each entry made them most attractive to your vote.
I will vote, at least in part, based on whichever entry I learnt the most from.
For what it’s worth (probably very little), if it were me, in the future I would consider encouraging collaborations with much more limited scope. It seems to me you would want to hone in on a single, very discrete fact and assess the evidence for and against. Rather than try to tackle whether vaccinations should be mandatory — which sweeps in not just broad questions of efficacy but also broad governmental policy aspects (which in turn are likely to depend on one’s broader views about libertarianism, etc.) — it seems likelier to be helpful to pose a question that narrows down to some single aspect, like the degree of efficacy of a particular vaccine x for condition y or something. That allows you to pull the relevant scientific literature on that narrow question and debate how to make sense of the studies. It’s hard to see how you can do that with a sweeping question like should vaccinations in general (which ones?) be mandatory (based on whose political theory?). The latter seems more like a broad debate rather than something one could meaningfully get one’s hands around for the purposes of an adversarial collaboration seeking to weigh the evidence pro and con. I worry that the other topics, too, are a bit on the broad and vague side. None of them seem susceptible to pulling a manageable universe of scientific publications and arguing for an empirical answer based on a discrete universe of evidence. I hope this doesn’t come across as critical of the teams who made these submissions. I admire them for putting in the work on these. But I do suspect that there might be more room for progress by taking a single fact question and collaborating to evaluate the evidence for it.
All valid points. On the vaccine collaboration specifically, these issues were definitely something we struggled with. The problem is if you’re dealing with a proposition and an opposition, you’re pulled in two different directions. On the one hand, with the broad topic you’re pulled in multiple different directions (and you’ll see later this week that we also dealt with added layers of complexity beyond just different vaccines you suggested). So the broad proposition makes it difficult to give adequate space to explore everything relevant due to time and space constraints. On the other hand, you don’t get as strong an opposition if you claim, “Issue X is driven by Y”. Then if X didn’t drive Y, and the opposition moves to the next explanation, you’re not getting to the core argument. You’re playing whack a mole with unimportant details.
In the scientific literature, the are lots of review articles summarizing questions such as those you talk about. We cite a few of them, but in general I don’t know how much more a collaboration on those grounds would be beyond simple signal boosting.
I also think the broad questions are really the ones people are more interested in answering. If there’s something wrong with the rotavirus vaccine, there’s evidence to that effect, and nobody does anything about it we should probably be asking questions of our regulatory safeguards (perhaps worldwide!)
My sense is that for many of these questions people are interested in seeing exactly what Scott is going for here. Most of the debate and sub-arguments will happen under the surface, and much of the discussion won’t make it into the final text (at least it didn’t in ours). But if I’m wondering, “I’ve heard arguments in favor of/against minimum wage, I wonder what experts think?” in my experience I’m not looking to adjudicate specific ideas, like under what circumstances MW will influence restaurant workers, or if that is offset in some way. I want to know if there’s evidence that MW actually has the poverty-reduction effects it’s intended to have, had undesirable side effects that outweigh any benefits, and whether it’s a policy that is better than alternatives. That’s more high level, and I suspect that part of the reason some people didn’t finish is because it takes longer to plow through all that material and only touch on a few specific ideas both sides can coalesce around.
FWIW, I disagree. Whether vaccine X is effective (or to which degree it is effective) for condition Y is something of an objective truth. We may or may not have the answer and it may be ‘controversial’ to the tiny amount of people working on vaccine X or with the condition Y but for the rest of us, it’s as boring as watching paint dry.
While the question as asked, by pitting the common good against individual freedom, does allow for a not-so-clear-cut debate where both sides can make valid points and that is of a broad interest to a lot of people.
For example, I am French so mandatory vaccines not only feel very natural to me but the reasoning behind seems unimpeachable to me. And I understand and accept the trade-off in terms of individual freedoms.
But while this is pre-approved b/c of my French upbringing, this is also a pragmatic issue. I wouldn’t agree to any invasion of personal/individual freedoms in the name of the common good.
Vaccination isn’t where I’d draw the line but it’s interesting to learn about the American psyche on such topics. I wouldn’t have nearly the same enthusiasm for a cut and dry technical discussion surrounding the efficacy of a given vaccine/medicine.
I think it’s important to be vague enough that one member of the collaboration doesn’t get “beaten”. If the scope is too limited and specific, then one member of the collaboration will be wrong and one will be right. The one who is wrong will be forced to admit that they’re wrong, or drop out of the collaboration, or just become difficult to work with.
Personally, what I hope to get out of reading these essays is an overview of the state of play regarding the evidence and arguments for some broad questions. I’m not hoping to read “the answer” to the question.
The exception to the above is when both collaborators are experts in a field and they both want to get to the bottom of why my studies always give one result and your studies always give the other.
I understand that concern. The smaller the scope the less adversarial the collaboration might be.
But if we assume that part of the goal is “cooling down the Culture War” and starting from adversarial positions before in depth research initiates, then I think we are supposed to be tackling these larger questions which academics don’t assay, no?
Would you say that Scott tried?
I stand corrected.
Will there be a place for discussion of those topics and teams that didn’t ultimately produce a full collaborative essay?
I was a member of such a team and would write a brief summary of what we found if I knew there would be a place for it. There may be some meaningful discussion to be had from our work.
On each of the collaborations and the poll, have you considered whether or not to allow comments? While the commentators here are high quality and usually have something valuable to add, they might unfairly influence the vote, especially when considering the order you post. Whichever is posted later in the week may have an advantage due to being more memorable and not exposed to criticism for as long.
As someone completely uninvolved, I’d be glad to hear those sorts of summaries too. @Scott, perhaps you could post another thread for those after the four finished entries?
It would also be interesting to have some discussion of why certain projects didn’t succeed (in the specific sense of writing a joint essay) if Scott is aiming to investigate adversarial collaboration more generally. So if you’d be willing to do a post mortem of why yours didn’t work that would be interesting
I’m a bit sad that no one took me up on my bid, but I’m quite interested in seeing the results here!
I’d be interested in looking at this topic and producing something like an adversarial collaboration with you if you are still interested.
I work for a private property developer so I’m very familiar with the issues at the coalface for housing delivery, and I pay a reasonable amount of attention to the YIMBY movement.
Caveats:
– I can’t commit to a time frame, I have spare time but life happens and there’s no point in doing this if it’s rushed (for example, I’m happy to read a few books along the way).
– I live in Melbourne, Australia – it’s a city where the American YIMBY arguments still have plenty of relevance, but there are some key differences too. I’m less interested in topics that are uniquely Californian.
Let me know if you’re interested!
Along with these results it would be great to see a writeup of the Franco-German history project, which managed to produce an account of WWII that satisfied both sides.
That doesn’t seem all that difficult; an account of WWI that satisfied both sides would seem to be the big challenge.
this project?
3 volumes: post-1945 (2006); 1815–1945 (2008); antiquity–1815 (2011, 2013)
I’m a little worried that either the first or last article will have a moderate advantage just positionally. (The first because it has the advantage of novelty, the last because it’s closest, both in time and page-proximity to the poll.) I doubt this will dominate quality, but it still may skew things slightly.
The only way to avoid that would be to divide the readers into 4! equal groups and have each one read the collaborations in a different order.
Or just randomize order for ex ante fairness.
Skewing things randomly is no better than skewing things systematically.
I’m looking forward to reading this… especially since three out of four seem so bloody obvious to me that I really want to see someone putting up a reasonable defense of the other side.
I agree, yet which three of the four are obvious?
Scott – great poll would be a pre/post reading question for the audience.
Example: Are mandatory vaccines good?
Answer before, answer after. (Yes/no)
Confidence in your answer (1-10, 1-10)
And correlation between who found which questions obvious.
More evidence for nomative determinism in entry #2! I wonder if Christian will be on the pro- or anti- side….
Well if we’re talking nominative determinism (adoptive nominative determinism, presumably) then I hope that Christian’s adversary in that collaboration does not make an ass of himself.
And I hope the transgender children one doesn’t turn into a flame war.
Bazing!
*rest of comment removed because it concerns an individual entry*
Safe to say Team #3 will hit the mark.
Lol
Please use approval or ranked choice voting for the poll. We don’t want any vote-splitting after all :-). Approval is easier to administer, understand, and explain, but RCV is more natural for voters to actually use (hint hint). https://www.opavote.com/ can do either.
Strongly agree.
I can’t recall what it’s called, but wouldn’t the best method be one often used in debates, where people select which position they supported before-hand, then select which one they support after the fact, and the winner is whoever “changes the most minds”
I strongly agree with this too.
This isn’t a debate. Changing minds is not the intention.
There’s a value in learning the nuances of a situation, even if it doesn’t change your overall opinion.
Plus, it incentives controversial collaborations more than we might want.
Though if there’s going to be a survey, it’d be interesting to to look at this kind of question as a secondary consideration.
Ah, fair enough. I got a little confused when we started talking about voting options.
That said, I do think that when it comes to controversial topics, changed minds is indicative of effective “learning” having taken place, even if that isn’t the stated and explicit goal of such an exercise.
I think a lot of learning happens outside of changed minds. You can learn new categories, new distinctions, important tensions, potential tradeoffs, a framework for thinking through your choices. And there is also learning that happens in passing from low confidence to higher confidence on issues. And why should we make the assumption that everyone has an existing view on every controversial topic?
This is more generally one way that philosophers might try to explain what philosophical progress looks like, even though people are fighting about the same things they have been for hundreds or thousands of years. (See the bottom of page 10 of this really interesting piece by David Chalmers: http://consc.net/papers/progress.pdf)
I like the idea of surveying the readers about their prior and posterior beliefs and confidence levels are about some questions related to each subject. But that would be excessively complicated to do, since Google Forms does not make it easy to track users between sessions.
Either approval or RCV is better than choose-one voting, but Star Voting (which can be administrated at star.vote) is better than either. Unlike approval voting, it allows fine-grained distinctions, and is strategy-free in any election where the two strongest candidates are clear (which, by the Gibbard-Satterthwaite theorem, is as strategy-free as possible). Unlike RCV, it’s immune to center-squeeze (a problem which RCV likely suffers in cases where options can be meaningfully placed on a 1-dimensional spectrum).
My partner and I were not expecting a democratic vote to determine the winner of the contest, and we were working under the assumption that Scott and a few of his closest would arbitrate. We worry our collaboration might be longer than even many SSC readers would have time to read. Ours is about the length of Meditations on Moloch.
Earlier Scott said:
.
Obviously editing for a small panel is different from editing for a large electing audience who has to read maybe 80 pages of guest contributions and take the time to weigh their vote. We are worried that the contest will be about who wrote the best SSC post, as opposed to best “adversarial collaboration” whatever that might mean. This is the same surprise for all parties, yet it is still true that had we known the process would be more democratic our editing process would have been different.
Do any of these concerns ring true to other collaborators?
Yes. I’m not unhappy about seeing the SSC audience vote for the essays as part of the process. It’s always interesting to see how public opinion turns. But you’re spot on with the ways it changes the dynamic, and I was a bit surprised to see how much sway the audience vote has here.
Honestly, though, the prize I was most excited about was the chance to be featured on SSC and discuss our ideas with a wider audience, so my uncertainty about the vagaries of an audience vote was outweighed pretty heavily upon seeing that all the essays would be featured.
Would it allay your concerns to say only vote if you’ve read everything?
What about to make my vote count for 10%?
What about to have two awards, one for my favorite and one for the readership’s favorite?
Which one would you prefer?
Being featured on SSC is great! I hope the goodness of the fruits incentivizes more adversarial collaborations in the future. We loved the process.
Sadly though, my motivation for writing is also mercenary, so second prize would mean a lot. I’ll circle back and confirm with my cowriter that he’s okay with that solution.
Christian Flanery agreed that a second prize is preferable, since it increases the chances of formal prize recognition for everyone.
I agree with JohnBuridan and Christian Flanery that a second prize is a good idea.
I’ll ask my adversary-collaborator flame7926 about it.
I’m afraid there will be very few voters in this case (our collaboration has 13 pages).
I also liked Jacob’s proposal:
flame7926 agreed (by mail):
I’m finding it helpful to go back and read the original threads to see what the collaborators’ positions and expectations were. In the spirit of “preregistering” and noting places where the evidence didn’t support the hypothesis, maybe that stuff should be documented somewhere?
Items 1-4 would be very convenient if they were links.
(I know that they weren’t available yet at time of posting, but now that they are, or at least most of them are….)
Seconded. It would be very helpful for sharing via social media.
It’s been interesting to read the first 3 essays, and the comments. I do wish the comments on the first essay hadn’t established a norm of making nasty comments about those people the reader/commenter disagreed with, including unnamed groups of prior responders, or at least that I hadn’t succumbed to temptation and conformed to that norm rather enthusiastically in the comment thread on the third essay 🙁 I’m a bit nervous about reading the comments on the fourth one, as I think it probably has the most potential to draw strong and nasty opinions, and I want to avoid being tempted into responding in kind.
I’ll probably vote for the one I find to have the best combination of referring to peer reviewed research (or established facts), considering all relevant issues, and producing a sensible outcome. That may mean it’ll be whichever best conforms to my own priors 😉 but I’ll try to avoid that.
My other considerations will be whether, IMO, it answers its own question, and whether the question as posed is worth answering. (An answer of “needs more research, of the following kind(s)”, would be acceptable to me. What I wouldn’t vote for would be a broad question being asked, and a narrower question actually being addressed.)
I might have a short attention span, but I couldn’t figure out the end result of the studies, expressed in a real simple “yes/no” answer.
It’s very hard to write as engagingly as Scott does.