Adversarial Collaboration Contest Results

Thanks to everyone who participated or voted in the adversarial collaboration contest. The winners are:

Grand Prize ($1000): Does The Education System Adequately Serve Advanced Students?

Editor’s Choice ($500): Should Transgender Children Transition?

Honorable Mentions ($250): Should Childhood Vaccination Be Mandatory?, Are Islam And Liberal Democracy Compatible?

I’m sorry for jerking the number and value of the prizes around so many times, but I wanted to balance my preferences, the contestants’ preferences, and readers’ preferences – and this was the best way I could think of to do it. Nobody has gotten less money than they expected, although some prize categories have gotten more money than I originally said. In the end I could not in good conscience let any of these escape without getting a prize. Thanks to this blog’s Patreon supporters for making this possible. All winners should email me with their preferred form of payment (I can do Paypal, Bitcoin, or donations to a charity of their choice).

The overwhelming winner of the popular vote was the collaboration on education. I agree this one was excellent. It cited a lot of research, analyzed it very well, and mostly came to conclusions. Its only flaw from my perspective was a lack of focus; it discussed many different educational interventions, some of which were similar enough that it was hard for me to keep track of what was going on.

I chose the collaboration on transgender children. I thought it did an exceptional job of addressing a specific hot-button issue many people are concerned about, presenting all the evidence on both sides, and mostly coming to conclusions. My strongest complaint was that it ignored some of the potential side effects of puberty blockers which commenters pointed out, and sort of trivialized bone problems that are not trivial; given that the side effects of puberty blockers was a major crux of this question, I found that to be a major weakness. I was still very impressed with the piece’s ability to break down and navigate such a controversial question.

The first honorable mention, the collaboration on childhood vaccination, was also a great example of navigating a controversial topic. It seemed to come closest to outright saying one of the two sides was correct – though see the commentary below for more on that. I had to take off points because the conclusion – that lots of countries get away without having mandatory childhood vaccinations, so surely America can too – seemed overly simplistic. Those other countries might maintain high childhood vaccination rates because of cultural differences, or a different (eg socialized) health system, or lots of other things that might not generalize to the US. Again, this was the crux of the issue, so minor flaws here are very important. The section on the hygiene hypothesis, while fascinating, seemed kind of unfocused to me and probably could have been much shorter.

The second honorable mention, the collaboration on Islam and democracy, was maybe the most thoroughly researched, but also seemed the least structured. The writers chose to list democratic and undemocratic features of six Islamic countries. Since they found one Islamic country (Tunisia) that seemed pretty much like a liberal democracy, they concluded the two were compatible. But in real life, when people ask whether Islam and democracy are compatible, I imagine them wondering things like “Are Muslim countries less likely to be democratic than other countries?” or “Is the Muslim religion one reason why there is so little democracy in the Middle East?” or “In mixed countries, do Muslim populations resist democratic norms?” The collaboration’s focus on listing the particular features of particular Muslim countries’ institutions seemed like a really complete answer to a really boring question.

I asked the various collaborators to send me their reports on the process. Their answers are below, or scroll down to the end of the blockquotes for my summary.

TracingWoodgrains from the education collaboration:

My adversarial collaboration was similar to how Kahneman describes his experience (https://pastebin.com/d6cbN765): a “failure to disagree.” When Michael and I zoomed into an issue at the object level, we usually agreed. Then we’d zoom out and realize we used that object-level detail as part of two drastically different narratives. I have no education credentials besides having been a deeply frustrated student; he was a happy student and now enjoys teaching. That dichotomy between teacher and student, happy and unhappy, defined our perspectives.

Going in, I was confident that tailored curriculum involving ability grouping could help advanced students progress up to 4-5 grade levels above where they are now, that school is a deeply harmful place for many advanced students academically and socially, and that education should follow the principles of game design (using online tools to allow students to progress at their own paces while encouraging them to progress quickly).

Our project was extensive and far-reaching, mostly due to our disagreement and Michael’s tireless willingness to keep working with me. We approached it as a research project more than anything else, which mostly meant “toss all relevant-looking research and related writing towards each other, read it manically, and see where and how disagreements bubbled up as we were learning.” At the start, I was coming from a position of near-complete ignorance towards the state of specific research on ability grouping, though I had passing familiarity with acceleration research and what I felt was a comfortable understanding of the (poor) state of research around online education. So a lot of my “position-shifting” was learning:

I learned that tracking, in a traditional sense, had much weaker evidence than I was anticipating, but the more it looked like what my intuitive view of its ideal, the better it fared. So: few peer effects, strong effects based on curriculum change, strongest impact based on diagnostic testing and precisely tailored curriculum. My views on acceleration as a useful, underfocused stopgap measure did not change. My understanding of the complexities of the education system, and why it was so far from what I hoped to see, grew tenfold. Michael’s pressing reminders of the competing purposes of education and experience in the field helped me a lot there.

That’s most of the small stuff. I had two genuinely paradigm-shifting experiences: First, on the happiness of advanced students. Typical mind here–I’d always just assumed that most advanced students were bored, held back, and miserable in school. It seemed one of the most obvious things in the world to me. But it’s really, really hard to find any of that in the research. Most studies indicate them enjoying school more than other students, (tautologically) performing better, and having lower incidence of mental health issues. My view shifted to intelligence as a potential intensifier for other risk factors (ADHD, autism, etc.), but not as the core cause of student frustration in school. My belief that the “smart kids” who are bored and miserable in school tend to be underprioritized and mis-served remains largely unchanged, but I have a very different view of this group.

Second: Michael pointed me towards noticing the apparent mediocrity of a lot of schools implementing some of the digital/in-person hybrid instruction I dreamed of. Rocketship Schools, School of One, Khan Academy, and others grounded my expectations of any sort of revolution. At the same time, we discovered Art of Problem Solving, which is probably the best thing in the world, and seeing what they’ve done with online education and specifically math made me realize that people smarter than me have been doing a really good job on “my” project. So now I see less need of creating the right resources, more of expanding their availability and sifting them out from the sea of garbage.

Less paradigm-shifting, but significant, is that I grew to trust experts more as I noticed that when you dug deep enough, most of the serious researchers came to pretty similar conclusions but wrapped them in different narratives, and really bad ideas that filtered through were less because the serious researchers had really bad ideas, more because their messages got distorted or ignored to better fit political agendas and the vagaries of people’s opinions.

(Bonus areas where I didn’t change my view, just properly grew one because I didn’t know enough to have a view before: direct instruction as intriguing and mostly good, Joplin plan and nongraded schools as intriguing and oddly ignored, early childhood education as incredibly high potential in specific fields, a growing frustration with just how much politics has infected and damaged education)

One example best outlines our differences: Direct Instruction. Michael actually brought it up to highlight problems he saw with online schools–basically, pointing out that it works, but isn’t attractive at all, except in educational emergencies. I hadn’t heard of it before and got about as far as “it works” before getting really fascinated by its whole story. I looked at schools that use it, interviewed someone closely involved with it, read into it, and concluded that it had a lot of potential that was being neglected since people felt like it wasn’t how they “should” learn. That whole arc was never supposed to happen. It was an offhand example that he raised to dismiss. And my response was basically “this is amazing; how have I never heard of this before?”

That happened again and again–where I would see an unusual, academically intensive approach that got good results, we’d talk about it, and he’d ask, “right, but what’s the point?” Sorting by aptitude over age level, high-intensity accelerated math programs, early (pre-K) academics, so forth. It’s worth repeating–we almost never disagreed about what studies showed, just on the importance of particular studies and particular points. And that was enough to fuel a hundred pages or so of disagreement.

So, advice for future adversarial collaborations: First off, it can only work if both are very, very willing to talk about the topic–the whole topic, not only their pet issues within it. If you find yourself uninterested in a part of the topic the other person is passionate about, pay close attention, since that’s often the most important part of disagreement. Second, it takes a lot of time. We took more time than strictly needed because we both enjoyed the research and conversations, but I’d guess 50-100 hours is a reasonable amount of time to set aside. Third–expect to find a lot of disagreements that boil down to differing priorities and interpretations of object-level facts you both agree on. If you’re both reasonable and willing to work together, you may find little factual ground you disagree on, even while telling two very different stories of the big picture.

One last thing: Adversarial collaborations are a phenomenal thinking tool. This was the most satisfying project I’ve worked on, bar none, and the best learning experience of my life. Talking in depth with someone who disagrees with you is incredible. If both agree, it’s hard to sustain a conversation, but the second someone says something wrong-seeming, it spurs a sort of need to respond, a creative burst. I read more research and focused more on that topic than I have at any other point, even though education has been one of my core interests for a while. Michael was the best partner I could have asked for on the project. I expect that we’ll still talk about wildly different things moving forward: I still love the potential in online learning, non-graded schools, and accelerated/intensive tools that depart dramatically from the current school model. Michael still loves teaching and working within the system mostly as it stands, and doesn’t trust most radical changes.

Thank you very much for organizing this contest. It was an incredible opportunity and a vital experience.

Michael from the same education collaboration:

I really want to give you something really detailed and good about our collaboration, but that would take a lot of work. Instead I’ll just give you the quick, stream of consciousness version.

When I first saw TW’s comment on your post, I thought his views on education were absolutely naive. I heard him expressing a love for game-based education, which I thought to be faddish and flashy in education but without much value. He also expressed that the strongest students were especially poorly-served by education. I thought that was dead wrong; smart kids would do fine, in general, out of school. Without checking, I can’t remember if he said anything about ability grouping. That wasn’t the point, anyway. What I heard was (sigh) another technophile education utopian who thinks that their own particular experience should be the basis of a system tasked with educating millions of children.

So we started chatting, and pretty soon we’d created an 150 page google doc. The doc contained our debates, annotated bibliographies, brief memoirs documenting our own experiences in education, etc.

I’m coming from all of this as a guy who was pretty conventionally successful in my academic life, and ended up in teaching almost by accident. In my professional life I’ve given a lot of attention to students who aren’t succeeding in my classes. Those are the situations that have made me the most passionate to put in extra efforts to help them. So things like volunteering to tutor a kid outside of class, to go over homework during my planning period, rethinking my pedagogical approaches.

At the same time, I had recently experienced a few cases in my honors elementary age classes that were making me think. And I was going back to work at a camp for talented math students.

So I’d say the real thing that changed me, through this experience, is I spent a lot of time thinking about students who find a class too easy, and what that can feel like. That was the most significant way that my views changed through working with TW. He consistently drew my attention towards students who are talented but unhappy.

I feel as if with what we wrote about technology, my views didn’t significantly develop through our collaboration. I think TW quickly learned that his tech optimism is future-optimism, not present-optimism, and that’s a change for him. But otherwise what we did there was more along the lines of finding a common narrative that we both thought important for people to understand the situation with technology and learning. That involved a lot of back and forth between us, as it’s hard to put everything in a way that satisfies both a pessimist (me) and an optimist (him). That’s how we conceived of this work, though.

A lot of the other things we did were more about fleshing out new conceptual territory than changing my views. I didn’t really have strong opinions about tracking or ability grouping. I was eager to learn something new, and TW was a great partner for working through the literature.

There was a lot of debate during this project, and we still disagree deeply. It’s hard to say who “won” our collaboration or who pushed who further etc. I’ve been reading about a lot of this stuff over the past 8 years. I don’t know how old TW is but he’s younger than me. I certainly urged him to read certain people who I thought were foundational, like Larry Cuban or David Labaree. A lot of this was very new to me though, and I was trying to integrate my experience in the classroom with my reading. I feel as if my strong views on gamification were quickly confirmed, and I also think that my view that smart kids mostly turn out alright were vindicated too. I’m sure TW has complementary ways in which he feels he shifted my views, though I’m not exactly sure what he’d say.

One thing that I think people haven’t given enough thought to is the differences between adversarial collaborations in different fields. Certainly there is a difference between medical research, economics research, comparative politics and education research. Each field has a different epistemology, and education research is a big tent containing many individual subfields. That means that TW and I had to do something that I love doing, which is figuring out how cog psych and big metanalysis and economic studies and case studies and history of education all fit together in a coherent whole.

So while I can imagine a briefer more focused write up for our collaboration, I think it’s possible that making sense of classroom questions is going to involve a lot more fuzziness and wandering than for certain other questions. We can’t just easily answer the question of whether top students are harmed by schooling. Compared to what? What’s the vision? What’s the purpose of schooling? That’s a question you don’t really need to ask about e.g. vaccines. The goal is to keep people alive without harming anybody else. I know there is some fuzziness there, but it’s just not the same magnitude.

One more thing, before I call it a night: by design, TW were non-adversarial in our collaboration. Early on, I sent TW an email where I said, you know what? We’ve figured out that we deeply disagree. I’m not interested in tallying points on whether my beliefs have been weakened or changed or your’s have. Instead, let’s just try to figure things out together. And that’s what we did.

I think a major question going forward is whether adversarial collaborations work best when you have an opponent or when things are framed so as to reduce opposition.

John and Christian from the Islam collaboration:

WHAT WERE YOUR INITIAL POSITIONS?
JohnBuridan’s initial position was that it was highly unlikely that Islam consistently opposed liberal democracy. ChristianFlanery’s position was that intrinsic components of Islam operate contrary to liberal democracy.

HOW MUCH DID YOUR POSITION SHIFT, AND HOW?

JohnBuridan’s position did not shift towards the other side of the aisle; he still think the opposing position holds too large a generalization. He did shift his approach to the question from a theoretical approach to a more descriptive one. ChristianFlanery convinced JB that the logic of surveying the variability in Islamic political thought would require immense time and resources. It would end up requiring us to determine what different groups of Muslim thinkers and activists think of each other. CF convinced JB that a descriptive and historical framework would come closer to providing insights into how the diversity of Islamic political thought caches out in real modern day politics.

CF became more conscious about how new Islam’s interactions with liberal democracy are. This made him less confident in his initial position. Furthermore, CF now has high confidence that liberal democratic polities can be achieved in the right conditions. CF continues to believe that a fundamentalist component in Islam will always exist, but he is now agnostic about whether Muslim populations will be experimenting with liberal democracy well into the future.

HOW MUCH DEBATE AND ARGUMENT WAS THERE?

Constant unceasing debate, but in a context of high levels of trust. We did not fall victim to one-upsmanship or point scoring debates. Preexisting intellectual rapport allowed us to trust each other not to load the dice too much. Although each of us probably gave too much ground to other person in certain places throughout the paper – many of which the commentariat picked up on – we think that such tradeoffs made the work possible.

The key to resolving debate for us was an established process for the paper. Each had three countries to research and do the write up on, and each of us edited each other’s rough drafts. We wrangled over essentially every line of the introduction but had the patience for very long phone calls to resolve them. Oftentimes we would disagree and circle back to the disagreement a week later and come to a solution. We both agreed beforehand to try to only pick battles we thought worth fighting. Tense moments surrounded our discussion of the introduction, the conclusion, constitution, Lebanon, and Iran. Our longest back-and-forths concerned the introduction and conclusion.

IS THE CONCLUSION CLOSER TO ONE PERSON’S ORIGINAL POSITION?

Our conclusion is closer to the side of commensurability. Nonetheless it is fairly neutral, and we try to leave the door open for the reader’s own knowledge and expertise to affect the conclusion. Our goal was to provide enough background knowledge of some slice of the Islamic world from which a general reader could inform their own conclusions.

– Lots of readers were deeply concerned about how we chose the countries we did. We chose the countries before doing deep research, and we were looking for a few key traits:
1) majority-supermajority Muslim
2) culturally diverse
3) geographically distant from one another.
4) tractable to discuss (we did not feel that we could do Saudi Arabia or Pakistan justice, and we considered the possibility of many other countries.)

WHAT ADVICE WOULD YOU GIVE FUTURE ADVERSARIAL COLLABORATORS?

Our main points were:
1) Establish high levels of trust beforehand. Discuss all sorts of topics and allow for a lot of freewheeling conversation about fundamentals before you buckle down to work.
2) Design a method and framework and develop a clear division of labor.
3) Split the difference wherever possible. We veered towards choosing the least controversial phrasing, but we also believe that a noncontroversial statement said well does a lot of good. Consistently work then discuss.
4) The fact that our topic was broad gave us plenty of thought-space to work with each.
5) Have a partner that is incentivized. We were both were enthusiastic about the Ad Collab format, confident we could make a contribution to the field of our topic, and the possibility of money was meaningful.
6) We both had time in our schedules. It was summer, and we both worked in academic and educational institutions at the time.

From Mark W, who wrote the pro-vaccine side of the vaccination collaboration:

WHAT WERE YOUR INITIAL POSITIONS?

My initial position was that herd immunity is a compelling government interest, and that it is well within the power of a reasonable government to implement a mandatory policy in order to achieve this end.

Interestingly, the first part of this statement (that herd immunity is a compelling government interest) was not challenged by my collaborator, so I think we spent less time making the case in favor of herd immunity than we probably should have. This was likely a mistake on my part, as I should have been spending more time shoring up my side – even those parts we agreed on – as this was one of the areas some commentators found lacking.

(Of course, if you are adversarially collaborating in bad faith, you could choose to agree quickly and dismiss significant parts of your opponent’s argument so as to focus less energy defending their strongest points. I’m not saying my collaborator did that; I think he was arguing in good faith. I’m just saying that you could…)

On the second point, Mark Davis began by saying that he believes vaccines are a major causal factor in autoimmune and autoinflammatory disease. I responded that we looked at that angle back in grad school (about 10 years ago for me) and my recollection of the evidence was that vaccines are not a good explanation for this phenomenon.

HOW MUCH DID YOUR POSITIONS SHIFT, AND IN WHAT WAYS?

My position that mandatory vaccinations is good policy shifted to the extent that we showed it’s not required to achieve herd immunity. I would say that as a matter of public policy influence, the burden falls on the side opposed to mandates to persuade the public that there’s a better way. On the side of vaccines causing autoimmunity, I was not swayed much. I think it’s possible that mycobacterial vaccines contribute a little, but the problem is that there are too many types of mycobacteria, and families of protective/sensitizing commensals are so broad that a highly-specific intervention such as vaccination is unlikely to be able to wipe out a whole family like that and leave you sensitized to developing autoimmunity. Mark Davis wanted to put forward the hypothesis that not only are vaccines bad (because, he believes, they cause autoimmunity) but that the bad outweighs the good they otherwise cause. I felt that one point we made in the essay was that even if it turns out that vaccines contribute significantly to autoimmunity (a claim I don’t find much evidence for) he is presenting a false choice. The public health implications are not either disease control and unchecked autoimmunity, or unchecked disease and autoimmunity control. There are populations that don’t have disease exposure and that don’t have autoimmunity, and there’s no good reason to jettison vaccines and the positive good that they bring (which we both agreed on) out of a belief that otherwise we have to live with autoimmunity.

There’s one point that didn’t make it into the final essay – due to time constraints and narrative flow – I wanted to add. If you believe that autoimmunity is prevented by exposure to pathogens that are vaccinated against, this has dire ethical implications. It’s one thing to believe that the measles vaccine caused your child to get asthma. Even if you’re wrong and unpersuadable, your proposed solution will be to not vaccinate your child. If you believe that a lack of disease exposure caused your child to get asthma, you’ll want to actively work to ensure that all your children contract each disease you refuse to vaccinate against. Otherwise, it makes no sense to not vaccinate against something, if you think the causal mechanism requires exposure, too. Thus, there is an ethical problem here, not with allowing parents to opt-out of vaccination, but with allowing parents to actively seek to spread disease through their communities. If a parent drives around with an 18-month-old in the back seat with no seatbelt, then intentionally gets into a car wreck, they would be criminally liable – even if they were only caught attempting such a feat. There’s a good case to be made that intentionally infecting your child with a disease known to cause morbidity and/or mortality should carry criminal liability. Davis didn’t make this exact case, but it’s really the only natural extension of his claim. If you oppose vaccination on the grounds that you believe it’s necessary to experience disease exposure in order to protect against autoimmunity, you have to be willing to expose your children to disease.

HOW MUCH DEBATE AND ARGUMENT WAS THERE?

Most of the debate was about whether vaccines can lead to autoimmunity. This was the driving factor behind our time delays. We went through multiple drafts surrounding this point as a main focus of conflict. Davis sent a couple of articles suggesting some causal link between vaccines and autoimmunity. I sent back about a dozen more that argued causal link, no effect, and protection. I also pointed out that the review he sent me actively argued against his point, as did every other review on the subject I could find and multiple meta-analyses.

HOW DID YOU RESOLVE IT?

Davis maintained that this body of evidence was not sufficiently strong to persuade him. At the outset, we had agreed that the burden of proof lay with his side in defending an active harm perpetrated by vaccination. Given that the evidence trended in the opposite direction, he conceded to allow language that public policy should not discourage vaccination – on the condition that private individuals be enabled to disagree with public consensus.

IS THE CONCLUSION CLOSER TO ONE PERSON’S ORIGINAL POSITION?

I’m not sure. I’d say the first part (about mandatory vaccination) preserves my stipulation that elimination of mandates is only okay insofar as the objective of herd immunity is maintained. Thus, even though I capitulated, I did so within the realm of broader public policy objectives I believe in. For the second part, we probably stuck closer to my side, in that there was simply no way to advise public policy changes based on weak evidence of hypothesized harms. That said, there were lots of caveats and hedging.

WHAT ADVICE WOULD YOU GIVE FUTURE ADVERSARIAL COLLABORATORS?

Some thoughts on my experience:

– Time was definitely a factor. Repeat back-and-forth drafts took the most time, because one person would usually have it for a week or more before turning it back to the other person.
That means at least two weeks per draft. If something like this were to become a regular feature, I would advise future collaborators to allow extra time, and maybe wait to submit in a future round. – Every time I tried to add a strong statement, such as “we have no evidence to support [X]” “there is no reason to believe [X] and ample reason to believe [Y] instead” he would object. I suspect there’s a strong tendency as a collaborator to not capitulate your side.
– That said; I did end up capitulating on part of my side (the mandate part) because we stated at the outset under what conditions we would do so. My conditions were met (other countries without mandates achieve herd immunity, so clearly it’s not an absolute requirement, and some countries with mandates do not achieve herd immunity, so clearly it’s not sufficient), so I conceded that point.
– My collaborator stopped there, however, so the most we could recommend was, “sure they do things differently elsewhere, but we don’t really know what that entails; go look it up yourself.”
– We spent most of our time on the particularly contentious second part of the collaboration. This may have been why the first part was weaker than it could have been.
– The assigned titles tended to focus public discussion on what they thought was the main debate, but often that was either not the whole debate, or the debate had shifted as a result of the collaboration.

If you choose to do this again in the future, I would recommend a few changes:
– Have collaborators choose titles.
– Require collaborators to stake out one or two specific claims they wish to refute/defend a priori, and have that be outlined in the first part of the final submission. I know we mostly did this with the bolded statements, but not all of the positions made it into the final piece, so it was hard to tell where people were coming from.
– Have the collaborators include conditions that would be necessary for them to concede their positions. I feel like it would be within the role of a moderator to remand any concessions that are unrealistic, or not well-matched to the claim. (i.e. “I will be convinced once 1,000 years’ of RCTs have been completed.”)
– Require a concluding section that states whether the conditions were met, rejected, or uncertain – as well as what should happen moving forward.
– Where possible, include a third, impartial, moderating collaborator. This is theoretically the person with no horse in the race. If there’s disagreement, they can say to one side, “sure, but you’re not arguing based on the evidence anymore, you’ve been bested on that front; we need to draft stronger language that reflects the actual state of the evidence.”
– The SSC community is a great resource. In retrospect, it would have been nice to have used part of an open thread to discuss elements of the ACC. For example, asking an
international audience about what their on-the-ground vaccination experience is like in the UK/Ireland/France/etc. I should have done this unofficially, but it would also have been helpful to have an official, “first comment is for ACC contributors to ask the community for input”. That would also allow the rest of the community to contribute to the contest.

From a_reader, who wrote the anti-transition side of the transgender collaboration:

WHAT WERE YOUR INITIAL POSITIONS?

My position was that “transgender children” (gender dysphoric children) shouldn’t transition in childhood – neither socially nor by suppressing puberty at 12 (or even earlier!)

Because a lot of them – percent differs among sources, but something between 30-90% – desist, cease to feel and act like the opposite sex. And that’s a better outcome: not needing surgery and hormones for the entire life, having less problems in society acceptance (even as gays/lesbians, as most desisters become) and in finding partners.

If dysphoria persists from 3 to 18, that’s it, transition may be necessary, but it’s better if it’s not necessary, if the problem ceases spontaneously.

Social transition in childhood (before adolescence) – that means name, pronouns, hairstyle, clothes of the desired gender – makes desisting more difficult. But puberty blockers at 12, I thought, can do worse: prevent the desistance from happening, stopping exactly the natural process that may have cured gender dysphoria, by the action of puberty hormones on the brain.

HOW MUCH DID YOUR POSITIONS SHIFT?

My position didn’t shift that much, but now I understand better why so many parents choose to let their children transition and take puberty blockers, how distressful gender dysphoria is for those children – anxiety, depressing symptoms, sometimes (at puberty) even suicide attempts – and social transition and even puberty blockers seem to alleviate it substantially.

I still consider (spontaneous) desistance a better outcome. And I still tend to think that maybe puberty blockers may stop that process. But I admit I didn’t found conclusive evidence (only circumstantial: all children on puberty blockers persist).

HOW MUCH DEBATE AND ARGUMENT WAS THERE?

Not much. In the beginning, flame7926 created a document shared by Google Docs, for documentation, where both of us wrote about the studies (and media articles and blog posts describing cases) we found. One found a study and wrote a few lines about it, then the other added their own observations – using different colors each of us at different times, so that in the end that document looked like a rainbow, matching the subject.

HOW DID YOU RESOLVE IT?

After the documentation stage, flame7926 asked me if I changed my position, I said not really, so after a short discussion, we agreed to not recommend for or against childhood transition, to just present all the evidence, the risks and benefits.

IS THE CONCLUSION CLOSER TO ONE PERSON’S POSITIONS?

The conclusion seems to be that there isn’t yet enough evidence for a firm, definitive conclusion. So it tries to keep a balance between our positions, resuming the evidence and admitting uncertainties.

WHAT ADVICE WOULD YOU GIVE FUTURE ADVERSARIAL COLLABORATORS?

I think besides conscientiousness, you need a degree of agreeableness and being a mistake theorist, not a conflict theorist (at least not about the subject); it is also important to have enough time (that is, quite a lot) and to find the subject interesting (but not essential, not among your main values – otherwise you risk to become a conflict theorist).

I was really lucky to have such a good “adversary” like flame7926, who in the end worked more than me: he wrote most of the final entry (as you probably have already noticed, I’m not a native English speaker). Seeing how many people were abandoned by their partners, I realize how lucky I was.

And from flame7926, who wrote the pro-transition side of the same collaboration:

I enjoyed the adversarial collaboration process and like the idea a lot. I think that it has the potential to produce very productive discussions and papers. I think it definitely can moderate individuals’ opinions – get them to recognize that many things aren’t so black and white.

To begin, I thought that transgender children should be allowed whatever options they want in terms of socially transitioning, while I had less strong of an opinion on puberty blockers (mostly because I didn’t know much about them). Honestly, I didn’t know that much about gender dysphoria or transgender expression in children and youth before commencing with this project, so my opinion wasn’t very formed. I did feel strongly though that we not present “desistence” as a preferred option because I think it is at least reminiscent of transphobic and homophobic attempts to convert individuals away from their identity – an often very traumatic and distressing process. And I thought that once you remove the idea that desistence is preferred (because I think it is very difficult to have that macro-level preference while remaining supportive of individual children on a micro-level) then social transitioning becomes preferred. I also thought that these cases of “desistence” might just be people expressing cross-gender behavior and not actual transgender identities.

I think my position moderated some towards a more even approach, and I can understand why parents and others would want children to desist from transgender identities (as long as this happens without trauma). I also stand stronger behind current research on desistence, in the sense that I do think there is strong evidence that a sizable fraction of children who express desire to either be the other gender or that they are the other gender will not express those same desires later in life. Yet I still think (and believe the evidence shows) that social support of children is extremely important (in terms of mental health), and I think denying their desire to socially transition can run counter to that. I also think the example of Samoa and other societies shows that gender dysphoric feelings are not constrained to our culture, but that the mental health associations are culturally contingent.

I would say there was some debate and argument. As a_reader describes in their response, we had a collaborative Google Doc broken down by topic, like social transitioning, puberty blockers, mental health etc. One person would pull some evidence from an article, the other person might pull some contrary evidence. Debate proceeded in that fashion, with more and more sources being brought in. There wasn’t that much overarching debate, but I think our topic was broad enough that it needed to be broken down in components in which we actually examined the evidence.

When it came time to write our paper, it seemed that neither of our positions had shifted that much. We presented the evidence, but I don’t believe there is currently enough evidence on puberty blockers to make a recommendation either way. Given that we did not end up agreeing on social transitioning, we could not recommend either way. I did end up writing most of the paper (at the request of my co-author) and am glad that we were able to produce something that moderated and expressed both viewpoints. I’m also proud of how we integrated quantitative evidence with more qualitative sources and anecdotal accounts, which I also feel are important when dealing with issues that are, necessarily, subjective.

In terms of advice, I think just being able to put in the time is important. Seeing the small number of projects that were completed, I wonder why so many dropped out. I’m glad I had such a motivated partner (I would say that they did more work than I did). I think that it is also important to be able to moderate your viewpoints and understand where your disagreements come from, even if you don’t end up reaching the same conclusion. I think that having someone else working can keep you on track, if you see them making process. I am curious how other groups divided up the writing on their projects (especially since I believe that most of them ended up longer than ours?).

My executive summary: almost no one changed their mind on the overall issue (a few people changed on smaller subpoints), but almost everyone moderated their opinion at least a little.

Some of the collaborations raised concerns I hadn’t considered. Although John and Christian were happy with their result and many other people liked their collaboration, I still can’t shake the feeling that they solved their intractable disagreement by avoiding their actual cruxes and searching under a streetlight they didn’t lose their keys near. If you force two people to write a mutually agreeable essay on a topic they can’t agree on, it’s probably hard to prevent them from inadvertently shifting to a slightly different topic where everything is cut and dried and factual and agreement is easy.

I was originally very excited to see the vaccine collaboration come out strongly in favor of “vaccines do not raise disease risk”, since it looked like someone had really changed their mind on a controversial topic – then disappointed again once I learned that vaccine opponent Mark Davis had not been convinced, but just agreed that the burden of proof was on him and he hadn’t met it by convincing his adversary. If I were a vaccine opponent, I would be pretty upset that the adversarial collaboration comes out strongly in favor of vaccines even though the two collaborators weren’t both on board.

And as a vaccine proponent, I worry that Davis’ magnamity in allowing the final result to be mostly pro-vaccine convinced Webb to “reciprocate” by being less forceful than he could have been in other areas, and not fight back too hard against the technically-true-but-fraught claim that other countries had higher vaccination rates than the US without mandatory vaccination programs. I don’t know if this really happened in this collaboration (from the write-up it looks more like they agreed on the preconditions under which they would give in, and maybe those weren’t thought out too well) but I think the two adversaries agreeing to “compromise” by each giving up on one of their key points, rather than really come to agreement or present their disagreement exhaustively, is a failure mode to be watched for.

This particularly bothered me in the vaccine collaboration because of another factor I hadn’t considered. I previously hoped adversarial collaborations could be a good match for “pseudoscience”, ie questions where the mainstream is overwhelmingly on one side but there are still some holdouts with an alternate narrative. Certainly these topics cry out for explanation, but the current paradigm – RationalWiki-type “skeptics” who make fun of pseudoscience with cheap shots – are really bad at convincing believers and usually mangle the theories they try to criticize. An adversarial collaboration format could make sure that debunkers have to address pseudoscientists’ strongest arguments – and that pseudoscientists’ supposed counterarguments to the debunking get addressed also.

I now worry that such collaborations would systematically end up legitimizing pseudoscience. If nearly everyone believes that some theory (UFOs, Bigfoot, etc) is false, and some prominent proponent and opponent of the theory do a collaboration, then even if everything goes great and the end result is mostly skewed to the opponent’s side, then there will still be a few compromises and places where the opponent gives into a little social pressure to frame things in a less-than-maximally-hostile way. So if the end result comes out 90% anti-Bigfoot – well, believing in Bigfoot 10% is more than most people currently believe in Bigfoot, so most credulous readers would update in a pro-Bigfoot direction. True, any debate between pro- and anti- Bigfoot factions will involve some pro-Bigfoot arguments that you wouldn’t have heard otherwise, but the adversarial collaboration format risks importing them into the final product in a way that makes it seem like they have a stamp of approval. I’m not sure this format is a good match for pseudoscience unless somebody finds a way around this.

Thanks to everyone who participated. I would like to do more with this format and will probably devote at least a similar amount of time and money to it next year; if anyone has ideas to improve the contest, or better ideas for promoting adversarial collaborations than another contest like this one, please let me know.

This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

56 Responses to Adversarial Collaboration Contest Results

  1. bcg says:

    I now worry that such collaborations would systematically end up legitimizing pseudoscience. … there will still be a few compromises and places where the opponent gives into a little social pressure to frame things in a less-than-maximally-hostile way. … the adversarial collaboration format risks importing [some new pro-Bigfoot arguments] into the final product in a way that makes it seem like they have a stamp of approval.

    Perhaps a case for adversarial collaboration.

    This really stuck in my craw. First, pseudoscience is where adversarial collaboration is supposed to shine. Second, this problem seems self-correcting; P(Bigfoot) can only get so high before future adversarial collaborations would cap it at some equilibrium. Third, whose “stamp of approval”? New pro-Bigfoot arguments should update in a pro-Bigfoot direction; that’s what it means for an argument to be pro-Bigfoot. If you mean, “some people will update their priors in a pro-Bigfoot direction, more than I think they should”, then I really have to trust you to know and communicate the difference between pseudoscience and “really strong arguments that devastate my case.”

    Fourth, isn’t this just, like, straight partisanship? Aren’t we guided by the beauty of our weapons, and all that?

  2. yaolilylu says:

    Repeat back-and-forth drafts took the most time, because one person would usually have it for a week or more before turning it back to the other person.
    That means at least two weeks per draft.

    I’m surprised that the collaborators chose to do it this way, a Google Doc with open editing privileges for both parties would solve this problem; both parties can write in the same document at the same time, and suggest edits to what the other has written, and open up discussions under each proposed edit. We did this for our HPMOR translation project, and it worked great. If we had to email a draft back and forth it would never get done.

  3. sustrik says:

    Fascinating how people were able to admit facts but still mostly stuck to their original narrative. It would be interesting to revisit post-modernist thought (at least Umberto Eco with his eternal theme of how narratives form our reality) with these results in mind.

  4. a reader says:

    Thank you, Scott. I am proud and honored that our work is your preferred choice and the prize helps me in a very difficult moment, I just came back from my father’s funeral when I read this announcement. I’ll write you an email in a few days, after I put myself together.

  5. JohnBuridan says:

    Scott has made a strong allegation against our Adversarial Collaboration, one against which I take great umbrage. He called our topic boring. This is a most serious accusation, one to which I feel compelled to respond with my word hoard unlatched. I don’t mind being corrected or being critiqued, for in all these cases there are diligent readers engaged with topic, but to be accused of unleashing a boring topic upon the world – that to me is a poison barb.

    😛

  6. Maxander says:

    Regarding the second-to-last paragraph (“I now worry that such collaborations would systematically end up legitimizing pseudoscience.” and onwards); I suspect this is an inherent risk with subjecting beyond-Overton-window issues to rational debate, and further, an inherent risk with the rationalist project in general. Shutting out taboo views is reasonably idiot-proof, as long as your pesky higher faculties don’t get in the way; engage with a view substantively, give it the presumption of “innocence,” and you’re faced with a reasoning problem that you can just get wrong (and note that this possibility doesn’t go away if the view in question is very very taboo or even very simple- everyone rolls a 1 on an int check occasionally.)

    I’ve observed that in more ideologically “pure” subcultures, rationalists often have a reputation as some pretty terrible things- racists, historical revisionists, nihilists, etc- and I suspect that it comes from the minority of rationalists who made mistakes when substantively engaging with various taboo issues and found themselves forced, by their own commitment to rationality, to take up elements of the “wrong” side’s view. (Or maybe they did it correctly and everyone else is wrong- but refer back to the “trust expert opinion -vs- trust your own reasoning” debate from awhile ago.) I don’t have suggestions for solutions to this problem, I just want to point out its importance/magnitude.

    • sclmlw says:

      Maybe the more important question about platform-provided influence is, “Does it matter?” Scott is worried that, say, a movement with 5% support is bolstered to 10% support because it now has a platform in which it is soundly defeated, but shown a modicum of respect. Meanwhile, what about that other 90%? The problem with avoiding certain topics as too taboo to give a platform to is that you cede the debate to a private, unexpected conversation somewhere – a situation that by definition is not likely to have any features of balanced and informative debate.

      I don’t think Scott is concerned that a majority of people are going to be convinced by the side of an argument that’s less supported. He seems more concerned that some small percent are going to be persuaded in the wrong direction. So what? Should a slightly larger but still powerless minority dissuade the rest of us from seeking a solid understanding of complex subjects?

      If your objective is to prevent legitimizing pseudoscience, I don’t see the problem. If your objective is to stamp out all pseudoscience, this format would probably not be well-suited to that task.

  7. Jiro says:

    We chose the countries before doing deep research, and we were looking for a few key traits:

    … 4) tractable to discuss

    This strikes me as a lot like looking under the streetlight for your keys because it’s brighter there.

    • brmic says:

      This impression is however wrong.
      Answering the tractable question among others (i) goes someway towards answering the original question (unlike the key example) and (ii) provides a calibration sample. That is to say, you’ll find out whether the things you considered tractable are in fact so, whether this subset already answers your question and at the very least, you have a better estimate of the search field involved and the time required even if you do find out selecting the tractable stuff was a poor decision. (Correspondingly finding you hit your head several times and slipped in your own vomit which you then had to sieve to search under the lamppost presumably also tells you that continuing the search under less favourable conditions is a bad idea. Maybe you need to come back tomorrow or call a friend.)

      • Jiro says:

        Answering the tractable question among others (i) goes someway towards answering the original question (unlike the key example)

        It doesn’t literally provide no information like the key example, but it certainly provides less information. You need to test the most relevant examples–not the easiest-to-test examples–if you really want to answer the question.

        whether this subset already answers your question

        If your question is about Islamic countries in general, the subset can’t answer the question unless it’s sufficiently similar to Islamic countries in general. And you really can’t show that. You can handwave it by claiming they are similar cultures or whatever, but you have no way to actually determine that aside from looking at them and saying “I think they must be similar enough”.

  8. fr8train_ssc says:

    Congratulations to all the collaborators. I agree with Scott that all were high quality, even if my preference did not win the grand prize.

    Idea for improving the next competition: While it would take more effort, maybe a pre-screen or have an initial round where individuals who want to participate start by participating in an ideological Turing test for a set number of controversial topics. Top winners of either side of a topic can then participate in an adversarial collaboration for that topic as the finals. In this case, anyone participating in one will be very familiar with both sides of an issue, while still recognizing themselves having enough of a stake or interest in promoting their own side. For example, for “Is Islam reconcilable with Democracy?” we’d have an initial ideological Turing test competition of a Nationalist/Conservative viewpoint vs. an Internationalist/Liberal viewpoint. The two individuals who can fool the most people (one fooling liberals as a conservative, the other vice-versa) would then do the adversarial collaboration. The idea behind this is that from what it sounds, it took a lot of research effort to generate these high quality results, but a lot of pairs abandoned the effort. This first round could serve as a good pre-screen to participants: if one knows enough to imitate an ideological opponent without actually committing to that viewpoint, they are probably well informed on the issue.

    Also an opportunity for a social science experiment: I have a hypothesis that people who perform better on an ideological Turing test are more likely to modify their position after doing an adversarial collaboration, but any “shift” or distance between their original and new position is less than someone who doesn’t perform well on such a test.

    • Watchman says:

      Whilst the Turing test sounds fun, the preconceived divisions it would involve would be too narrow. Remember despite the best efforts of the Guardian and its ilk, the liberal/conservative dichotomy you suggest does not work in a European context, and many commentators here may tend to a political pole but have a lot of atypical views for that pole: I tend libertarian but support gun control for example. And it’s that diversity of viewpoints that should make for the best discussion, rather than the ability to understand a limited and often self-parodying set of beliefs that can be associated with typical liberals or nationalist.

  9. FoxLisk says:

    Scott, you say

    Since they found one Islamic country (Tunisia) that seemed pretty much like a liberal democracy, they concluded the two were compatible. But in real life, when people ask whether Islam and democracy are compatible, I imagine them wondering things like “Are Muslim countries less likely to be democratic than other countries?” or…

    I think this is dangerously naive. There are certainly intellectual, good-faith, essentially “moderate” people who actually mean those questions. There is also a tremendous amount of very real vitriol based around the idea that Islam and liberal democracy are, very literally, incompatible. The cries for borders to be shut down, the “Muslim ban” — people support these things because they believe Islam is an existential threat to democracy.

    • Le Maistre Chat says:

      I support those things because Islamic neighbors can be an existential threat to infidels’s lives.
      So like, Tunisia is 99%+ Sunni Muslim and currently a liberal democracy, so if France becomes likewise it could be as liberal as Tunisia! But getting from Point A to Point B involves lots of French people getting murdered by Muslim immigrants and converts because the brain-dead government didn’t shut down the borders and allow criticism of Islamic theology.

  10. Hoopyfreud says:

    While I enjoyed all of the essays, I really felt like there were some points in the transgender kids one that were never addressed and are central to the whole debate, and that really detracted from it for me.

    The essay is clear that desistance is correlated with the best mental and physical health outcomes, which is expected, but then the anti- side frames much of the ensuing conversation in terms of what is most correlated with desistance, on the argument that “what’s good for desistance is what’s good for the most trans kids.” This is, strictly speaking, true, but leaves non-desisting kids in second place a lot of the time; it seems a bit like saying that we should be giving people placebos for chronic pain because people who aren’t addicted to painkillers have better outcomes.

    That’s not to say that desistance shouldn’t be encouraged, but just to say that it seems like there’s a lot of room to be asking about the children that don’t desist that being underexplored. If your argument is, “social transition and puberty blockers lead to lower desistance rates, so we shouldn’t allow them,” “what about the people who don’t desist?” is a valid response. It feels like the anti- side’s response is “what about them?” To which the answer is, “they kill themselves/experience trauma/self-harm at a statistically significantly elevated rate.” I feel like the conversation didn’t move forward from there, which disappointed me. I still enjoyed going all the evidence that got us there, but it feels to me like that’s the point at which the really interesting debate re: child transition starts, not stops.

    • flame7926 says:

      As one of the authors of that, I felt like we (intentionally) sidestepped the issue of whether desistence was the preferred outcome because I don’t think there was much hope of us coming to an agreement on that. I think if analyzing from an object level or evidence oriented perspective it may come out that it would be better if people spontaneously stopped expressing transgender desires and thus didn’t suffer associated mental health effects etc, but from another perspective, favoring desistence inherently devalues trans lives and experience, kind of saying they are less valuable when people really don’t have any control over whether they are transgender or not.

      I don’t know if it didn’t come across well, but at least when writing I tried to highlight the benefits of social transition (to both those who desist and those who don’t). One of the reasons I wasn’t willing to recommend against social transition and puberty blockers is because of how important these are to children and youth who are transgender and remain transgender.

      • Hoopyfreud says:

        I think it came across, but I also think it made it seem like you two weren’t really talking to each other. There would be one point saying “we should do this because it’ll lead to desistance” and a reply saying “we should do this because it’ll improve quality of life for trans kids,” but those statements don’t really reply to each other. It was almost like reading two different essays based on the same material.

    • Sebastian_H says:

      I think the problem is that for some value of ‘true’ desistance is the preferred outcome for people who aren’t ‘truly’ transgender, but it isn’t for those who are ‘truly’ transgender. The issue is that we don’t know ahead of time which are which. For those who aren’t truly transgender, exploring gender in an affirmative way seems to be helpful and going on puberty blockers is harmful. For someone who is truly transgender, exploring gender in an affirmative way seems to be helpful and going on puberty blockers is helpful.

      The problem is that we are super crappy at figuring out which case is which. In the real world we are probably going to have to screw it up for a few decades while we gather more evidence.

      • sclmlw says:

        I’d narrow that down from looking for “true” anything to something that’s more in line with what should be publicly debated. I don’t really care if your favorite color is blue, or if you like to wear baseball caps.

        Why should I care what gender you identify as? I think the sky-high suicide rates are the real concern here. So if desistance for one sub-group leads to lower suicide rates, but desistance of another sub-group does not, we might conclude that we’ve identified an accurate identification and treatment method to help an at-risk population.

        • Hoopyfreud says:

          This makes sense, but it seems to me that interventions which encourage desistence are likely to preclude interventions which encourage possibly-transgender kids to develop mechanisms for dealing with dysphoria, like social/future hormonal transition. People who don’t go through natal puberty as kids seem to have better outcomes among those who transition than those who do (this is less well-substantiated than what follows), those who are supported in their transition by their family and friends have better outcomes among those who transition than those who aren’t, and people who transition have better outcomes among those with dysphoria that persists than those who don’t. [Citations needed, but they’re available and I’m lazy; if you’d rather, consider this a hypothetical exercise in devising interventions with branching mutually exclusive options]

  11. sclmlw says:

    Thanks for the opportunity to participate in this collaboration, Scott. It was a great learning experience. I think I learned as much from the post-submission reactions and responses as I did during the writing and debate part. I’d point out a couple of things:

    1. I have renewed respect for Scott, who puts out lots of thoughts online and then vigorously, honestly engages with the commentary afterward. I always respected posts that he later went on to completely retract, admitting he was wrong. It’s hard to put out something that you worked a lot on and get negative feedback about, but then admit, “yes you had a point and your point was correct and mine was not.” That takes character.

    2. There was no reciprocal concession, as you suggested above, and I hope it didn’t look like there was. The concession about making vaccination mandatory came very early, as I noted my precondition had been met, and much of our later focus was on whether vaccination resulted in a positive harm (the autoimmunity angle). The concession that public policy should be made based on the evidence against a link between autoimmunity and vaccination literally came last minute.

    3. I don’t think we could have really avoided that part of the collaboration (the autoimmunity angle), which you and others found to be tangential. And this, I think, is important to understand – specifically to those who thought it was a waste of time. The vaccination opponent I collaborated with was especially concerned about vaccination causing autoimmunity. He believed that this harm was sufficient to merit a reversal of public policy against vaccination fundamentally, not just that it should be used as a case for reversal of mandates. Not fully addressing that point would have been to basically ignore his concerns altogether. In essence he was saying, “if I’m right then we have to totally rethink vaccination”. And to not address that would have been to ignore the entire reason he was there to debate in the first place.

    But perhaps when you are part of the mainstream side of an argument, the non-mainstream concerns seem tangential and beside the point, because you don’t believe it; therefore those arguments look like a distraction from the larger debate you should be having. The fact that the minority position wants to talk about it seems like either they’re unfocused and unable to discuss the solid science of the issue (mistake theory) or that they’re trying to cynically distract from all the real problems their proposals imply by bringing up some fringe idea that nobody believes (conflict theory).

    Meanwhile, from the minority viewpoint that ‘tangential point’ is the entire debate, and the fact that your opponents are wont to dismiss your arguments out of hand feels like they’re either uninformed and unwilling to debate you (mistake theory) or that they’re cynically attempting to deflect away from what you perceive are strong opposing arguments (conflict theory).

    3. I think it’s possible to be an advocate of vaccination without strongly endorsing mandates. It’s true that Davis didn’t persuade me that vaccines are actively harmful, and I didn’t persuade him that the evidence swings strongly against any identified harm from vaccines. But we did both agree that vaccines have huge benefits, and he conceded that they prevent much morbidity and mortality. I think pro-vaccination is legitimate public policy, and as such should seek the best strategies for boosting both overall population-level and sub-population-level vaccination rates to prevent the spread of disease. To the extent that mandates promote both those goals, mandates are great. To the extent they aren’t the optimum strategy, we should be willing to pursue others instead. Otherwise, we’re just interested in ideology and not practical solutions. For example, let’s say we figured out how to get mandates to increase population-level vaccination rates to 95%. But say that last 5% reacts to those mandates by banding together and refusing vaccination. Now, the problem here isn’t that they’re refusing, but that they’re banding together. In other words, your policy has generated a community that refuses vaccination, and as a community will thereafter be susceptible to outbreaks.

    I’m not saying mandates cause vaccination opposition. I think that’s a factual question (for a future adversarial collaboration, perhaps). I’m saying there’s this community that refuses to vaccinate. And the problem is that they all hang out together and think that not only should they not vaccinate, but that contracting the diseases we vaccinate against is a good thing. I don’t know that doubling down on one specific strategy that they vehemently oppose is going to help us achieve sub-population level herd immunity here. I’m not saying we should abandon all our policies here in the US, but it would be instructive for one or two states to experiment with some of the approaches mentioned by some of the European commentators, and potentially a less contentious approach could help allay the fears of wary parents.

    Or maybe not. All I’m saying is that support for mandates is not a prerequisite for support for vaccination in general. Mandates are a method that is secondary to the goal of vaccination, which is itself secondary to the goal of reduction of disease burden. That last thing is the thing we really care about.

  12. IsmiratSeven says:

    I now worry that such collaborations would systematically end up legitimizing pseudoscience. If nearly everyone believes that some theory (UFOs, Bigfoot, etc) is false, and some prominent proponent and opponent of the theory do a collaboration, then even if everything goes great and the end result is mostly skewed to the opponent’s side, then there will still be a few compromises and places where the opponent gives into a little social pressure to frame things in a less-than-maximally-hostile way. So if the end result comes out 90% anti-Bigfoot – well, believing in Bigfoot 10% is more than most people currently believe in Bigfoot, so most credulous readers would update in a pro-Bigfoot direction. True, any debate between pro- and anti- Bigfoot factions will involve some pro-Bigfoot arguments that you wouldn’t have heard otherwise, but the adversarial collaboration format risks importing them into the final product in a way that makes it seem like they have a stamp of approval. I’m not sure this format is a good match for pseudoscience unless somebody finds a way around this.

    Inspired by hearing about the Google Doc from the a_reader/flame7926, I was thinking one solution to this would be for each participant to start off by writing their own “submission” – something in a condition that could be submitted to the contest as a “final essay”, outlining their position in absolute, uncompromising terms – framed more as a “thesis”, answering the prompt, than a “rebuttal” which would address the opposing side.

    Then each participant reads the other’s overview, and can highlight portions of contention. After this, the original author of each overview can list what it would take for them to consider the point of contention falsified. I think this would also be beneficial in terms of getting a collaborator to challenge their own side – “what would it take to get me to believe black holes aren’t real?”, and may help shine the light on bad-faith collaborators if these are made available for public viewing (“the only way Mr. Squiggles would back down from this claim that the moon is made of green cheese would be to go there himself and take a bite of it”).

    This keeps things on track, and gets the “adversarial” bit out of the way from the get-go, thus making it a bit easier to not succumb to social pressure (“I like you, so I won’t bring my A game”).

    Collaborators could also take it one step further and make each “overview” completely modular – i.e., the individual papers are written in such a way that sentences can be freely swapped between them, or they can both be summed to a single, bigger essay. If you don’t want a particular sentence in the essay, falsify it.

    Could be a bit daunting for any paper more complex than a standard high school six-paragraph thesis paper, but it’s a possibility that’s fun to dwell on.

    • sclmlw says:

      Interesting idea. This format could presumably be more publicly observable than the private collaborations that were done in this round. It would be interesting to see how a public adversarial collaboration’s results would differ.

    • imoimo says:

      I’d worry about anchoring. Changing one’s mind is hard enough, having to do so after writing a manifesto on your starting position would, I expect, make things harder. Also no one should be committing to anything before the research-discovery phase of collaborating.

      • albatross11 says:

        It seems like a useful way to proceed might be to start out by agreeing on the structure of the argument. For example, if you want to talk about whether vouchers improve quality of education, you might need to start by agreeing on stuff like “How can we tell when quality of education has increased or decreased?” Or if you want to talk about whether the gender gap in STEM is due to discrimination, you might need to start by agreeing on what would be sufficient evidence to decide that question, or maybe on what the plausible components of an explanation for that gender gap look like, and what evidence we have.

        It feels really important, to me, to have some notion of “what would actually cause me to change my mind?” If there’s no evidence or argument that could ever do so, then there’s no useful collaboration possible.

        Unfortunately, this would be easy to game, either intentionally (I’ll believe the other side of this claim only when it’s used to make cold fusion and FTL travel work) or unintentionally (I just make the goal for the other side very, very narrow so that there is no practical way to get the evidence needed to prove it. (An example of this is Turkheimer’s idea that one should not consider any evidence in favor of genetic causes for racial IQ differences until people can point out individual genes and show the path of causation for each one to raise/lower IQ.)

        But collaborators in good faith ought to be able to come to an agreement about at least the parameters of the disagreement, what definitions make sense, and what evidence would be sufficient to lead to a conclusion.

        • marshwiggle says:

          Statements of what would cause participants to change their minds would be ideal, yes, but I fear that would be very difficult.

          Merely coming to agreement on what the question is and what would count as an answer is hard enough, but I do think it is necessary. Without that, what can you really accomplish besides talking past each other to the parts of an audience that share enough of one participants preconceptions?

  13. Le Maistre Chat says:

    Glad to see the education system collaboration win! You two were amazing!

    But in real life, when people ask whether Islam and democracy are compatible, I imagine them wondering things like “Are Muslim countries less likely to be democratic than other countries?” or “Is the Muslim religion one reason why there is so little democracy in the Middle East?” or “In mixed countries, do Muslim populations resist democratic norms?” The collaboration’s focus on listing the particular features of particular Muslim countries’ institutions seemed like a really complete answer to a really boring question.

    That was my reaction to this paper too. “A really thorough report on what six Muslim countries’s governments are like this minute. Huh.”

    • Hazzard says:

      Reading through it I softened slightly. I was fairly hard in the direction of them not being compatible, but my opinion after reading the piece was still they’re not especially compatible and any progress in favour of democracy will end up being despite Islam, rather than because of it. But Tunisia did convince me progress can happen, albeit slowly and with an iron fist

      • A different way of putting the question would be “is Islam less compatible with democracy than Christianity?” A lot of the impression that it is comes from comparing modern Christian countries, in most of which the population doesn’t take their religion very seriously, with modern Muslim countries, in most of which the population takes their religion considerably more seriously.

        On the empirical side, you would want to look at countries, possibly in the past, which were hard core Christian—Deseret would be one recent but brief example, assuming we count Mormons as Christians. Also look for countries that are nominally Muslim but where a large fraction of the population doesn’t fast through Ramadan, say the required prayers, etc.—I’m not sure if there are examples. That would be equivalent to modern Catholic countries where contraception and divorce are pretty much taken for granted.

        On the theoretical side one would look at to what extent the doctrines of the two religions were or were not consistent with democracy. As best I can tell, Islamic doctrine is not consistent with treating all religions equally, but I think one could argue that democratic election of the caliph is actually closer to what the religion requires than the way historical caliphates actually developed.

  14. Alex Zavoluk says:

    Congratulations! I enjoyed reading all of the essays.

    The lesson I would draw from John and Christian is probably more that their original question is somewhat ill-formed and should be broken down; the question itself is half category dispute, half flag-waving. I’m not really blaming them, though; it’s the way the question is usually framed in public discourse. And like in regular arguments, attempting to look into the argument in any detail quickly reveals how many details it is covering up. I also would have liked a look at Islamic theology, history of the Islamic world, a wider variety of Islamic countries, non-state actors, Muslims in western countries, etc. but given that our collaborators are doing this in their free time over the course of a few months I think that can be forgiven.

  15. JohnBuridan says:

    I think in the initial prompt the “present a unified view” created the dynamic where social courtesy would on occasion inhibit strength of the true position. In the ideal American justice system each side argues as strenuously as possible for their position, and there is no redacted unified presentation of the evidence.

    In the future, collaborations on controversial issues should be converts to the same position or they should write a dialogue together with stand-in characters?

  16. ana53294 says:

    I have shifted my opinions from one view to the polar opposite. In my teens, I was very much anti-GMO (because a big part of the green movement said things that made sense to me, every other thing made sense, so I thought that they were also right on GMOs). Then I went and studied Biology in University, with a specialization in plants, and I had professors who talked about the scientific facts about GMOs for four years. And I eventually changed my views.

    The thing is, I didn’t go to sleep anti-GMO and wake up pro-GMO the next day. It was rather a slow drip of small shifts in my thinking. And even when I wasn’t so sure about the things I was saying the previous year, because the knowledge I learned made me doubt what I thought I knew, I never, ever, publically backed down from views I had expressed before, even once I decided they were wrong.

    People have pride. Changing a belief you hold fundamental to who you are, and doing it consciously and publically is a really, really tall order. A lot of people who change their viewpoints will then erase all their memories of holding the previous opinion. Remembering that you once held a different opinion is hard, and makes you doubt the opinions you have right now. Will they change with time, as you learn new facts, also?

    I was only able to become pro-GMO once I was in an environment where people did not know opinions I strongly endorsed beforehand. My pride and feelings of self-worth would not let me back down, even if it meant I would have to publicly endorse something I did not believe in anymore. So the fact that nobody changed their opinions after the adversarial collaboration does not surprise me at all.

    If we are not talking about some minor quibble, but a view that makes you part of a community, such as the anti-vaxxer community, not only do you have to convince the people about the facts of the matter – you have to convince them to abandon their community. Because they won’t be able to hold their head high while staying in that community and abandoning that community’s belief system. Because communities police thought quite effectively, and kick out people who reject their key ideas.

    And whenever I see people who fundamentally changed their beliefs, such as the people who left gangs, or left a cult, they do all eventually leave their community.

  17. Matt M says:

    I am now interested in reading a collaboration on the existence of Bigfoot.

    • albertborrow says:

      I know, right? I almost regret not suggesting my own when the contest was announced, but I understood that I wasn’t really ready to invest that amount of time and research into it.

    • Watchman says:

      I’m assuming that would require a SSC reader to rationally believe in the existence of bigfoot, which might be a bit of a difficult task to find. And I think that might be the point that Scott’s concern about adversarial collaboration on paranormal areas becomes moot: it’s not possible to have a collaboration of this nature if one party’s contribution is based on paranormal investigations and David Icke (full disclosure: I know nothing of David Icke’s views on bigfoot, and I really don’t care to know…) whilst the other is rationally considering evidence. To actually engage in meaningful debate will require one or both parties to abandon all their priors.

      There is probably more room for such a collaboration on technical issues: would a proposed mechanism for a UFO work; is a particular piece or class of evidence valid; is the government really that competent at covering things up etc. Basically anything that doesn’t require initial abandonment of priors might work; anything where the scope is too wide to allow this is unlikely to get to a point where concessions need to be made.

    • Gazeboist says:

      You can’t really answer that question without defining what you mean by “Bigfoot”, and further determining what it would mean for “Bigfoot” to “exist”.

  18. Jacobethan says:

    First off, a hearty congratulations to all the participants. And thanks also for these very thoughtful postmortems, which I found in some ways as illuminating as the contributions themselves.

    If you put the entries on a spectrum of “hot-button”-ness (i.e., tribal affective loading, difficulty of constructive discussion in the culture at large), transgender kids probably has the “hottest” and gifted education the “coldest” button temperature, and in that light it’s interesting how those collaborations also appear to have diverged most sharply at the level of process.

    That is, Michael and TracingWoodgrains seem to have been by far the most interested in debating each other in an open-ended, richly recursive way where there’s a continual attempt to move toward higher-level generators of disagreement. And that’s reflected in the structure of their actual entry, which devotes a lot of attention to how the viability of any intervention needs to be assessed in the context of a system whose participants have competing and perhaps incoherent normative aims.

    A_reader and Flame7926, by contrast, seem to have developed a fairly refined method of dividing their topic up into analytically separate components, and to have limited debate to asking each other in a granular way whether a particular piece of evidence had changed their view of the issue in that specific bucket. There seems to have been very little interest in systematically exploring whatever higher-level premises might be responsible for consistent divergences in interpretation across different empirical domains.

    I don’t want to say that either approach is intrinsically superior to the other, though I guess it might be evident that in this case I found the former a bit more interesting. It may be that something like A_reader and Flame7926’s method is necessary for working productively on an issue where the ideological loading is so out-front that it requires some conscious routing around. I just found the difference itself pretty striking and worth underlining explicitly.

  19. Somethatname says:

    It’s not surprising that nobody shifted sides. The structure of the contest puts them into teams of 1 and pits them against each other. Usually the person feels that their views are representative of a larger tribe. So why would they endure shame and betray that tribe by changing their opinion? It’s like a gladiator fight, only we somehow expect one of the gladiators to defeat themselves and join the other gladiator. I agree that it makes for good reading, however. I personally learned a lot from these entries, particularly the transgender one. It has changed my view about the options transgender children should be offered, and when these should be offered.

    I actually thought that the vaccination one came pretty close to coming to agreement. They had accepted the merit of each others view, and separated them so that the views weren’t in direct opposition. Perhaps in a different setting one of them might have felt more willing to come to a complete agreement.

    My suggestions for future competitions if encouraging contestants to change their minds is important:
    – Remove the salience of the contestants identity. This might mean them adopting pseudonyms and not having any history about them.
    – More emphasis on pathos and ethos. I realise that this is a bit of a heresy to suggest, but they are important to consider when trying to change peoples minds.
    – Require the contestants to meet in person.
    – Encourage a non-judgemental atmosphere. People are more willing to change their deeply held beliefs when they do not feel like they will be critiqued and judged for doing so.
    – Set an example. If people see that your your mind and be vulnerable about it, it may turn into a group norm that you adopt.
    – Unite them against a common enemy. This is already done to a degree by having the groups complete against each other.

    I realise that these aren’t very realistic suggestions. And they would make for a crap competition. which is the point of making them. It might be better to focus on how effectively the entries change the mind of the audience. Which includes accepting that they might convince people to accept things you know are wrong.

    I’d also suggest that instead of giving your personal award and opinion, you create a panel to do that. It’d feel less subjective and authoritarian that way. Which is a bit weird considering the nature of the contest.

  20. Robert Jones says:

    I think I said this before, but I thought the vaccines collaboration skirted the real question, which is, “Does mandatory vaccination increase the vaccination rate?” It seems plausible to me that if parents are told, “If you bring your child to X place at Y time, we will give them a free vaccination to protect them from [nasty disease]”, they will do so, whereas “You are required by law to bring your child to X place at Y time for vaccination” might cause alarm. The fact that some countries that don’t have mandatory vaccination have higher vaccination rates than the US is at least suggestive that the mandate may be counterproductive, but the authors didn’t really explore this possibility (presumably because Mark Davis was opposed to the mandate for different reasons).

    • albatross11 says:

      ISTM that the public policy question is not whether mandatory vaccination raises vaccination rates–it’s hard to imagine how it wouldn’t. Instead, I think the public policy question comes down to the impact of your choice not to vaccinate yourself/your kids on:

      a. Your kids’ safety
      b. The safety of your neighbors

      For the first one, we should probably judge this the way we do other child endangerment sorts of cases–what added level of risk are you taking on for your kids, and is it a level where we’d normally charge you with endangering your kids. It’s hard to imagine (say) refusing the MMR for your kids rising to that level of extra risk.

      For the second one, we need to have some way to estimate the impact of your individual decision to vaccinate on your neighbors. The easy way to think about this is in terms of herd immunity. So the relevant question there seems to me to be about whether or not mandatory vaccination is needed to get herd immunity. If so, then there’s a case for mandatory vaccination (which you might argue on other grounds, but at least the case is pretty clear for why you want it). If mandatory vaccination doesn’t get you herd immunity (either because you’d have it anyway or because you can’t get it even with mandatory vaccination), then it seems really hard to make the case for it.

      ETA: In (a), another precedent might be car seat and bike helmet laws. I’m not sure how that corresponds to added risks from not vaccinating your kids.

      Also, FWIW, my kids get all their shots, and so do I. IMO, this is the best tradeoff between risks/costs.

      • sclmlw says:

        Interesting question, since the answer is basically, “it depends”. For both (a) and (b) if the vaccination rate is, say, 12% then the burden would be on the parents to say, “look, you had this intervention that could have saved your child/neighbor significant risk of morbidity/mortality but you didn’t take it.” But if the vaccination rate is 12% then liability levels are very broad and you have to go around prosecuting large numbers of parents. This would be like police officers pulling over every car that goes even 1 mph over the speed limit.

        Meanwhile, once you reach the level of herd immunity you start getting to the point where, outside certain fringe cases, avoiding vaccination is not really expected to increase your risk of disease appreciably. So if a public policy is sufficient to influence vaccination rates, shouldn’t that be the place we look to, as opposed to individual, private liability? Especially since private liability is either meaningless or impractical to implement?

    • Sebastian_H says:

      You might be right that it is the mandate itself that is counterproductive, but you should be open to the possibility that the mandate is a response to lower vaccination rates (due to other societal/cultural issues) or that the mandate is not particularly effective and the difference in rates is due to other societal/cultural factors. It seems plausible that different countries have different levels of trust in their elites (or their medical elites) and that this will cause varying rates of vaccination somewhat independent of whether or not vaccination is mandatory.

      • sclmlw says:

        I think, to synthesize all these points, that there’s a concern that mandates are effective at raising vaccination rates in a path-dependent manner. Thus, you can raise rates through mandates only to a certain level. And if you’ve taken that path, you may be shutting off a path that will get you to a higher vaccination rate than you might had you taken a different path that didn’t include mandates.

        But that doesn’t mean that you started with mandates and ended up with a less-comprehensive vaccination rate. It could be that other institutional features lead you down a path where mandates became inevitable, and exploring entirely new paths of social norms, etc., would be required to get you to a higher vaccination rate. You’d have to pack up multiple steps to get to that point, and it’s really not clear how you get there from here.

  21. Jack V says:

    Oh, that is interesting. I feel like I got more out of reading the experiences than the collaborations. In particular, “we almost always agreed at the object level, but we originally had different generalisations.” I need to remember that in many, many circumstances.

    I didn’t read the articles well enough to start with. I was impressed at the amount of research in them, even though I didn’t fully appreciate it at the time.

    The problem I had from the start was that many of the titles seemed hard to answer. Islam and democracy was a great collaboration, but it’s hard to make it a question you can answer, too much depends if you mean “harder to be compatible” or “less likely” or “almost impossible”. Schools suck for a lot of people, are we just answering that, or answering if they’re worse for academic inclined children than non-academic-inclined children, and how to compare those? The contestants did an excellent job summarising the evidence, but in a position where it’s hard to come to a clear conclusion, even a tentative one. C.f. your comment about shifting the ground, which is good to come up with an agreed question, but also potentially dodges the issue.

    And even more so with the vaccine question. When I first saw it, I mentally interpreted the question as “should children be vaccinated”. But if you’re specifically asking whether it should be mandatory, that’s more like, “if you already accept that it should be done, what’s the most effective way of making that happen, making it mandatory or something else?” which is a very important question, but very difficult to answer, it’s the sort of sociology which is hard to research. I’m used (in the UK) to vaccination being normal, offered and assumed by default but not actually mandated, and instinctively I honestly thought that’s what was meant by mandatory, occasional exceptions allowed (but more from “I’m too busy” then “I don’t want it”) although now that’s starting to change.

    I would guess that making it mandatory may make more backlash than benefit, and other measures (more education, requiring vaccination-or-specific-excuse to use schools, etc) are more likely to succeed, but I don’t really know.

    On the other hand, I’m in favour of everything, especially medical interventions, being non-mandatory if possible, and if a small number of people opt out, that’s fine as long as we get herd immunity. But if enough people want to opt out to break herd immunity then being optional suddenly matters and maybe stronger measures (whether mandating or not) are necessary.

    I’m sympathetic to people worried that vaccines are systematically causing medical problems. There are all sorts of widespread things that WERE a medical mistake. But it always seems like they’re LOOKING for a problem. There’s so many different problems suggested, and there doesn’t seem to be any particular evidence other than “this got more common over the last twenty years and so did vaccines”. But I’m also like, what’s the end game here? Do we find safer vaccines? Do we accept that the problems of vaccines, whatever they are, are sufficiently bad it’s better to go back to deaths from measles epidemics instead? Or does it only work as long as fairly small number of people think there’s a problem?

    • Robert Jones says:

      But it always seems like they’re LOOKING for a problem.

      That’s my impression as well, i.e. people have formed the opinion that vaccines are bad and are looking for evidence to support that opinion. That raises the question of why people would initially form the opinion that vaccines are bad, and I do wonder whether it might be precisely because they are mandatory.

      • Michael Watts says:

        I submit that the reason people initially form the opinion that vaccines are bad in general is the same reason people specifically try to personally avoid getting vaccines — vaccines involve sticking a needle into your vein. People hate this. Many people hate it a lot.

        The phenomenon of looking for a reason not to do something you really really don’t want to do is well known and doesn’t require further explanation.

        • eyeballfrog says:

          Aren’t most vaccines done intramuscularly, not intravenously?

          I mean, not that it matters too much in that I suspect people are also not fond of getting a needle in the shoulder. But just clarifying.

        • sclmlw says:

          I think it’s less a concern about getting stuck with a needle, and more a concern that the intervention is targeted universally toward children. I think it’s true opponents are “looking” for some problem they expect to be there, but I don’t think the issue is “I feel apprehensive about my kids getting stuck with a needle.” I mean, there’s not nearly as strong a movement against circumcision, and there’s a lot more pain involved in that intervention. If it was only about pain, I expect a nearly equivalent movement. I’m not saying there’s no anti-circumcision movement; but it’s clearly not matched to the anti-vaccine movement in intensity, distrust, etc.

          • Michael Watts says:

            I don’t think it’s only about pain. I think that circumcision consists entirely of actions that, while painful, are “external” and easy to comprehend in full. I think people view the needles that are part of vaccination as different in kind, and scarier; in my model, pain is not even a consideration.

            Which is only what you’d expect from the fact that vaccination does not usually hurt.

        • Evan Þ says:

          In addition to that (and the reasons offered in other replies), many people don’t trust the government. When the government rolls out a big program, many people will look behind the surface justification – is it really meeting the need? Is it subtly trying to do something else as well? Does it have other bad effects?

          I’m very sympathetic to this position myself, even though I conclude after investigation that the government actually got it right with vaccines.

    • cryptoshill says:

      My childhood is an interesting case – I was vaccinated, but not fully. My mother was absolutely convinced that a particular vaccine (which was at the time only a few years old) was too risky to give me considering that it was for a very uncommon disease. People like my mother are the sorts of people I try to defend when I defend vaccination opponents. My basic claim is you *really* think that you want to subsume all of your decisionmaking power about what medications to give or not give your child to the government? This claim focuses on systemic risks (bad quality assurance, bioterrorism, a particular group of doctors at the CDC being disastrously wrong about something). I admit that as your typical internet-libertarian outlier, I place more focus on systemic risks than maybe I should. I do, however feel like the way mandatory vaccination is forwarded (and the way vaccination opponents are attacked) is giving me more cause for this concern than less.

    • Joseph Greenwood says:

      It looks like a lot of these comments about vaccination are good-faith second hand speculation, so I want to interject with what I’ve seen of opposition to vaccines.

      At least two of my aunts are firmly in the anti-vaccination camp, and both of them started okay-vax or pro-vax. However, they each had children with developmental issues that became prominent just after they were vaccinated, and so they asked themselves “is something wrong here?”

      I think this is an understandable reaction. If you went to a restaurant and ordered a new dish, then went home and started vomiting, you probably wouldn’t order this dish again either.

      At this point, they started looking into vaccination, read the research put forth by both sides, and drew their own conclusions about how vaccinations are a ploy by big pharma to make money and the risks associated with the are systematically downplayed and suppressed. At least one of my aunts acknowledges that vaccines do have the good effects they are claimed to have, but counters that more than one of her kids experiences some degree of autism because of them, and says that is too high of a cost.

  22. educationrealist says:

    Congratulations to all the collaborators. I’m amazed at the work you put into it, and I’m so glad that everyone got an award, as you all deserved it.

    Michael and Tracing Woodgrains, your article spawned hundreds of passionate comments and some really good conversations. I enjoyed participating in the results of your work, and I was fascinated and honored to read your early drafts.