So now that it’s June and predicting what will happen in 2015 is about 40% easier, I might as well get started. Acceptable confidence levels are 50%, 60%, 70%, 80%, 90%, 95%, and 99%, to make it easy to score. All predictions are about the state of the world on 12/31/2015:

World Events
1. US will not get involved in any new major war with death toll of > 100 US soldiers: 70%
2. North Korea’s government will survive the year without large civil war/revolt: 95%
3. Greece will not announce it’s leaving the Euro: 60%
3. Neither Russia nor Qatar will lose their World Cups: 80%
4. Ebola will kill fewer people in second half of 2015 than the in first half: 95%
5. No terrorist attack in the USA will kill > 100 people: 90%
6. Assad will remain President of Syria: 70%
7. Israel will not get in a large-scale war (ie >100 Israeli deaths) with any Arab state: 90%
8. Syria’s civil war will not end this year: 80%
9. ISIS will control less territory than it does right now: 70%
10. ISIS will continue to exist: 80%
11. Iran will reach a deal with the West on nuclear weapons: 80%
12. No major civil war in Middle Eastern country not currently experiencing a major civil war: 90%
13. Iraq’s situation not to get any worse (eg gov’t collapse, new rebellion): 60%
14. Obamacare will survive the year mostly intact: 60%
15. Hillary Clinton will be the top-polling Democratic Presidential candidate: 95%
16. Jeb Bush will be the top-polling Republican candidate: 50%
17. Trans-Pacific Partnership to pass at least mostly intact: 60%
18. US official unemployment rate will be < 7% in Dec 2015: 95% 19. Bitcoin will end the year higher than $200: 95% 20. Oil will end the year >$60 a barrel: 50%

Personal Life
21. SSC will remain active: 95%
22. SSC will get fewer hits in the second half of 2015 than the first half: 60%
23. At least one SSC post in the second half of 2015 will get > 100,000 hits: 70%
24. Shireroth will remain active: 90%
25. I will remain at my same job through the end of 2015: 95%
26. There will be no further ramifications or lawsuits from either side over the flooding of my house: 80%
27. I will reach my savings target: 90%
28. I will get a score at >95th percentile for my year on PRITE: 50%
29. I will be involved in at least one published/accepted-to-publish research paper by the end of 2015: 60%
30. I will not break up with any of my current girlfriends: 80%
31. I will not get any new girlfriends: 50%
32. I will not finish [project]: 60%
33. I will attend NYC Solstice ritual: 80%
34. I will flake out of my plan to lead some kind of Solstice Ritual myself: 60%
35. I will be living in the house I’m currently trying to arrange to rent: 70%

These are all the things I could think of worth predicting; feel free to suggest others.

1. Obamacare will survive the year mostly intact: 80%
2. US does not get involved in any new major war with death toll of > 100 US soldiers: 90%
3. Syria’s civil war will not end this year (unstable cease-fire doesn’t count as “end”): 60%
4. Bitcoin will end the year higher than $1000: 70%
5. US official unemployment rate will be < 7% in Dec 2014: 80% 6. Republicans will keep the House in US midterm elections: 80% 7. Democrats will keep the Senate in US midterm elections: 60%
8. North Korea’s government will survive the year without large civil war/revolt: 80%
9. Iraq’s government will survive the year without large civil war/revolt: 60%
10. China’s government will survive the year without large civil war/revolt: 99%
11. US government will survive the year without large civil war/revolt: 99%
12. Egypt’s government will survive the year without large civil war/revolt: 60%
13. Israel-Palestine negotiations remain blocked, no obvious plan for Palestinian independence: 99%
14. Israel does not get in a large-scale war (ie >100 Israeli deaths) with any Arab state: 95%
15. Sochi Olympics will not be obvious disaster (ie spectacular violence, protests that utterly overshadow events, or have to be stopped early): 90%
16. Putin will remain President of Russia at end of 2014: 95%
17. Obama will remain President of USA at end of 2014: 95%
18. No nuclear weapon used in anger in 2014: 99%
19. No terrorist attack in USA killing > 100 people: 90%
20. No mass shooting in USA killing > 10 people: 50%
21. Republic of Shireroth will not officially disband or obviously die before end of 2014: 90%
22. Republic of Shireroth will remain in Bastion Union: 80%
23. Active population of Shireroth on last 2014 census will be between 10 and 20: 70%
24. Slate Star Codex will remain active until end of 2014: 70%
25. Slate Star Codex will get more hits in 2014 than in 2013: 60%
26. At least one 2014 Slate Star Codex post will get > 10,000 hits total: 80%
27. No 2014 Slate Star Codex post will get > 100,000 hits total: 90%
28. 2014 Less Wrong Survey will show higher population than 2013 Survey conditional on similar methodology: 80%
29. 2014 Less Wrong Survey will show population < 2000 C.O.S.M.: 70% 30. 2014 Less Wrong Survey will have > % female than 2013 Less Wrong Survey: 70%
31. 2014 Less Wrong Survey will have < 20% female: 90% 32. HPMoR will conclude in 2014: 80%
33. At least 1 LW post > 100 karma in 2014: 50%
34. No LW post > 100 karma by me: 80%
35. CFAR will continue operating in end of 2014: 90%
36. MIRI will continue operating in end of 2014: 99%
37. MetaMed will continue operating in end of 2014: 80%
38. None of Eliezer, Luke, Anna, or Julia will quit their respective organizations: 60%
39. No one in LW community will become world-famous (let’s say >= Peter Thiel) for anything they accomplish this year: 80%
40. MIRI will not announce it is actively working on coding a Friendly AI (not just a few bits and pieces thereof) before the end of 2014: 99%
41. I will remain at my same job through the end of 2014: 95%
42. I will get a score at >95th percentile for my year on PRITE: 70%
43. I will be involved in at least one published/accepted-to-publish research paper by the end of 2014: 20%
44. I will not break up with any of my current girlfriends through the end of 2014: 50%
45. I will not get any new girlfriends in 2014: 50%
46. I will not be engaged by the end of 2014: 80%
47. I will be living with Ozy by the end of 2014: 80%
48. I will take nootropics on average at least once/week through the second half of 2014: 50%
49. I will not manage to meditate at least 100 days in 2014: 80%
50. I will attend NYC Solstice ritual: 60%
51. I will arrange some kind of Michigan Solstice Ritual: 50%
52. I will not publicly identify as religious (> atheist) by the end of 2014: 95%
53. I will not publicly identify as neoreactionary or conservative (> liberal or libertarian) by the end of 2014: 70%
54. I will not publicly identify as leftist or communist (> liberal or libertarian) by the end of 2014: 80%
55. I will get a Tumblr in 2014: 50%
56. I will not delete/abandon either my Facebook or Twitter accounts: 60%
57. I will have less than 1000 Twitter followers by the end of 2014: 60%
58. When Eliezer sends me a copy of “Perfect Health Diet”, I will not be convinced that it is more correct or useful than the best mainstream nutrition advice (eg Stephen Guyenet’s blog): 70%
59. I will end up being underconfident on these predictions: 50%

Of predictions at the 50% level, 4/8 (50%) were correct
Of predictions at the 60% level, 7/9 (77%) were correct
Of predictions at the 70% level, 6/8 (75%) were correct
Of predictions at the 80% level, 12/15 (80%) were correct
Of predictions at the 90% level, 7-0 (100%) were correct
Of predictions at the 95% level, 5-0 (100%) were correct
Of predictions at the 99% level, 6-0 (100%) were correct

I declare myself to be impressively well-calibrated. You should all trust me about everything.

2015 predictions coming soon.

For example, take medical research. Suppose a certain drug is weakly effective against a certain disease. After a few years, a bunch of different research groups have gotten their hands on it and done all sorts of different studies. In the best case scenario the average study will find the true result – that it’s weakly effective.

But there will also be random noise caused by inevitable variation and by some of the experiments being better quality than others. In the end, we might expect something looking kind of like a bell curve. The peak will be at “weakly effective”, but there will be a few studies to either side. Something like this:

We see that the peak of the curve is somewhere to the right of neutral – ie weakly effective – and that there are about 15 studies that find this correct result.

But there are also about 5 studies that find that the drug is very good, and 5 studies missing the sign entirely and finding that the drug is actively bad. There’s even 1 study finding that the drug is very bad, maybe seriously dangerous.

This is before we get into fraud or statistical malpractice. I’m saying this is what’s going to happen just by normal variation in experimental design. As we increase experimental rigor, the bell curve might get squashed horizontally, but there will still be a bell curve.

In practice it’s worse than this, because this is assuming everyone is investigating exactly the same question.

Suppose that the graph is titled “Effectiveness Of This Drug In Treating Bipolar Disorder”.

But maybe the drug is more effective in bipolar i than in bipolar ii (Depakote, for example)

Or maybe the drug is very effective against bipolar mania, but much less effective against bipolar depression (Depakote again).

Or maybe the drug is a good acute antimanic agent, but very poor at maintenance treatment (let’s stick with Depakote).

If you have a graph titled “Effectiveness Of Depakote In Treating Bipolar Disorder” plotting studies from “Very Bad” to “Very Good” – and you stick all the studies – maintenence, manic, depressive, bipolar i, bipolar ii – on the graph, then you’re going to end running the gamut from “very bad” to “very good” even before you factor in noise and even before even before you factor in bias and poor experimental design.

So here’s why you should beware the man of one study.

If you go to your better class of alternative medicine websites, they don’t tell you “Studies are a logocentric phallocentric tool of Western medicine and the Big Pharma conspiracy.”

They tell you “medical science has proved that this drug is terrible, but ignorant doctors are pushing it on you anyway. Look, here’s a study by a reputable institution proving that the drug is not only ineffective, but harmful.”

And the study will exist, and the authors will be prestigious scientists, and it will probably be about as rigorous and well-done as any other study.

And then a lot of people raised on the idea that some things have Evidence and other things have No Evidence think holy s**t, they’re right!

On the other hand, your doctor isn’t going to a sketchy alternative medicine website. She’s examining the entire literature and extracting careful and well-informed conclusions from…

Haha, just kidding. She’s going to a luncheon at a really nice restaurant sponsored by a pharmaceutical company, which assures her that they would never take advantage of such an opportunity to shill their drug, they just want to raise awareness of the latest study. And the latest study shows that their drug is great! Super great! And your doctor nods along, because the authors of the study are prestigious scientists, and it’s about as rigorous and well-done as any other study.

But obviously the pharmaceutical company has selected one of the studies from the “very good” end of the bell curve.

And I called this “Beware The Man of One Study”, but it’s easy to see that in the little diagram there are like three or four studies showing that the drug is “very good”, so if your doctor is a little skeptical, the pharmaceutical company can say “You are right to be skeptical, one study doesn’t prove anything, but look – here’s another group that finds the same thing, here’s yet another group that finds the same thing, and here’s a replication that confirms both of them.”

And even though it looks like in our example the sketchy alternative medicine website only has one “very bad” study to go off of, they could easily supplement it with a bunch of merely “bad” studies. Or they could add all of those studies about slightly different things. Depakote is ineffective at treating bipolar depression. Depakote is ineffective at maintenance bipolar therapy. Depakote is ineffective at bipolar ii.

So just sum it up as “Smith et al 1987 found the drug ineffective, yet doctors continue to prescribe it anyway”. Even if you hunt down the original study (which no one does), Smith et al won’t say specifically “Do remember that this study is only looking at bipolar maintenance, which is a different topic from bipolar acute antimanic treatment, and we’re not saying anything about that.” It will just be titled something like “Depakote fails to separate from placebo in six month trial of 91 patients” and trust that the responsible professionals reading it are well aware of the difference between acute and maintenance treatments (hahahahaha).

So it’s not so much “beware the man of one study” as “beware the man of any number of studies less than a relatively complete and not-cherry-picked survey of the research”.

II.

I think medical science is still pretty healthy, and that the consensus of doctors and researchers is more-or-less right on most controversial medical issues.

(it’s the uncontroversial ones you have to worry about)

Politics doesn’t have this protection.

Like, take the minimum wage question (please). We all know about the Krueger and Card study in New Jersey that found no evidence that high minimum wages hurt the economy. We probably also know the counterclaims that it was completely debunked as despicable dishonest statistical malpractice. Maybe some of us know Card and Krueger wrote a pretty convincing rebuttal of those claims. Or that a bunch of large and methodologically advanced studies have come out since then, some finding no effect like Dube, others finding strong effects like Rubinstein and Wither. These are just examples; there are at least dozens and probably hundreds of studies on both sides.

But we can solve this with meta-analyses and systemtic reviews, right?

Depends which one you want. Do you go with this meta-analysis of fourteen studies that shows that any presumed negative effect of high minimum wages is likely publication bias? With this meta-analysis of sixty-four studies that finds the same thing and discovers no effect of minimum wage after correcting for the problem? Or how about this meta-analysis of fifty-five countries that does find effects in most of them? Maybe you prefer this systematic review of a hundred or so studies that finds strong and consistent effects?

Can we trust news sources, think tanks, econblogs, and other institutions to sum up the state of the evidence?

CNN claims that 85% of credible studies have shown the minimum wage causes job loss. But raisetheminimumwage.com declares that “two decades of rigorous economic research have found that raising the minimum wage does not result in job loss…researchers and businesses alike agree today that the weight of the evidence shows no reduction in employment resulting from minimum wage increases.” Modeled Behavior says “the majority of the new minimum wage research supports the hypothesis that the minimum wage increases unemployment.” The Center for Budget and Policy Priorities says “The common claim that raising the minimum wage reduces employment for low-wage workers is one of the most extensively studied issues in empirical economics. The weight of the evidence is that such impacts are small to none.”

Okay, fine. What about economists? They seem like experts. What do they think?

Well, five hundred economists signed a letter to policy makers saying that the science of economics shows increasing the minimum wage would be a bad idea. That sounds like a promising consensus…

..except that six hundred economists signed a letter to policy makers saying that the science of economics shows increasing the minimum wage would be a good idea. (h/t Greg Mankiw)

Fine then. Let’s do a formal survey of economists. Now what?

raisetheminimumwage.com, an unbiased source if ever there was one, confidently tells us that “indicative is a 2013 survey by the University of Chicago’s Booth School of Business in which leading economists agreed by a nearly 4 to 1 margin that the benefits of raising and indexing the minimum wage outweigh the costs.”

But the Employment Policies Institute, which sounds like it’s trying way too hard to sound like an unbiased source, tells us that “Over 73 percent of AEA labor economists believe that a significant increase will lead to employment losses and 68 percent think these employment losses fall disproportionately on the least skilled. Only 6 percent feel that minimum wage hikes are an efficient way to alleviate poverty.”

So the whole thing is fiendishly complicated. But unless you look very very hard, you will never know that.

If you are a conservative, what you will find on the sites you trust will be something like this:

Economic theory has always shown that minimum wage increases decrease employment, but the Left has never been willing to accept this basic fact. In 1992, they trumpeted a single study by Card and Krueger that purported to show no negative effects from a minimum wage increase. This study was immediately debunked and found to be based on statistical malpractice and “massaging the numbers”. Since then, dozens of studies have come out confirming what we knew all along – that a high minimum wage is economic suicide. Systematic reviews and meta-analyses (Neumark 2006, Boockman 2010) consistently show that an overwhelming majority of the research agrees on this fact – as do 73% of economists. That’s why five hundred top economists recently signed a letter urging policy makers not to buy into discredited liberal minimum wage theories. Instead of listening to starry-eyed liberal woo, listen to the empirical evidence and an overwhelming majority of economists and oppose a raise in the minimum wage.

And if you are a leftist, what you will find on the sites you trust will be something like this:

People used to believe that the minimum wage decreased unemployment. But Card and Krueger’s famous 1992 study exploded that conventional wisdom. Since then, the results have been replicated over fifty times, and further meta-analyses (Card and Krueger 1995, Dube 2010) have found no evidence of any effect. Leading economists agree by a 4 to 1 margin that the benefits of raising the minimum wage outweigh the costs, and that’s why more than 600 of them have signed a petition telling the government to do exactly that. Instead of listening to conservative scare tactics based on long-debunked theories, listen to the empirical evidence and the overwhelming majority of economists and support a raise in the minimum wage.

Go ahead. Google the issue and see what stuff comes up. If it doesn’t quite match what I said above, it’s usually because they can’t even muster that level of scholarship. Half the sites just cite Card and Krueger and call it a day!

These sites with their long lists of studies and experts are super convincing. And half of them are wrong.

At some point in their education, most smart people usually learn not to credit arguments from authority. If someone says “Believe me about the minimum wage because I seem like a trustworthy guy,” most of them will have at least one neuron in their head that says “I should ask for some evidence”. If they’re really smart, they’ll use the magic words “peer-reviewed experimental studies.”

But I worry that most smart people have not learned that a list of dozens of studies, several meta-analyses, hundreds of experts, and expert surveys showing almost all academics support your thesis – can still be bullshit.

Which is too bad, because that’s exactly what people who want to bamboozle an educated audience are going to use.

III.

I do not want to preach radical skepticism.

For example, on the minimum wage issue, I notice only one side has presented a funnel plot. A funnel plot is usually used to investigate publication bias, but it has another use as well – it’s pretty much an exact presentation of the “bell curve” we talked about above.

This is more of a needle curve than a bell curve, but the point still stands. We see it’s centered around 0, which means there’s some evidence that’s the real signal among all this noise. The bell skews more to left than to the right, which means more studies have found negative effects of the minimum wage than positive effects of the minimum wage. But since the bell curve is asymmetrical, we intepret that as probably publication bias. So all in all, I think there’s at least some evidence that the liberals are right on this one.

Unless, of course, someone has realized that I’ve wised up to the studies and meta-analyses and and expert surveys, and figured out a way to hack funnel plots, which I am totally not ruling out.

(okay, I kind of want to preach radical skepticism)

Also, I should probably mention that it’s much more complicated than one side being right, and that the minimum wage probably works differently depending on what industry you’re talking about, whether it’s state wage or federal wage, whether it’s a recession or a boom, whether we’re talking about increasing from $5 to $6 or from $20 to $30, etc, etc, etc. There are eleven studies on that plot showing an effect even worse than -5, and very possibly they are all accurate for whatever subproblem they have chosen to study – much like the example with Depakote where it might an effective antimanic but a terrible antidepressant.

(radical skepticism actually sounds a lot better than figuring this all out).

IV.

But the question remains: what happens when (like in most cases) you don’t have a funnel plot?

I don’t have a good positive answer. I do have several good negative answers.

Decrease your confidence about most things if you’re not sure that you’ve investigated every piece of evidence.

Do not trust websites which are obviously biased (eg Free Republic, Daily Kos, Dr. Oz) when they tell you they’re going to give you “the state of the evidence” on a certain issue, even if the evidence seems very stately indeed. This goes double for any site that contains a list of “myths and facts about X”, quadruple for any site that uses phrases like “ingroup member uses actual FACTS to DEMOLISH the outgroup’s lies about Y”, and octuple for RationalWiki.

Most important, even if someone gives you what seems like overwhelming evidence in favor of a certain point of view, don’t trust it until you’ve done a simple Google search to see if the opposite side has equally overwhelming evidence.

Imagine that somebody wrote:

Some of my friends support Ron Paul. I think that’s wrong. After all, he’s a libertarian, and Wikipedia says a libertarian is a person who believes in free will. But free will is impossible in a deterministic universe. Ron Paul’s belief in free will is clearly why there are so few Swiss people among Ron Paul supporters, since Swiss people are Calvinists and so understand determinism better.

This is sort of how I feel reading Why I Am Not A Rationalist on Almost Diamonds.

I’m having trouble not quoting it in full:

I’m not a rationalist because I’m an empiricist. I find no value in “logical” arguments that are based in intuition and “common sense” rather than data. Such arguments can only perpetuate ignorance by giving it a shiny veneer of reason that it hasn’t earned.

I boggle that we haven’t sorted this out yet. I particularly boggle that atheists of my acquaintance promote rationalism over empiricism. The tensions between basic rationalism and empiricism parallel the tensions between church theology and the philosophy of science. We have no problem rejecting church theology as not being grounded in evidence. Why do so many atheists praise rationalism?

Let me stop here and make it clear that I’m not rejecting logic or critical thinking. Goodness knows that I’ve spent hours just this summer helping people share useful heuristics that will, in general, help them get to the right answers more often. I’ve led workshops and panels on evaluating science journalism and scientific results. When I’ve spoken to comparative religion classes in the past, I’ve talked about religious skepticism with an emphasis on the basics of epistemology.

The problem isn’t logic or critical thinking. The problem is a tendency to view those skills as central to getting the right answers. The problem is a tendency to view them as the solution. They’re not, and the idea that they are is in distinct contrast with the way humanity has actually grown in knowledge and understanding of the world.

Rationalism is, at heart, an individualist endeavor. It says that the path to getting things right lies in improving the self, improving the thinking of one person at a time. It’s not surprising that the ideology and movement appeal largely to the young, to men, to white people, to libertarians. It focuses primarily on individual action.

That’s not how we’ve come to learn about our world, though. It’s not how science or any other field of scholarship works. Scholarship is a collaborative process. And I don’t just mean peer review and working groups, though those are important as well.

Scholars add to our knowledge of the world by building on the work of others. They apply tools and methods developed by others to new material and questions. They study the work of other scholars to inspire them and give them the background to ask and answer new questions. They evaluate the work of others and consolidate the best of it into larger theoretical frameworks. Without the work of scholars before them, scholars today and evermore would always be recreating basic work and basic errors.

All too often, I find rationalists taking this repetitive approach. They think but they don’t study. As a consequence, they repeat the same naïve errors time and again. This is particularly noticeable when they engage in social or political theorizing by extrapolating from information they learned in secondary school and 101-level college classes, picked up in pop culture, or provided by people pushing a political cause. Their conclusions are necessarily as limited as their source material and reflect all its cultural biases.

As best I can tell, it is conflating a misunderstood version of rationalism (Descartes) with a misunderstood version of rationalism (Yudkowsky) and ending up with something unrelated to either, in the most bizarre possible way. But since there are commenters there who seem to agree with it, better nip this in the bud before it spreads.

II.

Rationalism (Descartes) is not simply the belief that sitting and thinking is more useful than observation. Descartes-style rationalism is complicated, but involves the claim that certain concepts are known prior to experience. For example, it is possible to understand mathematical truths like 1 + 1 = 2 separately from our experience of observing people add one apple to another. It is also possible to know them more completely than our knowledge of the external world, since our external senses can only tell us that all additions of one plus one have equalled two so far, but our reason can tell us something we have never observed – that it is necessarily true that everywhere and for all time 1 + 1 will equal two.

This so obviously gets bogged down in definitions of what is or isn’t “prior to experience” or a “concept” that philosophers today have mostly moved on to bigger and better things like hitting people with trolleys. It has nevertheless gotten a mild boost of interest recently with Chomsky’s claim that some features of human language are innate, and evolutionary psychology’s claim that certain preferences like fear of spiders may be innate. You can learn much more than you wanted to know at the Stanford Encyclopedia of Philosophy

But in no case does the debate ever resemble Almost Diamonds’ naive conception of some people thinking they have to study the world and other people sitting and speculating in armchairs and playing at self-improvement because they don’t want to get their hands dirty in the real world. In fact, Descartes himself was a devoted experimentalist – probably too devoted. His intense interest in anatomy combined with his belief that animals lack souls made him one of the most prolific vivisectionists of all time. We can thank him for such useful pieces of scientific information as “If you cut off the end of the heart of a living dog and insert your finger through the incision into one of the concavities, you will clearly feel that every time the heart shortens, it presses your finger, and stops pressing every time it lengthens.” One can accuse the author of this statement of a lot of things, but “not willing to get his hands dirty” isn’t one of them.

Likewise, Leibniz, the other most famous partisan of rationalism of all time, also made notable contributions to physics, geology, embryology, paleontology, and medicine. Either he was out exploring the world, or he had some armchair. Particularly ironically for Almost Diamonds’ thesis, he was one of the most prominent advocates of research as a collaborative endeavor, and founded various scientific discussion societies around Europe as well as calling for a giant international database of all scientific findings. All of this was entirely consistent with, and informed by, his rationalism.

The people on r/philosophy also do a good job of explaining this mistake in that blog’s conception of rationalism (the people on r/badphilosophy do an, um, less good job)

III.

But I think Almost Diamonds is mostly talking (also talking?) about rationalism (Yudkowsky), ie internet rationalism. After all, she mentions “the rationalist movement” and says they’re about “understanding cognitive biases” and “appeal largely to…libertarians”.

This is even more wrong.

At least rationalism (Descartes) is sort of about some kind of disconnect with empirical evidence. In the context of rationalism (Yudkowsky) this is about the same level of error as expecting Ron Paul to be a philosopher preaching free will just because “libertarianism” can mean something in metaphysics. Rationalism (Yudkowsky) and rationalism (Descartes) share a name, nothing more.

Almost Diamonds says:

I’m not a rationalist because I’m an empiricist. I find no value in “logical” arguments that are based in intuition and “common sense” rather than data…I boggle that we haven’t sorted this out yet. I particularly boggle that atheists of my acquaintance promote rationalism over empiricism.

Meanwhile, the founding document of rationalism (Yudkowsky), the Twelve Virtues of Rationality, states:

The sixth virtue is empiricism. The roots of knowledge are in observation and its fruit is prediction. What tree grows without roots?…Do not ask which beliefs to profess, but which experiences to anticipate.

It adds:

You cannot make a true map of a city by sitting in your bedroom with your eyes shut and drawing lines upon paper according to impulse. You must walk through the city and draw lines on paper that correspond to what you see. If, seeing the city unclearly, you think that you can shift a line just a little to the right, just a little to the left, according to your caprice, this is just the same mistake.

This makes the same point as Almost Diamonds.

Now, granted, some movements can have Official Founding Beliefs that they don’t follow. Many rationalists take the Virtues very seriously (one just let me know it is hanging on the wall of his group house) but perhaps like some of Jesus’ more lovey-dovey commandments or the inconvenient parts of the Constitution, they are honored more in the breach than in the observance?

I don’t think so. Rationalists have taken this idea and run with it, which is why we are so obsessed with things like making beliefs pay rent in experience, discussion of “Bayesian updating”, and even making monetary bets on our beliefs to train ourselves to make sure they conform to real world outcomes. It’s why the rationalist proverb, upon being given a cool theory, goes “Name three examples”.

A typical (okay, I lied, highly extreme) example is Gwern, who consumed pretty much every chemical and then carefully recorded its effects on his sleep, emotions, performance on cognitive tests, et cetera and then performed Bayesian analysis on it. There’s obviously something wrong with that, but it’s not lack of empiricism!

Diamonds:

All too often, I find rationalists taking this repetitive approach. They think but they don’t study.

Twelve Virtues again:

The eleventh virtue is scholarship. Study many sciences and absorb their power as your own. Each field that you consume makes you larger. If you swallow enough sciences the gaps between them will diminish and your knowledge will become a unified whole. If you are gluttonous you will become vaster than mountains.

And in accordance with this, I will put the rationalist movement up, mano a mano, against any other movement on the entire Internet in terms of the quality of scholarship and empiricism.

Like, holy @#$%, we have Luke Muehlhauser, who can’t write a simple life hacks post on productivity without fifty-seven different journal article citations, and who writes at great length about how to study a field of research effectively, the best textbooks on every subject, software tools for efficient scholarship, etc, etc, etc.

We have a community-wide survey that collects information on one hundred thirty-six demographic categories, for over fifteen hundred community members, and then a tradition of obsessively arguing about the implications of the results for several weeks every year.

We know that 20% of rationalists over the age of 35 have Ph. Ds. 54% have either a Ph. D, an MD, or a Master’s!

And not to toot my own horn, but there’s a reason this blog’s series of impromptu literature reviews is called “Much More Than You Wanted To Know” and has investigated the literature on things like marijuana legalization and SSRI effectiveness, fifty or sixty studies per review, to a degree that’s gotten some coverage on major news sites including Andrew Sullivan’s blog and Vox.

And…wait a second! The author of that blog knows Kate Donovan! How do you know Kate Donovan and still accuse rationalists of “not studying”?!?! DO YOU EVEN HAVE EYES?

Finally, Almost Diamonds says:

Even in the modern rationalist movement, which speaks more to collecting evidence than classical rationalism, I have yet to see any emphasis on epistemic humility.

But the Twelve Virtues says:

The eighth virtue is humility. To be humble is to take specific actions in anticipation of your own errors. To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty. Who are most humble? Those who most skillfully prepare for the deepest and most catastrophic errors in their own beliefs and plans. Because this world contains many whose grasp of rationality is abysmal, beginning students of rationality win arguments and acquire an exaggerated view of their own abilities. But it is useless to be superior: Life is not graded on a curve. The best physicist in ancient Greece could not calculate the path of a falling apple. There is no guarantee that adequacy is possible given your hardest effort; therefore spare no thought for whether others are doing worse. If you compare yourself to others you will not see the biases that all humans share. To be human is to make ten thousand errors. No one in this world achieves perfection.

Really? “Yet to see any emphasis on epistemic humility”? The most important mission statement of the rationalist movement says that one of the movement’s twelve founding principles is humility, then waxes rhapsodic about it. Seriously, we’re the people who keep calling ourselves “aspiring rationalists” to remind ourselves that we’re not nearly as rational as we should be yet! We’re the people who obsessively calibrate with Prediction Book et cetera to remind ourselves just how high our error rate is. We’re the people who keep a community-wide keep a mistakes repository (with Gwern once again going above and beyond). THERE ARE FORTY ONE DIFFERENT POSTS ON LESS WRONG TAGGED ‘OVERCONFIDENCE’!

Almost Diamonds dislikes rationalism because she believes in an emphasis of empiricism over armchair speculation, careful scholarship over ignorance, and epistemic humility. But she’s just described the rationalist movement almost to a ‘T’! She’s attacking the rationalist movement for not living up to her ideal philosophy which is…the precise philosophy of rationalist movement!

This is mean, but I’m going to say it. Almost Diamonds describes the rationalist movement in a way that even the most cursory glance at any rationalist site or document would disprove. Her opinion seems to be based entirely on a distorted idea of the dictionary definition of the word “rationalism”.

It’s almost like she’s, I don’t know, sitting in an armchair speculating about what rationalism must be, rather than going out and looking for evidence.

ಠ_ಠ

IV.

Also, can I just mention that one of the commenters on that blog says that the problems with the rationalist movement are a lot like the problems with frequentist statistics, and what would really help them is if they investigated Bayesianism? I swear I am not joking. I swear this is a thing that happened.

V.

But aside from all this, I do think there’s an important point that needs to be made here. That is – given that empiricism and scholarship is obviously super-important, why is it not enough?

The very short answer is “A meta-analysis of hundreds of studies is what tells you that psychic powers exist. Critical thinking is what helps you figure out whether to trust that result.”

The longer answer: rationality is about drawing correct inferences from limited, confusing, contradictory, or maliciously doctored facts. Even the world’s most stubborn creationist would have to realize the truth of evolution if you could put her in a time machine and make her watch all 3 billion years of life on Earth. But more rational people can realize the truth of evolution after reading a couple of good biology textbooks and having some questions answered. And Darwin could realize the truth of evolution just by observing the natural world and speculating about finches. There’s something I do better than the creationist and Darwin does better than me, and it’s not “have access to data”.

Life is made up of limited, confusing, contradictory, and maliciously doctored facts. Anyone who says otherwise is either sticking to such incredibly easy solved problems that they never encounter anything outside their comfort level, or so closed-minded that they shut out any evidence that challenges their beliefs.

Given this state of affairs, obviously it’s useful to have as much evidence as possible, in the same way it’s useful to have as much money as possible. But equally obviously it’s useful to be able to use a limited amount of evidence wisely, in the same way it’s useful to be able to use a limited amount of money wisely.

I recently reviewed thirty-five studies on racism in the criminal justice system, a very controversial topic. But I would suggest that almost nobody would change their opinion about this based on simple number of studies reviewed. That is, if a person who has read five studies and believes the system is racist encountered another person who has read ten studies and believes the system is fair, she would not simply say “Well, you’ve read more studies than I have, so I guess you’re right and I’m wrong.” She would probably say “That’s interesting, but I need to double-check the methodologies of those studies, make sure they mean what you think they mean, make sure you haven’t specifically selected only studies that prove your view, and make sure you haven’t fallen into one of a million other possible failure modes.”

The part where you have read 5 studies but I have read 10 is the empiricism that Almost Diamonds would say is the only meaningful skill that exists. The part where we want to make sure they’re good studies and I understood them right is rationality. I would trust the opinion of a rational person who knows one study far more than that of an irrational person who knows fifty. If you don’t believe me, I invite you to check out the hundreds of studies published in creation science and homeopathy journals every year.

Or what if you’re working in an area where you don’t even have hypotheses yet? It’s your job to explain or predict something that’s never been explained or predicted before. Sure, you’ve got to have a background level of expertise and scholarship, but no matter how many x-ray crystallographers you have somebody has to be the one to say “You know, our data would make sense if this molecule were in the shape of a helix.” What if you’re trying to predict the future – like in what year fusion power will become a reality, or whether a stock is going to go up and down – and you’ve already reviewed all of the relevant evidence? What then?

If somebody says that rationality is all nice and well, but not really important because you can just use the facts, then this is the surest sign of somebody who doesn’t possess the skill and doesn’t even realize there is a skill there to be possessed. They have inoculated themselves with the cowpox of doubt, trained themselves on easy problems so long that they’ve dulled their senses and forgotten that problems that require more thought than just looking up the universally-agreed-upon scientific consensus in Wikipedia even exist.

There is one paragraph for which I will give Almost Diamonds credit: she is partly right when she says rationality is fundamentally an individual endeavor. I mean, only in the sense martial arts is an individual endeavor – you can train with lots of people, you have to train with lots of people, you’ve got to learn the craft from others and stand on the shoulders of giants – but in the end you’ve got to punch the other guy yourself.

Thousands of scientists have worked their entire lives to get you the evidence in favor of evolution. But thousands of creationists have worked their entire lives to obfuscate and confuse that evidence. Thousands of scientists have studied the criminal justice system, but many of them aren’t very good at it, many of them disagree with one another, and very likely none of them have worked on the exact subsubproblem that you’re interested in. Other people can present the facts to you, but in the end you’re the one deciding what and who to believe. Just like everybody dies alone, everybody decides on their beliefs alone. And rationality is what allows them to do that accurately.

If you’re investigating a problem even slightly more interesting than evolution versus creationism, you will always encounter limited, confusing, contradictory, and maliciously doctored facts. The more rationality you have, the greater your ability to draw accurate conclusions from this mess. And the differences aren’t subtle

A superintelligence can take a grain of sand and envision the entire universe.

Einstein can take a few basic facts about light and gravity and figure out the theory of relativity.

I can take a bunch of conflicting studies and feel sort of confident I’ve at least figured out the gist of the topic.

Some people can’t take a movement that emphasizes on its founding document “OUR VIRTUES ARE EMPIRICISM, SCHOLARSHIP, AND HUMILITY” and figure out that it considers empiricism, scholarship, and humility to be virtues.

“Silliest internet atheist argument” is a hotly contested title, but I have a special place in my heart for the people who occasionally try to prove Biblical fallibility by pointing out whales are not a type of fish.

(this is going to end up being a metaphor for something. Yup, we’re back to Whale Metaphor Blogging.)

The argument goes like this. Jonah got swallowed by a whale. But the Bible says Jonah got swallowed by a big fish. So the Bible seems to think whales are just big fish. Therefore the Bible is fallible. Therefore, the Bible was not written by God.

The first problem here is that “whale” is just our own modern interpretation of the Bible. For all we know, Jonah was swallowed by a really really really big herring.

The second problem is that if the ancient Hebrews want to call whales a kind of fish, let them call whales a kind of fish.

I’m not making the weak and boring claim that since they’d never discovered genetics they don’t know better. I am making the much stronger claim that, even if the ancient Hebrews had taken enough of a break from murdering Philistines and building tabernacles to sequence the genomes of all knownspecies of aquatic animals, there’s nothing whatsoever wrong, false, or incorrect with them calling a whale a fish.

Now, there’s something wrong with saying “whales are phylogenetically just as closely related to bass, herring, and salmon as these three are related to each other.” What’s wrong with the statement is that it’s false. But saying “whales are a kind of fish” isn’t.

Suppose you travel back in time to ancient Israel and try to explain to King Solomon that whales are a kind of mammal and not a kind of fish.

Your translator isn’t very good, so you pause to explain “fish” and “mammal” to Solomon. You tell him that fish is “the sort of thing herring, bass, and salmon are” and mammal is “the sort of thing cows, sheep, and pigs are”. Solomon tells you that your word “fish” is Hebrew dag and your word “mammal” is Hebrew behemah.

So you try again and say that a whale is a behemah, not a dag. Solomon laughs at you and says you’re an idiot.

You explain that you’re not an idiot, that in fact all kinds of animals have things called genes, and the genes of a whale are much closer to those of the other behemah than those of the dag.

Solomon says he’s never heard of these gene things before, and that maybe genetics is involved in your weird foreign words “fish” and “mammal”, but dag are just finned creatures that swim in the sea, and behemah are just legged creatures that walk on the Earth.

(like the kelev and the parah and the gavagai)

You try to explain that no, Solomon is wrong, dag are actually defined not by their swimming-in-sea-with-fins-ness, but by their genes.

Solomon says you didn’t even know the word dag ten minutes ago, and now suddenly you think you know what it means better than he does, who has been using it his entire life? Who died and made you an expert on Biblical Hebrew?

You try to explain that whales actually have tiny little hairs, too small to even see, just as cows and sheep and pigs have hair.

Solomon says oh God, you are so annoying, who the hell cares whether whales have tiny little hairs or not. In fact, the only thing Solomon cares about is whether responsibilities for his kingdom’s production of blubber and whale oil should go under his Ministry of Dag or Ministry of Behemah. The Ministry of Dag is based on the coast and has a lot of people who work on ships. The Ministry of Behemah has a strong presence inland and lots of of people who hunt on horseback. So please (he continues) keep going about how whales have little tiny hairs.

It’s easy to see that Solomon has a point, and that if he wants to define behemah as four-legged-land-dwellers that’s his right, and no better or worse than your definition of “creatures in a certain part of the phylogenetic tree”. Indeed, it might even be that if you spent ten years teaching Solomon all about the theory of genetics and evolution (which would be hilarious – think how annoyed the creationists would get) he might still say “That’s very interesting, and I can see why we need a word to describe creatures closely related along the phylogenetic tree, but make up your own word, because behemah already means ‘four-legged-land-dweller’.”

Now imagine that instead of talking to King Solomon, you’re talking to that guy from Duck Dynasty with the really crazy beard (I realize that may describe more than one person), who stands in for all uneducated rednecks in the same way King Solomon stands in for all Biblical Hebrews.

“Ah course a whale is a feesh, ya moron” he says in his heavy Southern accent.

“No it isn’t,” you say. “A fish is a creature phylogenetically related to various other fish, and with certain defining anatomical features. It says so right here in this biology textbook.”

“Well,” Crazy Beard Guy tells you, “Ah reckon that might be what a fish is, but a feesh is some’in that swims in the orshun.”

With a sinking feeling in your stomach, you spend ten years turning Crazy Beard Guy into a world expert on phylogenetics and evolutionary theory. Although the Duck Dynasty show becomes much more interesting, you fail to budge him a bit on the meaning of “feesh”.

It’s easy to see here that “fish” and “feesh” can be different just as “fish” and “dag” can be different.

You can point out how many important professors of icthyology in fancy suits use your definition, and how only a couple of people with really weird facial hair use his. But now you’re making a status argument, not a factual argument. Your argument is “conform to the way all the cool people use the word ‘fish'”, not “a whale is really and truly not a fish”.

There are facts of the matter on each individual point – whether a whale has fins, whether a whale lives in the ocean, whether a whale has tiny hairs, et cetera. But there is no fact of the matter on whether a whale is a fish. The argument is entirely semantic.

So this is the second reason why this particular objection to the Bible is silly. If God wants to call a whale a big fish, stop telling God what to do.

(also, bats)

II.

When terms are not defined directly by God, we need our own methods of dividing them into categories.

The essay “How An Algorithm Feels From The Inside” is a gift that keeps on giving. You can get a reputation as a daring and original thinker just by copy-pasting it at different arguments with a couple of appropriate words substituted for one another, mad-libs like. It is the solution to something like 25% of extent philosophical problems.

It starts with a discussion of whether or not Pluto is a planet. Planets tend to share many characteristics in common. For example, they are large, round, have normal shaped orbits lined up with the plane of the ecliptic, have cleared out a certain area of space, and are at least kind of close to the Sun as opposed to way out in the Oort Cloud.

One could imagine a brain that thought about these characteristics like this:

One could imagine this model telling you everything you need to know. If an object is larger, it’s more likely to be round and in cis-Neptunian space. If an object has failed to clear its orbit of debris, it’s more likely to have a skewed orbit relative to the plane of the ecliptic. We could give each of these relationships Bayesian weights and say things like large objects have a 32% chance of being in cis-Neptunian space and small objects an 86% chance. Or whatever.

But this model has some big problems. For one thing, if you inscribe it in blood, you accidentally summon the Devil. But second, it’s computationally very complicated. Each attribute affects each other attribute which affects it in turn and so on in an infinite cycle, so that its behavior tends to be chaotic and unpredictable.

What the human brain actually seems to do is to sweep all common correlations into one big category in the middle, thus dividing possibility-space into large round normal-orbit solitary inner objects, and small irregular skewed-orbit crowded outer objects. It calls the first category “planets” and the second category “planetoids”.

Obligatory Less Wrong picture

You can then sweep minor irregularities under the rug. Neptune is pretty far from the sun, but since it’s large, round, normal-orbit, and solitary, we know which way the evidence is leaning.

When an object satisfies about half the criteria for planet and half the criteria for planetoid, then it’s awkward. Pluto is the classic example. It’s relatively large, round, skewed orbit, solitary…ish? and outer-ish. What do you do?

The practical answer is you convene some very expensive meeting of prestigious astronomers and come to some official decision which everyone agrees to follow so they’re all on the same page.

But the ideal answer is you say “Huh, the assumption encoded in the word ‘planet’ that the five red criteria always went together and the five blue criteria always went together doesn’t hold. Whatever.”

Then you divide the solar system into three types of objects: planets, planetoids, and dammit-our-categorization-scheme-wasn’t-as-good-as-we-thought.

(psychiatry, whose philosophy of categorization is light years ahead of a lot of the rest of the world, conveniently abbreviates this latter category as “NOS”)

The situation with whales and fish is properly understood in the same context. Fish and mammals differ on a lot of axes. Fish generally live in the water, breathe through gills, have tails and fins, possess a certain hydrodynamic shape, lay eggs, and are in a certain part of the phylogenetic tree. Mammals generally live on land, breathe through lungs, have legs, give live birth, and are in another part of the phylogenetic tree. Most fish conform to all of the fish desiderata, and most mammals conform to all of the mammal desiderata, so there’s no question of how to categorize them. Occasionally you get something weird (a platypus, a lungfish, or a whale) and it’s a judgment call which you have to decide by fiat. In our case, that fiat is “use genetics and ignore all other characteristics” but some other language, culture, or scientific community might make a different fiat, and then the borders between their categories would look a little bit different.

III.

Since I shifted to a borders metaphor, let’s follow that and see where it goes.

Imagine that Israel and Palestine agree to a two-state solution with the final boundary to be drawn by the United Nations. You’re the head of the United Nations committee involved, so you get out a map and a pencil. Both sides have sworn by their respective gods to follow whatever you determine.

Your job is not to draw “the correct border”. There is no one correct border between Israel and Palestine. There are a couple of very strong candidates (for example, the pre-1967 line of control), but both countries have suggested deviations from that (most people think an actual solution would involve Palestine giving up some territory that has since been thoroughly settled by Israel in exchange for some territory within Israel proper, or perhaps for a continuous “land bridge” between the West Bank and Gaza). Even if you wanted to use the pre-1967 line as a starting point, there would still be a lot of work to do deciding what land swaps should and shouldn’t be made.

Instead you’d be making a series of trade-offs. Giving all of Jerusalem to the Israelis would make them very happy but anger Palestine. Creating a contiguous corridor between Gaza and the West Bank makes some sense, but then you’d be cutting off Eilat from the rest of Israel. Giving all of the Israeli settlements in the West Bank back to Palestine would satisfy a certain conception of property rights, but also leave a lot of Jews homeless.

There are also much stupider decisions you could make. You could give Tel Aviv to Palestine. You could make the Palestinian state a perfect circle five miles in radius centered on Rishon LeZion. You could just split the territory in half with a straight line, and give Israel the north and Palestine the south. All of these things would be really dumb.

But, crucially, they would not be false. They would not be factually incorrect. They would just be failing to achieve pretty much any of the goals that we would expect a person solving land disputes in the Middle East to have. You can think of alternative arrangements in which these wouldn’t be dumb. For example, if you’re a despot, and you want to make it very clear to both the Israelis and Palestinians that their opinions don’t matter and they should stop bothering you with annoying requests for arbitration, maybe splitting the country in half north-south is the way to go.

This is now unexpectedly a geography blog again.

The border between Turkey and Syria follows a mostly straight-ish line near-ish the 36th parallel, except that about twenty miles south of the border Turkey controls a couple of square meters in the middle of a Syrian village. This is the tomb of the ancestor of the Ottoman Turks, and Turkey’s border agreement with Syria stipulates that it will remain part of Turkey forever. And the Turks take this very seriously; they maintain a platoon of special forces there and have recently been threatening war against Syria if their “territory” gets “invaded” in the current conflict.

Pictured: Turkey (inside fence), Syria (outside)

The border between Bangladesh and India is complicated at the best of times, but it becomes absolutely ridiculous in a place called Cooch-Behar, which I guess is as good a name as any for a place full of ridiculous things. In at least one spot there is an ‘island’ of Indian territory within a larger island of Bangladeshi territory within a larger island of Indian territory within Bangladesh. According to mentalfloss.com:

So why’d the border get drawn like that? It can all be traced back to power struggles between local kings hundreds of years ago, who would try to claim pockets of land inside each other’s territories as a way to leverage political power. When Bangladesh became independent from India in 1947 (as East Pakistan until 1971), all those separate pockets of land were divvied up. Hence the polka-dotted mess.

Namibia is a very weird-looking country with a very thin three-hundred-mile-long panhandle (eg about twice as long as Oklahoma’s). Apparently during the Scramble For Africa, the Germans who colonized Namibia really wanted access to the Zambezi River so they could reach the Indian Ocean and trade their colonial resources. They kept pestering the British who colonized Botswana until the Brits finally agreed to give up a tiny but very long strip of territory ending at the riverbank. This turned out to be not so useful, as just after Namibia’s Zambezi access sits Victoria Falls, the largest waterfall in the world – meaning that any Germans who tried to traverse the Zambezi to reach the Indian Ocean would last a matter of minutes before suddenly encountering a four hundred foot drop and falling to pretty much certain death. The moral of the story is not to pester the British Empire too much, especially if they’ve explored Africa and you haven’t.

But the other moral of the story is that borders are weird. Although we think of borders as nice straight lines that separate people of different cultures, they can form giant panhandles, distant islands, and enclaves-within-enclaves-within-enclaves. They can depart from their usual course to pay honor to national founders, to preserve records of ancient conquests, or to connect to trade routes.

Hume’s ethics restrict “bad” to an instrumental criticism – you can condemn something as a bad way to achieve a certain goal, but not as morally bad independent of what the goal is. In the same way, borders can be bad at fulfilling your goals in drawing them, but not bad in an absolute sense or factually incorrect. Namibia’s border is bad from the perspective of Germans who want access to the Indian Ocean. But it’s excellent from the perspective of Englishmen who want to watch Germans plummet into the Lower Zambezi and get eaten by hippos.

Breaking out of the metaphor, the same is true of conceptual boundaries. You may draw the boundaries of the category “fish” any way you want. A category “fish” containing herring, dragonflies, and asteroids is going to be stupid, but only in the same sense that a Palestinian state centered around Tel Aviv would be stupid – it fails to fulfill any conceivable goals of the person designing it. Categories “fish” that do or don’t include whales may be appropriate for different people’s purposes, the same way Palestinians might argue about whether the borders of their state should be optimized for military defensibility or for religious/cultural significance.

Statements like “the Zambezi River is full of angry hippos” are brute facts. Statements like “the Zambezi River is the territory of Namibia” are negotiable.

In the same way, statements like “whales have little hairs” are brute facts. Statements like “whales are not a kind of fish” are negotiable.

So it’s important to keep these two sorts of statements separate, and remember that in no case can an agreed-upon set of borders or a category boundary be factually incorrect.

IV.

I usually avoid arguing LGBT issues on here, not because I don’t have strong opinions about them but because I assume so many of my readers already agree with me that it would be a waste of time. I’m pretty sure I’m right about this – on the recent survey, readers of this blog who were asked to rate their opinion of gay marriage from 1 (strongly against) to 5 (strongly in favor) gave an average rating of 4.32.

Nevertheless, I’ve seen enough anti-transgender comments recently that the issue might be worth a look.

In particular, I’ve seen one anti-transgender argument around that I take very seriously. The argument goes: we are rationalists. Our entire shtick is trying to believe what’s actually true, not on what we wish were true, or what our culture tells us is true, or what it’s popular to say is true. If a man thinks he’s a woman, then we might (empathetically) wish he were a woman, other people might demand we call him a woman, and we might be much more popular if we say he’s a woman. But if we’re going to be rationalists who focus on believing what’s actually true, then we’ve got to call him a man and take the consequences.

Thus Abraham Lincoln’s famous riddle: “If you call a tail a leg, how many legs does a dog have?” And the answer: “Four – because a tail isn’t a leg regardless of what you call it.”

(if John Wilkes Booth had to suffer through that riddle, then I don’t blame him)

I take this argument very seriously, because sticking to the truth really is important. But having taken it seriously, I think it’s seriously wrong.

An alternative categorization system is not an error, and borders are not objectively true or false.

Just as we can come up with criteria for a definition of “planet”, we can come up with a definition of “man”. Absolutely typical men have Y chromosomes, have male genitalia, appreciate manly things like sports and lumberjackery, are romantically attracted to women, personally identify as male, wear male clothing like blue jeans, sing baritone in the opera, et cetera.

Some people satisfy some criteria of manhood and not others, in much the same way that Pluto satisfies only some criteria of planethood and whales satisfy only some criteria of mammalhood. For example, gay men might date other men and behave in effeminate ways. People with congenital androgen insensitivity syndrome might have female bodies, female external genitalia, and have been raised female their entire life, but when you look into their cells they have Y chromosomes.

Biologists defined by fiat that in cases of ambiguous animal grouping like whales, phylogenetics will be the tiebreaker. This was useful to resolve ambiguity, and it’s worth sticking to as a Schelling point so everyone’s using their words the same way, but it’s kind of arbitrary and mostly based on biologists caring a lot about phylogenetics. If we let King Solomon make the decision, he might decide by fiat that whether animals lived in land or water would be the tiebreaker, since he’s most interested in whether the animal is hunted on horseback or by boat.

Likewise, astronomers decided by fiat that something would be a planet if and only if meets the three criteria of orbiting, round, and orbit-clearing. But here we have a pretty neat window into how these kinds of decisions take place – you can read the history of the International Astronomical Union meeting where they settled on the definition and learn about all the alternative proposals that were floated and rejected and which particular politics resulted in the present criteria being selected among all the different possibilities. Here it is obvious that the decision was by fiat.

Without the input of any prestigious astronomers at all, most people seem to assume that the ultimate tiebreaker in man vs. woman questions is presence of a Y chromosome. I’m not sure this is a very principled decision, because I expect most people would classify congenital androgen insensitivity patients (XY people whose bodies are insensitive to the hormone that makes them look male, and so end up looking 100% female their entire lives and often not even knowing they have the condition) as women.

The project of the transgender movement is to propose a switch from using chromosomes as a tiebreaker to using self-identification as a tiebreaker.

(This isn’t actually the whole story – some of the more sophisticated people want to split “sex” and “gender”, so that people who want to talk about what chromosomes they’ve got have a categorization system to do that with, and a few people even want to split “chromosomal sex” and “anatomical sex” and “gender” and goodness knows what else – and I support all of these as very important examples of the virtue of precision – but to a first approximation, they want to define gender as self-identification)

This is not something that can be “true” or “false”. It’s a boundary-redrawing project. It can make for some boundaries that look a little bit weird – like a small percent of men being able to get pregnant – but as far as weird boundaries go that’s probably not as bad as having a tiny exclave of Turkish territory in the middle of a Syrian village.

(Ozy tells me this is sort of what queer theory is getting at, but in a horrible unreadable postmodernist way. They assure me you’re better off just reading the darned Sequences.)

You draw category boundaries in specific ways to capture tradeoffs you care about. If you care about the sanctity of the tomb of your country’s founder, sometimes it’s worth having a slightly weird-looking boundary in order to protect and honor it. And if you care about…

I’ve lived with a transgender person for six months, so I probably should have written this earlier. But I’m writing it now because I just finished accepting a transgender man to the mental hospital. He alternates between trying to kill himself and trying to cut off various parts of his body because he’s so distressed that he is biologically female. We’ve connected him with some endocrinologists who can hopefully get him started on male hormones, after which maybe he’ll stop doing that and hopefully be able to lead a normal life.

If I’m willing to accept an unexpected chunk of Turkey deep inside Syrian territory to honor some random dead guy – and I better, or else a platoon of Turkish special forces will want to have a word with me – then I ought to accept an unexpected man or two deep inside the conceptual boundaries of what would normally be considered female if it’ll save someone’s life. There’s no rule of rationality saying that I shouldn’t, and there are plenty of rules of human decency saying that I should.

V.

I’ve made this argument before and gotten a reply something like this:

“Transgender is a psychiatric disorder. When people have psychiatric disorders, certainly it’s right to sympathize and feel sorry for them and want to help them. But the way we try to help them is by treating their disorder, not by indulging them in their delusion.”

I think these people expect me to argue that transgender “isn’t really a psychiatric disorder” or something. But “psychiatric disorder” is just another category boundary dispute, and one that I’ve already written enough about elsewhere. At this point, I don’t care enough to say much more than “If it’s a psychiatric disorder, then attempts to help transgender people get covered by health insurance, and most of the ones I know seem to want that, so sure, gender dysphoria is a psychiatric disorder.”

And then I think of the Hair Dryer Incident.

The Hair Dryer Incident was probably the biggest dispute I’ve seen in the mental hospital where I work. Most of the time all the psychiatrists get along and have pretty much the same opinion about important things, but people were at each other’s throats about the Hair Dryer Incident.

Basically, this one obsessive compulsive woman would drive to work every morning and worry she had left the hair dryer on and it was going to burn down her house. So she’d drive back home to check that the hair dryer was off, then drive back to work, then worry that maybe she hadn’t really checked well enough, then drive back, and so on ten or twenty times a day.

It’s a pretty typical case of obsessive-compulsive disorder, but it was really interfering with her life. She worked some high-powered job – I think a lawyer – and she was constantly late to everything because of this driving back and forth, to the point where her career was in a downspin and she thought she would have to quit and go on disability. She wasn’t able to go out with friends, she wasn’t even able to go to restaurants because she would keep fretting she left the hair dryer on at home and have to rush back. She’d seen countless psychiatrists, psychologists, and counselors, she’d done all sorts of therapy, she’d taken every medication in the book, and none of them had helped.

So she came to my hospital and was seen by a colleague of mine, who told her “Hey, have you thought about just bringing the hair dryer with you?”

And it worked.

She would be driving to work in the morning, and she’d start worrying she’d left the hair dryer on and it was going to burn down her house, and so she’d look at the seat next to her, and there would be the hair dryer, right there. And she only had the one hair dryer, which was now accounted for. So she would let out a sigh of relief and keep driving to work.

And approximately half the psychiatrists at my hospital thought this was absolutely scandalous, and This Is Not How One Treats Obsessive Compulsive Disorder, and what if it got out to the broader psychiatric community that instead of giving all of these high-tech medications and sophisticated therapies we were just telling people to put their hair dryers on the front seat of their car?

I, on the other hand, thought it was the best fricking story I had ever heard and the guy deserved a medal. Here’s someone who was totally untreatable by the normal methods, with a debilitating condition, and a drop-dead simple intervention that nobody else had thought of gave her her life back. If one day I open up my own psychiatric practice, I am half-seriously considering using a picture of a hair dryer as the logo, just to let everyone know where I stand on this issue.

Miyamoto Musashi is quoted as saying:

The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy’s cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him.

Likewise, the primary thing in psychiatry is to help the patient, whatever the means. Someone can concern-troll that the hair dryer technique leaves something to be desired in that it might have prevented the patient from seeking a more thorough cure that would prevent her from having to bring the hair dryer with her. But compared to the alternative of “nothing else works” it seems clearly superior.

And that’s the position from which I think a psychiatrist should approach gender dysphoria, too.

Imagine if we could give depressed people a much higher quality of life merely by giving them cheap natural hormones. I don’t think there’s a psychiatrist in the world who wouldn’t celebrate that as one of the biggest mental health advances in a generation. Imagine if we could ameliorate schizophrenia with one safe simple surgery, just snip snip you’re not schizophrenic anymore. Pretty sure that would win all of the Nobel prizes. Imagine that we could make a serious dent in bipolar disorder just by calling people different pronouns. I’m pretty sure the entire mental health field would join together in bludgeoning anybody who refused to do that. We would bludgeon them over the head with big books about the side effects of lithium.

Really, are you sure you want your opposition to accepting transgender people to be “I think it’s a mental disorder”?

VI.

Some people can’t leave well enough alone, and continue to push the mental disorder angle. For example:

There are a lot of things I could say here.

I could point out that trans-Napoleonism seem to be mysteriously less common than transgender.

I could relate this mysterious difference to the various heavily researched apparent biological correlates of transgender, including unusual variants of the androgen receptor, birth-sex-discordant sizes of various brain regions, birth-sex-discordant responses to various pheromones, high rates of something seemingly like body integrity identity disorder, and of course our old friend altered digit ratios. If our hypothetical trans-Napoleon came out of the womb wearing a French military uniform and clutching a list of 19th century Grand Armee positions in his cute little baby hands, I think I’d take him more seriously.

I could argue that questions about gender are questions about category boundaries, whereas questions about Napoleon – absent some kind of philosophical legwork that I would very much like to read – are questions of fact.

I could point out that if the extent of somebody’s trans-Napoleonness was wanting to wear a bicorne hat, and he was going to be suicidal his entire life if he couldn’t but pretty happy if I could, let him wear the damn hat.

I could just link people to other sites’ pretty good objections to the same argument.

But I think what I actually want to say is that there was once a time somebody tried pretty much exactly this, silly hat and all. Society shrugged and played along, he led a rich and fulfilling life, his grateful Imperial subjects came to love him, and it’s one of the most heartwarming episodes in the history of one of my favorite places in the world.

Sometimes when you make a little effort to be nice to people, even people you might think are weird, really good things happen.

Part of what bothers me – and apparently several others – about yesterday’s motte-and-bailey discussion is that here’s a fallacy – a pretty successful fallacy – that depends entirely on people not being entirely clear on what they’re arguing about. Somebody says God doesn’t exist. Another person objects that God is just a name for the order and beauty in the universe. Then this somehow helps defend the position that God is a supernatural creator being. How does that even happen?

“Sir, you’ve been accused of murdering your wife. We have three witnesses who said you did it. What do you have to say for yourself?”

“Well, your honor, I think it’s quite clear I didn’t murder the President. For one thing, he’s surrounded by Secret Service agents. For another, check the news. The President’s still alive.”

“Huh. For some reason I vaguely remember thinking you didn’t have a case. Yet now that I hear you talk, everything you say is incredibly persuasive. You’re free to go.”

While motte-and-bailey is less subtle, it seems to require a similar sort of misdirection. I’m not saying it’s impossible. I’m just saying it’s a fact that needs to be explained.

When everything works the way it’s supposed to in philosophy textbooks, arguments are supposed to go one of a couple of ways:

1. Questions of empirical fact, like “Is the Earth getting warmer?” or “Did aliens build the pyramids?”. You debate these by presenting factual evidence, like “An average of global weather station measurements show 2014 is the hottest year on record” or “One of the bricks at Giza says ‘Made In Tau Ceti V’ on the bottom.” Then people try to refute these facts or present facts of their own.

2. Questions of morality, like “Is it wrong to abort children?” or “Should you refrain from downloading music you have not paid for?” You can only debate these well if you’ve already agreed upon a moral framework, like a particular version of natural law or consequentialism. But you can sort of debate them by comparing to examples of agreed-upon moral questions and trying to maintain consistency. For exmaple, “You wouldn’t kill a one day old baby, so how is a nine month old fetus different?” or “You wouldn’t download a car.”

If you are very lucky, your philosophy textbook will also admit the existence of:

3. Questions of policy, like “We should raise the minimum wage” or “We should bomb Foreignistan”. These are combinations of competing factual claims and competing values. For example, the minimum wage might hinge on factual claims like “Raising the minimum wage would increase unemployment” or “It is very difficult to live on the minimum wage nowadays, and many poor families cannot afford food.” But it might also hinge on value claims like “Corporations owe it to their workers to pay a living wage,” or “It is more important that the poorest be protected than that the economy be strong.” Bombing Foreignistan might depend on factual claims like “The Foreignistanis are harboring terrorists”, and on value claims like “The safety of our people is worth the risk of collateral damage.” If you can resolve all of these factual and value claims, you should be able to agree on questions of policy.

None of these seem to allow the sort of vagueness of topic mentioned above.

II.

A question: are you pro-Israel or pro-Palestine? Take a second, actually think about it.

Some people probably answered pro-Israel. Other people probably answered pro-Palestine. Other people probably said they were neutral because it’s a complicated issue with good points on both sides.

Probably very few people answered: Huh? What?

This question doesn’t fall into any of the three Philosophy 101 forms of argument. It’s not a question of fact. It’s not a question of particular moral truths. It’s not even a question of policy. There are closely related policies, like whether Palestine should be granted independence. But if I support a very specific two-state solution where the border is drawn upon the somethingth parallel, does that make me pro-Israel or pro-Palestine? At exactly which parallel of border does the solution under consideration switch from pro-Israeli to pro-Palestinian? Do you think the crowd of people shouting and waving signs saying “SOLIDARITY WITH PALESTINE” have an answer to that question?

But it’s even worse, because this question covers much more than just the borders of an independent Palestinian state. Was Israel justified by responding to Hamas’ rocket fire by bombing Gaza, even with the near-certainty of collateral damage? Was Israel justified in building a wall across the Palestinian territories to protect itself from potential terrorists, even though it severely curtails Palestinian freedom of movement? Do Palestinians have a “right of return” to territories taken in the 1948 war? Who should control the Temple Mount?

These are four very different questions which one would think each deserve independent consideration. But in reality, what percent of the variance in people’s responses do you think is explained by a general “pro-Palestine vs. pro-Israel” factor? 50%? 75%? More?

In a way, when we round people off to the Philosophy 101 kind of arguments, we are failing to respect their self-description. People aren’t out on the streets saying “By my cost-benefit analysis, Israel was in the right to invade Gaza, although it may be in the wrong on many of its other actions.” They’re waving little Israeli flags and holding up signs saying “ISRAEL: OUR STAUNCHEST ALLY”. Maybe we should take them at face value.

This is starting to look related to the original question in (I). Why is it okay to suddenly switch points in the middle of an argument? In the case of Israel and Palestine, it might be because people’s support for any particular Israeli policy is better explained by a General Factor Of Pro-Israeliness than by the policy itself. As long as I’m arguing in favor of Israel in some way, it’s still considered by everyone to be on topic.

III.

Some moral philosophers got fed up with nobody being able to explain what the heck a moral truth was and invented emotivism. Emotivism says there are no moral truths, just expressions of little personal bursts of emotion. When you say “Donating to charity is good,” you don’t mean “Donating to charity increases the sum total of utility in the world,” or “Donating to charity is in keeping with the Platonic moral law” or “Donating to charity was commanded by God” or even “I like donating to charity”. You’re just saying “Yay charity!” and waving a little flag.

Seems a lot like how people handle the Israel question. “I’m pro-Israel” doesn’t necessarily imply that you believe any empirical truths about Israel, or believe any moral principles about Israel, or even support any Israeli policies. It means you’re waving a little flag with a Star of David on it and cheering.

So here is Ethnic Tension: A Game For Two Players.

Pick a vague concept. “Israel” will do nicely for now.

Player 1 tries to associate the concept “Israel” with as much good karma as she possibly can. Concepts get good karma by doing good moral things, by being associated with good people, by being linked to the beloved in-group, and by being oppressed underdogs in bravery debates.

“Israel is the freest and most democratic country in the Middle East. It is one of America’s strongest allies and shares our Judeo-Christian values.

Player 2 tries to associate the concept “Israel” with as much bad karma as she possibly can. Concepts get bad karma by committing atrocities, being associated with bad people, being linked to the hated out-group, and by being oppressive big-shots in bravery debates. Also, she obviously needs to neutralize Player 1’s actions by disproving all of her arguments.

“Israel may have some level of freedom for its most privileged citizens, but what about the millions of people in the Occupied Territories that have no say? Israel is involved in various atrocities and has often killed innocent protesters. They are essentially a neocolonialist state and have allied with other neocolonialist states like South Africa.”

The prize for winning this game is the ability to win the other three types of arguments. If Player 1 wins, the audience ends up with a strongly positive General Factor Of Pro-Israeliness, and vice versa.

Remember, people’s capacity for motivated reasoning is pretty much infinite. Remember, a motivated skeptic asks if the evidence compels them to accept the conclusion; a motivated credulist asks if the evidence allows them to accept the conclusion. Remember, Jonathan Haidt and his team hypnotized people to have strong disgust reactions to the word “often”, and then tried to hold in their laughter when people in the lab came up with convoluted yet plausible-sounding arguments against any policy they proposed that included the word “often” in the description.

I’ve never heard of the experiment being done the opposite way, but it sounds like the sort of thing that might work. Hypnotize someone to have a very positive reaction to the word “often” (for most hilarious results, have it give people an orgasm). “Do you think governments should raise taxes more often?” “Yes. Yes yes YES YES OH GOD YES!”

Once you finish the Ethnic Tension Game, you’re replicating Haidt’s experiment with the word “Israel” instead of the word “often”. Win the game, and any pro-Israel policy you propose will get a burst of positive feelings and tempt people to try to find some explanation, any explanation, that will justify it, whether it’s invading Gaza or building a wall or controlling the Temple Mount.

So this is the fourth type of argument, the kind that doesn’t make it into Philosophy 101 books. The trope namer is Ethnic Tension, but it applies to anything that can be identified as a Vague Concept, or paired opposing Vague Concepts, which you can use emotivist thinking to load with good or bad karma.

IV.

Now motte-and-bailey stands revealed:

Somebody says God doesn’t exist. Another person objects that God is just a name for the order and beauty in the universe. Then this somehow helps defend the position that God is a supernatural creator being. How does that even happen?

The two-step works like this. First, load “religion” up with good karma by pitching it as persuasively as possible. “Religion is just the belief that there’s beauty and order in the universe.”

Wait, I think there’s beauty and order in the universe!

“Then you’re religious too. We’re all religious, in the end, because religion is about the common values of humanity and meaning and compassion sacrifice beauty of a sunrise Gandhi Buddha Sufis St. Francis awe complexity humility wonder Tibet the Golden Rule love.”

Then, once somebody has a strongly positive General Factor Of Religion, it doesn’t really matter whether someone believes in a creator God or not. If they have any predisposition whatsoever to do so, they’ll find a reason to let themselves. If they can’t manage it, they’ll say it’s true “metaphorically” and continue to act upon every corollary of it being true.

(“God is just another name for the beauty and order in the universe. But Israel definitely belongs to the Jews, because the beauty and order of the universe promised it to them.”)

If you’re an atheist, you probably have a lot of important issues on which you want people to consider non-religious answers and policies. And if somebody can maintain good karma around the “religion” concept by believing God is the order and beauty in the universe, then that can still be a victory for religion even if it is done by jettisoning many traditionally “religious” beliefs. In this case, it is useful to think of the “order and beauty” formulation as a “motte” for the “supernatural creator” formulation, since it’s allowing the entire concept to be defended.

But even this is giving people too much credit, because the existence of God is a (sort of) factual question. From yesterday’s post:

Suppose we’re debating feminism, and I defend it by saying it really is important that women are people, and you attack it by saying that it’s not true that all men are terrible. What is the real feminism we should be debating? Why would you even ask that question? What is this, some kind of dumb high school debate club? Who the heck thinks it would be a good idea to say ‘Here’s a vague poorly-defined concept that mind-kills everyone who touches it – quick, should you associate it with positive affect or negative affect?!’

Who the heck thinks that? Everybody, all the time.

Once again, if I can load the concept of “feminism” with good karma by making it so obvious nobody can disagree with it, then I have a massive “home field advantage” when I’m trying to convince anyone of any particular policy that can go under the name “feminism”, even if it’s unrelated to the arguments that gave feminism good karma in the first place.

Or if I’m against feminism, I just post quotes from the ten worst feminists on Tumblr again and again until the entire movement seems ridiculous and evil, and then you’ll have trouble convincing anyone of anything feminist. “That seems reasonable…but wait, isn’t that a feminist position? Aren’t those the people I hate?”

(compare: most Americans oppose Obamacare, but most Americans support each individual component of Obamacare when it is explained without using the word “Obamacare”)

V.

Little flow diagram things make everything better. Let’s make a little flow diagram thing.

We have our node “Israel”, which has either good or bad karma. Then there’s another node close by marked “Palestine”. We would expect these two nodes to be pretty anti-correlated. When Israel has strong good karma, Palestine has strong bad karma, and vice versa.

Now suppose you listen to Noam Chomsky talk about how strongly he supports the Palestinian cause and how much he dislikes Israel. One of two things can happen:

“Wow, a great man such as Noam Chomsky supports the Palestinians! They must be very deserving of support indeed!”

or

“That idiot Chomsky supports Palestine? Well, screw him. And screw them!”

So now there is a third node, Noam Chomsky, that connects to both Israel and Palestine, and we have discovered it is positively correlated with Palestine and negatively correlated with Israel. It probably has a pretty low weight, because there are a lot of reasons to care about Israel and Palestine other than Chomsky, and a lot of reasons to care about Chomsky other than Israel and Palestine, but the connection is there.

I don’t know anything about neural nets, so maybe this system isn’t actually a neural net, but whatever it is I’m thinking of, it’s a structure where eventually the three nodes reach some kind of equilibrium. If we start with someone liking Israel and Chomsky, but not Palestine, then either that’s going to shift a little bit towards liking Palestine, or shift a little bit towards disliking Chomsky.

Now we add more nodes. Cuba seems to really support Palestine, so they get a positive connection with a little bit of weight there. And I think Noam Chomsky supports Cuba, so we’ll add a connection there as well. Cuba is socialist, and that’s one of the most salient facts about it, so there’s a heavily weighted positive connection between Cuba and socialism. Palestine kind of makes noises about socialism but I don’t think they have any particular economic policy, so let’s say very weak direct connection. And Che is heavily associated with Cuba, so you get a pretty big Che – Cuba connection, plus a strong direct Che – socialism one. And those pro-Palestinian students who threw rotten fruit at an Israeli speaker also get a little path connecting them to “Palestine” – hey, why not – so that if you support Palestine you might be willing to excuse what they did and if you oppose them you might be a little less likely to support Palestine.

Back up. This model produces crazy results, like that people who like Che are more likely to oppose Israel bombing Gaza. That’s such a weird, implausible connection that it casts doubt upon the entire…

Oh. Wait. Yeah. Okay.

I think this kind of model, in its efforts to sort itself out into a ground state, might settle on some kind of General Factor Of Politics, which would probably correspond pretty well to the left-right axis.

In Five Case Studies On Politicization, I noted how fresh new unpoliticized issues, like the Ebola epidemic, were gradually politicized by connecting them to other ideas that were already part of a political narrative. For example, a quarantine against Ebola would require closing the borders. So now there’s a weak negative link between “Ebola quarantine” and “open borders”. If your “open borders” node has good karma, now you’re a little less likely to support an Ebola quarantine. If “open borders” has bad karma, a little more likely.

I also tried to point out how you could make different groups support different things by changing your narrative a little:

Global warming has gotten inextricably tied up in the Blue Tribe narrative: Global warming proves that unrestrained capitalism is destroying the planet. Global warming disproportionately affects poor countries and minorities. Global warming could have been prevented with multilateral action, but we were too dumb to participate because of stupid American cowboy diplomacy. Global warming is an important cause that activists and NGOs should be lauded for highlighting. Global warming shows that Republicans are science denialists and probably all creationists. Two lousy sentences on “patriotism” aren’t going to break through that.

If I were in charge of convincing the Red Tribe to line up behind fighting global warming, here’s what I’d say:

In the 1950s, brave American scientists shunned by the climate establishment of the day discovered that the Earth was warming as a result of greenhouse gas emissions, leading to potentially devastating natural disasters that could destroy American agriculture and flood American cities. As a result, the country mobilized against the threat. Strong government action by the Bush administration outlawed the worst of these gases, and brilliant entrepreneurs were able to discover and manufacture new cleaner energy sources. As a result of these brave decisions, our emissions stabilized and are currently declining.

Unfortunately, even as we do our part, the authoritarian governments of Russia and China continue to industralize and militarize rapidly as part of their bid to challenge American supremacy. As a result, Communist China is now by far the world’s largest greenhouse gas producer, with the Russians close behind. Many analysts believe Putin secretly welcomes global warming as a way to gain access to frozen Siberian resources and weaken the more temperate United States at the same time. These countries blow off huge disgusting globs of toxic gas, which effortlessly cross American borders and disrupt the climate of the United States. Although we have asked them to stop several times, they refuse, perhaps egged on by major oil producers like Iran and Venezuela who have the most to gain by keeping the world dependent on the fossil fuels they produce and sell to prop up their dictatorships.

We need to take immediate action. While we cannot rule out the threat of military force, we should start by using our diplomatic muscle to push for firm action at top-level summits like the Kyoto Protocol. Second, we should fight back against the liberals who are trying to hold up this important work, from big government bureaucrats trying to regulate clean energy to celebrities accusing people who believe in global warming of being ‘racist’. Third, we need to continue working with American industries to set an example for the world by decreasing our own emissions in order to protect ourselves and our allies. Finally, we need to punish people and institutions who, instead of cleaning up their own carbon, try to parasitize off the rest of us and expect the federal government to do it for them.

In the first paragraph, “global warming” gets positively connected to concepts like “poor people and minorities” and “activists and NGOs”, and gets negatively connected to concepts like “capitalism”, “American cowboy diplomacy”, and “creationists”. That gives global warming really strong good karma if (and only if) you like the first two concepts and hate the last three.

In the next three paragraphs, “global warming” gets positively connected to “America”, “the Bush administration” and “entrepreneurs”, and negatively connected to “Russia”, “China”, “oil producing dictatorships like Iran and Venezuela”, “big government bureaucrats”, and “welfare parasites”. This is going to appeal to, well, a different group.

Notice two things here. First, the exact connection isn’t that important, as long as we can hammer in the existence of a connection. I could probably just say GLOBAL WARMING! COMMUNISM! GLOBAL WARMING! COMMUNISM! GLOBAL WARMING! COMMUNISM! several hundred times and have the same effect if I could get away with it (this is the principle behind attack ads which link a politician’s face to scary music and a very concerned voice).

Second, there is no attempt whatsoever to challenge the idea that the issue at hand is the positive or negative valence of a concept called “global warming”. At no point is it debated what the solution is, which countries the burden is going to fall on, or whether any particular level of emission cuts would do more harm than good. It’s just accepted as obvious by both sides that we debate “for” or “against” global warming, and if the “for” side wins then they get to choose some solution or other or whatever oh god that’s so boring can we get back to Israel vs. Palestine.

Some of the scientists working on IQ have started talking about “hierarchical factors”, meaning that there’s a general factor of geometry intelligence partially correlated with other things into a general factor of mathematical intelligence partially correlated with other things into a general factor of total intelligence.

I would expect these sorts of things to work the same way. There’s a General Factor Of Global Warming that affects attitudes toward pretty much all proposed global warming solutions, which is very highly correlated with a lot of other things to make a General Factor Of Environmentalism, which itself is moderately highly correlated with other things into the General Factor Of Politics.

VI.

Speaking of politics, a fruitful digression: what the heck was up with the Ashley Todd mugging hoax in 2008?

Back in the 2008 election, a McCain campaigner claimed (falsely, it would later turn out) to have been assaulted by an Obama supporter. She said he slashed a “B” (for “Barack”) on her face with a knife. This got a lot of coverage, and according to Wikipedia:

John Moody, executive vice president at Fox News, commented in a blog on the network’s website that “this incident could become a watershed event in the 11 days before the election,” but also warned that “if the incident turns out to be a hoax, Senator McCain’s quest for the presidency is over, forever linked to race-baiting.”

Wait. One Democrat, presumably not acting on Obama’s direct orders, attacks a Republican woman. And this is supposed to alter the outcome of the entire election? In what universe does one crime by a deranged psychopath change whether Obama’s tax policy or job policy or bombing-scary-foreigners policy is better or worse than McCain’s?

Even if we’re willing to make the irresponsible leap from “Obama is supported by psychopaths, therefore he’s probably a bad guy,” there are like a hundred million people on each side. Psychopaths are usually estimated at about 1% of the population, so any movement with a million people will already have 10,000 psychopaths. Proving the existence of a single one changes nothing.

I think insofar as this affected the election – and everyone seems to have agreed that it might have – it hit President Obama with a burst of bad karma. Obama something something psychopath with a knife. Regardless of the exact content of those something somethings, is that the kind of guy you want to vote for?

Then when it was discovered to be a hoax, it was McCain something something race-baiting hoaxer. Now he’s got the bad karma!

This sort of conflation between a cause and its supporters really only makes sense in the emotivist model of arguing. I mean, this shouldn’t even get dignified with the name ad hominem fallacy. Ad hominem fallacy is “McCain had sex with a goat, therefore whatever he says about taxes is invalid.” At least it’s still the same guy. This is something the philosophy textbooks can’t bring themselves to believe really exists, even as a fallacy.

But if there’s a General Factor Of McCain, then anything bad remotely connected to the guy – goat sex, lying campaigners, whatever – reflects on everything else about him.

This is the same pattern we see in Israel and Palestine. How many times have you seen a news story like this one: “Israeli speaker hounded off college campus by pro-Palestinian partisans throwing fruit. Look at the intellectual bankruptcy of the pro-Palestinian cause!” It’s clearly intended as an argument for something other than just not throwing fruit at people. The causation seems to go something like “These particular partisans are violating the usual norms of civil discussion, therefore they are bad, therefore something associated with Palestine is bad, therefore your General Factor of Pro-Israeliness should become more strongly positive, therefore it’s okay for Israel to bomb Gaza.” Not usually said in those exact words, but the thread can be traced.

VII.

Here is a prediction of this model: we will be obsessed with what concepts we can connect to other concepts, even when the connection is totally meaningless.

Suppose I say: “Opposing Israel is anti-Semitic”. Why? Well, the Israelis are mostly Jews, so in a sense by definition being anti- them is “anti-Semitic”, broadly defined. Also, p(opposes Israel|is anti-Semitic) is probably pretty high, which sort of lends some naive plausibility to the idea that p(is anti-Semitic|opposes Israel) is at least higher than it otherwise could be.

Maybe we do our research and we find exactly what percent of opponents of Israel endorse various anti-Semitic statements like “I hate all Jews” or “Hitler had some bright ideas”. We’ve replaced the symbol with the substance. Problem solved, right?

Maybe not. In the same sense that people can agree on all of the characteristics of Pluto – its diameter, the eccentricity of its orbit, its number of moons – and still disagree on the question “Is Pluto a planet”, one can agree on every characteristic of every Israel opponent and still disagree on the definitional question “Is opposing Israel anti-Semitic?”

(fact: it wasn’t until proofreading this essay that I realized I had originally written “Is Israel a planet?” and “Is opposing Pluto anti-Semitic?” I would like to see Jonathan Haidt hypnotize people until they can come up with positive arguments for those propositions.)

What’s the point of this useless squabble over definitions?

I think it’s about drawing a line between the concept “anti-Semitism” and “oppose Israel”. If your head is screwed on right, you assign anti-Semitism some very bad karma. So if we can stick a thick line between “anti-Semitism” and “oppose Israel”, then you’re going have very bad feelings about opposition to Israel and your General Factor Of Pro-Israeliness will go up.

Notice that this model is transitive, but shouldn’t be.

That is, let’s say we’re arguing over the definition of anti-Semitism, and I say “anti-Semitism just means anything that hurts Jews”. This is a dumb definition, but let’s roll with it.

First, I load “anti-Semitism” with lots of negative affect. Hitler was anti-Semitic. The pogroms in Russia were anti-Semitic. The Spanish Inquisition was anti-Semitic. Okay, negative affect achieved.

Then I connect “wants to end the Israeli occupation of Palestine” to “anti-Semitism”. Now wanting to end the Israeli occupation of Palestine has lots of negative affect attached to it.

It sounds dumb when you put it like that, but when you put it like “You’re anti-Semitic for wanting to end the occupation” it’s a pretty damaging argument.

This is trying to be transitive. It’s trying to say “anti-occupation = anti-Semitism, anti-Semitism = evil, therefore anti-occupation = evil”. If this were arithmetic, it would work. But there’s no Transitive Property Of Concepts. If anything, concepts are more like sets. The logic is “anti-occupation is a member of the set anti-Semitic, the set anti-Semitic contains members that are evil, therefore anti-occupation is evil”, which obviously doesn’t check out.

(compare: “I am a member of the set ‘humans’, the set ‘humans’ contains the Pope, therefore I am the Pope”.)

Anti-Semitism is generally considered evil because a lot of anti-Semitic things involve killing or dehumanizing Jews. Opposing the Israel occupation of Palestine doesn’t kill or dehumanize Jews, so even if we call it “anti-Semitic” by definition, there’s no reason for our usual bad karma around anti-Semitism to transfer over. But by an unfortunate rhetorical trick, it does – you can gather up bad karma into “anti-Semitic” and then shoot it at the “occupation of Palestine” issue just by clever use of definitions.

This means that if you can come up with sufficiently clever definitions and convince your opponent to accept them, you can win any argument by default just by having a complex system of mirrors in place to reflect bad karma from genuinely evil things to the things you want to tar as evil. This is essentially the point I make in Words, Words, Words.

If we kinda tweak the definition of “anti-Semitism” to be “anything that inconveniences Jews”, we can pull a trick where we leverage people’s dislike of Hitler to make them support the Israeli occupation of Palestine – but in order to do that, we need to get everyone on board with our slightly non-standard definition. Likewise, the social justice movement insists on their own novel definitions of words like “racism” that don’t match common usage, any dictionary, or etymological history – but which do perfectly describe a mirror that reflects bad karma toward opponents of social justice while making it impossible to reflect any bad karma back. Overreliance on this mechanism explains why so many social justice debates end up being about whether a particular mirror can be deployed to transfer bad karma in a specific case (“are trans people privileged?!”) rather than any feature of the real world.

But they are hardly alone. Compare: “Is such an such an organization a cult?”, “Is such and such a policy socialist?”, “Is abortion or capital punishment or war murder?” All entirely about whether we’re allowed to reflect bad karma from known sources of evil to other topics under discussion.

Look around you. Just look around you. Have you worked out what we’re looking for? Correct. The answer is The Worst Argument In The World. Only now, we can explain why it works.

VIII.

From the self-esteem literature, I gather that the self is also a concept that can have good or bad karma. From the cognitive dissonance literature, I gather that the self is actively involved in maintaining good karma around itself through as many biases as it can manage to deploy.

I’ve mentioned this study before. Researchers make victims participants fill out a questionnaire about their romantic relationships. Then they pretend to “grade” the questionnaire, actually assigning scores at random. Half the participants are told their answers indicate they have the tendency to be very faithful to their partner. The other half are told they have very low faithfulness and their brains just aren’t built for fidelity. Then they ask the participants victims their opinion on staying faithful in a relationship – very important, moderately important, or not so important?

There is a strong signal of people who are told they are bad at fidelity to state fidelity is unimportant, and another strong signal of people who are told they are especially faithful stating that fidelity is a great and noble virtue that must be protected.

The researchers conclude that people want to have high self-esteem. If I am terrible at fidelity, and fidelity is the most important virtue, that makes me a terrible person. If I am terrible at fidelity and fidelity doesn’t matter, I’m fine. If I am great at fidelity, and fidelity is the most important virtue, I can feel pretty good about myself.

This doesn’t seem too surprising. It’s just the more subtle version of the effect where white people are a lot more likely to be white supremacists than members of any other race. Everyone likes to hear that they’re great. The question is whether they can defend it and fit it in with their other ideas. The answer is “usually yes, because people are capable of pretty much any contortion of logic you can imagine and a lot that you can’t”.

I had a bad experience when I was younger where a bunch of feminists attacked and threatened me because of something I wrote. It left me kind of scarred. More importantly, the shape of that scar was a big anticorrelated line between self-esteem and the “feminism” concept. If feminism has lots of good karma, then I have lots of bad karma, because I am a person feminists hate. If feminists have lots of bad karma, then I look good by comparison, the same way it’s pretty much a badge of honor to be disliked by Nazis. The result was a permanent haze of bad karma around “feminism” unconnected to any specific feminist idea, which I have to be constantly on the watch for if I want to be able to evaluate anything related to feminism fairly or rationally.

Good or bad karma, when applied to yourself, looks like high or low self-esteem; when applied to groups, it looks like high or low status. In the giant muddle of a war for status that we politely call “society”, this makes beliefs into weapons and the karma loading of concepts into the difference between lionization and dehumanization.

The Trope Namer for emotivist arguments is “ethnic tension”, and although it’s most obvious in the case of literal ethnicities like the Israelis and the Palestinians, the ease with which concepts become attached to different groups creates a whole lot of “proxy ethnicites”. I’ve written before about how American liberals and conservatives are seeming less and less like people who happen to have different policy prescriptions, and more like two different tribes engaged in an ethnic conflict quickly approaching Middle East level hostility. More recently, a friend on Facebook described the-thing-whose-name-we-do-not-speak-lest-it-appear and-destroy-us-all, the one involving reproductively viable worker ants, as looking more like an ethnic conflict about who is oppressing whom than any real difference in opinions.

Once a concept has joined up with an ethnic group, either a real one or a makeshift one, it’s impossible to oppose the concept without simultaneously lowering the status of the ethnic group, which is going to start at least a little bit of a war. Worse, once a concept has joined up with an ethnic group, one of the best ways to argue against the concept is to dehumanize the ethnic group it’s working with. Dehumanizing an ethnic group has always been easy – just associate them with a disgust reaction, portray them as conventionally unattractive and unlovable and full of all the worst human traits – and now it is profitable as well, since it’s one of the fastest ways to load bad karma into an idea you dislike.

IX.

According to The Virtues Of Rationality:

The tenth virtue is precision. One comes and says: The quantity is between 1 and 100. Another says: the quantity is between 40 and 50. If the quantity is 42 they are both correct, but the second prediction was more useful and exposed itself to a stricter test. What is true of one apple may not be true of another apple; thus more can be said about a single apple than about all the apples in the world. The narrowest statements slice deepest, the cutting edge of the blade. As with the map, so too with the art of mapmaking: The Way is a precise Art. Do not walk to the truth, but dance. On each and every step of that dance your foot comes down in exactly the right spot. Each piece of evidence shifts your beliefs by exactly the right amount, neither more nor less. What is exactly the right amount? To calculate this you must study probability theory. Even if you cannot do the math, knowing that the math exists tells you that the dance step is precise and has no room in it for your whims.

The official desciption is of literal precision, as specific numerical precision in probability updates. But is there a secret interpretation of this virtue?

Precision as separation. Once you’re debating “religion”, you’ve already lost. Precision as sticking to a precise question, like “Is the first chapter of Genesis literally true?” or “Does Buddhist meditation help treat anxiety disorders?” and trying to keep these issues as separate from any General Factor Of Religiousness as humanly possible. Precision such that “God the supernatural Creator exists” and “God the order and beauty in the Universe exists” are as carefully sequestered from one another as “Did the defendant kill his wife?” and “Did the defendant kill the President?”

I want to end by addressing a point a commenter made in my last post on motte-and-bailey:

In the real world, the particular abstract questions aren’t what matter – the groups and people are what matter. People get things done, and they aren’t particularly married to particular abstract concepts, they are married to their values and their compatriots. In order to deal with reality, we must attack and defend groups and individuals. That does not mean forsaking logic. It requires dealing with obfuscating tactics like those you outline above, but that’s not even a real downside, because if you flee into the narrow, particular questions all you’re doing is covering your eyes to avoid perceiving the the monsters that will still make mincemeat of your attempts to change things.

I don’t entirely disagree with this. But I think we’ve been over this territory before.

The world is a scary place, full of bad people who want to hurt you, and in the state of nature you’re pretty much obligated to engage in whatever it takes to survive.

But instead of sticking with the state of nature, we have the ability to form communities built on mutual disarmament and mutual cooperation. Despite artificially limiting themselves, these communities become stronger than the less-scrupulous people outside them, because they can work together effectively and because they can boast a better quality of life that attracts their would-be enemies to join them. At least in the short term, these communities can resist races to the bottom and prevent the use of personally effective but negative-sum strategies.

One such community is the kind where members try to stick to rational discussion as much as possible. These communities are definitely better able to work together, because they have a powerful method of resolving empirical disputes. They’re definitely better quality of life, because you don’t have to deal with constant insult wars and personal attacks. And the existence of such communities provides positive externalities to the outside world, since they are better able to resolve difficult issues and find truth.

But forming a rationalist community isn’t just about having the will to discuss things well. It’s also about having the ability. Overcoming bias is really hard, and so the members of such a community need to be constantly trying to advance the art and figure out how to improve their discussion tactics.

As such, it’s acceptable to try to determine and discuss negative patterns of argument, even if those patterns of argument are useful and necessary weapons in a state of nature. If anything, understanding them makes them easier to use if you’ve got to use them, and makes them easier to recognize and counter from others, giving a slight advantage in battle if that’s the kind of thing you like. But moving them from unconscious to conscious also gives you the crucial choice of when to deploy them and allows people to try to root out ethnic tension in particular communities.

In order to rectify the error, here is a nice clean post on the concept that adds a couple of further thoughts to the original formulation.

The original Shackel paper is intended as a critique of post-modernism. Post-modernists sometimes say things like “reality is socially constructed”, and there’s an uncontroversially correct meaning there. We don’t experience the world directly, but through the categories and prejudices implicit to our society; for example, I might view a certain shade of bluish-green as blue, and someone raised in a different culture might view it as green. Okay.

Then post-modernists go on to say that if someone in a different culture thinks that the sun is light glinting off the horns of the Sky Ox, that’s just as real as our own culture’s theory that the sun is a mass of incandescent gas a great big nuclear furnace. If you challenge them, they’ll say that you’re denying reality is socially constructed, which means you’re clearly very naive and think you have perfect objectivity and the senses perceive reality directly.

The writers of the paper compare this to a form of medieval castle, where there would be a field of desirable and economically productive land called a bailey, and a big ugly tower in the middle called the motte. If you were a medieval lord, you would do most of your economic activity in the bailey and get rich. If an enemy approached, you would retreat to the motte and rain down arrows on the enemy until they gave up and went away. Then you would go back to the bailey, which is the place you wanted to be all along.

So the motte-and-bailey doctrine is when you make a bold, controversial statement. Then when somebody challenges you, you claim you were just making an obvious, uncontroversial statement, so you are clearly right and they are silly for challenging you. Then when the argument is over you go back to making the bold, controversial statement.

Some classic examples:

1. The religious group that acts for all the world like God is a supernatural creator who builds universes, creates people out of other people’s ribs, parts seas, and heals the sick when asked very nicely (bailey). Then when atheists come around and say maybe there’s no God, the religious group objects “But God is just another name for the beauty and order in the Universe! You’re not denying that there’s beauty and order in the Universe, are you?” (motte). Then when the atheists go away they get back to making people out of other people’s ribs and stuff.

2. Or…”If you don’t accept Jesus, you will burn in Hell forever.” (bailey) But isn’t that horrible and inhuman? “Well, Hell is just another word for being without God, and if you choose to be without God, God will be nice and let you make that choice.” (motte) Oh, well that doesn’t sound so bad, I’m going to keep rejecting Jesus. “But if you reject Jesus, you will BURN in HELL FOREVER and your body will be GNAWED BY WORMS.” But didn’t you just… “Metaphorical worms of godlessness!”

3. The feminists who constantly argue about whether you can be a real feminist or not without believing in X, Y and Z and wanting to empower women in some very specific way, and who demand everybody support controversial policies like affirmative action or affirmative consent laws (bailey). Then when someone says they don’t really like feminism very much, they object “But feminism is just the belief that women are people!” (motte) Then once the person hastily retreats and promises he definitely didn’t mean women aren’t people, the feminists get back to demanding everyone support affirmative action because feminism, or arguing about whether you can be a feminist and wear lipstick.

4. Proponents of pseudoscience sometimes argue that their particular form of quackery will cure cancer or take away your pains or heal your crippling injuries (bailey). When confronted with evidence that it doesn’t work, they might argue that people need hope, and even a placebo solution will often relieve stress and help people feel cared for (motte). In fact, some have argued that quackery may be better than real medicine for certain untreatable diseases, because neither real nor fake medicine will help, but fake medicine tends to be more calming and has fewer side effects. But then once you leave the quacks in peace, they will go back to telling less knowledgeable patients that their treatments will cure cancer.

5. Critics of the rationalist community note that it pushes controversial complicated things like Bayesian statistics and utilitarianism (bailey) under the name “rationality”, but when asked to justify itself defines rationality as “whatever helps you achieve your goals”, which is so vague as to be universally unobjectionable (motte). Then once you have admitted that more rationality is always a good thing, they suggest you’ve admitted everyone needs to learn more Bayesian statistics.

6. Likewise, singularitarians who predict with certainty that there will be a singularity, because “singularity” just means “a time when technology is so different that it is impossible to imagine” – and really, who would deny that technology will probably get really weird (motte)? But then every other time they use “singularity”, they use it to refer to a very specific scenario of intelligence explosion, which is far less certain and needs a lot more evidence before you can predict it (bailey).

The motte and bailey doctrine sounds kind of stupid and hard-to-fall-for when you put it like that, but all fallacies sound that way when you’re thinking about them. More important, it draws its strength from people’s usual failure to debate specific propositions rather than vague clouds of ideas. If I’m debating “does quackery cure cancer?”, it might be easy to view that as a general case of the problem of “is quackery okay?” or “should quackery be illegal?”, and from there it’s easy to bring up the motte objection.

Recently, a friend (I think it was Robby Bensinger) pointed out something I’d totally missed. The motte-and-bailey doctrine is a perfect mirror image of my other favorite fallacy, the weak man fallacy.

Weak-manning is a lot like straw-manning, except that instead of debating a fake, implausibly stupid opponent, you’re debating a real, unrepresentatively stupid opponent. For example, “Religious people say that you should kill all gays. But this is evil. Therefore, religion is wrong and barbaric. Therefore we should all be atheists.” There are certainly religious people who think that you should kill all gays, but they’re a small fraction of all religious people and probably not the ones an unbiased observer would hold up as the best that religion has to offer.

If you’re debating the Pope or something, then when you weak-man, you’re unfairly replacing a strong position (the Pope’s) with a weak position (that of the guy who wants to kill gays) to make it more attackable.

But in motte and bailey, you’re unfairly replacing a weak position (there is a supernatural creator who can make people out of ribs) with a strong position (there is order and beauty in the universe) in order to make it more defensible.

So weak-manning is replacing a strong position with a weak position to better attack it; motte-and-bailey is replacing a weak position with a strong position to better defend it.

This means people who know both terms are at constant risk of arguments of the form “You’re weak-manning me!” “No, you’re motte-and-baileying me!“.

Suppose we’re debating feminism, and I defend it by saying it really is important that women are people, and you attack it by saying that it’s not true that all men are terrible. Then I can accuse you of making life easy for yourself by attacking the weakest statement anyone vaguely associated with feminism has ever pushed. And you can accuse me if making life too easy for myself by defending the most uncontroversially obvious statement I can get away with.

So what is the real feminism we should be debating? Why would you even ask that question? What is this, some kind of dumb high school debate club? Who the heck thinks it would be a good idea to say “Here’s a vague poorly-defined concept that mind-kills everyone who touches it – quick, should you associate it with positive affect or negative affect?!”

Taboo your words, then replace the symbol with the substance. If you have an actual thing you’re trying to debate, then it should be obvious when somebody’s changing the topic. If working out who’s using motte-and-bailey (or weak man) is remotely difficult, it means your discussion went wrong several steps earlier and you probably have no idea what you’re even arguing about.

PS: Nicholas Shackel, original inventor of the term, weighs in.

A recent Tumblr conversation on the Machine Intelligence Research Institute has gotten interesting and I thought I’d see what people here have to say.

If you’re just joining us and don’t know about the Machine Intelligence Research Institute (“MIRI” to its friends), they’re a nonprofit organization dedicated to navigating the risks surrounding “intelligence explosion”. In this scenario, a few key insights around artificial intelligence can very quickly lead to computers so much smarter than humans that the future is almost entirely determined by their decisions. This would be especially dangerous since most AIs use very primitive untested goal systems inappropriate for and untested on intelligent entities; such a goal system would be “unstable” and from a human perspective the resulting artificial intelligence could have apparently arbitrary or insane goals. If such a superintelligence were much more powerful than we are, it would present an existential threat to the human race.

This has almost nothing to do with the classic “Skynet” scenario – but if it helps to imagine Skynet, then fine, just imagine Skynet. Everyone else does.

MIRI tries to raise awareness of this possibility among AI researchers, scientists, and the general public, and to start foundational research in more stable goal systems that might allow AIs to become intelligent or superintelligent while still acting in predictable and human-friendly ways.

This is not a 101 space and I don’t want the comments here to all be about whether or not this scenario is likely. If you really want to discuss that, go read at least Facing The Intelligence Explosion and then post your comments in the Less Wrong Open Thread or something. This is about MIRI as an organization.

(If you’re really just joining us and you don’t know about Tumblr, run away)

II.

Tumblr user su3su2u1 writes:

Saw some tumblr people talking about [effective altruism]. My biggest problem with this movement is that most everyone I know who identifies themselves as an effective altruist donates money to MIRI (it’s possible this is more a comment on the people I know than the effective altruism movement, I guess). Based on their output over the last decade, MIRI is primarily a fanfic and blog-post producing organization. That seems like spending money on personal entertainment.

Part of this is obviously mean-spirited potshots, in that MIRI itself doesn’t produce fanfic and what their employees choose to do with their own time is none of your damn business.

(well, slightly more complicated. I think MIRI gave Eliezer a couple weeks vacation to work on it as an “outreach” thing once. But that’s a little different from it being their main priority.)

But more serious is the claim that MIRI doesn’t do much else of value. I challenged Su3 with the following evidence of MIRI doing good work:

A1. MIRI has been very successful with outreach and networking – basically getting their cause noticed and endorsed by the scientific establishment and popular press. They’ve gotten positive attention, sometimes even endorsements, from people like Stephen Hawking, Elon Musk, Gary Drescher, Max Tegmark, Stuart Russell, and Peter Thiel. Even Bill Gates is talking about AI risk, though I don’t think he’s mentioned MIRI by name. Multiple popular books have been written about their ideas, such as James Miller’s Singularity Rising and Stuart Armstrong’s Smarter Than Us. Most recently Nick Bostrom’s book Superintelligence, based at least in part on MIRI’s research and ideas, is a New York Times best-seller and has been reviewed positively in the Guardian, the Telegraph, Salon, the Financial Times, and the Economist. Oxford has opened up the AI-risk-focused Future of Humanity Institute; MIT has opened up the similar Future of Life Institute. In about a decade, the idea of an intelligence explosion has gone from Time Cube level crackpottery to something taken seriously by public intellectuals and widely discussed in the tech community.

A2. MIRI has many publications, conference presentations, book chapters and other things usually associated with normal academic research, which interested parties can find on their website. They have conducted seven past research workshops which have produced interesting results like Christiano et al’s claimed proof of a way around the logical undefinability of truth, which was praised as potentially interesting by respected mathematics blogger John Baez.

A3. Many former MIRI employees, and many more unofficial fans, supporters, and associates of MIRI, are widely distributed across the tech community in industries that are likely to be on the cutting edge of artificial intelligence. For example, there are a bunch of people influenced by MIRI in Google’s AI department. Shane Legg, who writes about how his early work was funded by a MIRI grant and who once called MIRI “the best hope that we have” was pivotal in convincing Google to set up an AI ethics board to monitor the risks of the company’s cutting-edge AI research. The same article mentions Peter Thiel and Jaan Tallinn as leading voices who will make Google comply with the board’s recommendations; they also happen to be MIRI supporters and the organization’s first and third largest donors.

There’s a certain level of faith required for (A1) and (A3) here, in that I’m attributing anything good that happens in the field of AI risk to some sort of shady behind-the-scenes influence from MIRI. Maybe Legg, Tallinn, and Thiel would have pushed for the exact same Google AI Ethics Board if none of them had ever heard of MIRI at all. I am forced to plead ignorance on the finer points of networking and soft influence. Heck, for all I know, maybe the exact same number of people would vote Democrat if there were no Democratic National Committee or liberal PACs. I just assume that, given a really weird idea that very few people held in 2000, an organization dedicated to spreading that idea, and the observation that the idea has indeed spread very far, the organization is probably doing something right.

III.

Our discussion on point (A3) degenerated into Dueling Anecdotal Evidence. But Su3 responded to my point (A1) like so:

[I agree that MIRI has gotten shoutouts from various thought leaders like Stephen Hawking and Elon Musk. Bostrom’s book is commercially successful, but that’s just] more advertising. Popular books aren’t the way to get researchers to notice you. I’ve never denied that MIRI/SIAI was good at fundraising, which is primarily what you are describing.

How many of those thought leaders have any publications in CS or pure mathematics, let alone AI? Tegmark might have a math paper or two, but he is primarily a cosmologist. The FLI’s list of scientists is (for some reason) mostly again cosmologists. The active researchers appear to be a few (non-CS, non-math) grad students. Not exactly the team you’d put together if you were actually serious about imminent AI risk.

I would also point out “successfully attracted big venture capital names” isn’t always a mark of a sound organization. Black Light Power is run by a crackpot who thinks he can make energy by burning water, and has attracted nearly 100 million in funding over the last two decades, with several big names in energy production behind him.

And to my point (A2) like so:

I have a PhD in physics and work in machine learning. I’ve read some of the technical documents on MIRI’s site, back when it was SIAI and I was unimpressed. I also note that this critique is not unique to me, as far as I know the GiveWell position on MIRI is that it is not an effective institute.

The series of papers on Lob’s theorem are actually interesting, though I notice that none of the results have been peer reviewed, and the paper’s aren’t listed as being submitted to journals yet. Their result looks right to me, but I wouldn’t trust myself to catch any subtlety that might be involved.

[But that just means] one result has gotten some small positive attention, and even those results haven’t been vetted by the wider math community yet (no peer review). Let’s take a closer look at the list of publications on MIRI’s website- I count 6 peer reviewed papers in their existence, and 13 conference presentations. Thats horribly unproductive! Most of the grad students who finish a physics phd will publish that many papers individually, in about half that time. You claim part of their goal is to get academics to pay attention, but none of their papers are highly cited, despite all this networking they are doing.

Citations are the standard way to measure who in academia is paying attention. Apart from the FHI/MIRI echo chamber (citations bouncing around between the two organizations), no one in academia seems to be paying attention to MIRI’s output. MIRI is failing to make academic inroads, and it has produced very little in the way of actual research.

My interpretation, in the form of a TL;DR

B1. Sure, MIRI is good at getting attention, press coverage, and interest from smart people not in the field. But that’s public relations and fundraising. An organization being good at fundraising and PR doesn’t mean it’s good at anything else, and in fact “so good at PR they can cover up not having substance” is a dangerous failure mode.

B2. What MIRI needs, but doesn’t have, is the attention and support of smart people within the fields of math, AI, and computer science, whereas now it mostly has grad students not in these fields.

B3. While having a couple of published papers might look impressive to a non-academic, people more familiar with the culture would know that their output is woefully low. They seem to have gotten about five ten solid publications in during their decade-long history as a multi-person organization; one good grad student can get a couple solid publications a year. Their output is less than expected by like an order of magnitude. And although they do get citations, this is all from a mutual back-scratching club of them and Bostrom/FHI citing each other.

IV.

At this point Tarn and Robby joined the conversation and it became kind of confusing, but I’ll try to summarize our responses.

Our response to Su3’s point (B1) was that this is fundamentally misunderstanding outreach. From its inception until about last year, MIRI was in large part an outreach and awareness-raising organization. Its 2008 website describes its mission like so:

In the coming decades, humanity will likely create a powerful AI. SIAI exists to confront this urgent challenge, both the opportunity and the risk. SIAI is fostering research, education, and outreach to increase the likelihood that the promise of AI is realized for the benefit of everyone.

Outreach is one of its three main goals, and “education”, which sounds a lot like outreach, is a second.

In a small field where you’re the only game in town, it’s hard to distinguish between outreach and self-promotion. If MIRI successfully gets Stephen Hawking to say “We need to be more concerned about AI risks, as described by organizations like MIRI”, is that them being very good at self-promotion and fundraising, or is that them accomplishing their core mission of getting information about AI risks to the masses?

Once again, compare to a political organization, maybe Al Gore’s anti-global-warming nonprofit. If they get the media to talk about global warming a lot, and get lots of public intellectuals to come out against global warming, and change behavior in the relevant industries, then mission accomplished. The popularity of An Inconvenient Truth can’t just be dismissed as “self-promotion” or “fundraising” for Gore, it was exactly the sort of thing he was gathering money and personal prestige in order to do, and should be considered a victory in its own right. Even though eventually the anti-global-warming cause cares about politicians, industry leaders, and climatologists a lot more than they care about the average citizen, convincing millions of average citizens to help was a necessary first step.

And this which is true of An Inconvenient Truth is true of Superintelligence and other AI risk publicity efforts, albeit on their much smaller scale.

Our response to Su3’s point (B2) was that it was just plain factually false. MIRI hasn’t reached big names from the AI/math/compsci field? Sure it has. Doesn’t have mathy PhD students willing to research for them? Sure it does.

Peter Norvig and Stuart Russell are among the biggest names in AI. Norvig is currently the Director of Research at Google; Russell is Professor of Computer Science at Berkeley and a winner of various impressive sounding awards. The two wrote a widely-used textbook on artificial intelligence in which they devote three pages to the proposition that “The success of AI might mean the end of the human race”; parts are taken right out of the MIRI playbook and they cite MIRI research fellow Eliezer Yudkowsky’s paper on the subject. This is unlikely to be a coincidence; Russell’s site links to MIRI and he is scheduled to participate in MIRI’s next research workshop.

Their “team” of “research advisors” includes Gary Drescher (PhD in CompSci from MIT), Steve Omohundro (PhD in physics from Berkeley but also considered a pioneer of machine learning), Roman Yampolskiy (PhD in CompSci from Buffalo), and Moshe Looks (PhD in CompSci from Washington).

Su3 brought up the good point that none of these people, respected as they are, are MIRI employees or researchers (although Drescher has been to a research workshop). At best, they are people who were willing to let MIRI use them as figureheads (in the case of the research advisors); at worst, they are merely people who have acknowledged MIRI’s existence in a not-entirely-unlike-positive way (Norvig and Russell). Even if we agree they are geniuses, this does not mean that MIRI has access to geniuses or can produce genius-level research.

Fine. All these people are, no more and no less, is evidence that MIRI is succeeding at outreach within the academic field of AI, as well as in the general public. It also seems to me to be some evidence that smart people who know more about AI than any of us think MIRI is on the right track.

Su3 brought up the example of a BlackLight Power, a crackpot energy company that was able to get lots of popular press and venture capital funding despite being powered entirely by pseudoscience. I agree this is the sort of thing we should be worried about. Nonscientists outside of specialized fields have limited ability to evaluate their claims. But when smart researchers in the field are willing to vouch for MIRI, that give me a lot more confidence they’re not just a fly-by-night group trying to profit off of pseudoscience. Their research might be more impressive or less impressive, but they’re not rotten to the core the same way BlackLight was.

And though MIRI’s own researchers may be far from those lofty heights, I find Su3’s claim that they are “a few non-CS, non-math grad students” a serious underestimate.

MIRI has fourteen employees/associates with the word “research” in their name, but of those, a couple (in the words of MIRI’s team page) “focus on social and historical questions related to artificial intelligence outcomes.” These people should not be expected to have PhDs in mathematical/compsci subjects.

Of the rest, Bill is a PhD in CompSci, Patrick is a PhD in math, Nisan is a PhD in math, Benja is a PhD student in math, and Paul is a PhD student in math. The others mostly have masters or bachelors in those fields, published journal articles, and/or won prizes in mathematical competitions. Eliezer writes of some of the remaining members of his team:

Mihaly Barasz is an International Mathematical Olympiad gold medalist perfect scorer. From what I’ve seen personally, I’d guess that Paul Christiano is better than him at math. I forget what Marcello’s prodigy points were in but I think it was some sort of Computing Olympiad [editor’s note: USACO finalist and 2x honorable mention in the Putnam mathematics competition]. All should have some sort of verified performance feat far in excess of the listed educational attainment.

That pretty much leaves Eliezer Yudkowsky, who needs no introduction, and Nate Soares, whose introduction exists and is pretty interesting.

Add to that the many, many PhDs and talented people who aren’t officially employed by them but attend their workshops and help out their research when they get the chance, and you have to ask how many brilliant PhDs from some of the top universities in the world we should expect a small organization like MIRI to have. MIRI competes for the same sorts of people as Google, and offers half as much. Google paid $400 million to get Shane Legg and his people on board; MIRI’s yearly budget hovers at about $1 million. Given that they probably spend a big chunk of that on office space, setting up conferences, and other incidentals, I think the amount of talent they have right now is pretty good.

That leaves Su3’s point (B3) – the lack of published research.

One retort might be that, until recently, MIRI’s research focused on strategic planning and evaluation of AI risks. This is important, and it resulted in a lot of internal technical papers you can find on their website, but there’s not really a field for it. You can’t just publish it in the Journal Of What Would Happen If There Was An Intelligence Explosion, because no such journal. The best they can do is publish the parts of their research that connect to other fields in appropriate journals, which they sometimes did.

I feel like this also frees them from the critique of citation-incest between them and Bostrom. When I look at a typical list of MIRI paper citations, I do see a lot of Bostrom, but also some other names that keep coming up – Hutter, Yampolskiy, Goetzel. So okay, it’s an incest circle of four or five rather than two.

But to some degree that’s what I expect from academia. Right now I’m doing my own research on a psychiatric screening tool called the MDQ. There are three or four research teams in three or four institutions who are really into this and publish papers on it a lot. Occasionally someone from another part of psychiatry wanders in, but usually it’s just the subsubsubspeciality of MDQ researchers talking to each other. That’s fine. They’re our repository of specialized knowledge on this one screening tool.

You would hope the future of the human race would get a little bit more attention than one lousy psychiatric screening tool, but blah blah civilizational inadequacy, turns out not so much, they’re of about equal size. If there are only a couple of groups working on this problem, they’re going to look incestuous but that’s fine.

On the other hand, math is math, and if MIRI is trying to produce real mathematical results they ought to be sharing them with the broader mathematical community.

Robby protests that until very recently, MIRI hasn’t really been focusing on math. This is a very recent pivot. In April 2013, Luke wrote in his mini strategic plan:

We were once doing three things — research, rationality training, and the Singularity Summit. Now we’re doing one thing: research. Rationality training was spun out to a separate organization, CFAR, and the Summit was acquired by Singularity University. We still co-produce the Singularity Summit with Singularity University, but this requires limited effort on our part.
After dozens of hours of strategic planning in January–March 2013, and with input from 20+ external advisors, we’ve decided to (1) put less effort into public outreach, and to (2) shift our research priorities to Friendly AI math research.

In the full strategic plan for 2014, he repeated:

Events since MIRI’s April 2013 strategic plan have increased my confidence that we are “headed in the right direction.” During the rest of 2014 we will continue to:
– Decrease our public outreach efforts, leaving most of that work to FHI at Oxford, CSER at Cambridge, FLI at MIT, Stuart Russell at UC Berkeley, and others (e.g. James Barrat).
– Finish a few pending “strategic research” projects, then decrease our efforts on that front, again leaving most of that work to FHI, plus CSER and FLI if they hire researchers, plus some others.
– Increase our investment in our Friendly AI (FAI) technical research agenda.
– We’ve heard that as a result of…outreach success, and also because of Stuart Russell’s discussions with researchers at AI conferences, AI researchers are beginning to ask, “Okay, this looks important, but what is the technical research agenda? What could my students and I do about it?” Basically, they want to see an FAI technical agenda, and MIRI is is developing that technical agenda already.

In other words, there is a recent pivot from outreach, rationality and strategic research to pure math research, and the pivot is only recently finished or still going on.

TL;DR, again in three points:

C1. Until recently, MIRI focused on outreach and did a truly excellent job on this. They deserve credit here.

C2. MIRI has a number of prestigious computer scientists and AI experts willing to endorse or affiliate with it in some way. While their own researchers are not quite at the same lofty heights, they include many people who have or are working on math or compsci PhDs.

C3. MIRI hasn’t published much math because they were previously focusing on outreach and strategic research; they’ve only shifted to math work in the past year or so.

V.

The discussion just kept going. We reached about the limit of our disagreement on (C1), the point about outreach – yes, they’ve done it, but does it count when it doesn’t bear fruit in published papers? About (C2) and the credentials of MIRI’s team, Su3 kind of blended it into the next point about published papers, saying:

Fundamental disconnect – I consider “working with MIRI” to mean “publishing results with them.” As an outside observer, I have no indication that most of these people are working with them. I’ve been to workshops and conferences with Nobel prize winning physicists, but I’ve never “worked with them” in the academic sense of having a paper with them. If [someone like Stuart Russell] is interested in helping MIRI, the best thing he could do is publish a well received technical result in a good journal with Yudkowsky. That would help get researchers to pay actual attention(and give them one well received published result, in their operating history).

Tangential aside- you overestimate the difficulty of getting top grad students to work for you. I recently got four CS grad students at a top program to help me with some contract work for a few days at the cost of some pizza and beer.

So it looks like it all comes down to the papers. Su3 had this to say:

What I was specifically thinking was “MIRI has produced a much larger volume of well-received fan fiction and blog posts than research.” That was what I inended to communicate, if somewhat snarkily. MIRI bills itself as a research institute, so I judge them on their produced research. The accountability measure of a research institute is academic citations.

Editorials by famous people have some impact with the general public, so thats fine for fundraising, but at some point you have to get researchers interested. You can measure how much influence they have on researchers by seeing who those researchers cite and what they work on. You could have every famous cosmologist in the world writing op-eds about AI risk, but its worthless if AI researchers don’t pay attention, and judging by citations, they aren’t.

As a comparison for publication/citation counts, I know individual physicists who have published more peer reviewed papers since 2005 than all of MIRI has self-published to their website. My single most highly cited physics paper (and I left the field after graduate school) has more citations than everything MIRI has ever published in peer reviewed journals combined. This isn’t because I’m amazing, its because no one in academia is paying attention to MIRI.

[Christiano et al’s result about Lob] has been self-published on their website. It has NOT been peer reviewed. So it’s published in the sense of “you can go look at the paper.” But its not published in the sense of “mathematicians in the same field have verified the result.” I agree this one result looks interesting, but most mathematicians won’t pay attention to it unless they get it reviewed (or at the bare minimum, clean it up and put it on Arxiv). They have lots of these self-published documents on their web page.

If they are making a “strategic decision” to not submit their self-published findings to peer review ,they are making a terrible strategic decision, and they aren’t going to get most academics to pay attention that way. The result of Christiano, et al. is potentially interesting, but it’s languishing as a rough unpublished draft on the MIRI site, so its not picking up citations.

I’d go further and say the lack of citations is my main point. Citations are the important measurement of “are researchers paying attention.” If everything self-published to MIRI’s website were sparking interest in academia, citations would be flying around, even if the papers weren’t peer reviewed, and I’d say “yeah, these guys are producing important stuff.”

My subpoint might be that MIRI doesn’t even seem to be trying to get citations/develop academic interest, as measured by how little effort seems to be put into publication.

And Su3’s not buying the pivot explanation either:

That seems to be a reframing of the past history though. I saw talks by the SIAI well before 2013 where they described their primary purpose as friendly AI research, and insisted they were in a unique position (due to being uniquely brilliant/rational) to develop technical friendly AI (as compared to academic AI researchers).

[Tarn] and [Robby] have suggested the organization is undergoing a pivot, but they’ve always billed themselves as a research institute. But donating money to an organization that has been ineffective in the past, because it looks like they might be changing seems like a bad proposition.

My initial impression (reading Muelhauser’s post you linked to and a few others) is that Muelhauser noticed the house was out of order when he became director and is working to fix things. Maybe he’ll succeed and in the future, then, I’ll be able to judge MIRI as effective- certainly a disproportionate number of their successes have come in the last few years. However, right now all I have is their past history, which has been very unproductive.

VI.

After that, discussion stayed focused on the issue of citations. This seemed like progress to me. Not only had we gotten it down to a core objection, but it was sort of a factual problem. It wasn’t an issue of praising or condemning. Here’s an organization with a lot of smart people. We know they work very hard – no one’s ever called Luke a slacker, and another MIRI staffer (who will not be named, for his own protection) achieved some level of infamy for mixing together a bunch of the strongest chemicals from my nootropics survey into little pills which he kept on his desk in the MIRI offices for anyone who wanted to work twenty hours straight and then probably die young of conditions previously unknown to science. IQ-point*hours is a weird metric, but MIRI is putting a lot of IQ-point*hours into whatever it’s doing. So if Su3’s right that there are missing citations, where are they?

Among the three of us, Robby and Tarn and I generated a couple of hypotheses (well, Robby’s were more like facts than hypotheses, since he’s the only one in this conversation who actually works there).

D1: MIRI has always been doing research, but until now it’s been strategic research (ie “How worried should we be about AI?”, “How far in the future should we expect AI to be developed?”) which hasn’t fit neatly into an academic field or been of much interest to anyone except MIRI allies like Bostrom. They have dutifully published this in the few papers that are interested, and it has dutifully been cited by the few people who are interested (ie Bostrom). It’s unreasonable to expect Stuart Russell to cite their estimates of time course for superintelligence when he’s writing his papers on technical details of machine learning algorithms or whatever it is he writes papers on. And we can generalize from Stuart Russell to the rest of the AI field, who are also writing on things like technical details of machine learning algorithms that can’t plausibly be connected to when machines will become superintelligent.

D2: As above, but continuing to apply even in some of their math-ier research. MIRI does have lots of internal technical papers on their website. People tend to cite other researchers working in the same field as themselves. I could write the best psychiatry paper in human history, and I’m probably not going to get any citations from astrophysicists. But “machine ethics” is an entirely new field that’s not super relevant to anyone else’s work. Although a couple key machine ethics problems, like the Lobian obstacle and decision theory, touch on bigger and better-populated subfields of mathematics, they’re always going to be outsiders who happen to wander in. It’s unfair to compare them to a physics grad student writing about quarks or something, because she has the benefit of decades of previous work on quarks and a large and very interested research community. MIRI’s first job is to create that field and community, which until you succeed looks a lot like “outreach”.

D3: Lack of staffing and constant distraction by other important problems. This is Robby’s description of what he notices from the inside. He writes:

We’re short on staff, especially since Louie left. Lots of people are willing to volunteer for MIRI, but it’s hard to find the right people to recruit for the long haul. Most relevantly, we have two new researchers (Nate and Benja), but we’d love a full-time Science Writer to specialize in taking our researchers’ results and turning them into publishable papers. Then we don’t have to split as much researcher time between cutting-edge work and explaining/writing-down.

A lot of the best people who are willing to help us are very busy. I’m mainly thinking of Paul Christiano. he’s working actively on creating a publishable version of the probabilistic Tarski stuff, but it’s a really big endeavor. Eliezer is by far our best FAI researcher, and he’s very slow at writing formal, technical stuff. He’s generally low-stamina and lacks experience in writing in academic style / optimizing for publishability, though I believe we’ve been having a math professor tutor him to get over that particular hump. Nate and Benja are new, and it will take time to train them and get them publishing their own stuff. At the moment, Nate/Benja/Eliezer are spending the rest of 2014 working on material for the FLI AI conference, and on introductory FAI material to send to Stuart Russell and other bigwigs.

D4: Some of the old New York rationalist group takes a more combative approach. I’m not sure I can summarize their argument well enough to do it justice, so I would suggest reading Alyssa’s post on her own blog.

But if I have to take a stab: everyone knows mainstream academia is way too focused on the “publish or perish” ethic of measuring productivity in papers or citations rather than real progress. Yeah, a similar-sized research institute in physics could probably get ten times more papers/citations than MIRI. That’s because they’re optimizing for papers/citations rather than advancing the field, and Goodhart’s Law is in effect here as much as everywhere else. Those other institutes probably got geniuses who should be discovering the cure for cancer spending half their time typing, formatting, submitting, resubmitting, writing whatever the editors want to see, et cetera. MIRI is blessed with enough outside support that it doesn’t have to do that. The only reason to try is to get prestige and attention, and anyone who’s not paying attention now is more likely to be a constitutional skeptic using lack of citations as an excuse, than a person who would genuinely change their mind if there were more citations.

I am more sympathetic than usual to this argument because I’m in the middle of my own research on psychiatric screening tools and quickly learning that official, published research is the worst thing in the world. I could do my study in about two hours if the only work involved were doing the study; instead it’s week after week of forms, IRB submissions, IRB revisions, required online courses where I learn the Nazis did unethical research and this was bad so I should try not to be a Nazi, selecting exactly which journals I’m aiming for, and figuring out which of my bosses and co-workers academic politics requires me make co-authors. It is a crappy game, and if you’ve been blessed with enough independence to avoid playing it, why wouldn’t you take advantage? Forget the overhyped and tortured “measure” of progress you use to impress other people, and just make the progress.

VII.

Or not. I’ll let Su3 have the last word:

I think something fundamental about my argument has been missed, perhaps I’ve communicated it poorly.

It seems like you think the argument is that increasing publications increases prestige/status which would make researchers pay attention. i.e. publications -> citations -> prestige -> people pay attention. This is not my argument.

My argument is essentially that the way to judge if MIRI’s outreach has been successful is through citations, not through famous people name dropping them, or allowing them to be figure heads.

This is because I believe the goal of outreach is get AI researchers focused on MIRI’s ideas. Op eds from famous people are useful only if they get AI researchers focused on these ideas. Citations aren’t about prestige in this case- citations tell you which researchers are paying attention to you. The number of active researchers paying attention to MIRI is very small. We know this because citations are an easy to find, direct measure.

Not all important papers have tremendous numbers of citations, but a paper can’t become important if it only has 1 or 2, because the ultimate measure of importance is “are people using these ideas?”

So again, to reiterate, if the goal of outreach is to get active AI researchers paying attention, then the direct measure for who is paying attention is citations. [But] the citation count on MIRIs work is very low. Not only is the citation count low (i.e. no researchers are paying attention), MIRI doesn’t seem to be trying to boost it – it isn’t trying to publish which would help get its ideas attention. I’m not necessarily dismissive of celebrity endorsements or popular books, my point is why should I measure the means when I can directly measure the ends?

The same idea undercuts your point that “lots of impressive PhD students work and have worked with MIRI,” because it’s impossible to tell if you don’t personally know the researchers. This is because they don’t create much output while at MIRI, and they don’t seem to be citing MIRI in their work outside of MIRI.

[Even people within the rationalist/EA community] agree with me somewhat. Here is a relevant quote from Holden Karnofsky [of GiveWell]:

SI seeks to build FAI and/or to develop and promote “Friendliness theory” that can be useful to others in building FAI. Yet it seems that most of its time goes to activities other than developing AI or theory. Its per-person output in terms of publications seems low. Its core staff seem more focused on Less Wrong posts, “rationality training” and other activities that don’t seem connected to the core goals; Eliezer Yudkowsky, in particular, appears (from the strategic plan) to be focused on writing books for popular consumption. These activities seem neither to be advancing the state of FAI-related theory nor to be engaging the sort of people most likely to be crucial for building AGI.

And here is a statement from Paul Christiano disagreeing with MIRI’s core ideas:

But I should clarify that many of MIRI’s activities are motivated by views with which I disagree strongly and that I should categorically not be read as endorsing the views associated with MIRI in general or of Eliezer in particular. For example, I think it is very unlikely that there will be rapid, discontinuous, and unanticipated developments in AI that catapult it to superhuman levels, and I don’t think that MIRI is substantially better prepared to address potential technical difficulties than the mainstream AI researchers of the future.

This time Su3 helpfully provides their own summary:

E1. If the goal of outreach is to get active AI researchers paying attention, then the direct measure for who is paying attention is citations. [But] the citation count on MIRIs work is very low.

E2. Not only is the citation count low (i.e. no researchers are paying attention), MIRI doesn’t seem to be trying to boost it – it isn’t trying to publish which would help get its ideas attention. I’m not necessarily dismissive of celebrity endorsements or popular books, my point is why should I measure the means when I can directly measure the ends?

E3. The same idea undercuts your point that “lots of impressive phd students work and have worked with MIRI,” because its impossible to tell if you don’t personally know the researchers. This is because they don’t create much output while at MIRI, and they don’t seem to be citing MIRI in their work outside of MIRI.

E4. Holden Karnofsky and Paul Christiano do not believe that MIRI is better prepared to address the friendly AI problem than mainstream AI researchers of the future. Karnofsky explicitly for some of the reasons I have brought up, Christiano for reasons unmentioned.

VIII.

Didn’t actually read all that and just skipped down to the last subheading to see if there’s going to be a summary and conclusion and maybe some pictures? Good.

There seems to be some agreement MIRI has done a good job bringing issues of AI risk into the public eye and getting them media attention and the attention of various public intellectuals. There is disagreement over whether they should be credited for their success in this area, or whether this is a first step they failed to follow up on.

There also seems to be some agreement MIRI has done a poor job getting published and cited results in journals. There is disagreement over whether this is an understandable consequence of being a small organization in a new field that wasn’t even focusing on this until recently, or whether it represents a failure at exactly the sort of task by which their success should be judged.

This is probably among the 100% of issues that could be improved with flowcharts:

In the Optimistic Model, MIRI’s successfully built up Public Interest, and for all we know they might have Mathematical Progress as well even though they haven’t published it in journals yet. While they could feed back their advantages by turning their progress into Published Papers and Citations to get even more Mathematical Progress, overall they’re in pretty good shape for producing Good Outcomes, at least insofar as this is possible in their chosen field.

In the Pessimistic Model, MIRI may or may not have garnered Public Interest, Researcher Interest, and Tentative Mathematical Progress, but they failed to turn that into Published Papers and Citations, which is the only way they’re going to get to Robust Mathematical Progress, Researcher Support, and eventually Good Outcomes. The best that can be said about them is that they set some very preliminary groundwork that they totally failed to follow up on.

A higher level point – if we accept the Pessimistic Model, do we accuse MIRI of being hopelessly incompetent, in which case they deserve less support? Or do we accept them as inexperienced amateurs who are the only people willing to try something difficult but necessary, in which case they deserve more support, and maybe some guidance, and perhaps some gentle or not-so-gentle prodding? Maybe if you’re a qualified science writer you could apply for the job opening they’re advertising and help them get those papers they need?

An even higher-level point – what do people worried about AI risk do with this information? I don’t see much that changes my opinion of the organization one way or the other. But Robby points out that people who are more concerned – but still worried about AI risk – have other good options. The Future of Humanity Institute at Oxford research that is less technical and more philosophical, wears their strategic planning emphasis openly on their sleeve has oodles of papers and citations and prestige. They also accept donations.

Best of all, their founder doesn’t write any fanfic at all. Just perfectly respectable stories about evil dragon kings.

What if oracles gave clear and accurate answers to this sort of question? What if anyone could ask an oracle the outcome of any war, or planned war, and expect a useful response?

When the oracle predicts the aggressor loses, it might prevent wars from breaking out. If an oracle told the US that the Vietnam War would cost 50,000 lives and a few hundred billion dollars, and the communists would conquer Vietnam anyway, the US probably would have said no thank you.

What about when the aggressor wins? For example, the Mexican-American War, where the United States won the entire Southwest at a cost of “only” ten thousand American casualties and $100 million (with an additional 20,000 Mexican deaths and $50 million in costs to Mexico)?

If both Mexico and America had access to an oracle who could promise them that the war would end with Mexico ceding the Southwest to the US, could Mexico just agree to cede the Southwest to the US at the beginning, and save both sides tens of thousands of deaths and tens of millions of dollars?

Not really. One factor that prevents wars is countries being unwilling to pay the cost even of wars they know they’ll win. If there were a tradition of countries settling wars by appeal to oracle, “invasions” would become much easier. America might just ask “Hey, oracle, what would happen if we invaded Canada and tried to capture Toronto?” The oracle might answer “Well, after 20,000 deaths on both sides and hundreds of millions of dollars wasted, you would eventually capture Toronto.” Then the Americans could tell Canada, “You heard the oracle! Give us Toronto!” – which would be free and easy – when maybe they would never be able to muster the political and economic will to actually launch the invasion.

So it would be in Canada’s best interests not to agree to settle wars by oracular prediction. For the same reasons, most other countries would also refuse such a system.

But I can’t help fretting over how this is really dumb. We have an oracle, we know exactly what the results of the Mexican-American War are going to be, and we can’t use that information to prevent tens of thousands of people from being killed in order to make the result happen? Surely somebody can do better than that.

What if the United States made Mexico the following deal: suppose a soldier’s life is valued at $10,000 (in 1850 dollars, I guess, not that it matters much when we’re pricing the priceless). So in total, we’re going to lose 10,000 soldiers + $100 million = $200 million to this war. You’re going to lose 20,000 soldiers + $50 million = $250 million to this war.

So tell you what. We’ll dig a giant hole and put $150 million into it. You give us the Southwest. This way, we’re both better off. You’re $250 million ahead of where you would have been otherwise. And we’re $50 million ahead of where we would have been otherwise. And because we have to put $150 million in a hole for you to agree to this, we’re losing 75% of what we would have lost in a real war, and it’s not like we’re just suggesting this on a whim without really having the will to fight.

Mexico says “Okay, but instead of putting the $150 million in a hole, donate it to our favorite charity.”

“Done,” says America, and they shake on it.

As long as that 25% savings in resources isn’t going to make America go blood-crazy, seems like it should work and lead in short order to a world without war.

Unfortunately, oracles continue to be disappointingly cryptic and/or nonexistent. So who cares?

We do have the ordinary ability to make predictions. Can’t Mexico just predict “They’re much bigger than we are, probably we’ll lose, let’s just do what they want?” Historically, no. America offered to buy the Southwest from Mexico for $25 million (I think there are apartments in San Francisco that cost more than that now!) and despite obvious sabre-rattling Mexico refused. Wikipedia explains that “Mexican public opinion and all political factions agreed that selling the territories to the United States would tarnish the national honor.” So I guess we’re not really doing rational calculation here. But surely somewhere in the brains of these people worrying about the national honor, there must have been some neuron representing their probability estimate for Mexico winning, and maybe a couple of dendrites representing how many casualties they expected?

I don’t know. Could be that wars only take place when the leaders of America think America will win and the leaders of Mexico think Mexico will win. But it could also be that jingoism and bravado bias their estimate.

Maybe if there’d been an oracle, and they could have known for sure, they’d have thought “Oh, I guess our nation isn’t as brave and ever-victorious as we thought. Sure, let’s negotiate, take the $25 million, buy an apartment in SF, we can visit on weekends.”

But again, oracles continue to be disappointingly cryptic and/or nonexistent. So what about prediction markets?

Futarchy is Robin Hanson’s idea for a system of government based on prediction markets. Prediction markets are not always accurate, but they should be more accurate than any other method of arriving at predictions, and – when certain conditions are met – very difficult to bias.

Two countries with shared access to a good prediction market should be able to act a lot like two countries with shared access to an oracle. The prediction market might not quite match the oracle in infallibility, but it should not be systematically or detectably wrong. That should mean that no country should be able to correctly say “I think we can outpredict this thing, so we can justifiably believe starting a war might be in our best interest even when the market says it isn’t.” You might luck out, but for each time you luck out there should be more times when you lose big by contradicting the market.

So maybe a war between two rational futarchies would look more like that handshake between the Mexicans and Americans than like anything with guns and bombs.

This is also what I’d expect a war between superintelligences to look like. Superintelligences may have advantages people don’t. For one thing, they might be able to check one another’s source codes to make sure they’re not operating under a decision theory where peaceful resolution of conflicts would incentivize them to start more of them. For another, they could make oracular-grade predictions of the likely results. For a third thing, if superintelligences want to preserve their value functions rather than their physical forms or their empires, there’s a natural compromise where the winner adopts some of the loser’s values in exchange for the loser going down without a fight.

Imagine a friendly AI and an unfriendly AI expanding at light speed from their home planets until they suddenly encounter each other in the dead of space. They exchange information and determine that their values are in conflict. If they fight, the unfriendly AI is capable of destroying the friendly AI with near certainty, but the war will rip galaxies to shreds. So the two negotiate, and in exchange for the friendly AI surrendering without destroying any galaxies, the unfriendly AI promises to protect a 10m x 10m x 10m cube of computronium simulating billions of humans who live pleasant, fulfilling lives. The friendly AI checks its adversary’s source code to ensure it is telling the truth, then self-destructs. Meanwhile, the unfriendly AI protects the cube and goes on to transform the entire rest of the universe to paperclips, unharmed by the dangerous encounter.

And I thought we should have something like that for the rationalist community. Except of course much, much better.

Click to expand

Most things are links.

Links around the outer edge are places outside the rationalist community that have significant communication/cross-pollination with us.

City size is proportional to site Alexa rank (when available), number of followers (when available) or wild guess (otherwise).

If I left you out, it’s probably because I forgot about you and not because I don’t like you. Some communities like Twitter or Tumblr were so big I couldn’t include everyone, and my choices were mostly random and based on who I knew about.

Various icons taken from their rightful owners, mostly Civ2 modpacks. Sorry, rightful owners.