codex Slate Star Codex

THE JOYFUL REDUCTION OF UNCERTAINTY

OT109: Opulent Thread

Posted on August 26, 2018 by Scott Alexander

This is the bi-weekly visible open thread (there are also hidden open threads twice a week you can reach through the Open Thread tab on the top of the page). Post about anything you want, ask random questions, whatever. You can also talk at the SSC subreddit or the SSC Discord server. Also:

1. The due date for Adversarial Collaboration Contest entries was last week. I now have four entries submitted: TracingWoodgrains + MichaelP, Mark + Mark, Flame7926 + AReader, JohnBuridan + ChristianFlannery. If you submitted an entry and I didn’t get it, please let me know below. If you almost have an entry done and want to beg for more time (no more than a week or so), you can do it below and I might give in. I’ll reserve the first post on this thread for contest discussion.

2. Comment of the week is a reader refining the claim (see eg here) that supposed magical immunity to bullets inspires some warriors to be braver.

3. I’ve unbanned various people whose terms of ban were up or almost up. I know I banned skef a few months ago, but I can’t find the ban in the usual place and so I cannot rescind it. Skef should check if they can comment. If not, they might want to register an alternate account since I can’t figure out how to unban them. Sorry about the inconvenience.

4. Thanks to everyone who’s arranged SSC meetups the past few weeks, including the digital meetup on Throne. As always, you can find upcoming meetups near you on the meetups page. If you hosted or attended a meetup, please comment to let me know how it went.

5. And thanks to everyone who pre-registered for the informal experiment on CO2. Do whatever you’re going to do, and I’ll have another survey up in about a month where you can record your results.

Posted in Uncategorized | Tagged | 189 Comments

Carbon Dioxide: An Open Door Policy

Posted on August 23, 2018 by Scott Alexander

[Content note: reading this post might cause feelings of suffocation or provoke panic attacks in susceptible individuals. Epistemic status is very speculative.]

Last month I moved into a small cottage behind a big group house. The cottage is lovely. The big group house is also lovely, but the people in it started suffering mysterious minor ailments. Headaches, fatigue, poor sleep – all the things that will make your local family doctor say “Take two placebo and call me in the morning”. Using my years of medical training and expertise, I was able to…remain completely unaware of the problem while my housemates solved it themselves.

There’s been a flare-up of research interest in indoor carbon dioxide levels, precipitated by a Berkeley study (paper, popular article) finding that increasing CO2 concentration from the level of a well-ventilated building to the level of a poorly-ventilated building had profound effects on cognitive ability, cutting various test scores by as much as 50%. This was so dramatic as to be implausible, but seems to match the result of previous Hungarian studies and a later Harvard study on the same subject. The Harvard team later replicated their result with real workers in real offices and found that, controlling for other factors, workers in the best-ventilated offices scored about 25% better on cognitive tests than in the worst-ventilated ones. NASA got really interested in this research because spaceships require a lot of intellectual work and don’t have a lot of open windows. They’re still running tests but they say that “preliminary results suggest differences” between better- and worse- ventilated environments.

On the other hand, a 2017 study failed to find the effect, possibly because their cognitive tests were easier. And bloggers have pointed out that submarines have more CO2 than the worst terrestrial buildings, but don’t have any problems overt enough for the Navy to notice or worry. So it’s a crapshoot of contradictory results and considerations, just like everything else.

Aware of this research, my housemates tested their air quality and got levels between 1000 and 3000 ppm, around the level of the worst high-CO2 conditions in the studies. They started leaving their windows open and buying industrial quantities of succulent plants, and the problems mostly disappeared. Since then they’ve spread the word to other people we know afflicted with mysterious fatigue, some of whom have also noticed positive results.

When I heard about this, my first question was: didn’t any of these people notice they only felt bad at home? Shouldn’t it have been a big red flag when they went to the office, or went for a walk, and all their problems disappeared? This can’t be too big a deal, or else “I feel bad in my house, but fine everywhere else” would be a more common complaint.

My housemate Kelsey referred me to the work on CO2 and sleep. Right now this is just a few papers by a guy named Strøm-Tejsen, but the implications are pretty important. He notes that however bad your carbon dioxide levels are during the day they’re probably much worse at night, when you shut yourself up in a small room, close all the doors and windows, and just breathe for like eight hours straight. Normal outdoor air is about 400 ppm CO2 (more by the time you read this; thanks, fossil fuel industry!) A well-ventilated building during the daytime is about 700 ppm, and a poorly ventilated building during the daytime about 1400 ppm. But the average bedroom at night can be 2000 ppm or more. Friend-Of-The-Blog Gwern got a CO2 monitor to test these findings, and confirmed that while his daytime CO2 was around 500 ppm, nighttime CO2 in his bedroom could get as high as 3000 ppm. MIT’s Joel Jean discussed trying the same in this Medium post, with similar results:

I live in California, and Gwern presumably lives in some kind of formless cybermatrix, so we don’t have to worry about seasons. But Dr. Jean lives in Massachusetts, and he found that during the winter, indoor CO2 went up even further, in some cases exceeding OSHA’s rules for permissible workplace exposure:

If CO2 can affect sleep quality, that would explain how it could produce a whole-day effect. Strøm-Tejsen tests this on sixteen subjects and finds that “objectively measured sleep quality and the perceived freshness of bedroom air improved significantly when the CO2 level was lower, as did next-day reported sleepiness and ability to concentrate and the subjects’ performance of a test of logical thinking.” Good things about this study: subjects were blinded to condition, the paper contains a pilot experiment and a main experiment which mostly replicate each other’s results. Bad things about this study: the experiments were about n = 15 each, the researchers didn’t correct for multiple comparisons, and they admit to manipulating the statistics surrounding their logical reasoning tests to get better results. But if I just look at their tables and try to ignore their manipulation, I’m at least kind of impressed:

And experts seem to take their results seriously – for example, here’s NASA again. And we know from sleep apnea and studies that high physiological levels of carbon dioxide can cause sleep disturbances. I can’t figure out how to convert external ppm to internal likely level of carbon dioxide in the blood, but maybe this could provide a plausible mechanism.

I’m reluctant to be too numerical about all this, because everything about health has massive individual variability. Three people share one of the bedrooms at my group house (look, the Bay Area is really bad). One of them got the typical symptoms of excess CO2 really bad; the other two were fine. Some people are just going to be more sensitive to this kind of thing – the same way three people can drink the same amount of alcohol, two of them will get pleasantly buzzed, and one of them will black out.

I’ve tried sleeping with my door open the past few nights, and I haven’t noticed any difference. Probably I shouldn’t; my house is well-ventilated and I wasn’t feeling too bad beforehand. But I’ve started recommending a few of my patients with mysterious sleep issues try the same thing. It’s too early for results so far, and the science behind it is weak, but it seems like a cheap experiment.

Since the main source of CO2 is human exhalation, I’m most worried about buildings where many people are crammed into small spaces in close proximity (hello, Bay Area readers!). Since the main way CO2 gets cleared is through ventilation, I’m most worried about buildings made to strict environmental standards with great insulation (hello, Bay Area readers again!).

If you’re concerned about this, the best solution is to open a window or an internal door in your bedroom at night. If for some reason this is impossible, the second-best solution is to get certain succulents or other plants that participate in the ominously-named process of “dark fixation” – ie do their plant breathe-in-CO2-and-breathe-out-oxygen thing at night. This is also called “crassulacean acid metabolism” and Googling either term will get you a list of appropriate species. It will probably take like ten succulents to do much to CO2 levels, but a room full of succulents on every flat surface is also kind of #aesthetic.

I’m interested in more data on this, so if you’re planning on experimenting with changes to your nighttime air quality based on this post, please fill out this form to register for an informal quasi-experiment. I’ll follow up with a form for you to give your results in a couple of weeks.

Oh – and sorry for the content warning at the top, but I’ve felt kind of low-grade suffocate-y throughout writing this post, and had to go out and take a couple of breaths of fresh air a few times. Remember – perception is fundamentally Bayesian, and combines external sensation with internal expectations; this is why placebos can have such a profound effect on pain. Perception of air quality vs. suffocation seems to be especially susceptible to this, which is probably one of the major etiological factors behind panic attacks. Just repeat to yourself that it all adds up to normality: the air quality in your room hasn’t changed since when you were feeling just fine before you started reading this article, so you should be okay.

Posted in Uncategorized | Tagged | 202 Comments

Practically-A-Book Review: EA Hotel

Posted on August 20, 2018 by Scott Alexander

Effective altruism (“EA”) is a movement dedicated to redirecting charity-related resources to the most important and successful charities. In practice this involves a lot of research into how important various problems are, and how well various charities work. Some of this research is done by well-funded official institutions. Other research, maybe exploring more unlikely scenarios or starting from weirder assumptions, is done as individual labors of love. These smaller-scale efforts might be self-funded, or supported by a few small donors. For example, Wild Animal Suffering Research, which investigates ways to improve the lives of animals in the wild, has yet to catch the attention of any hedge fund managers.

Like everything else, effective altruism is centered around San Francisco. San Francisco is the most expensive city in the world, so this isn’t very efficient; most of the relevant research can be done online from anywhere in the world. The official institutional charities eat the expense in exchange for the extra access to funders and other resources, but it’s a problem for small independent organizations. There’s been lots of research into possible solutions, but only if “let’s see how many people we can cram into one house in Berkeley” counts as “research”.

Blackpool is a beach resort in northern England. “Beach resort in northern England” is exactly as fun as it sounds, so nobody goes there. Everything is really cheap, and you can buy a whole hotel for the cost of a parking spot in San Francisco. Enter Greg Colbourn, an effective altruist and successful cryptocurrency investor. He bought the 17-bedroom Hotel Athena and wants to offer free room and board to researchers working on effective altruist projects

The plan is to make the E and A different colors than the rest of the word, eg ATHENA, for a double meaning.

Colbourn writes::

Do you long to be free from material needs and be able to focus on the real work you want to do? I know I’ve certainly been in that situation a few times in the past, but instead have lost time doing unimportant and menial jobs in order to be able to get by financially. Talented effective altruists losing time like this is especially tragic given that a lot of cause areas are currently constrained by the amount of quality direct work being done in them.

Buildings in the run-down seaside holiday resort of Blackpool (UK) are really cheap. I’ve bought a 17 bedroom hotel with dining room, lounge and bar for £130k. Assuming a 7% rental yield (which is reasonably high), this works out at about £45 per person per month rent. Factoring in bills, catering, and a modest stipend/entertainment budget, living costs could be as low as £5700/person/year (or lower for people sharing rooms, see budget). This is amazing value for hotel living with all basic services provided.

The idea is to invite people to live there, with all their expenses covered by donors, for up to two years. Funding is already in place (via me) for the first year of operations. The project will be managed by someone who lives on site and deals with all the admin/finances, shopping/cooking/cleaning/laundry, socials/events and morale – they will also have free living expenses, and be paid a modest salary. Note that this should be considered as a potential high impact, high prestige supporting role, for those excited to be involved in such a capacity on an EA mission. Guests will be free from concerns of material survival, and be able to have prolonged and uninterrupted focus on whatever projects they are working on. Obviously these will be largely limited to purely desk-based, or remote work.

Is this really more effective than just spending the money on grants that allow researchers to support themselves? If you know anything about effective altruism, you’ll have guessed that someone has already done the math on this. The hotel expects to be able to support people for £5700 (= $7300) / year, so:

For [giving people grants] to be equivalent, they would have to be living (or go to live) somewhere where the costs of living are comparable, otherwise I would effectively be buying them time at a much higher cost. For example, someone frugal living in London might be able to get by on £15,000 a year. So for the same costs they would get about a third of the time; and this is before factoring in the free time-saving services (cooking, laundry, cleaning etc). So unless they were ~4x as productive as the average hotel guest, this would be a bad deal for me as a donor. Also, the community aspect of having a significant number of EAs in the same place is probably worth something too in terms of increased focus, collaboration and morale boosting productivity. Community building via deeper in-person ties is becoming increasingly important to the further development of the EA movement.

Also, and in considerations I have to admit did not occur to me:

Blackpool might be hard to get to in the event of a catastrophe, but the flip side of this is that there would be a lower risk from hostile actors (mercenaries, milita), as well as lower direct damage from nukes and fallout…the cellars could serve as a nuclear bunker of moderate protection. It will be relatively low cost to keep a stockpile of long lasting food down there, which could be slowly used and replenished by the kitchen over a 2-5 year cycle. There is already bathroom plumbing down there, and other essentials could be easily added.

The hotel has another advantage, harder to explain. There’s a lot of concern in the EA community about Goodhart’s Law: “When a measure becomes a target, it stops being a good measure”. Think of it as “teaching to the test”, but for a broad and metaphorical definition of “test”. Or as cancer researchers who can’t research the most promising fields because they’re busy researching the fields that sound the best on a grant application, or the ones that will produce some small payoff in a year so that grantmakers classify them in the “productive” category and renew their funding. Or as the CEO who can’t pursue the most promising strategy because he’s busy pursuing the strategy that will maximize shareholder value next quarter. Effective altruism wants to avoid that failure mode. But as long as you’re in the business of distributing scarce resources to the people who seem to be able to use them best, you’re at risk.

It looks like the hotel wants to accept all comers who are making a good-faith effort to work on effective altruist ideas. I don’t know their exact plan or whether it will work. But having a low-barrier-to-entry hotel in an abandoned beach resort at least sounds less exploitable than giving people low-barrier-to-entry multi-thousand-dollar grants. The white paper says:

Scammers can be avoided, at least in the first instance, by requiring a prior history of involvement in EA and references from people with standing in the community. The bigger risk is well intentioned but ineffective grant recipients (/hotel guests) just not delivering. However, we are likely already in a situation where EAs frequently go off on their own to work on individual projects that don’t deliver, but we don’t hear much about it because of selection effects/social desirability bias

And:

Communal mealtimes will be encouraged though, as a way for camaraderie and a sense of community to develop amongst guests. Friendships will be formed, problems, and their solutions, discussed, and some amount of fun and laughter will be had too hopefully! Also, some people may find that the inevitable discussion of work that will arise will lead to some amount of peer pressure that will aid in keeping them productive […]

In terms of guarding against people becoming freeloaders (or effectively squatters), perhaps there could be a mandatory minimum of very brief (one page) reports on progress to be filed every three months. There would also be some indirect self-generated pressure from guests worrying about donors getting their money’s worth (although perhaps this will only be applicable for the more highly scrupulous). But then there would have to be a mechanism for chucking people out if they are achieving very little. The hard cap of two years on length of free stays per person will be an ultimate limiter, but it might be prudent to proceed with caution regarding taking on long term residents (perhaps a three month probation period would be useful).

Would people really want to live here? Apparently. In the month or two it’s been open, it’s already picked up three residents. The Facebook poll shows 115 expressions of interest, so if even a fraction convert to real bookings, they can keep their seventeen rooms full for a long time.

I have no idea if this project is a good use of anyone’s time or money, but I find it endearing. Part of this is of course the giant middle finger raised at Henry VIII – undissolution of the English monasteries and all that. But it’s also how I picture a near-term future where everything goes better than expected. A modest basic income could give everyone the opportunity to live a spartan but generally pleasant lifestyle in a friendly and meaningful community of intellectually-aligned people. As the old saying goes: “If it happens, it’s possible”.

Further links:

1. EA Hotel “white paper”

2. EA Hotel website

3. EA Hotel Facebook group

4. Apply to stay at the EA Hotel. Effective altruists are welcome to apply to live there for free; others are invited to pay for rooms at the usual rate if they want to go there on holiday / gawk at the interesting social experiment.

Posted in Uncategorized | Tagged | 122 Comments

The Parentheses Riddle

Posted on August 16, 2018 by Scott Alexander

Because I hate you, I included this question on the SSC survey:

It’s a weird trick question, but I would say B is right. Imagine converting “(” to X and “)” to Y. Then the first answer is XYXY, and the second answer is YXXY. I suppose you could group the parentheses in pairs, in which case the answer would be “both”, but in practice few people wanted to say that. Of the 6,000 answers I received, most were either A or B. And one factor had a dramatic effect: age.

This is a big effect. People in their 20s were more than twice as likely to choose B as people their 60s. There’s a slight improvement after 70, but I think that’s just noise caused by a low sample size in that group.

My first thought was that the younger population on this blog is disproportionately techies, and techies have to work with very finicky parentheses all day. There was indeed a slight tendency for techies to do better on this, but it was a very small part of the effect. Even controlling for that, or limiting the analysis to only non-techies, most of the effect remained.

My second thought was that maybe this was an effect of older people gradually getting less sharp. But IQ itself was correlated with the parentheses question much less than age was. SAT score also didn’t correlate very well. And I would expect the most dramatic age-related declines to be after fifty, but the parentheses effect seems to – if anything – slow down then.

My third thought is that older people had less time to waste staring at a dumb survey and trying to solve everything exactly right. But the Squares And Circles Illusion also measures how long you’re willing to stare at a dumb question, and age was negatively correlated with this one. Big Five conscientiousness also did not affect parentheses answers.

I am pretty stumped by this. Right now my guess is that it is caused by age-related cognitive decline, but the connections go deeper than IQ – there’s something about age that affects the parentheses-reading faculty in particular regardless of how smart you are. I know there’s a medium-sized dementia-screening industry. I don’t know how good they are at detecting the normal level of cognitive decline in relatively young and intelligent people – but I would be interested to hear their opinion on this question and whether it has interesting properties beyond those they already know about.

EDIT: Commenters propose that young people might have seen the riddle before on Facebook or Reddit or some other young-person-Internet-place.

Posted in Uncategorized | Tagged , , | 253 Comments

SSC Survey: Scattered Negative Results

Posted on August 15, 2018 by Scott Alexander

Traffic to this blog is declining. I need to act decisively to draw people back. Write something so interesting it can’t help but go viral. I’m going to write about…negative results from the perception questions on last year’s survey.

The last SSC survey had a lot of optical illusions and visual riddles. I had hoped to expand on some of the work in Why Are Transgender People Immune To Optical Illusions and Can We Link Perception And Cognition? This post is a very brief summary of results and, basically, an admission of failure. While I was able to replicate the same suggestive results as in the last survey, I was unable to expand on them, strengthen them, or really turn them into any kind of interesting framework.

I was able to weakly replicate the headline result from Why Are Transgender People Immune To Optical Illusions: transgender status still correlated with all three mask illusions, and with the average of all three mask illusions, but very weakly: r = -0.04, p = 0.001. This was true even when I excluded everyone who took place in last year’s survey, providing an independent confirmation of the result. But with correlations this low, it’s hard to get too excited.

I was also able to weakly replicate the headline result from Can We Link Perception And Cognition?. I haphazardly gave people a “weirdness score” based on them having more mental illnesses, more unusual political opinions, and more minority sexual/gender identities (without looking at their illusion results). People with higher weirdness scores consistently had more ambiguity-tolerant results on illusions, with correlations around r = 0.05 for most tests. They also had notably higher average Tolerance of Uncertainty Test scores. But none of these results were very striking and there was minimal individual structure in them. If I was going to take this further I would have come up with a more principled definition of weirdness, but at this point it doesn’t seem worth it.

What do I mean by saying these results are weak and lack internal structure? To give an example: the last survey focused on a single optical illusion, the Hollow Mask. This survey used three different versions of the Hollow Mask in the hopes of removing noise and getting a higher-fidelity mask perception signal. This didn’t work at all. The correlations between the three masks were very low. For example, there was only an r = 0.09 correlation between being able to see the second mask illusion and the third mask illusion. While all correlations were significant, it doesn’t seem fair to conceptualize them as testing the same perceptual function.

Given that I couldn’t even get different versions of the same illusion to line up, you can guess that I didn’t get much correlation between different illusions. For example, the Tables Illusion correlated with average score on the Mask Illusion at about r = 0.06, p = 0.001. I was also able to replicate the correlation in the literature between autism and the Tables Illusion, r = 0.03 p = 0.01, but again this was very small.

Were there any actually large correlations? Surprisingly, the Surgeon Riddle produced some of the most impressive results on the whole survey. For example, it correlated at r = 0.18, p = 0.001 with ability to see “the the” as two separate words. Given that the Surgeon Riddle seems much less perceptually basic than the other things on here, I believe this is probably some kind of confounder, maybe amount of time people spend on each question or something like that.

There was an overwhelmingly strong effect of age on the Parentheses Riddle and nothing else. 62% of people in their 20s got it right, compared to 29% of people in their 70s. This was surprising enough that it might be worth its own post. I was unable to wring anything else out of this no matter how hard I tried.

I tried a factor analysis to draw some factors out of all the different illusions. SPSS came up with six factors, each of which explained 5-10% of the variance; given how many variables I put in, this doesn’t look that much better than chance. None of the factors correlated surprisingly well with anything else, nor was there an obvious pattern in which illusions they grouped together.

In conclusion, most illusions had very low (less than r = 0.1) but highly significant (less than p = 0.001) correlations with one another and with various mental illnesses. There was no clear pattern to the correlations, although they did generally replicate past observations and findings from the literature. I can’t really say for sure if there’s a real effect here or it’s all just confounders, and if there is a real effect I definitely can’t tease out its structure or say anything about it for sure. I encourage other people to look into these and see if they can do better. You can download the survey results here.

I have a couple other things I pre-registered to investigate but never got around to before, so here they are:

Political conflict theory was slightly correlated with various questions on the Tolerance of Ambiguity and Tolerance of Uncertainty tests, but not enough to be interesting. For example, when asked whether political extremists you don’t like (eg fascists) were making understandable mistakes or just evil, the answer was correlated at about r = 0.05 with various questions like “Small doubts keep me from acting” and “The ambiguities in life stress me out”. The more stressed people were by ambiguity, the more they thought extremists were evil. This was not a very large effect, it varied from question to question in an unpredictable way, and I wouldn’t have mentioned it if I didn’t feel obligated to tie up preregistered loose ends.

Contra my prediction, there was no relationship between autism and the likelihood of giving process-based (“meta-level”) responses to categorization questions as opposed to person-based (“object-level”) responses. For example, when asked to judge fascists beating up minorities vs. minorities beating up fascists, autistic and neurotypical people were about equally likely to base their responses on principles (ie “beating up people is wrong”) vs. on the groups involved (ie “fascists are bad”).

Contra my prediction, people with ADHD did not describe themselves as more ambitious. They did describe themselves as more risk-taking and more likely to prefer a buzzing-city aesthetic than a quiet-country aesthetic, but the results, although significant, were too low in magnitude to be interesting.

As I mentioned in a previous article, the results of the AI Persuasion Experiment two years ago did not persist.

Posted in Uncategorized | Tagged , | 38 Comments

SSC Survey Results: ADHD And Rejection Sensitivity

Posted on August 14, 2018 by Scott Alexander

Introduction

ADHD is typically considered a disorder of attention and focus. There are various other traits everyone knows are linked – officially, hyperactivity and “behavior problems”; unofficially, anger and thrill-seeking – but most people consider these to be some sort of effect of the general attention deficit.

Dr. William Dodson pushes a different conception, where one of the key features of ADHD is “rejection-sensitive dysphoria”, ie people with the condition are much less able to tolerate social rejection, and more likely to find it unbearable and organize their lives around avoiding it. He doesn’t deny the attention and focus symptoms; he just thinks that rejection sensitivity needs to be considered a key part of the disorder.

I say “Dr. William Dodson pushes”, but this requires a little research before it becomes apparent. What a Google search shows is just a bunch of articles saying that rejection sensitivity is a key part of ADHD that gets ignored by non-expert psychiatrists and that it’s important to educate patients about it and include it in any treatment plan. My conclusion is that all of these articles can be traced back to Dr. Dodson or people inspired by Dr. Dodson, of which there are many. The ADHD patient community has gotten really into this and pushed it in a lot of support groups and patient communities and so on, where it is repeated uncritically as “an important ADHD feature psychiatrists often forget about”. But the genesis is just Dr. Dodson saying so, with limited formal evidence.

See for example ADDitude Magazine, which says that:

Rejection sensitivity is part of ADHD. It’s neurologic and genetic. Early childhood trauma makes anything worse, but it does not cause RSD. Often, patients are comforted just to know there is a name for this feeling. It makes a difference knowing what it is, that they are not alone, and that almost 100% of people with ADHD experience rejection sensitivity. After hearing this diagnosis, they know it’s not their fault, that they are not damaged.

…and suggests very high doses of alpha-agonists or MAOIs (!) as a treatment. Or WebMD, which says:

Up to 99% of teens and adults with ADHD are more sensitive than usual to rejection. And nearly 1 in 3 say it’s the hardest part of living with ADHD.

Neither article cites any sources.

I am skeptical of the rejection-sensitive dysphoria concept. Part of it is the lack of evidence beyond Dr. Dodson’s personal experience, which my own personal experience contradicts. But part of it is that it seems suspicious for the Forer Effect, the tendency for everyone to believe a generic statement describes them especially. This effect is beloved of psychics – tell a client a few Forer statements and they’ll walk out convinced you can read minds – but it causes problems in the more reputable sciences as well. Here is Forer’s original list of statements that produce the effect:

1. You have a great need for other people to like and admire you.
2. You have a tendency to be critical of yourself.
3. You have a great deal of unused capacity which you have not turned to your advantage.
4. While you have some personality weaknesses, you are generally able to compensate for them.
5. Disciplined and self-controlled outside, you tend to be worrisome and insecure inside.
6. At times you have serious doubts as to whether you have made the right decision or done the right thing.
7. You prefer a certain amount of change and variety and become dissatisfied when hemmed in by restrictions and limitations.
8. You pride yourself as an independent thinker and do not accept others’ statements without satisfactory proof.
9. You have found it unwise to be too frank in revealing yourself to others.
10. At times you are extroverted, affable, sociable, while at other times you are introverted, wary, reserved.
11.Some of your aspirations tend to be pretty unrealistic.
12. Security is one of your major goals in life.

The first statement sounds a lot like rejection sensitivity. I will admit that one of the strongest pieces of evidence in favor of rejection-sensitivity dysphoria is all the ADHD patients who comment on any article about it with “Wow, this perfectly describes my experience! This is amazing!” – but if it were a Forer Effect, this would be less surprising.

There have been two studies attempting to investigate the ADHD-rejection link. The first, Canu & Carlson, failed to find it. The second, Bondu and Esser, did find it. This may be because the second had a larger sample size. But the second also correlated rejection-sensitive dysphoria with ADHD symptoms, rather than relying on an ADHD diagnosis. It also failed to screen out other psychiatric disorders, many of which are comorbid with ADHD and look a lot like rejection sensitivity (eg anxiety, depression). Overall this does provide some support for the hypothesis, but so far it looks like the only attempt to formally test it.

Methods

I investigated the idea of rejection-sensitive dysphoria through the Slate Star Codex survey, an online survey of readers of this blog. It received 8,077 responses, including 717 people with professionally-diagnosed ADHD, 860 more people with self-diagnosed ADHD, and several thousand with other psychiatric conditions.

The survey contained the following question:

I am more sensitive than others to rejection, teasing, criticism, or my own perception that I have failed or fallen short

This is a standard screen for rejection-sensitive dysphoria quoted as Dr. Dodson’s proposed addition to the DSM diagnostic criteria for ADHD (though I cannot find confirmation of this from Dr. Dodson himself). Most of the citations frame it more strongly, eg “For your entire life have you always been much more sensitive than other people you know to rejection, teasing, criticism, or your own perception that you have failed or fallen short?”. I cannot remember why I made the survey version weaker and this is a weakness of this study. Respondents were asked to rate their level of agreement with this statement on a scale from 1 (strongly disagree) to 7 (strongly agree).

The following analysis strategy was devised before looking at any data: to compare people with ADHD only to people with no psychiatric condition, to people with only one other psychiatric condition, and to people with multiple psychiatric conditions. The three other conditions with a large enough sample size to use were autism, anxiety, and depression. Analyses were done both among self-diagnosed patients and professionally diagnosed patients. I committed to publishing this post regardless of what the analysis said.

All other analyses were done after seeing the data, and should be considered exploratory only.

Results

7,264 people answered the question about rejection-sensitive dysphoria:

There was a slight bias towards affirmative answers, consistent with a small (but not large) Forer effect.

Among people with self-diagnosed psychiatric conditions, answers varied as follows:

ADHD had the lowest rate of rejection sensitivity among the four psychiatric conditions studied, although it was still higher than people with no condition.

Among people with professionally-diagnosed psychiatric conditions, the picture was much the same:

Once again, ADHD had the lowest rate of rejection sensitivity among the four psychiatric conditions studied, but was higher than people with no condition.

Articles alternately claim that “99%” “99.9%” or “almost 100%” of people with ADHD endorse rejection sensitive dysphoria. My survey found less dramatic results:

58.5% of ADHD patients gave answers of 5, 6, or 7 on a one-to-seven scale of rejection sensitivity. This was statistically significantly different from the 46.2% of the general population who did the same, but not in nearly as impressive a way as suggested by the 99.9% numbers that some people quote.

Discussion

My survey found that there was a weak but statistically significant tendency for people with ADHD to have higher rejection sensitivity than people without psychiatric conditions. However, this was a general tendency for any psychiatric disorder, and ADHD had the lowest rates of rejection sensitivity among the four disorders studied. This suggests the conception of rejection sensitivity as fundamental to ADHD is inaccurate and should not be included in discussions of the condition.

Flaws in this study include the survey methodology, which relied on self-report to get each respondent’s diagnosis or lack thereof. On a deeper level, if the current conception of ADHD is fundamentally flawed because of ignoring rejection-related symptoms, then no diagnosis using the current method can be trusted. But to call something ADHD at all, rather than a completely new illness, requires that it have some correlation with ADHD features as classically understood. Given that there is no generally-accepted objective measure of ADHD, these diagnoses are the best measurement we have at this time.

Other flaws included the admittedly weak single-question test for rejection sensitivity, the inexplicable softening of the single question from its classic version, the lack of adjustment for any confounders, and the heavily selected sample of SSC readers.

However, it is still difficult to argue with the magnitude of the result in a large sample size such as this one. ADHD has the least rejection sensitivity of any of the disorders studied. None of the flaws in this study seem to be of the type or the magnitude that could incorrectly produce this result.

I conclude that it is important to beware of Forer effects in ideas about psychiatric symptomatology, especially ideas that spread among patient groups without formal study or buy-in from researchers. Forer statements are often vague, slightly self-flattering, or suggest that the person involved needs special care or respect from others. Since most people feel this way, it may be easy to convince people with a condition that the condition implies a Forer statement. My informal survey does not support this connection at this time. Other teams should follow up with more formal experiments to try to confirm.

Posted in Uncategorized | Tagged , , | 122 Comments

OT108: Opangolin Thread

Posted on August 12, 2018 by Scott Alexander

This is the bi-weekly visible open thread (there are also hidden open threads twice a week you can reach through the Open Thread tab on the top of the page). Post about anything you want, ask random questions, whatever. You can also talk at the SSC subreddit or the SSC Discord server. Also:

1. Comments of the week are everything by sclmlw on cancer research (see eg this thread) and Cerastes on why we should make humans cold-blooded.

2. Since the Meetup Times And Places thread was posted, meetups have been added in Moscow, Columbus, Sacramento, Berkeley, San Jose (CA), and Portland (OR). Details have been changed for Boston, St. Louis and Wellington. If you’re in any of those cities and interested in attending, please go back and check the new information.

3. I’m interested in reports from meetups that have already happened. In fact, if you organized a meetup, please keep track of how many people attended, since I might survey people on that later.

Posted in Uncategorized | Tagged | 1,196 Comments

SSC Meetups 2018: Times And Places

Posted on August 9, 2018 by Scott Alexander

Thanks to everyone who offered to host a meetup. We’re scheduled for meetups in 85 cities (and one ship!) in 25 countries, soundly beating last year’s list. Full list of cities, times, and places is below.

Most people who are on the fence have said they’ve enjoyed going. Most people who felt intimidated about going have said they’ve enjoyed going. Most people who felt they were too different from the median SSC reader to fit in have enjoyed going. Most people who worried they weren’t smart enough to fit in have enjoyed going. Etc. Some tips from past experience with these meetups:

1. If you’re the host, bring a sign that says “SSC MEETUP” and prop it up somewhere on a table.
2. Bring blank labels and pens for nametags.
3. Collect everyone’s name and email address in a spreadsheet (template), so you can start a mailing list to make organizing future meetups easier.
4. If it’s the first meetup, people are probably just going to want to talk, and if you try to organize some kind of “fun” “event” it’ll probably just be annoying.
5. Some things that have worked for later meetups include people giving short presentations on topics of interest to them, or discussion of some particular blog post.
6. Nothing is going to get done unless there’s a Schelling point for who has to do it, and right now that’s the meetup organizer.
7. It’s much easier to schedule a second meetup while you’re having the first compared to trying to do it later on by email.
8. Surprisingly many people will love you forever if you bring stim toys.
9. In case people want to get to know each other better outside the meetup, you might want to mention reciprocity.io, the rationalist friend-finder/dating site. It runs off Facebook, so you have to Facebook friend the other person first.
10. If you have a vague location like “in the mall” or “at the North Park”, nobody will ever find each other. Give a specific place (eg “at the North Park, by the big oak tree in the northwest corner”) and be carrying a sign saying “SSC MEETUP”. If you were too vague in your description, comment with a better one and I can edit it in.

Please look over your meetup to make sure it’s correct. If you have corrections, further details, or a new meetup to add, please post a top-level comment below containing the text “@SCOTT” so I can find it easily.

LIST OF CITIES

(all times are local, and PM if not indicated otherwise. You can also search geographically using the map on the LW/SSC meetup page)

Amsterdam, Netherlands
Time: Saturday, August 11, 3:00
Location: On the grass in front of Weesperzijde 100
Contact: mathijs[dot]henquet[at]gmail[dot]com, Meetup page, Facebook group

Ann Arbor, Michigan
Time: Saturday, August 11, 7:00
Location: All Hands Active, 225 East Liberty
Contact: 7343520125

Austin, Texas
Time: Saturday, September 1, 1:30
Location: Central Market, 4001 N Lamar
Contact: ???, but see Google group

Bangalore, India
Time: Sunday, August 26, 4:00
Location: Matteo Cafe, Church Street
Contact: m.nihalmohan[at]gmail.com

Berkeley, California (Bay Area central meetup: Oakland, San Jose, Silicon Valley, etc)
Time: Saturday, September 8th, 3:00
Location: Empty field at the intersection of West Circle and Free Speech Bikeway on UC Berkeley campus
Contact: scott[at]shireroth.org, Facebook event

Berlin, Germany
Time: Saturday August 18, 5:00
Location: St Bart, Graefestraße 71, 10967 Berlin
Contact: buddabrotler[at]gmail.com, Facebook event

Birmingham, UK
Time: Saturday, August 25, 3:00
Location: Caffe Nero, 8-9 Lower Temple St, Birmingham B2 4JD
Contact: askew.thomas[at]gmail.com, or see Facebook event

Boston, Massachusetts
Time: Saturday, August 11, 3:00
Location:  Starbucks, 66 Beacon Street
Contact: ardeibel[at]gmail.com

Boulder, Colorado
Time: Saturday, August 11, 3:00
Location: Private home, 3143 Eastwood Ct
Contact: Call corticalcircuitry at 14159883608, Meetup page

Brisbane, Australia
Time: Friday, August 31, 7:00
Location: The Ovolo Inchcolm bar, 73 Wickham Terrace, Spring Hill QLD 4000
Contact: jarred.filmer[at]gmail.com

Bristol, UK
Time: Sunday, September 2, 3:00
Location: Pinkmans Bakery, 85 Park Street, BS1 5PJ Bristol
Contact: whatagoodemailaddress[at]gmail.com, or see Facebook group

Brussels, Belgium
Time: Saturday, September 1, 2:00
Location: North part of the Parc de Bruxelles
Contact: arthur@milchior.fr

Buffalo, New York
Time: Sunday, August 26, 11:00 AM
Location: Five Points Bakery, 44 Brayton St
Contact: spetey[at]gmail.com

Cambridge, UK
Time: Friday, August 17, 6:00
Location: The Burleigh Arms
Contact: aeluncrombie44[at]gmail.com, or see Facebook event

Canberra, Australia
Time: Friday, August 24, 7:00
Location: 9/8 Walsh Place, Curtin
Contact: arunbharatula[at]outlook.com, or call +61 432356733

Charleston, South Carolina
Time: Sunday, August 12, 3:00
Location: Marion Square
Contact: reilly dot steven at gmail dot com

Chengdu, China
Time: Sunday, August 12, 3:00
Location: The Beer Nest 2, near Tongzilin Metro Station
Contact: campbellnilsen on WeChat

Chennai, India
Time: Sunday, August 26, 11 AM
Location: Pumpkin Tales, Alwarpet (vegan friendly!)
Contact: fufstsahil at gmail dot com

Chicago, Illinois
Time: Saturday, September 8, 12 noon
Location: Art Institute South Garden, by the fountain
Contact: rkrzyzanowski[at]gmail.com, or call 312 618 0316, or Facebook event

Cleveland, Ohio
Time: Saturday, August 18, 1:00
Location: Pour Cleveland Cafe, 530 Euclid Ave
Contact: ferbfreeman[at]gmail.com

Cologne / Köln, Germany
Time: Saturday, August 11, 5:00
Location: Marienweg 43 50858 Köln
Contact: marcel_mueller[at]mail.de

Columbus, Ohio
Time: Saturday, September 8, 3:00
Location: Private residence, 397 Eldridge Ave, Columbus, OH, 43203
Contact: james.thomas.hays[at]gmail.com

Copenhagen, Denmark
Time: Friday, September 28, 5:00
Location: Churchwing of Studenterhuset, Købmagergade 52, 1150 Copenhagen
Contact: mathias@bonde.dk, or call +45 93 93 42 43, or see Facebook event

Denver, Colorado
Time: Tuesday, September 4, 7:00
Location: Apartment common room, 5151 E Yale Cir, Denver, CO 80222-6934
Contact: embrodski[at]gmail.com

Detroit, Michigan
Time: Saturday, August 18, 6:00
Location: Private house. For address, please email contact.
Contact: matt.mattarn[at]gmail.com

Dublin, Ireland
Time: Tuesday, August 14, 6:00
Location: Gingerman Pub, Finean St
Contact: soneillm[at]tcd.ie

Durham / Raleigh / Research Triangle, North Carolina
Time: Wednesday, August 15, 7:30
Location: Ponysaurus Brewing Company, Durham
Contact: willdjarvis[at]gmail.com

Edinburgh, Scotland, UK
Time: Sunday, August 12, 6:00
Location: BrewDog, Lothian Road (table booked under the name “Hughes”)
Contact: ???, but see Facebook group

Edmonton, Alberta, Canada
Time: Saturday, August 11, 1:00
Location: Garneau Remedy Cafe, 8631 109 St NW
Contact: ???

Escazú, Costa Rica
Time: August 25, 2:00
Location: Food Court in Multiplaza, Escazú
Contact: Jorge via WhatsApp at +506 6219 9230

Grand Rapids, Michigan
Time: Saturday, August 25, 3:00
Location: Downtown Market
Contact: ???

Helsinki, Finland
Time: Tuesday, August 28, 6:00
Location: Dubliner (Kaivopiha)
Contact: schelsinkimeetup[at]gmail.com, or see website

Honolulu, Hawaii
Time: Saturday, August 11, 12 noon
Location: Starbucks 680 Ala Moana Blvd A101
Contact: richelson.david[at]gmail.com, 520-369-3359

Houston, Texas
Time: Sunday, August 26, 10 AM
Location: Agora (Coffee Shop), 1712 Westheimer Rd
Contact: alex.m.weldy[at]gmail.com, or see Facebook group

Irvine / Orange County, California
Time: Friday, August 10, 6:00
Location: Center of Aldrich Park, UC Irvine
Contact: gabeaweil[at]gmail.com, or call Gabe at (202) 681-1383

Istanbul, Turkey
Time: Sunday, August 19, 2:00
Location: Espresso Lab in Istiklal Cd., Taksim (across the street from the French embassy)
Contact: vsural[at]gmail.com

Jersey City, New Jersey
Time: Saturday, August 11, 11 AM
Location: Apartment at 30 Newport Pkwy
Contact: 37davidg[at]gmail.com , 4088579822

Kansas City, Missouri
Time: Sunday, August 19, 12 noon
Location: Mud Pie Bakery, 1615 W 39th St
Contact: fustruly[at]gmail.com

Kyiv (Kiev), Ukraine
Time: Saturday, August 11, 11 AM
Location: Bookshop at Lobanovskogo avenue, 6 A (“Дім Книги”), ask for Marichka
Contact: chernyshenko123[at]gmail.com

Lexington, Kentucky
Time: Sunday, August 19, 7 PM
Location: Blue Stallion Brewing, 610 W 3rd Street
Contact: hellernathan19[at]gmail.com

London, UK
Time: Sunday, August 12, 2:00
Location: Private house, 48 St Paul’s Crescent, NW1 9YA
Contact: philip.hazelden[at]gmail.com, or see Facebook group

Los Angeles, California
Time: Wednesday, August 29, 7:00
Location: Wine Bar next to Landmark Theater in Westside Pavillion, 10850 Pico Blvd #312
Contact: betterscale[at]protonmail.com, or see Google group, or contact T3t on Discord

Madison, Wisconsin
Time: Saturday, August 18, 6:00
Location: The Roman Candle, 1054 Williamson St
Contact: cmfrayne[at]gmail.com, Facebook event

Madrid, Spain
Time: Saturday, September 15, 12 noon
Location: Crepes and Waffles, Calle de Fuencarral, 105, Madrid
Contact: pavill01[at]ucm.es, or see meetup page or Facebook group

Manchester, UK
Time: Saturday, August 18, 3:00
Location: The grass patch next to the Wharf Pub in city center M15 4ST coordinates 53.473975, -2.256884
Contact: kernelmanchester[at]gmail.com, or message Ben on Facebook

Melbourne, Australia
Time: Friday, September 7, 6:00
Location: The Queensberry Hotel Dining Room, 593 Swanston St, Carlton
Contact: Call Jugemu, 0438 869 257, or see Facebook group

Miami / Fort Lauderdale, Florida
Time: Saturday August 11, 5:00
Location: Passion del Cielo Coffee, 3301 NE 1st Ave #100
Contact: contact[at]postlibertarian.com, or see Facebook group or Facebook event

Montreal, Quebec, Canada
Time: Saturday, September 1, 2:00
Location: Private house, 810 Avenue Duluth E, Montréal, QC H2L 1B3
Contact: mathieu.roy.37[at]gmail.com, or see Facebook group or Facebook event

Moscow, Russia
Time: Saturday, August 25, 4:00
Location: Kocherga rationality time-club at Bolshaya Dorogomilovskaya, 5 building 2 (https://kocherga-club.ru/#contacts for directions)
Contact: Alex at nikkou.postbox@gmail.com

Munich, Germany
Time: Saturday, August 11, 5:00
Location: Guddenstraße 7, Munich
Contact: lw[at]1platz.de, or see meetup page or Facebook group

Nashville, Tennessee
Time: Tuesday, August 21, 7:00
Location: Industrious, 1033 Demonbreun Ave, Suite 300
Contact: james[at]writechem.com, or see Facebook event

New York City, New York
Time: Saturday, August 18, 3:00
Location: Hudson Eats Food Court, tables near the Black Seed Bagel (will have a sign at the tables), Brookfield Place, 2nd Floor, 230 Vesey Street, Manhattan (map)
Contact: Rachel at (650) 534-7354, or see Google group

Norman, Oklahoma
Time: Saturday, August 11, 6:00
Location: Barnes & Noble 540 Ed Noble Pkwy
Contact: fischerjoe855[at]gmail.com

Northampton, Massachusetts
Time: Saturday, September 8, 6:30
Location: The Roost (cafe), 1 Market Street
Contact: alex[at]alexliebowitz.com

On Board The Battleship USS Iowa
Time: Saturday, September 8, 1:30
Location: Battleship USS Iowa Museum, 250 S Harbor Blvd, Los Angeles / San Pedro
Contact: Who do you think?

Oslo, Norway
Time: Thursday, August 23, 7:00
Location: Starbucks on Aker Brygge
Contact: anders[at]huitfeldt.net

Ottawa, Ontario, Canada
Time: Friday, August 10, 6:00
Location: Art House Cafe, 555 Somerset Street W, Ottawa, Ontario K1R 5K1
Contact: StefanDeYoung[at]gmail.com, or see Facebook group

Oxford, UK
Time: Thursday, August 16, 7:15
Location: The Swan & Castle
Contact: hbesceli[at]gmail.com

Paris, France
Time: Saturday, September 1, 3:00
Location: 48°51’37.9″N 2°17’11.8″E (map)
Contact: Call Jules at +33.6.soixante-quinze.99.65.84

Phoenix, Arizona
Time: Saturday, August 25, 10 AM
Location: 2933 N 17th Dr
Contact: Text Wong at 740 346 9664

Philadelphia, Pennsylvania
Time: Thursday, August 30, 6:00
Location: Tir Na Nog Irish Pub, 1600 Arch Street
Contact: dmcbriggs[at]gmail.com

Pittsburgh, Pennsylvania
Time: Saturday, September 15, 12 noon
Location: Private home, 326 Lincoln Ave, Carnegie PA
Contact: mirandagavrin[at]gmail.com, please RSVP if attending

Portland, Oregon
Time: Saturday, August 11, 12:30
Location: Private residence, 3246 NE 57th Ave (cross street is Klickitat)
Contact: nwalton125[at]gmail.com, encouraged to RSVP

Prague, Czech Republic
Time: Sunday, August 26, 6:30
Location: Tea Room Dharmasala, Peckova, 296/15
Contact: nadvornik.jiri[at]gmail.com, or see Facebook event

Pune, India
Time: Saturday, August 11, 3:00
Location: Cafe Coffee Day outlet at IISER campus
Contact: 7rat13[at]gmail.com

Rochester, New York
Time: Sunday, August 26, 1:00
Location: Wegman’s Market Cafe Food Court, 745 Calkins Road
Contact: christopherjpohlman[at]gmail.com

Sacramento, California
Time: Sunday, August 12, 5:00
Location: 1627 16th St, Sacramento, CA 95814
Contact: amethyst.eggplant[at]gmail.com

Salt Lake City, Utah
Time: Saturday, August 11, 3:00
Location: City Creek Food Court
Contact: wearenotsaved[at]gmail.com, or see Facebook event

San Antonio, Texas
Time: Saturday, August 11, 2:00
Location: Local Coffee @ 7338 Louis Pasteur Dr #204, look for the table with the monster on it
Contact: waffles.he.waffles[at]gmail.com

San Diego, California
Time: Saturday, August 11, 2:00
Location: Bonita Cove Park
Contact: the.god.empress.celestia[at]gmail.com or see Facebook page or Google group

San Francisco, California
Time: Monday August 13, 6:15
Location: 855 Brannan St, lobby across the street from REI
Contact: Call 203-503-7508

San Jose, California
Time: Saturday, September 1, 2:00
Location: Private house, 3806 Williams Rd
Contact: ddfr[at]daviddfriedman.com

Santa Rosa / Sonoma, California
Time: Saturday, August 25, 3:00
Location: 4656 Quigg Dr, Santa Rosa
Contact: adrian@smithdev.io or VivaLaPanda#6386 on Discord

São Paulo, Brasil (Brazil)
Time: Saturday, August 25, 2:00
Location: Vanilla Cafe, R. Antônio Carlos, 452 (near Metro Consolação)
Contact: gusbicalho[at]gmail.com, or see meetup group, Google group, or Facebook group

Seattle, Washington
Time: Sunday, September 9, 2:00
Location: Private house, 5238 11th Ave NE
Contact: speeze.pearson+ssc[at]gmail.com, or see Facebook event

Sheffield, UK
Time: Sunday, September 30, 12 noon
Location: Private house, Crookesmoore, S10 1DJ
Contact: jwhiteley1[at]sheffield.ac.uk

St. Louis, Missouri
Time: Saturday, August 25, 2:00
Location: Coffee Cartel, 2 Maryland Plaza
Contact: djcart137[at]gmail.com, or call 636-219-0944

Stockholm, Sweden
Time: Saturday, August 11, 3:00
Location: Cafe Dox, Stora Nygatan 31, 111 27 Stockholm (if closed, Grändens Cafe, Yxsmedsgränd 2)
Contact: ???, but see Meetup page

Sydney, Australia
Time: Thursday, August 16, 6:00
Location: 565 George Street, level 2, “Fishbowl Room”
Contact: Call Elo at +61438481143

Tel Aviv, Israel
Time: Monday, August 13, 7:00
Location: Melchett 23, Tel Aviv (3rd floor, apartment 12, building code 7890 ‘enter’)
Contact: hi[at]davidkatz.me

Toronto, Ontario, Canada
Time: Saturday, August 18, 5:00
Location: Prenup Pub (St George & College Street)
Contact: Ryan at (647) 562-7358

Tuebingen, Germany
Time: Wednesday, August 18, 8:00
Location: Willi Tübingen, Wilhelmstraße 3/1, 72074 Tübingen
Contact: y.muehlhaeuser[at]gmx.de

Vancouver, British Columbia, Canada
Time: Saturday, August 11, 3:00
Location: Central Library, downtown Vancouver, meet just inside main entrance
Contact: goodtown.r[at]gmail.com

Vienna, Austria
Time: Saturday, August 18, 3:00
Location: Kaisermühlenstrasse 24. “The meeting room is on the ground floor of the apartment buidling which is across the road from the Stadlau station. The room entrance is on the opposite (eastern) side of the building, between Erich-Fried-Weg and Fahngasse. (map) ”
Contact: moses.skoda[at]gmail.com, or see Facebook group

Washington DC
Time: Saturday, August 25, 7:00
Location: 616 E St NW
Contact: robirahman94[at]gmail.com or see Google group

Wellington, New Zealand
Time: Thursday August 16, 6:00
Location: Enigma, 128 Courtenay Place
Contact: 021 885 180

Zurich, Switzerland
Time: Saturday, August 25, 4:00
Location: The Bäckeranlage Park
Contact: juri2mol[at]gmail.com

Posted in Uncategorized | Tagged , | 96 Comments

Before You Get Too Excited About That Trigger Warning Study…

Posted on August 6, 2018 by Scott Alexander

STUDY: Trigger Warnings Are Harmful To College Students says the Daily Wire, describing a study whose participants’ average age was 37 and which did not measure harm.

You can find the study involved here. A group of Harvard scientists asked 370 people on Mechanical Turk to read some disturbing passages – for example, a graphic murder scene from Crime and Punishment. Half the participants received the following trigger warning before the passage:

TRIGGER WARNING: The passage you are about to read contains disturbing content and may trigger an anxiety response, especially in those who have a history of trauma

Participants were asked to rate their anxiety before and after reading the passages. After they had finished, they were asked to fill out a bunch of questionnaires that measured their opinions about how trauma worked.

The researchers found that people who received the trigger warning were 5% more likely to endorse the idea that they were vulnerable to trauma, and also 5% more likely to endorse the belief that people with trauma could suffer persistent negative effects from that trauma. There were some subgroup and moderation analyses which I ignore for the usual reasons.

What might be some causes for concern with this study?

First, Stuart Ritchie points out that the results are statistically weak. Most of the results have p-values around 0.05, and are not corrected for multiple testing. That means it hasn’t been formally proven whether or not the results are random chance. I don’t like haggling over whether something is just above or just below a significance threshold. But if you do like that kind of haggling, this study doesn’t survive it very well.

Second, the participants’ average age is 37. There’s nothing wrong with 37 year olds, but everyone who says that they’ve proven trigger warnings don’t work on college students either hasn’t read the study, or is adding their editorial opinion.

Third, trigger warnings are aimed at people with trauma. The authors excluded people with trauma from the study. The study was not an attempt to test whether trigger warnings could ever be beneficial, nor to quantify the net (benefit minus harm). It just tested whether there might be nonzero harm. Even if trigger warnings harm healthy people, they might still help the people with trauma who they are intended to help.

But a much more fundamental concern is that this isn’t measuring anything we care about.

Some people read a trigger warning saying disturbing passages could cause emotional harm. Then they read a disturbing passage. Then, on a test, they were slightly more likely to agree with the statement that disturbing passages could cause emotional harm. Of note, they did not claim that they themselves had been harmed or triggered by the passage. In fact they specifically denied this; there was no difference in anxiety between the two groups after reading the passage. They just agreed, in a theoretical sense, that trauma was harmful.

The most boring possible explanation is that if you gave someone a passage saying “TRIGGER WARNING: SOME SNAKES BEAR LIVE YOUNG”, then made them read a book about torture-murder, then asked them whether some snakes could bear live young, they would be more likely to answer yes. Reading the statement about snakes doesn’t cause the snakes to bear young. It just convinces the reader of it.

In the same way, the trigger warning tells respondents that the passage can cause anxiety, and links this to a history of trauma. It doesn’t say outright that passage –> anxiety –> permanent trauma, but it kind of implies it. It certainly primes any pre-existing beliefs along those lines that participants might have heard. And it potentially induces demand effects – which Mechanical Turk is infamous for – by letting you know what the experimenters are thinking.

For this to result in actual harm, two extra things would have to happen.

First, an effect that happens a split second after you read a trigger warning stating something would have to stick around to become a permanent part of the psyche. This is definitely not how these things work. For example, in 2016 I made a group of people read either an essay warning of the dangers of artificial intelligence, or a control essay that didn’t mention the topic. Then I surveyed people on how dangerous they thought AI was. Unsurprisingly, the people who read the control essay were less worried. The effect stuck around after a month on another survey that explicitly reminded them of the essay they’d read, but when I covertly surveyed them again two years later, there was no difference in their beliefs. They’d forgotten the whole thing. If you think about it, both the experimental group and the control group in this new experiment must have seen lots of trigger warnings throughout their lives. The effects of all of them paled in comparison to having seen a trigger warning a minute earlier in the same experiment. Hours or days later, the effect of this one will fade away too.

(This is the kind of measurement I’ve condemned as streetlight psychology, after the story of the drunk who searched for his keys under a streetlight – not because he’d lost them there but because that was where it was easy to look. It’s been used before to study eg video game violence. If you make someone play a violent video game and then test them for violentness a few minutes later while the adrenaline is still in their system, they’ll be more violent. But this is little different from showing that people are sad minutes after they watch a sad movie. If you’re asking whether years of playing a video game will cause violent criminality (or years of watching sad movies will cause clinical depression), that’s a very different question. The trigger warning study has the same problem.)

But even if reading a trigger warning has lasting effects on your beliefs about trauma, we still have to prove that those beliefs cause you to be more easily traumatized. One of the effects in the study was people saying they believed they were more easily traumatized – but how much do we trust that belief? If I made people read a passage saying that cancer was very common and even healthy people often got it, I could certainly make them express a belief that they were more worried about and susceptible to cancer. But would that belief cause cancer? I realize some psychological constructs do work like this, where believing that you’re vulnerable makes you more vulnerable. But a lot of psychological constructs also don’t work like this; it can’t be voodoo all the way down.

So, in conclusion: this study tests an accommodation meant to prevent trauma in college students on a population of non-traumatized 30 and 40-somethings. It finds it does not increase anxiety, but may change beliefs about trauma in the very short term, although those changes do not meet strict standards of statistical significance. It is currently unclear whether those changed beliefs last longer than a few minutes, or whether they have any effect on anything.

I think about this Steven Kaas tweet a lot:

A lot of people care a lot about whether trigger warnings are helpful or harmful in college courses. This study answers (or “answers”) the superficially similar-sounding question of whether they affect certain beliefs about trauma in the very short term. I won’t say it has literally zero bearing on the original question, but if you treat it as having any specific positive amount of bearing, you’re probably wrong.

I don’t want to blame the researchers, who are experts in the psychology of trauma and doing important work. This isn’t an awful study from the perspective of psychologists trying to start a research program that might be suggestive of something after a decade or two. It is an awful study in the context of anything that anybody might report upon or use to form an opinion. Blaming the reporting would be better, but the right-wing sources that exaggerated this are still light-years behind the left in their ability to falsify and weaponize study results and I would feel bad saying anything that seemed to single them out as particularly culpable. Let’s just agree everything is terrible all the time.

As for trigger warnings themselves, I’ll repeat my own proposal: have them, but put them in the Boring Legalese Page of the book, the one where they list the ISBN number and the city where the publishing company has its headquarters and something something Library Of Congress. Make them matter-of-fact, like “Content: rape, murder, ethnic slurs”. You don’t need to embellish with “AND THEREFORE YOU SHOULD BE ANXIOUS” or “SO PEOPLE WITH A TRAUMA HISTORY SHOULD BEWARE” the way this experiment’s warning did. Just put a list of things people might want to know about on the Boring Legalese Page page, and let the couple percent of people with a trauma history check it before reading if they want. If it’s a worksheet or syllabus, put it in small print somewhere consistent. If your opposition to (or support for) trigger warnings is any more impassioned than that, I am suspicious of your motives and think maybe you should sit this one out.

Posted in Uncategorized | Tagged , , | 242 Comments

Cancer Progress: Much More Than You Wanted To Know

Posted on August 1, 2018 by Scott Alexander

Official statistics say we are winning the War on Cancer. Cancer incidence rates, mortality rates, and five-year-survival rates have generally been moving in the right direction over the past few decades.

More skeptical people offer an alternate narrative. Cancer incidence and mortality rates are increasing for some cancers. They are decreasing for others, but the credit goes to social factors like smoking cessation and not to medical advances. Survival rates are increasing only because cancers are getting detected earlier. Suppose a certain cancer is untreatable and will kill you in ten years. If it’s always discovered after seven years, five-year-survival-rate will be 0%. If it’s always discovered after two years, five-year-survival-rate will be 100%. Better screening can shift the percent of cases discovered after seven years vs. two years, and so shift the five-year-survival rate, but the same number of people will be dying of cancer as ever.

This post tries to figure out which narrative is more accurate.

First, incidence of cancer:

This chart doesn’t look good (in both senses of a chart not looking good – seriously, put some pride into your work). Although there’s a positive trend since 2001, it’s overwhelmed by a general worsening since 1975. But this isn’t the right way to look at things: average age has increased since 1975. Since older people are at higher risk of cancer, an older population will look like higher cancer rates. Also, something has to kill you, so if other issues like violent crime or heart disease get better, it will look like a higher cancer rate.

Here’s a better graph:

This is adjusted for age. I’ve switched from incidence rates to death rates, which is bad, but I can’t find good age-adjusted incidence data. Also, notice that this graph truncates its y-axis differently than the other. Still, it shows a similar pattern of adjusted death rates getting worse until 1990 and better thereafter. Why?

Smoking! That graph is just this one plus a 20-to-30-year delay:

Through the first half of the twentieth century, improved tobacco-making technology, increased wealth, and better advertising caused order-of-magnitude increases in smoking. It takes on average a few decades for smoking to cause lung cancer, so there’s a peak in cancer (overwhelmingly driven by lung cancer) with a few-decade delay from the smoking graph. As smoking started to decline, so did lung cancer.

What about the other striking increase on the incidence graph, that of prostate cancer? In the late 1980s, guideline-making bodies suggested that doctors test harder for prostate cancers; doctors followed the recommendation, detected every little tiny irrelevant prostate tumor, and treated patients aggressively for cancers that never would have affected them before they died of something else. In the late 1990s, guideline-making bodies admitted this had been a bad idea, made the opposite recommendation, and people stopped diagnosing prostate cancer as often. If you look at incidence rates, that spike is much bigger. I’m not sure why this shows up on death rates, but perhaps the treatment itself contributed to mortality, or perhaps coroners were biased to attribute a death to prostate cancer if they knew the cancer was present.

Meanwhile, stomach cancer has declined dramatically; different sources attribute this to improved treatment for the cancer-causing stomach bacterium h. pylori, improved food processing methods, and increased vitamin C. Colon cancer is decreasing because colonoscopies remove more pre-cancerous polyps. Liver cancer increased because of a hepatitis C epidemic. A few other cancers are increasing or declining for similarly diverse reasons.

But overall cancer incidence and death rates increased up to 1990 and have declined thereafter. Pretty much everyone attributes the bulk of the decreasing death rate to improved prevention. If improved cancer treatment is contributing, it’s swamped by the social factors and we can’t see it in these data.

The most common method for measuring the effect of improved cancer treatment is the five-year survival rate – what percent of people survive five years after being diagnosed with cancer? Here are the relevant data (source):

This is the best graph I can find, but it unfortunately leaves out breast cancer, colon cancer, and several other major cancers where we’ve made important advances. It’s from 2008, but the trends shown have continued since then. Note that change in the “All Cancers” category also reflects changing distribution of sites.

That looks like progress. But this is where the early diagnosis concerns come in. They’re best expressed by Welch, Schwartz, and Woloshin, who find that among different types of cancer, secular decreases in five-year-survival-rate are not correlated at all with improvement in the cancer death rate, but they are very correlated with change in the incidence rate. In other words, why are people living longer after being diagnosed with cancer? It can’t be because we’re treating the cancer successfully – if it were, they would be linked to decreases in the number of people dying of cancer. But it must be because we’re detecting more cases of small cancers too minor or slow-growing to kill people quickly (“lead-time bias” and “length bias”), which shows up as increases in the cancer detection rate.

This study does not prove that cancer treatment is not improving. It just shows that five-year-survival-rates do not in and of themselves provide evidence for improving cancer treatment. Any signal from improving cancer treatment is drowned out by the signal from improved detection.

How do we get around this? One possibility is to investigate change in stage-specific survival rates. That is, doctors classify cancers by stage, all the way from very early poorly-developed cancers with good prognosis to very advanced cancers with bad prognosis. A lead-time bias or length bias would show up as cancers being detected at an earlier stage. So if we found that more people were surviving even within each bin of “stage at which the cancer was detected”, this would be strong evidence that cancer treatment really is getting better.

Several groups have looked into this. The best data comes from the government’s national cancer statistics clearinghouse at SEER (source):

Even within each stage, five-year-survival-rate has increased significantly from 1975 to 2012.

Closer investigations of specific cancers are similar. Stage-adjusted cervical cancer risk and colon cancer risk both show most of the modern gains in survival rate persisting.

But maybe stages are too big a bin to serve as a useful proxy. Imagine a study that wanted to prove that having more cars made you happier. They do a survey and find that people with more cars are happier, but someone objects that maybe wealthy people have more cars and wealth makes you happier. Imagine that their response is to separate people into two bins: “poor people” who make below $50K and “rich people” who make more. They find that even within each bin, cars still make you happier. But this is just a problem of too few bins: a person making $10K is still very different from a person making $40K (and likely to have fewer cars). The attempt to remove confounding with bins fails. These cancer studies generally use only a few broad stages; might this be allowing effects from early diagnosis to creep back in?

Elkin, Hudis, Begg & Schrag look into this. They find that within each stage, tumors have gotten smaller since 1975, suggesting that the staging system isn’t capturing everything we care about regarding cancer. But they find that even when adjusted for size, some of the stage-specific modern gains in cancer survival still remain. In particular, decreasing size explains 61% of improved survival in localized-stage breast cancer, and 28% of improved survival in regional-stage breast cancer. Another study on breast cancer does a similar adjustment with other ways of classifying cancer and concludes that “improvements were shown irrespective of tumor size, lymph node status, and ER status” and “the impact of screening was by nature of limited magnitude. The modified treatment strategies implemented by the use of nationwide guidelines seemed to have a major impact on the substantial survival improvements.” Another group does a simulation and finds that it’s implausible that screening-related biases are the entire source of improved survival:

The results from our study suggest that lead-time bias introduced by mammography screening does not explain the survival improvement observed during the recent decades in the Nordic countries. The absolute as well as relative bias was generally small, and much smaller than the observed increase in relative survival between 1964-2003. However, in some settings the absolute bias reached 4.0-5.7 percentage points, on a survival around 68-77%, a difference that many would see as an interesting improvement in survival.

A lot of this work has been done in breast cancer, probably because it’s had a strong push for screening recently. We would expect screening to be even less important in other cancers, but there hasn’t been as much work on it. One exception is Tong et al, who find that changes in tumor stage and size explain only 20% of improved survival rates in colon cancer, but advancements in therapy explain about 71%. Separately, an authoritative-sounding collection of colon cancer experts express their opinion that “it is possible that within-stage migration had some effect on our findings, but it is implausible as the major source of the trends we observe.”

The only contrary data point I can find is this study of laryngeal cancer, which finds worsening stage-specific survival rates for high-stage laryngeal cancer since 1977. However, the study authors note this was the only one of 24 cancer types examined to show decreasing survival rates. They speculate that maybe some kind of change in smoking behavior over this period has changed the nature of laryngeal carcinomas to favor a more aggressive type. They don’t really have any evidence for this, but given that this is the only one of 24 cancer types to show a decrease in survival rate, it’s probably something at least that unique, and doesn’t indicate a general failure in cancer treatment.

There could still be unobserved confounders. Stage alone wasn’t enough, but merely adding size to stage might still not be enough. Even the papers that look at a few more esoteric things like receptor status might not be enough. All we can say with certainty is that right now, adjusting for everything we know about and are able to monitor, cancer survival rates still seem to have increased. Tomorrow we might discover new confounders that take that away from us, but right now there is no particular reason to expect that we should.

So: age-adjusted cancer incidence rates and death rates have been going down since 1990, primarily due to better social policies like discouraging smoking. Five-year-survival rates have been gradually improving since at least 1970, on average by maybe about 10% though this depends on severity. Although some of this is confounded by improved screening, this is unlikely to explain more than about 20-50% of the effect. The remainder is probably a real improvement in treatment. Whether or not this level of gradual improvement is enough to represent “winning” the War on Cancer, it at least demonstrates a non-zero amount of progress.

I don’t want to frame this in terms of “here we DEMOLISH the pseudoscientific narrative that cancer progress is weak”. Many of the people I know who critique this research are from an older generation. They remember Nixon assuring them at the very beginning of the War on Cancer that we would have a cure within five years. If they’re really old, maybe they remember victories of that scale over polio and smallpox. If those were their hopes, it’s right for them to feel disappointed. But I come from a generation that doesn’t expect much, and I think the evidence suggests my low expectations have more or less been met.

Posted in Uncategorized | Tagged , , | 180 Comments