Links 1/18: Helink Of Troy

Did you know: the world’s largest Hindu temple is in the city of Robbinsville, New Jersey.

Among Antarctica’s few rivers is the Alph, so named because it runs through caverns down to a sunless sea.

Norway joins Portugal in decriminalizing all recreational drugs. I look forward to years of biased studies exaggerating the effects of this on both sides.

More on monopolies of older orphan medications: the cost of periodic paralysis cure Daranide went from pocket change to $109,500/year.

A really detailed article on how Google is using computer vision technology to provide unprecedented level of detail in Google Maps. Also: automatic identification of urban interestingness, principles of effective cartography, and the nth-level foundational work for self-driving cars.

How would you like to have your legal case judged by a guy called Justice Force Crater? Bonus: he disappeared mysteriously and was never found.

Guillermo del Toro says he saw a real UFO and it was “horribly designed”.

Jacobite: AI Bias Doesn’t Mean What Journalists Say It Means. Lots of people I know have been playing a giant game of chicken hoping someone else would write this article first so they didn’t have to – and it looks like Chris Stucchio and Lisa Mahapatra lost.

To Unlock The Brain’s Mysteries, Puree It. By blending a brain in a way that preserves cell nuclei, you can see how many nuclei there are per cc of brain soup and so get a more accurate count of number of brain cells.

The forgotten history of the African-American community’s pro-eugenics movement.

Alice Maz recounts her days as a Minecraft simulated tycoon. Highly recommended.

Related: Eighty common scams in the Runescape MMORPG economy. Although apparently a lot of Runescape players are clueless twelve-year-olds, and saying “Give me 1000 coins and I’ll give you a cool item” and then not giving them the item makes you some kind of wildly successful criminal mastermind.

AI researcher Rodney Brooks makes specific dated predictions on the future of AI. Too bad he doesn’t give confidence levels and so there won’t be a fair way to judge him.

The brief postal correspondence between Charles Darwin and Karl Marx. Spoiler: Marx was an embarassing-level Darwin fanboy and sent him a copy of Das Kapital along with glowing praise. Darwin politely answered that Marx’s work was “in the long run sure to add to the happiness of Mankind” but never actually read it.

Highlights of Wikipedia’s List Of US Medical Associations include the American Radium Society, Flying Doctors of America, and World Doctors Orchestra. Also, which of these would it make you most nervous to learn your doctor wasn’t in – American Board Of Legal Medicine, Physicians Committee For Responsible Medicine, or Physicians For Life?

Sean Carroll on Twitter: “Does one eventually reach an age where one stops having the anxiety dream about not being allowed to graduate from high school because one signed up for a math class then never attended it?” Reposting here because I’d never heard anyone mention this as a common dream before, but I absolutely have it (though not necessarily math class). Any theories why this happens?

Does a higher minimum wage decrease criminal recidivism? (h/t Marginal Revolution)

The US flag is probably partly based on the flag of the British East India Company, which appealed to the colonists as an model of British colonies enjoying partial independence.

You’ve probably heard that memo writer James Damore has sued Google for discrimination against conservative white men. It seems like a complicated case: political discrimination is generally legal but might not be in California (see here), and discriminating against white men seems hard to distinguish from affirmative action and various societywide diversity campaigns universal enough that I assume someone would have noticed before now if they were illegal. Some people are suggesting it’s more of a publicity stunt (possibly externally funded like Peter Thiel’s support of Hulk Hogan?) to embarrass Google and raise awareness. Which it’s doing – read the the whole 161 page lawsuit here if you want a look at the salacious accusations, which one commenter summarized as “industry-wide blacklists, cash bonuses for condemning wrongthink, calls to summarily fire white men accused of bad behavior, calls for ‘unfair’ (the exact word used) treatment of white men, openly booing the presence of white men, employees using company mailing lists to plot violent antifa actions” etc. Related: this profile of Damore’s (female, Indian-American) attorney. Also: apparently Mencius Moldbug having lunch with a Google employee “triggered a silent alarm, alerting security personnel to escort him off the premises”. Also: a commenter suggests an inside story in which the Damore memo was allowed to blow up because of office politics among top Google leadership.

Bloomberg: The Most Awful Transit Center In America Could Get Unimaginably Worse. They’re talking about Penn Station, a “debacle” which “embodies a particular kind of American failure”. And the unimaginable worseness is that the tunnels under the Hudson are decaying and might need to be closed, which would throw New York’s transportation grid into chaos.

Also Bloomberg: Study: Lobbying Doesn’t Help Companies Or Their Shareholders. This is well within a large body of work finding that money doesn’t really matter in politics, but if true it means both that popular wisdom is so wrong we should be thrown into near-Cartesian doubt about everything, and that corporations are idiots and throw away money for no reason. I put this alongside the “medical care doesn’t improve health outcomes” papers in “well, either this is false or everything else is”. Still give it a 50-50 chance of being true, though.

For the past forty years, every time new works were about to come into the public domain, Congress altered extended copyright law to keep them private. Now with the growing power of open access movements, there doesn’t seem to be any political will for this and works from 1923 will finally enter the public domain next year, followed by one year’s worth of works every year thereafter. Biggest potential loser is Disney, since Mickey Mouse would become public domain in 2024. H/T MR.

Van Bavel, Feldman-Hall & Mende-Siedlecki’s paper on ethics includes (first paragraph) a description of how a real-life trolley problem happened in Los Angeles in 2003 – transportation officials chose to switch tracks, saving dozens of lives. H/T Siberian Fox.

A Medium article by some Google AI scientists and a professor critiques that “AI can analyze your face to tell if you’re gay or not” paper from a few months ago. They find that the AI was most likely just looking at a couple of non-physiological features – glasses, makeup, facial hair style, tanning. Then they show you can get pretty accurate gay/straight classifications just by doing this manually to an image database. [This link previously was more condemnatory, but commenters have pointed out the original study specifically admitted this might be true]

Cambridge University’s Center for the Study of Existential Risk has created a Civilization 5 mod that adds superintelligent AI dynamics to the endgame.

The newest addition to GiveWell’s top charities is No Lean Season, which gives poor rural Bangladeshis enough money to pay for a bus ticket to work in the city during the season without as much farm work.

Primatologists: Bonobos prefer jerks.

NYT: As Labor Pool Shrinks, Prison Time Is Less Of A Hiring Hurdle. Included as a sort of nod to market optimism, where if there’s enough demand all of these problems like discrimination against former convicts will solve themselves. Of course, it would be nice if we didn’t have to wait for economic booms.

In the early 1900s, Sears sold over 70,000 build your own house kits, including some for really impressive mansions.

New research suggests the Sicilian Mafia originated from a sudden increase in the demand for lemons after they were found to cure scurvy; the resulting need to control lemon theft created an entire new dimension of underworld economics. Insert your own “market for lemons” or “lemon-stealing whore” joke here.

Chris Dillow of Stumbling And Mumbling joins the Hereditarian Left.

New entry to the linking perception and cognition research program: acuity of color vision correlates highly with IQ. And of course Michael Woodley is using this to try to demonstrate dysgenic effects.

New study finds more evidence that small class size improves test scores. Especially interesting: in addition to improving them the normal way by students doing better, it improves them because when public school class sizes are smaller, high-SES parents are more likely to send their kids to public schools. But there’s an effect even beyond this.

Larry Sharpe is the new (black, raised in poverty) face of the Libertarian Party. Key insane quote: “In the short run, Republicans will break, we’ll get a bunch. In the long run, we’ll absorb the Democratic Party. The Democratic Party will go away in 30 years. It won’t exist. It’ll be the Libertarian Party.”

This entry was posted in Uncategorized and tagged . Bookmark the permalink.

825 Responses to Links 1/18: Helink Of Troy

  1. Michael Climek says:

    Yes – the High School can’t graduate Dream is very common, though I too didn’t know it was common (I thought it was just me!) until XKCD did a comic about it (which should have been enough of a tip off), I shared it, and most everyone I knew said they have it too.

    the key is Anxiety right? I haven’t had it in a while, haven’t been anxious about anything in a while. I’m settled. Has anyone tracked basic ‘mood’ states that trigger it?

    • Alkatyn says:

      Oddly enough I’m 27 and started having a very similar dream in the past few months, after as far as I’m aware never having had it before. It often comes with a theme of me not being able to find the right classroom or not having the schedule for the classes. And/or me being my current age but having to be back in school for a technicality.

      Even more oddly, in my school system there was no requirement to pass all your classes to graduate.

      • gbdub says:

        I have a similar dream, and it’s always going back to high school to retake a class. Or not making it to class,or realizing that I have totally forgotten to attend a class for a semester. I graduated almost 14 years ago.

        • Michael Climek says:

          Yes this. The dream is always like – You should up to a class on the last day of class, and You had no idea you were even supposed to be attending, and now you have to pass the final – or you’ll not graduate, or something else terrible will happen.

          • Evan Þ says:

            I never had this dream till I graduated college, but since then I’ve had it several times.

          • Murphy says:

            Since it’s good to also include dis-confirmations: I’ve never had that dream though I did once experience something similar in real life getting an email to let me know I got a zero on a class I never attended. Thankfully it was a records error and the same professor taught the other class I had actually been attending and it was cleared up in a few emails.

            If I were to guess I’d bet it’s because class schedules are something I never really got anxious about.

          • Desertopa says:

            I don’t remember a single specific dream about this, and yet…

            I feel that I must have had dreams about this, and rather vivid ones, because when you describe that dream, my immediate reaction is “hasn’t that actually happened to me at least a few times? That was awful.” But I’m pretty sure that it never actually happened.

            I’ve had situations where there was some important event or assignment I didn’t take notice of until it was late enough to constitute a crisis. But I also have a pretty vivid emotional recollection of there being times when I realized that there were classes I was signed up for where it was now months into the semester and I hadn’t attended a single class or done a single assignment, and it didn’t matter whether I passed a final or anything else because it was already impossible for me to pass or drop the class. But, I’m almost certain this never actually happened in reality.

          • AnonYEmous says:

            I’ve had dreams like this, but like…I destroyed what I needed to pass, or I failed to study, or…something. It’s the same general framework people describe, but never the specifics.

        • CatCube says:

          I never actually had a dream about not attending class, but about once per year, near the middle of a semester, I’d be sitting in my dorm room watching TV or surfing the internet and suddenly have a feeling that there was a class that I forgot I was enrolled in and hadn’t been attending. I’d log in to the scheduling website for my college and confirm that no, I’ve not forgotten about any of my classes.

      • JulieK says:

        I also have some other recurring anxiety dreams- that I have to catch a plane and haven’t packed, or it’s almost Passover and I haven’t started preparing.

      • Randy M says:

        It often comes with a theme of me not being able to find the right classroom

        I’ve had this dream but that’s probably not a representative example, since I actually had this happen to me on the first day of sixth grade and was still wandering around nearly crying looking for the room ten minutes into the period.

    • Alkatyn says:

      For me it’s definitely anxiety provoked. I tend to get them when for some reason i am having more physiological anxiety than normal, but not any particular real world thing I’m anxious about (e.g. if i haven’t exercised for a while, issues with medication). So possibly what’s happening is that the brain is generating scenarios to ‘explain’ the anxiety the body is feeling.

    • BBA says:

      I have the dream often, but in my case the fact that I left high school a credit short of graduating (to attend college, which I graduated from in the normal length of time) is a complicating factor. Sometimes it’s a different stage in my long, convoluted education. Oddly it’s almost never the “final exam, didn’t study” variant that people bring up the most, it’s usually that I’m signed up for a class I’ve never attended and I can’t find the classroom.

      • anonymousskimmer says:

        but in my case the fact that I left high school a credit short of graduating (to attend college, which I graduated from in the normal length of time)

        I wish I had done this. I wish I knew it was even a possibility. So much fucking time wasted trying to meet HS credit requirements when I had the actual option to take two years of community college in lieu of the last two years of HS. I’m pissed off that my HS didn’t even mention this as a possibility.

        • mupetblast says:

          I went to college late, at 24. First community then transferred to a 4-year (which I was at for only 2; never took the SAT and all that jazz). Graduated at 30 from San Jose State U. only to find I was missing one PE credit. Ended up taking the only thing available the following summer (2009): a basketball training camp at an Oakland junior college for the school’s team.

          I was the only white guy in the class and ten years older than everyone else. When the coach found out why I was weirdly in attendance, he told me to just run laps around the gym for the hour and I’m good. He didn’t want me getting in the way of his team anymore than I did, I suppose.

          When I think of the above, I’m glad I wasn’t born in Europe even as I acknowledge they do a lot better than the US. “Reinventing” yourself or blooming late doesn’t come as easy over there, or so I hear. They track you early on into a certain strata of the adult working world.

          • Lambert says:

            As a Brit, I’m just glad that PE has never reckoned into a supposedly serious measurement of my academic performance.

    • Bobby Shaftoe says:

      Can we think of this in terms of the kind of “control theory” stuff Scott was posting about a little while ago?

      In order to successfully make it to class all the time in real life, there had to be a “check for class attendance obligations” routine that ran constantly in the background for years and years. It is counterbalanced by an “actually attended class” feeling that usually keeps it in check.

      But “check for class attendance obligations” gets so embedded that even years later it pops up from time to time, and causes panic because there have been no recent “attended class” signals at all!

      • Brandon Berg says:

        Data point consistent with this hypothesis: After more than fifteen years in the labor force, I’ve never had this dream about work except during an lengthy spell of unemployment several years ago. We need to find some retirees.

        • I’m now retired. I don’t think I have had the “I haven’t been showing up for the class I’m supposed to be teaching” dream since, but just last night I had a dream where I was visiting with friends and suddenly realized that it was evening already of the day I was supposed to be flying home and I was still there, hadn’t even packed.

          Then I woke up and realized I was home.

          But of course, I still do things for which that anxiety might be appropriate, take trips which require me to fly somewhere at a particular time.

          Of possible relevance … . Quite recently, I got a phone call from someone asking why my wife and I were not at a dinner engagement an hour or so away. It eventually turned out that the caller had forgotten to tell me that the date for a dinner we had discussed had become definite–or exactly what time and where. But until I discovered that I was seriously upset, feeling depressed for having made just the same sort of mistake I sometimes have anxiety dreams about. Which suggests that it is something with serious emotional weight for me.

          • Brandon Berg says:

            You said in a recent blog post that you retired at the end of last year, so it’s only been a month or two, right? That’s probably not enough time to trigger the mechanism Bobby suggested. It’s been a long time, but I don’t recall ever having had this dream while actually in school. Can any current students comment on this?

          • JulieK says:

            Do you have any idea if people who were homeschooled have these dreams?

          • @Brandon:

            I retired at the end of the 2016-17 academic year, so my last class was about eight months ago. I don’t remember having any of the relevant dreams since, but I might not–dreams vanish pretty fast.

      • cuke says:

        I’m in my mid-50s and still have this dream, even though I’ve been working for a long time in pretty engrossing (and sometimes stressful) careers. I rarely have workplace-based stress dreams.

        My husband, also in his mid-50s, never has this dream, but often has work-based stress dreams having to do with organizing big meetings and forgetting one key thing or having some insurmountable last-minute obstacle arise.

        I think my identity was more attached to being a good student than my husband’s was and his identity is more attached to being a good worker than mine is. I was also a student for a lot more years than he was, so I’m inclined to support your theory.

        I also have a recurring one about being expected to do some kind of big on-stage performance without having the script, etc — though I have no history as a performer.

        And these days I’m having a recurring dream about needing to find housing in a city I used to live in and for some reason need to move back to. I’m wondering what it’s about since I’m happy where I live.

    • nelshoy says:

      I have this dream too.

      A theory: maybe we have a lot of anxiety-droven dreams but these are the ones that are specific enough and with enough real world details that we remember them upon waking.

      At the risk of sounding like a psychopath: Does anyone have dreams where they start killing people and other people who could be witnesses a la one of the new Black Mirror Episodes?

      • WashedOut says:

        Yes. I do not have a violent or aggressive temperament but roughly 50% of my dreams are violent, and a lot of those are as you describe.

        And holy moly, Crocodile was a bad episode. Wonderfully shot, but plot-wise he completely phoned it in.

      • howardtreesong says:

        At the risk of sounding like a psychopath, I was once holding my then-infant daughter while standing on a balcony, several feet from the edge. I had a momentary impulse to hurl her from the balcony. I of course ignored the impulse and went inside. There was zero chance I’d ever have acted on it, but it was a very definite impulse all the same. She’s now seventeen. Also: she wasn’t screaming or crying or anything, nor was I frustrated in any sense by her existence or anything else at that moment. It was a perfectly happy day.

        It was quite strange: I’ve never felt anything like it, before or after. It very definitely made me feel like a psychopath for a few days, and if it had ever repeated itself I likely would have sought therapy.

        • anonymousskimmer says:

          Some people with OCD have these kinds of intrusive thoughts all the time.

          • howardtreesong says:

            Thanks for that link; it has some reasonably specific descriptions of OCD, which until now I had only thought of in very general laymen’s terms. I don’t have either obsessions or compulsions that seem like they’re out of norm. In fact, I can only think of one other time that I had an obsession, and that was when I came down with a high fever while on a work trip. I stayed in my hotel room and hallucinated that I was a sphere for about thirty-six hours. For most of that, I was caught in a thought loop that went kind of like this: “I’m a sphere. WTF. No, I’m not a sphere. I’m me, but I have a high fever and this thought will go away when the fever breaks. No, I’m a sphere . . . ” Rinse and repeat perhaps a thousand times.

            Very definitely one of the worst days in my life.

          • Loquat says:

            I don’t have OCD, but I still get thoughts like that from time to time. Probably the most disturbing one was the time several years ago when I was sorely tempted to kick an co-worker’s baby – several of us were standing around the office chatting, she’d brought in her baby and it was sitting on the floor near my feet, and the thought of punting it across the room just would not leave my head. I think I wound up leaning on a pillar and crossing my feet to stifle the impulse.

        • Protagoras says:

          I believe a decent number of people feel similiarly strange urges to jump themselves in such circumstances. Possibly this is related.

          • Aapje says:

            Perhaps it is fear/worry resulting in vividly visualizing such an event, which the mind then confuses with visualizing a course of action.

          • add_lhr says:

            The French have a very evocative name for that – “l’appel du vide”. It seems nearly universal among my friends and after googling, quite common among the general population as well. I have experienced it most commonly when being around a police officer and feeling a sudden strange urge to try to grab his gun.

          • Aapje says:


            Do you have high fear of guns?

            My fear of heights seems to trigger it.

          • lvlln says:

            I too have often felt the urge to grab a policeman’s gun if one happens to be nearby. I’ll often stick my hands in my pockets and consciously focus on verifying that I can feel the inside of my pockets with my hands until the policeman is out of sight, in order to make sure that I can’t perform the action.

            I grew up in a blue tribe environment where it was pretty much taken as granted that guns were evil scary things that should be avoided at all costs. I have gone to a couple shooting ranges in my adulthood and enjoyed the experiences. I don’t know if this history plays into that at all.

            I also have a fear of heights and do feel that urge to throw myself off a cliff/balcony/bridge if I happen to be on one and near the edge. I also commonly feel the urge to throw myself onto the subway tracks when waiting for a train (which happens a couple times every workday), and to throw myself into the road in front of a moving car when I’m waiting to cross it. I usually grab onto a railing or a signpost and focus on keeping contact with that thing, as I rationally know that it’s impossible for my body to be simultaneously grabbing it and in the middle of the road or falling off a bridge or whatever at the same time.

            I’ve never felt the urge so strongly that I’ve ever taken even a single step in the direction of actually performing it, which seems contradictory with it being an urge. But I still experience the feeling as an urge. At the least, knowing that I’ve never actually acted on the urges and having come up with fairly unobtrusive coping mechanisms, I don’t feel too much negative emotion from the fear that I’ll act on those urges.

          • Randy M says:

            Scott had some posts here describing a neuroscience theory that, paraphrased, anticipation is very similar to desire. It seems plausible that a fear, vividly imagined, could be experienced as an urge to do it. Although this seems like a pretty kludgy adaptation, give the usefulness of a clear and obvious feeling of fear.

          • moonfirestorm says:

            Arachnophobia is fairly common. Has anyone with it ever experienced a desire to reach out, grab a nearby spider, and put it on their face?

            It seems like some fears could work like this, but I’m not convinced it’s generalizable to all fears. Which raises the question, what separates fear of heights from fear of spiders?

          • Aapje says:

            I think that with regard to spiders, the fear is typically that the spider does something to you, not that you do something to the spider.

            I think that the fear has to be about something that you do (by mistake), not something done to you, for it to result in an urge.

            For example, the police gun example also involves the possibility of doing the wrong thing to provoke the officer into violence against you.

        • Rusty says:

          Not one for everyone I suspect but I enjoyed Piercing a lot. Anyway, the protagonist of the novel had much the same problem as you. I wonder if you used his technique at all? Let’s hope not.

      • I’ve never seen Black Mirror and I don’t think I have ever had dreams where I “start killing people and other people who could be witnesses.” But I have dreams where I have committed a murder–I’m not sure whether any which actually include the murder, or only knowing afterwards that I have committed one. I don’t remember the details of any of those at all clearly, but I don’t think they were connected to any real world desires to kill anyone in particular.

      • Jade Nekotenshi says:

        I’ve never specifically had a dream where I committed any kind of non-defensive act of violence (outside of war dreams which are pretty rare but have happened), but just recently I had a dream where I was in some kind of very unconventional minimum-security prison. They wanted to move me to medium security, and I was trying to bargain with the guards to do any of a number of horrible things to me instead of that – I specifically remember saying “for the love of god, not that – chop my feet off with a band saw, burn my eyes, but don’t do that!”. I was scared of someone, but I don’t know who or why, or what I was in prison for, other than being absolutely sure that I deserved it, and deserved far worse treatment than what they were actually doing to me.

        Left me unsettled and nerve-jangly for the rest of the week, really.

    • Steve Sailer says:

      In college, I stopped going to my 8am Business Law II class because the lecturer was incoherent. After a month or so of not going, I happened to wake up early one morning, so I dropped in. They were handing out pieces of paper. “Wow, this lecturer has decided to print up outlines for his lectures. That’s a big improvement,” I thought.

      But it turned out to be a test I hadn’t studied for, just like in the classic nightmare.

      I still got a 91 on the test (although the average score was 93).

      It was a bad class.

      • themightypuck says:

        I skipped the past 2 months of a class and then showed up for the scheduled exam and no one was there. The prof had used the last two classes for the final.

    • Lillian says:

      Not only do i not have that dream, but i don’t have any of the other common dreams people talk about, nor any recurring dreams of any sort whatsoever. Sure there are thematic similarities between my dreams, but it’s very rarely the same one twice, and never thrice. Though, when i was a child i did have that dream where you’re suddenly naked in public, but even then the public would be a surreal dreamscape, and it never ended with me being embarrassed by everyone else. Instead either nobody cared, or i hid before anyone noticed, or i would suddenly realize i could fly and nothing else mattered.

      That’s not to say my mind doesn’t troll me when i’m sleeping. Just not with the dreams other people talk about. One of its best efforts thus far involved coming up with an entire fantasy setting with its own magic system in which i was an apprentice to a travelling judge-wizard. All so it could have an interesting setting for me to be tremble in terror as the tyrannical autocratic king announces that he will take me as his concubine. Fortunately, they thought i was going to come quietly so they put me in a luxury cabin instead of a cell with wizard suppression runes, while allowed me to overload the ship’s magical engines and blow it up as a distraction to cover my escape. Unfortunately the man guarding me was supernaturally enhanced and utterly relentless, so even though i blew up a second ship under him, he still managed to recapture me. At least i got to watch him get chewed out for letting me escape, letting me blow up two of their ships, and injuring me in the course of recapturing me. Serves him right, the jerk. Then they did throw me in with the anti-wizard runes, and things got rather unpleasant after that.

      Like i said though, that was one of the best efforts. More typical ones are more like me being a jerk and pissing off all my friends, or whatever. Boring stuff.

    • liskantope says:

      I think the near-universal experience isn’t necessarily dreaming of high school itself but dreaming about having forgotten to prepare for something, the obvious cause being that this is a near-universal anxiety. For instance, I’ve heard a lot of reports of dreams where they’re about to go on stage for some theater performance and realize they’ve forgotten to go to all the rehearsals, from people who have done a lot of theater. (I occasionally have both versions of the dream.)

      • Protagoras says:

        Yeah, I’ve had school dreams about being unprepared/having utterly forgotten to do something critical, but both about being a student and (since I started teaching) about teaching. So if people also have it about theater, do people have this about work in general? Or is it maybe especially likely to be about areas where you get constant scrutiny (from teachers and peers as a student, much more so than for the most part in working life, and theater and teaching are both areas where you have an audience to judge you)?

      • Icedcoffee says:

        Although I haven’t had this happen in a while, I have absolutely had the dream about going on stage without knowing any of my lines. (E.g. I time travel back into my past self, in a role that I actually played, but have now forgotten.)

        • Lillian says:

          Had this happen in real life during my high school drama class final exam, which consisted of putting on a performance at a school theatre. A filled to capacity school theatre since everyone wanted to see the performances, and every teacher was entitled to to cancel one class for that purpose.

          However, i hadn’t forgotten my lines, but rather had simply not prepared any beforehand. While i did have a costume and an idea, i had neither written nor rehearsed anything. Curtain comes up, spotlight comes on, i walk forward, freeze, request a do over. Step back, spotlight off, curtain down. Pause. Curtain up, spotlight on, i walk forward, freeze again, and i sob, “i’ve never done this before.” Entire audience cheers for me. Okay, deep breath, one more time. Curtain up, spotline on, i walk forward and deliver an entire on the spot improvised monologue. Not only do i get a massive cheer, but i also get a coveted and rare A+ grade that entitles me to participate in the “Best of” show in the final week of school. For that one, i did actually prepare my monologue the previous day, and rehearse it the same day. Definitely better than freezing on stage

          Relatedly, this may be why i don’t get anxiety dreams about school. Pretty much all the things, from missing a class, to not doing the homework, to not studying for the test, are things i’ve already done. The world didn’t end. A dream can’t make me anxious by showing me something that i already reacted to with an indifferent shrug.

      • schazjmd says:

        That makes sense to me. I’ve had the forgotten class dream many times (which I always attributed to guilt over all the classes I cut in high school), but I’ve also had frequent dreams where I’m being reassigned and the movers show up and I haven’t packed anything yet. I’ve been out of the military for more than 20 years, and I still get this dream. I never thought to connect the two dreams to a common theme of anxiety over being unprepared for something.

        • B Beck says:

          A couple times I’ve had a dream in which the DoD had screwed up my discharge paperwork and I had to go back in briefly to satisfy some requirement.
          I’ve been out of the military for around 20 years, as well, and I was my present age (40’s) in the dream.

          • Dissonant Cognizance says:

            I used to get this dream a lot shortly after I got out, with the added imagery of my former boat sort of stalking me like a predatory animal. I’d just be going about my normal dream-business, turn a corner, and there’s a Los Angeles class submarine with the chief of the boat ready to tell me my DD-214 is invalid. One time it was parked in my dad’s driveway on a ridiculous giant speedboat trailer, dripping seawater, with the crew running around pissed at having to deal with such an absurd situation.

        • jmiettin says:

          Here in the land of compulsory military service it is very common for male adults to have nightmarish dreams about being back in the military.

          Usually in the dreams the dreamer ends up as a new recruit because of some mistake and needs to go through all the rookie stuff and no one believes him that he’s already been through all that.

          For some reason, I have not had those dreams for years even though I’ve been to multiple refreshment trainings after my discharge to the reserves.

          • Jade Nekotenshi says:

            Good grief, yes. I was in the military for 12 years, been out for 7, and I still occasionally have dreams about being stuck in boot camp. Still had ’em while I was an active-service NCO, even.
            I actually was stuck in boot camp for far longer than usual because I got injured, so this might have some origin in bad memories rather than anxiety as such. Boot camp is, usually, calculated to suck as much as it reasonably can for as long as you’re supposed to be there: if you have to stay much longer, the suck starts to accumulate.

    • Peter says:

      The details seem to vary from person to person, but the basic theme of being in school and about to fail due to not having done work seems to be pretty common. In mine there’s some exam that I haven’t revised for (or attended classes for, generally due to having lost interest and sort-of forgotten I was even meant to be turning up to them) and theoretically my exam performance is all that matters, but how am I to do well in the exam if I haven’t prepared properly? The dream rarely if ever gets as far as the actual exam. There’s a variant where there’s a long project (more than a term long) that I have to hand in (people in the UK system who were doing GCSEs in the mid-90s might remember Technology), and I’d kind of lost interest in going to those classes or doing the project work months before, but the dream never actually reaches handing-in time. Like Alkatyn, sometimes the dreams have me being my current age or a bit younger, and inexplicably back in school (or sometimes at university, sometimes even doing a PhD) .

      I get these things all the time; I have Generalized Anxiety Disorder, and most of my memorable dreams are anxiety dreams, and the remainder are sufficiently surreal that that they might as well be. All these people who talk about how to have a “dream wedding” – I do not want a dream wedding, it would not end well. In my last dream about a wedding the bride wandered off halfway through due to having lost interest.

      In fact that seems to be a common theme, randomly losing interest in something important and going off and doing something else.

    • bean says:

      Never had that one. The closest I’ve ever come is one about forgetting to start college a few months before I went. That was a rather interesting dream, because at one point I paused and started to try to figure out how it had happened. I realized I didn’t remember anything from the past 6 months, and then worked out that it was a dream. At which point my brain, apparently panicked by the thought of being in an illogical universe, punched the eject button, and I woke up.

      • Lillian says:

        This is how every instance of lucid dreaming ends for me. As soon as i realize i’m dreaming, i panic and force myself awake, often by committing dream suicide. Being aware you are in a universe that makes no sense is one of the most viscerally unpleasant feelkngs i know. Which is funny because i love surreal dreamscapes in fiction. It hasn’t happened for a long time though, so i guess my mind built better safeguards.

        • liskantope says:

          Interesting, this is the first report I’ve heard of lucidity being an intensely negative experience rather than a pleasantly thrilling one. Lucid dreaming isn’t for everyone, I guess.

          • Lillian says:

            The interesting thing about it is that i very much would like to lucid dream, for pretty much the same reasons every one else does. What’s more i am fascinated by surreal dreamscapes, and love tales of dreams and dream worlds. In all respects i should seem like the sort of person who would be an avid lucid dreamer. And yet, every time i realize i am dreaming, i am instantly seized upon by an utter blind panic that chases away all thought save that of awakening.

        • bean says:

          Oh, it wasn’t that. I didn’t commit suicide or anything. It’s just that as soon as I realized it was a dream, I was awake. I’ll admit that my assumption it was my subconscious trying to get out is conjecture, but I think it’s the most likely explanation.

        • Edward Scizorhands says:

          When I realize I’m in a dream, which is rare[1], I’ll have to be careful not to break the rules of the universe too much or else I will wake up.

          [1] I will often say “oh, this must be a dream, obviously, because I can fly,” only to have some other part of my brain say “no, that’s always been true before, but now you really can fly; isn’t all that dream experience useful, huh?”

      • Nick says:

        My brain hits the eject button whenever I realize too, but often I just end up in another dream. I had this three or four times in a row one night a few months ago.

    • peacetreefrog says:

      I used to have that dream all the time, but stopped having it once I got into programming. Is it because credentials don’t really matter in coding or I’m generally less anxious now? My guess is the former, but who knows. Also, my grandfather still has this dream sometimes and he’s 92.

    • howardtreesong says:

      I’ve never had the dream with respect to high school, but I did have it with respect to undergraduate school. My institution had a swim test requirement, which involved swimming 100 meters or some other short distance, and also treading water for something like fifteen minutes. I was a perfectly competent swimmer and had no trouble taking and passing the swim test, but I’ve woken up in a cold sweat half a dozen times in thirty years with a dream that I never took the swim test and thus that my undergraduate degree claims are fraudulent.

    • The Nybbler says:

      I’ve had the dream, though it was always college and never a math class (English or History usually). Also it was sometimes a master’s degree, which I don’t have and never tried for. But it DID stop, sometime in my 30s.

    • RDNinja says:

      I actually had a similar problem in high school. I was in the gifted program, and was given special permission to take two years of high-school algebra in middle school, along with my twin brother and our friend. When my junior year rolled around, I found out that they wouldn’t give me high school credit for the classes I took in middle school, so there were now not enough math classes offered to get the math credits required to graduate.

    • anonymousskimmer says:

      Yeah, it undoubtedly had something to do with anxiety. I’d also have a version where I had forgotten to do my newspaper route for a couple of days, or woke up late to do it. This lasted for a few years after no longer doing that job.

      My academic one was college based. The last one was a few years ago, I haven’t had one since. In that last dream I said “screw it, I’ll wing the test. Worse case scenario I have to retake the class.”, and the dream ended with me feeling fine. This basically describes how I actually finished college. My anger, frustration and annoyance at the tertiary educational system, and a willingness to just deal with whatever life threw at me in its regard, finally fully trumped any anxiety I’ve had about it.

    • Icedcoffee says:

      I have the dream too, except it is an undergraduate math course. (I never skipped high school classes. College? Um . . .) Mine normally isn’t graduation focused, though, it’s more of an “oh crap I haven’t been attending and my grade is gonna be so bad.”

      But yes it is always math. And the classroom is normally in a basement.

    • fossilizedtreeresin says:

      Maybe had it once or twice, but my repeated dream of choice is being chased by something, usually zombies, but sometime witches, serial killers, etc. I used to be sure that everybody had those kind of dreams until I shared mine with a friend and he was all “what the hell are you talking about?”.

      Maybe it’s not as common? Did anyone of you have repeated dreams about being chased?

      • Berna says:

        I did, it was my most frequent nightmare. Being in a complicated building where I can’t find the exit, and having to hide/run, because I must not be found in there.

      • A Definite Beta Guy says:

        I used to have those dreams pretty consistently. I decided I would try to stand my ground in such dreams, and…uhhh…weapons don’t work too well in dreams.

        I get zombie dreams pretty frequently. Nothing with werewolves or witches or anything else. Just zombies.

        Learned how to lucid dream just so I could stop getting eaten by zombies.

      • I think I had dreams about being chased or somehow threatened when I was a child, although by now I can’t remember any details.

      • methylethyl says:

        “Being chased by a malevolent entity” was my most common nightmare as a child. Not so much as an adult.

      • Hyzenthlay says:

        I don’t have those dreams often, thankfully, but I’ve definitely had them. The most recent one involved being chased by the clown from IT (the miniseries with Tim Curry, not the new movie) though in real life I don’t find either version scary.

        I had another dream where I was on a street and saw this random guy walking toward me in this strange, inhumanly fast way, and knew he was after me. I ran into the nearest house’s backyard and tried to lose him by running through a series of interconnected backyards (not sure if it worked because I woke up shortly after that).

      • cuke says:

        Yes, that was the main recurring nightmare I had for a decade until my early 20s. They stopped when I managed to confront the person chasing me in the dream and asked him what he was up to. That seems to have taken the wind out of the sails of that dream. Thank god.

      • Tarpitz says:

        Always a (large, dark, wolf/Alsatian-shaped) dog for me. In real life I love dogs and have never been frightened of one, but I was quite badly scared at the age of 8 or so by The Hound of the Baskervilles (possibly a retelling for children – I’m unclear on this point) which I think must be where it comes from. It quite often segues into sleep paralysis, with the sense that the dog is in the room.

        This is just another of the many things that lead me to suspect that JK Rowling and I have similar styles of dreaming.

    • Urstoff says:

      It’s always an undergrad course for me that I enrolled in and then meant to drop but never did. I wonder how much these dreams correlate to performance in school. Do bad students or students who didn’t care about school have these dreams?

      • anonymousskimmer says:

        or students who didn’t care about school

        No, once you stop caring about school the dreams end.

      • A Definite Beta Guy says:

        I did not give a crap about my classes and I still have this dream all the time. Normally it’s high school, not college, and I have to go back to high school at my present age. But I’m not allowed because I’m an old man, so I have to sneak in, and pass my classes, without anyone knowing I am there.

      • Lillian says:

        I didn’t care much about school, and i don’t have any such dreams. As i speculated above, it’s hard to make me anxious about missing a class, not doing homework, or not studying for a test. Those are all things i’ve already done, and it wasn’t the end of the world.

    • secret_tunnel says:

      I have this exact dream all the time, and had a similar dream last night thanks to reading this blog post before bed.

      I don’t suffer from anxiety at all, though.

    • yossarian says:

      My particular anxiety dreams from the last half a year or so that I remember, with approximate number of times happening:
      1) Teeth falling out (2)
      2) Shitting myself or showing up naked to class/work/whatever (3)
      3) Killing or hitting someone who I am REALLY not supposed to be killing or hitting and now I have to kill everyone who was around (2)
      4) Being tasked to program code that analyzes data in a world with two time dimensions and time travel in each one possible (4? seriously?)
      5) Missing a train or plane to somewhere I really have to be (2)
      6) Being kicked off work/school due to wrong sexual behaviour (3)
      Interesting thing is, out of those only two happened IRL and/or could reasonably happen within a reasonable time frame from now – 4, 5 (well, 1 happened back in childhood, obviously, and probably 2, though I don’t remember an instance of me shitting myself)

      • cuke says:

        Yes, me too to many of these themes. And I hear them from lots of clients as well.

        The teeth-falling-out one I have often heard people interpret as anxiety about growing up/older. I have come to associate it with worry over impending change.

        • Desertopa says:

          The “anxiety over growing up/growing older” explanation sounds to me like the sort of explanation people would come up with based on symbolism and what sorts of causes tend to result in tooth loss in real life, but I’m very skeptical that dream associations track the sort of symbolism we use in the waking world that strongly. These dream association correspondences almost all seem to be drawn from people deciding what some psychologists decided were compelling symbolic representations, but not from any sort of rigorous research into what sort of life circumstances people with various dream features tend to be undergoing.

          Personally, I used to have teeth-falling-out dreams fairly often when I was younger, along with more frequent dreams in which my finger joints would bend off the proper directions until my fingers became gnarled and inoperable. In my case, I don’t think the dreams represented anxiety about anything in particular; they would occur regularly when I was under states of high generalized anxiety (most often related to schoolwork,) and rarely occur now that my anxiety levels are lower.

          I wouldn’t be surprised though, if a lot of dream features don’t represent shared symbolism between individuals, and insofar as they indicate something in particular, it’s specific to the person having them, rather than having a consistent meaning across the population.

          • Protagoras says:

            So you’re a Freudian rather than a Jungian when it comes to interpretation (well, Freud would speculate when the person wasn’t there to question, but his recommended method of interpretation of symbols was to ask the patient what they meant to them).

          • Desertopa says:

            I don’t think individuals necessarily know what the symbols mean to them either, and for that matter I’m not even sure “symbolic” is the right term, since I think the connection is likely less representation, more free association. But I do think individuals are probably in a better position to guess what their associations are than an outsider is, because they can spot under what conditions the dream imagery tends to recur.

      • Edward Scizorhands says:

        I very very frequently find myself needing to be somewhere but the mode of transport I need to take (car, subway) simply doesn’t work. Like it always goes the wrong way, won’t start, never moves, something.

        • cuke says:

          Also what’s up with phones in dreams? Mine are always broken and I spend what feels like hours trying to call the same person over and over and can’t get through.

          • Edward Scizorhands says:

            Don’t get me started on trying to read!

          • Randy M says:

            Maybe your unconscious knows how hard it is to do a good horror movie when the protagonist can call for help.

          • Paul Zrimsek says:

            On the other hand, the ability to glide frictionlessly for long distances just above the ground, as I can often do in dreams, would be pretty cool.

          • Randy M says:

            On the other hand, the ability to glide frictionlessly for long distances just above the ground, as I can often do in dreams, would be pretty cool.

            Oh, this one, I think I’ve had this so much normal walking sometimes feels weird.

          • Iain says:

            I have the same frictionless gliding thing. I speculate that it has something to do with swimming.

          • Edward Scizorhands says:

            For some reason it makes perfect sense as me just deciding “oh, gravity isn’t going to work on me any more. I should have thought of this long ago!”

    • dndnrsn says:

      I can’t recall any school-based anxiety dreams. My anxiety dreams are either more “horror scenario” dreams but I wake up extremely anxious, or, surreal anxiety dreams about situations that can’t possibly be real.

    • neciampater says:

      I have this dream. I either didn’t realize a paper was due or I didn’t do my Latin reading.

      Should this appear on the next SSC survey?

    • herculesorion says:

      I sometimes had “skipped class, can’t graduate” dreams. Then I actually did skip one class for most of a semester and I still graduated. (In fact I got a “B” in the class, suggesting that I wasn’t the only one who skipped it.)

      After that I stopped having that dream, because apparently my subconscious was like “oh, really? Well, fuck *this* then, let’s come up with something else…”

    • Claw21 says:

      Yes, I have this kind of dream all the time except it isn’t high school but college and I forgot I have to do one more semester to graduate and I haven’t been attending class. It has been years since I’ve been in any kind of school and I still have this dream.

      I also have dreams about other high-stress, long-term, and life-altering periods of my life such as the 18 month religious mission I did in my early 20s. I wonder if this dream stuff is some kind of lingering, mild form of PTSD.

    • Fossegrimen says:

      I’ve never had any dreams resembling this, but since we’re talking about dreams I recently learned that people don’t have recursive dreams. By recursive I mean that I dream that I dream that I dream X and when I ‘wake up’ either from realising I’m dreaming or from some nightmare, I just pop one level of dream off the stack and keep dreaming. Being forcefully awakened from several levels deep is really disconcerting.

      Anyone else experiencing similar or am I really weird?

      • Winter Shaker says:

        I had that happen once (at least that I can remember). Each time I ‘woke up’, I was dreaming that I was in my actual real-life bed, wanting to reach down to the foot of my bed and turn off the CD player (which, in the real world, I had left on when I went to bed), and finding that pressing stop had no effect, until a good two or three iterations (at least that’s what it felt like), when I did manage to wake up for real and switch the CD player off (which was playing Ween’s Alcan Road, in case you’re curious – kind of a weird and trippy song, especially once the loud, slow guitar comes in).

        That said, it was only recursive while waking up; I don’t think I experienced several layers of going to sleep while already asleep.

      • A Definite Beta Guy says:

        I have recently had dreams like this. I am not a huge fan of the whole “inception” thing. It’s really disorienting to “wake up” from one dream and find yourself in another dream.

        FTR, when I say “recently,” I don’t mean post-Inception (release date 2010). This is more like a 2015-2016 development.

      • cuke says:

        The only time I had this kind of nested dreaming was a brief stint of taking Effexor. I stopped taking it because this dream effect was so disturbing.

      • Tarpitz says:

        Not often, but I’ve had this a few times, yes.

      • jeqofire says:

        I once had a recursive dream where I kept trying to tell people about the first dream, only to realize that I had no explanation for riding a 1993 Isuzu Trooper across Brazil, and was still dreaming. This happened like 3 times before I really woke.
        My first nightmare was a recursive dream, where I thought I woke up, only for the room to start shaking and an enormous bat-owl-thing was hovering above me, flapping its devilish wings like it owned the place. This was also my first sleep paralysis dream, hence why it was so bloody terrifying.

    • Edward Scizorhands says:

      I constantly have a dream about my old paper route. They are so common they have built up their own timeline and remembered event in the dreamline, where I’m certain this is a dream but someone tells me “by the way this isn’t a dream” and I buy that for some reason. I’m still fed up enough that I usually don’t bother delivering the newspapers.

      Also, I haven’t done any collections from customers in over a year in the dreamtime. I guess it’s a charity operation or whatever.

    • flye says:

      That dream is common enough to have been an excellent joke in the 1984 movie Top Secret. Val Kilmer’s character has that anxiety dream, and is extremely relieved when he wakes up to find he’s not still in high school but merely being tortured as a suspected spy.

    • JulieK says:

      XKCD just did another comic on “universal dreams”:

    • kboon says:

      Had one of those dreams just the other day (night).

    • jeqofire says:

      I’ve had some weird back-to-school dreams lately, but not of the anxiety variety, more the “Most of my dreams lately have been set at my elementary school and sometimes they want it to try and make sense”.
      I only really had one dream I’d call a nightmare before I was 26 and living alone, and the latter started after some attempts at wiping out invasive insects left the smell of the associated chemical weapons in the building for longer than expected. And these almost all involved some otherwise normal dream being interrupted by demons (or in one case, Luke Skywalker and Princess Leia watching the debris cloud from the first Death Star and trying to summon demons from it…).
      I didn’t have the “naked in public” dream until I actually started wondering why that was supposed to be a big deal (this did not make it less embarrassing, but I somehow got over it and tried to fight a giant Ivan Ooze. … Being that he was several stories tall and I was 7, I just kinda bounced off and landed something like 100yards away). I only had the teeth thing after I broke a tooth, at which point there were a few dreams about noticing other teeth being broken in the same way without explanation.
      I don’t think I’ve ever had any recurring dreams, although a recurring element has involved getting trapped in elevators that won’t stop, or the doors/threasholds of elevators malfunctioning or having huge gaps between the elevator and the floor.
      There was a period—I forget when, but might be able to check?—when I had dreams about trying to go back to high school, and running into conflict because I only remembered after I got there and tried doing things that I had long-since graduated.
      Mostly, there’s a lot of influence from whatever has had my attention lately, with whatever other references my subconscience comes up with to fill in the gaps. The worst anxiety stuff was mostly occasional bits in later high school of the “people will not stop sexualizing everything and this makes dreams involving people I like but don’t like uncomfortable” variety. I wonder if half the reason I took a level in Lucid Dreaming around that time had to do with escaping those situations. Weirdly enough, the one that avoided going down that path involved someone I actually was attracted to, presumably because there was no “zomg but what if the world is right?” element when the answer was yes.
      From this, I can only conclude that I was more afraid of sexuality than anything else at the time. This took a weird turn recently, when someone I was trying and failing to avoid falling for showed up in a dream, but turned out to be a demon in disguise and tried to eat my face. Then it turned out in reality that she liked me, too, and that was the end of that problem.

    • Jody Lanard says:

      Here is a 2014 Psychology Today informal survey of people’s school dreams. Nearly all the responders rated the dreams unpleasant. According to the authors, “The most common school dream themes are (a) missing classes all term and therefore being likely to fail, and (b) being unable to find the classroom.”

      I have not gone down the rabbit hole to find more academic studies of this topic. As a person perpetually struggling against procrastination, I have had lots of the “missing classes all term” dreams. In real life, I do not have panic attacks. But from these dreams, I think I have an inkling of what they might be like!

      From a retired psychiatrist.

      Jody Lanard MD

    • howardtreesong says:

      I’ve had only one other powerful repeat dream, and that occurred for about a year after my father died in a plane crash. I went to identify his body, which was a mistake: he was recognizable but barely. In the dream, he’s running a chainsaw in the entranceway of our barn. I’m at the other end of the barn and I for some reason know he’s about to run the saw into his leg. I run down the length of the barn, screaming and waving my hands, but he can’t hear me over the noise of the saw and he’s looking down at what he’s cutting. As I run, the saw slips and saws into his leg, and I wake up in a sweat with an elevated heart rate and sweats.

      Thankfully, that stopped after three or four episodes. There is no doubt it was anxiety-related.

    • nestorr says:

      I dropped out of college and thought that dream was due to the unfinished nature of my tertiary education. I am glad to know people who did finish are also having it, this means I don’t have to go to college just to get that monkey off my back.

    • As a nod against selection effects, I will denote that despite being a fairly anxious sort of person, I have never (as far as I can recall) had any dreams about being insufficiently prepared for an important academic event or something like that.

      In general the dreams I remember are madcap romps, collections of disparate images and fancies, though usually with an underlying narrative thread. See for examples. So dreams like people are describing here seem like ones I would be unlikely to remember, because they just make too much sense.

    • Lasagna says:

      I’ve had that dream since high school – I thought everyone did. Turns out I was right, I guess.

      Mine always takes the same form: it turns out that I forgot to go to a class for the entire year, and one of my classmates mentions that the final is today. I spend the dream trying to figure out how to get a day or two to study for it, but I can never track down the right people.

      One interesting thing is the class itself. After high school, the dream was always a high-school science or tech class; after college it was usually a pre-med class. After law school it switched to “Introduction to Intergalactic Law” (not remotely kidding). The high school and college class names changed, but it was always something insanely esoteric, like “Principles of Aviation”, that I would have no shot at faking my way through.

      Law school was almost twenty years ago, and I still regularly have the dream.

    • raj says:

      I commonly have a variation where I have to go back to middle or high-school and take some sort of remedial classes. Emotionally it is a mixture of excitement in getting to go back, and a deep sense of failure/wrongness for needing to

    • Mark Atwood says:

      I have a recurring repeat of the “I didn’t go to a class” dream, for one particular class, that feels so real I’m not sure that it’s not a real memory. Part of the problem is that that particular school year was real life very chaotic, and it’s actually probable that I really did forget to attend that particular class, and I have no good way to tell now if it really had happened or not.

  2. How would you like to have your legal case judged by a guy called Justice Force Crater? Bonus: he disappeared mysteriously and was never found.

    You mean Justice Joseph Force Crater. He was known as Joseph, not as Force.

    His title was Justice, but in pop culture he became “Judge Crater”.

    When I was younger, this case (though dating from 1930) was so still so well-known that it was often mentioned in comedy routines and such.

    • Steve Sailer says:

      Yes, Judge Crater was the paradigmatic case of disappearance in mid-20th Century American culture. Old “New Yorker” comic essays by the likes of James Thurber or Robert Benchley would often mention Judge Crater having disappeared without feeling any need to fill you in on any details about who he was, the way I reference the OJ Simpson case without bothering to explain who OJ was.

    • SteveReilly says:

      Yeah, there’s a Sopranos episode from probably 2002 or so where an older character refers to Judge Crater and a younger character (maybe mid-30s) has no idea what he’s talking about. Neither did I at the time, though I grew up in a suburb of New York.

  3. Paul Crowley says:

    “Political Discrimination

    California law prohibits employers from controlling their employees’ political activities. This means that an employer may not punish an employee for being a member of a specific political party. Nor may employers forbid employees from going to political rallies or becoming candidates for public office.

    Employers are also prohibited from trying to coerce or influence their employees to take any sort of political action. And employers are prohibited from retaliating against employees who oppose such practices.

    Political discrimination can be serious. In some cases, it is criminally punishable as a misdemeanor. There are also fines, fees, and civil damages that can be imposed against the employer (and sometimes recovered by the employee).”

    • gbdub says:

      Affirmative action is one thing, but could Damore, assuming his allegations are true, make a good case for creating a hostile work environment based on a protected class (race and gender)?

      • maxaganar says:

        For anyone looking to hear the other side of this — the argument that Damore does not have a strong case based on being a protected class — two useful references:

        And the first segment of:

      • Matt M says:

        I feel like “discriminating against white people is obviously okay because after all that’s what affirmative action is” is a huge statement. Getting the left to admit that would be a huge win in the culture war for the right. Because I’m pretty sure the official position on AA is that no, it’s not discriminating against white males, it’s just giving minorities “a fair shot.” I think this is true both at the high-level of pop culture (arguing with people on Twitter) as well as at the legal level.

        If they were to concede that yes, affirmative action is the same as discrimination against white males, I think that helps quite a bit.

        • My impression is that affirmative action generally involves biasing the admissions process in favor of a group but not by enough to make the percentage of that group admitted larger than their percentage of the relevant population. It sounded, from the part of Damore’s claims that I read, as though Google was trying to hire more women than men. I’m not sure of the legal implications of the difference.

          • The Nybbler says:

            If we look at certain highly ranked schools (e.g. CMU, MIT, Harvey Mudd), we can see they blatantly bias their admissions process; CMU and Harvey Mudd both boast about having ~50% of their computer science classes being female, but they do it by having an acceptance rate 1.5x (CMU) to 2.5x (Harvey Mudd) times higher for women as for men. No doubt they define the “relevant population” as the population as a whole rather than applicants. This is the sort of thing some people proposed at Google as well. However, I don’t think it’s legal in employment.

          • One Name May Hide Another says:

            I recently read that as far as admissions to highly selective private colleges go, it’s usually females who are discriminated against, and especially Asian females. This will, of course, be different for technical schools. But if you’re a male applying to Yale/Harvard/Princeton, etc, and you have a great GPA and fantastic test scores, you’ll have an easier time getting admitted than a female with equivalent grades/scores. This is because girls who apply are better prepared, having been better students in high school, and accounting, for example, for 70 percent of valedictorians. Public schools that prohibit gender discrimination in admissions, such as Berkely, have more women in their undergrad program as well as a higher acceptance rate for women. (Women are less likely to apply to great programs if their application isn’t competitive.)

            So, somewhat counterintuitively
            to some, perhaps , if you want to go to Harvard, you’re far better off being a white male than an Asian female.

          • DyingOfLot49 says:

            @The Nybbler

            There are far fewer female applicants to CS so female CS applicants are a much more self-selected group. As a male (conservative) CS student at CMU I’ve noticed no competence gap between genders.

          • The Nybbler says:


            Are you sure they’re not selecting the N/2 top women, then selecting, rather than the best N/2 men, a _matched set_ of N/2 men?

          • Gobbobobble says:

            If there’s one thing a uni loves more than scoring political points, it’s generating Very Successful alumni. Maybe they could get away with deliberately underplaying their hand for an undergrad program but it still sounds awfully difficult to achieve

          • The Nybbler says:

            If there’s one thing a uni loves more than scoring political points, it’s generating Very Successful alumni.

            Very Successful Alumni come from law programs, business programs, or research science programs. Mere computer science people, most of whom go into industry, aren’t much of a factor.

            Maybe they could get away with deliberately underplaying their hand for an undergrad program but it still sounds awfully difficult to achieve

            I don’t trust these people. For years they tried everything but quotas and couldn’t move the needle much. Now they’re trying what looks like blatant quotas (which of course work), and the story is that self-selection completely balances much lower selectivity? If I can think of selecting the men to match the accepted women, so can they. It wouldn’t be a huge effect, not for a school like CMU.

          • Gobbobobble says:

            I was under the impression that most startups come from Bachelor’s-level CS grads. Which are far from universally Very Successful, but enough are Sufficiently Successful to be good poster children and some even make it big enough to buy their name on a building.

            It’s possible a school that a school would throw away its chance at hosting the next Zuck because he might make their quota’d students look bad, but I wouldn’t jump to that conclusion.

          • rlms says:

            It’s obvious what colleges would gain from having a higher/lower admissions bar for overrepresent/underrepresented groups (good publicity), but how would they benefit from rejecting the best students in the overrepresented group? No journalist is going to write a hit piece with the headline “CMU male CS students are smarter than female ones!”. It seems that the only thing it would achieve is to let the world fit your beliefs.

            I agree that something unusual is probably going on, as the situation described doesn’t match my experience (my CS cohort is 13% female, my impression is that the women in it are slightly stronger than the men (the top quartile is 16% female) but certainly not by enough to allow a gender balance in both numbers and ability). But I think the most likely explanation is that CMU have simply managed to attract a high number of strong female applicants (quite possibly by having an even gender balance).

          • The Nybbler says:


            One criticism of affirmative action is the beneficiaries tend to be less-qualified than the non-beneficiaries. This would show, e.g., by the beneficiaries having lower test scores or some other such thing. And this would be noticed; it’s very hard to sweep under the rug, and it leads to problems (like higher drop-out and failure rates later on). By selecting the non-beneficiaries once you know the characteristics of the beneficiaries, you can avoid that problem.

            The diversity people have made gender equality in male-dominated STEM fields an overriding priority. I find it easy believe they would manipulate admission in that way to achieve their goal.

            The alternative claim is that the top men and top women applying are the same, and the excess of men is entirely due to underqualified men applying. This is a convenient world for the diversity people. But if it’s true, why is it only true now?

          • rlms says:


            One criticism of affirmative action is the beneficiaries tend to be less-qualified than the non-beneficiaries.

            Indeed. But as far as I know, colleges with affirmative action programmes don’t reject highly qualified non-minority applicants to avoid the problems this causes (or for that matter the same problems with regard to athletic scholars and legacies). So I don’t see why they would do so in this case.

            The alternative claim is that the top men and top women applying are the same, and the excess of men is entirely due to underqualified men applying.

            That is one alternative. I actually think it is pretty plausible: it seems unlikely when you phrase it like that, but that’s because you are using the loaded (and I think incorrect) term “underqualified”. Even though the “underqualified” men might be weaker, if they’re accepted I think that makes them pretty qualified. So it would be more accurate to just say that relatively weaker women don’t apply, whereas relatively weaker men do. That seems likely to me.

            But that’s only one alternative. I gave another possibility, which is that the pool of qualified men at the relevant level *is* larger than that of women, but women are more likely to apply to/accept offers from CMU than MIT/Stanford. Given that CMU has a 50/50 gender balance and MIT/Stanford don’t, I think it would be odd if that *wasn’t* true to some extent.

          • albatross11 says:


            If you make it so that women get into your CS program with lower qualifications than men, then assuming those qualifications mean about the same things (as predictors of success or ability) for men and women, you’re going to end up with less qualified and capable women than men in your CS program. If you make the qualifications a *lot* lower, that difference will be large and pretty obvious to both teachers and students.

          • The Nybbler says:


            Yes. A *lot* lower probably isn’t on the table here considering the institutions. But my point is you can eliminate this problem by choosing the best women along with a matched set of men. So if your women have an average SAT-M of 780, you choose a men with an average SAT-M of 780, even if you could choose an all-800 set of men. (note this is made easier by the students topping out the test; probably a large number of both men and women applicants have an 800)

      • Hyzenthlay says:

        Getting the left to admit that would be a huge win in the culture war for the right.

        I dunno, I routinely hear people on the left say things way worse than that. Though I guess most of those remarks are coming from the far left and moderates would probably say that those people don’t speak for them.

        • Matt M says:

          Yeah, I very much mean the mainstream left in this instance.

          Can you even imagine say, Joe Biden saying something like “Well of course affirmative action involves racial discrimination against white people! Of course diversity efforts at major corporations make it more difficult for men to get jobs! Everyone already knew that!”

          • Edward Scizorhands says:

            I can easily imagine Joe Biden saying that. Before our current POTUS he was the poster-child for “direct path from brain to mouth” in politics.

          • Matt M says:

            And my point is that such a thought wouldn’t even be in his brain in the first place.

            Let me put it this way: Walking around saying “affirmative action is racial discrimination against white people” strikes me as something that could get you in a lot of trouble at Google.

          • The Nybbler says:

            Let me put it this way: Walking around saying “affirmative action is racial discrimination against white people” strikes me as something that could get you in a lot of trouble at Google.

            You bet. In one incident that I think is mentioned in the lawsuit, someone claimed that the practice of sending “hiring packets” from minority candidates through a second hiring committee if the packet was a weak no-hire constituted “lowering the bar” for such candidates. This ignited a firestorm, where all sorts of high personages swore up and down that this did no such thing and the person who suggested it was bad for even thinking it did.

            (It does, though, under reasonable assumptions.)

          • keranih says:

            someone claimed that the practice of sending “hiring packets” from minority candidates through a second hiring committee if the packet was a weak no-hire constituted “lowering the bar” for such candidates.

            I was (briefly) on a department diversity committee at uni and this practice was confirmed as being an example of how hard the department was trying to get ‘under represented’ minorities enrolled/hired.

            While my (faint, hesitant) questioning of the legitimacy of the practice was (politely) brushed aside, I do count it as a win that the department chair straight up said that there weren’t enough qualified candidates and “all the best ones, nearly all the good ones, and most of the decent ones get snatched up by the Ivies with full rides, and [uni] can’t compete with that.”

            And [uni] *still* let the activist groups accuse them of anti-POC bias.

    • Scott Alexander says:


      As written, it sounds like it only prohibits discrimination based on party. So it’s unclear if you could fire someone for having the wrong views on diversity, as opposed to being a member of the Anti-Diversity Party.

      I’m curious if there’s any case law on this, and how far it’s supposed to go (ie can you fire someone for being a Nazi? For posting inflammatory political trolling online? Etc.)

      • Le Maistre Chat says:

        If there’s an actual American National Socialist Workers Party, it sounds like it would be a de jure misdemeanor to penalize them for it, just like any other third party.
        Of course Paul didn’t clarify whether the law allows creating a politically hostile environment through the use of shibboleths rather than specifically saying “Republicans will be subject to disciplinary action.”

        • Paul Crowley says:

          I’m not an expert on California law, I just did a Google search. You now know pretty much what I know.

      • Brad says:

        There is a case law, and it’s quite a bit broader than just based on party. There’s a case from the 1970s that held that coming out as gay counted as a protected act of political activity.

        Gay Law Students Ass’n v. Pac. Tel. & Tel. Co., 595 P.2d 592, 610 (Cal. 1979)

        These statutes cannot be narrowly confined to partisan activity. As explained in Mallard v. Boring (1960): “The term `political activity’ connotes the espousal of a candidate or a cause, and some degree of action to promote the acceptance thereof by other persons.” (Italics added.) The Supreme Court has recognized the political character of activities such as participation in litigation (N.A.A.C.P. v. Button (1963)), the wearing of symbolic armbands (Tinker v. Des Moines School Dist. (1969)), and the association with others for the advancement of beliefs and ideas (N.A.A.C.P. v. Alabama (1958).)[17]

        Measured by these standards, the struggle of the homosexual community for equal rights, particularly in the field of employment, must be recognized as a political activity. Indeed the subject of the rights of homosexuals incites heated political debate today, and the “gay liberation movement” encourages its homosexual members to attempt to convince other members of society that homosexuals should be accorded the same fundamental rights as heterosexuals. The aims of the struggle for homosexual rights, and the tactics employed, bear a close analogy to the continuing struggle for civil rights waged by blacks, women, and other minorities. (See, e.g., Gay Students Org. of Univ. of New Hampshire v. Banner (1st Cir.1974); Acanfora v. Board of Education of Montgomery County (4th Cir.1974); Aumiller v. University of Delaware (D.Del. 1977))

        A principal barrier to homosexual equality is the common feeling that homosexuality is an affliction which the homosexual worker must conceal from his employer and his fellow workers. Consequently one important aspect of the struggle for equal rights is to induce homosexual individuals to “come out of the closet,” acknowledge their sexual preferences, and to associate with others in working for equal rights.

        In light of this factor in the movement for homosexual rights, the allegations of plaintiffs’ complaint assume a special significance. Plaintiffs allege that PT&T discriminates against “manifest” homosexuals and against persons who make “an issue of their homosexuality.” The complaint asserts also that PT&T will not hire anyone referred to them by plaintiff Society for Individual Rights, an organization active in promoting the rights of homosexuals to equal employment opportunities. These allegations can reasonably be construed as charging that PT&T discriminates in particular against persons who identify themselves as homosexual, who defend homosexuality, or who are identified with activist homosexual organizations. So construed, the allegations charge that PT&T has adopted a “policy … tending to control or direct the political activities or affiliations of employees” in violation of section 1101, and has “attempt[ed] to coerce or influence … employees … to … refrain from adopting [a] particular course or line of political … activity” in violation of section 1102.

        (citations omitted)

        I don’t know how this case is going to come out, but it doesn’t look frivolous.

    • fnord says:

      Here‘s a blog post by an attorney noting the California law on political discrimination but still rather negative on Damore’s chances.

      As I read them, these statutes clearly protect off-duty, off-site political activity. They protect running for political office. They protect registration with a particular partisan preference. But I don’t think they necessarily protect an on-duty or otherwise work-associated expression of a political opinion.

      • Scott Alexander says:

        Thanks, added to the post.

      • herculesorion says:

        ha, yes, well. That guy doesn’t actually provide an argument that Damore has no case beyond “Damore’s a big wiener and I don’t like him”.

        It takes a certain kind of doofy ex post facto reasoning to argue that “work-associated expression of political opinion” is not something that the California antidiscrimination statues were specifically written to protect.

        • Anonymous Bosch says:

          That guy doesn’t actually provide an argument that Damore has no case beyond “Damore’s a big wiener and I don’t like him”.

          Private companies may engage in voluntary affirmative action programs. United Steelworkers of America v. Weber (1979) 443 U.S. 193. Nor, under California law, is an employer required to be blind to or ignore the demographics of its workforce and may permissibly take action to ensure balance and diversity within that workforce. Bolin v. San Bernardino City Unified School District (1984) 155 Cal.App.3d 759. While according to Damore this was phrased strongly and with generous use of popular buzzwords, what’s substantively described seems to me to at least plausibly fit within the guidelines permitted by Bolin’s interpretation of California anti-discrimination law.

          More simply put, reaching out to women and minority groups is not invidious discrimination. Affirmative action is not invidious discrimination. Damore appears to be laying the groundwork to argue that these cases be overturned.

          Not being a CA employment attorney myself I cannot say whether he’s misreading those cases. I can say that they constitute a stronger legal argument than “Damore’s a big wiener and I dont like him.”

        • howardtreesong says:

          The man walks through the statutory basis for Damore’s claims and the relevant case law. Yes, it’s clear that he doesn’t like Damore or his politics, but his analysis is substantially deeper than what you suggest.

        • Vorkon says:

          ha, yes, well. That guy doesn’t actually provide an argument that Damore has no case beyond “Damore’s a big wiener and I don’t like him”.

          To be fair, that’s almost certainly the basis upon which the case will be decided, one way or the other, so it makes a rather persuasive legal argument! :p

      • mupetblast says:

        So then the Berkeley hot dog guy has a chance. But not Damore.

    • Nornagest says:

      This and this appear to be the applicable sections of law, and while IANAL, they look a little narrower than the language used for other types of discrimination. It is not in the big list of protected classes here, for example.

    • The Nybbler says:

      I don’t think it was in the lawsuit, but there was one case where an employee was disciplined (given a warning only) for posting on social media a picture of himself with a television showing Rand Paul, while he was wearing a Google T-Shirt. (This discipline may have eventually been withdrawn.)

      I have no doubt that if you were to reverse genders and races this would be a slam-dunk “hostile workplace environment” case, even without considering the political angle. However, I have no faith that the courts will enforce the law in a non-discriminatory manner, despite the fact that the statutes are non-discriminatory on paper.

      • Le Maistre Chat says:

        Judges can’t be trusted to apply the law as written, because higher education is controlled by the Left. But not all judges leave law school brainwashed, so…

    • keranih says:

      I was told the CA law was originally put into place in order to protect members of the American Communist Party from workplace reprisals.

      Which would make Damore’s lawsuit – let alone the thought that he might win – a delightful case of being careful what you wish for.

      • Le Maistre Chat says:

        OTOH, a judge could easily enforce the spirit of the law rather than the letter, de facto ruling that it protects Communists and Democrats but not Republicans.

    • Mark Atwood says:

      I need to research those laws in Washington State…

      I recently discovered that teachers in the local public high school has been offering extra credit to create banners advancing various progressive causes, and to march in the various progressive parades and protests here in Seattle. The school’s printshop and arts supplies room was made available for this at specific times after school hours. To get the credit, the students have to have a picture taken of themselves at the protest or a parade, posted to social media, with appropriate hashtags.


      • A Definite Beta Guy says:

        This sounds pretty bad. You can just as easily offer extra credit for attending local council or planning meetings. This was actually a requirement for my political science classes (and which I determined my village board had stupid freakin’ planning requirements that was killing business development).

      • Deiseach says:

        Let me take a wild guess: a student who wanted to make a banner or placard for their participation in the March for Life, or who wanted extra credit and provided ” a picture taken of themselves at the protest or a parade, posted to social media, with appropriate hashtags” is not going to get the same response from the teachers? 🙂

  4. JulieK says:

    I have a recurring dream of wandering around a very large high school, trying to find my classroom. (The high school I attended in real life was quite small.) Sometimes I find the room, but I’m uneasy because I’ve missed the last few classes and haven’t done the homework.

  5. bestburough says:

    I wish we would stop giving attention to James Damore (the Google memo guy). Not because of his controversial memo, but because that guy appears to be a liar. Links:

    • Evan Þ says:

      Okay, it sounds like he is a liar, but it also sounds like he exposed a seriously toxic culture at Google. Can we still talk about his lawsuit (which doesn’t appear to be impacted by his lies) and about Google’s culture?

      • lambda_calculus says:

        Excuse my ignorance, but what is the seriously toxic culture that Damore exposed?

        • Aapje says:

          Workers fighting the culture war among themselves, including by refusing to work with wrongthinkers. The management rewarding people who write political opinions for one side with ‘peer bonuses.’ Having a worker tell a coworker that he will try to harass him out of his job. Having a worker argue on the intranet that people on the other side of the culture war should be threatened with violence. Workers discussing on the intranet how they can effectively use violence against the other side of the culture war.

        • Matt M says:

          Would like to draw attention to this response, for all of the “This will be bad PR for Google because nobody wants to work in this kind of environment” people…

    • christhenottopher says:

      Chess is often used as a signal of intelligence, large numbers of people lie on their resumes, scoring a Google programming job means a six figure salary with great benefits, and signaling intelligence likely helps a person get Silicon Valley programming jobs. End result? I’m pretty underwhelmed by this. It means almost nothing to the story of Damore’s firing or the wider argument about the validity of the points in his memo. Is it good to lie even given the incentives I point out above? No. Does this particular lie signal a person who will compulsively lie? Not really, even the doubling down might be an attempt to not make himself harder to hire elsewhere in Silicon Valley (or give a potential excuse for Google to say “we didn’t fire him over politics but over lying on his resume”). It’s not a great doubling down, but that could be panic as much as anything else.

      So really, I don’t see what’s been proven besides the guy isn’t a perfect paragon.

      • Brad says:

        A typical example of a lie on a resume is: “Expert in C++ templates” when really you only have a moderate amount of experience with C++ templates. Or spearheaded a project that saved the company $100 million in cloud costs, when really you weren’t the driving force behind the project, but instead just one of several people on the project.

        Not something concrete, verifiable, and strictly untrue. Those kind of lies are exactly the ones that suggest someone may be a compulsive liar. Nor would I would so blase about the doubling down. That’s the kind of blustery defensiveness that, again, is associated with a habitual liar.

        I’m reminded of Paul Ryan’s claim while running for VP of once having run a sub three hour marathon.

        • vrostovtsev says:

          You’ve chosen pretty poor examples given that you sometimes have to claim you are an ‘Expert in C++ templates’ so that a recruiter will match you with a job requirement of ‘C++ templates wizard’ when in reality what they need and what you are is the same: a person with moderate experience with C++ templates.

          In other words, these have nothing to do with being a ‘compulsive liar’. And neither does Dalmore’s misconception that he is a chess master.

          • Brad says:

            In other words, these have nothing to do with being a ‘compulsive liar’.

            I didn’t say or imply otherwise. And your last sentence has precisely the same semantic content as “nuh-uh”.

    • suntzuanime says:

      “Can we stop listening to liars please?” No. Everybody lies. I’m sorry, I really am.

    • bbeck310 says:

      He definitely lied about the FIDE Master thing, which is weird–and I’m not surprised that Greg Shahade, a top scholastic US chess player who is still a top US player, called him out on it. For many of us former scholastic champions (I wasn’t in Greg’s class, but I have a few scholastic state championships), media reporting on self-reported strong players who demonstrably aren’t is a pet peeve. I get particularly annoyed with the common “inner city kid/school wins national chess championship” stories, which are invariably about kids or teams that win reserve or novice sections and would get slaughtered by the kids who won 3 out of 7 in the championship sections. (Note: IS 318 in Brooklyn is an exception–they really were that good). There’s also a common version of that in Chicago of “Local urban school wins National Girls’ Championship” – while official, the National Girls’ Championship is held every year in Chicago and the older age group sections tend to not be attended by teams, so usually the 15-16 year old section has only two schools fielding a full team, both of which are local and both of which contain only students rated under 1000.

      That being said, almost all the claims in the Damore complaint are supported by primary documents, so I don’t need to rely on Damore’s truthfulness–and if the goal is to damage Google in discovery, his eventual roasting in cross-examination won’t matter too much.

      • Deiseach says:

        I don’t know anything about chess, I don’t know if this was “pump up your CV/resumé, don’t say ‘I was on the chess team and won a tournament’, say ‘I was ranked as Whatever'” and so forth.

        I will say, I’ll be remembering this argument the next time somebody on the progressive side is tripped up over a similar detail and the “oh come on, does that really matter?” comment is made. (And I have seen that exact point being made about “so what if George claimed X and it was really Y, or X never happened – that’s only a minor detail, the larger point stands” in debates where George is maintaining that he knows all about being a poor Albanian goatherder who was exiled for being gay, so he is entitled to speak about homophobia, and it turns out he never went near a goat in his life).

        Yes it does, since it’s been made to matter here. If Damore deliberately and knowingly lied that is dishonesty. If that’s the biggest detail about “And so his entire case against Google is worthless!” then it’s not relevant (it would be relevant if Damore claimed he was, say, physically threatened by a manager in Google and that was shown to be a lie). If in future someone making a claim that the Department of Health and Human Services is being over-run by Evangelicals pushing a religious agenda* lied about winning a spelling competition in third class, then the same rule applies: either we junk everything they say because THEY ARE A LIAR, or we look at the larger claim and judge what is happening independently of the ‘did they lie about this thing that has nothing to do with that thing?’.

        *”Ugh, does it matter if the guy is a Melkite not an Evangelical? Either way, he’s a bigot!” Why yes, yes it does, since you’ve made it so that “it matters if the guy lied about a chess ranking since that would invalidate his claim of constructive dismissal”.

        • moscanarius says:

          Indeed, it is funny how personal honesty becomes the most relevant thing in the world when one can use it to bash the outgroup. “Damore lied about his chess achievements! OMG I CANT EVEN WHAT A LYING LIAR HIS MEMO IS FALSE HIS LAWSUIT IS WRONG!!1!!”

          Meanwhile, when some people actually fabricate stories we are supposed to focus on the bigger picture.

          • pdbarnlsey says:

            I’m not sure the chess guy above can be taken to speak for “the left” or even if he’s a member of it.

            FWIW, as someone who’s left of America’s centre I think this lie is morally bad behaviour (don’t claim achievements you haven’t earned and don’t double down when called on it) and probably means I would trust Damore’s veracity slightly less in unrelated circumstances. But it doesn’t impact much, if at all, on how I judge his treatment by Google or the validity of his law suit.

            The examples you quote, as far as I can tell by just mousing over the links, make me think less of the perpetrators, trust them less, or not at all, in the future and, in at least one case, also reflect badly on the journalist and publication involved, and make me trust them less in future.

            But my authority to speak for the entire Left recently lapsed, so YMMV…

          • bestburough says:

            Your assumptions are built on thin air. I am a chess player, which is how I found about this, and I value personal integrity, which is why I care about this.

          • moscanarius says:


            The conversation drifted a bit; I was replying to Deiseach, who made a side point about an argument with some other person of progressive leanings, not chess guy; and she was replying to bbeck310, who mostly talked about Greg Shahade (author of the first link), not about the original commenter. So yeah, maybe at this point everything is a bit confusing.

            Chess-guy Greg Shahade can be definitely be called a member of the left, in politics and culture, as per his own blog posts. About commenter bestburough I cannot say much for now. We can’t say Shahade speaks for “the left” because he is not that famous, but he shares the most loud of their common views.

            FWIW, as someone who’s left of America’s centre […] his treatment by Google or the validity of his law suit.

            I don’t disagree with anything there. Greg Shahade does, however, and I presume bestburough also does, as he shared Shahade’s post without any sort of disclaimer.

            The examples you quote, […] make me trust them less in future.

            I find this all very good and don’t disagree at all. Perhaps I was a bit heavy-handed there, so maybe I should try to put in better terms:

            I just get annoyed when a personal flaw (like lying), even if serious, gets blown out of proportion when it can be weaponized. Shahade was basically calling for Damore to never ever be listened to anymore because of a minor lie; he was assuming all sorts of character flaws to be upstream of that one lie he could find Damore sustaining. That’s bad. I am pretty sure that Shahade would never make this Isolated Demand of Honesty in any of the cases I quoted.

            The examples I gave serve two purposes: one is to illustrate how even much more egregious forms of lying are often excused if the liar has the right political colors; but another purpose is to show that, indeed, we have to keep an eye on the big picture. If some lying liar Muslim girl lies about having been attacked by a Trump supporter, my trust in her goes down by a lot, and my trust on the next hate crime accusation goes down by a bit; but neither goes down to zero. I still will neither discard all the girl says as a lie, nor call every instance of denounced hate crime a hoax; and I don’t think anyone should. If even in a direct case of lying about important stuff we don’t ostracize the liar and ignore the message, why should we do this in Damore’s case?

            I hope I was more clear this time.

          • moscanarius says:


            Your assumptions are built on thin air. I am a chess player, which is how I found about this, and I value personal integrity, which is why I care about this.

            Please notice that this was my reply to Deiseach’s comment, in which she mentioned hearing that sort of argument pushed by progressives, in answer to bbeck310’s comment about Greg Shahade, which you linked. This comment you replied to is not discussing your position on these matters; the discussion just drifted, it was no longer about you. You can see my direct reply to you downthread; you will find no unflattering assumptions about you there.

            However, though I’m not (yet) assuming you hold these double standards, I’m confident to say that Shahade does. I searched his blog; there are two instances of him denouncing personal dishonesty: this Damore post and one against a certain Azmaiparashvili. He says he has lots of thoughts about Ferguson, but none of them involves calling out the liars. He wrote exactly one thing about Trump, and Trump’s lies were not even mentioned. Doesn’t sound like an integrity enthusiast. Then, all of sudden, he just finds out that lying is unberable when he needs to criticize a person he disagrees politically with.

            That looks a lot like what I described previously.

            By the way, about this part:

            …I value personal integrity, which is why I care about this.

            You’re hardly unique; we all value personal integrity here. The question is about how much we should mistrust and/or mistreat a person who was found not to be 100% integrous. I think this should lower our trust on them, but not make us completely skeptical of everything they say. I already explained my reasons for this.

            Shahade, which you linked, seems to think otherwise:

            If you think about the people in your life that you respect and look up to, none of them will just lie to you about petty bullshit. But James decided he would. And because of that, his voice is no longer relevant. He has shown himself to have a low moral character and shouldn’t be taken seriously about anything.

            Do you agree with him?

          • bbeck310 says:


            Chess-guy Greg Shahade can be definitely be called a member of the left, in politics and culture, as per his own blog posts. About commenter bestburough I cannot say much for now. We can’t say Shahade speaks for “the left” because he is not that famous, but he shares the most loud of their common views.

            This is correct–and it’s the reason why I don’t think the parts of Shahade’s post beyond the chess analysis are relevant, interesting, or worth arguing about. I knew of Greg, mostly through his younger sister Jennifer, because the two of them were extremely strong scholastic players about 4-5 years older than me, and Jennifer taught at a summer chess camp I attended around 5th-6th grade. Greg and Jennifer are excellent authorities about chess, both the game itself and its governing institutions in the U.S. and the world. When it comes to politics, I have no reason to take Greg as having any more expertise than any other random person on the street, and I don’t think it’s productive to focus on the non-chess aspects of his post. (FWIW: I am definitely on “the right,” and agree with you on the general issue–I just don’t like it when discussions get derailed into debating a third party’s irrelevant opinions).

          • moscanarius says:


            I have no reason to take Greg as having any more expertise than any other random person on the street, and I don’t think it’s productive to focus on the non-chess aspects of his post.

            That’s fine. Let me be clear that I would not bring his political opinions to the discussion if we were talking only about chess. I would not (as I have not) disqualify anything he said about his sport just because he is wrong about some other things.

            But he was not talking only about chess, he was trying to talk about morality.

            bestburough wants us to stop talking about Damore because he is a liar. Shahade wants to make character inferences about Damore based on his lie about chess.

            That’s a moral call. I didn’t question his chess-ranking discussion, as I both don’t understand it and trust he is not lying to us. But when he decides to take a moralistic stand in public, well… a moralist like me feels like jumping into the fray.

            So yes, I don’t think we should take him seriously outside his area of expertise. But since someone though it good to expose his moral thought to us (presumably agreeing), I think it deserved a countercomment. The moment someone here agrees with Shahaden, his opinion becomes more relevant in my opinion. Publically stated moral opinions – specially those so uncharitable as the ones he expressed – invite some sort of response. I’m not saying “hate this guy” or “strip his chess titles”, I’m saying “he is mischaracterizing Damore for political divergences, here’s the proof”. Let all the interested parties keep following his chess exploits, now well warned that his moral judgement is not to be relied upon.

    • herculesorion says:

      No True Scot would argue using an ad hominem fallacy!

    • Squirrel of Doom says:

      If we stopped listening to liars, how would we keep up with anything our politicians said?

    • My comment posted in response to the first of the two stories linked to above:

      Your story is, I presume, correct. My objection is to the headline–the implication that we should not listen to anyone who has been demonstrably dishonest. As you probably know, it is now well established fact that Martin Luther King plagiarized his doctoral thesis, copied large parts of it from a thesis done by another doctoral candidate a little earlier. That’s dishonesty of the same sort you describe in a matter substantially more serious than lying about how good a chess player is in an application for a job that isn’t about playing chess. Do you conclude that nobody should have listened to anything King said?

      Obama, in order to pass a major and controversial piece of legislation, lied about it, promised the public that if you like your present insurance you can keep it. Should nobody listen to anything he said?

      FDR, in his first presidential election, attacked the incumbent for spending too much. He then proceeded to spend much more. Woodrow Wilson campaigned as a peace candidate and then brought the U.S. into WWI. Both of those are acts of dishonesty more serious than Damore lying about his chess accomplishments.

      Do you think nobody should have listened to anything any of those people said? If not, you are applying a double standard–”don’t listen anything a liar says if he says things I don’t like.”

      Most people will lie if the reason to do so is strong enough. The sensible policy is to take nobody you do not know well on faith, listen to what people say with the knowledge that it might not be true and then try, if possible, to check whether it is.

      • Lillian says:

        While i don’t disagree with you in principle, i do dearly wish nobody had ever listened to anything Woodrow Wilson ever said. The title for worst President ever is a dead heat between him and Buchanan.

        • howardtreesong says:

          I nominate Warren G. Harding and Lyndon Johnson. Harding was entirely corrupt. And Johnson was reputed to be extraordinarily crass — introducing people to his penis, forcing Cabinet members to follow him into the crapper to continue a conversation while he defecated, peeing on a secret service agent, etc. etc.

          I’d put Wilson in the second tier, right behind those guys. I don’t know anything about Buchanan, however.

          • quaelegit says:

            Buchanan was the president before Lincoln, so he gets blamed for letting the civil start. He was nominated because he’d been out of the US for years (diplomatic stuff, don’t remember the details) and so had avoided the recent political fights that had left everyone else in the party with too many enemies.

            I don’t know if he could have held things together by 1857, but the impression given in my high school US history class was that he didn’t try very hard/effectively.


            For Harding — I know there was a lot of corruption during his term but was he in on it or did he fail to keep a handle on his staff and administration (like with Grant, IIRC)? Lucky for Harding he died before facing much scrutiny.


            Hadn’t heard of these stories of Johnson (but I don’t know much about him except what they say at the LBJ ranch, where they obviously present him in the best light). It complicates the question of best/worst president on how much to weight personal behavior vs. policy decisions vs. public perception/leadership (JFK really good at this, Carter really bad at this).

            Edit: I wouldn’t weight personal behavior as strongly as policy personally, because there’s so many presidents that I don’t know about the personal behavior of.

          • Evan Þ says:

            @quaelegit, more than that – Buchanan stood back and didn’t even say anything while traitors in his Cabinet traitorously evacuated military bases in the South, and moved military supplies to other poorly-defended Southern bases so that the Confederacy would be able to seize them. Without this aid, I’ve heard, the Confederacy would probably have fallen at least a year earlier.

            A lot of people will also criticize Buchanan for personally not using military force against secession. I’ll give him a pass on that one, though: he was a strict constructionalist, and he sincerely believed that the governments of the seceded states remained legal state governments, and the Constitution did not allow the Federal government to use military force against them.

            That said, he stood back and winked at literal traitors in his Cabinet who literally gave aid and comfort to literal enemies of the United States, resulting in the death of tens of thousands more United States soldiers.

          • sourcreamus says:

            Harding was a good president. He came into office during a depression and when he died the economy had turned around and was strong. The combined unemployment and inflation rate went down the most during his administration than any other president.

          • Lillian says:

            It boggles the mind that someone would call Lyndon Baines Johnson a bad President, let alone the worst President, on his private personal behaviour. His Presidential administration affected the lives of tens of millions of people, it is on that basis, whether he affected them for good or ill, that he should be judged. If you think he was a terrible President because of Vietnam, or the Great Society, or Civil Rights, then fine there’s an argument to be made there. But calling him literally the worst ever because he shoved his dick into the faces of other men who walked the halls of power? Really?

            Similarly the fact that there were a bunch of corruption scandals in the Harding administration hardly makes him the worst. It certainly doesn’t make him a good president, sure, but the fact is that none of those scandals hand much of a lasting negative effect on the United States. Unlike say, Wilson and Buchanan, whose action or inaction cost America tens of thousands of lives.

      • anonymousskimmer says:

        Obama, in order to pass a major and controversial piece of legislation, lied about it, promised the public that if you like your present insurance you can keep it. Should nobody listen to anything he said?

        As I said before: The insurance companies chose to stop offering the grandfathered plans. Yes, the PPACA did incentivize this choice of theirs, but it was an insurance company choice none the less. Obama never stated that the insurance companies would stop making the same business decisions they had made in the past.

        I don’t argue with the broader point. Obama should have made it clear this is what was likely to happen. His lie was through wishful thinking and possibly omission, not commission.

        • but it was an insurance company choice none the less.

          Since it would have been illegal to offer the same policies on the same terms after the change, it was not their choice.

          I don’t argue with the broader point. Obama should have made it clear this is what was likely to happen. His lie was through wishful thinking and possibly omission, not commission.

          You have a lower opinion of his intelligence than I do. It was perfectly obvious that if an insurance company was required to add coverage of things it had not before covered and if it was required to accept all applicants of the same age and sex at the same price independent of their condition of health, it could not continue to offer the same policies as before at the same price.

          Obama very much wanted the bill passed and was willing to tell a deliberate lie for the purpose. That may or may not have been immoral, depending on your view, but it was dishonest.

          • anonymousskimmer says:

            Since it would have been illegal to offer the same policies on the same terms after the change, it was not their choice.

            Of course it was their choice. They could have kept offering the same plans to the same people (no new comers) under a grandfathered status. As the people wouldn’t change, the actuarial tables for those plans wouldn’t have changed either (except to the extent people dropped out of the plans), thus prices could also have been maintained with only standard increases, adjustments for drop-outs, and adjustments for the ending of lifetime limits and child coverage.

            This would have meant higher overhead for the insurance companies as they suddenly have to manage their old plans and create new plans for the PPACA-compatible markets, but that in no way prevented them from maintaining the old plans (it merely incentivized them not to).

            Now you’re the one being disingenuous with your statement.

          • Thegnskald says:

            anonymous –

            There were specific requirements applied which, combined, made it impossible for most grandfathered plans to continue.

            In particular, the extension of benefits to young adults, the elimination of exclusions on pre-existing conditions, and the elimination of lifetime limits, along with strict limits on the growth rates of costs to consumers, meant most plans could not, in fact, just keep the same actuarial tables. The risks we’re changed substantially.

            It is possible, even likely, that Obama lacked sufficient understanding of the healthcare industry to anticipate this. However, before making that sort of promise to the public, maybe he should have asked somebody who did understand the industry?

            A convenient lie justified by easily-remedied ignorance is not exactly an improvement over an outright lie in full knowledge. I don’t see the purpose in incentivizing politicians to not understand things so they get a free pass when they say something dumb.

        • Edward Scizorhands says:

          It wasn’t just insurance companies deciding things on their own. They were sometimes forced to abandon their plans because the grandfathering rules were strict.

          The Affordable Care Act tried to allow existing health plans to continue under a complicated process called “grandfathering,” which basically said insurance companies could keep selling plans if they followed certain rules.

          The problem for insurers was that the Obamacare rules were strict. If the plans deviated even a little, they would lose their grandfathered status. In practice, that meant insurers canceled plans that didn’t meet new standards.

          Obama’s team seemed to understand that likelihood. U.S. Health and Human Services Secretary Kathleen Sebelius announced the grandfathering rules in June 2010 and acknowledged that some plans would go away. Yet Obama repeated “if you like your health care plan, you can keep it” when seeking re-election last year.

          • anonymousskimmer says:

            If the plans deviated even a little, they would lose their grandfathered status.

            If the plans deviated even a little they would no longer be “your plan” that you could keep if they liked it.

    • Glen Raphael says:

      Is it possible he just confused FIDE Candidate Master with FIDE Master? Is there a way to look up the former?

      Some of Damore’s chess-related claims seem easily confirmed (like coming in second in a youth competition) so the issue seems puzzling.

      • Aapje says:

        Candidate Master requires a rating of 2200, but Damore only got to 1817. So there is no way he got that title.

        • bestburough says:

          He’s actually never had a FIDE rating of 1817. He had a rating from his local chess federation (USCF), which does not translate to FIDE and does not suffice for any FIDE titles.

        • Glen Raphael says:

          Are you guys SURE that chess rating info can’t exist and then expire and get removed from the database? I press the point because, dammit, I had a USCF chess rating back when I was in high school, and when I do a player search today on my own name all rating and game info is missing. That tells me that sometimes USCF data does get misplaced or lost or otherwise doesn’t get into the national database. Or at least it did at some point in the past, so I’m reluctant to confidently assert it’s never happened since then. If the claim Damore is trying to make is “I briefly had a score above 2200, which would render me an FIDE Candidate Master…but for some reason that score level doesn’t show up in a search today”, that might be false but I’m not sure we can regard it as proven false.

          My USCF Player Info

          • bestburough says:

            I don’t know how USCF is running things, but FIDE ratings do not expire. And more importantly, FIDE titles never expire. The only way that someone could conceivably lose their FIDE title is if they were exposed for fraud or cheating.

          • Edward Scizorhands says:

            As someone kind of sympathetic to Damore, Damore is the one who can provide some evidence that he used to have that ranking. An old email, an old program, something. The fact that he hasn’t makes me think he’s lying.

          • Glen Raphael says:

            @Edward Scizorhands

            I suspect Damore is just confused and misremembered his chess status and has no proof to establish it either way. I have sympathy for that last part of his status because I share it.

            My own story is that in high school I joined the chess team junior year (playing ~6th board) and became President of the chess team senior year (playing ~2nd board, partly because I’d improved but also because so many better players had graduated). We played a bunch of tournaments. Most were just between two schools and not officially rated, but there were occasional regional school tournaments or local public tournaments that were officially rated, and if you played in one of these and won at least one match, they’d send you a paper saying what your rating was. IIRC the first rating I received was in the mid-1600s, quite unimpressive by the standards of my school – our best player was rated over 1900.

            The thing is…the tournaments blur together. I don’t remember the name of any of the tournaments, much less the names of specific ones that resulted in specific scores from specific organizations. A few are notable for other reasons. Like…there was an event (possibly held at UC Berkeley? Or maybe at SRI?) where I got to be among a couple dozen people playing simultaneous games against chess master George Koltanowski and he signed for me a copy of his book. Or there was…an event somewhere in Silicon Valley (maybe Sunnyvale?) my junior year where the winning teams from several local school divisions met and I won a huge blue-and-goid trophy for being, the second-best C division player. That is to say: of all the players who were so bad they were only 5th, 6th, or 7th boards at their school, I was the second-best of those players, and that somehow merited a trophy – I think that event might have produced my first rating.

            Point is, I do remember having an official USCF rating – having a couple in fact – and I probably did but I don’t remember exactly what events the ratings came from. And I have no physical proof – I didn’t keep a slip of paper with my rating on it in a scrapbook somewhere and the rating info is not on the old USCF site when you look up my name. Honestly, my chess rating wasn’t that important at the time and memories associated with it have grown hazy. So if somebody were to demand today that I prove what my chess rating was then, or produce an email documenting it, or give the name of the event that produced it, I can’t.

            So…does this make me “a liar”?

          • arabaga says:

            Were all of the tournaments you played in before January 1990? If so, that explains it – if you look at the various tabs, they all give a cutoff date, e.g. “There are no tournament results for this member since Jan. 1, 1991” in the Tnmt. Hst tab.

          • Brad says:

            So…does this make me “a liar”?

            It depends. If on the basis of that history you claimed in writing to be a FIDE Master (i.e. has a 10% to beat a grandmaster, less than 10k worldwide as of 2010) and when challenged on it you doubled down, yep that would certainly make you a liar.

            I fenced in high school, even won from time to time, but I don’t go around claiming to have been a junior olympian. If somehow I managed to forget whether or not I was in the junior olympics I think I’d err on the side of saying I wasn’t. That doesn’t seem like the kind of thing I would have forgotten.

          • Edward Scizorhands says:

            I can totally see how he thought something about himself, misremembered, and then came to believe the best possible story about himself. It’s natural and human memory is fallible, especially about oneself.

            But, as far as I know, he hasn’t said anything like that. When pushed on the issue, he said “nope, definitely had it.”

            If he comes out and says “fudge, I guess I was wrong,” at least I could go either way between lying and honestly misremembering. But he kind of just leaves it there without doing anything, letting his backers try to come up with possible reasons.

            The ball is in his court and he really could do something about it.

            EDIT: I see in a link below he has raised the possibility that he was just misremembering. Okay.

          • bbeck310 says:


            And I have no physical proof – I didn’t keep a slip of paper with my rating on it in a scrapbook somewhere and the rating info is not on the old USCF site when you look up my name.

            But the old USCF site does tell you why the rating info isn’t there–it lists your expiration date as 5/31/87, and USCF doesn’t have its tournament records online from before 1991. That’s not the case with Damore, all of whose tournaments were after 2000.

            IMHO, the most likely “innocent” explanation is that Damore played on a reputable Internet chess site like ICC, got his rating up to 2200 there, and thought that counted as a Master rating.

          • Glen Raphael says:


            the old USCF site […] lists your expiration date as 5/31/87, and USCF doesn’t have its tournament records online from before 1991

            arabaga and bbeck310 are correct, all my tournament data would have been before 1990. So that seems like one mystery solved.


            If on the basis of that history you claimed in writing to be [status] and when challenged on it you doubled down

            I don’t think you’re using the term “doubled down” correctly. Damore sounded more vague and less certain about it when challenged. As I read it, he retroactively reconstructed why he thought he had that status, but has admitted some confusion and has also minimized how solid that status was. He seems to have thought he briefly, just barely, while in middle school qualified. He hasn’t admitted being entirely wrong yet, but he has backpedalled a bit.

            Speaking as a former serious blackjack player, I’d call this behavior either “reducing the bet” or “standing pat”, not “doubling down”.

          • Mark Atwood says:

            I have the same status about my chess rating. I was in the chess club in high school in the mid to late 80s, and was an ok player. (“ok” meaning that could almost always beat someone in a random public match, but always lost in interesting and embarrassing way when I played against the better people in the chess club.)

            I played in a BUNCH of multischool matches, and even ended up going up a couple of levels in state and regional tournaments. I ended up with a stack of official looking certificates with my ranking printed on them. They were all lost in a basement flood in the early 1990s.

            I did, in fact, put my chess club membership and those ranking numbers on my applications to universities. And did put it on my first-entering the job market resume. If I had done better at chess, I also would have left it on my resume for longer, especially if I was applying for grad school or a corporate software research gig.

        • Part of the evidence that he was a repeated liar was that in the AMA he claimed that the requirement used to be 2200. If that was true for Candidate Master then it isn’t as clear a case of lying as it was claimed to be.

          Obviously that still leaves the rest of the evidence.

          • Glen Raphael says:

            Further complication: 2200 does make you a “Master” under USCF rules, just not under FIDE rules. USCF also calls 2000 (not 2200) a “Candidate Master” (Source: the “Norm and Title Rules” table on page 3 of this pdf).

            Damore says on reddit that he probably confused FIDE with USCF when claiming 2200 was “master”.

          • bestburough says:

            I fail to see how that helps his case in any way. Yes, there is a less prestigious title (Candidate Master) with a requirement of a FIDE rating of 2200, but Damore has never even had a FIDE rating, let alone a FIDE rating anywhere in that vicinity.

            As for the more obscure USCF title: he does not have that one either! It requires a USCF rating of 2000, but his USCF rating peaked in the 1800s. If his case is that he confused the FM title with a much worse title that he never even had either, then I think it’s time for Damore to just admit that he lied.

      • bestburough says:

        He’s never had a FIDE rating, let alone a rating above 2200, which is the minimum for the Candidate Master title. FIDE ratings can be found here:

        And as a chess player, I would like to mention that those titles are very hard to achieve.

        • Squirrel of Doom says:

          As another chess player, let me reemphasize that even 2200 means you’re very seriously good at the game!

      • bbeck310 says:

        Glen – He might have confused CM with FM, but the FIDE rating site shows no listing at all for the last name “Damore.” Unless he played in FIDE-rated tournaments under a pseudonym, he doesn’t have a FIDE rating. It would also be very odd if he showed as having a FIDE rating for games played after 2005 but the USCF system didn’t register it; almost any FIDE-rated game in the US goes towards USCF ratings as well.

        There’s no payment required for FIDE, and no expiration. For example, I have a listing on FIDE from playing in 1 FIDE-rated tournament in 2012. While FIDE password-protects its tournament history, you can find the same tournament record at USCF.

        (And, since I was trying to do chess-blogging at the time, 3 of my annotated games from that tournament are still online here, here, and here, though the site that supported the diagrams I posted has since broken. The title of the last one, “FIDE Rating Established!,” turned out to be wrong–I thought all you needed to establish a FIDE rating was 3 games with 1 total point, but that’s incorrect.)

      • Rob K says:

        Short answer: no, it’s very unlikely. Even weak FIDE rated players are searchable online. To achieve a FIDE rating anywhere in the range of FM/candidate master titles you’d have to play a significant number of FIDE rated games, and it’d be possible to find your player profile.

    • Edward Scizorhands says:

      That second site uses the claim that he had “PhD” in his LinkedIn when he was just a candidate and evidence he lied about his chess ranking. But IIRC that’s the default way that LinkedIn displays people in PhD programs, and Google recruited him out of his PhD program so it wasn’t like he was trying to fool them.

      I worry about a citogenesis being created here.

    • moscanarius says:

      As others already said, if we stop giving any attention to liars, we won’t be giving attention to anyone anymore. People lie sometimes; but few lie all the time about all things, big and small. Finding out that Damore lied about his chess achievements may lower our trust in him, but should not make us assume that he is always lying, nor that his complaints are all lies, nor that he has nothing relevant to say.

      That said, I took some time to look at the first link. It’s a mess. Yes, Damore lied, but the inferences the guy makes from this are unacceptable. Let’s see:

      Recently James Damore, was fired for a memo he wrote while working at Google. I didn’t read it as I really don’t have time to read more pseudo science about the differences between men and women from random guys.

      He won’t read it, but he is just sure it is pseudoscience; and he can barely disguise his dislke for Damore. Great start.

      The point of this blog is to expose James Damore is a liar. And not only is he a liar, but he’s the type of liar who lies for sport about extremely minor things.

      Let me correct that: the point is that he found one lie. From this lie, he is trying deduce Damore’s motivations for lying and to establish him as a big fat liar.

      His case is that Damore lied about being a FIDE Master, and when questioned, doubled down. I understand nothing of chess ratings, but will trust the blogger on it.

      The problem is that he tries to make a character inference from this episode:

      This is the behavior of a lying narcissist who is used to getting whatever they want in life, and is also used to there being no consequences for his actions. Given the fact that no one seems to care that he’s a liar but instead is more concerned on his uneducated opinion of the differences between men and women, he’s being proven correct.

      When someone is willing to tell a blatant and calculated lie about something so inconsequential, what are they going to do when they have the chance to lie about something with real importance? Of course they are going to lie, because that’s what liars do. They lie, and lie and lie again, if they think it will help them.

      None of this is warranted. You can’t call it “narcissism” by meaninful definition, or say this shows him to be “used” to whatever, based on this single data point. There is no basis for assuming that the guy who tells small lies is certain to tell big lies. And most importantly, there is no reason to assume that a liar will only tell lies; even the biggest liar of them all can still be telling the truth sometimes.

      His interpretation is very uncharitable and unreasonable. Were he doing this to, say, Obama, everyone would dismiss him a nutcase partisan. And they would be right.

      If you think about the people in your life that you respect and look up to, none of them will just lie to you about petty bullshit. But James decided he would. And because of that, his voice is no longer relevant. He has shown himself to have a low moral character and shouldn’t be taken seriously about anything.

      This is the worst. A liar is still a human being, and even a immoral person deserves at least the fair treatment from members of the community. A liar can still say the truth sometimes. A liar still deserves to get justice if his case is proven fair. This Greg Shahade is making the same discourse a fanatical evangelical preacher would make to disqualify a rape victim based on her promiscuity.

      By the way, the guy eventually read the memo one day latter (I guess this makes him a liar too? He said he didn’t have time to read pseudoscience…). In his post, we find this gem:

      So how can we talk about the differences between two large groups of people in a constructive way? My advice: Just don’t do it

      Let’s be honest, have we ever really gained anything from having this discussion?

      I’m not making this up. The guy basically wants to call off any discussion on the topic. Maybe we shouldn’t be taking him seriously about some things.

      • Ratte says:

        Glancing at his critique of the memo, I can’t help but notice this:

        Google seems to be doing just fine with the practices they’ve been using. Maybe if a company like Google, which is one of the biggest and most well known companies in the world, is championing diversity to this degree, there’s some merit behind it?

        The whole reason comments like Damore’s were solicited in the first place was because Google’s diversity efforts were not, in fact, ‘doing just fine.’

        His dismissal of gender differences in chess is baffling as well, seeing as the #1 woman wouldn’t even break the top 20 for men (consistent for juniors) and this has been extensively commented on elsewhere. Glancing at Wikipedia, I also notice that (as elsewhere) countries who are generally worse in terms of women’s rights do much better than expected. Qatar, for example, is the #3 ranked country for women.

  6. ahartntkn says:

    One note on the Mickey Mouse going into the public domain; there’s a huge asterisk there. Mickey as the character which appeared in Steam Boat Willy is going into the public domain. The Mickey with red shorts and the iconic three-circle silhouette is not going into the public domain. The image and name of Mickey Mouse are also protected by trademark on top of whatever copyrights, and that can’t expire so long as it maintains an association with Disney. Something similar in the past happened with Tarzan, where the work itself was protected by copyright, but the name was protected by trademark, so you could make all the derivative works you wanted, but couldn’t use the name without permission. The most significant work I know of that’s going into the public domain soon, without any caveats, is Winnie the Pooh, so there’s that.

    • Brett says:

      That makes a lot more sense. I was wondering why Disney was being sanguine on that, since even if they rarely use the classical characters for anything other than merchandise and the amusement parks, it still has brand value and importance there.

    • BBA says:

      I mentioned this last time the topic came up: trademark and copyright have different scopes, and there was a case about Peter Rabbit that set out the difference. Warne & Co, the original publisher, could certainly trademark the cover picture of Peter Rabbit for use on its books, but it couldn’t sue other publishers for reprinting the public domain book, only for the possibility of confusing their edition with Warne’s.

    • John Schilling says:

      but the name was protected by trademark, so you could make all the derivative works you wanted, but couldn’t use the name without permission

      IANAL, but I’m pretty sure that’s not the case. You can use the name, as long as you’re not using the name in a way that is intended to confuse people as to whether or not you are Disney. Trademark cannot absolutely monopolize a name, except possibly the name of Disney itself (and maybe not even that if your given name is Joe Disney and you’re selling e.g. weapons or trash-disposal services).

      The bigger obstacle would be that putting Steamboat Willie in the public domain only gives you Steamboat Willie’s version of Mickey Mouse. Who looks somewhat different than the current version of the character, had a wife or girlfriend named Minnie but not the rest of the supporting cast, etc, and you’re not allowed to make derivative versions that incorporate any of the subsequent additions or visualizations.

      • pdbarnlsey says:

        My lawyer days are behind me, and in any case relate to Australian law, but I think you’re conflating passing off and trademark. Trademark ought to deal only with actually misleading conduct, but in practice does build a pretty significant moat around the trademarked property unless it is challenged on distinctiveness grounds, which is not a precondition for registration in the first place.

        A range of other legal mechanisms directly address misleading marketing – but perhaps the situation in the US is different.

    • Steve Sailer says:

      Disney continues to invest a lot of money in Mickey Mouse, so I don’t have a problem with their keeping intellectual property rights in their creation. In contrast, F. Scott Fitzgerald’s estate isn’t investing in The Great Gatsby (published 1925), so it would seem reasonable for that book to enter the public domain.

    • JPNunez says:

      I read the article and still harbor doubts Steamboat Willie will be allowed to go into public domain.

      It is still 2018, Disney has a few years to decide to move their asses to protect one their most iconic property. Let’s say 80% of confidence they lobby for it at some point. Sure, maybe right now it looks dubious, but that’s only because there’s still a lot of time to act.

    • sandoratthezoo says:

      So my kid watches a lot of Disney movies. I notice that the current Disney “production company brand thing” is a short clip of Steamboat Willy (Mickey steering and whistling). That didn’t used to be the trademark that Disney used, was it?

      I hypothesize that Disney is making sure that when Steamboat Willy goes into public domain, they have as weaponizable a trademark as possible. By making it an actual clip, including sound, rather than just a still or a logo, they can shut down more uses of Steamboat Willy under trademark law than they otherwise would.

      (I am not a lawyer, this is completely speculative and I’m not 100% sure trademark law actually works that way.)

      • Edward Scizorhands says:

        I’ve had a lawyer tell me the same thing. He also told me that Mickey Mouse isn’t trade marked, only the very very specific poses that Disney submitted as trademarks. Overall I’m not sure how much I trust that guy.

  7. Ilya Shpitser says:

    That AI bias article was awful. The author is correct that statistical bias is different from discriminatory bias — but in the kinds of settings propublica is talking about we don’t want the latter.

    I happen to have gone to a talk by the author of the propublica expose in question, and I can assure you, she knows what she is talking about, and surely understands the difference between these two types of bias.

    Journalists aren’t misleading people, they are correctly pointing out regression models can encode awful biases in our society. And other ideas are needed than regression models.

    Simple example: regressions will fail to parole folks with big criminal records. But in places like Baltimore it’s very easy to get a long criminal record for no better reason than cops loving to hassle African Americans even if they are doing nothing wrong.

    • Evan Þ says:

      In that case, though, wouldn’t better AI notice that “black Baltimorians with criminal records” are outliers and should be disaggregated from the group “people with criminal records”?

      • Montfort says:

        The AI won’t notice (or won’t notice as much as it “should”), because more-or-less-law-abiding black Baltimoreans will continue to be hassled if paroled, thus increasing their risk of reoffense (even though, in the hypothetical, the reoffense is bogus).

        In theory, a sufficiently competent human might be able to account for this by accepting higher reoffense rates for black Baltimoreans than other demographics; I’m not sure if they do, or how we would check if they’re allowing the correct amount of slack.

        • Symmetric says:

          This isn’t a problem with the algorithm, it’s a problem with the system described by the algorithm, and/or the metric being targeted.

          If you really want to predict which individuals are going to be arrested again, then an unbiased algorithm will (and should) measure any bias in the police’s own processes that affect the arrest rate, and use that to make an accurate prediction.

          If you change the target to “chance of violent reoffending” or “chance of reoffending excluding trivial nonviolent charges”, then your algorithm should be able to pick up the delta between these two probabilities.

          And furthermore, if we run a number of different classifiers, or pick a regression algorithm that outputs the weights on a decision tree used to produce predictions, then we should be able to see what factor is contributing to the disproportionate number of blacks in the population of reoffenders. For example in this case given all the numbers, the classifier could output the skewed arrest-rate/crime-rate, and if those weights were published, then it would be much easier to spot this kind of bias in the underlying system described by the algorithm.

      • Ilya Shpitser says:

        A regression model, of the type Northpointe uses, learns an optimal mapping from inputs to outputs in a given class. It can’t notice anything.

        Anyways, here’s our first take on the problem:

        • Ghillie Dhu says:

          In discussing the extent to which a particular approach is “fair,” we believe the gold standard is human intuition. That is, we consider an approach inappropriate if it leads to counter-intuitive conclusions in examples.

          I think this might be the root cause of the pushback you’re receiving; color me skeptical that there’s any approach for which no examples produce counter-intuitive conclusions.

      • Tenacious D says:

        Better AI or a jury of their peers. This article describes a study comparing COMPAS with a wisdom-of-crowds approach using people from Mechanical Turk to estimate risk to reoffend:

        The humans did slightly better.

        (On the topic of the wisdom of crowds–and the prev SSC post–I wonder if anyone has ever tried a dating app based on that concept? Like, users could be required to suggest some matches between profiles from a distant city to access more of their own matches).

        • Conrad Honcho says:

          Crowd-sourced dating suggestions is a really neat idea.

        • [Thing] says:

          (On the topic of the wisdom of crowds–and the prev SSC post–I wonder if anyone has ever tried a dating app based on that concept? Like, users could be required to suggest some matches between profiles from a distant city to access more of their own matches).

          I was very skeptical of Luna (especially the cryptocurrency aspect), but if they implemented this feature, it would be really interesting. You could reward users with stars for correctly guessing which profiles other users would want to see (as determined by how the latter users rate their matches). That sounds fun! Who wouldn’t want to play a little matchmaker for random internet strangers? I bet some people would sign up for that even if they weren’t looking to date, just to play it as a game. You could also have leaderboards for top matchmakers, and an option for satisfied couples to send thank-you notes to all the users who matched them. I suppose there could be issues with perverse incentives, privacy, or whatever, but I don’t see why it would be any worse than baseline for internet dating.

          • Tenacious D says:

            “leaderboards for top matchmakers”

            I can definitely see some people getting really into that!

          • Randy M says:

            I wonder if that could even work as a third party app that used a bot or something to scour eharmony or match for profiles and showed them to players semi-anonymously to match.

          • dndnrsn says:

            @Tenacious D

            If you get a streak, do you get a supply package, or an airstrike, or something?

          • Randy M says:

            You have the ability to award flowers, nice perfume, or, at high levels, a string quartet to the next pair you set up on a date.

            Of course, matches occurring as a result of a match-streak reward don’t count towards your next reward, but they do help you level up faster!

          • dndnrsn says:

            This sounds way cooler than Luna. The waiter walks up to the table. The couple sitting at it, a year after their first date their, out for their anniversary, look up. “Excuse me, but I have a bottle of champagne for you, courtesy of killerdawg42069. He thanks you for continuing his streak.”

          • Tenacious D says:

            @ dndnrsn and Randy:

            I like your streak reward idea!

    • Scott Alexander says:

      I haven’t read the ProPublica piece, but most people I’ve seen write about this, or talked to about this, seem to have the misunderstanding that the authors think people have.

      • pdbarnlsey says:

        My sense is that there’s a fairly wilful attempt to conflate the two, or even a refusal to recognise any meaningful distinction between “this algorithm leads to worse outcomes for [protected group] than for others” and “this algorithm unfairly penalises members of [protected group]”.

        • wintermute92 says:

          I’ve linked a number of people to Scott’s Framing for Heat Instead of Light as a tech-agnostic explanation of what’s wrong with these stories. It’s often helpful just to distill down the insight that “ties into correlates with race” and “creates new action based on race” are vastly different events.

          And yes, I agree that many people actively fail to employ this distinction. I haven’t read ProPublica on COMPAS, but their series on bias in auto insurance made the error – even after acknowledging that case and claiming to avoid it!

          • Sniffnoy says:

            It’s often helpful just to distill down the insight that “ties into correlates with race” and “creates new action based on race” are vastly different events.

            In some cases this can be thought of as “introduces error” vs “propagates racism error”. (We could also I suppose have a category for “amplifies”.) It’s very hard (usually impossible!) to do better than propagating error…

        • Ilya Shpitser says:

          Yup, a lot of people who write fairness papers in ML are ?confused about/aware of? this point, also. Luckily, this is changing.

        • ADifferentAnonymous says:

          The whole concept of disparate impact is about recognizing facially-neutral rules with worse outcomes for protected groups as unfair. There’s a principled version of this position, but I think it’s fair to say popular articles aren’t going to a ton of effort to explain “this is biased in the disparate-impact way, which is different from classic bias but still important”.

          • Brad says:

            It is worth noting that the disparate impact legal test is part of burden shifting frameworks, rather than being the sole element needed to be proved to win a case under the civil rights laws. There seems to be some willful misstatements of how the law works in this area (when pushed back on, often defended on the grounds that it is de facto sufficient without any evidence for that claim).

          • The Nybbler says:

            If protected groups differ from other groups in ways besides the protected characteristic, disparate impact may not be unfair.

          • ADifferentAnonymous says:

            Nybbler: You’re right, and I realize the way I phrased that made it sound like I meant disparate impact is always unfair. It’s not. But it’s sometimes unfair, even in some cases where the rules don’t seem overtly unfair.

            In the case of COMPAS, the actual effect depends on how the judge uses the score. But if the algorithm’s risk score were the sole factor in parole decisions, I do think it would be unfair, despite being accurate.

            Why? Well, do we want parole decisions to be based solely on recidivism chance? Maybe we want it to accomplish other goals, like incentivising good behavior in prison. And most people feel that justice decisions should be based on things people have done, rather than things we have reason to believe they’re going to do (as I learned anew when I brought up Robin Hanson’s private justice proposal last open thread). Throwing out these considerations because we have an accurate algorithm for recidivism chance would be wrong, and I suspect wrong to the disproportionate detriment of black people.

          • whales says:

            All else aside, this whole discussion is a good example of why I tried to write up some exposition on unfair outcomes from fair tests, which I find myself linking over and over. An algorithm that “fairly” penalizes a group in terms of the point estimates it produces will be unfair (in a mathematical sense that’s sometimes a “fairness” we care more about) as an aggregate decision process.

      • Ilya Shpitser says:

        By “most people” do you mean “random people on the street”? If so, of course they will misunderstand, random people on the street don’t know what EITHER type of bias even is. People on the street will also think that quantum computers can solve NP-complete problems in polynomial time, regardless of how well someone like Scott Aaronson might write a user-friendly explanation that this isn’t the case.

        But I don’t think it would then be fair for a physicist-turned-data-scientist to then write a post about how Scott Aaronson is misleading people.

      • matthewravery says:

        The problem is with the language. Stucchio and Mahapatra think that “people” understand bias the way they do. People don’t. People understand it the way the journalists doing reporting do. (Source: Am a statistician who has to talk to people that aren’t all the time about things like statistical bias.) SSC readers may be more literate in statistics jargon, so that might not apply here, but for the general populace, including plenty of people with graduate degrees in technical fields, the definition they’ll think of is the colloquial one.

        As for the substance of the article, I’m nonplussed. Their argument in its entirety is that, “An algorithm that minimizes loss functions succeed at minimizing the loss function it was given, therefore it’s a good algorithm.” This is a tautology. The criticism with which they need to engage is, “The algorithm may not be minimizing the correct loss function.”

        • J Milne says:

          The ‘common use bias’ vs ‘statisticians bias’ is a red herring, journalists obviously aren’t intending to use ‘bias’ in the statistical sense.

          • matthewravery says:

            Yeah, that’s my point. The article Scott linked says the following:

            “The media is misleading people. When an ordinary person uses the term “biased,” they think this means that incorrect decisions are made — that a lender systematically refuses loans to blacks who would otherwise repay them. When the media uses the term “bias”, they mean something very different — a lender systematically failing to issue loans to black people regardless of whether or not they would pay them back.”

            This is of course absurd. When “an ordinary person” uses the term “biased”, they mean it in exactly the same way the “media” is using it here.

          • J Milne says:

            Sorry I should have been more clear that I was agreeing with your point, but just suspected that it was deliberate on the author’s part to pretend that the definition of ‘bias’ was the source of the issue.

          • tscharf says:


            bias: prejudice in favor of or against one thing, person, or group compared with another, usually in a way considered to be unfair.

            discrimination: the unjust or prejudicial treatment of different categories of people or things, especially on the grounds of race, age, or sex.

          • Aapje says:

            I expect that most common people don’t even understand the distinction and believe that either both are the case or neither.

      • J Milne says:

        It would still be better to not link articles that claim that all journalists talking about AI bias are peddling junk. My understanding is that people are concerned about situations where:

        There’s a difference in groups A and B due to some upstream factors, e.g. group A tends to have lower socio-economic status; and instead of learning to discriminate using the (useful) socio-economic status variable, the AI learns to simply distinguish using the (also useful, but more obviously unfair) labels A and B.

        • The Nybbler says:

          And that’s _not_ what’s happening here. The labels A and B aren’t even given to the algorithm.

          • J Milne says:

            Are you talking about the ProPublica piece? Because they Jacobite piece isn’t just responding to that:

            COMPAS is not the only computer algorithm that has been accused by journalists of being biased. Google adwords was accused of unfairly showing ads for high paying jobs to men. Financial algorithms for predicting loan repayment are regularly accused of being racist. The media regularly decries algorithms as “biased,” “sexist,” and “racist.”

            Though as an aside, you could hide the labels A and B and still run into trouble, if your AI has access to variables that correlate with being A or B but still don’t directly influence the measure actually targeted. E.g. ‘likes hiphop music’ will still be a useful variable even if it’s not the ‘correct’ one.

        • Garrett says:

          I’m not sure if it’s relevant to your argument, but this gets into another aspect of justice which I suspect is more complicated.

          There are two questions here which might need to be separated:

          1) What elements of socioeconomic status, racial discrimination, etc., led this person to the point where they were convicted of the given crime?

          2) Give the person in front of them, what is the likelihood that they will [violently] re-offend.

          I think it’s possible that the answer to #2 may be calculated with a reasonable degree of accuracy/precision, but people are mostly worried about #1 and don’t have a good way to address that. So they attempt to address #2 to address #1 because it’s a more tractable problem

        • Steve Sailer says:

          I would distinguish between trying to prevent crime before it happens and trying to punish crime after it happens. I would be more sympathetic to using race, as in “racial profiling,” to prevent crime. I’m wary of using race as a factor in determining punishment for crime.

          The problem is that the two issues overlap in questions like determining when to let a prisoner out of prison, which has both a punishment for a past infraction aspect and a prevention of future crime aspect.

          That, say, an Asian Indian-American in prison for a violent crime would be less likely statistically to commit another violent crime than an African-American with the same history of violence would suggest that it would be a statistically more efficient use of prison resources to let the Indian out before the black.

          On the other hand, the idea of punishing one individual more than another individual due to racial differences seems distasteful and not in sync with the best principles of Anglo-American jurisprudence. I don’t really know how to resolve the two feelings.

          • Matt M says:

            I feel like modern parole rules are a nonsensical black box that we’d be better off eliminating entirely.

            Have standard sentences that are the same for everyone, based on the crime. You cannot get out early for “good behavior” or whatever, and once you’re out, you’re out, none of this “check in with your officer and take three drug tests a week” nonsense. Bad behavior (escape attempts, assaulting a guard, etc.) carries standard additions to your sentence. Good behavior is the expectation and carries no “reward.” Bad behavior carries extra punishment. No elaborate system for letting some people out early with a ton of strings attached while others continue to languish in jail.

            What would be so wrong with this?

          • keranih says:

            @Matt –

            Eh. I think that it’s reasonable to have flex in the system. Firstly, because there is probably value in ‘stepping down’ confinement/oversight on severe criminals to help guard against relapse. Prove that one can be on good behavior with a little freedom and one gets a bit more freedom.

            Secondly, because situations do change and a society might end up with more prisoners (or fewer facilities) than they had planned on, and having an alternative correction scheme already coded in can prevent log jams and rash decisions.

          • John Schilling says:

            Thirdly, because plea-bargaining and prosecutorial discretion are real things, and if prosecutors and judges can’t arrange for the guy convicted of a sympathetic manslaughter to spend only five years in prison, they’ll arrange for him to be convicted of negligent homicide or whatever else it is that results in the desired lesser punishment.

            Fourthly, because even if you do away with plea bargains juries are still a thing, and so is Google, and if there’s a mandatory twenty-year sentence for manslaughter then juries will know that even though you tell them not to. And then the guy who committed a sympathetic manslaughter won’t be convicted at all.

            Fifthly, because the Bastille was in fact stormed. People don’t want everyone who commits the same crime to do the same time, and they won’t tolerate it. People want the guy who in a fit of passion shoots his daughter’s rapist to spend less time in prison than the guy who in a fit of passion shoots his more successful business rival, and one way or another that’s what they are going to get. Whatever you put between them and that objective, is going to get broken.

          • albatross11 says:


            I see your point, but I think something like that is inevitable if you do any statistical forecasting to decide which prisoners to release early. You’re basically going to put each prisoner who’s up for parole into some kind of bin based on people like them–people my age, sex, education level, people who committed the same crime I did, etc.–and then, you’re going to judge how likely I am to commit future crimes based on what other people who were similar to me did.

          • Steve Sailer says:

            One possible angle on this difficult question is the matter of time: racial profiling is more reasonable under time pressure.

            For example, your daughter’s car breaks down at night in a bad neighborhood and her phone is dead. She can walk either north or south to get help. North of her she can see six African-American teens loitering on the corner, while south of her she can see six Laotian-American teens loitering on the corner. Which way should she go? Is she wrong to use race as a major input in her decision-making?

            In contrast, a parole board has time to research in some depth into individual characteristics, so putting a high value on race would seem more questionable.

      • Douglas Knight says:

        I have read the ProPublica piece and I assumed that it was ignorance. But if Ilya insists that it is malice, I will believe him.

      • Ilya Shpitser says:

        That’s a good question, actually. I think perhaps folks victimized by this don’t have the resources, and aren’t aware enough to contact the ACLU. This absence of lawsuits might change, actually…

        • Aapje says:

          Or they may fear that they will lose (a belief in institutional racism/sexism implies a belief that a judge will not be fair) & believe more in ‘direct action.’

    • Steve Sailer says:

      I went through local newspaper reports for 2017 homicide totals in the 51 biggest cities in the country and Baltimore had the second highest homicide rate (after St. Louis) last year, so apparently somebody is doing something wrong in Baltimore:

      • Thomas Jørgensen says:

        Unusually racist policing could really easily create a crime-wave. Both from people with high melanin going “I´ve been arrested before, I´ll get arrested again. If I am going to do the time, might as well actually do the crime” and robbing the nearest bank when they cant get a job due to their bogus criminal record.
        And also from the police arresting the wrong people, or failing to catch career criminals simply because they have made themselves so very unpopular that noone tells them anything, even when they are looking for awful people.

        • vV_Vv says:

          This could also apply if the police is unjustly perceived as racist.

        • Steve Sailer says:

          Have you looked at pictures of Baltimore City’s elected leadership and its police force?

          • The Nybbler says:

            That they are black does not mean they are not racist against blacks.

          • Hyzenthlay says:

            That they are black does not mean they are not racist against blacks.

            I think black people can absolutely have biases against other black people. In general, people are quite capable of having prejudices against their own group; I’ve encountered some hardcore SJ white people with pretty strong anti-white bias.

            So yeah, a cop being black doesn’t mean he’s not racist, but I do think that introduces a level of nuance and complexity that tends to get brushed under the rug in the “racist white cops murdering blacks in the name of white supremacy” narrative. Sometimes people will try to spin it as “well, those anti-black black cops are just tools of white supremacy; they’re puppets being manipulated by white people, so it’s still the doing of white people.” But I think this is a pretty transparently desperate tactic. Racism, and prejudice in general, are complex psychological issues that can’t generally be reduced to one group oppressing everyone else.

      • Nancy Lebovitz says:

        I’ve wondered whether random policing is worse than no policing. People are less likely to think that obeying the rules/living decently will do them any good, and there’s no chance for local policing to develop.

    • Anon. says:

      I highly recommend this comment from the subreddit. It’s clear that the author either does now know what she is talking about or set out to deliberately mislead.

      Also, Inherent Trade-Offs in the Fair Determination of Risk Scores.

      • Ilya Shpitser says:

        > According to their own data, black offenders and white offenders with a given risk score were equally likely to be convicted of another offense within two years of release. But that wasn’t consistent with the narrative they wanted to tell, so they buried that part.

        This person is confused, this sort of equality is completely irrelevant to whether discrimination is occurring (because conditional distributions tell you nothing about discrimination in general). This person, in addition to being confused, has a political axe to grind, as well.

        Now, the fact is, there _is_ a conversation taking place within the algorithmic fairness community on how to formalize biases of interest properly. However, one thing that probably is not going to help is ask_the_Donald level hot takes.

        • Anon. says:

          The entire premise of the propublica piece is about conditional distributions. Their big finding was that:

          The formula was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants.

          • Ilya Shpitser says:

            The entire premise of the propublica article is that Northpointe is selling proprietary regression models which people use in sentencing and parole hearings now. How sure are we that these regression models aren’t discriminating? I don’t think Northpointe thought about this very hard.

            There is a huge conversation now in statistics and ML about what properties, precisely, a statistical procedure should have to avoid bias, of the discrimination variety. Lots of papers, and so on. I am going to the “ethics and AI” conference colocated with AAAI this year, and (aside from alignment and so on), addressing these types of discriminatory biases is a big topic at this conference:


            All due to articles like the propublica one. The entire premise of the author’s talk, by the way, is that academics were falling down on the job, so journalists like her had to learn a bit of data science to start the conversation. Academics are listening now, and working on the problem, so her goal was accomplished.

            By the way, we analyzed COMPAS data, and based on what we were able to get our hands on, we concluded that what they are doing is really bad (by our lights). If you want, we can talk about that in detail, or you can just read our paper.

            You are trying to kill the messenger.

          • Anon. says:

            How sure are we that these regression models aren’t discriminating?

            Given that, conditional on risk score, people were about equally likely to reoffend regardless of race…pretty sure?

            I’m not sure about where you are suggesting discrimination even enters into the question here. Are you saying that white people are less likely to be caught if they reoffend, so the equal reoffense rates are hiding discrimination beneath them? If so, that’s hardly a matter to be solved at the level of risk scoring…

          • Ilya Shpitser says:

            I am saying I don’t think you thought very hard about what the legal definition of discrimination is or ought to be. As it happens, folks in law _have_ thought fairly hard about this (similarly to how they thought fairly hard about causation, with their “but for” test). It might be worthwhile for you to read about that stuff. It’s sort of directly relevant.

          • AlephNeil says:

            Anon – I used to agree 100% with that reddit comment you linked to, but after a bit of thought, I think we can meet Shpitser half way. See my comment here.

          • Ilya Shpitser says:

            I think we have to start with a definition of discrimination we both agree on. Once we have that down, it’s just math.

          • Anon. says:

            I suppose the _legal_ definition in the US basically comes down to disparate impact, but that is because they start with HNU as an axiom. Certainly, if we optimize for the legal structure instead of reality we would end up with very different things, I don’t disagree with you on that point. But the legal definition is no different than the ideal reality of the journalists.

          • Ilya Shpitser says:

            The definitions of discrimination that appear in the legal literature are not all based on disparate impact. It’s sort of similarly complicated to causation in law, see David Friedman’s comment in this thread. We use a particular counterfactual one in our paper.

            We could talk about whether it’s sensible (e.g. captures human moral consensus on discrimination) I think it is, and it does.

          • tscharf says:

            “I think we have to start with a definition of discrimination we both agree on.”

            The entire point of the article was that journalism misinterprets the term bias in an (intentionally) inaccurate way. Hand wringing over academic definitions is the motte / bailey doctrine in action as far as this debate goes. The plain reading of journalism’s use of bias and discrimination is one of immoral intent / outcome.

          • Steve Sailer says:


            You seem to be reluctant to spell out what you mean by “bias” and “discrimination” even though you seem offended that other commenters aren’t on the same wavelength as you as to whatever it is you mean. Could you illuminate us by copy and pasting from something else you’ve written what exactly it is you mean?

          • Ilya Shpitser says:

            “Bias” is an overloaded term. There are lots of kinds of bias — I gave some examples of selection bias in data generation, did you read it? There is also “statistical bias,” which is expected error.

            There’s cognitive biases, which LW folks like to think about — the classification project for those started with Francis Bacon, I suppose.

            In epidemiology there is a whole taxonomy of biases — Berkson’s bias, selection bias, confounding bias, and so on. Do you want to get into this? I am not avoiding discussing this stuff, I teach this stuff to undergrads. I just didn’t think a full discussion was relevant here. Although I did notice the author of the article that started this seemed to conflate statistical bias of an estimator and selection bias in the data.

            “Discrimination” is not so easy to define. Which I think is one of the main takeaways here. If someone thinks it’s easy, I think that’s an indication they have not thought about the problem very hard. This is mirroring an earlier difficulty people had defining what “actual causation” means. It seems like this should also be easy, but it’s really really not.

            We have a definition in our paper linked to causal paths. There are about a million papers that do other definitions. Obviously, we think ours is better motivated (in part because of a link to existing legal definitions), but there’s room for argument.

            I think it’s sort of ironic you accuse me of reluctance where I literally linked a paper I wrote spelling this out, and spent all of yesterday trying to discuss it. Are you reading the same thread as me?

            I could have said “we define discrimination as the presence of a path-specific effect. Don’t know what that is? Here are three papers on this, but trust me it’s really good.” But I don’t think that would be a very effective way for me to communicate.
            I did try to describe informally, in a few places.

            I wouldn’t say I am offended, but it is true that there is a certain class of slatestar poster that I don’t really wish to engage, that I don’t think contribute anything interesting, and that I wish went away.

          • moscanarius says:

            Although I did notice the author of the article that started this seemed to conflate statistical bias of an estimator and selection bias in the data.

            Well, that’s what the folks here were complaining about. The journalists were conflating some concepts and this is misleading to the public.

            Summing that with your first comment:

            I happen to have gone to a talk by the author of the propublica expose in question, and I can assure you, she knows what she is talking about, and surely understands the difference between these two types of bias.

            If you are talking about the same person (sorry I’m getting confused), then (1) either there is a contradiction between the statements, or (2) she made a mistake when writing, for candid reasons, or (3) she made a mistake when writing, for malicious reasons. Elsewhere in the thread, you seem to be defending (2) and specially rebuking (3). That’s fine, we can believe it was candid; even the Stucchio piece was not saying they were disonest ill-willed people.

          • rlms says:

            I think “the article that started this” is Stucchio’s.

          • moscanarius says:


            I don’t know, he mentioned in another comment that he thinks the propublica authors discussed poorly. And it makes more sense that “the article that started this” is the propublica one, at least the way I read.

            In fact, I think this most of this confusion sums up to Ilya objecting to Stucchio saying the propublica piece was “misleading”, on the grounds that he thinks “misleading” implies “bad intentions”, and he believes the original authors are not up to evil (because he’s met one of the authors).

          • Ilya Shpitser says:

            Hi I meant the person replying here is confused, based on their first comment.

            ProPublica folks probably _also_ are confused. And just to be clear — I don’t blame people for being confused, because what we are talking about is difficult. I am not playing gotcha. The fact that we shouldn’t criticize people for trying to address hard topics, even if they get things wrong, is precisely what my entire point is.

          • moscanarius says:

            The fact that we shouldn’t criticize people for trying to address hard topics, even if they get things wrong, is precisely what my entire point is.

            Then in all fairness, why not to state it clearly and stick to it? Because I’m not the only one who noticed you shooting to all sides on this thread.

            I didn’t read Stucchio as criticizing the propublica people for “trying to adress a hard topic”, but for making statements Stucchio thought could mislead the public. Sure that’s allowed? If their criticism is wrong (as you think they are) and the propublica people are right (which you don’t think? You have not been very clear), then of course it is fair for you to defend the propublica piece. But otherwise, your only criticism is that you think Stucchio was too heavy-handed and not super precise herself. No need complicate the situation.

            Let’s see:

            1. You think the propublica piece was good because it brought attention to an important problem, even if some people could take the wrong conclusions.

            2. You think despite being good, the propublica piece made some poor characterizations (yes? probably? not? you have not been clear). You acknowledge that this may lead to people getting the wrong ideas.

            3. You think the Stucchio piece in response was bad because:

            3a. It says the journalists are misleading people, and you object to this characterization because you don’t believe the journalists have bad intentions.

            3b. It says the journalists are ignorant of the key concepts, and you object to this because you know one of the propublica journalists and can vouch for her knowledge and honesty.

            3c. It is too harsh a criticism, and may disincentivize other journalists from writing about important topics. That’s killing the messenger, somehow.

            3d. It is an irrelevant criticism, as every text can be misleading, and it doesn’t matter very much if the common people are getting the wrong ideas, since the scientists that matter got the right one.

            3e. You think the Jacobite piece is advancing the wrong arguments itself.

            If that’s it, here’s my take:

            3a. “Misleading” doesn’t always imply bad intent. I think the only part of Stucchio’s piece that could be seen as pushing to the assumption of bad faith are the last paragraphs. And notice that the piece does not refer only to the propublica article. Overall, I think they mean “misleading” in the (very common) sense of “phrased in a way that induces the audience to error, despite intentions on the contrary”.

            3b. There are 4 authors in the propublica piece; you can vouch one. Can you be sure this friend of yours was involved in all of the writing? Can you be sure the other 3 are as qualified? Can you vouch that every author, editor, and advertiser involved was as candid?

            3c. I highly doubt the disincentive would be so strong. Jacobite mag is not exactly the backbone of the press; its readers are not so influentional. Besides, harsh criticism is available for everything journalists write. I think you are overestimating the critics power here.

            3d. The fact that all texts can be misleading is no justification to deny criticism of any one specific text for being misleading. I disagree that if the Smart People got it right there is no reason to point out what could mislead the other folks. ProPublica is not an academic publication, supposed to b read by scientists with the proper qualifications. You may not want to spend your time doing this, but what’s so wrong about other people doing it?

            3e. That was not your original criticism, though.

            EDIT: I see that Stucchio has said in a comment here that she thinks the propublica piece was click-motivated. That adds more weight to your criticism, even though it does not appear in the original piece.

          • Ilya Shpitser says:


            1. Agreed.

            2. Yes, I think propublica was not thinking about “machine bias” (in their title) in the right way, and has been criticized for this, by lots of of folks. That is ok. This is hard to get right. I think they were thinking about this problem in the wrong sort of way without an intent to mislead.

            3a. Agreed.

            3b. Journalists who work on science reporting may sometimes be bad (e.g. innumerate). This particular person whose talk I went to did not strike me as innumerate. She struct me as well-read, and well-intentioned. This does _not_ of course imply that she ought to be able to crack the “machine bias” problem. Numeracy is necessary but not sufficient. I don’t view cracking the problem as her job in the ecosystem.

            3c. Agreed (well, on the margin).

            3d. Agreed. Also there is a correlation/causation issue here. Just because a layperson read A and got confused about B does not imply reading A is what did it! It’s possible (and in fact is what usually happens) the layperson lacks the background to get the proper takeaway from A, in which case we have to decide how to attribute their bad takeaway to their lack of background vs A itself.

            It is possible to communicate to a broad audience badly, of course, and we should try to do it well. So a criticism I would accept is “you did popular science badly, because of X,Y,Z.” But even here, the issue is usually incompetence rather than malice.

            Folks who tend to mislead maliciously about science topics are, by my lights, more often academics themselves (for career reasons), rather than journalists. Of course journalistic fraud is also a thing.

            3e. I may or may not agree. I noticed the author said some things that really did not seem right to me, though. But here, in the comments.

            I also noticed that my exchange with the author ended with the author saying I am trying to bury with obscurantism.

            You know, I am pretty relaxed about disagreement. I disagree with lots of people about lots of things. Many of these people are my friends and colleagues. I don’t think we need to posit bad intent every time there is intractable disagreement on the internet.

          • moscanarius says:


            That’s OK, my doubts are answered. And I agree that the problems with any piece are most likely to come from incompetence than malice. It’s just that I don’t see much reason for the beef with the word “misleading”, or for going around throwing side arguments when the main point could be more clearly stated. Sorry, but it looks like derailing, even when it is not the intention. Well, that may be just my preference, you are allowed to have yours.

            You know, I am pretty relaxed about disagreement. I disagree with lots of people about lots of things.

            Maybe I am bad about tone reading, but your first comments did not struck me as “relaxed”, and I am sure I was not the only one to think this. Maybe much of the latter misunderstandings followed from there.

            I don’t think we need to posit bad intent every time there is intractable disagreement on the internet.

            I agree, but who’s this directed at? Well, nevermind.

          • But even here, the issue is usually incompetence rather than malice.

            I know nothing much about this particular issue, having read neither the original article nor the critique. But I think the usual problem is bias, not incompetence or malice. The journalist sees an academic article which can be interpreted in a way that fits the conclusion he wants to push (or which makes a particularly striking story). Because he agrees with that conclusion he is not inclined to look very carefully to see if that’s really what the article implies, still less to look carefully to see if the article itself has problems. So he writes up a popular piece that presents the article as supporting the conclusion he wants, with rhetoric designed to make the case look as strong as possible.

            Folks who tend to mislead maliciously about science topics are, by my lights, more often academics themselves (for career reasons), rather than journalists.

            I think deliberate dishonesty is probably more common there, if only because the journalist has more wiggle room to interpret things the way he wants due to ignorance. On the other hand, the journalist may be being dishonest not in the facts he reports but in how he reports them. For example (from my blog).

            An example of the scientist problem (from my blog).

          • albatross11 says:

            I think this is a very general bias that applies as much to readers as to journalists. If I am making a case for some idea that you already accept, or want to accept, then in general I can make a really weak argument, produce a graph that doesn’t really agree with what I’m saying as evidence, maybe link to a couple papers or some data that doesn’t really support my argument, and a distressing number of people will nod their heads and say “yep, case closed.”

            I also think it’s likely to be a mix of journalist fooling themselves and journalists convincing themselves that they’re telling the truth despite some noisy data and a somewhat confusing set of facts. I don’t have strong evidence here, but as best I can tell most journalists don’t want to lie even for a good cause. (There are journalists who lie or make up stories or just flat write propaganda, but I think most journalists don’t want to do that kind of thing.)

            I suspect this is rather similar to what you can see from the replication crisis in the social sciences–rarely, there’s some actual fraud (I think some of the original priming research was faked, for example), but far more often there were social scientists convincing themselves they’d discovered something interesting out of some very messy data and complicated experiments, using sophisticated statistical tools that they maybe didn’t 100% understand. And I suspect they found it easier to convince themselves of results that re-enforced their existing political and social beliefs, and that would also be appealing to their advisors and colleagues and tenure committees and such.

            This phenomenon makes me uneasy when I see news reporting on a culture-war-heavy story. It’s not that I think there are all that many journalists who are willing to lie, but there do demonstrably seem to be a fair number who will omit a lot of information that doesn’t fit the story they want to tell, or who will structure their stories so that the comfortable narrative is told in the headline and the first couple paragraphs of the story, while the facts that complicate or undermine the narrative only appear in the last paragraph or two of the story. It’s one reason I wish there were more right-leaning news sources that I actually thought were trying to do real reporting rather than propaganda–someone whose journalists were mostly Republicans would have a different set of blind spots and expectations and comfortable narratives, and maybe we’d get a better picture of reality as a result.

          • tscharf says:

            Journalism isn’t homogeneous. My observations have been that environmentalism and race relations are areas where monocultures now dominate. The biggest form of bias here is selection bias, consistently telling only one side of the story. This is very evident if one examines which anecdotal stories are reported.

        • AlephNeil says:

          To my mind, being ‘discriminatory’ would mean treating people who were otherwise alike differently based on their race, which in this context would mean that given a black and a white person with equal recidivism risk, the former would have a higher expected risk score than the latter.

          Now, if the risk scores perfectly reflected recidivism risk then (i) they would not be ‘discriminatory’ in the above sense and (ii) it would also be true that black and white people with the same score had the same recidivism risk.

          But let’s suppose instead that score = recidivism risk plus an error term (which has the same distribution for everyone). Then this is still “non-discriminatory” in the above sense. However, a black person with a given score is now *more* likely to recidivize than a white person with the same score.

          Therefore, if a black person with a given score is equally likely to recidivize as a white person with the same score, then the system *is* discriminatory in the above sense.


          (However, the fairness criterion ProPublica seems to be interested in – whether the proportion of non-recidivists who get high scores is the same between black and white defendants – has even less to do with “discrimination” as I’ve defined it above. It’s also a less intuitive thing to look at than ‘whether people with the same risk scores have the same reoffending rates’. It seems like the only reason to look at it at all is to generate propaganda for consumption by the mass public.)

          • peterispaikens says:

            It all comes down to the old issue about defining fairness and equality – you can have equality of opportunity (or a fairness of the process) or equality of outcomes (or a fairness of the results), but you can’t have both, they are inherently incompatible for any substantially different demographics. Given equal opportunity, you’ll have different expected outcomes and if you want equal expected outcomes, you must grant unequal opportunities; so also in this case, if you want a system to have results according to the true risk of recidivism, it’ll have to take the protected criterion (or a correlate) into account; and if you will ensure that the process is race-blind, it’ll have unequal results.

            Thus, every system will be unfair in one way or another, and you can only pick the side or some tradeoff in the middle. You *can* argue that the side/tradeoff should get chosen differently, but simply criticizing a system by saying that it’s unfair by one definition or the other is silly, it’s like criticizing water for the (very inconvenient) property of making stuff wet.

          • AlephNeil says:

            Not really. What I’m saying is that there are two separate notions of “race-blindness” here that conflict with each other:

            (1) People of different races with same risk score should have same risk of recidivism.

            (2) People of different races with same risk of recidivism should (on average) have same risk score. [Better: they should have the same probability distributions of possible risk scores.]

            The COMPAS algorithm is claimed to have property (1), but I would argue that

            – Property (2) is intuitively what we mean by “non-discrimination”
            – Perhaps surprisingly, (1) and (2) are not logically equivalent.
            – If property (2) is true then property (1) will fail in a way that looks biased against whites.
            – If property (1) is true then property (2) will fail in a way that looks biased against blacks. And this is more important because (2) is the ‘true meaning’ of non-discrimination (I claim).

            Note: if in fact black people of a given risk score turn out to have a higher (not equal) risk of recidivism then it’s possible that property (2) holds after all.

            (Finally, I will remark that this distinction between (1) and (2) has nothing to do with ProPublica’s findings here which are misleading and disingenuous for exactly the reasons Northpointe identified in their response.)

          • The Nybbler says:

            People of different races with same risk of recidivism should (on average) have same risk score.

            How do we measure this, even in principal? We don’t know the true risk of recidivism of a given individual, not even after the fact, and there doesn’t seem to be a natural way to bucket them.

          • Ghillie Dhu says:

            I’m having difficulty working out why (1) != (2), let alone why adhering to one produces the appearance of bias on the other in the specific directions you specify; would you be willing to walk through the logic?

          • AlephNeil says:

            To The Nybbler:

            That’s a critical question. It’s a ‘leap of faith’ to assume that there’s a single right answer (at a given time) to the question of what a person’s true recidivism risk is.

            One extreme but consistent approach is to argue that the people who actually did recidivize have ‘true recidivism risk = 1’ and those who didn’t have ‘true recidivism risk = 0’, and then I think my logic becomes identical to ProPublica’s, doesn’t it?

            The other extreme but consistent approach is to say that the true recividism risk is whatever you get once you calibrate the COMPAS score (modulo bucket error). In this case, there is no “discrimination” (by my definition).

            And between those two extremes I can’t see any *principled* way of defining the ‘true risk’ (unless we’ve just assumed it as a leap of faith).

            I concede: my entire line of thought doesn’t go anywhere useful.

            To Ghillie Dhu:

            At the outset, let’s switch recividism risk for height, so that we can unproblematically talk about normal distributions.

            Imagine height is the sum of two factors A and B. Assume that black people are taller on average. Imagine that when we try to measure height we can only see factor A. Say that A is the person’s “height score” whereas A+B is the true height.

            This will satisfy property (2) as long as the following holds: given A+B, knowing a person’s race tells you no more information about A.

            Is property (1) satisfied (under that same assumption)? Not necessarily, and in fact probably not. For instance, we could have A and B as i.i.d normal variables with one mean m for whites, and a higher mean m’ for blacks. Then having observed A = a, the expected height comes to a + m for whites and a + m’ for blacks.

            Let’s keep A and B as independent normal variables but now say that the mean of B is the same for blacks and whites, while the mean of A is higher for blacks than whites. Then property (1) is satisfied: if a black and a white person have the same “height score” then they have the same expected height. But property (2) now fails: given a white and a black person of the same height, the white person probably has a lower A than the black person.

            To go back to recividism risk “A” would represent everything that goes into a score like COMPAS whereas “A+B” would be “the true recidivism risk”. However, the only two meaningful intepretations of “the true recividism risk” in sight are:

            (i) “true recividism risk” = “the recividism risk conditional on the COMPAS score alone” in which case the whole thing becomes trivial and properties (1) and (2) are equivalent

            (ii) “true recividism risk” = “1 or 0 depending on whether the person actually recidivizes” in which case property (2) becomes: “conditional on whether white and black offenders recividize, they ought to have the same scores” which is the criterion that ProPublica were using.

          • David Speyer says:

            @Ghillie Dhu I’m trying to get an intuitive understanding of this for myself, so let me see if I can explain it to you.

            Let’s choose a non-racial scenario. Let’s say we have 100 rich kids and 100 poor kids interested in coding. 90 of the rich kids go to top colleges, and only 10 of the poor kids do. 50 of the kids in the top school learn to code well, and this is uncorrelated with their race, so the breakdown is 45 rich + 5 poor. Only 10 kids from the bottom school do the same, again uncorrelated, so the breakdown is 1 rich and 9 poor.

            Google hires the 100 students who went to top schools. Google argues position (1): “Our standard has 50% probability of predicting good coders, for both rich and poor students”.

            Google’s critics argue position (2): “14/60 good coders, or 23%, grew up poor, but only 10% of Google’s hires did!”

            If I were to try to say it with no numbers, I would say that the poor, good coders problem is that they are hard to find among the pool of poor, bad coders, while the rich, good coders are relatively easy to distinguish from the rich, bad coders.

          • Ghillie Dhu says:

            I think I’m getting there (thanks for indulging); to check my understanding:

            For criterion (1) there are four subpopulations of interest:
            (a) White high-scorers
            (b) Black high-scorers
            (c) White low-scorers
            (d) Black low-scorers

            Ideally the reoffense rate of (a) should match that of (b) AND the reoffense rate of (c) should match that of (d).

            For criterion (2) the four subpopulations of interest are:
            (i) White reoffenders
            (ii) Black reoffenders
            (iii) White non-reoffenders
            (iv) Black non-reoffenders

            Ideally the distributions of scores of (i) should match that of (ii) AND the distributions of scores of (iii) should match that of (iv), but if the relative sizes are such that (ii):(iv) > (i):(iii), then (b):(a)>(d):(c).

            Is this just Simpson’s Paradox then?

          • lemmycaution415 says:

            An algorithm that factors in race (or some proxy for race) to determine parole risk is in a real sense biased even if “people with the same risk scores have the same re-offending rates”. The algorithm is using race to keep people in jail- punishing some people for being a certain race. That is bad.

        • AnonYEmous says:

          However, one thing that probably is not going to help is…hot takes.

          If you took anything away from Stucchio’s article, it should be this, so why are you protesting it so hard?

          Your argument is worthy of engaging, but it’s not ProPublica’s. Your argument is that, without conscious counter-discriminatory measures, already-existing discrimination could be made worse, because the data inputs themselves could be tainted. ProPublica’s argument is that the algorithm thinks black people are more likely to re-offend but they’re not. I guess how you get from their argument to yours is saying “they’re not but the cops lie about them being so and so they are arrested”, but ProPublica cited evidence of black people not being in jail despite being labeled so; if they’re being harassed by police and jailed unjustly, wouldn’t they, uh, be in jail?

          As to your actual argument, I don’t like it for several reasons.

          First off, it’s just the generic progressive / social justice argument, and I say this not so I can blast it with my superweapon, but because it embodies a lot of the flaws of those arguments. Namely: it’s like that in Baltimore, but is it like that in other cities in Maryland? Is it even like that in all the parts of Baltimore, for all black people? And how do you know, anyways? Heck, how do I know if what you’re saying is accurate? There’s no easy answer to all of these questions, so introducing conscious counter-discrimination has potential to backfire drastically. In my opinion, social justice’s backfire has led to more racial tension; if the system is literally geared towards releasing black criminals onto the streets, you had better believe that there will be more racial tension !!!

          So maybe you avoid the problem of AI deciding altogether by letting a jury of peers do it instead; according to this comment thread that’s slightly more accurate and I believe it. OK, but we’re mostly back to the same problem: how do the jury of peers know to discount accurately? I guess if they have local knowledge that kind of helps (and this is probably your strongest argument), but even then they may not know well; moreover, while they may be counter-discriminatory on their own, they also may be discriminatory (i.e. racist), and either way creates problems.

          Moreover and finally, the biggest problem here is that Baltimore police are apparently harassing citizens. I don’t think that there will be a bigger problem than that, and it should be fixed regardless; as it is fixed, the issue of algorithmic bias fixes itself. So…why not more effectively target that? Yeah, I recognize that it’s difficult and it’s been worked on a lot, but still.

          • Ilya Shpitser says:

            “First off, it’s just the generic progressive / social justice argument”

            For folks who wonder why I don’t want to engage a certain segment of the commentariat here and want them to go away, it’s this type of stuff.

        • MB says:

          This is a purposefully bad way of framing the conversation. The public’s interest is in minimizing the (expected and actual) number and severity of crimes committed by felons upon early release. A felon’s interest is almost irrelevant here, since there is no right to early release and in any case a felon loses some rights due to the felony committed.
          Also, there is a difference between prejudice and discrimination. Whenever some, but fewer than all felons are granted early release, discrimination can be said to occur. The issue is what criteria do we use to discriminate between them. The best outcome is when this discrimination is not based on ill-conceived human prejudice (and I invite the interlocutor to consider the possibility that his opinions are also biased, no less than mine). And no, I don’t care about the deeply flawed US legal definition of discrimination.
          Probably a person on the street who is not a rabid leftist would agree with me. Anything else is an attempt to obfuscate the issue.

      • David Speyer says:

        @Ilya I am interested in reading your paper. Is the right one?

        On a quick skim, it looks like I may also need some help with the vocabulary so, if you want to post a summary for the “good at math but don’t know ML and stats” crowd, that would rock.

        • Ilya Shpitser says:

          Are you David Speyer I would be happy to chat on skype/g+/zoom about our paper, if you wish. Or by email.

          Yes, that arxiv url is the right one.

          There are some stats details that are possibly not relevant, but the basic point is we need to think about a mathematical formalization of a sensible definition of discrimination that people would agree with. We cite a legal opinion that gives a very clear counterfactual “had everything else been the same” definition, which I think is the discrimination analogue of the Humean counterfactual definition of causation. Hume won — we all use his definition now in statistics. I hope that legal opinion will also win.

          Once we have that counterfactual definition, it’s just a matter of formalizing it, using a (?the?) language of counterfactuals (which causal inference people happened to have developed already). The right way to do data analysis to avoid discrimination falls out of that.

          • David Speyer says:

            Yes, I’m that David Speyer. Feel free to e-mail if you want to. I don’t see a real reason to talk one on one rather than here, but am very willing to. I’m just someone whose been following the debate, been frustrated by the lack of mathematical clarity in earlier publications, and thinks the topic is interesting. And I usually respond to the challenge to “read our paper” by clicking over to the arXiv. 🙂

          • Hume won — we all use his definition now in statistics.

            If I am correctly following this, you mean the “but for” definition of causation.

            The situation in law is a bit more complicated. My standard example is the situation where you meet a friend on the street, stop to chat for a moment, he then goes on and, a minute later, is crushed by a falling safe. If you had not stopped him he would not have been under the safe when it fell, so you are the but for cause of his death. But you are not legally liable.

            One, I think the usual, explanation is that the consequence was not forseeable. Put in probabilistic terms, the conditional probability of his death was not affected by your conversation given that, from your standpoint, exactly when the safe fell was a random variable.

            The real case is Berry v. The Bourough of Sugar Notch, 191 Pa. 345 , 43 A. 240 (1899).

            Note that while I am a retired law professor I am not and have never been a lawyer, so my expertise is in the logic of the problem, not the current state of the relevant law.

          • Ilya Shpitser says:

            “But for” in law is for torts (correct me if I am wrong, IANAL). But causation in law comes up a lot, so there are lots of other definitions / criteria. I agree with that.

            I think fairness/discrimination is similar, there are some attempts to formalize in legal opinions. There is no one test. We picked the most reasonable one.

          • “But for” in law is for torts (correct me if I am wrong, IANAL).

            The case I cited was a tort case. The argument was that if the trolley had not exceeded the speed limit it would not have been under the tree when it fell. Exceeding the speed limit was illegal hence per se negligence. So both the but for requirement and the negligence requirement were satisfied.

            Despite which the court ruled against the plaintiff.

          • michelemottini says:

            The legal definition quoted in the paper is:

            “The central question in any employment-discrimination
            case is whether the employer would have taken the
            same action had the employee been of a different race
            (age, sex, religion, national origin etc.) and everything
            else had been the same.”

            I find it clear and correct. My (naive?) take would be that an algorithm that does not have the protected characteristic (age, sex, etc) as input cannot discriminate based on this definition – because the decision is based only on ‘everything else’.

            I suspect that the paper says something different,but following the math would take a while….an explanation or example (if possible) would be appreciated

          • Ilya Shpitser says:

            “My (naive?) take would be that an algorithm that does not have the protected characteristic (age, sex, etc) as input cannot discriminate”

            This is the first reading of this, and it (except in very special cases) doesn’t work.

            An informal explanation is proxies for race — even if you don’t include race in your model, you might include a causal consequent of race that essentially determines race (e.g. zipcode). So you have to think about _paths_ from race to your decision. For example the path race -> zipcode -> decision based on zipcode. Formalizing how to think about paths is what the paper is about.

          • The Nybbler says:

            @Ilya Shpitser

            It appears that your paper’s analysis of the COMPAS algorithm assumes that all effects of race which are not mediated by prior convictions are assumed to be bias. Is this so?

          • Ilya Shpitser says:

            The Nybbler: I wouldn’t take the preliminary COMPAS analysis we did too seriously. We only had access to a limited subset of their data, and had to approximate their predictions by learning (Northpointe does not release theirs, proprietary info).

            But I think criminal record and similar types of things were the only mediators that seemed to us to be ok* to use.

            (*) In reality, of course, even criminal record isn’t perfectly ok, for reasons described elsewhere here — cops hassle differentially. We had some discussion of this very point in peer review.

            As I recall, though, we made some modeling assumptions (we weren’t looking for a scary result, just wanted to get the procedure to finish in a reasonable time), and got a fairly scary number. But again, the purpose of the paper wasn’t to give a definitive analysis, but to demonstrate the operation of the method.

            Actual convincing analysis is (a lot of) future work.

    • yossarian says:

      I remember some years ago I read a paper on criminology when I was in a forensics chem class. Basically, what it said, is that something like 90% of violent pre-mediated crimes and 95% of repeated violent pre-mediated crimes were commited by people who were easily recognised as criminals by average test subjects picked off the street. If so, it shouldn’t be that hard to have an AI that recognizes such signs.

      • Shion Arita says:

        Not necessarily, I think. I think that judging the intentions of other humans is a difficult task that I’d expect humans to be actually good at (as opposed to simple things that we’re crappy at, like arithmetic; a 1980s texas instruments calculator can knock the pants off anyone), both because of very strong evolutionary selection for being able to do so, both in a cooperative and competitive sense, and the fact that we have the architecture to emulate them.

    • James Koppel says:

      After seeing the article, I was about to jump in and post something similar, citing your advisor. So glad to see you beat me to it.

    • tscharf says:

      Journalists are absolutely misleading people when they talk about bias, infer it is intentionally immoral, and never print the underlying legitimate data that is the source of that bias. It is very rare to read a story about sentencing bias that even mentions racial disparities in violent crime (as the above comment does not). AI algorithms are there to remove disparities such as death sentences, not to insert them.

      There is a huge group think of racial disparities exist, therefore immoral racism. A burden of proof to establish immoral intent has left the building a long time ago.

      I have never once read a journalist’s analysis from the usual suspects of a culture war issue that stated the biases are factual and merit based but the real problem is they “encode awful biases in our society”. I think the issues would gather more traction if they were discussed like this, versus those who disagree are automatically labelled as x, y, z.

      Anyone can compete in the marketplace with “unbiased” algorithms against competitors with “awful biased” algorithms. Citizens should have a say in any regulations that distort this process. Perhaps people repaying loans is actually causally correlated to things other than skin color such as innumerable other cultural issues that just happen correlate to skin color. People mostly earn their bad credit ratings.

      To the extent that these problems are chicken / egg issues that should be addressed, but not necessarily by the heavy hand of government putting its thumb on the scale, and definitely not by the fourth estate hiding the root causes of an issue and only telling one side of the story.

      • Ilya Shpitser says:

        Just dropping here to say that this discussion is above your pay grade, based on what you said here.

        I have COMPAS data, it’s freely available (well, some of it). You can get it too. I bet you wouldn’t be able to make heads or tails of it, or whether there is discriminatory bias in it. Or what discriminatory bias even is.

        • tscharf says:

          “Journalists aren’t misleading people, they are correctly pointing out regression models can encode awful biases in our society.”

          I suppose journalists are experts that are on your illustrious “pay grade” and are experts on the academic definition of discriminatory bias and fully articulate those differences to their readers. My reading of how they handle the subject of discrimination and bias suggests they do not.

          I bow to your imminently high opinion of yourself.

        • cactus head says:

          Thanks for the ask_the_donald level hot take.

    • stucchio says:

      Author here. ProPublica did not show that the COMPAS algorithm exhibited discriminatory bias – in fact, they show it doesn’t. The COMPAS algorithm does not directly discriminate. It does not exhibit statistical bias. And it does exhibit calibration – a 30% risk score for a black person means the same thing as a 30% risk score for a white person.

      What ProPublica showed is that more blacks than whites have risk scores in the marginal range, e.g. 30-50%, and remain in jail as a result. People in this regime have a higher false positive rate – 50-70% of them would not have committed a new crime. As a result, blacks overall have a higher false positive rate.

      If we fixed this problem, then 9% more people will be murdered/robbed/raped by the black criminals we release. Or alternately, a lot of innocent white people will remain in jail.

      Simple example: regressions will fail to parole folks with big criminal records. But in places like Baltimore it’s very easy to get a long criminal record for no better reason than cops loving to hassle African Americans even if they are doing nothing wrong.

      If this were true, then it will show up as statistical bias. Criminal records will stop being predictive of committing new crimes.

      However, blacks are not wildly disproportionately hassled in proportion to their involvement in crime.

      Or similarly, see this article by Phil Lemoine which covers similar ground. In fact, blacks have contact with the police at roughly the same rate as whites:

      Similarly, a black man has on average only 0.32 contacts with the police in any given year, compared with 0.35 contacts for a white man. It’s true that black men are overrepresented among people who have many contacts with the police, but not by much. Only 1.5 percent of black men have more than three contacts with the police in any given year, whereas 1.2 percent of white men do.

      I fully believe that Julie Angwin knows what she’s talking about. In my view she’s being deliberately misleading because it generates clicks.

      • Ilya Shpitser says:

        Thanks for stopping by!

        “The COMPAS algorithm does not directly discriminate.”

        And how do you define this?

        “If this were true, then it will show up as statistical bias.”

        Statistical bias is expected error. This issue is NOT NOT NOT about the expected error of an estimator. It is about the data set itself being generated in a biased way. Us causal inference types think about this problem a lot (in another context than discrimination, usually).

        Now that you said this, I am extremely worried.

        • Douglas Knight says:

          [the issue] is about the data set itself being generated in a biased way.

          There are many issues. This is an issue to worry about, but it is not the issue considered by the ProPublica analysis.

        • stucchio says:

          By “directly discriminate” I mean “make use of the `race` flag in the data”. I understand that the literature has all sorts of fancy definitions of algorithmic fairness. That’s not my critique.

          My critique is that a casual reader of these articles will come away thinking that algorithms are making incorrect decisions, and that the solution is to somehow make more correct decisions and fewer incorrect ones.

          What do you think a casual reader will think?

          > It is about the data set itself being generated in a biased way.

          No, the ProPublica article is not about this at all. None of the articles I mentioned are. If so, they would have needed an entirely different methodology to demonstrate this.

          For example, one might do what Lemoine or SA did and compare arrest data to NCVS data, or look at other sources of data which remove certain sources of bias.

          • Ilya Shpitser says:

            Is this your definition? Why should I care about it as capturing anything reasonable? I know Northpointe excluded race from their model. Of course, this does not remove discrimination-we-care-about for what I hope are obvious reasons.

            The Northpointe regression also probably has expected error of 0. So what?

            The casual reader will come away thinking the algorithm is making incorrect decisions — and they would be correct. They are incorrect, in a relevant sense we care about.

            I think where we differ is I did not expect ProPublica to solve the problem for us, but to point out the problem exists. They did precisely that.

            Academics, in part because of the ProPublica piece, are now actively working on the issue. The fact that someone off the street might misunderstand the point is not really an interesting criticism, as it can be leveled against essentially any public-facing piece of text, on any topic.

            Your article, for example.

          • stucchio says:

            Ilya, You’ve completely dodged the main point of both the article and the comment you just replied to. Instead, you merely stated a non-sequitur (that you don’t care about my definitions).

            I’ll repeat the core point again:

            My critique is that a casual reader of these articles will come away thinking that algorithms are making incorrect decisions, and that the solution is to somehow make more correct decisions and fewer incorrect ones.

            What do you think a casual reader will think?

            Elsewhere I asked if you’d be willing to gamble your beliefs on this. (See for discussions of how a similar bet was set up.)

          • Thegnskald says:

            Illya –

            Please quit this low-quality nonsense. Being rude without contributing any arguments, as you have now done multiple times in this comment thread, doesn’t convince anybody of anything except your rudeness.

          • Ilya Shpitser says:

            Sorry, I edited the post. And I agree with you on what the casual reader will think, based on the ProPublica piece. (I think I disagree that they are being misled, though).

          • stucchio says:

            Ilya, I’ll reply to your edit.

            By “incorrect decision”, I mean incorrect in the sense of predicting a black person is likely to recidivate when he isn’t, or a white person is unlikely to recidivate when they are.

            With this definition, rather than the one you made up, do you think the casual reader will correctly interpret the article?

            Are you willing to gamble on this prediction?

            If you think the typical reader of my article will misunderstand something, can you state clearly what it is and why you think this?

          • Ilya Shpitser says:

            I am not sure what a person off the street might say. They may or may not conclude this?

            A person off the street is very confused about very basic probability things.

            I definitely agree people off the street could be misled by the propublica article (or your article, or my paper, or…)

            Why do you call the definition you mention “incorrect” — adapting propublica’s stance or is this your own? “Incorrect” is a loaded word.

          • stucchio says:

            Using the term “correct” to describe a prediction as matching reality is not “loaded”. It’s merely the dictionary definition of the adjective “correct”.

            If you feel the need to get into some postmodern debate about the nature of reality, I’ll bow out here.

          • Ilya Shpitser says:

            I don’t know what you mean by “matching reality.”

            Do you mean like, we learn the true E[Y | X] response curve? Sometimes you want this, sometimes you don’t want this.

            I have an algorithm that maps data to an answer. Another person has another algorithm. They both use data (e.g. reality). They have different properties. You chose some property based on probabilities you like, and called an algorithm with this property “correct.” Presumably you would say my algorithm, which lacks this property is “incorrect.” Presumably a third algorithm by some person who wrote a fairness paper at NIPS this year also lacks your property, so it will also be “incorrect.”

            What is “correctness” to you? Might be useful to taboo “correctness” and talk about real properties of estimators, e.g. unbiasedness, consistency, robustness, absence of some effect (what my paper is about), etc.

            You might say “when it comes to predicting recidivism, we want to learn the true response curve based on the data.” (e.g. have a consistent estimator for true regression parameters). I would say doing this would “match reality” (since we learned the true curve, after all), but I would disagree that this is what we want here.

            Because the data is not generated in a fair way, essentially.

            To give you a simple example that comes up in my neck of the woods, I want to learn if asbestos is bad for you, so I learn the true response curve of asbestos on health in folks who work. The response curve would then show that asbestos exposure is positively correlated with health, which is weird. (This is sometimes called the “healthy worker survivor effect.”)

            Here the regression model is “correct,” but answering the wrong question. We want the causal effect of asbestos on health, and the regression model, while correct, isn’t giving us the answer we want. Because the data was not generated in a fair way — in particular folks who were sickened by asbestos dropped out of the work force.

            Another method is needed (there are a few actually), that would presumably be “incorrect” by your lights, since they aren’t mimicing the true response surface of asbestos/health in the data you see.

            Discrimination is a similar problem, we need to define what we want, first. What we want is not always the true regression function.

            A lot of discussions/papers on fairness have this missing piece of not doing the required analytic philosophy first. They just pick some property they like, and just automatically assume it’s what we want out of a fair algorithm. A lot of these properties seem self-evident to the people who propose them, but interestingly, many disagree with each other. They can’t all be right.

            I do think a lot of the philosophical legwork was already done in the legal literature, it’s just a matter of extracting it out of their language. I have a lot of respect for the philosophical chops of legal scholars.

          • stucchio says:

            Ilya, that’s fine. My criticism is that a lot of people read these articles and actually believe the algorithm is incorrectly computing E[Y | X].

            The main point my article makes is “no, they are correctly computing E[Y | X], don’t let the word ‘bias’ mislead you”. You don’t seem to disagree with this point, correct?

            Is your sole objection to my article my choice of topic?

          • Ilya Shpitser says:

            Perhaps it might be easier if I told you what I thought of the propublica article, and you told me what you think is off:

            (a) I think the article pointed to a real issue,

            (b) I think the article formalized and addressed it poorly,

            (c) I think laypeople reading it might come away with all sorts of wrong impressions about what’s going on.

            I think these are probably uncontroversial points. The interpretation is, I suppose, where we disagree.

            I think they did a lot of good bringing attention to a possibly real problem.

            I don’t think they had a malicious intention, nor do I think they are stupid. They didn’t solve the problem properly but solving it properly is very hard.

            I don’t think it was their intention to prey on laypeople’s misunderstandings to generate clicks/outrage/$$$/who knows.

            I think you disagree on some of these.

            I think if some people read the article and conclude that Northpointe incorrectly computed E[Y | X] and that this is a problem, then they correctly concluded that there is a problem, but are incorrect on what the problem is, and for an incorrect reason.

            I also think that if some people read the article and conclude there might be a serious problem with how regression models are used for parole and sentencing, _without understanding the details_, I think that would be a good conclusion, and arguably the entire point of the article. You can’t expect a higher bar than this from a layperson.

            I also think a layperson is much more likely to think of “bias” in the way we informally mean when we talk about discrimination than in the “expected error” sense. So I wouldn’t expect a layperson to be mislead about propublica asserting things about “expected error” bias, as they don’t know what that is. They likely don’t even know what expected value is.

          • Trofim_Lysenko says:

            “Sometimes you want this, sometimes you don’t want this.”

            Am I understanding correctly that you are arguing there are cases where we want to sacrifice calibration/precision of an algorithm like this in favor of other values? If I’m misunderstanding, can you clarify, and if I’m not, can you please explain?

            Because I can’t think of any circumstances in which I set the goal “Tell me how likely outcome X is for individual/group Y” in which my stated goal is better served by being LESS accurate…

          • rlms says:

            The goal “predict measurements of a variable for a certain population” is tautologically best satisfied by making your model as accurate as possible if “accurate” means “good at predicting measurements of a variable for a certain population”. But that goal is usually an approximation to/instrumental step towards another: see Ilya’s asbestos example.

          • Trofim_Lysenko says:


            The Asbestos example shows that in some cases using a regression analysis cannot answer a question like ‘how dangerous is asbestos for me?'”. The equivalent question here is “How likely is person X to reoffend?”. I have to say that so far, stucchio’s point seems to be the most relevant.

            We know the regression analysis can’t answer the question about asbestos because it says that asbestos exposure is correlated with health, I can install it in all my buildings and observe that later on the people living there will have elevated incidence of mesothelioma relative to the population living in the non-asbestos buildings.

            Likewise, if a predictive algorithm fails to correctly predict recidivism, we would expect the rates of false positives and false negatives to be worse than or the same as chance.

            The goal here is “correctly identify high risk-of-recidivism individuals” with the larger goal of “minimize repeat offenses” which leads to “minimize crime”. So far, I have not seen a response to Stucchio’s claims that in order to tweak it to get the desired results (equalized false pos/false neg rates across races) you’ll have to accept either a 9% more crime from reoffenders, or higher rates of low-risk-of-reoffense whites getting harsher sentences.

            To my mind, the obvious thing to do then is to either refute Stucchio’s assertion and show that we can equalize those rates without the effects he claims, or bite the bullet and argue explicitly that, for example, the 9% crime increase is worth it for the sake of the principle of justice and not being overly punitive to those who dont’ deserve it due to structural oppression or what have you.

          • Ilya Shpitser says:

            “Am I understanding correctly that you are arguing there are cases where we want to sacrifice calibration/precision of an algorithm like this in favor of other values? If I’m misunderstanding, can you clarify, and if I’m not, can you please explain?”

            The way to understand our proposal is via these name-swapping experiments people run. The idea is, if we send resumes to evaluators, and you keep the resumes the same for one evaluator, and swap the name from white-sounding to black-sounding, but keep the resume the same otherwise for another evaluator. If you see a difference, that means something bad is happening — evaluators are using race itself, which seems like it shouldn’t be relevant, controlling for qualifications.

            (This is an oversimplification, because the way I said this makes it seem like you should drop race as a feature in your regression. But this doesn’t work, and explaining why will quickly get us into the weeds of the proposal. Basically it’s proxies for race, or “causal pathways” as we say, that matter).

            It is quite simple to construct cases where the optimal regression does pay attention to the name. But we don’t want it to do that. We want it to do as well as possible, as long as it doesn’t do that.

            Re: “and show that we can equalize those rates without the effects he claims, or bite the bullet and argue explicitly that, for example, the 9% crime increase is worth it for the sake of the principle of justice”

            I think you are still trying to judge a non-regression problem by regression standards. The point of the asbestos example (a real example in epidemiology, by the way), is that its easy to construct cases where regressions are entirely irrelevant. As in, we shouldn’t use any part of them as a yardstick for success. Why should we? They are giving the wrong answer.

            Equalizing conditional odds and so on are all regression concepts.

            Your intuition of “well, if we want to check if asbestos is bad, run a trial” is a great one! A good related question to get at discrimination is “what trial might we run to check for it”. This is how we started too.

          • It is quite simple to construct cases where the optimal regression does pay attention to the name. But we don’t want it to do that. We want it to do as well as possible, as long as it doesn’t do that.

            Two points:

            1. Are you saying that you want to rule out rational discrimination? Imagine the case where the only information you have about someone is his race, and where race correlates with unobservable characteristics that are relevant to what you are doing. Suppose, for instance, that on average medicine A works better for blacks and medicine B for whites, but for some whites and some blacks it’s the other way around. You haven’t yet figured out what the underlying characteristic is that determines which medicine works better.

            Am I correct in interpreting what you wrote about “we don’t want” as saying that, in that context, you should ignore race in deciding who to give which medicine to? I assume that wouldn’t be your actual view in that situation. If so, what defines the subset of decisions for which what you wrote applies?

            2. The name switch example you offer has a problem–names correlate with things other than race. Suppose you picked a white name that is common for the children of college educated whites, a black name common among high school dropout blacks. If switching names affects the results, that might mean discrimination by race but it also might mean discrimination by parental education level.

          • Trofim_Lysenko says:

            EDIT: I’m going through the linked slides that were posted while I was at work. It looks like they may address part of my point about loss of meaningful information when you bar “proxies”.

            @Ilya Shpitser

            First, thanks for replying to me. I know you’re dealing with a ton of simultaneous critiques/questions and I appreciate that you’re trying to clarify your position here.

            “The way to understand our proposal is via these name-swapping experiments people run.”

            Right, I read through your paper. The problem I see with that analogy is that the test you chose in the paper very clearly specifies that ONLY race can be changed, and in the counterfactual everything else must be the same. That means by definition that any and all PROXIES for race must be held constant. If you throw out proxies for race as well as race itself you are no longer abiding by the terms of the test. This is even more important if “proxies for race” include variables that actually provide us with meaningful information. If I’m devising a test for prospective truck-unloaders at a warehouse (standing in a truck and manually hauling crates out until they can be placed on a pallet or conveyer belt and automated assistance can take over), it will probably involve ensuring they can lift and maneuver a certain weight of boxes unassisted for an extended period of time without slowing down. It turns out that “good upper body strength and endurance” is a pretty decent proxy for sex. If I throw that data out as invalid, I’m also discarding a fairly important piece of data if what I care about is “Pick people who will be able to shift heavy boxes around fast”.

            “As in, we shouldn’t use any part of them as a yardstick for success. Why should we? They are giving the wrong answer.”

            I’ll be honest, I do not understand your claim here at all. The yardstick for success in the asbestos example is “Am I reducing the number of people getting sick?”. That’s why it’s so easy to demonstrate that the regression is failing to give you the correct result. It fails the yardstick, because when we follow the prescriptions of the regression, more people get sick, not less. The yardstick here is “Are we reducing the number of recidivist offenses over using a different algorithm or not using the algorithm at all?”. So if it fails the yardstick, we’d expect either no difference in recidivist offenses or an increase. I fail to see how that can possibly be “the wrong question” in this context. Concerns like “people can have long criminal records despite being innocent because of police bias” (assuming for the sake of argument that this is a factual claim) are properly addressed by addressing problems with policing and arrests, not with design of algorithms used at sentencing.

          • moscanarius says:

            The fact that someone off the street might misunderstand the point is not really an interesting criticism, as it can be leveled against essentially any public-facing piece of text, on any topic.

            I’m sure you understand that just because the same criticism can be levelled against other texts, it doesn’t follow that this specific text should not be criticized at all.


            I get that you think the propublica piece did more good than harm, despite its flaws; I get it, and it’s fine. Some people still thought these flaws were worth exposing for the greater public. That’s fine too. Why the flamewar?

          • moscanarius says:

            By the way, this a very confusing thread. Ilya seems to be arguing that nobody understands what bias, or incorrect, or reality, or anything means (except himself, of course)…

            …while at the same time arguing that nobody is being misled by what journalists are writing. If this discussion is so complicate, what are the odds that the journalists got it right in the first place, so as not to mislead their readers?

          • Ilya Shpitser says:

            I am not arguing that nobody understands bias. Lots of people understand bias. Understanding bias is one wikipedia click away. This is not esoteric knowledge by any means.

            I am arguing a specific person doesn’t understand the difference between two types of biases, based on very specific things that specific person said, which were wrong.

            I am also not arguing that people aren’t being misled by reading stuff online. I am saying that assigning fault for them being misled to journalists (because journalists are being deliberately misleading for various ends) is probably something I will generally disagree with. Serious journalists rarely do this.

            I really don’t think you understand my position, but maybe that’s fine.

          • Ilya Shpitser says:

            I think I am getting sufficient number of meaningful and interesting questions that I may soon declare “thread bankrupcy” and ask if people might be willing to take it to emails, with a longer turn around time.

            But very quickly:

            (a) The aim of the proposal is not normative but operational. Which is why I went with the formula of “sometimes we want something and sometimes we don’t.” In cases where heart medication gives side effects to [ethnic group] we definitely want to take ethnicity heavily into account.

            It is not my place to tell you which paths are fair (perhaps all are fair for heart medication). My place is to tell you what to do if you decided the direct path from race to hiring is bad, and you want to run a regression to make hiring decisions for you.

            (b) We do not use the precise legal definition in our procedure, because our procedure aims to avoid discrimination on average. There is a unit-specific version of this, to be addressed later. This is why we don’t fix things to constants, but leave post-race characteristics at the distributions they would have had, counterfactually, had the race stayed the same, but changed for the purposes of the direct path.

            (c) “are properly addressed by addressing problems with policing and arrests, not with design of algorithms used at sentencing.”

            It’s not that I disagree, but this is hopelessly unrealistic. Compare also to: “problems with sampling bias in surveys that determine people’s opinion of who they will vote for are properly addressed by asking every US citizen their opinion” or “problems with learning whether smoking is bad for you are properly addressed by running a randomized controlled trial on smoking.”

            It is unrealistic to survey everyone, or do RCTs on probably harmful exposures, or get the police not to be awful in some locales. However, by being smart with data analysis algorithms, we can compensate for these problems, anyways.

          • However, by being smart with data analysis algorithms, we can compensate for these problems, anyways.

            Perhaps, but I don’t yet understand how.

            Take a simple case which I think, although I might be mistaken, is the sort of thing you are thinking of. The legal system in a jurisdiction is more willing to arrest and convict a black than a white on the same probability of guilt. Probability of reoffending is estimated using an algorithm whose input is number of past convictions. The result is that a black and a white who have both committed ten burglaries and are both equally likely to commit another get recorded as having different past convictions, the white having been convicted for five of his, the black for six. So the algorithm predicts the black is more likely to offend. And it appears to be correct, if we measure offenses by convictions, because when both of them reoffend the black is more likely to be arrested and convicted. Is this the sort of thing you are thinking of?

            If so, the problem is how to avoid it. The direct way would be to find some measure of legal system bias, but none occurs to me. A tempting solution is simply to look at what percentage of blacks and what percentage of white get arrested and convicted and attribute any difference to legal system bias. But that’s wrong because it depends on assuming that blacks and whites actually offend at the same rate, and there is no reason to assume that.

            Part of your solution seems to be to forbid the algorithm from using race as an input, even though that might improve both its apparent accuracy and its actual accuracy. Beyond that you seem to want to forbid it from using proxies for race. But if blacks and whites actually offend at different rates, then number of offenses is itself a proxy for race.

            I haven’t adequately followed all of this long discussion, but part of your point seems to be to demand a causal link for any variable used. But it’s hard to think of a proxy for which one couldn’t offer a possible causal link, and it’s hard to see how one could do better than that. If we had a fully adequate theory of causation, after all, we wouldn’t need to use statistics, we could just derive probability of reoffending from the accurate model.

          • Ilya Shpitser says:

            David, I think I am too tired for a full answer to your very good question, but a quick note: the fact that we don’t necessarily forbid the use of race as input is an essential feature of the proposal. We forbid the use of race via certain “bad” causal pathways only.

          • stucchio says:

            @DavidFriedman, if you want to detect biased measurements in one part of a process, the typical way is to look for other proxy measurements that must (by fundamental relationships) exclude the bias.

            For example, if you think (as Ilya seems to) that police are biased in arrest rates against blacks, you should look for crime measurements that exclude the police. For example, the NCVS or crime *reports*. In fact, when you analyze the data this way, you discover that arrests and reports are pretty well correlated.

            Here’s a blog post where this analysis is attempted, though the actual data strongly disagrees with Ilya:

            Another thing to do is look for fundamental relationships that should be reflected in the data (absent bias), but are not. Here’s an example: most crime is intraracial. So now imagine blacks were more likely to be arrested than whites for similar crimes. That would suggest that crimes against whites (which are mostly also perpetrated by whites) are disproportionately not being resolved.

            Here’s an example of this technique in action, albeit not in a criminology context:

            The short answer is that there are a lot of statistical techniques that can be used to measure bias in measurements. Many of them have been studied, but very few of them explain crime disparities. I have no idea why Ilya keeps appealing to police hassling black men, since the data doesn’t support it.


          • Trofim_Lysenko says:


            I’m perfectly fine taking this to e-mail if you’d prefer. My personal e-mail is a gmail one for brerwolf. I’m re-reading the paper you co-authored now specifically to try and better follow your claims about how to preserve useful data from proxy inputs (that is, inputs that are correlated with the sensitive characteristic but also strongly correlated with with the probability you’re wanting to determine), and will probably have follow-up questions based on that re-reading.

            On the big picture question of system reform and fixing pieces other than sentencing decisions I have to strongly disagree that I’m being unrealistic. I mean, are you basically of the opinion that the police and court system process blacks have not changed in the past 40-50 years? That there’s no real difference in bias or prejudice between Bull Connor’s Birmingham PD circa 1963 and the Baltimore PD circa 2018? I think that’s both unduly pessimisstic and not supported by the data we have to hand.

            EDIT: Oh, and for those interested in the specific COMPAS inputs, I believe I found the actual assessment form. obviously we don’t know how these are weighted or processed, but as far as raw categories of inputs this seems to cover them:

          • Ilya Shpitser says:

            “Is this the sort of thing you are thinking of?”

            Generally, I am worried that if one is training regressions for predicting recidivism (or loan defaults, or…), the questions those regressions are answering might not be helpful for the kinds of decision-making we want to do, if the data fed into those regressions has various selection biases built in.

            This is similar to how an asbestos/health regressions cannot guide decision-making of the “let’s avoid asbestos in buildings” type, if given biased data, where sick people are missing.

            Your question is, what sorts of biased data might we have for recidivism. I don’t actually have a full answer — but there are people who work on this type of data who probably have a better idea. Some examples though:

            Races could have vastly different rates of minor offenses (like traffic stops, “resisting arrest,” disrespecting cops in various ways). So a black person and a white person who are from similar backgrounds and have similar personalities would look very different in terms of their criminal record.

            Drugs popular in African American communities carry much larger sentences, essentially for no good reason, compared to other types of drugs. So incarceration records might look different, for no good reason.

            Recidivism is defined as a subsequent arrest, not subsequent conviction. It is easy to imagine that some races would be arrested more (cops love to arrest folks even if they know a conviction will not stick, as an intimidation tactic, there was a recent news story of someone getting arrested for criticizing an official getting a raise).

            People who work on fairness disagree on what to do when faced with trying to do regressions from data with biases of the above types in it. One difficulty is defining what bad feature of the data represents “unfairness” and how to make this bad feature go away.

            Our take is we should think about things people intuitively think shouldn’t happen — like directly using race in a decision (because what one’s race is, directly, is not relevant for recidivism prediction or loan decision). Then we say, ok if these things shouldn’t happen in data generated from a hypothetical “fair world” where above biases are not built in to data generation, let’s try to find a world close to ours where those things people intuitively think shouldn’t happen in fact do not happen.

            I suppose, one way to think about the problem is this:

            We want to find a way to get data from a “fair world” where we can just do regression on the data in peace, and use it to decide who to give loans to and who to let out of jail.

            edit: An important point about selection bias — the type of selection bias you have in your data is not a testable claim. In other words, if I think people who get sick from asbestos are missing at random, this is an assumption, not a function of the observed data. I needed to point this out because one naive question one might have is whether we can, if given some data, run some sort of procedure on it to figure out what sorts of selection biases it has.

            Information on biases in data generation is “side information” from domain experts. This is similar to how causal models are “side information” on the causal story behind the data.

            I wanted to point this out, because if one were a hard-nosed Popperian empiricist, this entire business would look insane and unscientific. However, a lot of information that guides data analysis you want to do is not in the data itself, and cannot be checked with the data.

          • albatross11 says:

            One pretty obvious problem with correcting for an assumed bias that can’t be tested for is that the people assuming the bias may be wrong, and in fact may just end up importing the researcher’s biases into his conclusions in a way that looks impressively mathy from the outside.

          • Ilya Shpitser says:

            Absolutely. Which is why you use trusted domain experts for your side information.

            But in general, statistical analysis a lot more tentative than a layperson might think it is, for these reasons. Even ordinary kind of analysis, not to do with discrimination.

          • Murphy says:

            “Which is why you use trusted domain experts for your side information.”

            But the problem with that is that expert opinions are the second lowest tier of evidence, second only to layperson opinions and don’t have a great track record. For example: the fundie churches manage to line up hundreds of people with biology related academic titles to back ID and creationism vs evolution.

        • It is about the data set itself being generated in a biased way.

          Perhaps I missed it, but have you explained at some point in the thread what you mean by that? It isn’t clear to me.

          Also, are you saying that that is what the ProPublica article claims or are you saying that their claim was false but useful because it got academics to ask the right questions?

          • Ilya Shpitser says:

            I would say propublica didn’t address the problem properly. But this is a very high bar for a journalist who knows some data science to clear.

            I am fairly certain the journalist in question self-described their goal as getting academics interested.

            An example of data generated in a biased way is one I had above, where folks who were sickened by asbestos dropped out of the workforce, so you only see people in your data who were unusually healthy or resistant to asbestos. That is, the data generation process is suffering from confounding/selection bias. If you aren’t careful this will lead you to a silly conclusion about the relationship of asbestos exposure and health.

            You might imagine recidivism prediction data suffers from similar issues due to all sorts of bad things happening in policing and recording and so on.

          • quanta413 says:

            You might imagine recidivism prediction data suffers from similar issues due to all sorts of bad things happening in policing and recording and so on.

            But then you might happen to check and the causal issues may run in the opposite direction you expect or there might be no effect at all. You’ve shown a lot of “in theory maybe there is a causal model that makes this issue different in a specific way” but as far as I can tell you’ve provided little evidence for why we should expect the causal errors to run in one direction or the other.

            Tell me if I fill in the details wrong . You would say there are obvious reasons why we might expect something like: black –> lives in poorer neighborhood –> gets more parking tickets because police go there more –> some civil or criminal penalty. And then this last observed bit will influence the score given to predict recidivism. I only skimmed your paper, but if I’m understanding correctly, an inference that includes only the last piece of data and does not correct for this pathway being discriminatory (in the intuitive sense) will likely be discriminatory in your model.

          • Ilya Shpitser says:

            We tried our stuff on the part of the COMPAS data that is freely available, and the preliminary results aren’t encouraging.

            I think you got the idea in our paper right. Basically we first sit down and agree on which paths are “bad.” Then we do regressions in a way that make “bad” paths go away. A common “bad” path is using race directly in the outcome, not mediated by other variables. But other bad paths are possible, we run through some examples.

          • quanta413 says:

            @Ilya Shpitser

            Sorry if I missed something relevant to this. But a very speculative question. Say rather than a acyclic directed causal graph, you have a causal graph where part of the phenomenon is described by a time varying phenomenon where two variables affect each others changes over time. So the graph either has cycles or you could have each value of a variable at a given time be treated like its own node in the graph and you’d get sort of a long criss-crossing ladder subgraph. Is there any obvious way to extend the idea of bad paths to this case? Not that necessarily would work well mind you. Just some way or ways.

          • Ilya Shpitser says:

            Yes. Path-stuff extends to general cases that include time-varying confounders and so on. I have papers on this (either in annals or cognitive science, depending on what sort of style you like to read). If you are seriously interested, I can probably arrange to chat over skype also.

          • quanta413 says:

            @Ilya Shpitser

            Thank you for the offer. I don’t yet think I’d be spending your time well on skype. I may ask in the somewhat distant future if I find the time to read some of your papers more carefully. Maybe I’ll try to slip one of your papers in to the journal club in my lab. Your expertise is in a type of modeling I don’t work with, so it would take me some time and effort to ensure myself that I couldn’t answer any questions I had from using the tools. I usually find that’s better for my long term gains than asking someone even though asking can be faster.

          • Ilya Shpitser says:

            For future reference:



            The latter one has a little notation bug in an example (not important to central claim).

        • watsonbladd says:

          I didn’t agree with your definition of discrimination, but now that I write out the argument against it I think I’m more convinced it is right.

          We’ve got an instrument that spits out a number which supposedly predicts whether or not you commit another crime. And if this number is 30%, 30% of black people and 30% of white people with that number both reoffend.

          Your argument seems to me to be that to get this property the model had to add in a race based term, and that we should consider this discriminatory. And as it goes this seems intuitively convincing.

          But now suppose we remove this term. And now those people getting 30% marks are less likely than that to actually reoffend then 30%, say 25% of them are. Don’t they have a colorable case that “this algorithm is mispredictive for my race, and the judges are confining me for longer as a result”? And if there are proxy terms tightly correlated with race that get added, now things get hairy.

          I think that having the same predictivity for black and white defendants is a more defensible criterion. If 70% of those marked as 70% reoffend, whether white or black, and the judges sentence them accordingly, then if the black defendant was white and got the same number they would be treated the same.

          • Ilya Shpitser says:

            “I think that having the same predictivity for black and white defendants is a more defensible criterion.”

            What we do in practice is partly a political problem, and that’s a separate can of worms.

            However, one relevant slogan speaking to your proposal is “equal pay for equal work.” We don’t say “equal pay for both genders,” right?

            I think my take is, we should imagine a hypothetical world where everyone is race-blind in the right way. And we should look at the closest world to the actual world we observe with this property (race-blindness-in-the-right way). Probably we will predict less well in various ways if we do this, than had we used the observed data as well as possible (e.g. in the words of the article author “the regression isn’t correct.”)

            But I think we want race-blindness-in-the-right-way.

            By the way, it’s not true that this has to do with removing regression terms — it’s a bit more complicated.

            Here are some slides someone at Berkeley apparently did based on our draft (I didn’t actually realize this until fairly recently):


          • Aapje says:

            @Ilya Shpitser

            However, one relevant slogan speaking to your proposal is “equal pay for equal work.” We don’t say “equal pay for both genders,” right?

            The problem is that there are people who say the former and believe that the latter statement follows from this (and also people who do not believe that, of course).

            So the end result is that you have people using a very agreeable slogan, while they actually mean something more specific that far, far fewer people agree with.

            This is commonly called a motte-and-bailey here…

            I think my take is, we should imagine a hypothetical world where everyone is race-blind in the right way.

            Having the same goal doesn’t mean that people can agree on the path to that goal.

            There are people who want to reduce all bias as much as possible & there are those who believe that we (temporarily or permanently) need to introduce bias to counter other bias and/or preserve existing bias that works in the ‘right’ direction.

            Many of the disagreements among people are not about the goal they claim to have*, but the path to it.

            * Although I believe that quite a few people don’t actually have the agreeable goals they claim to have, but far more selfish ones that they legitimize by combining altruistic goals with cherry picking of facts and other fallacies. So they reason themselves into believing that achieving the selfish goals will also achieve the altruistic goals.

      • The bigger picture issue is algorithmic fairness. As you are clearly aware, there are multiple possible definitions of “fairness” including both the ones you focus on (statistical bias, calibration) and what ProPublica observes (difference in FP/FN rates).

        There is an interesting and subtle discussion to be had here, for example in the paper you cite and elsewhere in the academic literature, about what is the right definition and the resulting tradeoff. Your article does mention these more interesting issues, but does not seem to explore this in depth and ultimately takes the primacy of your particular choice of objective function for granted (e.g., use of terminology such as “correctness” vs. “wishful thinking”).

        This choice is your personal politics/philosophy, reasonable people (e.g., the ProPublica authors) may differ and offer other objectives.

        • stucchio says:

          It’s not about politics/philosophy so much as it is about topic. The topic of my article is journalists like ProPublica misrepresenting an academic topic.

          I did not advocate for any choice of objective function. I advocated against calling predictions inaccurate when they are merely inputs to a decision process one considers unfair. One should call predictions inaccurate if they fail to match reality.

          Note that if ProPublica wrote an article “A widely used algorithm accurately predicts blacks are a lot more likely to commit crimes if let out on parole, but we think you should let them out of jail anyway because it’s unfair”, I would not call them misleading.

          But that’s not the article they wrote.

          Also if you’d like to read my take on algorithmic fairness issues, rather than the media misleading people, I’ll be discussing fairness (in the Indain context) at 50p in Bangalore next month.

          • matthewravery says:

            “I did not advocate for any choice of objective function.”

            Implicitly, that’s exactly what your article does. You cannot separate the outcomes of something like COMPAS from the objective function you feed it.

            For example, you say:

            “ProPublica labelled the algorithm as biased based primarily on the fact that it (correctly) labelled blacks as more likely than whites to re-offend (without using race as part of the predictor), and that blacks and whites have different false positive rates. This is actually just a necessary mathematical consequence of having an unbiased algorithm — no decision process, whether implemented by an AI or a sufficiently diverse group of humans, could possibly avoid this tradeoff.”

            This is a result of the objective function they chose and could be corrected by choosing a different one. COMPAS decided to minimize the error in estimated recidivism probability full stop. They could have instead chosen to minimize this with a penalty applied for differentials in FP rates among protected classes.

            It’s also worth noting that you conflate statistical bias (in the second sentence) with the colloquial meaning of bias (in the first sentence, in reference to ProPublica) in this paragraph. You clearly know the difference (you do a great job of explaining it in the article), so it’s odd that you choose to ignore it here.

          • The Nybbler says:

            They could have instead chosen to minimize this with a penalty applied for differentials in FP rates among protected classes.

            Yes, they could have put their thumb on the scale. But then they’d be using race as an input to their algorithm, which would subject them to direct discrimination claims.

          • stucchio says:

            >They could have instead chosen to minimize this with a penalty applied for differentials in FP rates among protected classes.

            COMPAS does not have a utility function. COMPAS is computing P(will recidivate|data), which is basically just a posterior estimate. (Not exactly, mainly because for some reason they didn’t do isotonic regression to it.)

            In theory a decision process using COMPAS could look like:

            decision = argmax_{send_to_jail} U(P(recidivate), isBlack, send_to_jail)

            for some U(…) that varies based on isblack. But in reality, that decision process actually happens in the mind of a judge, not in COMPAS.

            Are you advocating that COMPAS should lie to the judge in order to manipulate this process to your favored outcome? Should other parts of the criminal justice system, e.g. crime labs, also deliver misleading information to judges in the interest of fairness?

          • Ilya Shpitser says:

            You use a lot of loaded words. Is not reporting the true correlational asbestos/health link in a public health study “lying to the public about the health effects of asbestos”?

          • juribe says:

            The predictions fail to match reality, because the data fed into the algorithm is a biased estimator of the generating function.

            The algorithm accurately predicts whether blacks are more likely to be convicted of a crime while on parole, NOT if they are more likely to commit a crime.

          • stucchio says:

            Juribe, why do you believe that one is not a good proxy for the other? Do you have any evidence for this?

            As cited upthread, there is quite a bit of evidence suggesting that arrest data/conviction data is more or less accurately correlated to offense data. The latter can be measured via crime *reports* (e.g. NCVS) and somewhat indirectly by data on victim demographics (most crime is committed by someone similar to you), in order to get an estimate of how biased arrest data might be.

            The general result I’ve seen is “it’s not very biased”. Do you have evidence to the contrary?

          • David Speyer says:

            What if ProPublica wrote an article saying “a widely used algorithm is much better at finding reformed white convicts than reformed black convicts?” That strikes me as a pretty accurate summary of the problem.

            I am torn between criterion (1) and criterion (2), but surely there is a fact pattern which would move your sympathies to (2). Suppose that we have 1,000,000 black prisoners, of whom 600,000 will reoffend, and 1,000,000 white prisoners, of whom 400,000 will reoffend. The algorithm returns:

            High risk:
            999,000 black prisoners of whom 599,600 reoffend
            1,000 white prisoners, of whom 600 reoffend

            Low risk
            1,000 black prisoners of whom 400 reoffend
            999,000 white prisoners, of whom 399,600 reoffend

            This meets criterion (1) almost perfectly. Is nothing wrong here?

          • stucchio says:

            @DavidSpeyer, that’s another fair way to describe it. I think we can all agree that making algorithms more predictive is a desirable goal.

            Literally every single person involved in the process is incentivized to make roc_auc go up. If they can figure out how, NorthPoint can go sell COMPAS 2.0 for more money. Communities can release more reformed criminals while keeping more dangerous ones in jail. The statistician building the algo can put “improved roc_auc from 0.65 to 0.85” on his resume and demand a big pay raise.

            The only people who might oppose this are prison guard unions.

            In the case of lending (which is what I work on), I can literally deploy an algo with higher roc_auc and my company makes more money.

            The only problem is that doing this is hard. It’s a lot less “we’re such woke baes standing up to racism” and a lot more “I ran a spark job that computes a KDA on geolocation data and discovered new crime hotspots”.

          • David Speyer says:

            @stucchio We certainly want to minimize error, but I think we also want to minimize the ability for a person to recognize in advance that he is at elevated risk of a certain sort of error.

            Would it not be rational for a white convict, hoping to go straight, to think “77% of white folks in my position got a low risk score. My odds are good, I better get my GED and keep out of trouble”? And for a black person in the same position to think “only 55% of black folks in my position got a low risk score. The algorithm can’t tell the difference between me and the gang bangers. I’m screwed from the start!”

            I think this is worse than if 65% of both populations got a low risk score.

            Numbers pulled from the first bullet point of Pro Publica’s analysis. My understanding is that you object to Pro Publica’s framing, but you aren’t saying their literal numbers are wrong.

          • quanta413 says:

            @David Speyer

            Wait, how do you think risk scores are being used? The article indicates they are primarily used at sentencing. Each convicted person will know the result of their sentencing which may or may not take into account the risk score. From their point of view, there should not be any uncertainty as to what group they are in. Your story makes no sense.

            And if you’re saying they take into account what risk score they might get before they commit any crime, I find that highly implausible. And even if they did, knowing you’re more likely to get assigned to the high risk group (who gets more severe punishments) would normally be assumed to either have no effect or deter crime. It would be incredibly surprising to see an increase in the punishment if caught and be more likely to commit a crime because of that.

          • stucchio says:

            Would it not be rational for a white convict, hoping to go straight, to think “77% of white folks in my position got a low risk score. My odds are good, I better get my GED and keep out of trouble”? And for a black person in the same position to think “only 55% of black folks in my position got a low risk score. The algorithm can’t tell the difference between me and the gang bangers. I’m screwed from the start!”

            This behavior would be completely irrational.

            The algorithm assigns each individual a risk score based on their behavior and history. An individual’s race is not used in the score at all. A black person who gets his GED, lines up a job for when he gets out of prison, and tells the psychologist evaluating him “no means no, rape is bad, never again”, will have the exact same risk score as a white person who does the same.

            Blacks have a higher risk score on average because they behave differently.

            I agree that their numbers are mostly correct (the p-values they get when they looked for statistical bias are incorrect, however – failed to correct for multiple comparisons). But it’s pretty weird how so many readers – yourself included – come away with totally factually incorrect beliefs.

          • David Speyer says:

            To quote from the propublica article, the score is also based on questions such as “Was one of your parents ever sent to jail or prison?” “How many of your friends/acquaintances are taking drugs illegally?” and “How often did you get in fights while at school?”. All of these are things which the convict brings with them from his pre-crime life, and are different for defendants of different races. Unless you mean that “if the black convict is smart enough to lie and give the answers to these questions a white person would give”, he will not get the same score as a white person, no matter what he does in prison.

          • David Speyer says:

            For the record, things which are clearly wrong in the ProPublica article (and which I have tried not to repeat):

            * Saying that the algorithm predicted people would reoffend, rather than saying that they had a high risk. “High risk” meant something like 50-60%, it shouldn’t
            be viewed as a failure of the algorithm that they didn’t all reoffend.

            * Not making very clear that simply adjusting scores by a racial factor would lead to more reoffending criminals being paroled — there is no reason to think that the black convicts who would benefit are the reformed ones who are being missed.

          • AnonYEmous says:

            Unless you mean that “if the black convict is smart enough to lie and give the answers to these questions a white person would give”, he will not get the same score as a white person, no matter what he does in prison.

            so there are no black people and white people who would already give the same answers to these questions?

            maybe the criteria themselves are dumb (especially as you can just lie about them, apparently), but what’s the big difference between this and, say, crimes committed? According to you it’s because these criteria are pre-crime, but that’s not that big of a difference, is it?

          • stucchio says:

            @DavidSpeyer, parents going to jail might be problematic. Plenty of white people have parents who went to jail too, though. I have no strong opinion on it.

            As for things like being friends with drug users and getting into fights, those are things an individual can control. So yes, COMPAS does incentivize people to avoid drug users and fights. Similarly criminal history – COMPAS gives you a higher risk score if you commit more violent crimes, and thereby incentivizes you to commit fewer crimes.

            If blacks disproportionately choose to commit violent crime and be friends with habitual criminals, how is that the fault of COMPAS? And how is using these choices that any individual can make somehow unfair?

        • Witness says:

          I don’t have a problem with the idea that we should consider different definitions of fairness when deciding whether or not to parole any given offender, or set parole policy in general.

          I take a bit of issue with the idea that an algorithm asked to calculate a recidivism rate, and appears to do so accurately, is called unfair or biased; it answered exactly the question it was asked to.

          • Aapje says:

            Indeed, blaming an algorithm for disliking how it is used is like blaming shovels, because someone got smacked on the head with one.

          • matthewravery says:

            “Indeed, blaming an algorithm for disliking how it is used is like blaming shovels, because someone got smacked on the head with one.”

            But if the government decides that a particular shovel is the tool they’re using, and that shovel smacks a protected class in the head more than another shovel might, saying, “This shovel is good at digging holes” is a non-sequitur. The point isn’t that you shouldn’t use any shovels, just that you should think carefully about the shovel you choose.

          • Ilya Shpitser says:

            Yes, I think Matthew stated the issue concisely.

            There is more to life than unbiased regression models. Even for things less tricky than discriminatory bias problems.

          • albatross11 says:

            It feels to me like there are two different things going on here:

            a. There’s a set of statistical/correct prediction issues that come up because of problems with the data or selection bias or whatever. Specifically, our crime data (and recidivism data) comes from the criminal justice system, which may have all kinds of biases built in. Some may be explicit racial biases (the cops arrest the black suspect first), others may be other kinds of correlations (blacks tend to be poorer, poorer people can’t afford private attorneys or bail, bail and private attorneys make you less likely to end up convicted or pleading guilty, etc.).

            These are pretty hard to untangle. Worse, there’s a huge amount of room for sophisticated statistical techniques that are easy to get wrong, especially when the researcher or organization using them really prefers some answers to others.

            b. There’s a question of values or law or policy, having to do with what our goals are with a predictive algorithm. For example, would we prefer a predictive algorithm for making parole decisions that:

            (i) Minimizes errors (according to the data we have, maybe with some attempted corrections for flaws in the data)?

            (ii) Minimizes errors subject to constraints on what information may be taken into account? (Race, religion, juvenile history, history of psychiatric treatment, etc.)

            (iii) Minimizes errors subject to some other constraints on the predictors’ recommendations (equal recommendations for matched black and white prisoners even when unequal recommendations would give a better prediction of whether they’d reoffend, longer recommended sentences for sex criminals to send the right message, etc.)

            and so on.

            I think it’s really important to treat these two issues separately. For the first issue, you’re dealing with a hard technical problem where neither the public nor judges can really do much more than try to follow along and understand what the issues are and what you can do to address them.

            For the second issue, we’re talking about an actual public policy question involving a tradeoff. We need to make that explicit–there’s nothing about that that need exclude non-experts. We want to know whether we should prefer more accurate predictions or ones that exclude some information, or that give some kind of equality of outcomes at the cost of worse predictions.

          • albatross11 says:

            A major problem with these algorithms, as I understand it, is that the users (judges) are mostly not at all sophisticated about statistics or machine learning. So you’ve got this magic black box that gives predictions about how likely someone is to reoffend, and it’s like it came down the mountain with Moses or something.

            What I think we need to do is both to try to do the technical work to make sure we’re as accurate as possible in our data and predictions, and also to surface the policy/legal tradeoffs to the judges and legislators and voters. I think there is a *huge* temptation to keep some of those tradeoffs hidden away, because they involve tradeoffs between important values and get lots of people mad, but if we don’t surface those tradeoffs to the policymakers and the public, we end up leaving it up to the technical experts to make policy, and there’s no reason to think they’ll do a good job of it.

          • Trofim_Lysenko says:

            saying, “This shovel is good at digging holes” is a non-sequitur. The point isn’t that you shouldn’t use any shovels, just that you should think carefully about the shovel you choose.

            I disagree. Shovels are, by definition, for digging holes, and you should optimize for that. Any other problems are, practically by definition, not problems with the choice of shovel but with how it is employed. To extend the metaphor, if I’ve got people getting whacked over the head by misemployed shovels, the answer is absolutely, categorically not “Buy nice soft foam rubber shovels. Sure, they’re less good at digging holes, but now people won’t get hurt!”.

          • Aapje says:

            A lot of the claims about how the male white programmers are introducing bias in the algorithms only make sense if the people who make those claims believe that the shovels are broken themselves.

            To me, there are two reasonable possibilities to intervene:
            1. Improve the algorithms if they are broken/have a clear bias
            2. Partially ignore/counter the algorithms, based on decent evidence of the bias you try to counter with the bias you introduce

            Instead, a lot of people want to make the algorithms worse. I understand why, because it’s very hard to make an objective argument for (2), for the exact same reason why improving the algorithms is hard (plus other reasons*). By fudging secretly, you don’t actually have to defend what you are doing, so you can get away with adding whatever bias you prefer (and don’t have to prove that it does a good job at countering other biases).

            However, is it good/fair/helpful to introduce secret biases that in themselves may be a huge source of unfairness?

            * Some people want to intervene by canceling out the exact biases with counterbiases, while many others want to do it on the group level. The latter can create huge variance with group members that suffer from the negative bias, yet who do not profit from the positive bias & group members who do not suffer from the negative bias, but who do profit from the positive bias. Individualists tend to not like it when the suffering and privilege equals out on the group-level, but there is huge unfairness on the individual level.

          • albatross11 says:


            I think the “white male programmers are building their biases into the models” is an easy rhetorical point that doesn’t really hold together when examined. (Among other things, an awful lot of those “white male programmers” are East Asian or South Asian.) While “ML algorithms trained on data with biases will make predictions that reflect those biases” is accurate, but it involves fairly subtle arguments about what biases exist (or if they exist). And that’s separate from policy-level decisions about whether we should accept less accurate predictions to avoid basing important legal or commercial decisions directly or indirectly on race.

      • Ketil says:

        ProPublica did not show that the COMPAS algorithm exhibited discriminatory bias.

        From the bullet points in the article (

        “Black defendants were often predicted to be at a higher risk of recidivism than they actually were.” and “White defendants were often predicted to be less risky than they were.” This sounds like bias to me?

        I’d just add that your proposed explanation (black being more often in the medium risk category) sounds like a reasonable explanation for different false-positive rates. It is also deeply suspicious that PP makes these claims without plotting the data – with a plot of estimated risk vs actual risk (similar to calibration curves) for white and black, it should be abundantly clear whether the algorithm is fair or not.

    • Douglas Knight says:

      This reminds me of Scott’s recent posts. When someone makes a stupid or mendacious argument that can be crudely rebutted, it should be crudely rebutted, even if more sophisticated analyses are possible. But it is wrong to claim that the crude argument is the whole story.

      • Ilya Shpitser says:

        Don’t think the ProPublica piece was either stupid or mendacious.

        You are asking too much of them. I would rather they sounded the alarm and were wrong about some stuff than stayed silent.

        • Jack Lecter says:

          I think some of this comes down to how much stock to put in “conscious intent”.

          If you write stuff you know will be misinterpreted, using language that’s technically false*, is it really important whether you also think to yourself, “Ha ha, now I’m going to lie to you!” as you do it?

          Benquo put this better than I could here: Bad Intent is a Dispostion, Not a Feeling

          Also, it’s tempting to round this

          I would rather they sounded the alarm and were wrong about some stuff than stayed silent.

          off to

          I’m okay with people playing fast and loose with the facts, as long as it helps my side achieve its political goals.

          Which is unlikely to be a popular attitude ‘round these parts.

          (At least Arthur Chu was up front about it.)

          *I haven’t read the ProPublica article, so this should be taken hypothetically.

    • lycotic says:

      FYI, the “Equality of opportunity” paper with broken links from the article above is here:

      A quick glance at COMPAS a while ago suggested that it was an overtrained mess, and recent investigators have suggested that it doesn’t even beat humans:


      Beating humans at this kind of a task is *usually* not hard:

    • Anon. says:

      Ilya: consider a scenario where we have an algorithm that is 100% accurate. It accurately predicts a higher reoffense rate for blacks than whites. Would using this algorithm be discriminatory under any circumstances? Why? Which circumstances?

      • juribe says:

        The idea itself is discriminatory because being black is not a measurable category. Is blackness self reported? Is there a threshold for melanin?

        Fair algorithms should work to correct the bias that already exist in society but have no basis in object reality (we are rationalists after all) so they would be harsher on privileged classes.

        • jonahkatz says:

          Yes, this would be ‘discriminatory’. Because what most people mean by ‘discriminatory’ has absolutely nothing to do with the statistical meaning of ‘bias’ (and pretending that the everyday meaning of the word has anything to do with the statistical meaning is deeply silly). What the average guy on the street understands ‘biased’ and ‘discriminatory’ to mean is that, in the cases in question, they treat black people worse than white people, whether or not that may be ‘justified’ by the data. And the assumption of progressives is *not* that black people and white people are equally likely to re-offend after prison release, or equally likely to pay back loans with comparable credit scores, or equally likely to do x, y, or z. There may be instances of individual leftists hopefully proposing incorrect things like this, but this is not the core of the argument. The argument is that even if you find statistical differences between racial categories in any of these regards, it is *fundamentally immoral* to treat individuals as exemplars of their racial groups, especially though not exclusively in cases where race is not inherently linked to the outcome in question and is clearly being used as a heuristic for a messy hodgepodge of socio-economic factors that would require hard work and independent thought to sort out on their own terms. I don’t think you’ll find many people on the left who are totally unwilling, in the face of convincing data, to acknowledge that algorithms that (indirectly) disfavor black people are more accurate or efficient than algorithms that don’t. Their belief system holds, rather, that it is morally wrong to use those algorithms. I can imagine arguments against this position (I am not a leftist, although I was raised by leftists and have a certain sympathy for their worldview). But those arguments are not being made here; instead we’re hearing a bunch of reasons why it’s more efficient and/or accurate to allow algorithms to penalize racial groups for things that covary with their racial group. That’s not going to convince anybody, because it’s not an argument against their position. And yes, this would extend in the limit to categories beyond race and to car insurance and health insurance and everything else.

          • stucchio says:

            Misconceptions like yours are exactly why I wrote my article; ProPublica and other journalists have led you to believe wildly incorrect things.

            The COMPAS algorithm does not use race as an input. Neither does any loan approval algorithm.

            The COMPAS algorithm treats every person as an individual. It uses a “messy hodgepodge of socio-economic factors that would require hard work and independent thought to sort out”. Data points include things like “how many crimes has this person committed before”, “were those crimes violent”, “does this person have a job lined up”, and “does this person admit to the prison psychologist that they feel lots of uncontrollable rage”.

          • Ilya Shpitser says:

            Not using race as a feature does not avoid the problem, for the obvious reasons of race proxies. Common one is zip code in segregated places like Baltimore city, but there are others.

          • Aapje says:

            @Ilya Shpitser

            I think that you’ve got three kinds of inputs that cause a racial difference:
            1. Those that are 100% relevant to the (non-racist) goal. For example, running up credit card debt may correlate negatively very strongly with paying back loans. Black people may more often run up credit card debt, but that doesn’t mean that this parameter is merely a race proxy, because it also works for other races.
            2. Those that only work because they correlate to race, like using the number of times someone rented a Madea movie as a parameter. Such a kind of parameter is not predictive for other races. So white people who watch Madea movies a lot would then not be worse at paying back loans.
            3. Those that are a mixture.

            Type 1 may indicate current racism, for example when black people are denied regular loans based on their race, so they resort to credit card loans. However, it can also indicate the after effects of racism that no longer is in effect or it can indicate that black people have sub-cultures that on average are different from other races.

            Type 1 can cause feedback loops, which are IMO the only legitimate reason to reject them, based on an equality of opportunity morality. To reject such a algorithm, it is then not sufficient to have disparate impact, but it has to be shown that the feedback loops exist.

            Type 2 treats people differently purely for their race, which is incompatible with equality of opportunity.

            Type 3 is the unpleasant consequence of not always having good quality data.

            The issue with trying to fix this is that erring in both directions is in some ways unfair. If a person has a history of paying back loans, then why should this person be (partially) treated like a person without such a history?

          • stucchio says:

            Ilya, it’s fine that there are other definitions of bias. But that’s totally not what is being discussed here (except by you).

            Jonahkatz is one of the many people who read the ProPublica article (and many similar articles) and as a result believed entirely incorrect things.

            My article aims to point out that a) the beliefs folks like Jonahkatz hold after reading them are incorrect and b) journalists are probably doing this on purpose. My issue is journalists propagating factually incorrect beliefs about what algorithms actually do, not disputes over assorted different conceptions of fairness.

            If ProPublica wrote the story “There’s an algorithm that accurately predicts blacks commit lots of crime but we think you should ignore it’s factually correct predictions in the name of fairness”, then I’d have no problem.

          • Steve Sailer says:

            One interesting question is whether race adds predictive power incremental to even a host of other factors. Across much of the social sciences, race remains a powerful factor that makes forecasts more accurate.

            It’s evident, for example, that NFL teams use race as a factor in addition to non-racial factors in determining who to play at cornerback. (Blacks have filled all 64 starting cornerback jobs since 2003.) Whites and Polynesians who would appear to have what it takes to play cornerback in the NFL tend to get routed to playing safety instead. (Bill Belichick, perhaps the all time smartest NFL coach, used Julian Edelman at cornerback at times, but eventually determined he was most valuable at receiver.)

          • Ilya Shpitser says:

            “Except by you.”

            Earlier in the thread:

            me: “Simple example: regressions will fail to parole folks with big criminal records. But in places like Baltimore it’s very easy to get a long criminal record for no better reason than cops loving to hassle African Americans even if they are doing nothing wrong.”

            you: “If this were true, then it will show up as statistical bias.”

            It seems to me the person who is confused about the difference between “statistical bias” and “selection bias in data generation” is you. I suppose it is also true you read the propublica piece. In the true “let’s confuse correlation and causation” style that you seem to be exhibiting here, you might then blame your confusion on having read the propublica article.

            But I think just because you first read A, and then got confused about B does not necessarily imply A caused you to be confused about B. You probably were confused about B to begin with.

          • stucchio says:

            Ilya, if you want to see that my claim regarding algos learning biases in their inputs is true, you can validate it in a few minutes with numpy.

            Here’s some rough pseudocode you can play with:

            df = DataFrame()
            df['race'] = Bernoulli(0.2).rvs(n_samples)
            df['crime_propensity'] = Norm(0,1).rvs(n_samples)
            df['violent_crime'] = (df['crime_propensity'] + Norm(0,25).rvs(n_samples)) > 1.0
            df['num_tickets'] = Norm(df['crime_propensity']).rvs(n_samples) + norm(0, 0.25).rvs(n_samples)
            df.loc[df['race'] == 1, 'num_tickets'] += 3

            Now do regression on that dataset, but exclude crime_propensity from the regression (since that’s an unobservable variable). The input data is explicitly biased – every black person explicitly gets 3 extra tickets.

            Yet relatively simple regression will learn and correct for the bias in the input data.

          • Ilya Shpitser says:

            Let’s talk about asbestos.

            In asbestos data, you will have missing data due to the fact that folks who drop out of the workforce are systematically different from those who do not. This is what I mean by selection bias in the data — your data is not a representative sample.

            Now, if you use the data, as you have it, and learn the true regression curve relating asbestos exposure and health, your statistical bias, is (by definition) 0. Selection bias is still there.

            Similarly, if I did have a representative population (without dropout due to poor health), but I used an incorrectly specified regression model for the asbestos/health link, I would have no selection bias in data generation, but I would have statistical bias in the estimator (due to model misspecification).

            Selection bias and statistical bias are different. You can have the first but not the second, or the second but not the first.

            As far as I can tell, you are proposing I look into a regression, and then another regression where I exclude a covariate. This is _not_ selection bias in data generation, this is just a marginal model (not including variables in a model is effectively marginalizing those variables out from your observed data distribution). This is again orthogonal to whether the distribution itself is a fair representation of the population you want or not.

            The classical example of selection bias in data generation is survey data (and missing data more generally).

            edit: omitting variables from analysis _could_ lead to things like confounding bias, if you are interested in causal analysis — but I don’t think you are talking about that here, you seem to just be interested in regression problems.

          • stucchio says:

            Ilya, you’re repeatedly harping on the possibility that a regression does not fully capture causality. No one disputes this.

            Can you explain the relevance to this case, and how this relates to any of the examples in the article?

            I kind of get the impression that you’re trying to Euler us. (In this sense: )

          • Ilya Shpitser says:

            You seem to be misunderstanding the difference between statistical bias and selection bias. I have yet to see you address this. Your python example did the opposite of convincing me. This showed up in your first comment here.

            I think based on talking to you we seem to have some general disagreements about the intent of the propublica authors, but aside from that, I am not even sure we are speaking the same language, yet, when it comes to biases.

            So either:

            (a) I am misunderstanding you (so please explain, if you have time), or

            (b) You are actually misunderstanding this, in which case I think we should have a conversation about biases, what you think propublica is saying, and it is actually saying.

          • albatross11 says:


            The thing is, race proxies may themselves be relevant for what causes crime. To use a simple example, if you grow up in a high-crime neighborhood with a lot of gang activity, you’re probably a lot more likely to end up joining a gang. I’m pretty sure a lot larger fraction of black kids than white kids grow up in such neighborhoods. So if I find out what neighborhood you grew up in and use that to predict the likelihood you’ll commit a crime, I’m going to both be getting a good estimate of your race and also to be learning something actually relevant about predicting whether you’ll commit a crime. Similar things apply to family income, whether your parents were married when you were born/raised, whether you graduated high school, etc.

          • Ilya Shpitser says:

            “The thing is, race proxies may themselves be relevant for what causes crime. ”

            I agree. This is why I think fine-grained causal path analysis is important — we have to try to disentangle all that stuff.

          • stucchio says:

            Ilya, you’ve now convinced me completely that you’re eulering. You repeatedly hint at certain topics, act superior when someone misunderstands you and thinks you were referring to something else, and refuse to engage with the concrete discussion.

            If you believe some particular bias is relevant to any of the articles under discussion, make that argument. If you want to appeal to things like “police hassling black men disproportionately” or “arrests are biased”, first engage with the data (that I’ve cited multiple times) and then make that argument.

            Otherwise have fun. I don’t think you’re fooling anyone here, and I don’t plan to waste more time on this.

          • Trofim_Lysenko says:


            I’ve disagreed with Ilya before on other issues, and I’m still somewhat skeptical of the proposed methodology for ensuring algorithmic fairness and the appropriateness of applying it to this particular case, but I am also entirely confident that he is not “eulering” in the sense of trying to snow us with specious jargon. I think you guys are mostly talking past one another.

          • quanta413 says:


            I’m also am pretty sure Ilya is not trying to Euler anyone. He’s being a bit oblique in some posts, but I’ve seen how he posts when he actually isn’t interested in discussing something (at least, as far as I can tell from his behavior then) and this isn’t even remotely close.

          • Ilya Shpitser says:

            I think my current take away is you seem to have a way of finding a link between not understanding something and ill intent. Thanks for your time, I appreciate trying to engage with me.

            I did call your article awful, but it’s just one guy’s opinion!

          • stucchio says:

            Trofim, Quanta, as I said I’m happy to engage with Ilya at a concrete level.

            I don’t see much point in responding to vague allegations that I’m confused and fail to understand selection bias or causal inference or whatever. I have no intention of engaging in some “prove your expertise” contest judged by him. The only contest I’m willing to engage in is a prediction contest (he was unwilling to do so).

            If Ilya wants to say “I think selection bias exists in ProPublica’s example because of X, and it’s relevant to the discussion even though ProPublica never mentioned it because Y”, I’m happy to discuss. If the specific claim is “arrest data is biased in some manner because cops arrest more blacks for no good reason”, then I’m happy to discuss it after he engages with Scott Alexander’s and Phil Lemoine’s data on the subject (which I’ve cited repeatedly).

          • quanta413 says:


            I agree that there’s no real reason for you two to engage each other. You want to discuss this specific topic. Ilya wants to talk about theoretical adjustments to remove the “wrong” types of information/inference so to speak from the data post hoc. It’s similar to how the U.S. already mostly doesn’t allow rules or laws to be explicitly race based, but extends this rule. But he seems mostly uninterested in trying to show anything about this specific problem from known data. I think his point is about what is philosophically desired from an inference. We can hypothesize cases where using race directly in an inference is extremely predictive of outcomes and maybe even causal, but we would ignore it anyways since there is a broader commitment to law being based on the individual and not what group they are part of. The nice thing about relatively deontological meta-rules is they’re hard to weasel out of and tend to be good protection against terrible mistakes at the cost of maybe not being optimal for every situation. But if you allow for easy proxies for race, then you’ve routed around the rule, which is bad.

          • Ilya Shpitser says:

            Ok, let’s see.

            Here’s how your game would be played. I find an article online, say this one:


            I look for things that aren’t quite right, in it:

            “In short, this procedure of building a new predictor out of the old predictor plus a biasing variable is a recipe for us to find bias.”

            “ProPublica ran the exact analysis I describe above. The main difference is that they used a Cox model for survival probabilities (i.e., the probability of a criminal not re-offending) rather than linear regression. The general idea is similar but the the functional form differs. From what I can tell this is the right thing to do.”

            “Looks like a big win for machine learning, right? The statistical algorithm predicts criminal recidivism and it does in a way that any racial bias present cannot be distinguished from random chance.”

            So the claim “this is a recipe for us to find bias” is misleading (it’s wrong sort of bias you are finding).

            “From what I can tell this is the right thing to do.” No, it’s not the right thing to do. Well, there’s argument to be made on what precisely the right thing to do is, but it’s not this.

            The reason is simple, removing a feature from the model but using a strong proxy from the model just goes back to using the feature again. There are lots of proxies, or tricky combinations of proxies that can recover the original feature. (The other point is we are only interested in causal-consequent proxies, intuitively).

            The last sentence basically defines racial bias in a way that has to do with removing the feature from a model. This makes it seem like (a) this is the right definition, and (b) is a settled issue. Neither is true.

            I think you would probably get super annoyed if I said “Chris Stucchio is deliberately misleading the public to generate traffic to his blog.”

            I think the fact that we should not lead with that is my main point (this is a point about how most effectively to have a conversation about algorithmic bias — lots of folks are confused and that’s ok).

            The secondary point is maybe let’s try to minimize saying things that are actually wrong.

        • The Nybbler says:

          Do you claim knowledge of “object reality”? If so, how can we obtain this knowledge to test these algorithms against it?

        • Edward Scizorhands says:

          being black is not a measurable category

          The algorithm doesn’t have “race” as an input. The people complaining about “race” are ProPublica and the like who feel that the ones they call “black” are being discriminated against. I hope you write ProPublica an angry letter for measuring something that isn’t measurable.

          • albatross11 says:

            More fundamentally, “black isn’t a measurable category” implies that there can be no possible evidence about racial discrimination.

          • Jack Lecter says:


            To both of you.

            On a personal note, I’m sick of people who want to debate the underlying ontology of race when- and only when- someone makes a well-supported argument they don’t like.

            It’s a complicated subject, and they could easily end up arguing for the right conclusion just for Gettier reasons, but it’s the principle of the thing.

        • Steve Sailer says:

          Actually, self-reporting of race often works pretty well. It’s usually good enough for government work.

          On the other hand, if you let people who self-report as Asian out of prison faster than people who self-report as black, eventually the jailbirds will figure it out and take steps accordingly.

      • 1soru1 says:

        If an algorithm is 100% accurate, it has zero false positive and false negative rates for all groups. 0 = 0, so it is fair.

        The fairness issue is that, given any real-world algorithm _will_ have error, how is that error to be distributed?

    • Tedd says:

      I happen to have gone to a talk by the author of the propublica expose in question, and I can assure you, she knows what she is talking about, and surely understands the difference between these two types of bias.

      If this is so, I am deeply concerned. My primary objection to the ProPublica article has always been its one table with numbers, which has the labels in one dimension “Labeled Higher Risk, But Didn’t Re-Offend” and “Labeled Lower Risk, Yet Did Re-Offend”, and in the other “White” and “Black”, and has a caption reading in part “But blacks are almost twice as likely as whites to be labeled a higher risk but not actually re-offend. It makes the opposite mistake among whites: They are much more likely than blacks to be labeled lower risk but go on to commit other crimes.”

      As pointed out in the subreddit, those labels look very much like they’re giving P(not reoffend | high score) and P(reoffend | low score) by race.

      In fact they are giving P(high score | not reoffend) and P(low score | reoffend) by race.

      I have asked at least a dozen reasonably mathematically sophisticated people about that label and not one has interpreted it as the authors intended or even offered theirs as a possible interpretation. I cannot imagine an author who sincerely understood what she was talking about using the label “Labeled Higher Risk, But Didn’t Re-Offend” to mean “P(high score | not reoffend)” in good faith.

      I’ve always assumed this was simply a misunderstanding on the part of the authors. You say it is not. If it’s not stupidity, then, that implies malice.

      • Ilya Shpitser says:

        I think it’s possible for people to be neither stupid nor evil, and still get things wrong. It’s possible that the propublica folks were tackling something genuinely hard, and they got something wrong.

        The question is, do you want journalists to look into stuff like this in the future. Because if you are going to crucify them for not doing Fancy Statistics That Handle The Problem Properly, you are incentivizing less propublica style articles in the future. I think it would be a shame if there was less of this stuff.

        • Tedd says:

          It’s possible that the propublica folks were tackling something genuinely hard, and they got something wrong.

          I suppose. I don’t think the problem of labeling your tables in a way which is not actively misleading to everyone is that hard of a problem, if you actually understand what your tables represent.

          Because if you are going to crucify them for not doing Fancy Statistics That Handle The Problem Properly, you are incentivizing less propublica style articles in the future.

          I don’t expect journalists to get everything right the first time. I do expect them to own up to their mistakes when they’re pointed out rather than doubling down, and to post corrections, so that their mistakes stop misleading people. And as long as they fail to do so I think it is correct to continue to hold them to account.

          I don’t think this should have ended, as it seems to have, with ProPublica getting crucified forever for their mistakes. I think ProPublica should have corrected their mistakes, and then we could have all been about our ways.

          I would in fact like to incentivize journalists not tacking genuinely hard problems if they’re not going to be willing to later correct mistakes they make in doing so.

          The question is, do you want journalists to look into stuff like this in the future

          I am honestly not sure. It’s not clear to me whether they’re on average doing more good than harm. The ProPublica article, uniquely, set off a wave of really good and valuable research across a number of fields (though it must be pointed out that it did so by making strong claims despite weak theory, thus triggering a great deal of discussion a la Toxoplasma of Rage), but this is not the common case.

        • stucchio says:

          Suppose I wrote an article about Hillary Clinton’s emails, and the average reader winds up believing that Hillary was molesting small children in a pizza shop.

          My article raises a very valid issue – Hillary Clinton’s behavior regarding state secrets and document retention is a big problem.

          Do you similarly see no problem with my article?

          I totally think people shouldn’t criticize my hypothetical article because it’ll disincentivize people from doing investigative reports about political figures.

        • wintermute92 says:

          I’m yet another person who looked at that table and misinterpreted it as @Tedd describes. I just asked three other people statistically-literate people to look it over, and all three reached the same wrong conclusion.

          I think framing this as “do you want more thoughtful fact-based analyses or fewer” is deeply unfair. ProPublica ran a table of P( B | A ), and presented it such that ~100% of readers interpreted it as P( A | B ). That’s not crucifying people for a lack of Fancy Statistics, it’s saying that their labelling mislead virtually every reader on what statistics were being used.

          ProPublica is vastly better than almost anyone else in their field. Nevertheless, each of their algorithmic bias articles has left me feeling mislead – about the basic numbers in play, not some grand point about types of bias. Their (much less popular) followup explanations of technique have been informative and useful, but I certainly don’t believe I’m squelching good journalism by asking to know what an ill-labelled number actually describes.

      • wintermute92 says:

        For whatever it’s worth, I read that same table and made the same mistake. I expect I’m at least 2.5 sigma for statistical literacy. I asked some other people who are at least 2 sigma. They all made the same mistake.

        This doesn’t look like an issue of ignorant laymen or complex statistics. It looks to me like ProPublica put a sloppy label on some data, and mislead virtually everyone because the obvious English-language meaning of the label was false.

        Hopefully, this is just a case where the label made sense to a writer who already knew the data, and they didn’t test it out on new readers. It’s certainly not a case of good communication.

    • ksvanhorn says:

      “regression models can encode awful biases”

      If it has predictive power, why is it an awful bias?

  8. Study: Lobbying Doesn’t Help Companies Or Their Shareholders. This is well within a large body of work finding that money doesn’t really matter in politics, but if true it means both that popular wisdom is so wrong we should be thrown into near-Cartesian doubt about everything, and that corporations are idiots and throw away money for no reason.

    Popular “wisdom” is often wrong; the political and business worlds adhere to many easily-disprovable folkways.

    Longtime commenters here are probably tired of seeing my arguments that the power of money is gigantically overrated in politics. I see no reason for an epistemological crisis.

    A corporate lobbying shop, surely, is constantly at work inventing reasons why it’s absolutely vital to the parent organization. The execs back in Dallas or Denver or Seattle, who rely on those same people for “inside” knowledge of D.C. politics, are unlikely to challenge those contentions.

    • Brett says:

      I don’t know if it’s a case of lobbying just “cancelling other lobbying out”, or lobbying just being nearly useless. It’s like the debate over campaign tactics and spending in political science – the evidence seems to suggest that most of it is useless or limited in effect compared to the “fundamentals” of an election, but is that because they’re simply useless, or because they become nearly useless when everyone is using them?

      • JPNunez says:

        Well the interpretations are wildly different tho.

        If lobbying just cancels each other out, it’s dangerous to cut off lobby budget, because in that case maybe your opponents’ lobbying won’t cancel out and it will work against you.

        I don’t doubt there are instances where lobbying is useless, maybe even lots of them, but I also find weird discussing this in the same page where we are discussing whether IP owners will lobby for another copyright extension before 2023. And they also lobbied for the DMCA, net neutrality, etc, etc.

        • Protagoras says:

          I don’t know, I’m fairly skeptical of the scale of the benefit of lengthy copyrights to copyright holders (or of piracy suppression). I recall studies showing that most profitable creative works make almost all of their profit in the first year after publication. Extracting profit from old works may be less worthwhile than investing in new works, though of course the lawyers will try to convince their employers that it’s worth doing. They might also make more money producing cool works based on the older property of others than they do milking their own old properties. In the case of piracy it may do more to give works publicity than to undermine sales (some studies have suggested that). That they get the legislation they want does not guarantee that the legislation actually profits them.

          • JPNunez says:

            It’s hard to say if Disney would hurt or benefit from losing Steamboat Willie. What if someone makes a Steamboat Willie-Mickey spinoff and it is wildly successful and popular? does that benefit Disney, who holds the rest of the IP/trademarks of Mickey? I would think so, as they’d be on a better position to exploit a sudden Mickey craze.

            But as time moves forwards, they would lose more and more of Mickey and that advantage goes away. There’s no good reason, in the long run, to give up such a monopoly, particularly if your loss only becomes bigger with time.

            Luckily the current Disney moneymaker IPs are far more recent (Marvel and Star Wars, and the Pixar films) with the oldest one being Spider-man from 1962 which puts losing him at a p good distance.

            But they also hold a v profitable stable of “Disney Princesses” whose loss to the public domain would start only 9 years after Steamboat Willie.

            I don’t see any reason to not at least try to get another copyright extension

          • albatross11 says:

            The present value of money to be paid to me 75 years in the future is really low, like three cents on the dollar or something (assuming I entered the formula right in Excel, and assuming a 5% discount rate). So adding years of copyright protection is an incredibly inefficient way to add extra incentive for me to produce things worth copyrighting.

          • @ albatross11:

            So adding years of copyright protection is an incredibly inefficient way to add extra incentive

            Not as inefficient as you think. The nominal interest rate includes an allowance for expected inflation. Real interest rates, nominal rate minus the inflation rate, tend to be about one or two percent, not five percent. At a one percent discount rate, a dollar 75 years in the future has a present value of about 47 cents. At two percent, about 23 cents.

            It’s the real interest rate that is relevant because inflation will raise the nominal value of what your copyrighted work sells for.

          • christhenottopher says:

            The long term profitability of the IP itself may be limited, but I could see a potentially important effect in holding off rival media companies. Audiences are more likely to seek out known IPs than new ones and thus a new company could have an easier time building a fan base by working off of known stories than trying to present a new one. Getting funds from investors is certainly easier enabling higher quality production from a new studio. So the effect may less be because “I want to make profits off this IP” and more “I want to make it harder for start-ups to achieve success and challenge me in the market over the long term.” An interesting test here would be the frequency of new media company starts in places with weak copyright vs places with stronger copyright.

          • albatross11 says:


            Wow, thanks for pointing that out. It never occurred to me that I should be thinking of the real interest rate there!

        • Swami says:

          Not sure why anyone is even slightly surprised by this. Classic Red Queen effect. In an arms race, there is frequently no net gain for the participants. Were the researchers not aware of or expecting this outcome?

    • Le Maistre Chat says:

      For most of my adult life, I falsely believed that the Presidency and Senate seats went to whoever spent more money, and the only thing that trumped being able to buy office was gerrymandering.

      • Well, the last data I saw was that every factor of 2 by which one candidate outspent another shifted the popular vote about 1% in their favor. Hardly decisive at the margin but it does mean you have to play the game so you don’t get outspent by a factor or 100 or something, especially since there are large contingents of ‘always votes Democrat’ and ‘always votes Republican’ that won’t be swayed making that 1% more valuable.

        • Squirrel of Doom says:

          These studies are easily destroyed by correlation:

          The most popular candidate will naturally get both the most votes and the most money donations. The natural naive analysis will then “prove” that the money caused the votes.

          If your study doesn’t explicitly mention they’ve controlled for this, you can assume they didn’t.

          • Yes, that’s why the study compared money raised by the politician versus money spent by previously wealthy politicians out of their own pocket. The former correlates with victory much more than the later but the later does still have an influence.

    • Brad says:

      I wonder if the juxtaposition of that link paragraph and the next one (re: Mickey Mouse copyright) was intentional. Surely, the latter is an example of lobbying effect that did help a company and its shareholders?

      Likewise other very concrete efforts. Surely the hedge and PE funds got their money’s worth in that the recent tax bill did not eliminate the carried interest loophole despite near unanimous agreement by everyone other than them that it ought to be.

      • Protagoras says:

        As far as I know, the carried interest loophole is something that benetifs the management of hedge funds, not the funds themselves directly or the shareholders of the funds. I’ve already commented on copyright. And in any event, for the thesis of the paper to be true, it would only be necessary that the revenue from lobbying be no greater than could have been earned by other uses of the time and resources, not that the revenue be zero. A final point; successful lobbying may carry a cost in bad publicity, as well as greater oversight and meddling from a government that thinks it deserves a return for the favor.

        • Brad says:

          As I understand the management of funds are generally separate entities from the funds themselves, those entities have profits and losses, and are the ones that paid the lobbyists. The major exception would be vanguard which has a co-op structure where the funds on the management entity.

          So I think it makes sense to look at the cost/benefit from the POV of view of the managing entities rather than the funds. That said, if the managing entities themselves have employees and owners, we’d want to look at the interests of the owners rather than the employees.

    • The Nybbler says:

      Microsoft didn’t start lobbying in a big way until the antitrust case against them. So lobbying may not help, but not-lobbying can be an existential threat.

    • herculesorion says:

      Lobbying is Pascal’s Wager for businesses (just like so much else in business operations, really.)

      Everyone *HAS* to do it, even if it doesn’t seem to work, because if it turns out that it actually does work than anyone who didn’t do it is boned.

      • baconbits9 says:

        Neither spending money at all on lobbying nor spending more money on lobbying over those years seem to help companies, and for that matter contributions to political action committees don’t work either.

        If everyone had to do it then spending money at all would be > not spending money and would show up in the analysis in some way.

    • Edward Scizorhands says:

      You don’t even need an understanding of politics. Just an understanding of how corporations work will point out that they waste a lot of money on whatever the insiders’ hobby-horses are.

    • shakeddown says:

      I’d also add that lobbying suffers from Moloch – a lot of companies end up lobbying for causes which are unprofitable for them (like Coca-Cola and Pepsi being part of a lobbying group that advocates against climate regulations even though their own climate record is actually pretty good).

  9. BBA says:

    For the past forty years, every time new works were about to come into the public domain, Congress altered extended copyright law to keep them private.

    True, but “every time” would be once, in 1998. The 1976 act (effective 1978) doesn’t really count in my book, because copyrighted works were expiring regularly up until then.

  10. Le Maistre Chat says:

    I wasn’t sure what to think of the Damore spectacle, but between him letting his lawyers do the talking and the revelation that Mencius Moldbug eating lunch is considered a security breach by Google, I have hopes of seeing Google suffer.

    • Scott Alexander says:

      Him letting lawyers do the talking sounds like a good idea, but the brief seemed to me to be really horribly written, in a way that suggested Damore writing it and using it to pontificate. Sentences like “Google employees and managers strongly preferred to hear the same orthodox opinions regurgitated repeatedly, producing an ideological echo chamber, a protected, distorted bubble of groupthink” (and there are lots of these) don’t strike me as normal legalese. I’m actually pretty confused by this given his lawyer’s apparently good reputation.

      • Le Maistre Chat says:

        Agreed, and the foolishness of representing oneself in court had me going “meh” about the case. This is curiously confusing.

      • Evan Þ says:

        I’ve heard that briefs can be written for public consumption more than for the court, and courts tend to just ignore the florid rhetoric.

      • bbeck310 says:

        Patent litigator here – the filing isn’t a brief; it’s a complaint, which is the starting document for a lawsuit. In most lawsuits, a complaint is a simple statement of the facts necessary to make a legal claim for the plaintiff (Federal Rule of Civil Procedure 8 technically requires that the complaint be a “a short and plain statement of the claim showing that the pleader is entitled to relief”). In high-publicity cases, however, it’s common for lawyers to prepare complaints with a lot more invective and and unnecessary details. This even happens in patent litigation – I once had a client who wanted a complaint filled with tons of unnecessary detail on the thieving Taiwanese manufacturer who was supplying all the infringing defendants.

        Now, if you see that kind of unnecessary invective in a brief, someone’s probably doing something wrong. But this is the kind of complaint I would expect to see in a high-publicity case like Damore’s brought as much to embarrass the defendant as to win damages.

        • howardtreesong says:

          Commercial litigator here. +1 to this comment. I generally believe that lengthy narrative complaints like this one can be a) effective, in that everyone knows about this complaint and Google is thus very likely to take it very seriously and b) risky, in that Damore risks credibility shots against him during discovery if there are any factual or context mistakes in any of what he has written.

          I suspect that most judges and clerks will read this complaint with some interest. Bbeck310 is exactly right, though, that judges almost universally dislike invective in briefs and arguments. Parties do it all the time, but it’s distracting to the merits and often creates invective feedback loops that almost always end up being a giant waste of time and money.

        • Squirrel of Doom says:

          Right. The goal here might not even be to win the case on the legal merits.

          It might be to dig up so much embarrassing dirt in discovery that even when Google “wins” the case in court they lose so much more in the court of public opinion.

          Which in turn might force Google to settle the case, thus actually providing Damore a legal win of sorts.

          At least that’s what I read somewhere. I’m not smart enough to know if that’s a reasonable prediction.

          • An alternative objective, if the motive for the suit is ideological rather than financial, is to cause Google to substantially reduce the behaviors that Damore is complaining about. That doesn’t require either a win or an out of court settlement, although one or the other would help.

            A lot of the logic of the situation from that standpoint depends on what one believes about the motivations of the people running Google. At one extreme we might suppose that the decision makers don’t themselves care about m/f ratio, ideology, or any of that stuff but think their employees do, and are creating an environment friendly to one ideology and hostile to another because they believe that attracts more people than it repels and can be done with less effort than the alternative, since lots of employees will voluntarily help with doing it.

            On this interpretation, if the case creates significant costs for Google in bad publicity, or if Damore wins and that verdict increases the future costs of continuing the policies he complained of, one would expect Google to shift to less ideologically biased behavior.

            Assume, at the other extreme, that current policy reflects the strongly held ideological views of the people running Google. Google is rich enough so that it can afford to lose quite a lot of money through bad publicity, out of court settlements current and future, and the like. If the people in control are committed social justice warrior types they are likely to accept those costs in order not to be seen, by themselves and their side, as giving in to pressure from the enemy.

          • Ratte says:

            It might be to dig up so much embarrassing dirt in discovery that even when Google “wins” the case in court they lose so much more in the court of public opinion.

            That’s likely to be very effective. As a programmer, I’ve lost much of my esteem for Google and any desire to work there due to the complaint alone – the political atmosphere sounds oppressive, and I can’t trust any employer that allows people in supervisory positions to openly express animus towards any race or gender, as Google appears to have done here. I’ve heard similar sentiments from other people in my field, as well.

          • And part of my reaction was to wonder if I should sell my Google stock–not because I expect the case to lower its value, although that’s possible, but because I would prefer not to be a partial owner of a firm that acts in ways I disapprove of.

          • Matt M says:

            As a programmer, I’ve lost much of my esteem for Google and any desire to work there due to the complaint alone – the political atmosphere sounds oppressive

            What percentage of programmers do you think might feel this way?

            And how would you guess that compares to the percentage of programmers whose esteem for Google might increase, because they think that Nazis like Damore deserve to be oppressed, and they actively desire to work for companies who are principled enough to stand up against such evildoers.

          • veeloxtrox says:

            Am at Google. The internal split (of people who cared to voice opinions) was split about equally between A: “He might be right”, B: “Anyone in group A is a bigot”, C: “I didn’t think Google was oppressive but group B makes me think it is an issue”, D: “We shouldn’t be talking about this/ I am tired of it”.

            I would guess that externally about 25% of programmers had a worse opinion (of some degree) and 25% have a better opinion to some degree.

          • Matt M says:

            I would guess that externally about 25% of programmers had a worse opinion (of some degree) and 25% have a better opinion to some degree.

            So, if your estimates are accurate, this is a net wash for Google. Any negative PR is counterbalanced by positive PR.

            I think too many rationalists are predicting bad things for Google here by assuming that most people think discriminating against people with non-leftist opinions is wrong. I am here to say that no, not only does about 1/3 of the country think discriminating against Damore is acceptable, they think it’s a very good thing and the only problem with society is that more people aren’t doing it.

          • Le Maistre Chat says:

            @Matt M: so let Google keep doing it, and let’s everyone who’s not a Leftist boycott Google.

          • Squirrel of Doom says:

            Addendum: My source was a Megan McArdle column:

            Also, my guess for why Google does this is not so much that the management has a strong personal devotion to intersectional feminism or whatever, but that of all the forces they’re squeezed between here they’re most afraid of some kind of of lawsuit/legislation over gender discrimination/imbalance.

            One thing is certain, when you have as much money as Google, everyone wants some of it!

          • Steve Sailer says:

            From what I’ve heard, Google CEO Sundar Pichai played Pontius Pilate on Damore. He really wouldn’t have wanted to fire Damore, but Youtube boss Susan Wojcicki, who was Sergey and Larry’s landlady back in the 20th Century and then was Sergey’s sister-in-law for awhile, had her feelings hurt when one of her five children asked her about Damore’s memo, so Damore had to go. (Susan sister Anne, who used to be married to Sergey before taking up with slugger Alex Rodriguez, is the CEO of 23andMe, which is pretty funny when you think about it. I’m not sure, however, that Susan ever got the joke.)

            And, yeah, reports of this kind of high-estrogen soap opera lowered my estimate of the chance that Google could successfully build self-driving cars.

          • Matt M says:

            And, yeah, reports of this kind of high-estrogen soap opera lowered my estimate of the chance that Google could successfully build self-driving cars.

            Do you think this sort of nonsense isn’t happening at every other company working on self-driving cars?

          • rlms says:

            I am definitely going to start describing things as “high-testosterone nonsense”.

          • The Nybbler says:

            I am definitely going to start describing things as “high-testosterone nonsense”.

            Stuff — including dominance games among managers — gets described that way all the time; it’s not taboo.

            As for Susan Wojciki’s little story about her daughter, if Steve Sailer believes that one is literally true, he’s the only one.

            BTW, the self-driving cars are over in Waymo (which is not part of Google), so separate from this bit of drama.

          • Randy M says:

            I am definitely going to start describing things as “high-testosterone nonsense”.

            Sure beats the phrase “toxic masculinity” anyway.

          • Ratte says:

            @Matt M: The ‘political atmosphere’ I was referring to wasn’t the specific beliefs being debated, but the web of organizational blacklists, loud public arguments, purity tests, grudgeholders archiving statements and passing them to their lawyers, physical threats, and other general nastiness revealed by the complaint.

            Even if you think Damore deserved to be fired, no one wants to work in an office where supervisors openly brag of discriminating against people, being friends with someone on a secret organizational shitlist could have serious impacts on your career, and any one of your co-workers could be combing through your social media and message history and archiving anything that could be characterized as hostile. Look at the people whose communications were named in the complaint – they’re certainly sweating it now (and I noticed that several no longer list their employment at Google on social media).

            It’s a freaking job, not an election. I just want to get paid and make cool stuff!

          • Deiseach says:

            Susan Wojcicki …had her feelings hurt when one of her five children asked her about Damore’s memo

            I’m agreeing with The Nybbler here – whenever some opininator or other person making a high-profile mountain out of a molehill drags out the anecdotes about their cute wobbling-lipped little tyke (or older child) asking “Mommy (or Daddy), why did the nasty man say that nasty thing?” and then Mommy (or Daddy) writes an entire screed about how this ripped out their heart and they had to curl up in a foetal ball on the kitchen floor because how can they explain to Cute Blond(e) Tyke about the cruelty of the world, how can they look into the eyes of innocence and destroy a child’s trust in the goodness of human nature – I’m calling bullshit. It’s just another way of using “Won’t anybody think of the children?” – I don’t care or it doesn’t affect me, that’s not why I’m putting out a hit on this guy, but my sweet little baby – I have to protect them!

            Remember all the stories about weeping toddlers convinced Trump was going to come kill their families in the night in the aftermath of the election? Or in the run-up to it? Like the tweets in this article? Yeah, I don’t think so – if any kids were saying stuff about Trump it was because their parents were having hysterics about the election in front of them and scaring their kids themselves:

            Our kids might remember this year’s craziness – my 6 yo asks about Trump. My 9 yo is terrified of him. Scaring my kids is not cool.

          • Matt M says:

            no one wants to work in an office where supervisors openly brag of discriminating against people,

            It’s a freaking job, not an election. I just want to get paid and make cool stuff!

            This, specifically, is what I disagree with.

            I think you are assuming your own feelings are universal. They are not. The reactions to the Damore memo indicate that there are, in fact, a whole lot of people who DO want to work in an environment where people are openly discriminated against, so long as those people are white males with conservative opinions.

            It’s not even that they “don’t care” that Damore is discriminated against, they care about a lot… in a positive sense! They want him to be discriminated against. They threatened to resign from Google if Damore wasn’t fired.

            Some people do want their workplace to be more political, so long as it’s the “right kind” of politics. They love that they’re allowed to spend work time and work resources discussing their experience as a giant yellow dragonkin – if that was taken away, they’d quit and then sue for discrimination against dragonkind.

            And I’d suspect that while these sorts of people might be minorities in the workforce as a whole, they probably aren’t minorities among the demographic Google cares about – tech people who live in the bay area.

            The values of fairness and non-discrimination and “leaving politics at home” are not universal. Not even close.

          • AnonYEmous says:

            Yeah, just look at veeloxtrox’s comment if you want to see how certain Googlers think. I guess technically group B might want to keep their politics out of the workplace, but that type doesn’t tend to – and judging from some details out of the Damore lawsuit, that type didn’t in many cases.

          • Ratte says:

            @Matt M: Again, I’m not referring to politics in the left-right sense but the Louis-XIV’s-royal-court sense, where you have to spend more time worrying about whether you can be seen with the person you’re talking to or if you agreed with the boss sufficiently loudly enough to be noticed. No one likes this and it’s the death of productivity.

            @AnonYEmous: “I don’t like this kind of political atmosphere” encompasses groups A, C, and D, for different reasons. Even Bs get tired of a nonstop drumbeat of politics, as many of my friends on Twitter have.

            edit: On an unrelated note, Steve Yegge just quit Google citing (among other things) the company being ‘mired in politics.’

          • Matt M says:

            I’m not referring to politics in the left-right sense but the Louis-XIV’s-royal-court sense.. No one likes this and it’s the death of productivity.

            Okay… but I don’t see how this is relevant to Damore. The only dimension on which anyone cares about him at all is left-right. The point here is that you don’t have to worry about the royal court… unless you’re a Nazi, in which case, thank God we have those royals to give you the punishment you so justly deserve!

            All the gay-trans-latinx-otherkin people aren’t looking over their shoulders at all. To the extent that they play politics, it’s because they want to. They have no fear of any of this stuff. Nor should they.

          • albatross11 says:


            I am extremely skeptical that you have an accurate picture of the views of the average Google employee. Is there data anywhere on this? Because what you’re providing seems like a caricature.

          • Nornagest says:

            had her feelings hurt when one of her five children asked her about Damore’s memo

            Yeah, if you believe that, I have a bridge to sell you. If your children can’t handle the cruelty of modern politics, it’s because you’ve been scaring them about the cruelty of modern politics. There is no one else to blame — as long as there aren’t tanks in the streets or secret police hauling off relatives, your kids don’t have the context to scare themselves. I remember being that age. I had zero independent political opinions.

          • albatross11 says:


            My kids (16, 12, 8) ask questions about politics and express opinions all the time. They don’t necessarily understand everything that’s going on, but they do sometimes get upset or worried about stuff, want to go to protests, etc. My eight year old daughter was very upset by the election of Trump, and while we discuss politics at home, my sense is that she picked this up from seeing or hearing stuff outside the home.

            One sideline: my 16 year old had the AP US Government class last year. I’ve been really impressed by how much that has changed the quality of questions he asks and observations he makes–the class seemed to really do a nice job of giving him a basic understanding of how various levels of the US government work and are supposed to work, and giving him at least a little context for understanding current events.

          • Nornagest says:

            Sixteen is a little older than I was going for, since I can’t see anyone considering it a horrifying revelation if their sixteen-year-old daughter asked them some awkward political questions. I’ll concede that younger children might end up getting scared outside the home if the other adult influences in their life are pointing that way, but I doubt that’s what happened in the Wojcicki/Damore case — an elementary school teacher would talk about Trump’s election, but probably not about hiring practices at Google.

            I recall my own AP Government class being about when I started developing an independent understanding of politics, too, so I guess they must be doing something right.

          • Ratte says:

            @Matt M: I guess your experience is different, then. I mostly work with adults.

          • taradinoc says:

            All the gay-trans-latinx-otherkin people aren’t looking over their shoulders at all. To the extent that they play politics, it’s because they want to. They have no fear of any of this stuff. Nor should they.

            According to the recent Wired article, they do have fear of this stuff being used against them. They didn’t think they had to look over their shoulders, but now they’re realizing they do, because the same forms of retaliation (leaks, HR complaints) are being directed at their faction too.

            But also, few if any of them were seeking out this environment when they joined Google. They may participate because it’s preferable to quitting, or because from up close, it feels more like righteous self-defense than a stressful, soul-crushing time sink. But had they known in advance what it’d be like, they might have avoided the situation.

          • The Nybbler says:

            According to the recent Wired article, they do have fear of this stuff being used against them. They didn’t think they had to look over their shoulders, but now they’re realizing they do, because the same forms of retaliation (leaks, HR complaints) are being directed at their faction too.

            Fong-Jones is a major culture warrior and should not be considered a reliable source. That said, it’s clear that leaks are now considered fair play by both sides; the leaking of the Damore memo seems to have broken the taboo. As for HR complaints, the Wired article claims that one employee got a verbal warning for posting “fire all the bigoted white men” (I believe this is the meme at page 121 of the lawsuit, labeled “47” at the top). Another person, on the other hand was publicly rebuked by top management (page 104, marked “30”) for posting a question to the “Diversity TGIF” dory for asking if diversity efforts would address the underrepresentation of whites. This isn’t in the lawsuit, but if I recall correctly it was a response to another question asked by an Asian Googler who wanted to know if the diversity efforts would result in discrimination against Asians, who were overrepresented; that Googler was also reprimanded.

            So indeed, it is possible for SJWs to do things which get them a response from HR and management. But they’ve got a LOT more leeway.

        • Lasagna says:

          Yup. Exactly right. I was a commercial litigator for about 15 years, and “know who your audience is” is a basic tenant when drafting anything.

          For most everything else, the audience is the court, so like Bbeck says, drafting motion papers with this kind of kitchen sink approach and emotional appeal instead of statements of facts and law is generally a bad idea. But the complaint? Embarrassing the opposing part with a salacious description of their misdeeds can help pressure them to settle, or to change their ways, or just infuriate them.

          The media would not be publishing articles on this complaint if it read like your typical cookie-cutter personal injury complaint. Add in enough of the lunacy going on at Google, though, and it’s a different story.

    • WashedOut says:

      I think I share the same wish as you, but I doubt any legal team on the planet can out-maneuver Google in court (and sustain it), if for no other reason than Google’s budget >>> everyone else’s. When it comes to high-profile, potentially Orwellian-nightmare-exposing lawsuits, would they not rather pay people to shut up forever and go away?

      • Le Maistre Chat says:

        Pay people to shut up? Google couldn’t pay Moldbug enough to shut up. Dude basically thinks he’s Zoroaster as far as wanting to destroy those who serve the Lie regardless of personal or social consequences.
        Crushing conservatives and sundry witches in court because of their budget is a different matter.

      • Le Maistre Chat says:

        I sincerely wish you weren’t joking.

      • howardtreesong says:

        I’ve never studied the issue formally, so I’ll just speculate that the amount of money that a company pours into litigation is only weakly correlated to success. Put a little differently, the impact of Google putting $6 million into defending this case won’t have much more impact, if any, than Google putting in $5 million. There likely wouldn’t be any way for Google to put $100 million into it under any circumstance, so the idea that massive legal expense necessarily leads to victory really isn’t right.

        I’m not a class-action expert, but I know at least a little about it. The rules don’t permit a payoff to the named plaintiff in order to get rid of the entire class. The real key to these cases is class certification, which is the procedure in which a court determines if the set of plaintiffs described in the complaint is a proper class. The question there is whether individual issues predominate over common issues (which would result in no certification) or whether common issues predominate over individual ones (which would result in a certified class). There are other requirements, but that is the usual battleground.

        Here, there’s also the underlying question about whether or not there is liability even if you take all of Damore’s fact claims as being true. There are different procedures to tackle those issues — typically either motions to dismiss or motions for summary judgment.

        • WashedOut says:

          I take your point and thank you for stating it.

          When I picture Google vs. Some Guy et. al. i’m reminded of the cases I came across when I worked in forensic engineering. When Joe Schmoe’s Landrover did something unexpected on the highway and killed his family, Landrover would respond to his lawyer’s request for information by sending them a truckload of documents that would take 1000 lawyer-hours to sift through. I know there are some procedural rules around it but that was the gist of it.

          Similarly, Google can probably afford to make Damore’s (and anyone else’s) life as miserable as possible in the process.

          • howardtreesong says:

            Almost all discovery is now done electronically and is machine-searchable, and there is commercial software available that is quite strong at sifting relevant documents from a mountain of trash. Your Landrover anecdote sounds entirely correct historically, if somewhat less relevant now. I also suspect without knowing that Damore’s case is sufficiently public and political in nature that he may be able to raise capital to fund expenses to deal with burden issues like this.

            I absolutely agree that Google is likely to find every single thing that Damore has ever said or done online, and he’ll have to answer for anything that undercuts his credibility. And if he has any serious skeletons, he will be miserable indeed.

          • John Schilling says:

            I absolutely agree that Google is likely to find every single thing that Damore has ever said or done online, and he’ll have to answer for anything that undercuts his credibility.

            Right, but the joys of discovery mean that Damore’s lawyers are likely to find every vaguely relevant thing that Google’s suits have ever said or done in the quasi-privacy of their corporate email accounts. If the worst Damore has to answer for is the manifesto and the FIDE fib, he may consider that a fair trade.

            Any third party paying for Damore’s lawyers, will likely consider that a very good trade.

          • AnonYEmous says:

            I also suspect without knowing that Damore’s case is sufficiently public and political in nature that he may be able to raise capital to fund expenses to deal with burden issues like this.

            For those unaware, he’s got a fundraising campaign for his lawsuit:


            only $25,000 as of writing this, but that’s $25,000 more than I’d be able to raise if my boss fired me so *shrug*

  11. The story on bias has apparently got the historical origin of the term wrong, at least judging by a quick google:

    In the old game of bowls, it was a technical term used in reference to balls made with a greater weight on one side (1560s), causing them to curve toward one side;

    the anxiety dream about not being allowed to graduate from high school because one signed up for a math class then never attended it?

    My version is the anxiety dream where I’m supposed to be teaching a class and realize I haven’t been showing up to do so.

    • pdbarnlsey says:

      Lawn bowling balls are still very much biased, in this sense. Whereas a titled lawn wouldn’t be much help to anyone, I can’t think.

      The sport has enjoyed a semi-ironic resurgence in Australia, frequently as a team building activity/excuse to drink in the sun.

    • CatCube says:

      My version is the anxiety dream where I’m supposed to be teaching a class and realize I haven’t been showing up to do so.

      It’s weirdly awesome to find out that professors have this from the other side.

    • ManyCookies says:

      Wait so nobody shows up for dream class!? That’s a huge weight off my dream shoulders.

      Short film idea: both the students and the professor aren’t notified about a small class until pre-finals week. The professor is trying to figure out what the (nonexistent) guest lecturer covered so he can write a final, while the students bluff their attendance and individually try to steer the final’s material towards their personal comfort area.

    • phisheep says:

      Although the bias in bowls was originally made by a weight in the ball, in modern times it is made by the *shape* of the ball – it is turned to a (very slightly) ovoid shape giving a predictable tendency to curve to one side or the other depending which way you roll it. You can buy bowls with different degrees of bias – I have a set of my grandfather’s bowls turned from lignum vitae with medium bias.

      Slanted – well, domed – bowling greens do exist in the game of “crown green” bowls played in the north of England, but although the slope of the green makes for interesting play (for example, shots with an S-shaped profile as the slope of the green and the bias of the bowl work against each other) it does not favour either player, other than the more skilled one.

      • Kipling has a story somewhere about a badly warped pool or billiards table somewhere in British India. People who have become skilled at playing on it regard playing the game on a flat table as much too easy.

    • Protagoras says:

      Also had that one, though I continued to have the studenty dreams well after I stopped being a student (I don’t think I’ve had the student one for a while, but like most people I don’t remember dreams terribly well so I can’t say for sure how long it has been).

    • MaxieJZeus says:

      Professor’s version of the anxiety dream: This one seems to be pretty common among a certain population as well.

      When I was in grad school, my advisor out of the blue asked me if I’d started having the dream yet where I’d forgotten about a class that I was supposed to be teaching, not taking. I bolted upright and said “Yes! Just a few weeks ago!” I was much struck at the time by it, as it was the opposite of the (for me common) dream about forgetting to attend a class I’d signed up for. The “teaching variant” wound up replacing the other one, and recurred too often for comfort.

      In my case, contra ManyCookies’s suggestion, I would dream that the students had been attending the class, and had been manfully trying to carry on without me, and boy were they disappointed with me when, shamefaced, I showed up on the last day to apologize to them — their expressions were always most reproachful. Rather worse were the administrators, and the dream would usually pass away with a sound like thunder as some dean or department chair asked me to come into their office …

  12. entobat says:

    Why is no one asking the important question: does political lobbying improve health outcomes?

    • sohois says:

      Well if lobbying is negatively linked to firm performance, and on average worse firm performance leads to a weaker economy, and a weaker economy leads to lower tax revenues, and lower tax revenues lead to less government spending, and less government spending means fewer resources for medical care, and fewer resources mean people receive less medical care overall, and less medical care leads to better health outcomes, then we can say that yes, political lobbying does improve health outcomes!

      (cue Libertarians pointing out that governments never reduce spending when tax revenues decrease or something, and the whole chain of reasoning falls apart)

  13. meh says:

    I’ve had the dream at different times for nearly every stage of education, including a program I transfered out of. I always my current age and stage in life though, but for some reason back at school.

  14. Charles F says:

    The commentary about the Ta-Nehisi Coates link seems pretty wrong. Maybe there are tweets somewhere accusing Coates of white supremacy, but they’re not mentioned at the link. The tweets shown there and the original criticism they’re quoting do not accuse Coates of white supremacy. They accuse him of blowing white supremacy out of proportion and taking an overly simplistic view of the world where everything can be reduced to a black/white tribal conflict.

    • antilles says:

      No, this is also incorrect. That’s Richard Spencer’s take, who is a moron. Cornel West wrote an article in The Guardian which was frankly pretty embarassingly bad, accusing Coates of being a neoliberal shill, one, and two being inadequately invested in the black freedom struggle and not putting enough focus on possible solutions and heroic figures vs. just describing the problems and history of white supremacy. This is mostly because Cornell West is an old man with a hyper-fragile ego and he has various grudges against Coates. Anyway because of the fracas this kicked up – and because many people will still side with Cornel West and/or leftier-than-thou-takes and/or against people they perceive as Obama fanboys – he left Twitter again.

      • Ninety-Three says:

        What’s wrong with West’s critique? His point seems to be “White supremacy is Coates’ Great Satan which he blames for everything wrong in the world. In doing so he implicitly absolves all other systems of wrongdoing.” West picks his hobby-horse of capitalism and calls Coates a neoliberal, but plug in whatever system you personally think is the real wrongdoer: isn’t Coates minimizing its evils?

        • Anonymous Bosch says:

          West was extremely sloppy with his specifics, and most of the things he accused Coates of ignoring (drone strikes, for instance, or patriarchy) were in fact issues he’d written and blogged about multiple times. I’m sure it’s always possible under West’s worldview to say one isn’t putting enough emphasis on this or that, but that would make the pointless concern trolling too obvious.

      • HeelBearCub says:

        is is mostly because Cornell West is an old man with a hyper-fragile ego and he has various grudges against Coates.

        My impression is that grudge holding is sort of West’s thing. Doesn’t really have anything to do with Coates.

      • Conrad Honcho says:

        I read West’s piece last week. Has anyone written a good rebuttal of it?

        • Iain says:

          This hits a couple of the big points. This Vox article is a pretty good summary.

          • Conrad Honcho says:

            Well that was interesting. I can’t really take sides because I disagree with both of their world views. They’re arguing over who’s better confronting white supremacy when I don’t believe white supremacy exists. White supremacist nations don’t have diversity lottery immigration systems, don’t give non-whites political power, don’t have legally sanctioned programs discriminating against white people and in favor of non-whites for university admissions or jobs, etc. And if you ask actual white supremacists, those are people like Richard Spencer who want a white ethnostate specifically because we don’t have white supremacy and they want it. I’m clearly living in a different world than West and Coates.

            But one thing I found very interesting, from The Stranger article and quoted in the Vox article:

            The concept, which West should have given a little more consideration even if he disagreed with it, is this: Whenever blacks show they can rule, white supremacists freak out and do everything they can to bury the evidence of “Good Negro Government.” Obama was “not a revolutionary.” His drones certainly killed thousands of innocent people. He did little to change the bonus culture of Wall Street. Nevertheless, many whites saw his presidency as an abomination: a black man governing the US well by conventional standards. This is one of the themes Coates explores in his book. It’s not so much about a fear of a black planet, but the fear of Good Negro Government.

            When my liberal friends have said Obama was the best president of their lives, I think they’re telling the truth. But Mudede seems to think that when conservatives criticize Obama for setting the Middle East on fire, driving healthcare costs through the roof with Obamacare, Title IX insanity, unconstitutional executive orders, etc etc, they’re lying. In our heart of hearts we love the way Obama governed and are just mad he was black. This is a definite failure in modeling the outgroup.

          • Iain says:

            They’re arguing over who’s better confronting white supremacy when I don’t believe white supremacy exists.

            Modern-day America is not, on paper, a white supremacist state. You don’t have to go all that far in the past, though, to find an America that deserves that label. (The Civil Rights Act is the obvious example; as a less well-known example, consider red-lining.) That doesn’t disappear overnight. You may not have white supremacy now, but you are living with its legacy. You could say that America is post-white-supremacist: no longer officially discriminatory, but still working through the aftermath of a time when it was.

            (At this point it is traditional to point out that black people have an incentive to overestimate the effect of past injustices on modern problems. This is true, but goes both ways: there’s just as much incentive for white people to underestimate it. It’s a lot easier to point out bias in others than to grapple with your own.)

            But Mudede seems to think that when conservatives criticize Obama for setting the Middle East on fire, driving healthcare costs through the roof with Obamacare, weaponizing the DOJ and the IRS against conservatives, etc etc, they’re lying.

            Note that Mudede’s article doesn’t say anything about “conservatives”. You filled that in yourself. Mudede just says “white supremacists” and “many whites”. Even if you disagree with his assessment of how many white supremacists there are, the underlying argument still holds: people who are invested in white supremacy are going to have a natural bias towards seeing Obama’s actions negatively.

            There’s lots of evidence to show that people assess policy along partisan lines. See, for example, how perceptions of the economy flip-flopped from October 2016 to January 2017. Similarly:
            – if you are upset about Obama’s handling of Libya and Syria, but not about Bush’s handling of Afghanistan and Iraq, you might be seeing policy through a partisan lens.
            – if you are upset about Obama politicizing the DOJ, but not about Bush or Trump doing the same thing, you might be seeing policy through a partisan lens.

            Obviously, “partisan” doesn’t equal “racist”. But it also doesn’t disprove it. The real answer is somewhere in the middle: some conservatives disliked Obama on pure policy grounds, while others (say, this guy) were also influenced by race. Your disagreement here is a quantitative one, not a qualitative one.

          • Conrad Honcho says:

            Mudede made two statements about who was upset about “Good Negro Government.” One was “white supremacists” and the other was “many whites.” Are these the same people, or not?

            And yes, I’m very unhappy with the Bush administration. I’m a conservative or paleoconservative, not a neoconservative. Bush/Cheney made me basically stop identifying with the Republican party. What I like is small, decentralized government, and Cheney said “the era of small government is over.” What I wanted was the elimination of the department of education, and instead got No Child Left Behind. What I wanted was government out of healthcare and instead got Medicare Part D. What I wanted was the end of the welfare state and instead I got “faith-based initiatives.” What I wanted was a republic, not an empire, and instead I got wars of choice started under false pretenses. I was not at all happy with the presidency of George W. Bush and it had nothing to do with his race. Similarly, I was not at all happy with the Obama presidency, and that also had nothing to do with his race.

            Still, I think it’s interesting that Mudede thinks that many whites (or perhaps only white supremacists?) secretly liked Obama’s administration, and are lying about it to cover up the fact that all the good stuff they secretly like was done by a black man. No, I think people who don’t like the Obama administration’s governance genuinely don’t like his policies and actions irrespective of his race, and the people who don’t like Obama because of his race don’t secretly think he did a great job and are lying about it because it was done by a negro.

          • Iain says:

            If you and I and Mudede were all asked to predict what percentage of white people were white supremacists, I’m sure we’d all have different numbers. My point is simply that, once you accept that some white supremacists exist, Mudede’s argument stands: those people are going to be very motivated (consciously or not) to see Obama’s presidency through a negative lens.

            I take you at your word that your opposition to Obama had no racial component. But you are just one person, and it is not safe to assume that everybody else opposed Obama for the same reasons as you. It is, of course, equally wrong to assume that everybody else opposed Obama for racist reasons, and I hope it is clear that I am not arguing for that position. My goal here is merely to convince you that Mudede’s position is not necessarily incompatible with your experience. It does not explain everything, but it is a piece of the puzzle.

            Edit to add: I think it is wrong to interpret Mudede as accusing people of lying about liking Obama’s policies. What he’s saying is this: if you are a white supremacist, you believe that black people should not be in charge. If a black person in charge did a good job, that would be evidence against your belief. People don’t like it when their beliefs are challenged, and they are motivated to explain away such challenges. Therefore, white supremacists will look for reasons to oppose Obama’s policies, even if they would have let them slide under a white president.

            The mechanism shouldn’t be controversial. It is not uncommon to see people here argue that [the left / Democrats / the mainstream media] are opposing Trump for actions that would have been acceptable from members of the in-group. As I mentioned in my previous post, we have plenty of evidence of this playing out in (for example) economic polling. Really, the only question is: of the people with kneejerk reactions to Obama, how many were caused by him being a Democrat, and how many were caused by him being black?

            We can quibble about the proportions, but once you accept that framing, you’re like 80% of the way to agreeing with Mudede.

          • baconbits9 says:

            It is not fair to extrapolate some white racists felt a certain way to using it to defend the position. The quote has the word “many” in it, and the overall tenor is about America, not about a specific sub population of the US, with its relative size and importance given. I have never identified as a conservative, but my reading of these pieces has been that they are in general painting a large portion of the population in a negative light without explicitly doing so.

          • Conrad Honcho says:

            All right then. I’m curious as to who, specifically, Mudede was talking about when he said “Nevertheless, many whites saw his presidency as an abomination: a black man governing the US well by conventional standards. *snip* It’s not so much about a fear of a black planet, but the fear of Good Negro Government.”

            I would think these whites would have to be some large, representative group to have any impact at all. It would be like me saying “Whenever capitalists show they can improve the standard of living for workers, communists freak out and do everything they can to bury the evidence of ‘Good Capitalist Government.’ Nevertheless, many people saw the Trump presidency as an abomination: a capitalist improving the standard of living for workers. It’s not so much about a fear of a Trump presidency, but the fear of Good Capitalist Government.”

            Yes, there are a handful of literal communists in the US (probably more than there are literal white supremacists) who would rather see the economy tank so badly that it would increase class consciousness to the point of sparking Glorious Revolution, but it strongly implies that I think the real reason Democrats (the people who don’t like Trump) say they don’t like Trump is because they’re filthy commies jealous of his success running the government and stimulating the economy, and not because of their clearly articulated positions that they hate Trump because they think he’s a racist, sexist buffoon/fascist.

            So if I could ask Mudede a question it would be: “who specifically were you thinking of that has a fear of Good Negro Government?” If it’s just people like Richard Spencer, then why bother mentioning it, because who cares what Richard Spencer thinks? He has no power or influence and his kind is rarer in America than literal communists. If it’s people beyond that, and you’re talking about Sean Hannity or National Review or Ann Coulter, then you’re deranged because all of these people have clearly articulated ideological differences with Obama, and the idea they think he actually did a great job and they’re all just lying about it because they’re scared of Good Negro Government is ludicrous.

            It all starts sounding like Arguments From My Opponent Believes Something.

            Edit: I think I meant to link the Fetal Attraction essay about how people will frame bogus argument about what my opponent really believes. “Many whites” don’t really dislike Obama’s policies, they’re just scared of a black person demonstrating he can govern well.

            Edit to respond to your edit:

            if you are a white supremacist, you believe that black people should not be in charge. If a black person in charge did a good job, that would be evidence against your belief. People don’t like it when their beliefs are challenged, and they are motivated to explain away such challenges. Therefore, white supremacists will look for reasons to oppose Obama’s policies, even if they would have let them slide under a white president.

            I don’t think that’s the right model of modern white supremacists or nationalists. I don’t think Richard Spencer believes that no black person could manage the government, or be smart or accomplished in something. I’m pretty sure he knows Ben Carson and Thomas Sowell exist. They think they have different interests that are incompatible and should live separately and be self-governed separately.

            I suppose that might just be white nationalists and not white supremacists, but I have no idea who these people are. That someone thinks blacks are so inferior that no black person would be able to do a good job as president is extreme beyond even Richard Spencer and David Duke. Perhaps Anglin, the Daily Stormer guy thinks that? But why Mudede or Coates would be bothering to engage with that extremely tiny fringe minority when discussing race in America I don’t know, unless their perceptions are very warped. But then again, I do think their perceptions are very warped. As I’ve said they see a white supremacist (not post white supremacist) nation that I don’t see.

          • Gobbobobble says:

            My point is simply that, once you accept that some white supremacists exist, Mudede’s argument stands: those people are going to be very motivated (consciously or not) to see Obama’s presidency through a negative lens.

            If the group is small enough then, regardless of its narrow correctness, the argument is a non-sequitur as far as policy or the fuzzier “national conversation relevance” criteria. See also: radical Muslims

          • AnonYEmous says:

            Modern-day America is not, on paper, a white supremacist state. You don’t have to go all that far in the past, though, to find an America that deserves that label.

            i’m not saying this rephrasing removes all the issues people have with the original claim, but it certainly removes a lot of them; it can’t be used to bash basically everyone alive today and it focuses on real and tangible things over feelings and beliefs about the beliefs of others. It’s still vulnerable to over-exaggeration of the current harm caused by the past harms, which you’ve already talked about, but it’s a huge improvement from a lot of the rhetoric of Coates and his ideological brethren.

          • Conrad Honcho says:

            but it’s a huge improvement from a lot of the rhetoric of Coates and his ideological brethren.

            Yes, the rhetoric I hear is “confronting white supremacy” or “smashing white supremacy” or “ending white supremacy.” They’re talking about current and future action. It strongly suggests they believe white supremacy is an ongoing thing and I don’t see that at all.

          • dndnrsn says:

            There are different definitions of white supremacy. It’s one of those things where a popular meaning – a “white supremacist” is someone who wants to keep anyone not white (with what “white” is varying) down through laws or extra-legal violence or some combination, for example. There’s also a meaning assigned to it, usually by someone with a certain sort of social sciences background, which is far more loose. It’s under the former meaning that someone can call Justin Trudeau a “white supremacist” and mean it. It’s a bit more obscure version of how “racist” has several different meanings.

            Personally, I don’t like this habit that has emerged in the social sciences – it makes it difficult and confusing to discuss these things, because having a term that simultaneously has an prescriptive technical meaning and an emotionally charged descriptive meaning is not conducive to reasoned discussion. (In my less charitable moments, I suspect that’s the point – that it’s motte and bailey; describe someone using a term with a charged colloquial meaning and when they object, rebut that you’re using the technical meaning).

          • Randy M says:

            Personally, I don’t like this habit that has emerged in the social sciences

            It’s similar to what Scott called the worst argument, isn’t it? Using a label that to bring in connotations that aren’t merited for the example on their own terms–one shouldn’t react to whatever it is we’re calling white supremacy (Ta-Neisha Coates, wasn’t it? Or disliking Obama) the same way one would to organized lynching–simply because the example is in that category through some technicality or fiat.

          • quanta413 says:


            (In my less charitable moments, I suspect that’s the point – that it’s motte and bailey)

            I have to say, if you can come up with a charitable explanation for the behavior that doesn’t seem laughably bad to me I’d be impressed. Most likely explanation, it’s the rough equivalent of calling a democrat a socialist and then talking about how communists killed millions of people in order to try to make democrats seem evil. It’s bad behavior but also pretty typical. Most charitably, it’s a way to score academic humanities points for bombast and style; all the other effects are just unintended consequences. Worst case, it’s some combination of what you say above with what I think is the most likely explanation.

          • Iain says:

            By way of indirect response: let’s tell a story about Trump.

            For generations, America has been governed by a certain kind of elite: cosmopolitan, intellectual, politically correct, you know the drill. Many Americans have grown accustomed to the idea that this is what American government looks like. (They fervently swear, of course, that they hold no bias against non-elite Americans, although you might have a hard time convincing non-elites of that. Still, I’m sure they’re very polite when they meet Arkansans face-to-face.)

            Suddenly, along comes Trump, flipping the bird to all those elites and their expectations. He’s not an intellectual. He’s not cosmopolitan. He’s certainly not politically correct. Worst of all: he wins.

            Look at the elites in a tizzy. Things are not going as planned. Something, they feel certain, must be wrong. Look at him, sitting there in the White House, as if he belonged. Can’t people see that there’s something wrong? That his presidency is illegitimate?

            He starts making policy. Obviously, those policies are bad — how could they be otherwise? Never mind that much of what he does is standard operating procedure, and they’d never have noticed if, say, Jeb Bush had done the same thing. (They certainly wouldn’t have objected to Obama.) Presidents don’t look like Trump! Presidents are polite to foreign leaders, and apologize when accused of sexism, and are careful to only say politically correct things about immigrants. What is the world coming to, if a person like this could get elected?

            “Tut tut,” say the elite supremacists.

            I hope I have not failed the Intellectual Turing Test too badly. If you do not recognize the story, then the rest of this post will probably not work.

            Assuming that it works at least a little: what’s my point? A few observations:

            1. You might ask: who exactly are the elite supremacists? Where do you draw the line? It’s a complicated question. Relatively few elites will come out and proudly proclaim their hatred of the non-elite; many more will deny it — and even believe their own denial — while acting in various ways that undermine that claim. It’s a spectrum, not a binary.

            2. The effect is not necessarily conscious, or even malevolent. Part of it is just a response to the violation of unstated expectations: “This is how things always worked, but now it’s different, and change feels bad. Explanations for why this changed thing really is bad are going to feel extra-compelling.”

            3. This is, at best, a partial explanation for opposition to Trump. Many people would oppose every last thing Jeb Bush or John Kasich did with just as much fervor. Other people have legitimate disagreements with his policies. It’s hard to discern the true motivation for any particular opponent, and realistically it’s going to be a combination of all three.

            4. Elites and non-elites are going to disagree massively on the size of the effect, and nobody is really in a position to evaluate it objectively.

            5. This story obviously won’t convince elites to support Trump. Realistically, it shouldn’t — even if you eliminate any bias, they still disagree vehemently on policy. Nevertheless, it would be good for elites to take this story seriously: and, in particular, to consider that their viewpoint is not objective.

            PS: Please don’t spend too much time pointing out the flaws in the analogy. I am aware; I could name quite a few myself. It’s an intuition pump, not a rigorous proof.

          • dndnrsn says:


            There’re many better explanations than “this is a deliberate scheme.” The explanation of “whoa, academia” is probably the best one. The tightening relationship between left-wing politics and the academy has been poisonous for both. In this case, it’s led to the volcanic-level hot take of “not being fully on board with The Program is on the same spectrum as being Richard Spencer; it’s just a lesser form”.


            Good post. I mean, my statements above aside, things do happen on a spectrum. However, spectrums end somewhere – definitions of white supremacy that have, at the lowest levels, things that are just indicators of being insufficiently with the program, are like definitions of the autism spectrum that include “occasionally being nervous in social situations” as being on the autism spectrum.

          • quanta413 says:

            The explanation of “whoa, academia” is probably the best one. The tightening relationship between left-wing politics and the academy has been poisonous for both. In this case, it’s led to the volcanic-level hot take of “not being fully on board with The Program is on the same spectrum as being Richard Spencer; it’s just a lesser form”.

            Not sure how this differs from my “most likely explanation”. My “most likely explanation” wasn’t that it’s a deliberate scheme. It’s just that if someone hates the outgroup and then they see a method of tarring the outgroup they’ll copy it. No cabal schemes to compare democrats to communists either; it’s just hating the outgroup. It’s intentional, and it’s poor behavior. But it’s also pretty common, and it’s not a scheme.

          • tscharf says:

            The tightening relationship between left-wing politics and the academy has been poisonous for both.

            I really don’t see how they can be so blind to this. I admired the intent and execution of academia for decades. Now I see an increasingly corrupt influence of politics. Academia is being voluntarily or involuntarily co-opted for political purposes. There is a left wing politics / academia / journalism nexus that is self reinforcing and has damaged these institutions in the minds of the public. Opinion polls bear this out. This nexus believes they are “leading”, but seem to be dropping followers like flies. It’s gotten to the point that if you want to know what a professor from Columbia thinks about a culture war issue you might as well check the DNC website.

            I have a feeling a voter referendum on funding the social sciences wouldn’t end well for the academy today. Public funding shouldn’t be a popularity contest, but the condescension of the public at large from this group is palpable.

            It’s mostly the public face of academia that needs fixed, my daughters are going through school now and they do not report any indoctrination, although they both voted Clinton, ha ha. I can easily tolerate this. There are plenty of people who care about academia being respected, they need to speak up.

          • dndnrsn says:


            Not sure how this differs from my “most likely explanation”. My “most likely explanation” wasn’t that it’s a deliberate scheme. It’s just that if someone hates the outgroup and then they see a method of tarring the outgroup they’ll copy it. No cabal schemes to compare democrats to communists either; it’s just hating the outgroup. It’s intentional, and it’s poor behavior. But it’s also pretty common, and it’s not a scheme.

            But consider: Academia is kind of closed and insular. What would actually work to harm the outgroup, to advance one’s ingroup’s interests out in the world in general, is not necessarily what will advance an individual’s interests within academia (be that in the faculty lounge, or in the fighting for advancement among PhD students, or in the sphere of left-wing campus activism, or whatever). Compare to a political party – while the party needs to advance its interests, overcome its political opponents, etc, within the party if you want to be the candidate or the chief of the party committee or whatever, you’re competing against other people within the party. To some extent, whatever gets said about an outgroup that isn’t present on campus/in campus politics, is really just a byproduct. Or, for another analogy, consider infighting between branches of the military, within branches of the military, etc – this continues to happen during major wars, even.


            I went to school somewhere there was a fair bit of culture warring, but it was easy to stay out of it, by and large. It seems that in the years since I graduated, the culture warring has increased, and has become harder to avoid, but it’s still probably pretty easy to avoid if you just stay out of, say, student politics, etc. People here overestimate the degree to which it’s present on campuses.

          • It’s mostly the public face of academia that needs fixed, my daughters are going through school now and they do not report any indoctrination, although they both voted Clinton, ha ha.

            My daughter’s experience at Oberlin was that the intolerant monoculture was much more the students than the professors. The professors were generally left as well but recognized an obligation not to be biased in their teaching. When one of them said something in class that implied that of course everyone there agreed with his political views my daughter pointed it out to him afterwards, he apologized, and (I think) later apologized in class.

            The attitude of the student culture, on the other hand, was that if you did not agree with the left you were either stupid or wicked.

            (My daughter can correct this here if she wishes–I’m working from memory of what she said.)

        • baconbits9 says:

          I’m tempted to link West’s piece with the suggestion to read it as if it was written ironically.

  15. meh says:

    seems hard to distinguish from affirmative action and various societywide diversity campaigns universal enough that I assume someone would have noticed before now if they were illegal.

    People have noticed, and they are trying. Maybe someone thinks he could be a good case.

  16. Yakimi says:

    Incidentally, there existed one biologist who Marx held in even higher esteem than Darwin: an obscure architect and photographer named Pierre Trémaux, whose deservedly forgotten theories of racial variation Marx once described as constituting “a very important advance over Darwin”. Specifically, Trémaux expounded a literal rendition of the magic dirt theory.

    From “In the Interests of Civilization”: Marxist Views of Race and Culture in the Nineteenth Century,

    Trémaux’s theory relates the nature of the soil to human racial types (a not uncommon kind of argument in the nineteenth century although Trémaux’s version is particularly crude). The nature of the soil, according to Trémaux , changes over time. Older—primary or secondary—rocks are less “perfect” than are rocks of more recent periods. It follows that persons who live on more recent terrain are themselves more perfect (except where recent soil is the product of the erosion of old rocks). Perfection in humans is defined largely in aesthetic terms; e.g., Negroes are ugly, not because of their color (which to Trémaux is an unimportant feature of race) but because of their shape, while white Caucasians, especially Greeks, are beautiful.

    What makes new terrain more perfect, except that it is in some sense more “complex” or “varied,” is not clear from the text. Moreover, Trémaux suggests no mechanism by which the perfection of the soil could be translated into improved human types. His entire argument is, in fact, based upon correlations: people with similar characteristics tend to live on the same kinds of soil. For example, Newfoundlanders (who live on old rocks) are “a sort of Negro.” American Negroes, however, are much closer to American whites than to Australian aborigines. There are, therefore, as many different human races as there are soils of different type. Even within France, claims Trémaux, the people of Brittany, who live on old soil, are religious, superstitious, traditional in their allegiances, and willing to place their government in the hands of a king, while the people of Paris, who live on recently developed terrain, are intelligent, industrious, independent, and favor representative government.

    From Marx’s letter to Engels,

    In its historical and political applications, Trémaux is much more important and fruitful than Darwin. Here alone is found a natural basis for certain questions, as of nationality, etc. For example, he corrects the Pole, Duchinski, whose concern over the geological differences between Russia and the West Slavs he otherwise confirms, but in this matter it was not as Duchinski thought, that the Russians were not Slavs being much more Tartars, etc., but that the prevailing geological formation itself tartarized and mongolized the Slavs. As he indicates, (he was in Africa a long time) the common Negro type is only a degeneration of a much higher one.

    Engels, to his credit, was far more scientifically grounded and sought to dampen Marx’s fondness for this obvious pseudoscience. From his reply to Marx,

    [Trémaux’s] book is not worth anything, a pure fabrication, which defies all facts and would have to give a proof for every proof which it adduces.

    • Steve Sailer says:

      Thanks. Fun stuff.

      Much as we joke about Magic Dirt vs. Tragic Dirt, differences in soil quality really do play a role in the world. Tanzania has lousier soil on average than Belgium. Indeed, a lot of the tropical world has fairly poor soil. A combination of geologically old terrain and torrential tropic rains tend to lower the crop-growing ability of soils (except in river deltas).

      • Aapje says:

        Very poor soil tends to result in nomadic tribes with animals, rather than farms. Some people have argued that farms were an important step towards greater civilization.

        • Steve Sailer says:

          Rep. Steve King (R-Iowa) represents Sioux County, Iowa, which comes in very high ranking on a number of positive metrics, such as best upward mobility for working class children in Raj Chetty’s study, also has the best soil in the Midwest, as measured by highest price paid for farmland (up to $20,000 per acre in 2013).

    • Randy M says:

      Older—primary or secondary—rocks are less “perfect” than are rocks of more recent periods. It follows that persons who live on more recent terrain are themselves more perfect

      It’s amazing how large a gap of actual mechanism one can paper over by brazenly employing the words “it follows”.

  17. John Schilling says:

    Note that in the real-life implementation of the Trolley Problem, per the actual LA Times article cited, nobody was seriously hurt

    Injuries were mostly cuts to the arms, scratches and scrapes

    Also, the people who threw the switch at least claim that they didn’t even expect that much.

    We didn’t know it was going to crash into houses, though

    Since pretty much all ethical systems other than consequentialism recognize a qualitative difference between endangering human life(*) and knowingly causing death, this is a pretty weak example of a real-life Trolley Problem. The Trolley Gods will not be satisfied with such meager sacrifices.

    * Yes, even when the danger amounts to p>0.5 of death and/or >1.0 expected fatal casualties

    • CatCube says:

      I hadn’t heard about this incident, so I looked up the NTSB report for it.

      TLDR: Another instance of, “You can get away with violating safety rules until one day you don’t.”

      Crews had been violating rules about setting brakes. When they took cars off one train and put them on another, they were supposed to set handbrakes per operating rules (the handwheels you see on railroad cars; these tighten the brake rigging mechanically and hold the brakes on with a ratchet mechanism, because air will eventually leak out and release brakes if no locomotive is keeping air supplied). However, this takes a long time and is tedious, so crews would just pull the first locomotive away and rely on the airbrakes going into an emergency application, where air pressure keeps the brakes applied. They’d use the air to hold the cars until another locomotive was attached. However, here they started bleeding the air off the brakes and forgot that they had no handbrakes set.

      • Luke Somers says:

        Air leaking releases the brakes? I thought all trains used brakes where pressure was needed to release the brakes, so that once detached they would come to a halt, and I thought that had been totally standard and required for like 70 years…

        • The Nybbler says:

          Pressure from the locomotive is needed to release the brakes, but unlike truck brakes where the emergency brake is applied via a spring (and is weaker than service brakes), train brakes use local compressed air for their emergency braking (which is actually stronger than service brakes). But if you bleed off the local compressed air — which they did intentionally — no brakes for you.

          It doesn’t say in the story, but I think that once train brakes have gone into emergency application, they have to be bled manually; simply reconnecting the brakes to the locomotive and re-pressurizing is not sufficient. So what they were expecting to do is to disconnect the locomotive, connect one to the other end, bleed the brakes, and move the cars with the other locomotive. What they were supposed to do is set the handbrake before disconnecting the locomotive. What they actually did is neither connect another locomotive nor set the handbrake before bleeding the brakes.

  18. spinystellate says:

    What I got from the Bonobo-jerk writeup was that Bonobos prefer to take food that is associated with the image of the jerk over food associated with the image of the nice person. I don’t understand why that is interpreted as “Bonobo prefers jerk” to “Bonobo prefers taking things away from jerks”. It seems like it would be depend on whether, to a Bonobo, taking something from someone is seen as cavorting or punishing.

    • AnarchyDice says:

      I had the same thought while reading it and was surprised the researchers didn’t try to come up with some way of showing which of the two was at work.

      ETA: They could have, for example, had the bonobos choose which person to give something to or which person to “punish” with a spray of water or something.

  19. 75th says:

    I actually in real life failed a required college class because I forgot to ever attend, AMA

    (It was a required PE elective that lasted half a semester. The second half. And there was no system to remind students of these classes other than maybe having mid-terms in other ones. University policy mandated an automatic withdrawal due to non-attendance; I was failed anyway.)

  20. Alraune says:

    “PCRM” redirects here. For the political party, see Party of Communists of the Republic of Moldova.

  21. flof says:

    Regarding the Hindu temple: according to this wikipedia list, with 162acres/655Km² this would be the largest functioning Hindu temple, the famous Angkor Wat in Cambodia still being larger. Though, these temples are very different some include commercial areas, markets, ponds, gardens (which makes them fascinating to visit). Perhaps measuring the area of the buildings would make more sense for this list!

  22. Steve Sailer says:

    “New study finds more evidence that small class size improves test scores.”

    My impression is that private schools generally spend their higher funds per student more on smaller class sizes than on, say, higher quality teachers. But I could be wrong. This would be an interesting subject to research.

    My son won a scholarship to an extremely good (i.e., extremely well-funded) private high school in part due to them putting a high value on classroom participation because they only had 15 in a class.

    In general, I think education research should pay more attention to private schools. Generally speaking, it’s hard to prove statistically that anything works in public schools, but studying private schools might be helpful. My impression is that private schools tend to believe in small class sizes, and the people running the best private schools aren’t stupid.

    • Aapje says:

      AFAIK, studies show a small negative effect of small class sizes for public schools, presumably because it means that poorer teachers are hired to teach the extra classes.

      More top tier teachers may be willing to work at private schools, than they have teacher spots, in which case they won’t have the same issue. But then their experiences cannot be replicated by public schools.

      • herculesorion says:

        I don’t think anyone is suggesting that bad teachers won’t do a good job.

        The idea is that as class size increase, the ability bar for “better than bad” rises quickly.

    • ADifferentAnonymous says:

      Could just be that parents believe in small class sizes, and they might be stupid.

      • Deiseach says:

        It’s not so much class sizes as class disruption; if the teacher is spending the majority of the lesson dealing with Tommy who won’t sit down, hasn’t his books, is talking and distracting the other students and so on, not much teaching will get done.

        Parents may well think “school with smaller class sizes = school less likely to have a lot of Tommies to deal with = my kid can get taught in class rather than the teacher doing Tommy-wrangling”, either because the school can expel Tommy and the public school has to take him, or that Tommy won’t get in there in the first instance.

  23. sohois says:

    Does anyone have an ungated copy of the lobbying study? I expect that the Bloomberg write up is probably accurate given that it’s Cowen who wrote it, but I’d still appreciate being able to read their results in more detail.

  24. BlindKungFuMaster says:

    I read the lemon mafia thing a decade ago. Maybe the scurvy connection is new.

  25. AlphaGamma says:

    On Justice Force Crater: From 2008 to 2013, the most senior judge in England and Wales was Lord Justice Judge.

  26. liskantope says:

    Regarding the drug-legalization experiment, Trump’s comment about Norway being an example of a non-s***hole country, along with the accompanying backlash from the Left, is becoming more and more ironic.

    • Winter Shaker says:

      Can I ask for more details on why you would consider this to be a move towards s***hole country status? Is it because they are decriminalizing* but not legalising?

      (‘Decriminalisation’ and ‘legalisation’ mean two quite distinct things in drug policy circles: ‘decriminalisation’ usually means that personal possession and use is not a criminal offence, but production and sale still is; ‘legalisation’ usually means that the production and sale is also brought under a legal regulatory framework – it’s the difference between Portugal and Uruguay with regards to cannabis – and decriminalisation is sometimes thought of as a ‘worst of both worlds’ situation, where you still have all of the harms that result from the drugs being produced and sold by the unregulated criminal market, plus the multiplier from there now being more people likely to buy them since you’ve removed the legal disincentive)

      *Actually, from reading the article, it’s not clear that they even are fully decriminalising, in the sense of “the government will do to you for owning and using, say, cannabis or MDMA, nothing worse than it would do to you for owning, say, alcohol, tomatoes or plywood” – it’s not very clear but it sounds like they may just be removing criminal penalties as the default first response to someone using a drug, and mandating coercive treatment with the possibility of criminal penalties if you don’t cooperate with the treatment.

      • liskantope says:

        I didn’t read the linked article, and my comment was only meant to reflect on my simplistic impression that Norway is moving in (what is typically considered to be) a very progressive direction. I was alluding to the fact that Trump’s party and Trump’s cabinet AFAICT has generally disavowed progressivism, in particular with regard to drugs, but also with regard to social safety nets and government-run healthcare which characterize how things work in Norway. Therefore, Trump using Norway to exemplify a “good” country seems ironic, as does Trump’s left-wing critics’ assumption that this is about Norwegian people being almost all white when liberals seem otherwise happy to bring up how nice things are in Norway (with its social and fiscal liberalism) all the time.

        Of course, to be fair, this is all confused by the fact that terms like “s***hole” usually refer to a bad/difficult place to live rather than a place containing bad people, yet Trump (perhaps being confused on the distinction himself) would seem to be using it in the latter sense in the context of discussing immigration policies. (And of course his comment was moronic for other reasons as well, but that’s nothing new.)

        • Ilya Shpitser says:

          Norway also has the lucky combination of good institutions and oil.

          So I think attributing Norway being great to Norway being very progressive is sort of misleading. It’s very easy to succeed if you have this much oil, and enough institutions to prevent looting, regardless of whether progressive or conservative cultures dominate discourse.

          Trump seems pretty obsessed with oil, actually. I think at one point he was suggesting on his vlog (did you know he had a vlog? It is amazing.) US intervene in ?Libya? on “humanitarian grounds,” but then demand oil as payment.

          • JPNunez says:

            It is not actually that easy to succeed with oil, tho Norway seems to have managed itself very well. See, the resource curse.

          • baconbits9 says:

            The resource curse is….. non rigorous in most of the cases that it is used. The US was blessed with massive oil reserves and was barely a net importer of crude oil until the 1950s. It is a combination of resource and institutional wealth that matters (with no true Scotsman possibilities since we don’t understand institutional wealth nearly as well).

          • A1987dM says: