You know how it’s impossible even in principle to compare people’s utilities and so utilitarianism is a pipe dream that can never possibly work?
Well, I just learned Tufts has a searchable public database of utilities for various health outcomes. It doesn’t seem very good – some of the entries in there make so little sense that I worry that it’s haphazardly combining QALYs (in which higher numbers mean higher utility) and DALYs (in which lower numbers mean higher utility). Either that, or catching the flu decreases your utility more than being totally paralyzed does. But the database exists and is full of numbers, and that’s inspiring.
But this still doesn’t fix my constant gripe that no one has tried to extend this to non-health states. Searches for QALYs for poverty or totalitarian government, for example, mostly just turn up me complaining in various places about how there aren’t any.
Just to see if there is some fundamental impossibility here, and to get enough data to whet my own curiosity, I’ve set up a very amateurish person-who-is-no-good-at-math version of a utility measurement test. If you have a while (15-30 minutes) and are willing to follow confusing instructions, can you take it for me?
(survey closed, thank you for participation)
I agree it’s extremely confusing and most questions are underspecified in various ways, but if you could just take it anyway, try to guess what I mean, and leave anything you don’t get blank, that will be good enough for me.
It’s a lot harder for me to mentally encompass death than the cell when making these comparisons? The 10 years one was especially hard for me to really think about, and I probably went too on those based on the hedonic treadmill and how I would probably get used to eg south korea.
The probability trade-off section was incredibly counter-intuitive. The way I ended up having to think about it was “you are trapped in this situation. You’re given a chance to escape that might kill you. How good do the survival odds have to be before you to accept it?”
I imagine you’re going to get a lot of ass-answers for that section. I am not entirely sure mine aren’t ass-answers.
This is how I thought of it as well.
Agreed. That was a huge mess.
It would have been nice if, in the end notes of the first survey, it was explicitly mentioned that the baseline in the second survey was the “Guantanamo Bay” scenario from the first survey. As it is, the second-survey baseline is described rather differently.
As a note, I was confused by the lack of anchoring in the first question batch of the first survey, and as a result came up with my own “visualization” for the question. Let’s say I’m spirited away by Omega, who gives me a choice: either I can have n copies of me alive but in the stated question scenario, or m copies of me living at baseline. The answer is the ratio m/n at which I’m indifferent about the choice.
I don’t think the discrimination question was very good. Remember that racial discrimination is extremely entangled with classism. It’s unclear how much difference just being a member of a discriminated-against minority group would make without also changing social class.
Hmmmm.
For me the idea of having or caring about having copies of myself doesn’t really make sense…
I’m so used to thinking in terms of VMN utilities that by default I’d take the questions in Section 1 as synonymous with those in Section 3, but with “10 years” replaced with “however long I’d naturally live”. Also, I think death is so bad that on that scale nearly all the values would be between around 0.8 and 1.2. Do I submit the survey anyway?
It’s hard for me to say which would be a better floor- I took A-M and a lot of my preferences were weighted using death as a time-limit-to-raise-funds-for-cryonics (although it’d work for time-limit-to-accomplish-whatever-supremely-important-task types as well?) In retrospect, that’s probably not the way the criteria was intended, and I’d imagine there’d be a gulf in results between theists and non-theists.
I suppose the issue is that death doesn’t make for an especially good statistical floor, since people weigh the disutility of death in a lot of different ways? Unknown whether the prison option has similar issues.
(May I just pretend that my last name started with Z? Pretty please?) 🙂
Oh, okay.
Done.
Huh, so far I’ve gotten 13 responses to A-M and only 1 response to N-Z. This seems unlikely to be coincidence but everything seems to be set up correctly. If you took the N-Z survey, could you post so here?
I did (I think, at least I should have).
I have. I had the secret prison version, which I liked except for the difficulty of deciding how to represent the utility of states less acceptable. The first section it was obvious (negatives), but I wanted to be cautious about the later two. Perhaps I should have thought about them more carefully and done the reversed experiment.
Is it possible that negative numbers are throwing you off somehow? Some of the stuff you have here seems worse than prison, especially a prison where you aren’t even tortured.
I took it – last name O’Kane.
took it. why is it split up by last name?
Are most of us on here in the “last name A-M” category? Maybe you need a headcount? My real surname starts with “D”, so that’s the survey I took.
I find it unlikely that last initials are split evenly between halves of the alphabet. It would probably have been better if you’d used a different split, such as whether your date of birth is in the first or second half of the month.
I did, and found the time and probability questions difficult — they’re a bit complicated, and it’s easy to get the two sides confused. I wouldn’t be surprised if people who started this version were less likely to finish than those taking the death version.
Given that I meditate, go on silent retreats regularly, and am an introvert with an active imagination, death as the floor is probably more useful for illiciting accurate responses from me.
Thats… another issue. If I can send letters and have access to a shit-ton of writing materials, then the prison suddenly becomes less bad than many of the really horrible options.
I’m not up for doing the survey at the moment, but there’s a problem– your survey assumes that none of the respondents are in a discriminated-against category.
Maybe Scott fixed it after your comment, but it looks to me like the survey avoids this problem.
Those instructions ask you to imagine being in “a non-oppressed majority group”. If you’re in an oppressed minority group, the instructions imply that you not use your current life as baseline, and instead imagine a slightly better life.
I see that, Rory, but the “minimum wage” question amused me since that was me in a lot of the jobs I worked, so assuming it as the “worst” state in a question made me smile wryly.
That is, having it as one of the choices in “imagine you’re not like you are now, but in a position of dying by famine/being tortured by a totalitarian state/having to work on a supermarket till”. Arson, Murder and Jaywalking?
Agreed. My reaction to Guantanamo bay was ‘oh, I could meditate all day, not so terrible’. Death is a better floor.
I found the first section easy but the rest incredibly tough to visualize.
Agreed, the second section was really, really tough.
Also, death is a really sharp floor, so I kind of ended up using ‘Nothingness’ rather than death as a floor.
It doesn’t seem very good – some of the entries in there make so little sense that I worry that it’s haphazardly combining QALYs (in which higher numbers mean higher utility) and DALYs (in which lower numbers mean lower utility).
These two say effectively the same thing? I assume something else was meant?
Yup, I’m dumb. Fixed.
I would prefer the boring prison cell, but I could certainly imagine coming to regret that choice at some point. I had a strong urge to make things consistent, when taking the second and third parts I realized I would more strongly prefer not to die than is probably indicated in part one.
Holy crap was that confusing. I don’t know why I had to keep thinking about whether the questions were meant to be answered the way I answered them, and I think I’m pretty good at this stuff.
Prediction: for the X/10-X sections at the end, some percentage of them are going to be completely reversed.
There’s an error in the end-of-survey message for the last name N-Z survey. It says
But on that survey itself (the N-Z survey), floor is the prison cell, not death. So the message is reversed. The message is what A-M people should be seeing, not N-Z people.
I have to say I prefer using death as a floor, and am glad I got the survey asking that. It seems easier and simpler to me to compare the different states with being dead than being in the prison cell, although that might just be something personal.
I was halfway through the last section before realizing that I had all my percentages reversed, and had to go back and fix them. That was even after specifically thinking in the beginning “I had better not make this mistake,” too. I also predict that you’re going to get a lot of reversed percentages there.
I would have liked to have some more text about “Being blind”.
Like: “You are blind. You [can/cannot] still have a job and support yourself, so you [do/don’t] have to move back in with your parents. You [are/aren’t] able to find friends to spend time with. You [are/aren’t] able to find a romantic partner.”
I don’t know what is the “standard” state for these, so I had to guess, and I think I sort of guessed different each time, depending on how hard I thought about it.
Depends how each person takes it. The one blind person I know has a good job, is married, and has kids, so that’s influencing my decision on those questions. If it were a case of “blind or otherwise handicapped so that couldn’t find employment/had no means of support other than family or living in state care home”, that would make a difference to how people answer it.
I took the first survey as you can guess from my name.
When I had to pick years of life or probability of death, the only way I could get my gut to provide an answer was to ask myself “How many years of life are you willing to give up to avoid this outcome?” or alternatively as sixes_and_sevens says, “How much of a chance of getting killed are you willing to take to get out of this?”
For many / most of the questions my answer was “none” — I would not give up years of life to avoid being blind, in prison, discriminated against, a woman, etc. I would give up years of life to avoid being in North Korea or Syria because most likely I’d be giving up years of life by being there.
(There’s no easy way to fix the “giving up years of life by being there.” Otherwise the conditions wouldn’t really apply — you’d be immune to starvation, death by civil war or prison camp, etc. So I think this is fundamental to the questions.)
Given this problem, I guess prison would be a much better baseline for me — although as a number of people have commented that doesn’t sound too bad. There’s a library!
I think the floor needs a lot of thought for this to work at all. It has to be viscerally real, not too extreme, and not vary much with people’s preferences. Not clear that can be done but that is what is needed.
Also I think the way a few of us reframed the questions, as “How much time would you give up?” or “What chance of that outcome would you accept?” would make them a lot easier to answer.
It turns out my brain craves safety, and the floor scenario in survey 2 is pretty safe (food, water, rec time, library, no torture). So it didn’t really make an effective floor for me, and I ended up using too many negative numbers.
Also, where you have used descriptions of real life examples, like ‘Living in a country similar to present-day North Korea. You are near-starving, the government controls every facet of your life, and you worry about being sent off to a death camp at the slightest provocation’ – I have an automatic skeptisicm about how bad it is to be an everyday person in Horrible Countries, which I should do more research to see if it is true or not. So the fact I have answered these questions as though ‘I don’t want to live near-starving, and at risk of death camps’ makes me feel I have been asked a political opinion on ‘Is everyone in North Korea near starving and at risk of death camps’. Obviously if you’re surveying people with a range of political beliefs and affiliations you might want to get rid of this problem.
Argh, I have just been proof of your ‘many muppets take surveys’ argument. I hit return by accidently leaning on the keyboard when switching tabs, and it submitted the survey when I was musing about how horrible it was to live in Syria. Please throw my stub-results away, and I’ll come back and do this properly when I have more time and am not half asleep.
I also think the prison library makes the “floor” too good. For many people being alone and reading and not having to worry about much else, is a good life. So maybe take the entertainment away to get a better “floor”.
For the introverts amongst us, switch it to “You are dragged to a party! With loud music! And have to talk to people! And persons in various states of intoxication ask you to dance and/or make out with them!”
🙂
So yeah, I took the tests. I had an urge to go back and change my answers in previous sections to make them more consistent with each other. But I didn’t. So you get my baseline intuitions about the answers, not the feeling I got after pondering the different ways to look at the questions.
Also, I had the same issue Jed Harris had, that being told by an omnipotent being that I’m going to live for 10 years is a confounding factor for stuff like the Syrian civil war question, since a good bit of disutility from living in a civil war is the chance I’d get killed.
A lot of those scenarios were well constructed, but I found myself considering two factors:
1) Transition costs for things like blindness or transplantation to another country. How long until I learn to function normally, have a normal social circle, figure out how to read/work?
2) If I know I’ll live for exactly 10 years, I have a hard time taking “fear for my life” seriously.
I assumed that I would forget what the omniscient being told me after I made my choice.
If I differ from the majority of the world on one question, not for clever philosophical reasons, but because of a specific phobia, should I try to correct for it? I ended up going with my gut, since I was asking it to make up numbers anyway.
Interesting survey. I probably messed up the probability bits, but I took it as “Given the choice between being as you are now, and the probability of (percentage preference for new state less percentage chance of dropping dead), what is your preferred minimum percentage chance of dropping dead?”
I think, in the follow-up question you asked, that it would make a difference if the choices were between “baseline state and being dead” versus “baseline state and being locked up”. Someone may prefer life as the greatest good; another may prefer liberty and, if faced with deprivation of what is considered desirable, a painless death would be preferable. So I think “death” as a choice might very well get the best answers, since I know that my answers would have been different if the choice were between “you can be a billionaire or dead” and “you can be a billionaire or in prison”.
Fascinating survey. I expect I was very inconsistent – especially in the last section, where I imagined it as survival odds like some commenters above, and ended up giving more conservative answers than would be consistent with my earlier answers.
Also, I have a young child, so years aren’t that fungible to me – even if I abstractly rated blindness at, say, 70% of baseline utility, I’d have a disproportionately strong preference for living another 10 years even blind or poor or whatever, than another 7 years at baseline, so that I could see (or not “see”) her grow up more. I tried to ignore this effect for the survey, and instead imagine 10 blind years or 7 sighted years with her staying at her current age either way.
My main criticism of this survey (which nobody has really brought up yet) is that this may have become a ‘First-worlders criticize various states of living’ survey rather than what is really intended.
I took the second survey. I actually quite like the idea of being in a prison cell with a prison library available. This made most things come out negative, and I couldn’t visualise at all what I was trading off in sections 2 and 3, sorry.
A more appropriate baseline for me would be death, or perhaps prison without a prison library. I still think North Korea would come out negative compared to both of those too though.
On the man/woman question, I wasn’t sure whether it referred to keeping your current gender and swapping your sex, or to swapping both sex and gender to match. I know from experience that if it referred to just swapping your sex, my personal utility comes out very negative (way below prison, and slightly below death if there is no opportunity to transition).
It’s also not clear how you should answer if you’re neither a man or a woman.
I skipped both questions.
If you interpret the instructions literally, you should answer neither question.
I clicked N-Z, it used death as a floor, but when I finished I got the message “On the other version of this survey, instead of using death as a floor, it used the state of being stuck in a clean and humane but extremely boring prison cell. Which of these would you prefer? Which survey do you think makes a better way of eliciting preferences? You can answer in the comments section of Scott’s blog.”.
Also, some of the options, like North Korea, are *below* the floor, which made some questions impossible to answer.
I guess the survey was at least titled accurately.
Sorry, I mean it used the prison as a floor.
Why call it a “Utility Measurement Test” if it asks about a subset of your utility (“effectively the quality of your life”)? Being a billionaire multiplies my total utility a lot more than my quality of life. There aren’t enough neurons to feel all of the utility I would get from that.
Meh, I started taking the survey, then stopped to ask Scott for clarification on one thing, and then when I had the time to get back to filling it out it was already closed. 🙁
My one comment from the bits that I managed to see would be that saying “a prison like Guantanamo Bay” and then “where you are not tortured or abused” felt confusing and contradictory, since my impression of Guantanamo Bay was that it’s pretty much a place that’s built for having prisoners tortured and abused. Don’t know whether that influenced anyone else’s answers.
It surely influenced mine, making the prison sound worse than if it hadn’t mentioned Guantanamo.