Imagine a kingdom ruled by a wise and benevolent king who, by reason of some strange tradition, is prohibited from ever leaving his palace. He only receives information on the affairs of the kingdom from his various Viziers. Like most Viziers, they are evil and power-hungry, and they are all conspiring with some of the most brutal and oppressive nobles in the land to preserve their reign of terror.
Also, you have an adorable pet bear
One day the Heroine speaks out against the current conditions in the kingdom. Taxes are too high, the peasants are starving to death, and people are being enslaved, all to enrich a few brutal nobles. The Heroine goes from town to town with her message: the people must beg the King to do something about this problem.
The Viziers hear of this and go to the king. “The Heroine,” they say, “is speaking against you. The whole kingdom is happy and prosperous, but this one woman wants to tear it apart and start a civil war for her own personal enrichment. Your people beg you to do something about her before she destroys the golden age they are currently enjoying.”
And so the king orders the Heroine executed, an order which the Viziers and the nobles are all too happy to carry out.
In this kingdom all the laws would be utterly selfish and show no regard for the average citizen. But this would be totally consistent with the King himself being a good person.
In fact, the King could be a perfectly good person, a person who attains moral heights of which other people never even dream, simply because he would never face a true moral dilemma. Suppose there were some problem that might prove morally difficult for the King – for example, the eastern states, which provide most of the kingdom’s silk, are rebelling, and the king could either choose to live in peace with the newly independent east, or brutally crush them. If he chose the first option, silk would cost a lot more, and the king really likes silk.
If he were aware of this situation, it would be a sort of moral dilemma – do I do the right thing and avoid a war, or do I do what’s convenient for me and lets me keep my luxury goods? Thanks to the Viziers, this problem disappears. If the Viziers want silk, they can tell the king that the eastern states have just launched a surprise attack, complete with atrocities – they must be dealt with as a matter of existential threat to the kingdom itself. And if the Viziers don’t want silk, they can just tell the king that the east ran out of silk, too bad, nothing we can do about it. In fact, the Viziers will never present a true moral dilemma to the king, because then they wouldn’t know which side he’d choose.
And so the king is faced only with easy, convenient moral decisions, and is able to preserve perfect innocence and purity. No matter how awful and tyrannical his decisions, the populace may at least take consolation that their king is, at heart, a good person.
These were some of the thoughts that went through my head when I read Ozy’s The Inherent Goodness of Human Nature, or Lack Thereof. Ozy worries that Hansonian explanations – in which people do nice things mostly for selfish reasons like signaling or self-signaling – mean that there’s no such thing as goodness. As ze puts it:
Robin Hanson (if I understand him correctly) would argue that the person giving money to the Make a Wish Foundation doesn’t actually want to help sick children; they want to feel nice, like the sort of person who helps sick children, and– more importantly– they want everyone else to believe that they’re nice people who help sick children.
My initial reaction to this is “No! That’s horrible! You terrible person!” Unfortunately, “you’re a terrible person!” is not actually an argument that something is not true. My sense of revulsion at that idea is nothing more than a sign that I’m biased in favor of the “humans: basically nice” explanation.
Robin himself commented by saying:
I love people, even if I don’t think they are as good as they like to let on. I hope others can love me under the same conditions.
This seems like one of the wisest things I have ever heard, and restores just a little of my faith in humanity. But I think I’m more optimistic than Robin is.
Like Ozy, I believe human nature is basically good even though people’s actions seem based on selfish and amoral motives. This is no more contradictory than the King being basically good, even though all his decrees will seem based on selfish and amoral motives. If the King has no access to accurate information, but can only make decisions based on information gleaned from biased sources, then the biases of those sources will be reflected in his words and deeds.
I cannot say why I identify other people with the Kings of their minds rather than with the Viziers of their minds (or with the creepy guys standing next to the king of their minds) save that this is who I feel I am in my mind, it is how I would like other people to see me, and so it seems both accurate and kind to see other people that way as well. Upon this view, people are good by nature, far better than their actions suggest, and it is really hard not to love and respect them.
This is not to say I think there’s no such thing as evil. I would prefer that evil be something different than mere stupidity, something more than “Osama bin Laden was dumb enough to believe his mental Viziers when they told him becoming a terrorist mastermind was the right thing to do, poor guy”, and indeed it seems there are lots of good stupid people and evil smart people, even lots of irrational good people and rational evil people. Although I have no clear answer, I think I would define evil as certain habits of mind which make it extremely easy for your Viziers to put one over on you, certain tendencies like “other people would do the same to me, so I’m just giving them a taste of my own medicine if I hurt them.”
This is still an attempt to be a good person – it’s an attempt to create a moral system in which you are just and virtuous for hurting others – but once you’re letting your Viziers use this kind of argument on you I think it’s pretty safe to say you’ve gone evil. This doesn’t quite correspond to my inner intuitive impression of evil, but if I turn it from a specific English-language assertion to a sort of preconscious sense of wrongedness and arrogant self-justification that expresses the same idea, it might.
You may notice how nicely this meshes with Trivers’ theory of consciousness.
> this is who I feel I am in my mind, it is how I would like other people to see me, and so it seems both accurate and kind to see other people that way as well.
*Puts on Hanson hat*
Identifying with this sub-agent is highly useful as humans are terrible at lying and thus allows you to genuinely signal altruism. And you don’t have to worry about not getting in on the amoral/hypocritical goodies since other sub-agents will activate when the time is right.
*Takes off Hanson hat*
I would not give myself huge amounts of optimization power currently if it were offered. My mix of subagents is probably unfriendly. But fortunately we can do better.
I don’t understand how ‘wanting to feel nice’ and ‘wanting everyone else to believe that they’re nice people who help sick children’ are any kind of refutation of ‘wanting to help sick children’. It’s like arguing that apples aren’t red because they’re actually round.
No. You need to distinguish between terminal and non-terminal goals. The position you are ridiculing is that helping sick children is not a terminal goal, but is an instrumental goal to ‘wanting to feel nice’ or ‘wanting everyone else to believe that they’re nice people who help sick children’ and will be pursued if and only if it furthers this other goal. This is not obvious, and it makes the relevant prediction that that person will stop helping sick children in a context where that won’t further the terminal goal.
The hamburger example sounds so stupid because nobody would ever dream of assuming that eating the hamburger was your terminal goal in the first place. Of course eating is always instrumental to satisfying hunger and/or having a pleasant gustatory sensation, and possible other things.
Distinguishing between terminal and non-terminal goals is interesting and not obviously wrong, but I don’t see it discussed in this post, Ozy’s linked post, or Hanson’s post Ozy linked. Can you elaborate?
Also,
Of course eating is always instrumental to satisfying hunger and/or having a pleasant gustatory sensation, and possible other things.
This seems like a distinction without a difference.
Let’s distinguish.
I have a drug that makes you feel as nice as though you had donated money to help sick children, and is much cheaper. How many people do you think would take it instead of making a donation?
I think the ‘wanting to feel nice’ is to be read as ‘wanting to feel like a nice person’; i.e. the desire to promote a particular self-image. The cheap pill example really doesn’t work for this.
The only thing that would make me feel as nice as if I had donated to sick children, rather than feeling like a person looking for a self-deceptive high, is something that actually fully convinced me that I had actually donated money to sick children. And I wouldn’t take a pill that did that, because of another terminal goal against (overly obvious) self-delusion. I tend to think that having the terminal goal “helping sick children” and having the terminal goal “being the sort of person who helps sick children” reduce to basically the same actions in pursuit.
Thereby demonstrating that you were not in fact trying to make yourself feel good. One of you wants to make yourself into a particular kind of person, that is, is pursuing virtue, and the other wants to live in truth, which is also not feeling good.
The thing is that people can and do optimize for having a self-image as someone who helps children without optimizing for helping children. It even seems possible to optimize for that self-image without helping any children at all, namely if you’re very much mistaken about what is helpful to sick children.
nevertheless, that is not the same thing as only desiring to feel good and not to help children.
I think most charity donors would still donate? I’m not sure the drug example is really a good one here. A better distinction might be ‘what if there was a magic spell that made everyone around you think you’d donated to charity?’
…in which case I think many, even most people would still donate. Call me unusually optimistic if you like 🙂
Re terminal and non-terminal goals, I don’t think it’s necessarily the case that one goal has to be terminal. Say X cares about sick children and wants to be popular. Donating to charity helps her achieve two of her major goals at once. Obviously this means she will be more likely to donate than if it only helped one of her goals. I don’t think this means one goal is terminal or that X is not a good person.
Thinking more about terminal goals (and I still am not convinced that there has to be a terminal goal at all): what is the test for a terminal goal?
From Creutzer’s comment above, it should be ‘will X stop donating to charity if it no longer makes her popular?’
Suppose X moved to Zlobonia, where there are lots of sick children but by law all charitable donations are anonymous. In fact there is a social taboo against talking about one’s charitable acts. My opinion of the human race is that Zlobonia will have many charity donors and there is at least, um, rough guess, 30% chance X will continue to donate. She will find other ways to be popular.
But suppose X moved to Mordor, where the people around her are really nice and pretty and intelligent but think that helping sick children is an idiotic sissy thing to do, and people who do it are actively humiliated. All charitable donations are made public by law.
I think Mordor would have a very very low number of native charitable donors, but X might well continue to donate.
In short, I think human beings are complicated. Many of them are also good.
Yes, indeed anonymous donations are a puzzle if you have the view that people do good things for their reputational consequences. The “self-image” thing I see as an attempt to remedy this so that you can still explain why it is that people do not optimize for the things they claim to care about. A ‘desire to be a person who helps children’ seems indistinguishable from a ‘desire to help children’, so we need a ‘desire to maintain a self-image as a person who helps children’ and show how that behaves differently from an actual desire to help children. And then we have to say something about what things fulfill these desires (roughly, the things that give you “warm fuzzies”), etc.
I’m note sure that, when looked at at a deeper level, this whole thing is rightly framed in terms of a goals/desires (especially desires of a unified person), but at least as a high-level description used to point out the inconcruence between professed and revealed preferences, this way of speaking seems legitimate to me.
Even less sure I am about how this translates to saying that a person is properly good/altruistic; these terms, like most of ordinary mental language, may cease to work once you look too closely.
Avantikia, I’d like to think that in Mordor, there will be an underground of people who help sick children.
Creutzer, the idea of reputation-based generosity can be somewhat saved if you assume that people have mental models of people they want to please/impress/satisfy– at that point, reputation doesn’t need to involve actual people.
For what it’s worth and from memory, when people are asked why they engaged in heroic generosity, they tend to answer that it seemed like the obvious thing to do.
That is an excellent point! Maybe the “self-image” is in fact just a simulation of a hypothetical person learning of your acts.
Empirically, a lot of people spend a lot of time on fiction that involves acts of fake generosity. Video games and TV shows often allow people to be or identify with heroes.
People don’t seem to optimize their actions to help others in the real world very much.
More concretely, if I eat a hamburger and someone tells me that I didn’t actually want to eat the hamburger, I just wanted to not feel hungry, that’s not an argument; that’s just some smug idiot making meaningless sounds.
Agree with the hamburger comment. Also there are plenty of people who defeat this argument by giving to charity and never publicly bragging about it. Though actually, from a consequentialist perspective it makes sense to publicly brag if that guilts other people into giving, so perhaps that should be taken into account?
Encouraging people to give can be done by showing how much is given and not by whom.
For instance, bringing non-perishable food items to a place where they’re collecting early, so that people notice them and feel encouraged to give, without the donor standing over them.
I don’t think it would work as well; certain kinds of motivations work a lot better if they come from someone you know rather than an anonymous source. If I see ‘someone donated so much money to fight malaria’, I might think, ‘Well, this could be some billionaire like Bill Gates, for whom this money is peanuts. I couldn’t possibly donate’. If I know the donation came from X my coworker from the next cubicle who earns the same salary as I do, it’s a much stronger guilt-motivation.
Err — people are what they do, not what they would like to think of themselves doing. Otherwise you end up with murderers arguing that it wasn’t the Real Me that murdered.
I’m not sure I would like to be evaluated by my delusions, and I’m quite, quite, quite certain that I would not to live in a society where people were evaluated by theirs. So I would do my little part to helping ensure that other people don’t have to, either.
Also, the charity issue founders on anonymous donations. How much social cachet does the average donor get from a donation?
Thank you Mary. Please keep it up.
Agreed strongly WRT charity. More generally, from where I sit it looks like people aren’t generally good. They aren’t bad. They aren’t lawful. They aren’t self-justifying. They aren’t self-interested. They just aren’t very conscious or directed in their behaviors at all, towards any ends. They aren’t moral agents. Maybe kids are. I can’t tell.
“Err — people are what they do, not what they would like to think of themselves doing.”
I’m pretty sure that’s not true. The Germans under Hitler did terrible things, but I’m pretty sure they were not especially better or worse people than the French under de Gaulle, who didn’t do terrible things. It would be odd if a national border just happened to divide Good from Evil people.
I would like a model of human goodness that allows me to say things like “The Germans probably weren’t any worse people than the French, but they really screwed up on that one Nazism thing.”
It’s complicated. It’s not as though Nazism was invented and imposed by non-Germans, and it amplified traits that were already present in the culture.
On the other hand, I’m quite sure that a lot of the Germans who committed atrocities under Nazism wouldn’t have done anything of the sort if Nazism hadn’t happened.
It would be odd if a national border just happened to divide Good from Evil people.
It wasn’t a national border, exactly, but it was circumstances. Because of the fundamental attribution error, we don’t want to admit that mere circumstances can make people evil. This is why people are unsympathetic to the criminal saying “it wasn’t the real me”… until they themselves are the criminal.
There is some base rate at which otherwise “good” people will do terrible things, simply because all the circumstances line up in the right way to bring the bad tendencies latent in everyone to fruition.
The Germans are what they did — and you don’t know what you are, because you do not know what you would do.
Incidentally, the French under German occupation did terrible things. Perhaps comparing to, er — Ecuador would avoid distractions.
Wouldn’t it be very surprising if good policies couldn’t make people better and if bad policies couldn’t make people worse? People aren’t magical essences. They are made of parts. Those parts can be altered, re-arranged, effected. It would be odd if a national border *didn’t* in some instances, divide Good from Evil somewhat effectively, though it would be odd if it doing that was a coincidence.
What you want, I imagine, is more like a model that says ‘the Germans who became Nazis very likely weren’t born any worse than the French who fought for de Gaulle, and those same people might very plausibly again not be any worse, or even be better, on average today, after many more things have happened to them.
If you can get social credit for helping sick children, this suggests some general benevolence, even if it’s not entirely located in the person who’s doing the helping.
This is a very important point.
I agree.
The “misguided or mislead but basically good intentions” explanation for evil is tempting and plausible in some situations, but I lack the imagination to see how it can apply universally. For an example, recall the news story recently about the man in Pennsylvania (I think?) who had three women kidnapped for about a decade, and had also killed various children he fathered on them by rape. I can’t conure a rationalization or justification that any one could use to convince themselves that they were not the bad guy there. He wasn’t “taking care of” them, there was no slight against him from them that merited this (they were tangentially known to him, like a friend of his daughter), no one else gained from their suffering, etc.
(If someone has heard him give an explanation, feel free to correct me. I did not and can’t think of one.)
The man who kidnapped three women was in Cleveland, and didn’t commit murder unless you count beatings and starvation to cause miscarriages as murder.
Depends on whether they have a fetal homicide law. Yup, you can get a first-degree murder charge and the death penalty on willfully killing the baby before it’s born.
Thank you for the corrections, however, does anyone disagree with the larger point?
He might be a psychopath or otherwise without conscience. That happens and doesn’t really say anything about Scott’s theory, except that it is wrong in something like 1% of humans.
There are many contexts in which I think people get full points for living their lives in the ways that their cultures proscribe for them and that they believe to be good. But the king example does not work for me. If you have significant power over other peoples’ lives, you choose to be uninformed, and as a result you do things with bad consequences, you are at fault. If we frame this as the king not actually having any control over his ignorance – if he had insisted on speaking to the Heroine the viziers would have killed him, or refused – then he isn’t really a king and he isn’t in any meaningful way making any decisions, and so he can’t really be blamed for them. (But if they refused, that should tell him something about them, and we need to invent something stronger than a tradition to keep him from leaving the castle – say, a magic force field – in order for him to be absolved for not doing so.)
Agreed.
In addition, due to ‘selective memory’, lapses of attention, etc, our situation would appear to me to be worse than that of the king, in that we don’t just have bad information, we use contradictory information, don’t propagate conclusions, etc. People also fail or refuse to fulfill obligations regularly, even when those obligations are taken on under no duress and even while those obligations are being brought to their attention, and they do this while offering no explanations at all.
More generally, I think that people who give justifications tend to fail to observe that most people simply don’t bother with justifying their behavior most of the time, and many many people do identify as ‘winners’ or as ‘cool’ and see being ‘good’ as being weak or as being a loser.
Seems rather obvious that *you* are both the King and the Viziers of your own kingdom (and also the peasants). Identifying with just one part of yourself doesn’t seem like a very healthy and wholesome idea to me (although, being a psychiatrist, you probably have much better knowledge on the subject).
This. Although some parts count more than others, depending on their integration or lack of integration with the core. A knee-jerk reflex circuit is only a teeny weeny bit “you”.
There are selfish rationalists, but no evil rationalists. You know this because you’re not dead.
Dead humans outnumber living ones by an order of magnitude or more. Any statement “You know X because you’re not dead” thus deserves careful examination for anthropic bias.
The most steelman interpretation of “because you’re not dead” I can come up with is “because terrorists haven’t killed you”, and I might agree with the argument. If there were rational people trying to cause suffering for its own sake with modern technology then you’d think they’d have been much more effective at it.
On the gripping hand: if benevolent rationalists are most concerned with reducing existential risks, presumably malevolent rationalists are most concerned with increasing existential risks, and the results of that might not be publicly obvious until the megadeaths start. Suppose some of our past ages’ conquerors enjoyed the pyramids of skulls for their own sake, and suppose some of their sadly-numerous descendants share that proclivity. Would their optimal (pessimal?) course of action be to pick up a sword and try to emulate great^n grandpa, or to pick up some biotech/nanotech textbooks and try to become the inventor of Gray Death? The latter strategy would go forever unnoticed if it failed and would go unnoticed until too late if it succeeded.
I am trying to see how this cashes out in our dealings with other people, and am failing utterly. It seems like the idea “Osama’s viziers are feeding him data such that he is dangerous, so we should oppose him,” will produce decisions indistinguishable from “Osama is evil, so we should oppose him.”
Is the upshot just that we should be more angsty about opposing evildoers? If so, why, if angst isn’t instrumentally valuable? Or perhaps angst is instrumentally valuable (to oppose a bias for bloodlust, for example) but then that would also be true if there are no viziers, and everyone is a morally imperfect king. Right?
What concrete decision actually rides on the difference between these two models?
It’s worth considering whether it’s possible to improve the viziers.
I can say why I do: even if “System 1” thinking dominates the majority of our actions, whenever “System 2” thinking becomes aware of a decision and has enough time to make it, System 2 tends to be able to override.
In your analogy, our King may be elderly and arthritic and in need of bed rest 22 hours a day, but he does manage to get out of the palace for a couple hours here and there. That’s not enough to really be in control of the whole kingdom, it’s not even enough for him to double-check on most of his Visiers or personally meet with many petitioner Heroines on most days, and if anything it’s just maddeningly enough to become aware of how badly he’s failing at his duties the rest of the time… but it is occasionally enough for him to take real charge of and fix a problem once he’s aware of it.
Historical monarchs were pretty much in this situation (restricted by travel speeds and human limitations, if not by illness) for most of history and we still felt pretty comfortable referring to the results as monarchies.
Having an unflinching eye towards human shortcomings while simultaneously loving humanity is close to the pinnacle of wisdom.
As is the awareness that our enemies’ opinion of us comes closer to the truth than our own.
Correction: When I said enemies, I should have said acquaintances.
I think your definition of good/evil is workable and your view of human goodness/evil fair, but I think a simpler measure could be applied as or more effectively. For example, one could say that humans are innately good, or at least prefer goodness, *because* we desire to be viewed as good–that is, goodness is viewed as a superior state of being. While case-by-case definitions of good may vary–there are cannibalistic cultures, etc.–evil could be defined using the Socratic interpretation of “injustice.” That is, you commit an evil action when you do something which runs counter to your culture’s view of goodness but with the intention of being viewed as good in that culture. Consequently, if you are simply trying to change your culture’s interpretation of good/evil or acting according to your culture’s norms within a culture that does not share those norms, your actions can be counter-cultural but not evil. Anyone acting with the intention of being viewed as evil is, in fact, acting on a different code of morality and not evilly, insofar as their actions are good and intended to appear good under a different moral system.
The problem is that everyone, by that example, does evil things. At what point, then, does one pass from good to evil? If evil is something you can be outside of external observation, it should be measurable. Perhaps you are evil if your wont is to commit greater than 50% evil actions, given a more-than-simple moral dilemma (as in the article); it bears consideration.
If there were no bees, there would be no bee flies. The birds would eat them all without a second thought. Similarly, if there were no altruists, there would be no point in signaling altruism.
“I cannot say why I identify other people with the Kings of their minds rather than with the Viziers of their minds (or with the creepy guys standing next to the king of their minds)”
It’s interesting that the linked quote uses the exact opposite metaphor that you’re using in this post. If we wanted to be consistent maybe we could say “You are the king of your brain, but you’re a really corrupt king that does whatever your much more powerful vizier says”.
Taboo good and evil people?