Comments on: Growing Children For Bostrom’s Disneyland

By: » Superintelligence Bayesian Investor Blog

» Superintelligence Bayesian Investor Blog — Mon, 28 Jul 2014 20:12:17 +0000

[…] the risks of competitive pressures driving out human traits (discussed more fully/verbosely at Slate Star Codex)? If WBE and AGI happen close enough together in time that we can plausibly influence which comes […]

Report comment

By: Data and Philosophy

Data and Philosophy — Wed, 16 Jul 2014 17:54:01 +0000

Why do you follow Bostrom’s Malthusian assumption? That to me was the biggest flaw in the piece, and I don’t think it necessarily true that engineered beings would have to try to reproduce as much as possible.

Report comment

By: Paul Torek

Paul Torek — Tue, 15 Jul 2014 16:59:43 +0000

Alternative suggestion: I found Jenann Ismael’s papers (e.g. here and here ) to contain much of the same wonderful stuff (and more) as the presentation I linked to above, without the annoying video jitter that one might get with typical internet connections.

Report comment

By: Noah

Noah — Tue, 15 Jul 2014 16:48:44 +0000

Agreed – the obvious response to this all-against-all scenario…

Any agent that doesn’t always take the path that maximizes its utility (defined in objective economic terms) will be outcompeted by another that does.

…is for the agents to unionize. If we have superintelligent agents, surely coordination will sometimes be economically superior to all-against-all competition. If the advantages of coordination are sufficiently large, then the society of agents could adopt a futuristic communism/anarchism where economic pressures face the society as a collective whole but individual members of the society are relatively free from economic pressure.

it’s worth asking how doomed we are when we come to this point. Likely we are pretty doomed, but I want to bring up a very faint glimmer of hope in an unexpected place.

Incidentally, this lends some intuitive plausibility to primitivism. If humanity were almost certainly doomed by following a technological path, then conceivably our long-term values could be best served by destroying technological civilization and any capacity to rebuild technological civilization.

It would just be a really weird volume of space that seemed to follow different rules than our own.

In any case, I don’t buy that this scenario makes sense. The goo would need a power source, and as soon as the power source was unavailable, it would return to normal physics.

Report comment

By: suntzuanime

suntzuanime — Tue, 15 Jul 2014 08:44:21 +0000

I agree that it’s a conflict of interest – confiscating your savings as punishment is also a conflict of interest. But if the conflict of interest is a problem, it’s because it causes people to be punished for illegitimate crimes, not because the method of punishment is brutal.

Report comment

By: Alrenous

Alrenous — Tue, 15 Jul 2014 05:34:27 +0000

Consciousness is ontological subjectivity. For example, this means it really is inherently private. Sharing a thought with someone directly means being that person directly. As another, you can’t be mistaken about first-order mental entities. That you think you’re seeing blue is what causes you to be perceiving blue.

Report comment

By: Ken Arromdee

Ken Arromdee — Tue, 15 Jul 2014 04:56:11 +0000

suntsu: It’s a conflict of interest to both punish prisoners and benefit from the punishment in ways unrelated to discouraging crime.

Report comment

By: anon1

anon1 — Tue, 15 Jul 2014 03:34:09 +0000

It’s a problem because making prisoners profitable creates an incentive to punish people for illegitimate crimes.

Report comment

By: Thomas Eliot

Thomas Eliot — Tue, 15 Jul 2014 03:13:08 +0000

That is in fact what Eldritch said, yes.

Report comment

By: Scott Alexander

Scott Alexander — Tue, 15 Jul 2014 02:57:48 +0000

Very degenerate self replicating patterns do arise in current economic activity – for example, the weird feedback loops among stock-trading-robots that occasionally shut down a market or two. Make the economic activity a zillion times more complex, and maybe we’ll get better ones.

“Also how would an economy like that be stable for so long? Eventually the best agent would outcompete all the others, or have enough resources to be self-sufficient and do whatever it wants without interacting with the others.”

Why doesn’t this happen to biological life forms? Insects were competing against each other for millions of years, and while there was some driving-to-extinction mostly they just stayed in their own niches and formed a nice stable system?

(this question does bother me, but given that it happened there must be some answer)

Report comment