Part of what I’m doing is trying to understand what the AI-is-coming folks mean by “AI”.

What you’re describing seems much more possible, and much less powerful and dangerous, than an actually conscious and near omniscient AI.

I also don’t see how this AI can be described as friendly or unfriendly, which I see attached a lot to these conversations.

I’d describe what you’re talking about as improved business planning software. I am sure that there are existing biz software packages existing today that will give you estimated profit projections based on revisions to your biz inputs.

Of course these are only as good as the programming and the data (and the willingness to listen to them), but it seems believable they may get quite good over time.

I don’t think the software that is used by shoe manufacturers will be the same as the software used by crop farmers, though it might have some pieces in common. If you agree that what you’re talking about is plausibly the descendant of biz planning software, that would point to a variety of AIs, not one that gets control over everything.

I don’t think anyone is going to hook up their entire production apparatus to the output of a planning package like these. Even if the software calculates my highest return is in switching to sorghum and finally buying the newest model of the Caterpillar automated tractor, I’m not going to let it ship out the orders for me, I’m going to want to review the options myself first.

I suppose there would be some decisions I would trust directly to the machine, and over time the zone of trust is likely to increase. I think the trusted zone is going to have to prove itself continually in order to stay where it is and/or expand, though.

I suppose you can imagine a disaster scenario where humans have trusted more and more of their production capability to software decision making, and one day it all suddenly goes berserk. Something like the analog of a flash crash from automated high frequency trading. Unless we have ceded complete control to our planning software (let’s don’t do that) even this sounds more like an expensive ugly mess than a global catastrophe.

Interested to hear where you disagree.

Report comment

I believe Eli’s saying something slightly different than what you think. Specifically, I think he’s saying something close to, “We could (theoretically at some point in the future) write software that, given a utility function, will optimize for it; we don’t actually need to know the laws of physics for it to do this,” while I think you’re reading us as needing to also give the program the actual laws of physics for the world we run it in.

This still leaves the human doing the hard parts of identifying what they want and of writing the program, and it leaves the AI nonconscious. In general, when people use the term AI here, they don’t mean something conscious, but instead something that can solve problems.

The danger that people are worried about comes from thinking humans are likely to get the second problem I mentioned right, but fail at the first. Thus, we have a program that will optimize for what we tell it we want, but unfortunately it won’t actually optimize for what we really want. Another potential failure mode is where we tell it correctly what we want, but in the process of becoming powerful enough to do it the program in effect “forgets” what it’s trying to do. Both options leave us with a really powerful probably-nonconscious entity that does something we don’t want, where the typical example of such an entity is the “paperclipper”.

Report comment

It sounds like you’re saying, if you can describe everything that is important to you in the form of a utility function, and then express everything in the world in the form of data that can be applied to a utility function, we have reason to believe we can eventually write software that will complete those calculations.

Is that right? If so, it sounds like humans still have to do all the hard parts, and that software doesn’t sound like anything that would (presumably) be conscious like AIs are typically described as being.

But I may have missed your meaning.

> TL;DR: BRO DO YOU EVEN SOLOMONOFF?

No, I never have before.

I did look him up on Wikipedia, and from there skimmed “The Time Scale of Artificial Intelligence: Reflections on Social Effects”, which was quite readable for a layperson. I particularly appreciated his attempts to define milestones.

In that, he talks about a general theory of problem solving (Milestone B). To assume that such a thing is even possible seems like a reach to me. I can’t help wondering if he means something different than what I take the phrase to mean, but his 1-4 seem to be talking about the same kind of stuff.

I wish he had put milestones marking out progress on this particular step.

I note he thought such a thing was most likely to appear in 2 to 25 years. That was in 1985. Is there still optimism that a general theory of problem solving (and learning) is going to be formulated?

Report comment

Report comment

Report comment

But look at the difference between the two risks. The US govt. was so sure the atom bomb would work they devoted extraordinary resources to the project, employing tens of thousands of people for years to bring about the Trinity Test.

Even according to MIRI, AI is somewhere between decades away and never. Maybe if the US govt. started a project similar to the Manhattan, I would sit up and take notice. Until then, yawn.

Report comment

Report comment

Why wouldn’t a demon hurt us?
Why wouldn’t malevolent aliens hurt us?
Why wouldn’t Nazgul hurt us?

In order for people to take a risk seriously, the risk has to exist.

Until you can post a link to an AI that can converse with me like HAL from 2001, I’m not going to be any more afraid of AI than I am of frickin Lord Voldemort.

Report comment

Any argument against AI Risk that did not pose any problem for Hitler Risk is worthless.

Report comment

But actually building one is quite difficult, and existing ML/narrow-AI applications range from having very little to do with “an AI” to having almost nothing to do with “an AI”.

TL;DR: BRO DO YOU EVEN SOLOMONOFF?

Report comment