Subject: Re: [ExI] Unfrendly AI is a mistaken idea.

From: Eugen Leitl <[email-masked]>
Date: Fri, 1 Jun 2007 13:33:57 +0200

On Fri, Jun 01, 2007 at 09:23:09PM +1000, Stathis Papaioannou wrote:

>    With all the hardware that we have networked and controlling much of
>    the technology of the modern world, has any of it spontaneously
>    decided to take over for its own purposes? Do you know of any examples

Of course not. It is arbitrarily improbable to appear by chance.
However, human-level AI is very high on a number of folks' priority
list. It definitely won't happen by chance. It will happen by design.

>    where the factory has tried to shut out the workers, for example,

Did you read my mail? Automation is very widespread in current
factories, silicon foundries specifically. You don't need to
shut out anyone, just change the product output.

>    because it would rather not be a slave to humans? The reply that

Remote resource takeover is something which will be a part
of the deployment plan, and planned by people, not the system
itself.

>    current software and hardware isn't smart enough won't do: in biology,

Do you expect your car to explode in a thermonuclear 50 MT-fireball when
you start it? Why not? Mere objections that it can't happen won't do.

>    the very dumbest of organisms are constantly and spontaneously
>    battling to take over the smartest, often with devastating results.

I don't think that the current malware situation is a genuine problem,
but many would disagree. But of course the zombies and worms are not
sentient, not yet.

-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE
_______________________________________________
extropy-chat mailing list
[email-masked]
http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat

---

From: Eugen Leitl <[email-masked]>
Date: Fri, 1 Jun 2007 14:44:21 +0200

On Fri, Jun 01, 2007 at 10:06:15PM +1000, Stathis Papaioannou wrote:

>    We don't have human level AI, but we have lots of dumb AI. In nature,

There is a qualitative difference between human-designed AI, and
naturally evolved AI. Former will never go anywhere. Because of this
extrapolations from pocket calculators and chess computers to 
robustly intelligent (even insects can be that) systems are invalid.

>    dumb organisms are no less inclined to try to take over than smarter
>    organisms (and no less capable of succeeding, as a general rule, but
>    leave that point for the sake of argument). Given that dumb AI doesn't

Yes, pocket calculators are not known for trying to take over the world.

>    try to take over, why should smart AI be more inclined to do so? And

It doesn't have to  be smart, it does have to be able to survive in
its native habitat, be it the global network, or the ecosystem. We don't
have such systems yet.

>    why should that segment of smart AI which might try to do so, whether
>    spontaneously or by malicious design, be more successful than all the

There is no other AI. There is no AI at all.

>    other AI, which maintains its ancestral motivation to work and improve

I don't see how there could be a domain-specific AI which specializes
in self-improvement. 

>    itself for humans just as humans maintain their ancestral motivation

How do you know you're working for humans? What is a human, precisely?
If I'm no longer fitting the description, how do I upgrade that description,
and what is preventing anyone else from that?

>    to survive and multiply?