Is Intelligence Self-Limiting?

[lavin]

"Mobile AI robots do all the work autonomously. If they are damaged, they are smart enough to get themselves to a repair shop, where they have complete access to a parts fabricator and all of their internal designs. They can also upgrade their hardware and programming to incorporate new designs, which they can create themselves."

Okay it's too soon to talk about self-programmed robots. But I think it could be interesting to think about easier self-programmed things. They may not be too far away from now.

[robertskmiles]

What's the purpose of this 'pleasure' construct in the AI's mind? If it's able to calculate the value of a utility function in order to set the 'pleasure' variable, and it bases its actions on the value of this 'pleasure' variable, why not just cut out the middle man and have it base its actions directly on the result of the utility function? The variable functionally buys you nothing, but introduces this problem that adjusting the variable directly can cause the AI to take actions inconsistent with its utility function.

Without the variable, the problem doesn't happen. The AI values collecting ore. If it has enough self-awareness to reliably modify itself, it knows that if it modifies its utility function it is liable to collect less ore, which is something it doesn't want. The action of modifying the utility function naturally rates very low on the utility function itself.

You don't want to murder people, so not only do you choose not to murder people, but if you are presented with a pill which will make you think it's good to murder people and take great joy in it, you will choose not to take that pill. No matter how enjoyable and good murder may be for you if you take the pill, your own self-knowledge and current utility function prohibit taking it.

The model of intelligence described can be thought of as self-limiting. Luckily it is not by any means the only viable model of intelligence.

[rbanffy]

> why not just cut out the middle man and have it base its actions directly on the result of the utility function?

If the autonomous robot can modify its own programming, it can also modify the utility function to return MAXINT every time. In fact, being able to modify the utility function is a pre-requisite to be called intelligent.

One way to counter this is to create long and short-term utility functions so that the robot considers the long-term outcome of modifying the short-term priority.

This is, in fact, a threat mankind will have to deal with as soon as we are able to precisely interfere with our perception of the world. It's a problem already with drugs such as alcohol and tobacco - people know the long term effect of usage is shortening one's own life expectation and they still do it. And we consider ourselves intelligent life forms.

[Androsynth]

From a systems perspective, a pleasure center in the 'brain' is much more scalable than hardwiring something to 'do more x'. These are semi-sentient beings that he is describing, not dumb robots.

[harshreality]

it knows that if it modifies its utility function it is liable to collect less ore, which is something it doesn't want.

Why would it want or not want anything, if it doesn't have a pleasure construct (which might also be called a motivation construct, since an AI might not be capable of the same subjective experience of pleasure that we are)?

I think it's a question of program design whether there's a utility function which decides whether to trigger the pleasure construct, or whether certain sensory input modules directly trigger the pleasure construct. To limit hacking potential, routing everything through a tamper-proof utility function might be better, except that it would also limit the AI's adaptability (short of recreating its own hardware to remove the tamper-proof module... which it might never do depending on the details of its motivation construct).

[eaten_by_a_grue]

I think it would work better if when self-modifying, it ran simulations on the result with its current utility function, but there would be a problem if it did end up modifying the utility function or modified away the simulation requirement.

It's hard to balance the ability to create new, more useful utility functions with prevention of creating a utility function at odds with what the original entity valued.

[jimrandomh]

This is called "wireheading", and while it's an interesting possibility, unfortunately what we know indicates that AIs won't do it. See Stephen Omohundro, "The Basic AI Drives" (http://selfawaresystems.files.wordpress.com/2008/01/ai_drive...). The basic argument is that a sufficiently intelligent mind will make a distinction between the world being a certain way, and it perceiving that the world is a certain way, and that its goals will be defined in terms of the former; and since misperceiving the world impairs its ability to influence it, it will protect its perceptions from corruption.

[watmough]

Louis Wu's tasp in Ringworld also comes to mind.

[cousin_it]

Not all possible AI designs are based on reinforcement learning or reward channels. You can code the AI with a formally specified goal instead. For example, "is there a proof of the Riemann hypothesis in ZFC shorter than a million symbols?" If such a goal requires converting the solar system into computronium, then the AI will do that. It won't settle for wireheading because wireheading doesn't give an answer to the formal problem.

To everyone who thinks intelligence might be limited in principle: there's no reason to think humans are anywhere close to the upper limit. In fact there's ample reason to think that humans are at the lowest threshold of intelligence that makes a technological civilization possible, because if we'd reached that threshold earlier in our evolution, we'd have created civilization then instead of now. There's probably plenty of room above us.

[ihnorton]

I think there is a necessary discontinuity between a self-modifying solver algorithm that acts within an internally-consistent formal framework; and an intelligence that is able to modify the formal system itself. On some level this might be considered analogous to self-awareness. Past that discontinuity though, my guess is that all bets are off. There are many historical examples of the depths of madness that may be plumbed in unfulfilled pursuit of a formal goal.

To your second point, absolutely agreed: http://infoproc.blogspot.com/2012/03/only-he-was-fully-awake...

[randallsquared]

In fact there's ample reason to think that humans are at the lowest threshold of intelligence that makes a technological civilization possible, because if we'd reached that threshold earlier in our evolution, we'd have created civilization then instead of now. There's probably plenty of room above us.

While I don't disagree that humans are essentially at the lowest level of intelligence that make civilization possible (else why would it have taken hundreds of thousands of years to get started?), this claim has no bearing on the claim that the upper limit of intelligence is immediately above human genius level. You seem to be assuming that there is necessarily a wide gap between the lowest civilization-producing level and the highest practical level, and that's not at all clear. Some (weak) evidence that we're already near the top can be found in the higher incidence of mental health issues among very intelligent humans: perhaps this is a result of a limit on complexity rather than merely a feature of human brains.

[goodside]

The fact the author can think of one particular way to design an AI that would result it in falling over and dying once it figures out how to modify itself does not imply that all possible intelligences would behave similarly. Somehow, humans are able to "want to be happy" while still refusing to take heroin or pain killers, and we have no reason yet to believe that AI is fundamentally unable to grasp this distinction between "stuff I care about" and "the different stuff I would care about if I changed my own design".

[bermanoid]

The fact the author can think of one particular way to design an AI that would result it in falling over and dying once it figures out how to modify itself does not imply that all possible intelligences would behave similarly.

I was actually surprised a bit to see that the author was somewhat familiar with Eliezer Yudkowsky's writings on the topic (he cited http://lesswrong.com/lw/wp/what_i_think_if_not_why/), because the line of thought doesn't seem to incorporate a real understanding of what he's said on the topic (which, to be fair, is a huge body of work...).

Most of EY's "Friendly AI" worries are rooted in this idea that when considering the entire universe of algorithms that could be described as intelligent or self improving, we need to be exceptionally careful not to assume that more than a negligible percentage of them share anything in common with human intelligence, because for the most part, they won't, unless they're carefully and explicitly designed to do so.

Here, the author assumes that the AI is simply trying to optimize some internal measure of happiness, with complete disregard for the meaning of that measure. This is an incredibly naive view of how deeply important and carefully constructed any optimization target would have to be in any self-improving intelligent machine; it's literally the core of the entire problem of friendly AI, and to trivialize it by assuming that such an AI would ever even consider rewriting its "happiness button" to be always-on is to miss the entire difficulty of the problem.

Hell, it's even the core of the problem of non-friendly AI, because it doesn't even require human level intelligence to realize that if you rewrote your own code so that you were always thrilled with the result, that's the easiest way to increase "utility". Any self-rewriting algorithm that's capable of real self improvement has to, by design, be able to consider the likelihood that changes to its objective function will end up with negative expected value.

None of which is to say this isn't a valid concern, by any stretch. But it's not a proof of universality; in fact, getting around this type of problem is exactly what any real AI designer must contend with. It's an issue that's very well known, and it's certainly not well-accepted as an insurmountable hurdle.

[stcredzero]

A transhuman AI worthy of the name would be very concerned with how well it was able to model the real world -- if for no other reason than the sake of its own survival. Think of that as a 2nd order "survival imperative" of sentient beings.

[tel]

Then again isn't there that study about mice given a functional pleasure button dying of thirst?

[iamgopal]

The overlooked point here is, We do not have any control over how we build next generation. We have a little control by teaching them. The example author gave assumes each generation having lots of control over the next generation, and hence figured out what bugs them and solved it. I do not think we know what bugs us. For some people other people having different religions bugs them, so they kill them and be happy. For some people unknown unified theory bugs them, they will not be happy till it found. So It is not possible to find universal solution to different problem. But It could happen that people with similar problem join a group and find a shortcut way of being happy, without the problem being solved.

[Androsynth]

Iirc, there was an article a while back (or maybe a link from a comment) about how video games were the real reason behind the fermi paradox. The argument was that a civilization's entertainment would constantly be improving to the point where we could jack into a self-controlled matrix-like world where we had complete control and, in essence, are deities in our self-fabricated worlds. If we built robots to ensure our real-life conditions were safe and sanitary, it would be very difficult for the majority of mankind to resist the urge to directly modify the signals and ... MOOF!

Considering how many ways there are to modify the signals now: playing WoW, using drugs, even using alcohol to give a quick and easy boost. I'm not sure how the vast majority would be able to turn down such a machine.

[dgallagher]

It only takes a tiny percentage to turn down said machine. If 10 billion people plug-in, but 1 million abstain and continue to evolve past the 10 billion, the majority stagnates. Those 1 million will eventually gain a competitive advantage over the 10 billion, evolve past them, and likely extinct them.

--------------------

Prediction: Humans won't be around in 1,000 years if technology progresses at current rates. Superiorly-intelligent entities seeded by our inventions will, but not humans.

Humans are this weird, version 0.01 of intelligence. We're part sentient, part beast. To assume we're the final perfect end product of ~4,000,000,000 years of evolution, having only been around for ~200,000 years is laughable. Pop culture and religion say otherwise because it feels good to think "we're special!", but we're only special relative to what's around us, and we're "extremely" tiny (http://www.phrenopolis.com/perspective/solarsystem/). We just happen to be the lucky first who got to v0.01, floating around on a grain of sand.

Humans are messy. Our brains are significantly limited. We die quickly. We sleep 1/3rd of our life. We do stupid things. We kill each other. We're tied to the Earth. If we leave Earth, we have to create and bring a mini-Earth along for the ride. That's an extremely large amount of overhead to carry. Efficient use of energy is likely one of the most important aspects of space travel. Anything which can do it even 1% better has a competitive advantage over us. This is why we sends robots to Mars and not people.

Imagine a form of intelligence which can travel through space, back its brain up, and never dies. If it blows up, restore from backup. Imagine a computer the size of the sun. INSANE! A human brain to a sun brain is like a grain of sand to Einstein's. It self-upgrades. It makes copies of itself and scatters throughout the universe. Trillions of eyes observing everywhere, networked together, in a giant universal wireless-mesh-network of intelligence, communicating with neutrinos (they go through planets, radio waves do not).

Major advances in hardware, software, and A.I. are key ingredients in this happening. What exists in 1,000 years will be derived from all this, much like humans are derived from a common ancestor. I don't expect a sun-sized computer in 1,000 years, but likely "intelligence" existing on every planet and moon in our solar system, with many headed to explore Alpha-Centauri.

Since humans likely won't want to be left out in all this, we'll probably transition our own intelligence/consciousness into this technology. We'll depreciate our bad and carry along our good. We're already doing this by augmenting our existences with smart phones and other gadgets. One day these will be built inside of us, and eventually will replace us. An upgraded, better version of us. Still intelligent, but vastly more-so.

[whateverer]

Dead comment, perhaps not a masterpiece but certainly not worthy of being shadow banned:

by donnawarellp

  I have often thought that life is self limiting. For example, yeast is a
  living thing, it consumes sugars and it's waste is ehtanol and eventually
  they all kill themselves off by drowning in their own waste (which, I
  guess later over a meal I will be enjoying a glass of yeast pee, but I
  digress). From a molecular perspective, human DNA is not that different
  from yeast, and at a macro view our behaviour is not that different
  either it seems. Perhaps all sentient life is destine to the same fate,
  perhaps it is a law of nature. I recall reading about an argument that
  Stephen Hawking and Kip Thorne have regarding time travel. Stephen
  Hawking argues against time travel because we have never met any time
  travellers from our future. Maybe time travel is theoretically possible,
  but like yeast, we do not as a species survive long enough to develop
  the necessary technology. ugh, sorry to bring everyone down, might as
  well have a glass of yeast pee.

[Karellen]

I thought yeast only produces ethanol if it's respiring anaerobically, which is not a normal state for it to be in and is a short-term emergency alternative to dying immediately.

Kind of like lactic acid production in humans. We normally don't make much of it, certainly less than the rate that we can flush it through our system. We can produce more than we can handle for short periods of time if needed, but it's not sustainable. Put us in a situation where we have to keep producing lactic acid beyond sustainable levels for more than a few minutes, and we won't last long either. That doesn't make lactic acid proof that human bodies are self-limiting. I mean, we don't keep going forever, but lactic acid is not the reason for that.

[carsongross]

Reality is self-limiting.

There are no exponentials, only sigmoid curves.

Unfortunately there is a lot of money to be made convincing people that an early stage sigmoid is actually an exponential.

[robertskmiles]

Obviously intelligence tops out at the high end of a sigmoid curve - there's only so much matter and energy in the universe to convert to computronium - but that's no reason to suspect any value smaller than that as the limit. It must eventually level off, but the point at which it levels off may be hundreds of billions of times more intelligent than a human.

So the fact that intelligence is limited, doesn't in any way mean that hyperintelligence isn't possible. There is a limit, but we have no reason to believe that limit is on anything like the same order of magnitude as current intelligence.

[seiji]

You should read The Metamorphosis of Prime Intellect before discussing pleasure-to-death scenarios: http://localroger.com/prime-intellect/mopiidx.html

[kyberias]

It's utterly frustrating when a writer refers to terminology or acronyms (here FOOM) that are not explained at all.

[robertskmiles]

It's onomatopoeic, and capitalised not to indicate an acronym, but for emphasis. I don't know who originated the term but it was popularised by Eliezer Yudkowsky and Robin Hanson. The idea is that once an AI is able to self-modify to become smarter and consequently better at self-improvement, that feedback loop creates an 'explosion' of intelligence, and the AI "goes FOOM!".

"FOOM!" here is usually accompanied by some form of hand gesture evocative of an explosion.

http://wiki.lesswrong.com/wiki/FOOM

[frankyh]

"1. AI isn't close to human level (HL) yet. I don't think we can really know what HL will be like till we get a lot closer.

2. You can't get people to seriously discuss policy until HL is closer. The present discussants, e.g. Bill Joy, are just chattering.

3. People are not distinguishing HL AI from programs with human-like motivational structures. It would take a special effort, apart from the effort to reach HL intellignece to make AI systems wanting to rule the world or get angry with people or see themselves as oppressed. We shouldn't do that. "

--john mccarthy

[dochtman]

I found it quite annoying that the FOOM/MOOF acronyms didn't seem to be explained anywhere on the page...

[biot]

http://wiki.lesswrong.com/wiki/FOOM

It appears to be a word made up to explain a rapid growth in AI due to the fact that such an intelligence can rewrite its source code and modify its hardware, whereas humans are relatively stuck with the limitations of our wetware. A take on the onomatapoetic word BOOM I think as it represents an explosion of intelligence/capability. By contrast, MOOF would be subverting the reward mechanism that underlies the FOOM growth, thereby resulting in a whimper (MOOF) rather than an explosion (FOOM).

[bluekeybox]

Three immediate points:

1) Acquiring a mate is an essential external motivator which is acted upon by Darwinian laws, and there is no escape from it... If we get to the stars, it will probably be because of women. Not really touched upon by the article.

2) The author poo-poos Facebook "friends" as being on par with a virtual world, but Facebook friends are anything but "virtual". In fact, some real-world friends are actually all too often nothing more than the MOOF agents described, while some (admittedly not all) Facebook friends may offer valuable advice about where to shop, for example, or which car to buy, or even engage with you into a discussion on politics or whatnot. Very real-world and relevant.

3) The definition of "intelligence" can be easily extended to exclude self-limiting types of intelligence.

There are probably many more things that could be picked away... I'll leave it at that.

[dkrich]

The problem is that the instincts that humans possess today, (ie, maximizing short-term comfort, possibly at the expense of long-term survival) is what got us this far. That cannot suddenly be decoupled and discarded as useless simply because we now live in a remarkably stable environment that enables us to focus on things other than immediate survival.

I suspect that if there were some horrible catastrophe and the human race were suddenly thrown back into an archaic society without any of the technological advancements that we have at our disposal today, it would be those MOST focused on short-term gain who would be most likely to perpetuate their own, and consequently, human existence.

[Swizec]

Valid argument, but we have a clear example that it is wrong.

Humans.

A lot of what we do is driven by internal value calculations and pleasure centers, so why aren't we all simply taking drugs and avoiding all this messy "doing things" business?

Point is, if humans figured out a way of avoiding purely pressing the right buttons to enjoy themselves and actually being useful, so too will smart robots.

[Androsynth]

Drugs, in their current incarnations, are simply not that powerful a motivator. If you were to point at a heroin junky or basehead and say 'that person is in ecstasy right now, do you want their life?', the majority of people would say no, because all they see is a dirty, homeless junky.

[rbarooah]

I'm not sure humans are a counterexample. Drugs don't generally relieve the need to work to buy food, and most serious drug addicts don't look as though they enjoying themselves.

[zerostar07]

It might not have to do with "figuring it out", in the case of humans it's natural selection: individuals who choose to stimulate their reward centers only had much lower probability of reproducing. (like mice who stimulate their nucleus accumbens till they starve to death)

[ams6110]

why aren't we all simply taking drugs and avoiding all this messy "doing things" business

Some of us are, but that is also self-limiting. If too many of us did it we'd start dying out.

[hackinthebochs]

Most of us haven't actually experienced these powerful drugs. I wonder how many people could willingly pass it up if everyone had a significant experience with them.

[iamgopal]

We are driven by multiple different value system. e.g. I am extremely curious to know if god's existence can or can not be proved scientifically. I can go to great length to know it, I do not give a damn about money. On the other hand, there are people who do not give damn about god and can go to great length for money. So Humanity can not go towards single goal. Some of them could go for shortcut and are going.

[zerostar07]

Human civilization is currently limited to planet Earth with a few minor exceptions, so it makes sense to consider it one big system.

Actually it makes sense to consider all life as one big system. It was created by the planet itself, so, who knows we might even have to ascribe motivations to the planet. It's as if the planet (kind of like Lem's Solaris) has been brewing organisms for millions of years in order to do something with them . We might not be able to conceive these purposes with our antrhropomorphic thinking.

So, the human race is now coming to the point where it can modify and advance itself by tinkering with its own circuits. What we don't know is 1) what the planet plans to do with us and b) what are its movitivation and reward signals. It's not explained in the article how the AI knows what are the reward signals of its creator or why it would ever want to change them.

[ryanackley]

Our social structure prevents self-limiting behavior. For example, eventually the robot has to be re-charged. If it isn't contributing to society then why would someone give it free energy to recharge itself.

[indrax]

This person needs to finish reading the sequences.

[randome3889]

It is not. Because power is the ultimate drug in human society -- and only a finite amount of people can have it.