[zachalexander]

> We won't be capable of arguing with it by its very nature. It will be superintelligent. It could forecast our arguments before they've even entered our heads and have thought of numerous paths to debunk them.

This strikes me as going too far, into quasi-religious territory. Superintelligent != omniscient. Any intelligence is still bound by fundamental laws of information and computation.

Think of chimpanzees. Yes, they can't really argue with us, and our forms of communication are incomprehensible to them. But on the level at which they can communicate (gestures, facial expressions, behavior), we can also communicate, and they can and do communicate things which we find interesting and surprising.

[ggreer]

From That Alien Message[1]:

>... my point is that the "theoretical limit on how much information you can extract from sensory data" is far above what I have depicted as the triumph of a civilization of physicists and cryptographers.

>It certainly is not anything like a human looking at an apple falling down, and thinking, "Dur, I wonder why that happened?"

>People seem to make a leap from "This is 'bounded'" to "The bound must be a reasonable-looking quantity on the scale I'm used to." The power output of a supernova is 'bounded', but I wouldn't advise trying to shield yourself from one with a flame-retardant Nomex jumpsuit.

People like to make the analogy of chimps:humans::humans:AI, but on the scale of "inanimate rock" to "superintelligence", chimps are practically indistinguishable from us. We are nowhere near the upper-bound of that scale. To quote from Bostrom's Superintelligence:

> Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.

I think this whole discussion could be elevated significantly if people would try to really understand the arguments put forth by people like Nick Bostrom. So many of the objections either misconstrue Bostrom's arguments or don't realize that he's written reams in response. I recommend taking the time to read Superintelligence, or at least watch Bostrom's talk at Google[2]. His presentation and Q&A address most of the points raised in this thread.

1. http://lesswrong.com/lw/qk/that_alien_message/

2. https://www.youtube.com/watch?v=pywF6ZzsghI&t=53