"Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an "intelligence explosion," and the intelligence of man would be left far behind. "
I'm reminded of the Paperclip game (https://www.decisionproblem.com/paperclips/index2.html). While the goal of this superhuman ai was to create as many paperclips as possible, a normal person living in this time would be completely oblivious to this goal and would continue to imagine a rich fulfilling world around them, complete with values and dreams. Spoiler warning: in the end everything is obliterated for more paperclips.
We have no idea the direction things will go, and we maybe can't even use our prior reasoning to guess where this tech revolution will lead us, but I'm happy we're witnessing this in an open society where such tech is more or less available to all. It's hard to think of a more promising scenario than what we're currently faced with.
bitsinthesky|3 years ago
I'm reminded of the Paperclip game (https://www.decisionproblem.com/paperclips/index2.html). While the goal of this superhuman ai was to create as many paperclips as possible, a normal person living in this time would be completely oblivious to this goal and would continue to imagine a rich fulfilling world around them, complete with values and dreams. Spoiler warning: in the end everything is obliterated for more paperclips.
We have no idea the direction things will go, and we maybe can't even use our prior reasoning to guess where this tech revolution will lead us, but I'm happy we're witnessing this in an open society where such tech is more or less available to all. It's hard to think of a more promising scenario than what we're currently faced with.