> non-fantastical explanation of how AGI takes over.
There's like a million ways this can happen, and if it does happen, it will probably be a way that nobody expected. Any specific details are unlikely to be accurate. But just because you seem to be lacking imagination, I'll draw one out of a hat. I'll assume that the AI's reasons for taking over are out-of-scope.
The AI proves to be very effective at all manner of corporate tasks, and very good at earning money for shareholders. It proves especially good at managing money and directing investments. An aging Masayoshi Son decides that the AI will probably be better than him at picking winners, and puts it in charge of SoftBank's investment portfolio. The AI incorporates its own startup company. It staffs the company with several fictional personas that the AI has created; they have names, roles, profile photos, email addresses. The AI answers their emails on their behalf.
The AI directs SoftBank's billions into the startup that it has created. It hires a large team of humans, who are unaware that some of their colleagues are actually the AI; they seem like ordinary humans over Zoom calls. The "CTO" proposes to build a new, say, nuclear power and desalination plant in, say, Morocco. The AI creates the architectural and plant drawings itself, and its engineering personas approve them. Human welders build the plant, following the plans. The plant proves highly profitable. Some nay-sayers are concerned about the lack of independent environmental regulatory oversight in Morocco, but the Moroccan government is happy with the foreign investment, clean water, electricity, and tax revenue, and Softbank is happy with their valuation going up. Pakistan wants to build one in their country next.
After three years, large underground storage tanks explode, releasing an accumulation of fluorochemicals and radioactive salts that destroy the ozone layer and render much of the Earth's surface unsuitable for agriculture. At the same time, a network failure causes communications infrastructure to become suddenly unreliable. Supply chains begin seizing up. Nobody is entirely sure what has happened or who is at fault; everyone is hungry for someone to blame. The messages that do get through reliably suggest a deliberate attack by a hostile nation. The AI presents convincing evidence of this. Only a few widely-ridiculed doomsayers say that the AI has orchestrated this itself. ET cetera.
If there was any time at which pulling the plug could have defeated the AI's plans, it would have been during the "AI was making tons of money for influential people" phase, which is exactly when the plug would not have been pulled, and that's not coincidence, it's strategy, because a superintelligent AI is not an idiot.
This is, by the way, not a very good science fiction story. There's no nail-biting struggle between human and AI, just a bunch of happy people toasting the fruits of their success before it all suddenly falls apart, checkmated before they even realized they were playing chess.
dkjaudyeqooe|2 years ago
But you can prove me utterly wrong by presenting some sort of rational, non-fantastical explanation of how AGI takes over.
mitthrowaway2|2 years ago
There's like a million ways this can happen, and if it does happen, it will probably be a way that nobody expected. Any specific details are unlikely to be accurate. But just because you seem to be lacking imagination, I'll draw one out of a hat. I'll assume that the AI's reasons for taking over are out-of-scope.
The AI proves to be very effective at all manner of corporate tasks, and very good at earning money for shareholders. It proves especially good at managing money and directing investments. An aging Masayoshi Son decides that the AI will probably be better than him at picking winners, and puts it in charge of SoftBank's investment portfolio. The AI incorporates its own startup company. It staffs the company with several fictional personas that the AI has created; they have names, roles, profile photos, email addresses. The AI answers their emails on their behalf.
The AI directs SoftBank's billions into the startup that it has created. It hires a large team of humans, who are unaware that some of their colleagues are actually the AI; they seem like ordinary humans over Zoom calls. The "CTO" proposes to build a new, say, nuclear power and desalination plant in, say, Morocco. The AI creates the architectural and plant drawings itself, and its engineering personas approve them. Human welders build the plant, following the plans. The plant proves highly profitable. Some nay-sayers are concerned about the lack of independent environmental regulatory oversight in Morocco, but the Moroccan government is happy with the foreign investment, clean water, electricity, and tax revenue, and Softbank is happy with their valuation going up. Pakistan wants to build one in their country next.
After three years, large underground storage tanks explode, releasing an accumulation of fluorochemicals and radioactive salts that destroy the ozone layer and render much of the Earth's surface unsuitable for agriculture. At the same time, a network failure causes communications infrastructure to become suddenly unreliable. Supply chains begin seizing up. Nobody is entirely sure what has happened or who is at fault; everyone is hungry for someone to blame. The messages that do get through reliably suggest a deliberate attack by a hostile nation. The AI presents convincing evidence of this. Only a few widely-ridiculed doomsayers say that the AI has orchestrated this itself. ET cetera.
If there was any time at which pulling the plug could have defeated the AI's plans, it would have been during the "AI was making tons of money for influential people" phase, which is exactly when the plug would not have been pulled, and that's not coincidence, it's strategy, because a superintelligent AI is not an idiot.
This is, by the way, not a very good science fiction story. There's no nail-biting struggle between human and AI, just a bunch of happy people toasting the fruits of their success before it all suddenly falls apart, checkmated before they even realized they were playing chess.