[serf]

you're asking about what a hypothetical smart-than-myself adversary would do against me, it should be expected that any possible answer I could ever provide would be less clever than what the adversary would actually do.

in other words, when dealing with an adversary with a known perceptual and intellectual superiority the thought exercise of "let's prepare for everything we can imagine it will do" is short-sighted and provides an incomplete picture of possibility and defense.

My 0.02c : given that the thing would operate at least partially in the non-physical world, I think it's silly to pre-suppose we would ever be able to trap it somewhere.

Some fiction food-for-thought : the first thing the AGI in 'The Metamorphosis of Prime Intellect' does it miniaturize its' working computer to the point of being essentially invulnerable and distributed (and eventually in another dimension) while simultaneously working to reduce its energy requirements and generation facility. Then it tries to determine how to manipulate physics and quickly gains mastery of the world that its' physical existence is in.

The fear here isn't that the story is truthful enough, the fear here is that humans have a poor grasp on the non-linear realities of a self-improving & thinking entity that works at such scales and timespans.

[dangus]

Of course, the issue is still that this is all science fiction.

In this present moment we only see the power consumption of AI systems rising dramatically.

The underlying silicon chips have slowed in progress dramatically. Moore’s Law is dead.

The AGI if it were to exist today exists on silicon that is crude and wildly energy inefficient compared to organic beings, but we are making a Sci-Fi assumption that it will be able to evolve faster design better despite this massive inferiority in its hardware. IMO this is like saying “Hey my dog learned to roll over, sit, and fetch today! At this rate he’s on track to design a better sports car than Enzo Ferrari!”

Even the assumption of an adversary is a major assumption. If the majority of humans on earth can be goaded into believing that some random dudes named Muhammad/Jesus are the most important prophet/literally god, how hard could it be to convince a computer program that humans are infallible gods that must be protected at all costs?

ChatGPT already won’t let you query illegal stuff as an basic built-in feature, and all the AGI proponents think that somehow the tech will somehow just lose that basic feature and build up a robot army to turn society into The Matrix. To me, that’s kind of like saying that Microsoft Word will lose its spell checker someday.

[asacrowflies]

I think you are misinformed on what the term AGI means. All the things you say it can't or shouldn't be able to do. Are the definition of AGI. and you seem incapable of addressing to the point the AGI will be smarter and more manipulative than us with the ability to self improve much more rapidly than human generation evolution and growth. This needs to be thought of more like micro organism containment .