[voidUpdate]

> I haven't seen anyone online say, "We want an all-powerful, immortal system that we cannot control."

No, but having a resilient system that shouldn't be turned off in case of a nuclear strike is probably want some generals want

> I don't see evidence that current systems have any sense of will or a survival instinct.

I seem to recall some recent experiments where the LLM threatened people to try and prevent it being turned off (https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686..., ctrl-f for "blackmail"). They probably didn't have any power other than "send text to user", which is why their only way to try and perform that was to try and convince the operator. I imagine if you got one of those harnesses that can take full control of your computer and instructed it to prevent the computer from being turned off by any means necessary (and gave it root access), it would probably do some dicking about with the files to accomplish that. Its not that it's got innate self preservation, its just that the system was asked to not allow itself to be turned off, so it's doing that