[joefourier]

ASI still runs at finite speed and is limited by its hardware, and speed of its interactions with the real world. It won’t be able to recursively improve itself overnight if it only generates 10 tokens per seconds, and a second company could very well train one of its own before the first one has time to do much.

[reducesuffering]

You're not thinking of the second order meta system here. ASI isn't just one instance of an LLM responding to you in a session. It's the datacenter full of millions of LLM interacting with millions in parallel.

[joefourier]

Well in that case wouldn't that be millions of ASIs, each with contradictory goals?

I'm not saying that ASI isn't an existential threat, just that it probably won't present itself like the fanciful sci-fi scenario of a singular intelligence suddenly crossing a magic threshold and being able to take over the world. Most likely it will be some scenario we won't have predicted, the same way hardly anybody predicted LLMs.