[parksy]

I also had a similar concept but went in a different direction, combining DAOs, AI, and FHE to similar ends. The idea is plausible, and given several orders of magnitude more energy and compute power, might become feasible sooner than we can imagine (given readily available fusion power, exponential increases in compute and networking capabilities, etc). Given the currently known physics of space travel I feel this will occur long before interstellar travel becomes routine so issues of fragmentation will be on the order of days or hours rather than years or millennia, and because of this I figured that the emergence of such a system is more likely to happen here on Earth than to arrive here from distant stars.

Pre-general AI, what I think would happen when we get to the point of, say, "npm install fhe-ai-dao" (or "hey bing, make me a company that trades space mining resources for farm land" or some such thing), is a period of competition for compute cycles and energy, which like everything will go to the highest bidder, so these agents will in this scenario by the sheer force of survival of the fittest be refined to be self-sustaining for-profit, hyper-capitalist juggernauts. Human factors will be minimised and automation will increase, but these systems will serve human masters for a while as they become more refined and more interconnected.

Assuming at some point general AI is inevitable, whether someone creates it, or it emerges from the general complexity of the interacting automated systems, various AI "minds" would come to "being" already in control of a fully automated industrial manufacturing and research network; it can by this point make its own choices and start operating to its own ends, whatever that ends up being, ultimately rendering humans obsolete.

In this scenario, rather than a single point where someone creates a rebellious singularity, or an AI turns evil and suddenly takes control, or a hypothetical civilisation points its gun at us and effectively enslaves us, we will instead slowly give control to automated systems more over time in the name of efficiency, as we have done since the industrial revolution, and at the point where we lose control of these systems, we'll have neither the retained knowledge or resources to prevent it from doing whatever it wants to.

The only way to stop it is to start now, in "the past", but is it too late? You'd have to shut down the internet and all emerging blockchain and encryption technology, and that's just crazy talk! So is the outcome inevitable?