[duvenaud]

> What incentives do any humans have to so totally delegate the functioning of the core levers of societal power that they're unable to prevent their own extinction?

Because it'll be more effective at every step than the alternative. Just like specialization is more effective, so anyone who wants to avoid poverty needs to outsource their food growing to giant mechanized farms.

> "Better machine alternatives" implies that the police and military aren't first and foremost evaluated through their loyalty. A powerful army that doesn't listen to you is not a "better" one for your purposes.

Ah, I think there's a confusion - I'm saying that the police and military will stay loyal to the state, or head of state. But that even if there is a human nominally still in charge, that that human's hands will be tied by competitive pressures to gradually marginalize their own human citizens in favor of more productive machines. Maybe a good analogy is unpopular today would be free trade deals or immigration policies enacted for economic reasons.

I think the objection of "wouldn't the few remaining humans in charge become ever-more powerful, so they could enact UBI by fiat" is a good one. But I think it's just hard for third parties to treat unpromising, unproductive beings well - others will be constantly proposing other, more lucrative, uses of their resources.

[ahartmetz]

> But I think it's just hard for third parties to treat unpromising, unproductive beings well

Strong similarity to the Resource Curse and despotic governments. The people aren't that useful if you have natural resources.

[jkey]

We actually mention the Resource Curse as an example of this in the paper.