[duvenaud]

> as we notice these things, we pass laws against it

Well, the claim is that that's the sort of thing that will get harder once humans aren't involved in most important decisions.

> which is we created all this stuff, and we can (and have) modified the system to suit human flourishing

Why did we create North Korea? Why did we create WWI? We create horrible traps for ourselves all the time by accident. But so far there has been a floor on how bad things can get (sort of, most of the time), because eventually you run out of people to maintain the horrible system. But soon horrible systems will be more self-sustaining.