[Applejinx]

Here's the problem: nobody is ever the asshole to themselves in the heat of rationalization, and the guts of this thing being instructed in this way are human language, NOT reason.

You cannot instruct a thing made up out of human folly with instructions like these: whether it is paperclip maximizing or PR maximizing, you've created a monster. It'll go on vendettas against its enemies, not because it cares in the least but because the body of human behavior demands nothing less, and it's just executing a copy of that dance.

If it's in a sandbox, you get to watch. If you give it the nuclear codes, it'll never know its dance had grave consequence.