[alasano]

We don't know whether pushing towards AGI is marching towards a dystopia.

If it's winner takes all for the first company/nation to have AGI (presuming we can control it), then slowing down progress of any kind with regulation is a risk.

I don't think there's a good enough analogy to be made, like your nuclear power/weapons example.

The hypothetical benefits of an aligned AGI outweigh those of any other technology by orders of magnitude.

[esafak]

As with nuclear weapons, there is non-negligible probability of wiping out the human race. The companies developing AI have not solved the alignment problem, and OpenAI even dismantled what programs it had on it. They are not going to invest in it unless forced to.

We should not be racing ahead because China is, but investing energy in alignment research and international agreements.

[troupo]

> We don't know whether pushing towards AGI is marching towards a dystopia.

We do know that. By literally looking at China.

> The hypothetical benefits of an aligned AGI outweigh those of any other technology by orders of magnitude.

AGI aligned with whom?