[salawat]

>only if AI gets competent enough to help them do this do they become a problem, but by that point hopefully it's also competent enough to help everyone stop them

Tell me something; have you ever built something you later regret having built? Like you look back at it, accept you did, but realize that if you'd just been a bit wiser/knowledgeable about the world you wouldn't have done it? In the moment you're doing the thing you'll regret, you don't know in that moment anything better to do until the unpleasant consequences manifest, granting you experience.

If you haven't experienced that yet; fine, but we shouldn't be betting on existential problems with "hopefully" if we can at all avoid it. Especially when that hopefully clause involves something we're making the decision to craft, with means and methods we don't fully understand/aren't predictively ahead of, and knowing that the way these methods work have a tendency to generate/provide the basis to generate a thoroughly sycophantic construct.

[ben_w]

Sure.

To your point, my P(doom) is 0.1, but the reason it's that low is that I expect a lot of people to use sub-threshold AI to do very dangerous things which render us either (1) unwilling or (2) unable to develop post-threshold AI.

The (1) case includes people actually taking this all seriously enough, which as per your final paragraph, I agree with you that people are currently not.

Things like Roko's basilisk are a strict subset of that 0.1; there's a lot of other dooms besides that one.