[Barracoon]

Who built the automated lab that allows it to create a deadly pathogen before someone presses reboot?

I see almost no chance that an AGI starts redirecting resources like energy and compute without someone who has an economic stake in what the AI does noticing and taking action to correct the behavior so they can maintain their investment.

[ben_w]

For the first point:

All of them, I think? Better question is: which labs check the DNA sequences are safe before sending them to the printer. I know some claim to check, I hope but doubt that it's all worldwide.

For the second:

At first this looks like a good thing, you asked for growth and it gave you growth — "Who cares about the poor complaining that energy prices have gone up, they can turn use less heating/AC, right?", Martin Antoinette, probably — and like the fossil fuel and cigarette industries, you are motivated to FUD anyone who tries to warn anyone of the dangers.

Sometimes, as seems to be the case with fossil fuels, change happens soon enough. But this isn't a universal, the British ignored the USA because they assumed democracy was stupid and the rebels would see sense and go back to the superior British aristocracy… and then it was too late: https://youtu.be/Zbku2ILzGlo?si=yihnBlifkaLLseyy

[Barracoon]

So the AI has overtaken all code and manual processes within a laboratory environment to hijack the ability to print DNA sequences without anybody saying “man it’s weird that my projects never finish now”? There’s also the modeling vs real-world aspect that Halvar talked about in the submission - the AI needs to do some degree of real world testing to make sure the pathogen properly kills humans. It then needs to distribute that pathogen around the world quickly enough to prevent humans from cutting power.

That’s all incredibly fantastical and sounds vaguely plausible unless you’ve had to interact with bureaucracies.

To your second: I’m not sure that we’re talking about the same thing? If I own $100mm worth of H100s and their electricity, networking, and engineering overhead, I’m very interested in their utilization and return on investment. I’m going to have specific goals and milestones. When those are not met because the AI is doing something else, I’m going to direct introspection (which presumably is what kicks off the reboot in the discussion on point one).

And since you mentioned the poor having to pay higher prices, that highlights the fact that AI is not operating in a vacuum. If AI starts making life that much worse for people, they are going to rise up. FUD works to some degree, but there are always people without Internet or willing to martyr themselves for a cause.