[CorpOverreach]

I'd really love to talk to someone that both really believes this to be true, and has a hands-on experience with building and using generative AI.

The intersection of the two seems to be quite hard to find.

At the state that we're in the AIs we're building are just really useful input/output devices that respond to a stimuli (e.g., a "prompt"). No stimuli, no output.

This isn't a nuclear weapon. We're not going to accidentally create Skynet. The only thing it's going to go nuclear on is the market for jobs that are going to get automated in an economy that may not be ready for it.

If anything, the "danger" here is that AGI is going to be a printing press. A cotton gin. A horseless carriage -- all at the same time and then some, into a world that may not be ready for it economically.

Progress of technology should not be artitrarily held back to protect automateable jobs though. We need to adapt.

[thurn]

Which of these statements do you disagree with?

- Superintelligence poses an existential threat to humanity

- Predicting the future is famously difficult

- Given that uncertainty, we can't rule out the chance of our current AI approach leading to superintelligence

- Even a 1-in-1000 existential threat would be extremely serious. If an asteroid had a 1-in-1000 chance of hitting Earth and obliterating humanity we should make serious contingency plans.

Second question: how confident are you that you're correct? Are you 99.9% sure? Confident enough to gamble billions of lives on your beliefs? There are almost no statements about the future which I'd assign this level of confidence to.

[tsimionescu]

You could use the exact same argument to argue the opposite. Simply change the first premise to "Super intelligence is the only thing that can save humanity from certain extinction". Using the exact same logic, you'll reach the conclusion that not building superintelligence is a risk no sane person can afford to take.

So, since we've used the exact same reasoning to prove two opposite conclusions, it logically follows that this reasoning is faulty.

[geysersam]

Isn't the question you're posing basically Pascals wager?

I think the chance they're going to create a "superintelligence" is extremely small. That said I'm sure we're going to have a lot of useful intelligence. But nothing general or self-conscious or powerful enough to be threatening for many decades or even ever.

> Predicting the future is famously difficult

That's very true, but that fact unfortunately can never be used to motivate any particular action, because you can always say "what if the real threat comes from a different direction?"

We can come up with hundreds of doomsday scenarios, most don't involve AI. Acting to minimize the risk of every doomsday scenario (no matter how implausible) is doomsday scenario no. 153.

[quietbritishjim]

> Given that uncertainty, we can't rule out the chance of our current AI approach leading to superintelligence

I think you realise this is the weak point. You can't rule out the current AI approach leading to superintelligence. You also can't rule out a rotting banana skin in your bin spontaneously gaining sentience either. Does that mean you shouldn't risk throwing away that skin? It's so outrageous that you need at least some reason to rule it in. So it goes with current AI approaches.

[pembrook]

You bring up the example of an extinction-level asteroid hurling toward earth. Gee, I wonder if this superintelligence you’re deathly afraid of could help with that?

This extreme risk aversion and focus on negative outcomes is just the result of certain personality types, no amount of rationalizing will change your mind as you fundamentally fear the unknown.

How do you get out of bed everyday knowing there’s a chance you could get hit by a bus?

If your tribe invented fire you’d be the one arguing how we can’t use it for fear it might engulf the world. Yes, humans do risk starting wildfires, but it’s near impossible to argue the discovery of fire wasn’t a net good.

[OtherShrezzing]

> Superintelligence poses an existential threat to humanity

I disagree at least on this one. I don't see any scenario where superintelligence comes into existence, but is for some reason limited to a mediocrity that puts it in contention with humans. That equilibrium is very narrow, and there's no good reason to believe machine-intelligence would settle there. It's a vanishingly low chance event. It considerably changes the later 1-in-n part of your comment.

[tempfile]

> There are almost no statements about the future which I'd assign this level of confidence to.

You have cooked up a straw man that will believe anything as long as it contains a doomsday prediction. You are more than 99.9% confident about doomsday predictions, even if you claim you aren't.

[digbybk]

> I'd really love to talk to someone that both really believes this to be true, and has a hands-on experience with building and using generative AI.

Any of the signatories here match your criteria? https://safe.ai/work/statement-on-ai-risk#signatories

Or if you’re talking more about everyday engineers working in the field, I suspect the people soldering vacuum tubes to the ENIAC would not necessarily have been the same people with the clearest vision for the future of the computer.

[coryfklein]

Sounds a little too much like, "It's not AGI today ergo it will never become AGI"

Does the current AI give productivity benefits to writing code? Probably. Do OpenAI engineers have exclusive access to more capable models that give them a greater productivity boost than others? Also probably.

If one exclusive group gets the benefit of developing AI with a 20% productivity boost compared to others, and they develop a 2.0 that grants them a 25% boost, then a 3.0 with a 30% boost, etc...

The question eventually becomes, "is AGI technically possible"; is there anything special about meat that cannot be reproduced on silicon? We will find AGI someday, and more than likely that discovery will be aided by the current technologies. It's the path here that matters, not the specific iteration of generative LLM tech we happen to be sitting on in May 2025.

[Retric]

> Does the current AI give productivity benefits to writing code? Probably.

> If one exclusive group gets the benefit of developing AI with a 20% productivity boost compared to others, and they develop a 2.0 that grants them a 25% boost, then a 3.0 with a 30% boost, etc...

That’s a bit of a stretch, generative AI is least capable of helping with novel code such as needed to make AGI.

If anything I’d expect companies working on generative AI to be at a significant disadvantage when trying to make AGI because they’re trying to leverage what they are already working on. That’s fine for incremental improvement, but companies rarely ride one wave of technology to the forefront of the next. Analog > digital photography, ICE > EV, coal mining > oil, etc.

[utbabya]

> At the state that we're in the AIs we're building are just really useful input/output devices that respond to a stimuli (e.g., a "prompt"). No stimuli, no output.

It was true before we allowed them to access external systems, disregarding certain rule which I forgot the origin.

The more general problem is a mix between the tradegy of the common; we have better understanding every passing day yet still don't understand exacly why LLM perform that well emergently instead of engineered that way; and future progress.

Do you think you can find a way around access boundaries to masquerade your Create/Update requests as Read in the log system monitoring it, when you have super intelligence?

[otabdeveloper4]

> are just really useful input/output devices that respond to a stimuli

LLMs are huge pretrained models. The economic benefit here is that you don't have to train your own text classification model anymore. (The LLM was likely already trained on whatever training set you could think of.)

That's a big time and effort saver, but no different from "AI" that we had decades prior. It's just more accessible to the normal person now.

[ev7]

alignmentforum.com

[voidspark]

> Progress of technology should not be artitrarily held back to protect automateable jobs though. We need to adapt.

So you don't mind if your economic value drops to zero, with all human labour replaced by machines?

Dependent on UBI, existing in a basic pod, eating rations of slop.

[TobTobXX]

Yes! Sounds like a dream. My value isn't determined by some economic system, but rather by myself. There is so much to do when you don't have to work. Of course, this assumes we actually get to UBI first, and it doesn't create widespread poverty. But even if humanity will have to go through widespread poverty, we'd porbably come out with UBI on the other side (minus a few hundred millions starved).

There's so much to do, explore and learn. The prospect of AI stealing my job is only scary because my income depends on this job.

[cik]

> So you don't mind if your economic value drops to zero, with all human labour replaced by machines?

This was the fear when the cotton gin was invented. It was the ear when cars were created. The same complaint happened with the introduction of electronic, automated, telephone switchboards.

Jobs change. Societies change. Unemployment worldwide, is near the lowest it has ever been. Work will change. Society will eventually move to a currency based on energy production, or something equally futuristic.

This doesn't mean that getting there will be without pain.