[YossarianFrPrez]

What a terrible, awful tragedy!

A few months ago, OpenAI shared some data about how with 700 million users, 1 million people per week show signs of mental distress in their chats [1]. OpenAI is aware of the problem [2], not doing enough, and they shouldn't be hiding data. (There is also a great NYT Magazine piece about a person who fell into AI Psychosis [3].)

The links in other comments to Less Wrong posts attempting to dissuade people from thinking that they have "awoken their instance of ChatGPT into consciousness", or that they've made some breakthrough in "AI Alignment" without doing any real math (etc.) suggest that ChatGPT and other LLMs have a problem of reinforcing patterns of grandiose and narcissistic thinking. The problem is multiplied by the fact that it is all too easy for us (as a species) to collectively engage in motivated social cognition.

Bill Hicks had a line about how if you were high on drugs and thought you could fly, maybe try taking off from the ground rather than jumping out of a window. Unfortunately, people who are engaging in motivated social cognition (also called identity protective cognition) and are convinced that they are having a divine revelation are not the kind of people who want to be correct and who are therefore open to feedback. Because one could "simply" ask a different LLM to neutrally evaluate the conversation / conversational snippets. I've found Gemini to be useful for a second or even third opinion. But this means that one would be happy to be told that one is wrong.

[1] https://www.bmj.com/content/391/bmj.r2290.full [2] https://openai.com/index/strengthening-chatgpt-responses-in-... [3] https://www.nytimes.com/2025/08/08/technology/ai-chatbots-de...

[JohnMakin]

It's probably an artifact of how I use it (I turn off any kind of history or "remembering" of past conversations), but when I started becoming really impressed by tools like claude/chatgpt/etc. was the first time I was chasing down some dumb idea I had for work, convinced I was right, and it finally gently told me I was wrong (in its own way). That is exactly what I want these things to do, but it seems like most users do not want to be told they are wrong, and the companies are not very incentivized to encourage these tools to behave that way.

I have identified very few instances where something like chatGPT just randomly started praising me (outside of the whole "you're absolutely correct to push back on this" kind of thing). I guess leading questions probably have something to do with this.

[Avamander]

In one recent thread about StackOverflow dying, some people theorized that the success of LLMs and thus failing of SO could mostly be attributed to the amount of sycophancy of LLMs.

I tend to agree more and more. People need to be told when their ideas are wrong, if they like it or not.

[okayGravity]

It all has to do with specific filler words you use when prompting, especially chatGPT. If you use words that suggest a heavy (and I mean you really have to make the LLM know you're questioning), then it will question to an extent as you imply. If you look at the chats that they do have from this incident, he phrased his prompts as more convincing rather than questioning (i.e "Shes doing this because of this!") So chatGPT roleplays and goes along with the delusion.

Most people will just talk to LLMs like they are a person, even though LLMs won't understand the difference in complex social language and reasoning. It's almost like robots aren't people!

[baranul]

Companies want the money and continual engagement. People getting addicted to AI, as trusted advisor or friend, is money in their pockets. Just like having people addicted to gambling or alcohol, it's all big business.

It's becoming even more apparent, that there is a line between using AI as a tool to accomplish a task versus excessively relying on it for psychological reasons.

[DocTomoe]

> A few months ago, OpenAI shared some data about how with 700 million users, 1 million people per week show signs of mental distress in their chats

Considering that the global prevalence of mental health issues in the population is one in seven[1], that would make OpenAI users about 100 times more 'sane' than the general population.

Either ChatGPT miraculously selects for an unusually healthy user base - or "showing signs of mental distress in chat logs" is not the same thing as being mentally ill, let alone harmed by the tool.

[1] https://www.who.int/news-room/fact-sheets/detail/mental-diso...

[zahlman]

Having a mental health issue is not at all the same thing as "showing signs of mental distress" in any particular "chat". Many forms of mental illness wouldn't show up in dialogue normally; when it would, it doesn't necessarily show up all the time. And then there's the matter of detecting it in the transcript.

[tehjoker]

I don't know the full details, but 700M users and 1 million per a week, means up to 52M per year though I imagine a lot of them show up multiple weeks.

[mmooss]

> Because one could "simply" ask a different LLM to neutrally evaluate the conversation / conversational snippets.

The problem is using LLMs beyond a limited scope, which is free ideas but not reliable reasoning or, goodness forbid, decision-making.

Maybe the model for LLMs is a very good, sociopathic sophist or liar. They know a lot of 'facts', true or false, and are can con you out of your car keys (or house or job). Sometimes you catch them at a lie and their dishonesty becomes transparent. They have good ideas, though their usefulness only enhances their con jobs. (They also tell everything you say with others.)

Would you rely on them for something of any importance? Simply ask a human.

[gaigalas]

Why do you think a breakthrough in AI Alignment should require doing math?

Many alignment problems are solved not by math formulas, but by insights into how to better prepare training data and validation steps.

[YossarianFrPrez]

Fair question. While I'm not an expert on AI Alignment, I'd be surprised if any AI alignment approach did not involve real math at some point, given that all machine learning algorithms are inherently mathematical-computational in nature.

Like I would imagine one has to know things like how various reward functions work, what happens in the modern variants of attention mechanisms, how different back-propagation strategies affect the overall result etc. in order to come up with (and effectively leverage) reinforcement learning with human feedback.

I did a little searching, here's a 2025 review I found by entering "AI Alignment" into Google Scholar, and it has at least one serious looking mathematical equation: https://dl.acm.org/doi/full/10.1145/3770749 (section 2.2). This being said, maybe you have examples of historical breakthroughs in AI Alignment that didn't involve doing / understanding the mathematical concepts I mentioned in the previous paragraph?

In the context of the above article, I think it's possible that some people are talking to ChatGPT on a buzzword level end up thinking that alignment can be solved via "fractal recursion of human in the loop validation sessions" for example. It seems like a modern incarnation of people thinking they can trisect the angle: https://www.ufv.ca/media/faculty/gregschlitt/information/Wha...