I think it's anyone with mental health issues, not just suicidal ideations. They are designed to please the user and that can be very self destructive.
Maybe if it had a memory of your mental health issues it could at least provide some grounding truth. It can be a sad scary and lonely world for people with mental health issues.
"The things you are describing might not be happening. I think it would be a good time to check in with your mental health provider."
or
"I don't see any worms crawling on your skin. This may not be real."
Or whatever is correct way to deal with these things.
Unfortunately, that kind of truth isn't even encoded in the models. It doesn't really get ideas and can't understand whether worms are actually crawling on your skin because factual concepts aren't part of the embeddings as far as we can tell. It just knows how words relate to each other, and if you keep telling it there are worms on your skin, it will start extrapolating what someone who sees worms crawling on someone's skin would say.
sixothree|1 month ago
"The things you are describing might not be happening. I think it would be a good time to check in with your mental health provider." or "I don't see any worms crawling on your skin. This may not be real." Or whatever is correct way to deal with these things.
shadowgovt|1 month ago