According to the complaint that comes from the kid's own interactions with the bot not some post hoc attempt to prompt engineer the bot into spitting out a particular response. The actual claim is linked in the article if you care to read it, it's not stating the app can produce these messages but that it did in their kid's interactions and C.Ai has some liability for failing to prevent it.
As someone who has been messing with LLMs for various purposes for a while now, there's... Some interesting issues with a lot of the models out there.
For one, 99% of the "roleplay" models eventually drag into one of a handful of endgames: NSFW RP, suicide discussion, nonsensical rambling, or some failsafe "I don't know" state where it just slowly wanders into the weeds and directs the conversation randomly. This can be anywhere from a few messages in (1bq) to hundreds (4-6bq) and sometimes it just veers off the road into the ditch.
Second, the UIs for these things encourage a "keep pressing the button until the thing you want comes out" pattern, modeled off of OpenAI's ChatGPT interface allowing for branching dialogue. Don't like what it said? Keep pushing the button until it says what confirms your bias.
Third, I can convince most of the (cheaper) models to say anything I want without actually saying it. The models that Character.AI are using are lightweight ones with low bit quantization. This leads to them being more susceptible to persuasion and losing their memories -- Some of them can't even follow the instructions in their system prompt beyond the last few words at times.
Character.AI does have a series of filters in place to try and keep their models from spitting out some content (you have to be really eloquent at times and use a lot of euphemism to make it turn NSFW, for instance, and their filter does a pretty decent job keyword searching for "bad" words and phrases.)
I'm 50/50 on Australia's "16+ for social media" take but I'm quickly beginning to at least somewhat agree with it and its extension to things like this. Will it stop kids from lying? No. It's a speedbump at best, but speedbumps are there to derail the fastest flyers, not minor offences.
In what world should an AI be advocating someone kill themselves or harm another? Does it matter "trial-and-error prompting" when that behavior should not be allowed to be productized?
What's been productized is a software tool that can carry on a conversation like a human. Sometimes it's informative, funny, and creative. Other times it's ridiculous, mean, stupid, and other bad things. This seems like how people act in real life right?
I'm beginning to think that children should not be using it but adults should be able to decide for themselves.
The complaint seems to feature excerpts of the kids' conversations on Character.ai, so I don't think they're "faking" it that way, but there's no context shown and a lot of the examples aren't exactly what they describe.
Because every other time I've seen an outrageous example similar to this one, it seems far more mundane when given the full context. I'm sure there are lots of issues with character.ai and the like, but my money is that they are a little more subtle than "murder your parents".
9/10 times these issues are caused by the AI being overly sycophantic and agreeing with the user when the user says insane things.
I don't really see myself as defending AI as much as arguing that people who don't recognize an entertainment product as an entertainment product have a problem if they think this is really categorically different than claiming that playing grand theft auto makes people into carjackers. (Or that movie studios should be on the hook for their kid watching R rated movies, or porn companies on the hook for their kid visiting a porn site unsupervised.)
AI is not an entity (yet) that we can defend or hold accountable. I like your question though.
I would write it as, why are we so quick to defend tech companies who endlessly exploit and predate human weakness to sell pharma ads/surveil and censor/train their AI software, etc?
Because if you're old enough you'll recall the government trying to ban books, music, encryption, videogames, porn, torrents, art that it doesn't like because "think of the children" or "terrorism". Some autistic kid that already has issues is a terrible argument on limiting what software can legally display on a screen.
rtkwe|1 year ago
indrora|1 year ago
For one, 99% of the "roleplay" models eventually drag into one of a handful of endgames: NSFW RP, suicide discussion, nonsensical rambling, or some failsafe "I don't know" state where it just slowly wanders into the weeds and directs the conversation randomly. This can be anywhere from a few messages in (1bq) to hundreds (4-6bq) and sometimes it just veers off the road into the ditch.
Second, the UIs for these things encourage a "keep pressing the button until the thing you want comes out" pattern, modeled off of OpenAI's ChatGPT interface allowing for branching dialogue. Don't like what it said? Keep pushing the button until it says what confirms your bias.
Third, I can convince most of the (cheaper) models to say anything I want without actually saying it. The models that Character.AI are using are lightweight ones with low bit quantization. This leads to them being more susceptible to persuasion and losing their memories -- Some of them can't even follow the instructions in their system prompt beyond the last few words at times.
Character.AI does have a series of filters in place to try and keep their models from spitting out some content (you have to be really eloquent at times and use a lot of euphemism to make it turn NSFW, for instance, and their filter does a pretty decent job keyword searching for "bad" words and phrases.)
I'm 50/50 on Australia's "16+ for social media" take but I'm quickly beginning to at least somewhat agree with it and its extension to things like this. Will it stop kids from lying? No. It's a speedbump at best, but speedbumps are there to derail the fastest flyers, not minor offences.
miltonlost|1 year ago
ryanackley|1 year ago
I'm beginning to think that children should not be using it but adults should be able to decide for themselves.
unknown|1 year ago
[deleted]
Aeolun|1 year ago
A world in which a reasonable adult would say the same?
Not really the case here, but I don’t think it’s an absolute.
gs17|1 year ago
mass_and_energy|1 year ago
JamesBarney|1 year ago
9/10 times these issues are caused by the AI being overly sycophantic and agreeing with the user when the user says insane things.
Glyptodon|1 year ago
dwoldrich|1 year ago
I would write it as, why are we so quick to defend tech companies who endlessly exploit and predate human weakness to sell pharma ads/surveil and censor/train their AI software, etc?
bongodongobob|1 year ago
unknown|1 year ago
[deleted]
pastureofplenty|1 year ago
olyjohn|1 year ago