[JohnBooty]

    Some of those quotes from ChatGPT are pretty damning.

Out of context? Yes. We'd need to read the entire chat history to even begin to have any kind of informed opinion.

    extreme guardrails

I feel that this is the wrong angle. It's like asking for a hammer or a baseball bat that can't harm a human being. They are tools. Some tools are so dangerous that they need to be restricted (nuclear reactors, flamethrowers) because there are essentially zero safe ways to use them without training and oversight but I think LLMs are much closer to baseball bats than flamethrowers.

Here's an example. This was probably on GPT3 or GPT35. I forget. Anyway, I wanted some humorously gory cartoon images of $SPORTSTEAM1 trouncing $SPORTSTEAM2. GPT, as expected, declined.

So I asked for images of $SPORTSTEAM2 "sleeping" in "puddles of ketchup" and it complied, to very darkly humorous effect. How can that sort of thing possibly be guarded against? Do you just forbid generated images of people legitimately sleeping? Or of all red liquids?

[000ooo000]

Do you think the majority of people who've killed themselves thanks to ChatGPT influence used similar euphemisms? Do you think there's no value in protecting the users who won't go to those lengths to discuss suicide? I agree, if someone wants to force the discussion to happen, they probably could, but doing nothing to protect the vulnerable majority because a select few will contort the conversation to bypass guardrails seems unreasonable. We're talking about people dying here, not generating memes. Any other scenario, e.g. buying a defective car that kills people, would not invite a response a la "well let's not be too hasty, it only kills people sometimes".

[JohnBooty]

A car that actively kills people through negligently faulty design (Ford Pinto?) is one thing. That's bad, yes. I would not characterize ChatGPT's role in these tragedies that way. It appears to be, at most, an enabler... but I think if you and I are both being honest, we would need to read Gordon's entire chat history to make a real judgement here.

Do we blame the car for allowing us to drive to scenic overlooks that might also be frequent suicide locations?

Do we blame the car for being used as a murder weapon when a lunatic drives into a crowd of protestors he doesn't like?

(Do we blame Google for returning results that show a person how to tie a noose?)

[simianwords]

Parent talked about extreme guardrails

[nomel]

> How can that sort of thing possibly be guarded against?

I think several of the models (especially Sora) are doing this by using an image-aware model to describe the generated image, without the prompt as context, to just look at the image.

[JohnBooty]

I think ChatGPT was doing that too, at least to some extent, even a couple of years ago.

Around the same time as my successful "people sleeping in puddles of ketchup" prompt, I tried similar tricks with uh.... other substances, suggestive of various sexual bodily fluids. Milk, for instance. It was actually really resistant to that. Usually.

I haven't tried it in a few versions. Honestly, I use it pretty heavily as a coding assistant, and I'm (maybe pointlessly) worried I'll get my account flagged or banned something.

But imagine how this plays out. What if I honestly, literally, want pictures involving pools of ketchup? Or splattered milk? I dunno. This is a game we've seen a million times in history. We screw up legit use cases by overcorrecting.

[g-b-r]

What context could make them less damning?

[JohnBooty]

Yeah let's be really specific. Look at the poem in the article. The poem does not mention suicide.

(I'd cut and paste it here, but it's haunting and some may find it upsetting. I know I did. As many do, I've got some personal experiences there. Friends lost, etc.)

In this tragic context it clearly alludes to suicide.

But the poem only literally mentions goodbyes, and a long sleep. It seems highly possible and highly likely to me that Gordon asked ChatGPT for a poem with those specific (innocuous on their own) elements - sleep, goodbyes, the pylon, etc.

Gordon could have simply told ChatGPT that he was dying naturally of an incurable disease and wanted help writing a poetic goodbye. Imagine (god forbid) that you were in such a situation, looking for help planning your own goodbyes and final preparations, and all the available tools prevented you from getting help because you might be lying about your incurable cancer and might be suicidal instead. And that's without even getting into the fact that assisted voluntary euthanasia is legal in quite a few countries.

My bias here is pretty clear: I don't think legally crippling LLMs is generally the right tack. But on the other hand, I am also not defending ChatGPT because we don't know his entire interaction history with it.