[DBNO]

I definitely agree that ChatGPT can be incorrect. I’ve seen that myself. In my experience, though, it’s more often right than wrong.

So when you say “in nearly every question on complex topics", I’m curious what specific examples you’re seeing.

Would you be open to sharing a concrete example?

Specifically: the question you asked, the part of the answer you know is wrong, and what the correct answer should be.

I have a hypothesis (not a claim) that some of these failures you are seeing might be prompt-sensitive, and I’d be curious to try it as a small experiment if you’re willing.

[avalys]

In one example, AWS has two options for automatic deletion of objects in S3 buckets that are versioned.

"Expire current versions" means that the object will be automatically deleted after some period.

"Permanently delete non-current versions" means that old revisions will be permanently removed after some period.

I asked ChatGPT for advice on configuring a bucket. Within a long list of other instructions, it said "Expire noncurrent versions after X days". In this case, such a setting does not exist, and the very similar "expire current versions" is exactly the wrong behavior. "Permanently delete noncurrent versions" is the option needed.

The prompt I used has other information in it that I don't want to share.