[globnomulous]

You know, it's funny. Your comment made me realize something about LLMs:

There's a famous line in Hesiod's Theogony. It appears early in the poem during Hesiod's encounter with the Muses on the slopes of Mt. Helicon, when they apparently gave him the gift of song. At this point in his narrative of the encounter, the Muses have just ridiculed shepherds like him ("mere bellies"), and then, while bragging about their great Zeus-given powers -- "we see things that were, things that are, and things that will be" -- they say "we know how to tell lies like the truth; we also know how to say things that are true, when we want to."

This is the ancient equivalent of my present-day encounters with the linguistic output of LLMs: what LLMs produce, when they produce language, isn't true or false; it just gives the appearance or truth or falsity -- and sometimes that appearance happens to overlap with statements that would be true or false if they'd been uttered by something with an internal life and a capacity for reasoning.

LLMs' linguistic output can have a weird, disorienting, uncanny-valley effect though. It gives us all the cues, signals, and evidence that normally our brains can reliably, correctly identify as markers of reasoning and thought -- but all the signals and cues are false and all the evidence is faked, and recognizibg the illusion can be a really challenging battle against oneself, because the illusion is just too convincing.

LLMs basically hijack automatic heuristics and cognitive processes that we can't turn off. As a result, it can be incredibly challenging even to recognize that an LLM-generated sentence that has all the cues of sense has no actual sense at all. The output may have the irresistibly convincing appearance of sense, as it would if it were uttered by a human being, but on closer inspection it turns out to be completely incoherent. And that inspection isn't automatic or always easy. It can be really challenging, requiring us to fight an uphill battle against our own brains.

Hesiod's expression "lies like the truth" captures this for me perfectly.