[tshadley]

The article referenced the Oxford semantic entropy study but failed to clarify that the issue greatly simplifies LLM hallucination (making most of the article outdated).

When we are not sure of an answer we have two choices: say the first thing that comes to mind (like an LLM), or say "I'm not sure".

LLMs aren't easily trained to say "I'm not sure" because that requires additional reasoning and introspection (which is why CoT models do better); hence hallucinations occur when training data is vague.

So why not just measure uncertainty in the tokens themselves? Because there are many ways to say the same thing, so a high entropy answer may only reflect uncertainty in synonyms-- many ways to say the same thing.

The paper referenced works to eliminate semantic similarity from entropy measurements, leaving much more useful results, proving that hallucination is conceptually a simple problem.

https://www.nature.com/articles/s41586-024-07421-0

[int_19h]

QwQ is really good at saying "I'm not sure", to the point where it will sometimes check the correct and obviously trivial answer a dozen times before concluding that it is, indeed, correct. And it does punch way above its weight for its size.

So, basically, the answer seems to be to give models extreme anxiety and doubt in their own abilities.

[jimmySixDOF]

funny I was just looking at this approach and see what's new in the wild (a work in progress) see Entropix [1] by [2]

[1] https://github.com/xjdr-alt/entropix

[2] https://x.com/_xjdr

[sfink]

> proving that hallucination is conceptually a simple problem.

...proving that this one particular piece of the hallucination problem may be conceptually simple.

FTFY

[tshadley]

> ...proving that this one particular piece of the hallucination problem may be conceptually simple.

Everything mentioned in the article boils down to that one particular piece-- non-detected uncertainty. The architecture constraints referenced are all situations that cause uncertainty. Training data gaps of course increase uncertainty.

Their solutions are a shotgun blast of heuristics that all focus on reducing uncertainty-- CoT, RAG, fine-tuning, fact-checking -- while somehow avoiding actually measuring uncertainty and using that to eliminate hallucinations!