[_jonas]

To try out an existing product that quantifies LLM uncertainty (accurately incorporating both aleatoric & epistemic uncertainty), you can try this Trustworthy Language Model I built (after similar research):

https://tlm.cleanlab.ai/

TLM is an API you can use to quantify uncertainty of any LLM model: https://help.cleanlab.ai/tutorials/tlm/

Benchmarks showing these estimates more reliably detect bad answers & hallucinations than logprobs, LLM-as-judge, Selfcheck-GPT, etc: https://cleanlab.ai/blog/trustworthy-language-model/