[seidleroni]

Is there any way to do this with the frontier LLM's?

[red75prime]

Ask them to mark low confidence words.

[akoboldfrying]

Do they actually have access to that info "in-band"? I would guess not. OTOH it should be straightforward for the LLM program to report this -- someone else commented that you can do this when running your own LLM locally, but I guess commercial providers have incentives not to make this info available.

[seidleroni]

interesting... I'll give that a shot

[criemen]

It used to be that the answer was logprobs, but it seems that is no longer available.