[carsoon]

These models don't even choose 1 outcome. They list probabilities of ALL the tokens outcomes and the backend program decides to choose the one that is most probable OR a different one.

But in practical usage, if an llm does not rank token probability correctly it will feel the same as it "lying"

They are supposed to do whatever we want them to do. They WILL do what the deterministic nature of their final model outcome forces them to do.