[gbanfalvi]

> 6. I'm not sure what tasks call for this kind of logical reasoning

Basically any tasks that fulfill legal or business requirements? Both companies and governments are rushing to put LLMs into anything they can to avoid paying people. It’s vital to ascertain that, say, a benefits application is assessed properly and the LLM doesn’t hallucinate its way into an incorrect decision.

I’d question if we really need LLMs in many of the places we’re sticking them at all (or if it’ll even be cheaper), but that’s more flawed human decision.

[ianbicking]

I think the best way to verify if a task could benefit from this is to try it by hand. What would a logical representation of legal requirements look like? If we can't model it by hand then we shouldn't expect an LLM to be able to do it either.

Probably 15 years ago I recall talking with someone at Columbia University who was working on an NLP project to help defendants understand the full implications of a plea deal. For instance if you plead guilty to a misdemeanor, what are all the implications elsewhere for someone who has that on their record. Or if you are on parole and plead guilty, the plea could have an effect on your parole.

The result might look a little more like search than logic. Like "rule 39.21 applies when (logical condition)". But then I can imagine growing the logical conditions... maybe you start with the most obvious formal definitions like crime severity, but then start to pull in other definitions as formal logical states as you see where that leads.