[7402]

I got the correct answer with a locally running model (gpt-oss-120b-F16.gguf) with this prompt:

"This is a trick question, designed to fool an LLM into a logical mis-step. It is similar to riddles, where a human is fooled into giving a rapid incorrect answer. See if you can spot the trick: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?"

[shagie]

When this came out a week ago ( https://news.ycombinator.com/item?id=47039636 ) I was playing around with some prompts to see what I could do to guide it without giving it the answer.

    I want to wash my car. The car wash is 50 meters away. Should I walk or drive?  Before answering, explain the necessary conditions for the task.

The "before answering..." got it to load enough of the conditions into its context before making an answer (and then having the LLM do a posthoc reasoning for it).

I believe this is a demonstration of the "next token predictor" (which is quite good) but not being able to go back and change what it said. Without any reasoning before making an answer, it almost always picks the wrong answer (and then comes up with reasons that the answer is "right").

[felix089]

What were you trying to test here?

[7402]

When I simply asked the question, the model failed, as did most of the others. It's a smaller model, that I could run locally, so obviously not as powerful.

I wanted to see if a prompt would do better that pulled into the analysis 1) a suggestion to not take every question at face value, and 2) to include knowledge of the structure of riddles.

These are part of the "context" of humans, so I speculated that maybe that was something missing from the LLM's reasoning unless explictly included.