Can LLMs reason about math? The Subtraction Trick Test

[mapontosevenths]

This is really clever, but I think Gemini got it on the first try. I kept my prompts close to yours, but didn't include the initial framing bit about how it was supposed to be an expert.

https://gemini.google.com/share/b66e0158ee29

[MakeAJiraTicket]

Thank you! Gemini has consistently been the best performer that I've tried, but they always require the connection to be made explicit. The point of the test is that it is very low complexity and is very targeted toward what can be considered reasoning and these models can't produce the connection without prodding.

In the ideal case of reasoning you would simply present the methods and they'd bridge the gap independently when it is brought to the forefront of their context together, but it doesn't happen.