[wbhart]

I've tested GPT-4 on this and it can be induced to give up on certain lines of argument after recognising they aren't leading anywhere and to try something else. But it would require thousands (I'm really under exaggerating here) of restarts to get through even fairly simple problems that professional mathematicians solve routinely.

Currently the context length isn't even long enough for it to remember what problem it was solving. And I've tried to come up with a bunch of ways around this. They all fail for one reason or another. LLMs are really a long, long way off managing this efficiently in my opinion.

[Davidzheng]

Weird time estimate given that a little more than a year ago, the leading use of LLMs was generating short coherent paragraphs (3-4 sentences)

[nijave]

I've pasted docs and error messages into GPT 3.5 and it's admitted it's wrong but usually it'll go through a few different answers before returning back to the original and looping