[jksk61]

funny paper, I still don't know what was the goal of it. It is evident to anyone that LLM can't perform any meaningful reasoning, why even bothering in building such an infrastructure to test whether it is able to become a "scientist".

[kkzz99]

> It is evident to anyone that LLM can't perform any meaningful reasoning

Its not and its pretty evident to anyone that has actually used SotA LLMs for more than 5 minutes.

[somenameforme]

They do a phenomenal job of guessing the next word, and our language is redundant enough that that alone, carried out recursively, can produce quite interesting results. But reasoning? I'm certain everybody has gotten in this pattern, because it happens on pretty much anything where the LLM doesn't answer right on the first shot:

---

LLM: The answer is A.

Me: That's wrong. Try again.

LLM: Oh I'm sorry, you're completely right. The answer is B.

Me: That's wrong. Try again.

LLM: Oh I'm sorry, you're completely right. The answer is A.

Me: Time to short NVDA.

LLM: As an AI language learning model without real-time market data or the ability to predict future stock movements, I can't advise on whether it's an appropriate time to short NVIDIA or any other stock.

---

[rcxdude]

I think it depends on your standards. LLMs are by far the best general purpose artificial reasoning system we've made yet, but also they aren't really very good at it. Especially more complex steps and things that require rigor (chain-of-thought prompting and such helps, but still, they have super-human knowledge but the reasoning skills of maybe a young child)

[throwup238]

There is zero chance anyone can rigorously evaluate reasoning ability in five minutes.

[imtringued]

Whenever you poke people about LLMs solving decidable/computable problems, they get defensive and claim that they are not good for that. You are supposed to generate code that solves the decidable problem, heavily implying that retrieval, approximation and translation are the only true capabilities of LLMs.

[jjtheblunt]

Are there any published scientific studies indicating such? If so, i have missed them so far.