LLMs are also good at tasks that are roughly "linear" in the sense that a group of input tokens corresponds to a group of output tokens and that translation moves from left to right.
In a hard programming problem, for instance, you have a number of actions that have to take place in a certain dependency order that you could resolve by topological sort, but in a problem like the above one bit of English corresponds to one bit of Python in basically the same order. Similarly if it couldn't translate
take the sine of...
to
Math.sin(x)
because the language didn't already have a sin function it would have to code one up. Similarly, translating between Chinese and English isn't really that hard because it is mostly a linear problem, people will accept some errors, and it's a bit of an ill-defined problem anyway. (Who's going to be mad if one word out of 50 is wrong whereas getting one word wrong in a program could mean it doesn't compile and delivers zero value?)
LLMs can do a bit of generalization beyond full text search, but people really underestimate how much they fake reasoning by using memory and generalization and how asking them problems that aren't structurally close to problems in the training set reveals their weakness. Studies show that LLMs aren't robust at all to the changes in the order of parts of problems, for instance.
The biggest problem people have reasoning about ChatGPT is that they seem to have a strong psychological need to credit it with more intelligence than it has the same way we're inclined to see a face inside a cut stem. It can do an awful lot, but it cheats pervasively and if you don't let it cheat it doesn't seem as smart anymore.
PaulHoule|1 year ago
In a hard programming problem, for instance, you have a number of actions that have to take place in a certain dependency order that you could resolve by topological sort, but in a problem like the above one bit of English corresponds to one bit of Python in basically the same order. Similarly if it couldn't translate
to because the language didn't already have a sin function it would have to code one up. Similarly, translating between Chinese and English isn't really that hard because it is mostly a linear problem, people will accept some errors, and it's a bit of an ill-defined problem anyway. (Who's going to be mad if one word out of 50 is wrong whereas getting one word wrong in a program could mean it doesn't compile and delivers zero value?)LLMs can do a bit of generalization beyond full text search, but people really underestimate how much they fake reasoning by using memory and generalization and how asking them problems that aren't structurally close to problems in the training set reveals their weakness. Studies show that LLMs aren't robust at all to the changes in the order of parts of problems, for instance.
The biggest problem people have reasoning about ChatGPT is that they seem to have a strong psychological need to credit it with more intelligence than it has the same way we're inclined to see a face inside a cut stem. It can do an awful lot, but it cheats pervasively and if you don't let it cheat it doesn't seem as smart anymore.