One thing that is clouding this discussion is that most people are mixing up a lot of different characteristics that animals like humans have as if they were all the same thing.
So for many people they don't really distinguish between things like "reasoning", "self-aware", "conscious", "alive", "sentient", "intelligent", "has world model". They also don't distinguish between different types or varying levels of cognitive abilities.
It seems clear that high functioning LLMs must have some type of world model. But that doesn't mean it's necessarily exactly the same type of highly grounded model that a human would have, especially if it was trained on only text. It might be less rich or different but still quite useful.
Another example, LLMs clearly don't have the same type of fast adaptation in a realtime 3d environment that animals have. (That's not to say that they can't mimic it in some rough ways).
But if you don't really break all of this stuff down carefully in your head then it can be hard to accept that LLMs are doing anything interesting. Because in that worldview, it's all the same thing, so they have to give the LLM all of the other characteristics at the same time.
It's not the first time this has been a problem to accurately specify either. Throughout history animals proved nigh impossible to classify in these terms and were considered so many things, from something that's intertwined with nature as a hive mind of sorts, to complete automatons by Descartes, and later to beings that are mostly completely identical to us by Darwin.
I suppose it's also an interesting juxtaposition, animals can do everything but talk, while LLMs can do nothing but talk.
There was an interesting thought from a Microsoft researcher in an episode of This American Life. He had been given early access to GPT4 and found that it had gained an understanding of gravity and balancing. 3.5 would fail to describe a safe order for stacking 3 eggs, a bottle, a book and a nail. But 4 would give robust answers with logical justification added to each construction step for the tower.
The researcher remarked (paraphrasing) “The models are built to predict the next token. At some point the only way to improve token prediction is to start modeling the world and building an understanding of how things work.” Since I heard this I see it everywhere with LLMs. Load up a 30B parameter model on your computer and toy around with it. When it fails I now see that it’s reached a concept it doesn’t understand and is in more-or-less Markov Chain mode. Give the same task to GPT4 and it proves it’s modeled the concept appropriately because it can pull the right words out of the ether.
Perhaps the most important public discussion, at least among technical people, is making sure we really understand the experimental setup.
(I admit I could be projecting my lack of understanding; I’m not rock-solid on the experimental setup yet.)
Still to me, it matters if the linearity been described can be observed directly _in_ the model. My understanding is that it is not; rather, it is tested with an additional model. The term probing can be misunderstood; it isn’t -just- an internal probe. There are ‘derivative’ models as well.
This raises questions about what we mean when we we are talking about internal representations! In my view, a derivative of an internal representation is not the same thing.
(I welcome people who can educate, correct me, or even reframe the topics. The above thoughts are subject to change.)
It's important to note that these probe models are very simple - they're "trained" by just doing a linear regression between the hidden activations and the desired output. This means that the probes can barely do any computation themselves, so if they work at all this is a strong indication that the signal they predict was already in the hidden activations.
For even more proof, see "Figure 5: Space and time neurons in Llama-2 models" for single neurons in LLMs that already encode this information, without even having to use a probe model to extract it.
A lot of HNers were so adamant that the LLMs understand absolutely nothing and that these models are just predicting the next most likely word.
I think with this paper it becomes clear that the adamant denial was just human bias talking. LLMs crossed a certain line here.
You would think people would be amazed or in awe or react in fear at technological break through a and they often are. The weird part for LLMs was the delusion. Many people were in denial that it was any sort of breakthrough.
My only experience so far is with chatgpt, and I am not an expert in AI, or even just in LLMs.
With those disclaimers, my interactions inform my opinion that the LLM behind chatgpt has no internal world model. It shows no understanding of basic facts and makes very silly mistakes very easily. I have my bias, like anyone else, but in the case of AI in particular, I should say that I don't think there's anything especially magical or sacred about conscience and the human brain (or the brain of other animals, for that matter), and I'm sure it must be possible to arrive at different forms of intelligence starting with the "inanimate" building blocks of hardware and software, but what I've seen so far in LLMs doesn't make me support your statement that they've crossed a certain line. In fact, I'm very much let down by the experience and I'm afraid once the hype goes down, we may be in for another AI winter.
As I said at the start of the comment, I'm well aware I'm not a subject-matter expert. I'm open to being wrong. I just wanted to point out that not all of us unimpressed are in denial. I'd love to be in awe at an AI breakthrough, and I kind of feel I will be in awe sometime during my lifetime, but LLMs are not (yet?) that for me.
It's the Chinese Room argument all over again. People hear "predicting the next token" and all they can imagine is some sort of a statistical database lookup, ignoring the fact that when you have a huge corpus of data with incredibly complex internal correlations and all that data also happens to correlate with some unknown external thing, it's almost certain that a powerful learner will end up modeling that external thing if and when doing so will cause a quantum leap in prediction performance! A model that includes the external-thing hypothesis will almost certainly be simpler, ceteris paribus, than a model that doesn't.
This is a preprint that was just uploaded to arxiv two days ago. Don't be hasty and assume that it settles any matter at all. Many such claims have been made before and many counter-claims also. There is still a lively debate on the subject and it will be some time before there is agreement.
More generally, any scholarly article is a claim, and should never be read as automatically true. That's something to keep in mind.
I think LLMs are presenting some uncomfortable philosophical questions for people about how our own brains work and admitting that there is any kind of "intelligence" (even if very basic) in an LLM is an admission that our own brains may work in a similar manner.
Seems like the hyperbolic conclusion to make? The paper doesn't make that claim. Simple ml models can take data and find patterns, like separate red and blue, or xy coordinates. Those are a small amount of dimensions and easy to reason about.
This seems like the model found on its own another dimension to segregate data in deeper layers by generalizing and using context when learning. Its cool it can do that, but it still to me seems like drawing a best fit line just with higher dimensions.
I'm sure someone will correct me if they think I am way off.
Indeed. I was offering examples of how you could bootstrap cognitive processes at the beginning of the year and people were just sticking their fingers in their ears. A lot of people exercise their intellectual, economic, or political freedoms by rejecting others' efforts to gain the same things.
If you actually read the paper, the analysis absolutely does not support the claim that LLMs are capable of maintaining a spatial or chronological internal world model.
The LLMs simply activated when locations or times were mentioned, with no understanding of the relationship between different places or different times beyond the fact that different places and times had different tags. Or in other words, no internal model of the world.
The next step in the research should be ablation of these time and space neurons to see how it affects accuracy on space/time completions which would help rule out the memorization of linear probes.
It would also be reassuring if they showed that they couldn't reach similar performance when they assigned random values to the target coordinates and used non-spatial words.
The interpretation of the LLM having a "world model" is a big stretch in terminology. Encoding a coherent or accurate set of spatial coordinates for places is not equivalent to understanding the actuality of world space and the relationship between items in that space. By this definition an accurate spreadsheet of the lat-long of every major city would also constitute a "world model". It shouldn't be surprising to anyone that a model trained on (likely many) such spreadsheets would also encode that coordinate data. What's notable is that it still takes a team of human researchers to plot these raw numbers onto a map and to interpret them as experiential differentiators in physical space.
There's no doubt that an LLM can uncover whatever is structurally included in its training data, even if that encoding is implicit. What's less believable is that the LLM somehow achieves a grounded understanding of physical aspects of experience purely from streams of raw text.
I think the paper engages in some equivocation here, as the abstract differentiates between "an enormous collection of superficial statistics" and "a coherent model of the data generating process", without admitting that the first should well imply the second. But that doesn't then further imply a "world model" in the sense that we understand it as sentient beings. For us, an internal model is useful, but when our model or texts don't agree with the actuality of the world, the actuality of the world takes precedence. For an LLM there is no distinguishing between its trained representations and an actual exterior place, or even any sense that an exterior space exists.
"Encoding a coherent or accurate set of spatial coordinates for places is not equivalent to understanding the actuality of world space and the relationship between items in that space."
What do you mean by the "actuality" of world space? It certainly has memorized an enormous data set of geographic points for various important locations and can do some pretty sophisticated reasoning and inference from them, and with the code interpreter it can compute various metrics derived from them. What can this thing do that humans can't?
Current LLMs clearly have much coarser models of the world than humans do. Their training data do not include as many modalities and training data with spatiotemporal dimensions are inadequate.
Up and coming multimodal models are changing this.
> coherent model of the data generating process -- a world model.
I'd never seen a definition of world model but this seems deficient in several ways because it does not mention anything about abstraction and logical reasoning.
I’m seeking a brief almost hands-on introduction to how probing is done: in earlier papers and here. Suggestions? (Preferably with some photographs of napkin drawings.))
"Tenet" is a 2020 science fiction action-thriller film written and directed by Christopher Nolan. The story revolves around concepts of time manipulation and inversion,
making it a complex narrative that can be challenging to understand on first viewing.
Here's a basic summary:
The protagonist (referred to as "The Protagonist", played by John David Washington) is a secret agent who gets involved in a mission to prevent World War III. He is introduced to a concept called "inversion," where the entropy of people or objects can be reversed, making them move backwards in time.
His mission leads him to cross paths with a Russian oligarch named Andrei Sator (played by Kenneth Branagh), who is collecting pieces of an algorithm that can invert the entropy of the entire world, effectively reversing time and destroying the present in favor of the future.
The Protagonist also meets Kat (played by Elizabeth Debicki), Sator's estranged wife, who becomes a crucial part of the mission. The Protagonist and his partner Neil (played by Robert Pattinson) use inversion to their advantage in several action sequences, including a car chase and a final battle at Sator's secret city, where they successfully get the algorithm and prevent the destruction of the present.
The twist at the end of the film reveals that Neil was recruited by a future version of The Protagonist, and that they have been working together for much longer than the duration of the film's events. This means that the organization "Tenet", which they work for, was created by The Protagonist himself in the future. Neil's character is seen sacrificing himself to ensure the mission's success, highlighting the theme of fate and predestination in the movie.
"Tenet" is a complex film that uses its time manipulation concept to construct a narrative that loops in on itself, with events and characters revealing their true significance only as the story progresses or even after the film ends. Nolan's movie plays with the concepts of time, fate, and free will, requiring viewers to actively engage with and decipher its narrative structure.
I bet if we were able to get better output from our brains we would see something similar, for those of us that have any spatial awareness capabilities
But we don't have the output to analyze aside from our language and assumption of shared experience
“Have an internal world model,” or “give the appearance of having an internal world model”?
(Disclaimer: I’ve only read the summary, not the full paper)
Recent papers about advanced Go-playing AIs have made it abundantly clear that, at best, their conceptual understanding of Go is deeply flawed, and at worst, completely absent. Yet under most circumstances they give the appearance of super-human understanding based on superhuman play.
Do we have any reason to think LLMs are different? That they understand anything beyond “this is the best word to go next here.”
To be clear, LLM output is remarkable, all the more so if the above is true.
I think sometimes it is good to zoom out a little bit and look at things on a higher level. The LLMs are not an organic creature. It doesn't in any shape or way relate to a human being except the knowledge it was trained on. A bacteria shares over 50% of its genetic sequences with a human and we don't see it as anything more than a glorified bionic automaton. An LLM shares 0% with us. In fact its "genetic sequence" is coded on a completely different physics altogether. It is unreasonable to expect it to understand or think the same way an organic creature do.
But the outcome of its action is irrefutable. It plays and does things better better than us. That is all we need to evaluate it on. I know that the topic is about "sentience" but my point is we are evaluating a complete alien being using human's standards. Of course we will see that it is lacking. It isn't human.
The papers cites earlier papers that posit a ‘linear representation hypothesis’. Does your surprise “factor in” these papers? Do you suspect something different in play for the current paper?
No, no they don't. People looking at the output of LLMs project their own internal models of the world, time, and space onto the output and project that model as if it were the LLM's.
[+] [-] ilaksh|2 years ago|reply
So for many people they don't really distinguish between things like "reasoning", "self-aware", "conscious", "alive", "sentient", "intelligent", "has world model". They also don't distinguish between different types or varying levels of cognitive abilities.
It seems clear that high functioning LLMs must have some type of world model. But that doesn't mean it's necessarily exactly the same type of highly grounded model that a human would have, especially if it was trained on only text. It might be less rich or different but still quite useful.
Another example, LLMs clearly don't have the same type of fast adaptation in a realtime 3d environment that animals have. (That's not to say that they can't mimic it in some rough ways).
But if you don't really break all of this stuff down carefully in your head then it can be hard to accept that LLMs are doing anything interesting. Because in that worldview, it's all the same thing, so they have to give the LLM all of the other characteristics at the same time.
[+] [-] moffkalast|2 years ago|reply
I suppose it's also an interesting juxtaposition, animals can do everything but talk, while LLMs can do nothing but talk.
[+] [-] teaearlgraycold|2 years ago|reply
The researcher remarked (paraphrasing) “The models are built to predict the next token. At some point the only way to improve token prediction is to start modeling the world and building an understanding of how things work.” Since I heard this I see it everywhere with LLMs. Load up a 30B parameter model on your computer and toy around with it. When it fails I now see that it’s reached a concept it doesn’t understand and is in more-or-less Markov Chain mode. Give the same task to GPT4 and it proves it’s modeled the concept appropriately because it can pull the right words out of the ether.
[+] [-] famouswaffles|2 years ago|reply
https://twitter.com/wesg52/status/1709551516577902782?t=3b2F...
[+] [-] nmarinov|2 years ago|reply
[+] [-] joshuahedlund|2 years ago|reply
[+] [-] civilitty|2 years ago|reply
This reminds me of all those fMRI studies that look at brain blood flow before the dead salmon experiment came out.
[+] [-] xpe|2 years ago|reply
(I admit I could be projecting my lack of understanding; I’m not rock-solid on the experimental setup yet.)
Still to me, it matters if the linearity been described can be observed directly _in_ the model. My understanding is that it is not; rather, it is tested with an additional model. The term probing can be misunderstood; it isn’t -just- an internal probe. There are ‘derivative’ models as well.
This raises questions about what we mean when we we are talking about internal representations! In my view, a derivative of an internal representation is not the same thing.
(I welcome people who can educate, correct me, or even reframe the topics. The above thoughts are subject to change.)
[+] [-] flaghacker|2 years ago|reply
For even more proof, see "Figure 5: Space and time neurons in Llama-2 models" for single neurons in LLMs that already encode this information, without even having to use a probe model to extract it.
[+] [-] canjobear|2 years ago|reply
[+] [-] corethree|2 years ago|reply
I think with this paper it becomes clear that the adamant denial was just human bias talking. LLMs crossed a certain line here.
You would think people would be amazed or in awe or react in fear at technological break through a and they often are. The weird part for LLMs was the delusion. Many people were in denial that it was any sort of breakthrough.
[+] [-] fipar|2 years ago|reply
With those disclaimers, my interactions inform my opinion that the LLM behind chatgpt has no internal world model. It shows no understanding of basic facts and makes very silly mistakes very easily. I have my bias, like anyone else, but in the case of AI in particular, I should say that I don't think there's anything especially magical or sacred about conscience and the human brain (or the brain of other animals, for that matter), and I'm sure it must be possible to arrive at different forms of intelligence starting with the "inanimate" building blocks of hardware and software, but what I've seen so far in LLMs doesn't make me support your statement that they've crossed a certain line. In fact, I'm very much let down by the experience and I'm afraid once the hype goes down, we may be in for another AI winter.
As I said at the start of the comment, I'm well aware I'm not a subject-matter expert. I'm open to being wrong. I just wanted to point out that not all of us unimpressed are in denial. I'd love to be in awe at an AI breakthrough, and I kind of feel I will be in awe sometime during my lifetime, but LLMs are not (yet?) that for me.
Edit: s/are not in denial/are in denial/
[+] [-] Sharlin|2 years ago|reply
[+] [-] YeGoblynQueenne|2 years ago|reply
More generally, any scholarly article is a claim, and should never be read as automatically true. That's something to keep in mind.
[+] [-] andrewguenther|2 years ago|reply
[+] [-] tayo42|2 years ago|reply
This seems like the model found on its own another dimension to segregate data in deeper layers by generalizing and using context when learning. Its cool it can do that, but it still to me seems like drawing a best fit line just with higher dimensions.
I'm sure someone will correct me if they think I am way off.
[+] [-] anigbrowl|2 years ago|reply
[+] [-] gamblor956|2 years ago|reply
The LLMs simply activated when locations or times were mentioned, with no understanding of the relationship between different places or different times beyond the fact that different places and times had different tags. Or in other words, no internal model of the world.
[+] [-] umanwizard|2 years ago|reply
It already becomes clear after playing with ChatGPT-4 for five minutes. But a lot of people still refuse to accept it.
[+] [-] OneDonOne|2 years ago|reply
[+] [-] ImHereToVote|2 years ago|reply
[+] [-] benlivengood|2 years ago|reply
[+] [-] SiempreViernes|2 years ago|reply
[+] [-] didibus|2 years ago|reply
[+] [-] lsy|2 years ago|reply
There's no doubt that an LLM can uncover whatever is structurally included in its training data, even if that encoding is implicit. What's less believable is that the LLM somehow achieves a grounded understanding of physical aspects of experience purely from streams of raw text.
I think the paper engages in some equivocation here, as the abstract differentiates between "an enormous collection of superficial statistics" and "a coherent model of the data generating process", without admitting that the first should well imply the second. But that doesn't then further imply a "world model" in the sense that we understand it as sentient beings. For us, an internal model is useful, but when our model or texts don't agree with the actuality of the world, the actuality of the world takes precedence. For an LLM there is no distinguishing between its trained representations and an actual exterior place, or even any sense that an exterior space exists.
[+] [-] ComplexSystems|2 years ago|reply
What do you mean by the "actuality" of world space? It certainly has memorized an enormous data set of geographic points for various important locations and can do some pretty sophisticated reasoning and inference from them, and with the code interpreter it can compute various metrics derived from them. What can this thing do that humans can't?
[+] [-] nopinsight|2 years ago|reply
Up and coming multimodal models are changing this.
[+] [-] haltist|2 years ago|reply
I'd never seen a definition of world model but this seems deficient in several ways because it does not mention anything about abstraction and logical reasoning.
[+] [-] xpe|2 years ago|reply
[+] [-] tomjakubowski|2 years ago|reply
[+] [-] jsty|2 years ago|reply
[+] [-] kridsdale3|2 years ago|reply
"Tenet" is a 2020 science fiction action-thriller film written and directed by Christopher Nolan. The story revolves around concepts of time manipulation and inversion, making it a complex narrative that can be challenging to understand on first viewing.
Here's a basic summary: The protagonist (referred to as "The Protagonist", played by John David Washington) is a secret agent who gets involved in a mission to prevent World War III. He is introduced to a concept called "inversion," where the entropy of people or objects can be reversed, making them move backwards in time.
His mission leads him to cross paths with a Russian oligarch named Andrei Sator (played by Kenneth Branagh), who is collecting pieces of an algorithm that can invert the entropy of the entire world, effectively reversing time and destroying the present in favor of the future.
The Protagonist also meets Kat (played by Elizabeth Debicki), Sator's estranged wife, who becomes a crucial part of the mission. The Protagonist and his partner Neil (played by Robert Pattinson) use inversion to their advantage in several action sequences, including a car chase and a final battle at Sator's secret city, where they successfully get the algorithm and prevent the destruction of the present.
The twist at the end of the film reveals that Neil was recruited by a future version of The Protagonist, and that they have been working together for much longer than the duration of the film's events. This means that the organization "Tenet", which they work for, was created by The Protagonist himself in the future. Neil's character is seen sacrificing himself to ensure the mission's success, highlighting the theme of fate and predestination in the movie.
"Tenet" is a complex film that uses its time manipulation concept to construct a narrative that loops in on itself, with events and characters revealing their true significance only as the story progresses or even after the film ends. Nolan's movie plays with the concepts of time, fate, and free will, requiring viewers to actively engage with and decipher its narrative structure.
[+] [-] yieldcrv|2 years ago|reply
But we don't have the output to analyze aside from our language and assumption of shared experience
[+] [-] gcanyon|2 years ago|reply
(Disclaimer: I’ve only read the summary, not the full paper)
Recent papers about advanced Go-playing AIs have made it abundantly clear that, at best, their conceptual understanding of Go is deeply flawed, and at worst, completely absent. Yet under most circumstances they give the appearance of super-human understanding based on superhuman play.
Do we have any reason to think LLMs are different? That they understand anything beyond “this is the best word to go next here.”
To be clear, LLM output is remarkable, all the more so if the above is true.
[+] [-] galaxytachyon|2 years ago|reply
But the outcome of its action is irrefutable. It plays and does things better better than us. That is all we need to evaluate it on. I know that the topic is about "sentience" but my point is we are evaluating a complete alien being using human's standards. Of course we will see that it is lacking. It isn't human.
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] klysm|2 years ago|reply
[+] [-] nyrikki|2 years ago|reply
In theory you could represent an entire LLM as a single 2 dimensional graph of linear line segments.
It wouldn't be useful for much as the parameters are clustered in dense patches.
[+] [-] xpe|2 years ago|reply
[+] [-] toxik|2 years ago|reply
[+] [-] alexpetralia|2 years ago|reply
[+] [-] cratermoon|2 years ago|reply