This is very close to some thinking I've had. As we're finding, there's very few fixed points between the model and its tokens, and reality. In a more traditional rules-based AI, the designers create fixed points as they go. A general knowledge database with things like (object: apple, properties: tasty) is a giant collection of such fixed points, linking real human things, with internal AI states and objects. When those links exist, the AI's reasoning can be comprehensible to a human, because they can be explained in reference to human-understandable points.
When you go looking for such fixed points in a LLM, they're quite conspicuously absent. A LLM has essentially no internal model of reality other than how tokens relate to each other. And it's pretty clear based on the 'hallucinatory' property of LLMs, that the relationship of those tokens, and actual reality, is more accidentally coincidental than anything else.
I suspect it's us humans fixing a particular token to something concrete, to something in reality or the human mental world, when we view the LLM's output. We project understanding and meaning on to it. There is none until a human views it. The mystery is how such a mindless process often results in useful, meaningful, seemingly-mindful output, when interpreted by a sentient mind. Our instincts tell us that sufficiently complex intelligence requires consciousness and sentience; but to lean into that, assume it's there and to look for it, is motivated, backward reasoning. It's not there.
A model that can continuously learn, has something like a short and long term memory, and multiple input modalities -- a sort of transformer embodied in a robot that can move around on its own (don't blame me when it happens - not my idea!) -- might be able to establish its own fixed points. I suppose that's how humans figure out reality -- manipulating it and interpreting it through multiple senses. But even then, it may not create something which has fixed points which are the fixed points humans use or value. Such a system might be completely incomprehensible to us in how it thinks, not sentient, and yet still intelligent.
With CNNs they definitely go up abstraction layers from pixels -> lines -> curves -> abstract shapes. So if CNNs can do that then I assume that transformers can do something similar with language. But it's tough to prove because the way you do that with CNNs is you just visualize the output at each layer into an image. With a language model you have to discretize the embeddings into tokens and that isn't straigtforward.
I wonder if multimodal LLMs will be able to ground these points with reality since it can connect language and images together.
retrac|2 years ago
When you go looking for such fixed points in a LLM, they're quite conspicuously absent. A LLM has essentially no internal model of reality other than how tokens relate to each other. And it's pretty clear based on the 'hallucinatory' property of LLMs, that the relationship of those tokens, and actual reality, is more accidentally coincidental than anything else.
I suspect it's us humans fixing a particular token to something concrete, to something in reality or the human mental world, when we view the LLM's output. We project understanding and meaning on to it. There is none until a human views it. The mystery is how such a mindless process often results in useful, meaningful, seemingly-mindful output, when interpreted by a sentient mind. Our instincts tell us that sufficiently complex intelligence requires consciousness and sentience; but to lean into that, assume it's there and to look for it, is motivated, backward reasoning. It's not there.
A model that can continuously learn, has something like a short and long term memory, and multiple input modalities -- a sort of transformer embodied in a robot that can move around on its own (don't blame me when it happens - not my idea!) -- might be able to establish its own fixed points. I suppose that's how humans figure out reality -- manipulating it and interpreting it through multiple senses. But even then, it may not create something which has fixed points which are the fixed points humans use or value. Such a system might be completely incomprehensible to us in how it thinks, not sentient, and yet still intelligent.
goldwelder42|2 years ago
I wonder if multimodal LLMs will be able to ground these points with reality since it can connect language and images together.