The "Why AGI will not happen" argument here seems to hinge a fair bit on hardware limits and "computation is physical" but the arguments don't seem very good.
Much better in my opinion is Hans Moravec's 1997 paper "When will computer hardware match the human brain?" which seems quite solid in it's reasoning - he was a roboticist and spending time trying to make robots do things like vision and given at the time they understood the retina quite well he could compare the amount of compute needed to do something equivalent to a given volume of neurons and then multiply up by the size of the brain. Conclusions were
> human brain is about 100,000 times as large as the retina, suggesting that
matching overall human behavior will take about 100 million MIPS of computer power
100 TFLOPS roughly. Or similar to a top graphics card now. Also:
> Based
on extrapolation of past trends and on examination of technologies under development, it is
predicted that the required hardware will be available in cheap machines in the 2020s.
which seems to have come to pass. I think we don't have AGI yet because the LLM algorithms are very inefficient and not right for the job - it was more a language translation algorithm that surprised people by getting quite smart if you threw huge compute and the whole internet at it.
General Intelligence has already been demonstrated to be possible by the human brain, so I don't really get how physicality is an argument against AGI. Who is to say biological computers won't be built?
You kind of have to read the article to understand the big bold headings.
He is postulating that the way we deal with physical memory (an example of L2 and L3 caches is provided) demonstrates that, as we proceed with trying to form AGI out of our classical computer architectures, there are some fundamental problems for example larger caches are slower. With human intelligence, this doesn't always seem to be a problem for some humans. If you understand the "attention" part of recent developments in this field, he's saying that transformers are the most efficient way we got to achieve that and it's starting to look like a problem from a "physicality" standpoint as the author puts it.
The "physically" here is that larger caches are not just computationally larger banks of data but are actually physically larger and, by Euclidean distance, further away. Yet paradoxically the elephant nor blue whale is not the smartest brain on the planet, the distance from the center of my head to broca's region seems to have no effect on my elocution. Etc. Studying Einstein's brain doesn't do much (I guess the insulators are somewhat important?) for understanding Einstein's intelligence ... but that is 100% critical when understanding L2 and L3 caches on a die.
> Who is to say biological computers won't be built?
No one is saying that. We're just pretty sure it's not happening in our lifetimes.
I think you misunderstand the author's intent. He's not saying "The things I can't imagine are not going to happen". He's trying to argue that, "Look, the way things are going, the diminishing returns we are already seeing, the way our hardware works, this isn't going to get us to AGI." Of course, if you had some new architecture or weird "wetware" that somehow solved these problems I'm sure this article would concede that that's not the point.
I like the insight about “we already have dark factories (or specialized AI) where we want to, making a robot that can stitch your t-shirt at home (AGI) is ineffective”.
One analogy for this is cars. We found out it’s enough to pave a network of roads for “good enough” cars, even though there are still pipe dreams about supercar so flexible it can navigate any terrain.
The issue might be that the West is delaying building the road infrastructure waiting for that AGI supercar to happen.
(Military might care about the latter though, but in reality, drones and quadrupeds and tanks will be a better choice.)
Interesting that this was flagged (I do wonder if that is reflexive or not). The main argument that context is essential to precision (in current architectures) is pretty solid.
Maybe I’m showing my roots in the symbolic AI era here, but for me it’s even simpler: the current tech hasn’t put us significantly closer to AGI because it’s still missing some key elements.
Hofstadter was on to something with his “strange loops” idea. He never really defined it well enough to operationalize, but I have a hard time faulting him for that because it’s not like anyone else has managed to articulate a less wooly theory. And it’s true that we’ve managed to incorporate plenty of loops into LLMs, but they are still pretty light on strangeness.
For example, something brains do that I suspect is essential to biological intelligence is self-optimization: they can implement processes using inefficient-but-flexible conscious control, and then store information about that process in fast but volatile chemical storage, and then later consolidate that learning and transfer it to physically encoded neural circuits, and even continue to optimize those physical circuits over time if they get signals to indicate that doing so would be worth the space and energy cost.
Comparing that kind of thing to how LLMs work, I come away with the impression that the technology is still pretty primitive and we’re just making up for it with brute force. Kind of the equivalent of getting to an output of 10,000 horsepower by using a team of 25,000 horses.
Especially when you consider that the author is apparently an "Assistant Professor at Carnegie Mellon University (CMU) and a Research Scientist at the Allen Institute for Artificial Intelligence (Ai2)".
Believing we even know what AGI "is", let alone that we understand a direct path to achieving it, is truly an article of faith. There is no scientific basis for a development of human-level intelligence within our lifetimes, let alone the buzzword du jour of "superintelligence".
I'm convinced that we'll get AGI within the decade. But only because these large AI companies are controlling the narrative for the most part, and it's in their best interest to actually be the first one to announce that they achieved AGI.
Whether the AGI that they announce is actually AGI or not is a completely different debate. The goalposts will just continue to be moved until the statement is true.
I will be more tempted to call something AGI when it can perform experimental interventions and imagine counterfactuals, a la Judea Pearl. I want it to be able to ask questions, design theoretical models, set up an experiment, and analyze what happened - all with no human intervention.
I've always wondered what motivates people to invest time taking these stances. It indicates a wildly different, nearly alien way of thinking to my own.
Real machine learning research has promise, especially over long time scales.
Imminent AGI/ASI/God-like AI/end of humanity hawks are part of a growing AI cult. The cult leaders are driven by insatiable greed and the gullible cult followers are blinded by hope.
And I say this as a developer who is quite pleased with the progress of coding assistant tools recently.
We've barely found 1-2 ways to make useful AI and there's already professors who've incorporated those into their view of the field to such a degree that they cannot even imagine we might discover another way down the line.
You usually see this lack of imagination in the crusty old sciences, not in something as fast moving as this field. Props to the guy for being ahead off the curve though.
I'm the first to shit on anyone who thinks current LLMs will take us to AGI, but I'm far from insane enough to claim this is the end of the road.
You can count me as an AGI sceptic to extent that I don't think LLMs are the approach that are going to get us there, but I'm equally confident that we will get there, and that predictive neural nets are the core of the right approach.
The article is a bit rambling, but the main claims seem to be:
1) Computers can't emulate brains due to architecture (locality, caching, etc) and power consumption
2) GPUs are maxxing out in terms of performance (and implicitly AGI has to use GPUs)
3) Scaling is not enough, since due to 2) scaling is close to maxxing out
4) AGI won't happen because he defines AGI as requiring robotics, and seeing scaling of robotic experience as a limiting factor
5) Superintelligence (which he associates with self-improving AGI) won't happen because it'll again require more compute
It's a strange set of arguments, most of which don't hold up, and both manages to miss what is actually wrong with the current approach, and to conceive of what different approach will get us to AGI.
1) Brains don't have some exotic architecture than somehow gives them an advantage over computers in terms of locality, etc. The cortex is in fact basically a 2-D structure - a sheet of cortical columns, with a combination of local and long distance connections.
Where brains are different from a von-neumann architecture is that compute & memory are one and the same, but if we're comparing communication speed between different cortical areas, or TPU/etc chips, then the speed advantage goes to the computer.
2) Even if AGI had to use matmul and systolic arrays, and GPUs are maxxing out in terms of FLOPs, we could still scale compute, if needed, just by having more GPUs and faster and.or wider interconnect.
3) As above, it seems we can scale compute just by adding more GPUs and faster interconnect if needed, but in any case I don't think inability to scale is why AGI isn't about to emerge from LLMs.
4) Robotics and AGI are two separate things. A person lying in a hospital bed still has a brain and human-level AGI. Robots will eventually learn individually on-the-job, just as non-embodied AGI instances will, so size of pre-training datasets/experience will become irrelevant.
5) You need to define intelligence before supposing what super-human intelligence is and how it may come about, but Dettmers just talks about superintelligence in hand-wavy fashion as something that AGI may design, and assumes that whatever it is will require more compute than AGI. In reality intelligence is prediction and is limited in domain by your predictive inputs, and in quality/degree by the sophistication of your predictive algorithms, neither of which necessarily need more compute.
What is REALLY wrong with the GPT LLM approach, and why it can't just be scaled to achieve AGI, is that it is missing key architectural and algorithmic components (such as incremental learning, and a half dozen others), and perhaps more fundamentally that auto-regressive self-prediction is just the wrong approach. AGI needs to learn to act and predict the consequences of it's own actions - it needs to predict external inputs, not generative sequences.
AGI will never happen for one simple and obvious reason: there will never be a consensus about what AGI is.
This has always been the issue. This is an argument I made more than 20 years ago. AGI, whatever it is as a technical problem, is mainly a TESTING problem. If you don’t solve that, then AGI is remains a matter of faith. A cult.
I would argue that AGI will happen because we have to consensus on what AGI is. This leaves it open for these large AI companies to throw their weight around and define what AGI is before they claim to have achieved it.
You and I might agree that it's not AGI, but that's not going to stop Sam Altman from using such a bogus claim to pump share prices right before an IPO.
No amount of fantastical thinking is going to coax AGI out of a box of inanimate binary switches --- aka, a computer as we know it.
Even with billions and billions of microscopic switches operating at extremely high speed consuming an enormous share of the world's energy, a computer will still be nothing more than a binary logic playback device.
Expecting anything more is to defy logic and physics and just assume that "intelligence" is a binary algorithm.
The article doesn't say anything along those lines as far as I can tell - it focuses on scaling laws and diminishing returns ("If you want to get linear improvements, you need exponential resources").
I generally agree with the article's point, though I think "Will Never Happen" is too strong of a conclusion, whereas I don't think the idea that simple components ("a box of inanimate binary switches") fundamentally cannot combine to produce complex behaviour is well-founded.
> Expecting anything more is to defy logic and physics.
What logic and physics are being defied by the assumption that intelligence doesn't require the specific biological machinery we are accustomed to?
This is a ridiculous comment to make, you do nothing to actually prove the claims you're making, which are even stronger than the claims most people will make about the potential of AGI.
So is the “binary” nature of today’s switches the core objection? We routinely simulate non-binary, continuous, and probabilistic systems using binary hardware. Neuroscientific models, fluid solvers, analog circuit simulators, etc., all run on the same “binary switches,” and produce behavior that cannot meaningfully be described as binary, only the substrate is.
Your criteria is the lack of randomness and determinism by the sound of that.
What if i had an external source of trye randomness? Very easy to add. In fact current ai algorithms have a temperature parameter that can easily utilise true randomness if you want it to.
Would you suddenly change your mind and say ok ‘now it can be AGI!’ because i added a nuclear decay based random number generator to my ai model?
Whew, someone has some /new stans. A mostly Claude written piece of meandering navel gazing that tries to punch above its philosophical weight and fails.
I don't disagree with your overall assessment, but I'm curious about the basis for specific attribution to Claude?
Also if that's actually the case, it's incredibly ironic that we have a person who, presumably, is supposed to possess "intelligence", relying on a language model to help formulate and express their ideas about why models can't be intelligent.
tim333|2 months ago
Much better in my opinion is Hans Moravec's 1997 paper "When will computer hardware match the human brain?" which seems quite solid in it's reasoning - he was a roboticist and spending time trying to make robots do things like vision and given at the time they understood the retina quite well he could compare the amount of compute needed to do something equivalent to a given volume of neurons and then multiply up by the size of the brain. Conclusions were
> human brain is about 100,000 times as large as the retina, suggesting that matching overall human behavior will take about 100 million MIPS of computer power
100 TFLOPS roughly. Or similar to a top graphics card now. Also:
> Based on extrapolation of past trends and on examination of technologies under development, it is predicted that the required hardware will be available in cheap machines in the 2020s.
which seems to have come to pass. I think we don't have AGI yet because the LLM algorithms are very inefficient and not right for the job - it was more a language translation algorithm that surprised people by getting quite smart if you threw huge compute and the whole internet at it.
(Moravec's paper which is not a bad read https://jetpress.org/volume1/moravec.pdf)
It also has
>This paper describes how the performance of AI machines tends to improve at the same pace that AI researchers get access to faster hardware.
My guess is that will come to pass. AI researchers have the hardware and the software will improve shortly.
KHRZ|2 months ago
eldavojohn|2 months ago
He is postulating that the way we deal with physical memory (an example of L2 and L3 caches is provided) demonstrates that, as we proceed with trying to form AGI out of our classical computer architectures, there are some fundamental problems for example larger caches are slower. With human intelligence, this doesn't always seem to be a problem for some humans. If you understand the "attention" part of recent developments in this field, he's saying that transformers are the most efficient way we got to achieve that and it's starting to look like a problem from a "physicality" standpoint as the author puts it.
The "physically" here is that larger caches are not just computationally larger banks of data but are actually physically larger and, by Euclidean distance, further away. Yet paradoxically the elephant nor blue whale is not the smartest brain on the planet, the distance from the center of my head to broca's region seems to have no effect on my elocution. Etc. Studying Einstein's brain doesn't do much (I guess the insulators are somewhat important?) for understanding Einstein's intelligence ... but that is 100% critical when understanding L2 and L3 caches on a die.
> Who is to say biological computers won't be built?
No one is saying that. We're just pretty sure it's not happening in our lifetimes.
I think you misunderstand the author's intent. He's not saying "The things I can't imagine are not going to happen". He's trying to argue that, "Look, the way things are going, the diminishing returns we are already seeing, the way our hardware works, this isn't going to get us to AGI." Of course, if you had some new architecture or weird "wetware" that somehow solved these problems I'm sure this article would concede that that's not the point.
tomaskafka|2 months ago
One analogy for this is cars. We found out it’s enough to pave a network of roads for “good enough” cars, even though there are still pipe dreams about supercar so flexible it can navigate any terrain.
The issue might be that the West is delaying building the road infrastructure waiting for that AGI supercar to happen.
(Military might care about the latter though, but in reality, drones and quadrupeds and tanks will be a better choice.)
sandinmyjoints|2 months ago
ryandvm|2 months ago
Sure, that's a sensible stance to take... assuming that there will be no further technological development.
jxmorris12|2 months ago
ChuckMcM|2 months ago
bunderbunder|2 months ago
Hofstadter was on to something with his “strange loops” idea. He never really defined it well enough to operationalize, but I have a hard time faulting him for that because it’s not like anyone else has managed to articulate a less wooly theory. And it’s true that we’ve managed to incorporate plenty of loops into LLMs, but they are still pretty light on strangeness.
For example, something brains do that I suspect is essential to biological intelligence is self-optimization: they can implement processes using inefficient-but-flexible conscious control, and then store information about that process in fast but volatile chemical storage, and then later consolidate that learning and transfer it to physically encoded neural circuits, and even continue to optimize those physical circuits over time if they get signals to indicate that doing so would be worth the space and energy cost.
Comparing that kind of thing to how LLMs work, I come away with the impression that the technology is still pretty primitive and we’re just making up for it with brute force. Kind of the equivalent of getting to an output of 10,000 horsepower by using a team of 25,000 horses.
IshKebab|2 months ago
taco_emoji|2 months ago
unknown|2 months ago
[deleted]
antonvs|2 months ago
jaredcwhite|2 months ago
beAbU|2 months ago
Whether the AGI that they announce is actually AGI or not is a completely different debate. The goalposts will just continue to be moved until the statement is true.
biophysboy|2 months ago
zingababba|2 months ago
jcims|2 months ago
thomasdziedzic|2 months ago
chrsw|2 months ago
Imminent AGI/ASI/God-like AI/end of humanity hawks are part of a growing AI cult. The cult leaders are driven by insatiable greed and the gullible cult followers are blinded by hope.
And I say this as a developer who is quite pleased with the progress of coding assistant tools recently.
chmod775|2 months ago
You usually see this lack of imagination in the crusty old sciences, not in something as fast moving as this field. Props to the guy for being ahead off the curve though.
I'm the first to shit on anyone who thinks current LLMs will take us to AGI, but I'm far from insane enough to claim this is the end of the road.
HarHarVeryFunny|2 months ago
The article is a bit rambling, but the main claims seem to be:
1) Computers can't emulate brains due to architecture (locality, caching, etc) and power consumption
2) GPUs are maxxing out in terms of performance (and implicitly AGI has to use GPUs)
3) Scaling is not enough, since due to 2) scaling is close to maxxing out
4) AGI won't happen because he defines AGI as requiring robotics, and seeing scaling of robotic experience as a limiting factor
5) Superintelligence (which he associates with self-improving AGI) won't happen because it'll again require more compute
It's a strange set of arguments, most of which don't hold up, and both manages to miss what is actually wrong with the current approach, and to conceive of what different approach will get us to AGI.
1) Brains don't have some exotic architecture than somehow gives them an advantage over computers in terms of locality, etc. The cortex is in fact basically a 2-D structure - a sheet of cortical columns, with a combination of local and long distance connections.
Where brains are different from a von-neumann architecture is that compute & memory are one and the same, but if we're comparing communication speed between different cortical areas, or TPU/etc chips, then the speed advantage goes to the computer.
2) Even if AGI had to use matmul and systolic arrays, and GPUs are maxxing out in terms of FLOPs, we could still scale compute, if needed, just by having more GPUs and faster and.or wider interconnect.
3) As above, it seems we can scale compute just by adding more GPUs and faster interconnect if needed, but in any case I don't think inability to scale is why AGI isn't about to emerge from LLMs.
4) Robotics and AGI are two separate things. A person lying in a hospital bed still has a brain and human-level AGI. Robots will eventually learn individually on-the-job, just as non-embodied AGI instances will, so size of pre-training datasets/experience will become irrelevant.
5) You need to define intelligence before supposing what super-human intelligence is and how it may come about, but Dettmers just talks about superintelligence in hand-wavy fashion as something that AGI may design, and assumes that whatever it is will require more compute than AGI. In reality intelligence is prediction and is limited in domain by your predictive inputs, and in quality/degree by the sophistication of your predictive algorithms, neither of which necessarily need more compute.
What is REALLY wrong with the GPT LLM approach, and why it can't just be scaled to achieve AGI, is that it is missing key architectural and algorithmic components (such as incremental learning, and a half dozen others), and perhaps more fundamentally that auto-regressive self-prediction is just the wrong approach. AGI needs to learn to act and predict the consequences of it's own actions - it needs to predict external inputs, not generative sequences.
satisfice|2 months ago
This has always been the issue. This is an argument I made more than 20 years ago. AGI, whatever it is as a technical problem, is mainly a TESTING problem. If you don’t solve that, then AGI is remains a matter of faith. A cult.
beAbU|2 months ago
You and I might agree that it's not AGI, but that's not going to stop Sam Altman from using such a bogus claim to pump share prices right before an IPO.
grantcas|2 months ago
[deleted]
jqpabc123|2 months ago
No amount of fantastical thinking is going to coax AGI out of a box of inanimate binary switches --- aka, a computer as we know it.
Even with billions and billions of microscopic switches operating at extremely high speed consuming an enormous share of the world's energy, a computer will still be nothing more than a binary logic playback device.
Expecting anything more is to defy logic and physics and just assume that "intelligence" is a binary algorithm.
Ukv|2 months ago
I generally agree with the article's point, though I think "Will Never Happen" is too strong of a conclusion, whereas I don't think the idea that simple components ("a box of inanimate binary switches") fundamentally cannot combine to produce complex behaviour is well-founded.
soulofmischief|2 months ago
What logic and physics are being defied by the assumption that intelligence doesn't require the specific biological machinery we are accustomed to?
This is a ridiculous comment to make, you do nothing to actually prove the claims you're making, which are even stronger than the claims most people will make about the potential of AGI.
seanw265|2 months ago
The article is about the constraints of computation, scaling of current inference architecture, and economics.
It is completely unrelated to your claim that cognition is entirely separate from computation.
guardian5x|2 months ago
unknown|2 months ago
[deleted]
AnotherGoodName|2 months ago
What if i had an external source of trye randomness? Very easy to add. In fact current ai algorithms have a temperature parameter that can easily utilise true randomness if you want it to.
Would you suddenly change your mind and say ok ‘now it can be AGI!’ because i added a nuclear decay based random number generator to my ai model?
NuclearPM|2 months ago
CuriouslyC|2 months ago
lysace|2 months ago
antonvs|2 months ago
I don't disagree with your overall assessment, but I'm curious about the basis for specific attribution to Claude?
Also if that's actually the case, it's incredibly ironic that we have a person who, presumably, is supposed to possess "intelligence", relying on a language model to help formulate and express their ideas about why models can't be intelligent.