>But even NorthPole’s 224 megabytes of RAM are not enough for large language models, such as those used by the chatbot ChatGPT, which take up several thousand megabytes of data even in their most stripped-down versions.
At a glance, the supplementary materials from IBM's paper claim even less:
> NorthPole's core array includes 192 MB of flexible memory (768KB of unified memory per core). Assigning 2/3rd of this memory to parameters, such as weights and biases, provides 128MB for network storage.
My understanding is that this is a small energy-efficient chip for edge computing, like to stuff in an IoT device. Way too little memory to expect to run any recent language models, but could maybe do some basic object detection in a doorbell camera, for example.
The article's author seems to believe 224 megabytes is a huge amount of memory, and is a few orders of magnitude too low on the ChatGPT estimate too.
I would expect LLM hardware to routinely support between 32 and 512GB memory in the Very Near Future. 1-4TB by the end of the decade. Custom hardware for GPT and LLM technology only started being developed in earnest in September 2022
Ukv|2 years ago
> NorthPole's core array includes 192 MB of flexible memory (768KB of unified memory per core). Assigning 2/3rd of this memory to parameters, such as weights and biases, provides 128MB for network storage.
https://www.science.org/doi/suppl/10.1126/science.adh1174/su...
My understanding is that this is a small energy-efficient chip for edge computing, like to stuff in an IoT device. Way too little memory to expect to run any recent language models, but could maybe do some basic object detection in a doorbell camera, for example.
The article's author seems to believe 224 megabytes is a huge amount of memory, and is a few orders of magnitude too low on the ChatGPT estimate too.
dartos|2 years ago
Small LLMs that have been quantized to optimize for space still sit at 4-9 gigs
hadlock|2 years ago
wrigglingworm|2 years ago
loufe|2 years ago