I want to know if this is any different than all of the AMD AI Max PCs with 128gb of unified memory? The spec sheet say "128 GB LPDDR5x", so how is this better?
The GPU is significantly faster and it has cuda, though I'm not sure where it'd fit in the market.
At the lower price points you have the AMD machines which are significantly cheaper, even though they're slower and with worse support. Then there's apple's with higher memory bandwidth and even the nvidia agx Thor is faster in GPU compute at the cost of worse CPU and networking, and at the 3-4K price point even a threadripper system becomes viable that can get significantly more memory
> The GPU is significantly faster and it has cuda,
But (non-batched) LLM processing is usually limited by memory bandwidth, isn't it? Any extra speed the GPU has is not used by current-day LLM inference.
andsoitis|4 months ago
Framework's AMD AI Max PCs also come with LPDDR5x-8000 memory: https://frame.work/desktop?tab=specs
Numerlor|4 months ago
At the lower price points you have the AMD machines which are significantly cheaper, even though they're slower and with worse support. Then there's apple's with higher memory bandwidth and even the nvidia agx Thor is faster in GPU compute at the cost of worse CPU and networking, and at the 3-4K price point even a threadripper system becomes viable that can get significantly more memory
yencabulator|4 months ago
But (non-batched) LLM processing is usually limited by memory bandwidth, isn't it? Any extra speed the GPU has is not used by current-day LLM inference.
BoredPositron|4 months ago