[sharms]

It looks like these models won't be as useful for LLM inference which are heavily memory bandwidth constrained. The Macbook Pro page shows M3 at 100GB/s,150GB/s, and 300GB/s vs M2 at 200GB/s and 400GB/s. 400GB/s is available for M3 if you opt for the high gpu config, but interesting to see it go down across all of these models.

[minimaxir]

The presentation mentioned dynamic GPU caching: that seems like something transformer models would like.

[monocasa]

Could be, but I'd like to hear more information about what it actually entails.

My gut feeling is that it's kind of like Z compression, but using the high amount of privileged software (basically a whole RTOS) they run on the GPU to dynamically allocate pages so that scare quotes "vram" allocations don't require giant arenas.

If that's the case, I'm not sure that ML will benefit. Most ML models are pretty good about actually touching everything they allocate, in which case, lazy allocations won't help you much and may actually get in the way startup latency.

[brucethemoose2]

In addition to what mono said, llama.cpp allocates everything up front with "--mlock"

Llama.cpp (and MLC) have to read the all the model weights from RAM for every token. Batching aside, there's no way around that.