[ryan-c]

I'm using an M3 Ultra w/ 512GB of RAM, using LMStudio and mostly mlx models. It runs massive models with reasonable tokens per second, though prompt processing can be slow. It handles long conversations fine so long as the KV cache hits. It's usable with opencode and crush, though my main motivation for getting it was specifically to be able to process personal data (e.g. emails) privately, and to experiment freely with abliterated models for security research. Also, I appreciate being able to run it off solar power.

I'm still trying to figure out a good solution for fast external storage, I only went for 1TB internal which doesn't go very far with models that have hundreds of billions of parameters.

[ProllyInfamous]

>trying to figure out ... fast external storage

Acasis makes 40gbps external nVME cases. Mine feels quick (for non-LLM tasks).

I also use 10gbps Terramaster 4-bay RAIDs (how I finally retired my Pro5,1).

>energy usage

This thing uses an order of magnitude -less- energy than the computer it replaced, and is faster in almost every aspect.

[ryan-c]

10gbps is slow enough to be annoying when you're loading a 200GB model, unfortunately.

[gneuron]

This is the way brother