top | item 45837602 (no title) enigma101 | 3 months ago what's the hardware needed to run the trillion parameter model? discuss order hn newest CamperBob2|3 months ago It's an MoE model, so it might not be that bad. The deployment guide at https://huggingface.co/moonshotai/Kimi-K2-Thinking/blob/main... suggests that the full, unquantized model can be run at ~46 tps on a dual-CPU machine with 8× NVIDIA L20 boards.Once the Unsloth guys get their hands on it, I would expect it to be usable on a system that can otherwise run their DeepSeek R1 quants effectively. You could keep an eye on https://old.reddit.com/r/LocalLlama for user reports. wg0|3 months ago Are such machines available in the A class clouds such as Azure/AWS/Google? trvz|3 months ago To start with, an Epyc server or Mac Studio with 512GB RAM. criddell|3 months ago I looked up the price of the Mac Studio: $9500. That's actually a lot less than I was expecting...I'm guessing an Epyc machine is even less. graeme|3 months ago How does the mac studio load the trillion parameter model? load replies (2)
CamperBob2|3 months ago It's an MoE model, so it might not be that bad. The deployment guide at https://huggingface.co/moonshotai/Kimi-K2-Thinking/blob/main... suggests that the full, unquantized model can be run at ~46 tps on a dual-CPU machine with 8× NVIDIA L20 boards.Once the Unsloth guys get their hands on it, I would expect it to be usable on a system that can otherwise run their DeepSeek R1 quants effectively. You could keep an eye on https://old.reddit.com/r/LocalLlama for user reports. wg0|3 months ago Are such machines available in the A class clouds such as Azure/AWS/Google?
trvz|3 months ago To start with, an Epyc server or Mac Studio with 512GB RAM. criddell|3 months ago I looked up the price of the Mac Studio: $9500. That's actually a lot less than I was expecting...I'm guessing an Epyc machine is even less. graeme|3 months ago How does the mac studio load the trillion parameter model? load replies (2)
criddell|3 months ago I looked up the price of the Mac Studio: $9500. That's actually a lot less than I was expecting...I'm guessing an Epyc machine is even less.
CamperBob2|3 months ago
Once the Unsloth guys get their hands on it, I would expect it to be usable on a system that can otherwise run their DeepSeek R1 quants effectively. You could keep an eye on https://old.reddit.com/r/LocalLlama for user reports.
wg0|3 months ago
trvz|3 months ago
criddell|3 months ago
I'm guessing an Epyc machine is even less.
graeme|3 months ago