[ahmedfromtunis]

Where do you run a trillion-param model?

[Gracana]

If you want to do it at home, ik_llama.cpp has some performance optimizations that make it semi-practical to run a model of this size on a server with lots of memory bandwidth and a GPU or two for offload. You can get 6-10 tok/s with modest hardware workstation hardware. Thinking chews up a lot of tokens though, so it will be a slog.

[simonw]

What kind of server have you used to run a trillion parameter model? I'd love to dig more into this.

[isoprophlex]

You let the people at openrouter worry about that for you

[MurizS]

Which in turn lets the people at Moonshot AI worry about that for them, the only provider for this model as of now.

[skeptrune]

Good people over there