[benchess]

If I understand correctly, Groq chips have 220MB SRAM and the next best level is DDR4? How many chips are needed to run Llama2-70B at those speeds?

[tome]

Cool that you know the tech specs of the GroqChip! Yes, that's right, 220 MB of SRAM per chip. I think the demo where we first broke 200 tokens / sec was running on 1 GroqRack, so 64 chips. The live public demo that's currently running at 275 tokens / sec I think might be running on two GroqRacks, so 128 chips. I'm not certain of either of these figures so please don't quote me! But those are the right ball-park.

[trsohmers]

This article from less than a month ago says that it is on 576 chips https://www.nextplatform.com/2023/11/27/groq-says-it-can-dep...