[aent]

For anyone wondering, it includes 4 models: 7/13/30/65 billion parameters, the smallest one is 14Gb, the largest one is 131GB, all four are 235Gb.

[q1w2]

I wonder how many people are scrambling to set this up on their startup infra.

6x24GB NVRAM on 6 GPUs linked with NVSwitch is a little pricey, but totally doable.

[arthurcolle]

I got it running using Colab Pro+ (immediately got a V100 40GB VRAM GPU) - the 7B model works with batch size of 8 and a max seq len of 1024

[exo-pla-net]

How pricey would you estimate?

[mlboss]

Is it possible to run the smallest one on a consumer gpu with 24gb ram ?

[MacsHeadroom]

You can do even better!. You can run the second smallest one (better than GPT-3 175B) on 24GB of vram, ie LLaMA-13B. https://github.com/oobabooga/text-generation-webui/issues/14...

[Tepix]

Running it is easy but you'll probably want to finetune it, too

[rihegher]

I would be surprised if you can't. The smallest weight file is 14gb apparently