[angry-tempest]

Probably not for a few years, you need a (maybe few) A100(s) to be able to backprop a model that big with float32.

[verdverm]

iirc, they tweeted about using around 3800 in parallel