[alanaan]

great post. could you apply this same framework to optimize training as well?

[varunshenoy]

Slightly different set of trade-offs, but similar mental model. You always use large batch sizes (compute bound) and the bottleneck usually ends up communication between GPUs/nodes.