[zhwu]

Cloud services, such as autoscaling EKS or AWS Batch are mostly limited by the GPU availability in a single region. That limits the scalability of jobs that can run distributedly in a large scale.

AI batch inference is one of the examples, and this post found that by going beyond a single region, it is possible to speed up the important embedding generation workload by 9x, because of the available GPUs in the "forgotten" regions.

This can significantly increase the iteration speed for building applications, such as RAG, and AI search. We share our experience for launching a large amount of batch inference jobs across the globe with the OSS project SkyPilot.

TL;DR: it speeds up the embedding generation on Amazon review dataset with 30M items by 9x and reduces the cost by 61%.