[alienthrowaway]

What are the pros of using CUDA-enabled devices for inference?

[bitexploder]

My limited understanding is that CUDA wins on smaller batches and jobs but TPU wins on larger jobs. It is just easier to use and better at typical small workloads. At some point for bigger ML loads and inference TPU starts making sense.