[pqn]

Disclaimer: I work at Exafunction

I empathize a bit with the cloud providers as they have to upgrade their data centers every few years with new GPU instances and it's hard for them to anticipate demand.

But if you can easily use every trick in the book (CPU version of the model, autoscaling to zero, model compilation, keeping inference in your own VPC, using spot instances, etc.) then it's usually still worth it.

[lowdose]

Not to mention AWS has had a GPU cloud offering monopoly because Google Cloud and Microsoft Azure were publicly available until 2019.

[fomine3]

GCP still provides NVIDIA K80. I wonder is it still worth to hold.

[varunkmohan]

I think you'd probably always want to go with T4's since they are the same price unless there's just no availability for them.