[markonen]

Do you mean to imply that cloud services at higher levels of abstraction are cheaper per unit of compute than simple VMs? I believe you’ll find that the opposite is true.

At the scale discussed here, there are no free lunches.

[eloisant]

It depends on the scale, but running containers over a k8s clusters means your load will be distributed among the nodes according to capacity.

Managing VMs with dedicated resources directly means you have to distribute the load manually, leading to unused and wasted resources.

[sgarland]

You absolutely do not have to distribute VMs manually. This [0] is a tiny Python script run as a cron that migrates VMs in a Proxmox (also free) cluster according to CPU utilization. You could extend it for other parameters.

While I don’t personally have experience with more enterprise-y solutions like VMWare, I have to imagine they have more complete solutions already baked-in.

[0]: https://gitlab.com/tokalanz/proxmox-drs