I do dread that exact scenario you're mentioning. I know it is a possibility but I'm hoping that should this day arrive I'll be able to put it to rest and hopefully have learned stuff along the way.
You should absolutely give it a try, and maybe you'll have better luck than me, and it's a fun little experiment if nothing else.
If you get frustrated and don't want to do it anymore, you might also look at Docker Swarm. For homelab stuff, I found it considerably easier to work with. I'm not sure how well it would work with hundreds of services, but for a homelab I found it considerably less frustrating to deal with than K8s. It can still be useful to play with distributed systems and it's a bit more intro-friendly in my opinion.
It's something that I think most engineers should try at least once, if nothing else to learn firsthand the annoyances of distributed computing. It's easy to read about these things in textbooks and blog posts and university courses (and you should!), but at least for me these things didn't really sink in until had to fix broken stuff, or deal with weird heartbeat issues, or see things that should go N times faster actually go slower because I didn't take into account latency etc.
tombert|1 year ago
If you get frustrated and don't want to do it anymore, you might also look at Docker Swarm. For homelab stuff, I found it considerably easier to work with. I'm not sure how well it would work with hundreds of services, but for a homelab I found it considerably less frustrating to deal with than K8s. It can still be useful to play with distributed systems and it's a bit more intro-friendly in my opinion.
It's something that I think most engineers should try at least once, if nothing else to learn firsthand the annoyances of distributed computing. It's easy to read about these things in textbooks and blog posts and university courses (and you should!), but at least for me these things didn't really sink in until had to fix broken stuff, or deal with weird heartbeat issues, or see things that should go N times faster actually go slower because I didn't take into account latency etc.