chakspak's comments

There is a real use case for a tool like this.

I once had to manage a large, locked-down installation of extremely custom Linux distros running on extremely custom legacy hardware.

Python was not available, and definitely not Ruby, so that eliminates a bunch of CM tools. Not sure if it'd even fit on the ROMs of some of them. The hardware was a mix of ancient MIPS and PowerPC and ARM and x86 in the same stack, so compiling anything for the targets would have been painful. Nevermind kernel/libc versions so old that lots of commonly available tools wouldn't even compile on them. Golang has a minimum kernel version, you know. :P

Guess what all these systems could run though. Good old sh and sshd. I could send/receive files with cat and tar. I could collect system info from /proc. I could do pretty much whatever I needed. I ended up writing a tool very similar to this one in spirit, though not nearly as polished, and I didn't get to take it with me when I left the company.

Shell syntax can be obscure and confusing, but when you have nothing else, a shell gets the job done. I'm happy to see someone had a similar problem and solved it, and then made it a thing. :)

I generally agree with you, though I want to comment on this line:

> Code that you don't touch doesn't suddenly change its behaviour one day.

This can and does happen all the time, when the platforms and abstractions your code builds on change underneath you. This is why a compelling environment / dependency management story is so important. "Code rot" is real. =P

I've got a couple of these:

- Stretch of freeway by my old house that looks and feels like any freeway you've ever been on, except they limit the speed to 45mph. Try to go 45 at your own peril as car and semi alike go 65 past you. Then the cop camps there. I avoid that freeway now.

- Shopping center exit onto a small side street had a "no left turn" sign, but trying to obey this meant a massive detour around if you needed to go left. The side street was empty most of the time (and low speed!), so people tended to just go for it. Cop camped out there. They removed the sign a year later.

Yeah, there are quite a few applications that actively discourage the use of anything other than their CLI, which I find profoundly bizarre. Istio had a big warning in their docs that their Helm charts were deprecated, even though istioctl uses them under the hood. Funny that Kubeflow has its own Istio deployment, and they make you use kfctl to deploy it, haha.

It's not strictly required, but might be for you, since A/B deployment actually is a thing Istio does. We only use Kubeflow Pipelines currently; looking into Katib.

My feeling about Kubeflow is that it's a package of a lot of things that exist, with nice things on top, and an easy way to deploy those things. Only, it ends up not being that easy, and not easy at all to configure, and various features of the underlying tools are hard to get to or completely unavailable.

If you deployed your own Istio, you'd understand it end-to-end and could solve problems with it when it goes south, and you'd even understand what exactly you're using it for and why. The biggest problem I have with kfctl is that it basically asks for system:masters privilege to install everything and then you get to figure out what it did on your own. I don't want Istio, Cert-Manager, Knative, Argo, etc. deployments that I don't understand and can't easily configure because they're buried 5 levels deep in Kustomize overlays. These are all things I can install from elsewhere and with more documentation.

Kubeflow Pipelines is still a little mysterious, but the footprint is much smaller and not as invasive.

Never used Polyaxon. Gonna be Googling that!

My company has deployed Kubeflow for production model training. Early on, we used their big deployment, but we got frustrated trying to manage it with kfctl, so we started using kustomize directly and deploying only what we need, like KFP. So no istio for us! YMMV with your use case.