guslees's comments

I feel like the KEP was never examined "adversarially". And that isn't the right word, because nobody wants to create conflict where there is none. Perhaps "skeptically" is a better word, but even that implies some level of external objection.

To pick on some almost unfairly cherry-picked examples:

- the KEP called out "don't want to run scripts from github" as a design goal _for_ kustomize, when kustomize itself clearly ran arbitrary scripts in well advertised situations. Nobody during the (lengthy and open) KEP review stage raised this. (Since fixed by removing this kustomize feature)

- the various "to plugin or not to plugin" discussions are highly subjective (and always will be) and dance around the elephant in the room: "We think the plugin mechanism is great" obviously doesn't marry with "nobody wants to use it".

- the scope and approach of kustomize vs other "less stylistically opinionated" alternatives was never (openly?) considered afaics. For example, perhaps we could _just_ have a stack of json/s-m-p patches that get applied, or a generic mechanism for applying arbitrary "post-process" transformations. The kustomize KEP didn't really consider "not using kustomize" as an alternative, which is understandable from a kustomize pov (we must presume the authors like their proposal!), but not from a sig-cli and overall process pov.

Afaics, the kustomize folks did everything right through this, as did the rest of sig-sli, and the above is only apparent in hindsight. The reality is that the sig contributor group is small, and necessarily can only consider proposals that actually exist. The KEP process is entirely human-based, with all the biases and fallibility that comes from that. I don't have any useful suggestions for how to institutionalise scepticism that won't also bring far worse chilling effects. I note the KEP template already includes "drawbacks" and "alternatives" sections, but it relies on cultural expectations to ensure these are used effectively.

TL;DR: I actually think this "kerfuffle" is an example of the larger process working _well_. A decision graduated to a larger audience, was questioned, and rolled back to be reexamined in greater detail. We must _expect_ to have to iterate on some decisions sometimes, and that's ok, normal and healthy.

Making an "easier" app-level experience very quickly becomes (necessarily) opinionated, because you need to anticipate which k8s parameters need to be exposed and which can be derived/assumed. Narrowing the configuration space in this way is entirely the point of "easier".

The way I've been approaching this is that you need a local "power user" who produces a simple abstraction that captures local patterns and policies, and the rest of the company then reuses that abstraction (or abstractions). Helm sort of lets you build this, but in practice it requires re-packaging helm charts with local customisations - which rapidly becomes a lot of overhead. The alternative is to expose every possible k8s option through the original helm parameters, which in turn means the helm chart becomes bewilderingly complex, and we're back to our original problem statement.

Instead, I've been advocating an "overlay" approach with jsonnet and the design of kube.libsonnet. The idea is that each consumer can import some upstream k8s manifests (described in jsonnet), apply some further jsonnet-based translations, and then publish the result as newer/simpler "templates". Someone else can then consume that, add another layer, republish, rinse, repeat. Importantly, each "republished" layer is still as easy to consume as the original. Eventually you end up with a jsonnet-based template that becomes highly opinionated and specialised to your actual problem domain, and hopefully is terse enough for local devs to use without having to learn all about k8s.

Example strawman:

  local mycompany = import "mycompany.libsonnet";
  mycompany.PhpApp {
    repo: "webgroup/guestbook",
    url: "https://mycompany.com/guestbook",
    requires_mysql: true,
  }

This might (hypothetically) turn into a:

- k8s Deployment that derived the docker image from the repo name (using knowledge of local build/publish conventions) and the command from the fact it was a php app

- k8s Service to point to the Deployment

- k8s Ingress from the provided URL (and local policy), pointing to the Service

- Bring in a mysql instance via any one of several approaches (eg: new standalone instance, or configure a new user/table on a centrally-managed DB server)

None of the above would be hard to do right now using kubecfg (or other approaches), but requires at least one person who understands both local policies and kubernetes - and for them to express that knowledge in "mycompany.libsonnet".

Importantly, whatever "mycompany.PhpApp" did would be quite different to "mycompany.PeriodicSparkJob" or "someothercompany.PhpApp" - so this isn't really something the _community_ can provide, without it rapidly becoming generic again and missing the whole point of the exercise. Coming back to your question, I think this is why you won't (and will never in the general case) find already-made tools that just happen to match your particular local needs.

The fork is my fault. I agree it's confusing.

I wrote (and still maintain) kubecfg. Heptio joined the project and started adding lots of great stuff. Eventually it was clear they wanted to take it in a direction that was different to the original borgcfg-like vision. I suggested we split that new functionality into a different tool so we could keep exploring both directions without trying to mash both into the same cli flags. Hence ksonnet/kubecfg and ksonnet/ksonnet.

They both use jsonnet internally, generate the same k8s resources in the end, and have that common code heritage, so have many similarities.

ksonnet/ksonnet has a bunch of extra ("rails-like") tooling to hold your hand while you generate jsonnet, and assumes it uses the k.libsonnet library (https://github.com/ksonnet/ksonnet-lib).

ksonnet/kubecfg is much dumber and really just conceptually `jsonnet | kubectl apply`. In particular kubecfg avoids having any opinion about which jsonnet libraries you use. We use it extensively with https://github.com/bitnami-labs/kube-libsonnet, but you can also use it with https://github.com/ksonnet/ksonnet-lib or anything else that is valid jsonnet.

I wrote kubecfg as an unashamed imitation of borgcfg, because for all that we like to complain - borgcfg + GCL + git works extraordinarily well. Kubecfg uses jsonnet, which is similarly openly inspired by GCL (but with some important changes).

https://jsonnet.org/ https://github.com/ksonnet/kubecfg Example real-world usage: https://github.com/anguslees/k8s-home

That's not quite true. More parts == more things that can fail, but whether those failures result in the entire system failing depends on how you've combined the parts.

If you make each of the pieces required parts of the whole, then yes - adding more of them will increase the chance that the whole system fails. But in kubernetes, the additional pieces (nodes) are all redundant parts of the whole, and can fail without affecting the availability of the whole system. The more nodes you add, the more redundancy you're adding, and the less chance that the system as a whole will be affected.

Mathematically:

If a component fails F% of the time, then adding N of them "in series" (all of them need to work) means your whole system fails with a (1-(1-F)^N)% chance. Iow, as N goes up, the system approaches (1-0)% => 100% chance of failure.

Otoh, if you combine the parts "in parallel", and you only need any one[1] of the components to work in order for the whole system to work, then the system has a F^N% chance of failure. As N goes up, this system approaches 0% chance of failure.

[1] Kubernetes (etcd) isn't quite this redundant, since etcd needs a majority quorum to be functional not just any single node. But the principle is similar and still gets more reliable as you add nodes.

^ This.

If it's helpful at all, here's a concrete example of a k8s nginx setup that exports-to/is-monitored-by prometheus: https://github.com/bitnami/kube-manifests/blob/master/common... (Start at https://engineering.bitnami.com/articles/an-example-of-real-... if you would prefer to approach that repo top-down)

If you (and others on this sub-thread) like managing kubernetes through version control, I humbly suggest trying out the ksonnet/kubecfg toolchain (https://github.com/ksonnet/kubecfg and the closely related ksonnet/ksonnet-lib repo - Disclaimer: I'm one of the contributors).

Helm is nice, but (imo, and as some of the comments here are also pointing out) oriented more towards consuming 3rd-party pre-built charts from some sort of centralised marketplace/repo. (This is an important use case, and yes I know it is certainly possible to use helm to manage your own internal software too.)

The idea is that we can support both simultaneously: - people who want/need full expressiveness can do that, and - people who don't need that can happily use higher-level templates that someone else has put together.

Previous efforts at "simplifying" k8s make you choose between one or the other use-case. ksonnet is trying to provide something that can be more continuous along this spectrum. .. At least that's the hope.

It's "alpha", as the version signifies - but there's a linux/x86-64 binary of 'kubecfg' you can play with on https://github.com/ksonnet/kubecfg/releases