[clx75]

Some of the resources are short-lived, including jobs and dev containers. The corresponding CRs are created/updated/deleted directly in the cluster by the project users through a REST API. For these, expansion of the CR into child resources must happen dynamically.

Other CRs are realized through imperative commands executed against a REST API. Prime example is KeycloakRealm and KeycloakClient which translate into API calls to Keycloak, or FSXFileSystem which needs Boto3 to talk to AWS (at least for now, until FSXFileSystem is also implemented in ACK).

For long-lived resources up-front (compile time?) expansion would be possible, we just don't know where to put the expansion code. Currently long-lived resource CRs are stored in Git, deployment is handled with Flux. When projects want an extra resource, we just commit it to Git under their project-resources folder. I guess we could somehow add an extra step here - running a script? - which would do the expansion and store the children in Git before merging desired state into the nonprod/prod branches, I'm just not clear on how to do this in a way that feels nice.

Currently the entire stack can be run on a developer's laptop, thanks to the magic of Tilt. In local dev it comes really handy that you can just change a CRs and the children are synced immediately.

Drawbacks we identified so far:

If we change the expansion logic, child resources of existing parents are (eventually) regenerated using the new logic. This can be a bad thing - for example jobs (which expand into Argo Workflows) should not change while they are running. Currently the only idea we have to mitigate this problem is storing the initial expansion into a ConfigMap and returning the original expansion from this "expansion cache" if it exists at later syncs.

Sometimes the Metacontroller plugin cannot be a pure function and executing the side effects introduces latency into the sync. This didn't cause any problems so far but maybe will as it goes against the Metacontroller design expressed in the docs.

Python is a memory hog, our biggest controllers can take ~200M.

[ec109685]

We've used an artifact store like Aritifactory to store the generated / expanded K8s yaml files, ending up with three pieces: 1) A versioned and packaged config generation system that your dev ops team owns. You'd have test and production versions of this that all applications use in their CI pipeline. 2) A templated input configuration that describes the unique bits per service (this configuration file is owned by each application team) 3) The output of #1 applied to #2, versioned in an artifact store that is generated by the CI pipeline.

And finally, a Kustomize step can be added at the end to support configuration that isn't supported by #1 and #2, without requiring teams to generate all the K8s config pieces by hand.