Kargo, a multi-stage application lifecycle orchestrator

[rjzzleep]

I've used Flux, Flux2 and Argo for GitOps, and they all had their quirks. In general it seems that most CloudNative projects were kinda set in their own vision with very opinionated devs and a lot of corner cases that are not accounted for and won't be accounted for.

I have a little bit of hope that this one is going to be different with the lessons they got from Argo, but I'm not holding my breath.

Ps. when I first saw Argo, I thought this is it. The solution to all my problems.

[msm_]

I miss Flux (not flux2). It did everything I needed, and was simple to understand. It had its problems, but I feel like they could be resolved without a complete rewrite/revamp the Flux2 is. I understand the need for Flux2, but for my simple use-cases it's too complicated.

And Argo and now Kargo are even more complex.

[wg0]

I don't understand one thing about GitOps.

Imagine 10 apps deployed. All are actively deployed let's say few times a day.

You want to go back 10 days for App A. But in doing so you would have reverted whole state and all apps as they were 10 days ago.

Only way is to cherry pick particular commit and revert it.

No? I mean how git can be useful in rolling back singel components back and forth?

[candiddevmike]

Akuity is getting their lunch eaten by Harness and need some kind of IP to sell besides Argo-as-a-Service. I don't really see a path forward for them in the very crowded CI/CD space, so I would wait to see how they will monetize this and who the first movers are before attempting to use it.

From a business standpoint, Akuity had their Series A in 2022 and raised $25M. They have yet to show up on anyone's radar, IMO. Maybe Kargo is their PMF but I wouldn't move my CI/CD over yet.

[debarshri]

I think that's the nature of any CI/CD tool out there. All of them have some opinion to bound the product otherwise it is very hard to tame usecases.

[greatfood230]

Agreed. But I do feel Argo gives a good foundation; something on top is highly anticipated.

[morey-tech]

Some of that is a challenge of product design.

Devs tend to be opinionated on these projects because without that you end up with feature sprawl to the point that projects become unmaintainable.

On the other side, new projects need to focus on their specific segment and solve that problem well without worrying about corner cases.

Either way, any solution for a corner case needs to be implemented in a way that solves it broadly for any user.

[abuckenheimer]

I've only skimmed the quickstart docs https://kargo.akuity.io/quickstart/ but this snippet immediately clicks for me as something that I'd want to try

    cat <<EOF | kubectl apply -f -
    apiVersion: kargo.akuity.io/v1alpha1
    kind: Promotion
    metadata:
      name: prod-to-${FREIGHT_ID}
      namespace: kargo-demo
    spec:
      stage: prod
      freight: ${FREIGHT_ID}
    EOF

Have enjoyed using argo in general in the past, its got a great model for k8s native workflows / events but never got to using it for CD.

[distortionfield]

I used Argo for some e2e test suites I wrote and it was a god send. Being able to trigger off of GitHub labels was so insanely powerful for building developer tooling.

[sleepybrett]

Yeah we use argo workflows to power some e2e testing and rollouts for canary blue/green, and their notification server, but don't actually use their cd product.

[ThinkBeat]

I can almost follow this but I am at the edge of where I see how it could sound like buzzword gibberish.

I dont blame the author(s) but things are getting more specialized and more terms are created. (which usually maps to an existing term which also maps to an existed term which also maps an existing term and so on.

I have though about trying to create a "terminator"(hah) where you can paste in something from a new product and it will map the terms to existing terms as a sort of translator or term machine.

"" We are thrilled to announce Kargo, a multi-stage application lifecycle orchestrator for continuously delivering and promoting changes through environments. Kargo, brought to you by the creators of the Argo Project, is a reimagining of CD pipelines for the cloud-native era, with first-class GitOps support, progressive delivery features, and is 100% open source. ""

[0xbadcafebee]

multi-stage: in this case they mean dev->test->prod

application lifecycle: CI & CD

orchestrator: a thing that runs jobs with dependencies

continuously delivering: CD

promoting changes: if the tests work in the staging environment, allow someone to click a button that says "deploy to prod"

environments: dev, test, staging, production

CD pipelines: a bunch of continuous delivery jobs

cloud-native era: microservices/SaaS/PaaS/IaaS/IaC/II/containerization/webhooks/OIDC

first-class GitOps support: if you push a commit, a job is run

progressive delivery: deploy to 10% of users, if lots of errors, roll back the deploy

100% open source: our code is [currently] available but we will charge you out the ass to manage it for you and Enterprise features will be locked up once we write them

[what-no-tests]

Weird thing is, when I read that quote it makes perfect sense.

But I understand how it could look like the undecoded bytefall in the Matrix for those outside the know.

[robertlagrant]

Every time there's a new tool, it's worth considering if it locks in previous choices. E.g. ArgoCD means I can't move off Kubernetes. Will Kargo mean I can't move off ArgoCD or Kubernetes?

[AaronM]

Using Kubernetes locks you into kubernetes. ArgoCD doesn't lock you into Kubernetes because its just a wrapper around managing kubernetes manifests

Its like arguing that cloudformation locks you into aws

[spacelad101]

Environment promotion is an area that has been severely lacking with GitOps tools. I've hacked together CI pipelines to do environment promotion with Argo CD before and it was far from my preferred approach, but I could never find a good GitOps way of handling it. Might check out the webinar tomorrow to see if this is something worth trying out.

[nickjj]

What about having a git repo that has a Kustomize base which has overlays for different environments?

This way each environment is in its own directory which can have its own patches such as using a private load balancer instead of public for a staging environment or setting whatever environment variables that need to be different.

Then at the Argo CD level, Argo CD running in prod will look for an Application in the prod/ directory and Argo CD running in staging will look for an Application in the staging/ directory.

All in all you end up deploying the same artifact across environments and all of this is managed and exposed by Argo CD. For example you might want AutoSync enabled for staging so developers can auto-deploy their code but prod requires a tech lead or manager to push the sync button to deploy it.

The above works well in practice and isn't too complicated to pull off.

[therealkrancour]

I am a lead on Kargo. That's a great approach you've described and Kargo helps you make that approach work even better. It's not trying to re-solve problems that are already solved well by config management tools or by Argo CD. It provides a higher layer that integrates with those to orchestrate the progression of changes from one environment to another...

So in your example, suppose you changed something in your Kustomize `base/` directory. With _just_ Kustomize and Argo CD, those changes roll out to all environments all at once. With Kargo, you can progress the change and (re-)validate it environment-by-environment.

[spacelad101]

This approach definitely works, but its not the most convenient imo. At least in my experience, this always resulted in separate commits to dev/stage/prod as you progress through each part of the promotion process. It works, sure, but depending on your exact situation it can get cumbersome pretty quickly (for example, start adding in multiple prod regions, or maybe you want to canary one prod region first).

With Kargo, it looks like it lets you define your preferred promotion process, and then lets you release each promotion with a single click. I think the part that is the most interesting to me though is that it writes the changes it is making directly to the git repo.

[narrator]

I mistakenly read that as "Kargo, a multi-stage .... lifestyle orchestrator" and thought, damn, that sounds interesting!

[esafak]

I'm not a CI/CD specialist, so help me out. Is this something that is more suitable for large, complex setups or is it something smaller shops should be considering too? How does it compare with competitors like Spinnaker?

[bingemaker]

Is this tool similar to https://dagger.io/?

[KenCochrane]

I think they are similar but a little different. From Dagger's website "Dagger is a programmable CI/CD engine that runs your pipelines in containers" Kargo is a tool that works with your existing CI and CD (Argo CD) to help promote your code from one stage (Dev, UAT, QA, Prod, etc) to the next.

Kargo has native support for Argo CD today, but one of it's goals is to be agnostic and could probably work with Dagger or other tools in the future.

[therealkrancour]

Not much overlap. Dagger wasn't on my radar until now, but it reminds me of something else I used to work on and it frankly looks awesome!

But it looks to allow you to defines pipelines using your preferred imperative programming language -- which is awesome. I would totally use that for CI.

Kargo is really about divorcing CI and CD and creating something that's actually CD-focused and compatible with established GitOps principles.

[drewda]

Dagger looks more comparable to Argo Workflows: https://argoproj.github.io/argo-workflows/ That's the first of the Argo projects, which can run multi-step workflows within containers on Kubernetes.

For what it's worth, my colleagues and I have had great luck with Argo Workflows and wrote up a blog post about some of its advantages a few years ago: https://www.interline.io/blog/scaling-openstreetmap-data-wor...

[jmorenoamor]

Wow, looks interesting.

You make per project makefile-like python pipelines that are executed by an installed runtime/docker container?

Then I will give it a try, I've been looking for a lightweight local ad-hoc jenkins substitute.

[pm90]

I swear to god I thought this was dagster and was confused for a bit.

[tonnydourado]

> Unlike CI, Kargo deployment pipelines are not generic “jobs” (...). Instead, Kargo Stages are used to model your environments (...)

I didn't see an explanation of how Stages are different than jobs. Every single usage of Stage could have been replaced with job and the meaning would stay the same.

The data and DevOps marketing people really need to drop the buzzwordism.

[reilly3000]

I am a fan of the term- after long naming wars internally we settled on promotion stage to signify the same concept. It is one or more targeted clusters that function as one of several environments. It works both in the sense of stage as venue and stage as phase. Teams promote their config from something like dev to QA to Preprod to Prod. Often there is a need for business logic to happen before advancing to another stage, or multiple environments in different regions that need to be healthy within a stage before promotion can happen.

Job doesn’t really encapsulate this at all. An alternative may be Metaenvironments.

[0xbadcafebee]

Stages are just quality gates. When the quality level is high enough, the gate is released and the next step in the pipeline is performed.

[0xbadcafebee]

It sounds like the solution all of us knew Argo needed to solve but didn't. I'm... hopeful? But besides the fact that it's not finished yet, it also sounds like it's going to be tightly integrated into their "Akuity Platform". Tightly integrated solutions are an anti-pattern; we'll end up with cobbled-together scripts again when something we need to do isn't allowed by the "opinionated" platform.

[reilly3000]

I had the good fortune to work with Akuity last year. I haven’t looked at Kargo yet but I’m sure it’s going to be solid based on my experience with their engineering and leadership. The problem it is trying to solve for is very real and historically expensive to solve for. Most companies that operate in multiple clusters and want to do GitOps will face this issue, especially when there are gates and approvals that happen between environments.

[hermanb]

If Kargo requires R/W access to GitHub, and auto updates charts/images, isn’t that asking for your production environment to be infected by a change prepared and cultured in your dev environment and then auto updating / hiding itself into prod freight?

We disallow writing back to GitHub to avoid this issue, and manage stages through branches, combined with directories for overlays. Things can get out of sync, but comparing branches is easily automated.

[cced]

Couldn’t this be mitigated against by using something like codeowners and only allowing access to manage versioning files?

[theptip]

This looks interesting, codifying and automating promotion looks cool.

In my experience it gets pretty hairy to build automated conditional promotion in a GitOps pipeline, especially if you don’t go all the way to continuous delivery and might have different versions at different stages between environments.

[KenCochrane]

For those who are curious here is the Github repo https://github.com/akuity/kargo

[Havoc]

Sounds promising. Vaguely terraform like in its application to outside subscription services.

Can’t say I’m a fan of “we call it freight” though. Artefact is a perfectly fine word.

[therealkrancour]

Freight often contains multiple artifacts. Container would have been a nice word, but it's taken. ;)

[morey-tech]

I tend to agree, the abstraction from common language (e.g. artifact) is fun because it stays on theme with the "kargo" name but can make it difficult for new users to understand what is happening. I've always loved that in Argo CD, it's just an Application. No special name for something users are often going to refer to.

[manojlds]

Kargo (cargo) and Freight makes sense no?

[freedomben]

As of writing this, there are 5 total comments, 4 of which are by very inactive or brand new accounts with no activity, adding unhelpful/low-effort posts like:

> Something like this is definitely needed in the GitOps space...always felt like something was missing between promoting things and rolling them out

> Interesting. Will share with my team and try it out.

> There is a webinar tomorrow with Kelsey Hightower! Here is the link if you want to join https:...

> Looks promising, I'm definitely going to take it for a spin!

Something tells me this isn't organically gaining traction...

[pvg]

If you suspect abuse, email the mods since this sort of meta tends to go to thread hell.

[k8svet]

Another project announcement, another situation where I would literally pay a coffee's worth to cut to the chase -- why is there a need for another GitOps tool when several already exist?

Tighter integration with other Argo products? Why is this not simple a new component of Argo CI/CD? How much is "thought-leadership" a part of it, it is Kubernetes-adjacent, after all.

So far even the answers in this thread leaving me asking these questions even more strongly? If Argo CD is a "continuous deployment" tool, why isn't the priority making staged-rollouts a first class feature in Argo CD itself?

This is coming from a place of curiosity, not simply being a miser, I promise.

[therealkrancour]

Depending who you ask, CD stands for one of two things: Continuous _Deployment_ or Continuous _Delivery_ and they're not actually the same thing.

Want to go camping? You go on Amazon and order a tent. An Amazon truck _delivers_ it to you. Not the same as deployment. Deployment is when you pitch the tent.

Argo CD and Flux are great at Deployment. "Make this run in my cluster."

Neither of them addresses the other need -- delivery -- getting artifacts from point A to point B with adequate quality gates, etc. along the way.

[wg0]

There's fluxcd and Argo. What other Gitops tools one should keep an eye on?

[InfamousSoup744]

Something like this is definitely needed in the GitOps space...always felt like something was missing between promoting things and rolling them out