Feature flags in a CI pipeline

[gurchik]

I've done something like this with CircleCI and Github labels. The Continuation orb[1] allows you to dynamically generate your CI yml rather than using a static workflow. A Python script is executed which uses information about the PR (author, timestamp, labels applied to the PR, etc) and a CI template yml to add, remove, or skip jobs from the workflow. Some use cases are:

  - Replacing a job with a newer job for testing purposes, as described in the article.
  - Opting out of jobs, like "skip-deploy" or "skip-unit" labels, to speed up CI in some cases when these jobs aren't needed.
  - Making some optional jobs required depending on the files that have been modified in the PR. You can get a list of modified files in a PR using the Github API. We use this for example if a new feature necessitates a new CI job but you want this job to be optional for branches that don't have this feature code yet.

The additional complexity can make this a non-starter for some teams. In our case these use cases are more convenient for us than confusing for the eng team.

1: https://circleci.com/developer/orbs/orb/circleci/continuatio...

[dottedmag]

It looks like an additional tool and a new remote service to talk to. Are there any CIs that don't support editable variables on scopes larger than a build?

The article talks about GH Actions, they have org/repo/env variables.

Even Jenkins has variables usable from the build.

[pondidum]

Sure you can use the environment variables, and if you have simple uses cases this is probably the best way to go.

In this case, we're using LaunchDarkly's targeting to serve different variations to groups of users, and also using the in built monitoring LaunchDarkly provides to see how often each is requested etc.

[verdverm]

We add a "branch config" JSON file for each branch, which is later merged with some CUE. We put metadata and "feature flag" like values in there. This full config value is passed to all subjobs and makes it easy to change and grow our CI.

The main problem I see with real feature flags in CI is that it can make reproducing a build nearly impossible. What happens if someone flips a flag in the middle of a build? (where that flag value is checked in multiple places, so the flip happens in between the checks)

[goodoldneon]

Would logging feature flag values be enough? Then you could look at stdout to see what the value was during each job

[mo_po2]

Flagon's exit codes are as follows:

0 the flag queried is on (true) 1 the flag queried is off (false) 1 an error occurred querying the flag

I think that it would be nice that error and false had different exit codes.

[pondidum]

It actually does have different error codes; I think I just forgot to update the docs:

0 - flag on 1 - flag off 2 - error

I'll update the docs soon!

[tail_exchange]

I've been thinking about a way to add feature flags to my project's CI pipeline. What makes me afraid of adding such thing is reliability: if someone breaks the feature flag service, now the CI pipeline is broken too. I could just give them a default value if the communication fails, but then we will have builds that are skipping checks potentially due to transient errors. I would be very curious to know the guidance for using this tool in my case.

[verdverm]

An easier solution is to add a json file per branch that holds these "flag" or CI config values. We have a dir `bfg/...` to hold them. Nice thing is they are version controlled and the dev can change values between commits like they do with any other code. This pattern has helped solve a lot of issues and we no longer have complex conditions that key off of branch name patterns.

[rtpg]

I feel like you can just add a table to your DB and be done with it for most systems. I feel like people with large enough systems where “stick it into the DB” is not reasonable also should probably have a bespoke feature flag system.

[raptorraver]

I’ve been looking for a way to do integration tests for our pipelines. So far haven’t found anything I could use.

[jillesvangurp]

I use matrix tests with github actions to test my kt-search client with different versions of elastisearch and opensearch. Pretty easy to set up: https://github.com/jillesvangurp/kt-search/blob/master/.gith...

Basically it fires up elasticsearch using docker-compose and then the integration tests run against that. You could use a similar strategy to test different feature flag combinations.

For some of our private projects, we use kts to generate the github action yaml files using this: https://github.com/krzema12/github-workflows-kt

Well worth checking out if you have more complex workflows. Yaml is just horrible in terms of copy paste reuse. Also nice to get some compile time safety and auto complete with our action files.

[marcosnils]

hey! Dagger (https://dagger.io/) contributor here! Dagger can help with that. If you have any questions, happy to connect!