[delsarto]

One of the most interesting parts of OpenDev is that the infrastructure that builds and deploys all the OpenDev services is open.

Anyone can propose changes to practically all of the infrastructure via the https://opendev.org/opendev/system-config project.

Anyone can look through all of the changes; https://review.opendev.org/q/project:opendev/system-config+s...

Every change gets run through an extremely thorough CI system that tests the change and reports results, e.g. https://zuul.opendev.org/t/openstack/buildset/50ce144851224b...

Most of these CI jobs do things like apply your proposed change, deploy the service, then connect up a headless client and take screenshots of the results so you can confirm the correct behaviour of your change; e.g. http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b62... is from a job that was modifying the gita deployment.

Humans approve changes for merge with Zuul, but Zuul commits the code. When Zuul merges the change, another set of jobs will push things to the production hosts automatically. There is a talk on the overall process at https://www.youtube.com/watch?v=apLHQ4DkIHU

There are obviously bastion hosts and private components to the production deployment, but as much as practical is completely open. There have been contributors with no special access that have developed the deployment of infrastructure services and admins helped only at the end committing some secrets and providing production hardware resources. More commonly, people who have CI issues can jump in and fix their own problems, especially relating to things specific to them (e.g. mirror setup on CI nodes, software versions used, etc.)

One cool thing is that the Zuul jobs publish their production deployment logs in public, but encrypted (these might contain secrets, so are not made open by default). If you work on a particular system, you can request to commit your public key to the service you like and access all the deployment logs. For example, say you are interested in maintaining codesearch.opendev.org. The "infra-prod-codesearch" job deploys changes relating to this service. You can look at all the jobs at https://zuul.opendev.org/t/openstack/builds?job_name=infra-p..., and every job has an artifact that lets you download the logs; e.g. https://zuul.opendev.org/t/openstack/build/c76ec695d19a4e9e9.... Details are at https://docs.opendev.org/opendev/system-config/latest/open-i...

Wikimedia is the only other platform I've seen with a comparable commitment to open infrastructure (unsurprisingly there has been collaboration between the two over the years)