[jacobwg]

A list of fun things we've done for CI runners to improve CI:

- Configured a block-level in-memory disk accelerator / cache (fs operations at the speed of RAM!)

- Benchmarked EC2 instance types (m7a is the best x86 today, m8g is the best arm64)

- "Warming" the root EBS volume by accessing a set of priority blocks before the job starts to give the job full disk performance [0]

- Launching each runner instance in a public subnet with a public IP - the runner gets full throughput from AWS to the public internet, and IP-based rate limits rarely apply (Docker Hub)

- Configuring Docker with containerd/estargz support

- Just generally turning kernel options and unit files off that aren't needed

[0] https://docs.aws.amazon.com/ebs/latest/userguide/ebs-initial...

[3np]

> Launching each runner instance in a public subnet with a public IP - the runner gets full throughput from AWS to the public internet, and IP-based rate limits rarely apply (Docker Hub)

Are you not using a caching registry mirror, instead pulling the same image from Hub for each runner...? If so that seems like it would be an easy win to add, unless you specifically do mostly hot/unique pulls.

The more efficient answer to those rate limits is almost always to pull less times for the same work rather than scaling in a way that circumvents them.

[jacobwg]

Today we (Depot) are not, though some of our customers configure this. For the moment at least, the ephemeral public IP architecture makes it generally unnecessary from a rate-limit perspective.

From a performance / efficiency perspective, we generally recommend using ECR Public images[0], since AWS hosts mirrors of all the "Docker official" images, and throughput to ECR Public is great from inside AWS.

[0] https://gallery.ecr.aws/

[philsnow]

> Configured a block-level in-memory disk accelerator / cache (fs operations at the speed of RAM!)

I'm slightly old; is that the same thing as a ramdisk? https://en.wikipedia.org/wiki/RAM_drive

[jacobwg]

Exactly, a ramdisk-backed writeback cache for the root volume for Linux. For macOS we wrote a custom nbd filter to achieve the same thing.

[seanlaff]

The ramdisk that overflows to a real disk is a cool concept that I didn't previously consider. Is this just clever use of bcache? If you have any docs about how this was set up I'd love to read them.

[yencabulator]

> - Configured a block-level in-memory disk accelerator / cache (fs operations at the speed of RAM!)

Everyone Linux kernel does that already. I currently have 20 GB of disk cached in RAM on this laptop.

[jiocrag]

have you tried Buildkite? https://buildkite.com