WingNews logo WingNews
top | item 44205279

(no title)

msdrigg | 8 months ago

There are some hilarious tidbits in here

> Starting in the first week of 2024, the FreeBSD boot process suddenly got about 3x slower. I started bisecting commits, and tracked it down to... a commit which increased the root disk size from 5 GB to 6 GB. Why? Well, I reached out to some of my friends at Amazon, and it turned out that the answer was somewhere between "magic" and "you really don't want to know"; but the important part for me was that increasing the root disk size to 8 GB restored performance to earlier levels.

discuss

order

jeffbarr|8 months ago

The original object size limit for S3 was 5 GB, as noted in my 2006 blog post:

https://aws.amazon.com/blogs/aws/amazon_s3/

I do not know if this has anything to do with the cliff that you saw.

cperciva|8 months ago

Pretty sure that's not related. For one thing I don't think EBS snapshots are stored in S3 as 5 GB segments.

bigiain|8 months ago

This is why I keep reading the comments here.

Deep deep greybeard wisdom from the founding fathers of modern computing.

xandrius|8 months ago

Now I really want to know though.

cperciva|8 months ago

My understanding is that EBS has some heuristics for deciding whether to keep data cached; an AMI which has a cached snapshot as its root disk will boot much faster than an AMI where all the data needs to be pulled from S3.

avidphantasm|8 months ago

Yeah, I am constantly curious about how the sausage that is cloud services like AWS is made. It seems generally slick on the surface, but what’s holding it all together? I imagine it as a tangled ball of tools like Puppet, Chef, etc. and custom glue.

polskibus|8 months ago

I wonder how long did it take to bisect such issue. Build image every time and reboot a vm?

cperciva|8 months ago

I can't remember exactly but it was a few hours. I already knew which week the issue arose (from comparing weekly snapshots) so that gave me a head start.

But yes, I built a lot of AMIs. And launched new EC2 instances for each of them -- it wasn't just a matter of rebooting since the first time an AMI launches there's different behaviour (both from FreeBSD, e.g. growing the root disk, and from EC2, e.g. disk caching).