top | item 45362993

(no title)

riknos314 | 5 months ago

Microservices for days.

I worked on lifecycle ~5 years ago and just the Standard -> Glacier transition path involved no fewer than 7 microservices.

Just determining which of the 400 trillion keys are eligible for a lifecycle action (comparing each object's metadata against the lifecycle policy on the bucket) is a massive big data job.

Always was a fun oncall when some bucket added a lifecycle rule that queued 1PB+ of data for transition or deletion on the same day. At the time our queuing had become good enough to handle these queues gracefully but our alarming hadn't figured out how to differentiate between the backlog for a single customer with a huge job and the whole system failing to process quickly enough. IIRC this was being fixed as I left.

discuss

order

rubiquity|5 months ago

I used to work on the backing service for S3's Index and the daily humps in our graphs from lifecycle running were immense!

jiggawatts|5 months ago

I work on tiny systems now, but something I miss from "big" deployments is how smooth all of the metrics were! Any bump was a signal that really meant something.