[Gasp0de]

Exactly, my first thought was "Why in earth would anyone think that S3 was the right service to store millions of tiny ephemeral files?" and now it seems they have invented their own in-memory store instead of just using something like Redis. I also wonder what happens if their DIY thingy crashes, are the videos lost? Why not send to Kinesis or SQS in the first place?

[alex_smart]

From the article, individual video segments were 2-6 MB in size and SQS and Kinesis have a 1MB limit for individual records so they couldn’t have used either service directly. At least not without breaking their segments into even smaller chunks.

[Gasp0de]

You're right, I didn't pay attention there. Still seems that there a many solutions better suited than S3. Probably a classic case of "We need an MVP fast, let's optimize later".