(no title)
shawnb576 | 3 years ago
Twitter runs their own data centers, which means they own all of the THOUSANDS of machines in them. These machines, and all of their parts, have shelf lives and CONSTANTLY need replacement.
When they are replaced you can't just go to Best Buy with a credit card. At scale VERY SMALL changes matter: oh look they changed something in the disk firmware and haha now your databases corrupt data one out of 1M writes.
New machines need to be tested, burned in, installed. Old ones need to be cycled out.
Same goes for power equipment, networking, all that.
Because you built your own data center, and you were an early scale company it ALSO means a huge percentage of your systems are home grown - asset management, deployment, health checking, metrics, you name it. There are no articles on Stack Overflow. There is no blog post. How that shit works is mostly a function of what people knew about it and, well, at least half of those people are now poof - gone.
This hasn't even gotten to the services themselves, many of which are now running without an owner or any person at the company who has ever looked at them before. The remaining people are now up to their eyeballs in drama, survivor syndrome, fear, and, oh yeah, the work of many of their laid-off peers.
Few people, pfft, give me a break.
No comments yet.