Before they'd be affected by Route 53 outages, Cloudfront outages, and S3 outages. Now they can add Lambda outages to that list too.
It's also unclear how this actually solves the problem. Now if S3 in _either_ region is unavailable they'll start to fail 50% of uncached requests. I'm guessing they're using Route 53 health checks with some cloudwatch alarm to cut over to one region when they think the other is unhealthy. Presumably this is covered in the unavailable part 2.
I'm mildly skeptical that this is worth the increased risks plus the increased cost from running Lambda@Edge on cache misses.
I think it's still a reduction in risk overall. In the old model, they were vulnerable to S3 failing in one region, a thing that's happened many times. Now they've mitigated the S3-failure-in-one-region issue, at least mostly (though as you point out, how they do so is unknown), and in exchange they've picked up a dependency on Lambda@Edge. But Lambda@Edge, like CloudFront, is a global service distributed across many regions, and to my knowledge AWS has never had a global Lambda@Edge outage.
It's not impossible, of course. Some kind of control plane error could probably knock the whole global service offline. But I'd rather bet on a multi-region service than have all my eggs in one regional basket.
If the "Cross-region replication" line in the picture is talking about the native S3 cross-region replication (as I assume it is), beware the replication latency in this setup. AWS recently released "replication with an SLA" for S3 [0], but at "99.99% of the objects will be replicated within 15 minutes", it's not a good enough SLA to rely on in setups like this.
Presumably Part 2 of this post will address this limitation, or maybe their product isn't affected by it. (I've never looked into Contentful, though maybe I will now -- blog post purpose achieved?)
I'm also not sure if "active-active" is the best name for this setup, since objects can't be written to the 2nd bucket (replication only goes one direction). Generally I associate "active-active" with "writes can happen anywhere", though maybe I'm wrong?
That's a relatively new feature (from November 2018... wow, has it been a year already?). My guess is that they implemented this stuff before that existed or maybe near to its release.
I have not used origin failover either, though I'm pretty sure you're right that this is its exact use-case.
Although it may make sense to this company in _majority_ of companies this would be over-engineering. S3 availability is some of the best in the business. If S3 is down, a good chunk of the internet is down with it.
A single region can go down (and has in the past), no matter how reliable S3 as a whole is.
If your business wants to avoid downtime, this is a simple solution to further reduce risks that cause downtime.
Just because other sites might go down, doesn't mean you have to accept it for your own.
Those are also our biggest pain points together with slow deployments (cloudfront distributions can take 10-20 minutes to update, which is required to rollout a new version) and no support for lambda aliases. We forwarded that to our contacts at AWS, please also do that :-)
If you are already using Cloudflare that is correct, if you don’t that is an alternative.
Adding another vendor into your stack can often come with none engineering complexity (e.g. data protection forms, contractual requirements from customers another vendor you need to work with, etc.) so this is an alternative to stay in AWS when you already use it.
The architecture described here is pretty simple. The article states the fix was 20 lines of code. If this is the hardest problem you have to solve at work, I envy you.
sciurus|6 years ago
It's also unclear how this actually solves the problem. Now if S3 in _either_ region is unavailable they'll start to fail 50% of uncached requests. I'm guessing they're using Route 53 health checks with some cloudwatch alarm to cut over to one region when they think the other is unhealthy. Presumably this is covered in the unavailable part 2.
I'm mildly skeptical that this is worth the increased risks plus the increased cost from running Lambda@Edge on cache misses.
ReidZB|6 years ago
It's not impossible, of course. Some kind of control plane error could probably knock the whole global service offline. But I'd rather bet on a multi-region service than have all my eggs in one regional basket.
paulddraper|6 years ago
Specifically Lambda@Edge though, which changes the math a little.
That said, currently the better solution is CloudFront Origin Groups.
guuguuguu|6 years ago
[deleted]
ReidZB|6 years ago
Presumably Part 2 of this post will address this limitation, or maybe their product isn't affected by it. (I've never looked into Contentful, though maybe I will now -- blog post purpose achieved?)
I'm also not sure if "active-active" is the best name for this setup, since objects can't be written to the 2nd bucket (replication only goes one direction). Generally I associate "active-active" with "writes can happen anywhere", though maybe I'm wrong?
[0] https://aws.amazon.com/blogs/aws/s3-replication-update-repli...
rynop|6 years ago
Full disclosure, I've never used, but pretty sure this feature was created for the scenario you are trying to solve.
ReidZB|6 years ago
I have not used origin failover either, though I'm pretty sure you're right that this is its exact use-case.
Zaheer|6 years ago
NathanWilliams|6 years ago
A single region can go down (and has in the past), no matter how reliable S3 as a whole is. If your business wants to avoid downtime, this is a simple solution to further reduce risks that cause downtime.
Just because other sites might go down, doesn't mean you have to accept it for your own.
advisedwang|6 years ago
reilly3000|6 years ago
knodi|6 years ago
johanneswu|6 years ago
Disclaimer: I work at Contentful.
zackbloom|6 years ago
johanneswu|6 years ago
Adding another vendor into your stack can often come with none engineering complexity (e.g. data protection forms, contractual requirements from customers another vendor you need to work with, etc.) so this is an alternative to stay in AWS when you already use it.
Disclaimer: I work at Contentful.
jugg1es|6 years ago