Harper Reed and Scott VanDenPlas did a talk / Q&A at the Chicago AWS meetup group a few months ago. My hazy recollection of the salient points of the talk:
1) This represents a heterogenous set of services where the engineers in charge of each service were more-or-less free to use whatever technology stack they were most comfortable with.
2) Use puppet for provisioning everything. (There is some subtlety Scott went into about being able to bring up a bunch of servers quickly which I have since forgotten).
3) Everything shuts down the day after the election. Not much time spent future-proofing the code when it has a well known expiration date.
4) Testing testing testing. System had serious problems and 2008 and they didn't want to repeat in 2012.
5) Engineers and sysadmins had a lot of latitude to make changes in order to respond to issues as they developed. It sounded like this is essentially Harper Reed's management style (he seems like a cool guy to work for).
The subtlety you are referring to in #2 is that we moved from puppet for deployments to puppet to configure template nodes. We baked AMIs from those template nodes and deployed those, drastically shortening our time from boot to serving traffic.
The diagram follows the style of the AWS Reference Architectures[0]. I have yet to find a published set of diagram components that compare well to that style.
Edit: Juan Domenech[1] has a small custom made component pallet in png form. And, it looks like the original diagrams are custom Adobe Illustrator drawings[2].
Wish this had a legend. Obviously the box sizes and positioning mean something.
What I can see makes me a bit curious.
- Frontend Web boxes are "big", where the (some) of the backend App boxes are small. I'd have though it would be the reverse -- frontend horizontally scaled first, backend vertically scaled first.
- Sometimes they use Zones ABC, sometimes AB. This is cool, but a lot of the "front" infrastructure is AB (including www), so not sure what advantage having ABC on backend pieces are. These are obviously super-critical for some special reason? I guess they also might be pieces that are-not/only-partially replicated to the secondary site.
- The failover US-West site is using Asgard along with Puppet, but the primary US-East one isn't. I guess this is for managing failure scenarios?
- Prodigious use of a lot of AWS services. Including internal ELBs all over the place. SQL Server and PostgreSQL sneak in a few places.
- Couldn't find the CDN!
- Looks like staging and testing are a complete replica (hard to tell, the resolution isn't quite there). They're big though. This is fine, but raises the question why the secondary site is just a part replica? If you provision the lot to stage, you'd figure you'd run the secondary the same way?
- The Data Warehouse runs on the secondary site, but it only accessed from the primary. Interesting. Wonder why they just didn't put it on the primary?
Box sizes: this system supported lots of different programming languages/framewoerks/pre-built OSS, and as a result, some parts ran nice and lean (python) others chewed memory with reckless abandon (magento). Another factor was supplying enough network bandwidth to some hosts, hence larger sizes.
Zones: most apps were built for 2 AZ's at the start; apps deemed "critical" and "doable" flipped to 3 near the election.
Asgard: deployments had been tested with Asgard in East, but the rapid deploy in west actually used this approach. Thanks Netflix!
Services: some which are critical but missed the chart are IAM, sns, cloudwatch (for autoscaling), and yes, a bit of cloudfront was used on images sent out on SES (transactional only!) emails.
The most surprising thing about this is that test and staging are 1:1 duplicates of production. At the companies I've worked at dealing in scale, that has never been the case.
"Oh, you run 10 servers and 2 load balancers across these 2 sub-systems? Here's 2 servers and a toy whistle for staging."
But then again, I suppose that's what you get when you can dump hundreds of millions of dollars on a problem over a couple years.
With Puppet, and with AWS, presumably it's possible to flip a switch, and 5 minutes later have a replica which is around for the 30m you want to test it for, at 1/250th of the monthly cost
That part is a a little more nebulous. When deploying the entire full stack of all applications is automated, rolling a new environment is very simple. So, for instance, if I wanted to load test a specific end point in our API, I would just build a new instantiation of the API and test against that in a protected and isolated environment.
So in reality, it might be 2 staging environments and 12 testing. There was no real set number of environments.
Really jazzed by CloudOpt, which I didn't know about. I had a multi-region architecture supporting AWS offshore development teams and HA between US-East & US-West, and this would have been a godsend. We had Windows Servers as dev workstations in Singapore and devs were based in India and Canada -- latency was actually fine to the desktop, and we used Amazon's internet connection to haul data back to US-East before hitting our data centre. This was satisfactory for most connectivity (throughput was fine) but occasionally the encapsulated routing would cause packet retry pains that traditional WAN acceleration may have helped with a bunch.
But, overall, this is great work, and shows what's possible to those in the enterprise that think that Amazon is still a bookseller, and clouds are basically things in the sky you don't want on your golf days. This will only help further the progression to way less wasted cost & time in IT. (Unless you're a republican, I suppose. Or maybe especially if you're a republican, you should be studying this for 2016 and considering an alternative to hiring Very Serious IT Consultants.)
I would really be interested in know a) if I had a copy of the Puppet scripts how long would it take get this server configuration up and running and b) how much would the AWS costs be per month?
A) That's a bit nuanced and depends on the application you are referring to (there are ~200 represented here), but if you had the scripts and the repos and built from a vanilla AMI, a box would boot and configure within a few minutes. We shortened the boot times greatly by using puppet to configure a template node which we then used to create a generic versioned AMI per autoscale environment and deployed with Asgard. If you had the AMIs, an EBS backed instance would boot and serve traffic in anywhere between 15 and 30 seconds or so.
B) For costs, you can check the FEC filings... I cannot recall what they were month to month, but it will be accurately reflected quarterly there, so you can get a rough estimate. It was like a kabillion times more in October 2012 than it was in June 2011. The important part for us is that the costs scaled with our demand.
Man, I didn't realize big sites were this complex under the hood. That site wasn't even complicated; if I recall correctly it was just some info pages and payment. I don't know anything about backend stuff. All my projects are like this:
That just drives home how much simpler things have become with no SLA IaaS. Instead of spending 20+ hours racking an east coast/west coast pair of small cages or perhaps just 2 full racks each, you can save time and money by going with simple and glitch proof scale out systems driven by one button provisioning of hundreds of low performing vms. As an added bonus if your san or ethernet collapses you can safely sleep through the pager - somebody else will undoubtedly post the problem on their well monitored support forum.
Let's not forget the capEx advantage - It's not like the Obama organization can afford heavy five or low six figure expenditures on infrastructure when who even knows if they'll be able to raise a series A or will even be around in a year.
Its interesting to note that the the boweb-production servers of "www" don't have read-replicas (asynchronous replication) for read scalability. Anyone know why? Typically, one would use read-replicas to alleviate read workloads on the master.
Most likely the www servers are serving almost entirely static, cached content. Hell, they might not actually see any actual public traffic - they might just serve as CDN origins.
Who would have thought the infrastructure behind Obama's site and whatever else they're running was so big. I always assumed it would be running maybe one or two instances, not require a whole freakin' diagram to explain.
It's interesting that apart from the DynamoDB and the SQL Warehouse (Redshaft?) there's a pretty large LevelDB instance serving the sites/services.
I also wonder if in reality there were only one or two instances in each zone for each site behind the ELBs and the others were added to show how scalable the infrastructure is; or if their number actually did correspond to the avg. number of instances they ended up having to spin up.
It was made by Miles Ward on the AWS team but you can made a similar one at http://www.madeiracloud.com/ (and provision directly from the diagram / overlay monitoring info)
I'd love to describe a network in (something like) graphviz dot notation and get a diagram that looks like this out the other end. Usually I wind up just using Visio and doing it by hand because work pays for it anyway, and it's actually very nice.
That is your logic? I don't care if this infrastructure could send Obama back in time; it doesn't mean it is right for your needs. Some people glorify "successful" architectures, but it is at least as important (if not more so) to pay attention to your particulars.
Serious question: does the 3d mode add any information? I am distracted by it but perhaps the size of the blocks have any attached meaning? Is there a 2d variant available?
Otherwise: very interesting to see such a large infrastructure used for the campaign. I wonder how it compares to an average campaign here in Germany. I'd assume only a 10th of the servers (while spending the same amount of money :)) used here. I can't see anything special on the web before elections other than some wordpress alike web servers.
It is in a way. The department heads were very worried about the scrutiny and tone of the press in covering project narwhal and actively discouraged its use as a codename for the project.
[+] [-] minimax|13 years ago|reply
1) This represents a heterogenous set of services where the engineers in charge of each service were more-or-less free to use whatever technology stack they were most comfortable with.
2) Use puppet for provisioning everything. (There is some subtlety Scott went into about being able to bring up a bunch of servers quickly which I have since forgotten).
3) Everything shuts down the day after the election. Not much time spent future-proofing the code when it has a well known expiration date.
4) Testing testing testing. System had serious problems and 2008 and they didn't want to repeat in 2012.
5) Engineers and sysadmins had a lot of latitude to make changes in order to respond to issues as they developed. It sounded like this is essentially Harper Reed's management style (he seems like a cool guy to work for).
[+] [-] scottvdp|13 years ago|reply
[+] [-] ROFISH|13 years ago|reply
[+] [-] jared314|13 years ago|reply
The diagram follows the style of the AWS Reference Architectures[0]. I have yet to find a published set of diagram components that compare well to that style.
Edit: Juan Domenech[1] has a small custom made component pallet in png form. And, it looks like the original diagrams are custom Adobe Illustrator drawings[2].
[0] http://aws.amazon.com/architecture/
[1] http://blog.domenech.org/2012/06/aws-diagrams-palette-v10.ht...
[2] http://blog.domenech.org/2012/05/aws-diagrams.html
[+] [-] edwinnathaniel|13 years ago|reply
Oh well, I guess GAE is for toy project ahem.
[+] [-] jwilliams|13 years ago|reply
What I can see makes me a bit curious.
- Frontend Web boxes are "big", where the (some) of the backend App boxes are small. I'd have though it would be the reverse -- frontend horizontally scaled first, backend vertically scaled first.
- Sometimes they use Zones ABC, sometimes AB. This is cool, but a lot of the "front" infrastructure is AB (including www), so not sure what advantage having ABC on backend pieces are. These are obviously super-critical for some special reason? I guess they also might be pieces that are-not/only-partially replicated to the secondary site.
- The failover US-West site is using Asgard along with Puppet, but the primary US-East one isn't. I guess this is for managing failure scenarios?
- Prodigious use of a lot of AWS services. Including internal ELBs all over the place. SQL Server and PostgreSQL sneak in a few places.
- Couldn't find the CDN!
- Looks like staging and testing are a complete replica (hard to tell, the resolution isn't quite there). They're big though. This is fine, but raises the question why the secondary site is just a part replica? If you provision the lot to stage, you'd figure you'd run the secondary the same way?
- The Data Warehouse runs on the secondary site, but it only accessed from the primary. Interesting. Wonder why they just didn't put it on the primary?
[+] [-] miles932|13 years ago|reply
Zones: most apps were built for 2 AZ's at the start; apps deemed "critical" and "doable" flipped to 3 near the election.
Asgard: deployments had been tested with Asgard in East, but the rapid deploy in west actually used this approach. Thanks Netflix!
Services: some which are critical but missed the chart are IAM, sns, cloudwatch (for autoscaling), and yes, a bit of cloudfront was used on images sent out on SES (transactional only!) emails.
Thanks for the good questions!
[+] [-] pfg|13 years ago|reply
[+] [-] mratzloff|13 years ago|reply
"Oh, you run 10 servers and 2 load balancers across these 2 sub-systems? Here's 2 servers and a toy whistle for staging."
But then again, I suppose that's what you get when you can dump hundreds of millions of dollars on a problem over a couple years.
[+] [-] peteretep|13 years ago|reply
[+] [-] scottvdp|13 years ago|reply
So in reality, it might be 2 staging environments and 12 testing. There was no real set number of environments.
[+] [-] miles932|13 years ago|reply
[+] [-] quellhorst|13 years ago|reply
[+] [-] parasubvert|13 years ago|reply
But, overall, this is great work, and shows what's possible to those in the enterprise that think that Amazon is still a bookseller, and clouds are basically things in the sky you don't want on your golf days. This will only help further the progression to way less wasted cost & time in IT. (Unless you're a republican, I suppose. Or maybe especially if you're a republican, you should be studying this for 2016 and considering an alternative to hiring Very Serious IT Consultants.)
[+] [-] miles932|13 years ago|reply
[+] [-] stevewilhelm|13 years ago|reply
[+] [-] scottvdp|13 years ago|reply
B) For costs, you can check the FEC filings... I cannot recall what they were month to month, but it will be accurately reflected quarterly there, so you can get a rough estimate. It was like a kabillion times more in October 2012 than it was in June 2011. The important part for us is that the costs scaled with our demand.
[+] [-] lifeformed|13 years ago|reply
[traffic] -> [hosted server] -> [Amazon CloudFront files]
[+] [-] thrownaway2424|13 years ago|reply
[+] [-] JulianMorrison|13 years ago|reply
[+] [-] miles932|13 years ago|reply
[+] [-] trotsky|13 years ago|reply
Let's not forget the capEx advantage - It's not like the Obama organization can afford heavy five or low six figure expenditures on infrastructure when who even knows if they'll be able to raise a series A or will even be around in a year.
[+] [-] bonertruck|13 years ago|reply
[+] [-] dahumpty|13 years ago|reply
[+] [-] ceejayoz|13 years ago|reply
[+] [-] DigitalSea|13 years ago|reply
[+] [-] georgecalm|13 years ago|reply
I also wonder if in reality there were only one or two instances in each zone for each site behind the ELBs and the others were added to show how scalable the infrastructure is; or if their number actually did correspond to the avg. number of instances they ended up having to spin up.
[+] [-] gulfie|13 years ago|reply
[+] [-] danoprey|13 years ago|reply
[+] [-] maxerickson|13 years ago|reply
http://aws.amazon.com/solutions/case-studies/obama/
(which then links back to the map)
[+] [-] Ogre|13 years ago|reply
I'd love to describe a network in (something like) graphviz dot notation and get a diagram that looks like this out the other end. Usually I wind up just using Visio and doing it by hand because work pays for it anyway, and it's actually very nice.
[+] [-] WestCoastJustin|13 years ago|reply
[1] http://www.youtube.com/watch?v=X1tJAT7ioEg
[+] [-] miles932|13 years ago|reply
[+] [-] unknown|13 years ago|reply
[deleted]
[+] [-] dj-wonk|13 years ago|reply
[+] [-] adam-f|13 years ago|reply
I think the only architecture that can be copied is the replication in staging and test.
[+] [-] patrickg|13 years ago|reply
Otherwise: very interesting to see such a large infrastructure used for the campaign. I wonder how it compares to an average campaign here in Germany. I'd assume only a 10th of the servers (while spending the same amount of money :)) used here. I can't see anything special on the web before elections other than some wordpress alike web servers.
[+] [-] pm24601|13 years ago|reply
[+] [-] scribu|13 years ago|reply
Edit: D'oh: http://en.wikipedia.org/wiki/Project_Narwhal (disappointed it's not an easter egg)
[+] [-] bonertruck|13 years ago|reply
[+] [-] TallboyOne|13 years ago|reply
[+] [-] Kiro|13 years ago|reply
[+] [-] pm24601|13 years ago|reply