I don't understand why people think that companies can get special pricing from AWS but not drive feature development. Large customers like Netflix, GE & friends get to drive feature development. Heck, if you're the CIA and give AWS $600 MM they will build a private region for you!
This post has two contradictory quotes in my opinion:
"It’s worth noting that this AWS price wouldn’t be what Facebook paid in this hypothetical situation. Much like Snapchat and Netflix, Facebook would be a heavy and influential user"
"Facebook’s years of specialization for running Facebook are in contrast to AWS, whose storage is designed for multi-purpose (albeit-heavy) use." - Amazon already has several tiers of storage (S3, S3 - Infrequent Access, Glacier, CloudFront)... who is to say that they wouldn't be motivated to introduce more classes by a large customer like FB.
Therese aren't so much contradictions as indicators the reasonable answer to the question posed is really a simple 'No'. The only sane way it becomes yes is 'Yes it's possible if Facebook rewrote a bunch of Facebook AND Amazon rewrote a bunch of AWS'. That kind of possible isn't very interesting, though. It's surely possible to host Facebook on AWS if Amazon built an exact replica of each FB data center, trucked FB's servers over and then called the result part of AWS.
I think a lot of the smaller details this article glosses over contribute to FB's ability to run on so few machines. For example, comparing my anecdotes of FB's internal network (they probably have the most organized/mature IPv6 network on the planet) with those of my AWS public cloud experience, I'd reckon that a lot of FB applications would require architectural redesigns to operate in that kind of environment.
The real question isn't "can FB be hosted on AWS?", it's "why isn't FB competing with AWS?" because what they've already got is much better for the range of applications that they deploy.
I suppose the reason Facebook doesn't resell their infrastructure is because their endgame is to _be_ the internet. From their point of view, any site you're on that isn't Facebook is their competition.
They went from $3.6 billion to $10.2 billion in net income in a year. The $3.6 billion is probably more net income than Amazon has generated combined in the last two decades. The $10 billion is more than Amazon will earn a decade from now.
Their focus should be solely on what is going to likely end up being one of the three or four greatest money producing machines in world history (of those owned by a traditional corporation that is).
Cloud computing profits are - and will remain - a joke compared to the $20 billion in net income they'll yield in just a few more years from what they're already doing. Their focus should remain fairly narrow, they're a mere 13 years in at this point. Diving into cloud computing would be making the same exact mistake that Microsoft made with Bing, and that Google made with all their laughable social attempts, and so on. Facebook would split the market further and likely still end up #2 or #3 at best. Their best engineering talent should be solely focused on the golden goose social monopoly, which nobody else has and nobody else can compete with.
> The real question isn't "can FB be hosted on AWS?", it's "why isn't FB competing with AWS?" because what they've already got is much better for the range of applications that they deploy.
This. When they bought Parse I thought for sure they would get into the same business as AWS / Azure / etc but then they phased it out. Their have so much network infrastructure I thought for sure they would want to offer such services and yet they still have not.
I'm starting to think they won't ever get into the same business.
One of the big things it glosses over is core counts... counting "servers" ignores that AWS has current-gen instance families with as few as 32 VCPUs in their largest instance type versus 128 VCPUs in the x1 family. If you look at multiple generations of families the difference between "servers" becomes even more stark.
It could be relevant if there is some fixed refresh cycle of servers. That is, newer servers can serve more end users because of the faster cpu, drives, etc. The author seems to have skipped the concept though.
In 2012 Facebook had 1BN users and 180K servers.
1,000,000 users / 180,000 servers = 5,556 users per server
In 2017 Facebook has close to 2BN users.
2,000,000 users / 5556 users per server = 360,000 servers
Notice how the 1B has only 6 zeros. The awesome part is that this mistake evens itself out later, yielding a correct number for "The of servers".
Keep in mind, you can bet your ass that prices would be drastically lower than the public prices posted.
Large enterprises of any significance, and especially flagship/strategic customers, don't pay list prices.
Every cloud provider has private pricing and enterprise discount programs. That goes for all hardware vendors (e.g. Cisco, Palo Alto Networks, Oracle, etc selling to any company) as well.
I'm sure Netflix pays nowhere near public pricing on AWS and I'm sure AWS pays less than anyone in the world for an Intel CPU.
Reddit and Netflix are hosted on AWS so yes you could.
You wouldn't run it the same as if you had your own infrastructure though, and probably you would run it more horizontal and cached but it could be done much cheaper than this estimate. On your own environments you can use larger servers, dbs and sharding or clustering is closer. But the cloud is different and you'd be able to run it but differently. The cloud definitely influences architecture.
> Reddit and Netflix are hosted on AWS so yes you could.
Those are read-heavy properties (yes, even Reddit, a miracle of aggressive caching). Don't underestimate Facebook's write load, which is the bloody difficult thing to scale.
Some big players with their own hardware host overflow capacity in cloud data centers, e.g. so that in DDOS they can dip into additional capacity. I don't know if Facebook does this, but I expect that many of the big players already peer with cloud data centers.
"We estimate that Facebook has 830,000 servers in 2017."
No way this can be true. Facebook scaled primarily using CDNs like Akamai. Akamai is the largest CDN in the world and has no more than 300k-350k servers all over the world. It's one of the largest distributed system in the world. I worked there, so I know this for sure.
Facebook has recently been moving their data in house, but they do not have 830,000 servers. That's almost 3 times Akamai. At this point, Facebook would be more profitable letting people use their server infrastructure instead of using it for themselves.
Running Facebook over AWS is a CDN problem that was solved by Akamai!
Since we're guesstimating numbers a lot, it would be interesting to also guesstimate the human resources costs required to run the infrastructure: X electrical engineers, Y cooling engineers, Z SREs, W infrastructure devops ppls, etc.
Using the cloud you don't need X,Y,Z and presumably will need less of W. Assuming a 100+k/year salary, that's like 200+k/year total cost of employment. If X+Y+Z-ΔW ~= 1000, then the cloud provides a 200M savings on that front. The real number is probably 2x 3x higher, but I'm not going to bother going back and updating the guesstimates.
You are also thinking about this in a perfect vacuum. Global scale presents an entirely different --non-technical-- challenge.
Take for example, Dropbox stated they are moving off AWS -- except in Europe (and other non-US areas I'm sure). Having boots on the ground abroad, operating 24/7, dedicated to the mission of the company where they have not even visited ... near impossible.
Why? Hiring people around the globe is hard who are going to remove that HDD 24/7 is pretty hard. Hiring lawyers to understand foreign law is hard. Keeping your company focused abroad and building a culture abroad is very hard.
How long does it take for a company to expand to India on their own? In AWS, you just change your CloudFormation template to another region.
Right; but it doesn't change the fact that any company capable of operating technical infrastructure at scale (aka any company with compute/storage needs above a half-dozen long rows of racks at a datacenter) can do it for about 60% of what it would cost on AWS.
AWS is still a huge boon to startups from a cash flow perspective - it's like leasing equipment rather than buying it, except you're also leasing a very small fraction of all the folks you need to build/operate the equipment. It's also very cheap compared to trying to build and run the infrastructure yourself at small scale.
And don't kid yourself -- you still need SREs in AWS. They're working on different issues, but AWS introduces as many issues in an SRE's scope as it removes.
> We need to take into account that Facebook not only has double the users but also more data created per person - photos, videos, live streams etc. Plus it now hosts Instagram. So let’s double the number.
... but apparently no efficiency gains in 5 years to reduce the number again? Faster processes, more cores, more ram, better storage, more senior-developer-hours, bespoke datacentres... ?
I think you could easily build the NEXT facebook on AWS or any of the big cloud computing platforms. But obviously moving facebook to AWS at its current size wouldn't be practical, and no one would probably ever do something like that.
The answer is no. This is because you can't compare commodity hosting like AWS with specialized hosting like Facebook's data centers and infrastructure software. This question of possibility has nothing to do with 'number of servers' or 'amount of money'. It's architecturally incompatible.
> I'm guessing HHVM only really powers the API, business logic, etc.
So "only" everything except UI (which is run by the browser, not FB's servers), then. And that's only for browsers that use it at all, as they still need to render a lot of stuff from the servers both for speed and for people who turn off js or use older or limited browsers.
And this is a weird thing to say anyway, nothing runs "on" react; both the framework(s) and the business logic run on something else.
[+] [-] discodave|8 years ago|reply
This post has two contradictory quotes in my opinion:
"It’s worth noting that this AWS price wouldn’t be what Facebook paid in this hypothetical situation. Much like Snapchat and Netflix, Facebook would be a heavy and influential user"
"Facebook’s years of specialization for running Facebook are in contrast to AWS, whose storage is designed for multi-purpose (albeit-heavy) use." - Amazon already has several tiers of storage (S3, S3 - Infrequent Access, Glacier, CloudFront)... who is to say that they wouldn't be motivated to introduce more classes by a large customer like FB.
[+] [-] BurningFrog|8 years ago|reply
[+] [-] pvg|8 years ago|reply
[+] [-] jzelinskie|8 years ago|reply
The real question isn't "can FB be hosted on AWS?", it's "why isn't FB competing with AWS?" because what they've already got is much better for the range of applications that they deploy.
[+] [-] ryandvm|8 years ago|reply
[+] [-] adventured|8 years ago|reply
Their focus should be solely on what is going to likely end up being one of the three or four greatest money producing machines in world history (of those owned by a traditional corporation that is).
Cloud computing profits are - and will remain - a joke compared to the $20 billion in net income they'll yield in just a few more years from what they're already doing. Their focus should remain fairly narrow, they're a mere 13 years in at this point. Diving into cloud computing would be making the same exact mistake that Microsoft made with Bing, and that Google made with all their laughable social attempts, and so on. Facebook would split the market further and likely still end up #2 or #3 at best. Their best engineering talent should be solely focused on the golden goose social monopoly, which nobody else has and nobody else can compete with.
[+] [-] BinaryIdiot|8 years ago|reply
This. When they bought Parse I thought for sure they would get into the same business as AWS / Azure / etc but then they phased it out. Their have so much network infrastructure I thought for sure they would want to offer such services and yet they still have not.
I'm starting to think they won't ever get into the same business.
[+] [-] jsolson|8 years ago|reply
[+] [-] deepnotderp|8 years ago|reply
[+] [-] vacri|8 years ago|reply
A naive guess would be that providing services for users you don't control is harder than for users you do control.
[+] [-] OJFord|8 years ago|reply
As well as inflation, and all manner of other totally irrelevant concepts.
[+] [-] tyingq|8 years ago|reply
[+] [-] jxramos|8 years ago|reply
[+] [-] TheChetan|8 years ago|reply
Notice how the 1B has only 6 zeros. The awesome part is that this mistake evens itself out later, yielding a correct number for "The of servers".
[+] [-] p0rkbelly|8 years ago|reply
Large enterprises of any significance, and especially flagship/strategic customers, don't pay list prices.
Every cloud provider has private pricing and enterprise discount programs. That goes for all hardware vendors (e.g. Cisco, Palo Alto Networks, Oracle, etc selling to any company) as well.
I'm sure Netflix pays nowhere near public pricing on AWS and I'm sure AWS pays less than anyone in the world for an Intel CPU.
[+] [-] zeep|8 years ago|reply
[+] [-] virtuallynathan|8 years ago|reply
Disclamer: I work for AWS.
[+] [-] kondro|8 years ago|reply
[+] [-] p0rkbelly|8 years ago|reply
Hope you don't work with any anything to do with scale at AWS.
You're looking more towards 1 Exabyte a month...
[+] [-] drawkbox|8 years ago|reply
You wouldn't run it the same as if you had your own infrastructure though, and probably you would run it more horizontal and cached but it could be done much cheaper than this estimate. On your own environments you can use larger servers, dbs and sharding or clustering is closer. But the cloud is different and you'd be able to run it but differently. The cloud definitely influences architecture.
[+] [-] jsmthrowaway|8 years ago|reply
Those are read-heavy properties (yes, even Reddit, a miracle of aggressive caching). Don't underestimate Facebook's write load, which is the bloody difficult thing to scale.
[+] [-] Thaxll|8 years ago|reply
[+] [-] unknown|8 years ago|reply
[deleted]
[+] [-] philip1209|8 years ago|reply
[+] [-] jaypaulynice|8 years ago|reply
No way this can be true. Facebook scaled primarily using CDNs like Akamai. Akamai is the largest CDN in the world and has no more than 300k-350k servers all over the world. It's one of the largest distributed system in the world. I worked there, so I know this for sure.
Facebook has recently been moving their data in house, but they do not have 830,000 servers. That's almost 3 times Akamai. At this point, Facebook would be more profitable letting people use their server infrastructure instead of using it for themselves.
Running Facebook over AWS is a CDN problem that was solved by Akamai!
[+] [-] ivansavz|8 years ago|reply
Using the cloud you don't need X,Y,Z and presumably will need less of W. Assuming a 100+k/year salary, that's like 200+k/year total cost of employment. If X+Y+Z-ΔW ~= 1000, then the cloud provides a 200M savings on that front. The real number is probably 2x 3x higher, but I'm not going to bother going back and updating the guesstimates.
[+] [-] p0rkbelly|8 years ago|reply
Take for example, Dropbox stated they are moving off AWS -- except in Europe (and other non-US areas I'm sure). Having boots on the ground abroad, operating 24/7, dedicated to the mission of the company where they have not even visited ... near impossible.
https://blogs.dropbox.com/business/2016/09/making-european-i...
Why? Hiring people around the globe is hard who are going to remove that HDD 24/7 is pretty hard. Hiring lawyers to understand foreign law is hard. Keeping your company focused abroad and building a culture abroad is very hard.
How long does it take for a company to expand to India on their own? In AWS, you just change your CloudFormation template to another region.
[+] [-] exelius|8 years ago|reply
AWS is still a huge boon to startups from a cash flow perspective - it's like leasing equipment rather than buying it, except you're also leasing a very small fraction of all the folks you need to build/operate the equipment. It's also very cheap compared to trying to build and run the infrastructure yourself at small scale.
And don't kid yourself -- you still need SREs in AWS. They're working on different issues, but AWS introduces as many issues in an SRE's scope as it removes.
[+] [-] deepGem|8 years ago|reply
1,000,000 users / 180,000 servers = 5,556 users per server
Should be
1,000,000,000 users / 180,000 servers = 5,556 users per server
[+] [-] vacri|8 years ago|reply
... but apparently no efficiency gains in 5 years to reduce the number again? Faster processes, more cores, more ram, better storage, more senior-developer-hours, bespoke datacentres... ?
[+] [-] crispytx|8 years ago|reply
[+] [-] johnsmith21006|8 years ago|reply
[+] [-] luord|8 years ago|reply
[+] [-] oneplane|8 years ago|reply
[+] [-] accountyaccount|8 years ago|reply
[+] [-] curiousgal|8 years ago|reply
[+] [-] pier25|8 years ago|reply
This is surprising since 1) Facebook is most likely using Hack and 2) it could save a lot of money by moving to Go.
> Facebook’s entire site runs on HHVM (desktop, API and mobile)
Isn't Facebook running mostly on React? I'm guessing HHVM only really powers the API, business logic, etc.
[+] [-] cbhl|8 years ago|reply
That's why they invented Hack and HHVM in the first place -- it was a cheap-enough compromise that didn't require rewriting the whole code base.
[+] [-] ot|8 years ago|reply
[citation needed]
[+] [-] sgslo|8 years ago|reply
The majority (all?) of the front end and mobile apps are powered by React and React Native, respectively.
[+] [-] luord|8 years ago|reply
So "only" everything except UI (which is run by the browser, not FB's servers), then. And that's only for browsers that use it at all, as they still need to render a lot of stuff from the servers both for speed and for people who turn off js or use older or limited browsers.
And this is a weird thing to say anyway, nothing runs "on" react; both the framework(s) and the business logic run on something else.
[+] [-] pryelluw|8 years ago|reply
[+] [-] kondro|8 years ago|reply