I love GCP but honestly have no idea how these big internet properties can make the $ case for “the cloud”, unless they’re also planning to fire 90% of the Ops/SysAdmin team (ala Evernote).
We run on bare metal and to move to GCP would triple our hosting cost, not including bandwidth (which is free* for us).
Do people get discounts for doing PR? I’m really curious.
The hardware and bandwidth costs are only small parts of the full cost of running your own infrastructure. There are so many other benefits:
- you don't have to support your own low level services like Pub/Sub, Cloud Storage, Spanner, etc.
- you don't have to worry as much about things like HIPAA compliance
- developers don't have to wait for you to allocate more hardware, greatly slowing down the software development process
- it's much easier to test your applications, since you can spin up a full copy using rented infrastructure, and then throw it all away once your tests are complete
- many other benefits that I'm forgetting
And of course if you buy enough from them, you do get additional discounts.
For many (most? I don't have data) companies, the actual cost of the infrastructure and bandwidth itself is tiny compared to all the other "hidden costs".
I don't have very detailed knowledge, but my impression is that this move doesn't make long-term financial sense for the company, but it's being pushed by activist investors / board members who want to appear like they're good stewards of money because investors think cloud is cheaper / etc. (Note that this is on investors.etsy.com, not their tech blog; you can also find a call for public cloud in the Black & White Capital letter, in the NYTimes story from a couple weeks ago, etc.) I heard some rumors about this when I was applying for a job right before the layoffs happened, and I didn't hear anyone expressing interest in moving out of their datacenters before the layoffs.
To answer your question, people get discounts just for negotiating, no PR required. I would guess the conversation went something like this: "___ is how much we pay to host Etsy on bare metal right now. Can you match that?" "Yes."
I'm oversimplifying - there are a bunch of other considerations, like guarantees about future prices and capacity availability and whether having more flexibility makes it worthwhile for Etsy to pay more, but fundamentally Etsy looked at their current costs and asked Google to make an offer. (I would assume they actually had all three major players bid, and Google won.)
Also to your point, the announcement specifically called out Google's AI and big data capabilities. Those are workloads that can involve a lot of bursting, so the cloud might be especially attractive. Imagine trying to implement BigQuery in bare metal - you can do it, but you have to have a lot of extra servers lying around just in case someone tries to run an analytics query. The same goes even more strongly for something like a model training workload. For bursty enough workloads, running in the cloud can be cheaper than bare metal even given the cloud provider's profit margin.
Depending on how much model training Etsy wants to do, something like TPUs might have also given Google a fundamental edge in costs over bare metal or other providers. I don't know enough to know exactly what factors went into Etsy's decision, but I think these are reasonable ideas.
As with everything else, the prices will reduce over time. But to be honest: you are not the market. If you have a functioning, reliable infrastructure that isn't too expensive or PITA to maintain, you have a good enough infra and should continue using that.
If you're a startup who wants to minimize capex and quickly spin up a reliable, secure infra to host your app on to get it out of the door ASAP, you would go with cloud.
If you're a non-technical company with a small IT department that wants to have an online presence and/or build internal online tools without having to build out DC's everywhere, you would go with cloud.
I could come up with dozens and dozens more. Cloud isn't optimal for everyone, but it is VERY helpful for certain organizations.
I work for a company similar to GrubHub outside the US. 'The Cloud' works great for us. Think, how many people order food at 9AM on a Monday? Therefore, we only have a very small amount of servers running. However, 7PM on a Friday... thousands.
Yes, it's still mega expensive. But allows us to scale when we need it and not have to worry about the hardware, cost, or management of it.
There's an entire other side of building a web server that devs, thankfully, never have to see. OpenCompute [1] aims to standardize what that process looks like. To think that you can run bare metal, and do better than industry standards is naive IMO.
In the end, if you're worrying more about anything than business logic, you're not really delivering value -- you're solving engineering problems. You can sacrifice that control and solving problems for greater abstractions, i.e. serverless, fargate, etc.
Unless you have some extreme security requirements (which most don't) you probably don't have a good reason to run bare metal.
GCP is an operational expense, whereas bare metal is operational + capital expense. It might be more expensive for cloud, but the accounting looks a little cleaner.
I worked for a company that had bare metal. They were always shipping parts around, lots of movement back and forth between the data center, lots of night deploys for ops team which meant a burden on the HR department. There was a general anxiety about security.
It might be "cheaper" on paper to run bare metal, but it means your managers and executives have to make a lot of dinky little decisions all the time. I'd happily pay more to "push and forget".
I would be shocked if Etsy/Spotify/whoever else wasn't cutting a deal for a significant discount, getting these name brands on GCP is great PR for the platform.
Lifetime costs for much of this technology is more people costs. Also, much of the benefit comes from what's on top of the hosting. (Not having to worry about old versions of software, containers, better security, no issues if the one person who knows what's going on quits, etc)
> I love GCP but honestly have no idea how these big internet properties can make the $ case for “the cloud”, unless they’re also planning to fire 90% of the Ops/SysAdmin team (ala Evernote).
What kind of price differences are you thinking about? How much hardware and how many people are employed to take care of this? Does the tripling of cost take salaries into account?
Having to hire a team that will maintain hardware, deal with scalability and give guaranteed uptime isn't cheap and requires a lot of management. I'm not saying it never makes sense, but offloading all that work on to someone like AWS and Google who have teams doing this for you reliably can be well worth the cost.
most cost cuts come from automation (as you already noticed).
but not just more automation in ops/sysadmin space, but mostly because you can just free resources you don't need.
especially one-off jobs (analytics, etc.) take a huge amount of resources for a very short amount of time and after that time the resources aren't needed.
on clouds you can hope to schedule them preemtivly.
also most cost savings come at night, if you are not active on the whole globe you probably only need half of your resources at night. even if your global the chances are high that at some time you still need way less resources.
etsy is a marketplace on most marketplaces there are times where demand is really high and demand is certainly not so high. (christmas, black firday, regressions, etc..)
also if you self host, high availability can and is a pain.
monitoring your database is a pain, it takes a huge amount of time, which most companies do not have.
p.s. working at a small company and services like RDS/gcloud postgres! (thanks google) google buckets (not sure how they are called their name is not as cool as s3)/S3 are a huge win for us.
* elastic demand, either in the sense that your global traffic has peaks and troughs, or your global traffic is steady but its global distribution changes such that the balance of where you're deployed would ideally change over time; in either case you can benefit enormously from automatic scaling
* workloads that can take advantage of the spot market (or whatever GCP's equivalent is)
With bare metal you pay for servers you don't use. Cloud is more expensive on the compute side but you can really scale on your usage pattern, also security on the cloud is way better that on bare metal.
These calculations don't make sense unless you really consider TCO over any appreciable amount of time.
It's easy to say you can buy a huge server for a few thousand but the "cloud" provides incredible agility, security, and basically removes operational overhead. When compared to buying, maintaining, and decommissioning servers and software along with the time and personnel costs, it becomes a much closer race. Include global scale, elastic capacity, managed services, and support policies, and it's closer still.
Rarely are companies paying 3x IT spend just to run in the cloud, especially for large companies like Etsy.
Found some interesting contrasting statements from 2015 [1]:
> The company performed big data jobs early on in the cloud, using Amazon's Elastic Map Reduce service.
> “That [EMR] got us to a place where we think about what data we collect. At the time it was really inefficient from a cost perspective, but there weren’t any creative constraints,”
> The team eventually decided to bring it in house. “Our Hadoop cluster wields a huge amount of flexibility," said Rembetsy. "We can get a lot done on bare metal, and we know very much what we’re looking for. Cloud services, that’s where efficiencies fall away. Spin up and spin down doesn’t have the same value proposition.”
Sounds good. I was an early Etsy seller, as a glass artist and bead vendor, As well as a developer who worked with them as closely as I could for a few years.
They have always seemed a bit disconnected from their customers. The first regime, Rob Kalin, Haim and Revolving Dork, had an age and culture disconnect from most of their customers. The next group, headed by Chad Dickerson and Allspaw, seemed like they wanted to focus on technology for technology’s sake, more than what Etsy was actually doing with that as a company. They even made some statements to the effect of how they wanted Etsy to be known as a technology company, like Facebook, and be a platform, like Amazon.
All fine and good, but so far, what all of the leaders of Etsy have had in common is that they don’t really understand what it’s like to be a micro business person selling handmade art online. If I was an investor, I wouldn’t want to be paying for a bunch of guys to experiment with Mongo DB search engines and shards. As long as they’re still not understanding their core business, it’s a distraction.
Etsy should be focused on one thing: connecting buyers and sellers of handmade art with each other.
> The next group, headed by Chad Dickerson and Allspaw, seemed like they wanted to focus on technology for technology’s sake, more than what Etsy was actually doing with that as a company.
I think you're missing quite a bit of this story. I was around during the transition of Chad from CTO -> CEO, and the truth is that Etsy needed to focus on the technology badly during that time. The site would go down constantly. We had very little insight into what was going on with sellers or buyers. Every busy day was basically us crossing our fingers that the site wouldn't go down.
We needed someone focused heavily on the technology. We could never have continued on that path and grown as much as we did.
> Etsy should be focused on one thing: connecting buyers and sellers of handmade art with each other.
Deciding what potential buyers are interested in and what items to display to them is a pretty difficult technology problem.
Detecting mass-produced goods in a way that's more subtle than "ban all Chinese sellers" is a pretty difficult technology problem.
If anything I would argue that Etsy didn't focus nearly enough on that, and focused way too much on "low-hanging fruit" (minor workflow adjustments for sellers and buyers).
Not much meat for the technically interested here unfortunately. The respective business departments were clearly in charge of this press release. Which was a business mistake in my opinion: these press releases are primarily interesting for the CTOs and similar - the fluff presented here would serve to remind them of how they are kept at arms length i many board rooms even in their areas of expertise, more than anything. My guess is both Google Cloud and Etsy would dip in their opinion as result of it.
Probably because Google extends really really good deal to them for the PR effect.
I don't think Etsy should consider itself as a competitor to Amazon, they simply can't. On the other side, AWS already has so many success stories that they are unlikely cater to Etsy specifically. That makes GCE the best option out there. As to Azure, I think people trust Google more as to cloud-based technology than Microsoft.
Price breaks would be my guess. Famous names can get discounts in exchange for going public with the transition. I don't have evidence to back this up, but I get the sense that Google offers the deepest discounts, possibly because AWS and Azure have more mindshare and Google wants to catch up.
Google's marketing engine just remembered earlier this year that they have a cloud product that they kind of forgot to push for the last 3-4 years and decided to cut the DevOps poster child a better deal in exchange for a high-profile migration to the exclusion of the other cloud providers.
> Etsy's transition to the Google Cloud Platform is expected to be complete within the next two years.
I have no idea what Etsy looks like internally, but that sounds like an awfully long time in this day and age. Anyone have an inkling as to why this is a multi-year project?
Having worked for two companies that have shifted to cloud providers, multi-year timescales is fairly usual even just for like-for-like migrations. You can’t just shut down the servers, image them and start them in the cloud, you need to have a plan to progressively migrate services while maintaining site operation.
I would bet part of this has to do with the support cycle of the hardware they currently have, and other contracts: they have warranty/service/support on their current hardware, and contracts in place for their colocation facility, so they'll keep using those until they're (near) the end.
I'd imagine that the move would require some re-engineering of their services to make the best use of GCP. I'd also imagine that their would be a non-trivial migration process to ensure uptime is maintained and customers are not impacted during the move.
I would love to see a cost analysis of a company moving to GCP/AWS.
We were a 6M customer flash sale company sold for $200M. We had 3 devops engineers mostly helping developers with designing the system and pager duty - mainly because 24/7 was to demanding for only 2 people. We were SOX/PCI compliant.
Hosting costs all in all by year were around the same as the salaries I'd say.
People who moved to AWS/GCP, would I save costs in this scenario moving to AWS/GCP?
Etsy seems to be fine for the most part. The only criticism I have of Etsy is that they've moved away or diluted their core business (small business/home business sales) by allowing mass manufactured goods to be sold on the site.
Etsy still has a place, and I've ordered several items from Etsy vendors this last year. They just need to keep a check on what they want to be otherwise they'll be subpar eBay clone.
Act 1) Yay, we did it, we're gonna save so much money!
Act 2) [a year later] Wait, we're spending how much money?!?
Act 3) It's going to cost us how much to leave our current cloud?!?
Haven't people figured out yet that cloud companies make a profit? Sure, companies like Etsy can get a huge discount, but everyone else who isn't a big name is going to pay out the ass.
[+] [-] foobarbazetc|8 years ago|reply
We run on bare metal and to move to GCP would triple our hosting cost, not including bandwidth (which is free* for us).
Do people get discounts for doing PR? I’m really curious.
[+] [-] blaisio|8 years ago|reply
- you don't have to support your own low level services like Pub/Sub, Cloud Storage, Spanner, etc.
- you don't have to worry as much about things like HIPAA compliance
- developers don't have to wait for you to allocate more hardware, greatly slowing down the software development process
- it's much easier to test your applications, since you can spin up a full copy using rented infrastructure, and then throw it all away once your tests are complete
- many other benefits that I'm forgetting
And of course if you buy enough from them, you do get additional discounts.
For many (most? I don't have data) companies, the actual cost of the infrastructure and bandwidth itself is tiny compared to all the other "hidden costs".
[+] [-] geofft|8 years ago|reply
[+] [-] noahl|8 years ago|reply
I'm oversimplifying - there are a bunch of other considerations, like guarantees about future prices and capacity availability and whether having more flexibility makes it worthwhile for Etsy to pay more, but fundamentally Etsy looked at their current costs and asked Google to make an offer. (I would assume they actually had all three major players bid, and Google won.)
Also to your point, the announcement specifically called out Google's AI and big data capabilities. Those are workloads that can involve a lot of bursting, so the cloud might be especially attractive. Imagine trying to implement BigQuery in bare metal - you can do it, but you have to have a lot of extra servers lying around just in case someone tries to run an analytics query. The same goes even more strongly for something like a model training workload. For bursty enough workloads, running in the cloud can be cheaper than bare metal even given the cloud provider's profit margin.
Depending on how much model training Etsy wants to do, something like TPUs might have also given Google a fundamental edge in costs over bare metal or other providers. I don't know enough to know exactly what factors went into Etsy's decision, but I think these are reasonable ideas.
[+] [-] pm90|8 years ago|reply
If you're a startup who wants to minimize capex and quickly spin up a reliable, secure infra to host your app on to get it out of the door ASAP, you would go with cloud.
If you're a non-technical company with a small IT department that wants to have an online presence and/or build internal online tools without having to build out DC's everywhere, you would go with cloud.
I could come up with dozens and dozens more. Cloud isn't optimal for everyone, but it is VERY helpful for certain organizations.
[+] [-] azurezyq|8 years ago|reply
* Well designed security / encryption may cost a lot.
* Compliance may be a pain.
* multi-DC and HA
* Surge resource usage over holidays.
* "Sleeping" resources at night or wasted.
[+] [-] 7ewis|8 years ago|reply
Yes, it's still mega expensive. But allows us to scale when we need it and not have to worry about the hardware, cost, or management of it.
[+] [-] olingern|8 years ago|reply
There's an entire other side of building a web server that devs, thankfully, never have to see. OpenCompute [1] aims to standardize what that process looks like. To think that you can run bare metal, and do better than industry standards is naive IMO.
In the end, if you're worrying more about anything than business logic, you're not really delivering value -- you're solving engineering problems. You can sacrifice that control and solving problems for greater abstractions, i.e. serverless, fargate, etc.
Unless you have some extreme security requirements (which most don't) you probably don't have a good reason to run bare metal.
[1] - http://www.opencompute.org/
[+] [-] jbob2000|8 years ago|reply
I worked for a company that had bare metal. They were always shipping parts around, lots of movement back and forth between the data center, lots of night deploys for ops team which meant a burden on the HR department. There was a general anxiety about security.
It might be "cheaper" on paper to run bare metal, but it means your managers and executives have to make a lot of dinky little decisions all the time. I'd happily pay more to "push and forget".
[+] [-] spencerhakim|8 years ago|reply
[+] [-] NelsonMinar|8 years ago|reply
[+] [-] mathattack|8 years ago|reply
[+] [-] seanwilson|8 years ago|reply
What kind of price differences are you thinking about? How much hardware and how many people are employed to take care of this? Does the tripling of cost take salaries into account?
Having to hire a team that will maintain hardware, deal with scalability and give guaranteed uptime isn't cheap and requires a lot of management. I'm not saying it never makes sense, but offloading all that work on to someone like AWS and Google who have teams doing this for you reliably can be well worth the cost.
[+] [-] cobookman|8 years ago|reply
(Disclamer: I'm a Cloud Solutions Engineer at Google Cloud)
[+] [-] merb|8 years ago|reply
also most cost savings come at night, if you are not active on the whole globe you probably only need half of your resources at night. even if your global the chances are high that at some time you still need way less resources. etsy is a marketplace on most marketplaces there are times where demand is really high and demand is certainly not so high. (christmas, black firday, regressions, etc..)
also if you self host, high availability can and is a pain. monitoring your database is a pain, it takes a huge amount of time, which most companies do not have.
p.s. working at a small company and services like RDS/gcloud postgres! (thanks google) google buckets (not sure how they are called their name is not as cool as s3)/S3 are a huge win for us.
[+] [-] apendleton|8 years ago|reply
* elastic demand, either in the sense that your global traffic has peaks and troughs, or your global traffic is steady but its global distribution changes such that the balance of where you're deployed would ideally change over time; in either case you can benefit enormously from automatic scaling
* workloads that can take advantage of the spot market (or whatever GCP's equivalent is)
[+] [-] Thaxll|8 years ago|reply
[+] [-] manigandham|8 years ago|reply
It's easy to say you can buy a huge server for a few thousand but the "cloud" provides incredible agility, security, and basically removes operational overhead. When compared to buying, maintaining, and decommissioning servers and software along with the time and personnel costs, it becomes a much closer race. Include global scale, elastic capacity, managed services, and support policies, and it's closer still.
Rarely are companies paying 3x IT spend just to run in the cloud, especially for large companies like Etsy.
[+] [-] dboreham|8 years ago|reply
I think you can imagine a scenario under which the end goal to have this headline was someone’s primary goal for the past couple of years.
[+] [-] gregmac|8 years ago|reply
> The company performed big data jobs early on in the cloud, using Amazon's Elastic Map Reduce service.
> “That [EMR] got us to a place where we think about what data we collect. At the time it was really inefficient from a cost perspective, but there weren’t any creative constraints,”
> The team eventually decided to bring it in house. “Our Hadoop cluster wields a huge amount of flexibility," said Rembetsy. "We can get a lot done on bare metal, and we know very much what we’re looking for. Cloud services, that’s where efficiencies fall away. Spin up and spin down doesn’t have the same value proposition.”
[1] http://www.datacenterknowledge.com/archives/2015/05/11/how-e...
[+] [-] interskh|8 years ago|reply
[+] [-] code_duck|8 years ago|reply
They have always seemed a bit disconnected from their customers. The first regime, Rob Kalin, Haim and Revolving Dork, had an age and culture disconnect from most of their customers. The next group, headed by Chad Dickerson and Allspaw, seemed like they wanted to focus on technology for technology’s sake, more than what Etsy was actually doing with that as a company. They even made some statements to the effect of how they wanted Etsy to be known as a technology company, like Facebook, and be a platform, like Amazon.
All fine and good, but so far, what all of the leaders of Etsy have had in common is that they don’t really understand what it’s like to be a micro business person selling handmade art online. If I was an investor, I wouldn’t want to be paying for a bunch of guys to experiment with Mongo DB search engines and shards. As long as they’re still not understanding their core business, it’s a distraction.
Etsy should be focused on one thing: connecting buyers and sellers of handmade art with each other.
[+] [-] BlanketApple|8 years ago|reply
> The next group, headed by Chad Dickerson and Allspaw, seemed like they wanted to focus on technology for technology’s sake, more than what Etsy was actually doing with that as a company.
I think you're missing quite a bit of this story. I was around during the transition of Chad from CTO -> CEO, and the truth is that Etsy needed to focus on the technology badly during that time. The site would go down constantly. We had very little insight into what was going on with sellers or buyers. Every busy day was basically us crossing our fingers that the site wouldn't go down.
We needed someone focused heavily on the technology. We could never have continued on that path and grown as much as we did.
> Etsy should be focused on one thing: connecting buyers and sellers of handmade art with each other.
Deciding what potential buyers are interested in and what items to display to them is a pretty difficult technology problem.
Detecting mass-produced goods in a way that's more subtle than "ban all Chinese sellers" is a pretty difficult technology problem.
If anything I would argue that Etsy didn't focus nearly enough on that, and focused way too much on "low-hanging fruit" (minor workflow adjustments for sellers and buyers).
[+] [-] stareatgoats|8 years ago|reply
[+] [-] josephhurtado|8 years ago|reply
Could it be a price break they got, and also the AI advantages of Google Cloud's offerings? I am sure that played a big part.
[+] [-] bckygldstn|8 years ago|reply
Source: work for a retail company, we went with Azure.
[+] [-] oh-kumudo|8 years ago|reply
I don't think Etsy should consider itself as a competitor to Amazon, they simply can't. On the other side, AWS already has so many success stories that they are unlikely cater to Etsy specifically. That makes GCE the best option out there. As to Azure, I think people trust Google more as to cloud-based technology than Microsoft.
[+] [-] Analemma_|8 years ago|reply
[+] [-] synicalx|8 years ago|reply
[+] [-] codingdave|8 years ago|reply
I have no idea what Etsy looks like internally, but that sounds like an awfully long time in this day and age. Anyone have an inkling as to why this is a multi-year project?
[+] [-] sitharus|8 years ago|reply
[+] [-] gregmac|8 years ago|reply
[+] [-] sjeanpierre|8 years ago|reply
[+] [-] tty7|8 years ago|reply
[+] [-] clintonb|8 years ago|reply
[+] [-] whitepoplar|8 years ago|reply
[+] [-] unknown|8 years ago|reply
[deleted]
[+] [-] _Codemonkeyism|8 years ago|reply
We were a 6M customer flash sale company sold for $200M. We had 3 devops engineers mostly helping developers with designing the system and pager duty - mainly because 24/7 was to demanding for only 2 people. We were SOX/PCI compliant.
Hosting costs all in all by year were around the same as the salaries I'd say.
People who moved to AWS/GCP, would I save costs in this scenario moving to AWS/GCP?
[+] [-] vpol|8 years ago|reply
[+] [-] Someone1234|8 years ago|reply
Etsy seems to be fine for the most part. The only criticism I have of Etsy is that they've moved away or diluted their core business (small business/home business sales) by allowing mass manufactured goods to be sold on the site.
Etsy still has a place, and I've ordered several items from Etsy vendors this last year. They just need to keep a check on what they want to be otherwise they'll be subpar eBay clone.
[+] [-] cracell|8 years ago|reply
[+] [-] damm|8 years ago|reply
Etsy has been hurting; why would they move infrastructure now unless t hey got kind of kickback for the PR?
This isn't an easy move; so there has to be a huge benefit to them.
[+] [-] unknown|8 years ago|reply
[deleted]
[+] [-] CaptSpify|8 years ago|reply
It's like watching a play in 3 acts:
Act 1) Yay, we did it, we're gonna save so much money!
Act 2) [a year later] Wait, we're spending how much money?!?
Act 3) It's going to cost us how much to leave our current cloud?!?
Haven't people figured out yet that cloud companies make a profit? Sure, companies like Etsy can get a huge discount, but everyone else who isn't a big name is going to pay out the ass.
[+] [-] btian|8 years ago|reply
Boeing makes a profit, but how much money will it cost to design and make a plane from scratch?
[+] [-] merb|8 years ago|reply
[+] [-] illumin8|8 years ago|reply
[+] [-] ktta|8 years ago|reply