37signals spends more than $3M a year on cloud. So while it definitely isn't a major enterprise. It is also a lot more than a a few dozen servers.
I am not anti-cloud and pro cloud. My major problem with the new trend is that a lot of people are basically rediscovering pre "Cloud" era. which is VPS, Dedicated server and Colocation. And people are suggesting Hetzner or OVH or many other players are equivalent to AWS. While I dont disagree AWS is charging a lot for their offering, putting AWS to other services isn't even a valid comparison.
Completely ignoring the basics such as Server / CPU / RAM / SSD quality. Network quality such as interconnect, redundancy, as well as Data Center quality. If you rally want to do simple price and spec comparison you might as well go to Lowendbox to find a low cost VPS which some people have been doing since 2008.
I really wish there is a middle ground somewhere before using Hyperscalers. Both DO / Linode couldn't reach a larger scale. Hetzner is expanding their Cloud offering only and no dedicated outside EU.
How much of that is what technologists would consider "cloud" (IAAS, PAAS) versus what someone on the business side of things would consider "cloud" - office365, google gsuite, etc?
I sincerely doubt 37 signals has "a few dozen servers." Every company I've been in has a huge, sprawling cloud and that no one has governance over. New instances are stood up by individual teams in order to avoid bureaucratic delay, and these propagate indefinitely.
"In parallel, GEICO, one of the largest automotive insurers in the United States, is actively repatriating many workloads from the cloud as part of a comprehensive architectural overhaul."
> Their Cloud exodus can't have been more than a few dozen servers, right?
"At the moment we have somewhere between 20-25 servers in each cab, or about 90 servers in each site. Here’s what the rack layout looks like in Chicago, for instance."
aws and other hyperscalers will keep growing, no doubt. Public cloud adoption is at around 20%. So the new companies that migrate into the cloud will keep the growth going. That doesn't deny the fact that some might be repatriating though. Especially ones that couldn't get the benefits out of the cloud.
> That doesn't seem compatible with any kind of major cloud repatriation trend.
Agreed. I don't think this is a real trend, at least not right now.
Also, fwiw, I'm really not a fan of these types of articles that identify like a small handful of people or organizations doing something different and calling it a "trend".
GEICO is moving away from the cloud because their IT is a joke. They had a horrible on-prem infrastructure, so they moved to the cloud not knowing how, and they made the same mistakes in the cloud as on-prem, plus the usual mistakes every cloud migration runs into. They are moving away from the cloud because their new VP's entire career is focused on running her own hardware. What we know about their new setup is absolutely bonkers (like, K8s-on-OpenStack-on-K8s bonkers). Look to them for what not to do.
37signals is like the poster child for NIH syndrome. They keep touting cost savings as the reason for the move, but from what I have gathered, they basically did nothing to save cost in the cloud. It is trivial to save 75% off AWS's list price. They will even walk you through it, they literally want you to save money. That, plus using specific tech in specific ways, allows you to reap major benefits of modern designs while reducing cost more. 37signals didn't seem to want to go that route. But they do love to build their own things, so servers would be a natural thing for them to DIY.
Almost every argument against the cloud - cost inefficiency, fear of vendor lock-in, etc - has easy solutions that make the whole thing extremely cost competitive, if not a way better value, than trying to become your own cloud hosting provider. It's very hard to estimate the real world costs, both known and unknown, of DIY hosting (specifically the expertise, or lack of it, and the impacts from doing it wrong, which is very likely to happen if cloud hosting isn't your core business). But it's a 100% guarantee that you will never do it better than AWS.
AI is the only place I could reasonably imagine somebody having an on-prem advantage. At the moment, we still live in a world where that hardware isn't a commodity in the way every other server is. So you might just be faster to deploy, or cheaper to buy, with AI gear. Storage is similar but not nearly as tight a market. But that will change eventually once either the hype bubble bursts, or there's more gear for cheaper for the cloud providers to sell.
> Almost every argument against the cloud - cost inefficiency, fear of vendor lock-in, etc - has easy solutions that make the whole thing extremely cost competitive, if not a way better value, than trying to become your own cloud hosting provider. It's very hard to estimate the real world costs, both known and unknown, of DIY hosting (specifically the expertise, or lack of it, and the impacts from doing it wrong, which is very likely to happen if cloud hosting isn't your core business)
Please define your concept of self-hosting here. Does it mean you need to have your very own DC? Renting a few racks that you fill yourself? Rent CPU, storage and networking, with remote hands and all the bells and whistles?
Depending on the scenario it changes dramatically the burden of ownership (at a monetary cost, obviously). And depending on the size of the company and the variability of the workload, it can (or can not) make sense to be on-prem. But being like "cloud is perfect for everyone and everything, if you tune it well enough" seems a bit too much black&white to me.
It's very easy to estimate the real-world cost of on-prem or dedicated hosting - there is a wide range of providers that will quote you fixed monthly prices to manage it for you (including me) because we know what it costs us to manage various things for you.
AI is the only place I don't currently see much on-prem advantage, because buying SOTA equipment is hard, and it gets outdated too quickly.
For pretty much everything else, if you can't save 70%+ TCO, maintenance/devops included, over an optimized cloud setup, you're usually doing something very wrong, usually because the system is designed by someone who defaults to "cloud assumptions" (slow cores, too little RAM, too little fast storage, resulting in systems that are far more distributed than they need be is the typical issue).
The main problem with AWS is their outrageous pricing on some aspects like traffic. And some very unexpected pricing nuances which could burn thousands of dollars in a blink of an eye.
While AWS engineers are more competent, may be you don't need that much competency to run simple server or two. And expense structure will be more predictable.
I know nothing about Geico's IT but I find your comments surprising. GEICO is one of the most profitable insurance companies in the world which, of course, is the end goal of every company.
> Weekly explains that “just running legacy applications in the cloud is prohibitively expensive,” highlighting how lift-and-shift approaches often fail to deliver expected benefits.
Yes, if you have a mature business without active development at a scale where compute/storage costs is a substantial accounting line item, then it makes sense to run on hardware that doesn't have the flexibility and cost of the cloud.
There is an in-between that makes much more sense for most though. Running on provisioned bare metal. Lots of providers offer this as a better performance/price option where you don't have to deal with provisioning hardware but do everything else from the OS+maintenance and up.
At one company we used large bare-metal machine instances provisioned for stable parts of the application architecture (e.g. database and webapp instances) and the cloud for new development where it made sense to leverage capabilities, e.g. DynamoDB with cross-region replication.
I can't tell you how often I've run into cloud deployments that were lift-and-shifts, pushed on by bean counters wanting OPEX instead of CAPEX. They then run into actual cashflow expenses, less stability, more complex security (now you get IAM on top of basic networking!), and the ability for one underpaid person to easily do a lot of damage - because you're certainly not going to hire top-tier cloud talent - these are bean counters running things after all.
It makes it really clear why you so many data leaks via badly configured s3 buckets of dynamo tables...
Very large mature businesses that don’t see IT as a core function have probably outsourced management to a third party. There’s not much daylight between that third party’s margin and just paying a hyperscaler.
There are certain workloads that have never been really economical to run in cloud. Cloud economics is based on multi-tenancy, eg if you have a lot of hardware that is sitting idle a lot of the time, then cloud may be economical for you as the cloud provider can share it between you and others.
Cloud is also good for episodic use of expensive exotic systems like HPC and GPU fleets, if you don’t need them all the time- I call this serial multi-tenancy.
Cloud is not economical for massive storage, especially if you’re not willing to use backup solutions and reduced availability. For example, AWS S3 default keeps multiple copies of uploaded data; this is not comparable to typical on-premises RAID 1 or RAID 3. You can save money with reduced redundancy storage but then you have to take on more of the reliability burden. Likewise compute is cheap if you’re buying multi-tenant instances, but if you want dedicated instances or bare metal, then the economics aren’t nearly as attractive.
Cloud is also good for experimentation and rapid development - it’s so much faster to click a few buttons than to go through the hardware acquisition processes at many enterprises.
The companies that regret cloud due to financial concerns usually make two mistakes.
First, as noted above, they pay for premium services that are not directly comparable to on-prem, or they use workloads in cloud that are not cloud economical, or both.
Second, they don’t constrain random usage enough. It is super easy for a developer doing some testing to spin up thousands of dollars of bill. And it’s even worse if they leave it at the end of the day and go home- it’s still racking up hourly usage. And it’s downright ugly if they forget it and move on to something else. You have to be super disciplined to not spin up more than you need and turn it off as soon as you’re done with it.
What I was surprised to find in some big orgs is the processes have not evolved to be cloud first. There is lack of maturity, still a chain of committees, approvals, and manual processes; risk management still treats the services as a giant intranet, deployments are not scripted, ad hoc designs. Resources are placed in vnets so that they resemble a system they already know, and comes with all the associated risks.
I've never worked in a data center that did cooling and power correctly. Everyone thinks they're doing it right, and then street power gets cut - there's significant impact, ops teams scramble to contain, and finally there's the finger-pointing.
It’s the same old MBA cycle we had with onshoring / offshoring. Everyone wants to build their resume so they have to change things.
In this cycle a new MBA comes in wants to make an impact so does a cloud transition. Then they move on and the next guy comes in, wants to make an impact so moves things back in house. Repeat until some new fad comes along.
You can have a 100Gb uplink on a dedicated fibre for less than 1000$/month now. Which is insanely less than cloud bandwidth. Of course there are tons of other costs, but that alone can suffice to justify moving out of the cloud for bandwidth intensive app.
We went to cloud because 1) we only need 3 infra guys to run our entire platform and 2) we can trivially scale up or down as needed. The first saves us hundreds of thousands in skilled labor and the second lets us take on new customers with thousands of agents in a matter of days without having to provision in advance.
Just curious where do you get 100Gb with Internet transit and dedicated fiber for 1000$/month? I'm in a small town in eastern Germany and looked for a simple Gigabit fiber access for our office without any bandwidth guarantees and it's 1000€/month for 1Gb here with the most budget provider but with some nebulous bandwidth guarantees. I'm not talking about residential fiber that also very expensive after a certain threshold. I know there is init7 in Switzerland but it's the exception to the rule in Europe it seems. Getting a fast fiber and good transit is still expensive?
Chat, feeds and moderation run on AWS for us. Video on the other hand is bandwidth intensive. So we run the coordinator infra on AWS, but the SFU edge network on many different providers.
I think the cloud is good for some things, and not so great for others. S3 is fairly cost effective. RDS is expensive, bandwidth is crazy etc.
1. Egress cost. Cloud hosting providers have absolutely insane egress pricing. It's beyond stupid at this point, if you want to host anything bandwidth-intensive.
It doesn't seem to say in the article and it's not really discussed in these "LEAVING THE CLOUDS!!" articles, but what are these orgs doing for on-prem? Given the broadcom acquisition of vmware, rebuilding massive vsphere clusters like it's 2010 doesn't seem like a good long term play. Are they moving to kubernetes? Some other hypervisor?
At least in the case of 37signals, they went with colocated servers, some type of KVM and their own tool, Kamal, for containerized deployments without the complexity of kubernetes.
You can find one post here with many links at the bottom
Well, major companies aren't ditching the cloud and there is no evidence for a trend otherwise. And 37signals isn't a major organization for any of the big cloud providers. They are just a rounding error.
* AWS segment sales increased 19% year-over-year to $27.5 billion.
That means AWS brought in $4.3 BILLION more dollars in Q3 2024 vs 2023.
That's a huge amount of incremental revenue growth. If the net movement of workloads were out of the cloud, then it would have to show up in the results of Intel / TSMC / Equinix et. al.
I just took a look, and Equinix quarterly revenue is $2.1B.
Almost any story about cloud repatriation is a story about a failure of the market to act competitively rather than someone actually able to do it for less money than the cloud providers can.
The big providers margins are crazy, like over 50% which is normal for a software / service business but they are essentially hardware businesses.
I think control is maybe a bigger factor than cost these days. Being able to hold anyone accountable at all seems to be an operational superpower. Working with cloud vendor support is a torturous experience on a good day. It also doesn't matter how expensive the virtual machine is if there isn't one available to be provisioned.
I know it's kind of harsh, but owning the whole vertical and having the power to instantly fire anyone for giving an Azure-tier response is why these companies are doing it in my mind. Waiting on a 3rd party to find their own ass with a whole S&R team every time you need help is quite exhausting. I've never worked with an IT vendor and thought "damn these people are so responsive I can't dream of doing it better myself".
I work for a small orgthat is owned by a very large corp. Our spending is managed by large corp.
If I want to buy a $10 domain, the process takes a month and requires escalating to a purchasing director. If I want to rent a new server from hetzner, same thing.
If I want to spin up a bedrock instance for $1000/day on AWS - it’s already a line item in the budget so as long as I have a cost tag on the resource it’s pre-approved. As long as something is on the software catalog on AWS it’s ok to use.
> “Ten years into that journey, GEICO still hadn’t migrated everything to the cloud, their bills went up 2.5x, and their reliability challenges went up quite a lot too.”
yes this would make cloud cost a lot without any of the benefits lol
They will want cloud-like APIs on-premises and most will implement OpenStack. The second wave of migrations to the cloud will be even quicker for these companies making their way back to on premises.
Recently, i've come to realize one real use of those clouds was to provide a good US-EU network connection. If you want to provide both continent users with correct bandwidth to your service, you have no choice but to have them connect to a datacenter on their own continent. Public data transit across the atlantic is simply miserable.
Then, because they probably have private atlantic cables, you can replicate at good reliable speed.
> "While the cloud repatriation debate often focuses on the binary choice between cloud and on-premises, there’s a third path emerging: staying in the cloud while embracing alternative technologies and architectures."
I think the potential of using "less managed" with K8s etc. rather than fully self-managed is relatively untapped.
[+] [-] jsnell|1 year ago|reply
Meanwhile AWS is growing at 20%/year, Azure at 33% and GCP at 35%. That doesn't seem compatible with any kind of major cloud repatriation trend.
[+] [-] ksec|1 year ago|reply
I am not anti-cloud and pro cloud. My major problem with the new trend is that a lot of people are basically rediscovering pre "Cloud" era. which is VPS, Dedicated server and Colocation. And people are suggesting Hetzner or OVH or many other players are equivalent to AWS. While I dont disagree AWS is charging a lot for their offering, putting AWS to other services isn't even a valid comparison.
Completely ignoring the basics such as Server / CPU / RAM / SSD quality. Network quality such as interconnect, redundancy, as well as Data Center quality. If you rally want to do simple price and spec comparison you might as well go to Lowendbox to find a low cost VPS which some people have been doing since 2008.
I really wish there is a middle ground somewhere before using Hyperscalers. Both DO / Linode couldn't reach a larger scale. Hetzner is expanding their Cloud offering only and no dedicated outside EU.
[+] [-] PittleyDunkin|1 year ago|reply
[+] [-] WaxProlix|1 year ago|reply
[+] [-] everdrive|1 year ago|reply
[+] [-] 1vuio0pswjnm7|1 year ago|reply
Is GEICO a major enterprise
[+] [-] leftcenterright|1 year ago|reply
"At the moment we have somewhere between 20-25 servers in each cab, or about 90 servers in each site. Here’s what the rack layout looks like in Chicago, for instance."
- https://dev.37signals.com/37signals-datacenter-overview/
Their "server count" is definitely much higher than what you are thinking.
[+] [-] panrobo|1 year ago|reply
[+] [-] joshdavham|1 year ago|reply
Agreed. I don't think this is a real trend, at least not right now.
Also, fwiw, I'm really not a fan of these types of articles that identify like a small handful of people or organizations doing something different and calling it a "trend".
[+] [-] andrewstuart|1 year ago|reply
Revolutions cannot start huge, they must start small.
[+] [-] hulitu|1 year ago|reply
Forcing people to use OneDrive surely has an effect. /s
[+] [-] 0xbadcafebee|1 year ago|reply
37signals is like the poster child for NIH syndrome. They keep touting cost savings as the reason for the move, but from what I have gathered, they basically did nothing to save cost in the cloud. It is trivial to save 75% off AWS's list price. They will even walk you through it, they literally want you to save money. That, plus using specific tech in specific ways, allows you to reap major benefits of modern designs while reducing cost more. 37signals didn't seem to want to go that route. But they do love to build their own things, so servers would be a natural thing for them to DIY.
Almost every argument against the cloud - cost inefficiency, fear of vendor lock-in, etc - has easy solutions that make the whole thing extremely cost competitive, if not a way better value, than trying to become your own cloud hosting provider. It's very hard to estimate the real world costs, both known and unknown, of DIY hosting (specifically the expertise, or lack of it, and the impacts from doing it wrong, which is very likely to happen if cloud hosting isn't your core business). But it's a 100% guarantee that you will never do it better than AWS.
AI is the only place I could reasonably imagine somebody having an on-prem advantage. At the moment, we still live in a world where that hardware isn't a commodity in the way every other server is. So you might just be faster to deploy, or cheaper to buy, with AI gear. Storage is similar but not nearly as tight a market. But that will change eventually once either the hype bubble bursts, or there's more gear for cheaper for the cloud providers to sell.
[+] [-] darkwater|1 year ago|reply
Please define your concept of self-hosting here. Does it mean you need to have your very own DC? Renting a few racks that you fill yourself? Rent CPU, storage and networking, with remote hands and all the bells and whistles? Depending on the scenario it changes dramatically the burden of ownership (at a monetary cost, obviously). And depending on the size of the company and the variability of the workload, it can (or can not) make sense to be on-prem. But being like "cloud is perfect for everyone and everything, if you tune it well enough" seems a bit too much black&white to me.
[+] [-] vidarh|1 year ago|reply
AI is the only place I don't currently see much on-prem advantage, because buying SOTA equipment is hard, and it gets outdated too quickly.
For pretty much everything else, if you can't save 70%+ TCO, maintenance/devops included, over an optimized cloud setup, you're usually doing something very wrong, usually because the system is designed by someone who defaults to "cloud assumptions" (slow cores, too little RAM, too little fast storage, resulting in systems that are far more distributed than they need be is the typical issue).
[+] [-] vbezhenar|1 year ago|reply
While AWS engineers are more competent, may be you don't need that much competency to run simple server or two. And expense structure will be more predictable.
[+] [-] cdchn|1 year ago|reply
Do what now???
[+] [-] keernan|1 year ago|reply
[+] [-] karmakaze|1 year ago|reply
> Weekly explains that “just running legacy applications in the cloud is prohibitively expensive,” highlighting how lift-and-shift approaches often fail to deliver expected benefits.
Yes, if you have a mature business without active development at a scale where compute/storage costs is a substantial accounting line item, then it makes sense to run on hardware that doesn't have the flexibility and cost of the cloud.
There is an in-between that makes much more sense for most though. Running on provisioned bare metal. Lots of providers offer this as a better performance/price option where you don't have to deal with provisioning hardware but do everything else from the OS+maintenance and up.
At one company we used large bare-metal machine instances provisioned for stable parts of the application architecture (e.g. database and webapp instances) and the cloud for new development where it made sense to leverage capabilities, e.g. DynamoDB with cross-region replication.
[+] [-] hylaride|1 year ago|reply
It makes it really clear why you so many data leaks via badly configured s3 buckets of dynamo tables...
[+] [-] coredog64|1 year ago|reply
[+] [-] efitz|1 year ago|reply
Cloud is also good for episodic use of expensive exotic systems like HPC and GPU fleets, if you don’t need them all the time- I call this serial multi-tenancy.
Cloud is not economical for massive storage, especially if you’re not willing to use backup solutions and reduced availability. For example, AWS S3 default keeps multiple copies of uploaded data; this is not comparable to typical on-premises RAID 1 or RAID 3. You can save money with reduced redundancy storage but then you have to take on more of the reliability burden. Likewise compute is cheap if you’re buying multi-tenant instances, but if you want dedicated instances or bare metal, then the economics aren’t nearly as attractive.
Cloud is also good for experimentation and rapid development - it’s so much faster to click a few buttons than to go through the hardware acquisition processes at many enterprises.
The companies that regret cloud due to financial concerns usually make two mistakes.
First, as noted above, they pay for premium services that are not directly comparable to on-prem, or they use workloads in cloud that are not cloud economical, or both.
Second, they don’t constrain random usage enough. It is super easy for a developer doing some testing to spin up thousands of dollars of bill. And it’s even worse if they leave it at the end of the day and go home- it’s still racking up hourly usage. And it’s downright ugly if they forget it and move on to something else. You have to be super disciplined to not spin up more than you need and turn it off as soon as you’re done with it.
[+] [-] teyc|1 year ago|reply
[+] [-] kjellsbells|1 year ago|reply
We seem to have replaced cooling and power and a grumpy sysadmin with storage and architects and unhappy developers.
[+] [-] jimt1234|1 year ago|reply
[+] [-] gtirloni|1 year ago|reply
[+] [-] badgersnake|1 year ago|reply
In this cycle a new MBA comes in wants to make an impact so does a cloud transition. Then they move on and the next guy comes in, wants to make an impact so moves things back in house. Repeat until some new fad comes along.
[+] [-] kuon|1 year ago|reply
[+] [-] Salgat|1 year ago|reply
[+] [-] nisa|1 year ago|reply
[+] [-] gwbas1c|1 year ago|reply
[+] [-] ttt3ts|1 year ago|reply
[+] [-] tschellenbach|1 year ago|reply
I think the cloud is good for some things, and not so great for others. S3 is fairly cost effective. RDS is expensive, bandwidth is crazy etc.
(5M a year spend on AWS atm.)
[+] [-] cyberax|1 year ago|reply
In my experience, it comes down to two factors:
1. Egress cost. Cloud hosting providers have absolutely insane egress pricing. It's beyond stupid at this point, if you want to host anything bandwidth-intensive.
2. Storage pricing.
[+] [-] denkmoon|1 year ago|reply
[+] [-] weikju|1 year ago|reply
You can find one post here with many links at the bottom
https://basecamp.com/cloud-exit
[+] [-] lemme_tell_ya|1 year ago|reply
[+] [-] siva7|1 year ago|reply
[+] [-] vidarh|1 year ago|reply
Even at 37 signals size you're paying negotiated rates.
And 37 signals may not be a "major" organization to you, but they're bigger than the vast majority of companies.
[+] [-] discodave|1 year ago|reply
* AWS segment sales increased 19% year-over-year to $27.5 billion.
That means AWS brought in $4.3 BILLION more dollars in Q3 2024 vs 2023.
That's a huge amount of incremental revenue growth. If the net movement of workloads were out of the cloud, then it would have to show up in the results of Intel / TSMC / Equinix et. al.
I just took a look, and Equinix quarterly revenue is $2.1B.
[+] [-] hackit2|1 year ago|reply
[+] [-] deely3|1 year ago|reply
[+] [-] theginger|1 year ago|reply
[+] [-] chromanoid|1 year ago|reply
I can see how AI workloads makes clouds look expensive.
[+] [-] bob1029|1 year ago|reply
I know it's kind of harsh, but owning the whole vertical and having the power to instantly fire anyone for giving an Azure-tier response is why these companies are doing it in my mind. Waiting on a 3rd party to find their own ass with a whole S&R team every time you need help is quite exhausting. I've never worked with an IT vendor and thought "damn these people are so responsive I can't dream of doing it better myself".
[+] [-] maccard|1 year ago|reply
If I want to buy a $10 domain, the process takes a month and requires escalating to a purchasing director. If I want to rent a new server from hetzner, same thing.
If I want to spin up a bedrock instance for $1000/day on AWS - it’s already a line item in the budget so as long as I have a cost tag on the resource it’s pre-approved. As long as something is on the software catalog on AWS it’s ok to use.
[+] [-] michaelt|1 year ago|reply
But remember even when you're doing everything on-prem with your own employees, you're still running software written by third parties.
So you might still have an unresponsive third party, just they might be a database vendor instead of a cloud vendor.
[+] [-] asdasdsddd|1 year ago|reply
yes this would make cloud cost a lot without any of the benefits lol
[+] [-] gtirloni|1 year ago|reply
[+] [-] bsaul|1 year ago|reply
Then, because they probably have private atlantic cables, you can replicate at good reliable speed.
[+] [-] matt_johnston|1 year ago|reply
I think the potential of using "less managed" with K8s etc. rather than fully self-managed is relatively untapped.