[danols]

I don't understand how it is possible in a functioning market economy to have 8000% markup on a utility product like bandwidth. Seems more like an oligopoly where a few big actors with a big moat has agreed to keep prices inflated. How can a sustained 8000% markup for a product like bandwidth not be considered price fixing?

source: https://blog.cloudflare.com/aws-egregious-egress/

[amluto]

Because AWS doesn’t actually want to collect the egress fees. They want you to avoid egress entirely for high-value services, which has all kinds of excellent implications for AWS:

Third party SaaS offerings that move large amounts of data are effectively forced to host in AWS.

Want a small number of high-value servers (e.g. big GPUs, etc) in your own data center or colo to use for non-availability-critical purposes integrated with the rest of your AWS stack? You’d better price in egress!

Want to gradually transition to a competing cloud? Good luck, egress will bankrupt you before you finish the transition.

[StopHammoTime]

amluto is spot on. AWS don't want to make money off of egress, they want to make it non-viable to move anything out of AWS that isn't being served to an end consumer.

[torginus]

I wonder, if you made a video streaming app on top of AWS you could negotiate a significant reduction of fees, since you're not ruining their business by transferring out valuable data to competitors. And if that's the case, I wonder if that constitutes a breach of net neutrality in practice, since in practice, a major part of bandwidth costs will depend on what kind of information you are transferring.

[dwild]

> Want to gradually transition to a competing cloud? Good luck, egress will bankrupt you before you finish the transition.

They do offer the Snowball to lower transfer out cost, they also offer Direct Connect which is more convenient and can be cheaper.

For sure it won't be cheap, but being able to transfer 100 TB for less than 3k isn't too bad. Storing that on S3 would cost you nearly that per month... if you can't spare a month of cost to migrate, you may want to rethink your pricing.

[colordrops]

That still sounds like price fixing, but for a different purpose.

[netwo233gur]

The Cloudflare blog post really only looks at wholesale cost of bandwidth and compares it to the price AWS charges. But I think it's missing a huge component of all of the magic that happens inside AWS between those two things.

I've seen some of the inner workings of the big cloud providers' networking stacks. The networking infrastructure, the software that runs it, the software that exposes it to customers, the thousands of engineers working at any given moment in AWS/GCP/Azure's NOCs to maintain uptime are truly some of the most impressive technical marvels I have ever seen. They aren't as sexy to discuss on HN as something like the managed containers services, functions as a service, EC2 etc, but the networking stacks like the VPC, NAT gateways, subnet routing, privatelinks, security groups, ENIs, nitro cards, etc are pure magic as far as I'm concerned and are so so so much more complicated than a standard data center's networking stack, or even Cloudflare's stack.

To use Cloudflare's "bucket of water" metaphor, AWS isn't even close to just being a dumb bucket of water that you fill with water and then get charged to take out the water. There is so much that happens inside of that bucket to segment your water into different pipes, routing your water in all kinds of customer-customizable ways for many different use cases, mixing/heating/cooling your water as you need, all while guaranteeing things like making sure your water arrives exactly where it is supposed to arrive and doesn't get contaminated or leaked along the way.

Does AWS make a big markup on bandwidth? Yea, surely they do. But is it as simple as Cloudflare says it is? Not even close.

[lewisl9029]

Yes, their network stacks are definitely complex and cost a lot to maintain, I'm sure. But that doesn't necessarily make it a good deal if the customer isn't able to derive enough additional value from all that complexity. In fact it makes the offering less attractive if the complexity isn't sufficiently abstracted away and distract from product work or if their abstractions are leaky.

Recently I've been working with https://fly.io/ for a new app and it's a breath of fresh air compared to working with the big cloud providers. They offer simple but robust networking primitives built on top of ipv6 and WireGuard and provide a ton of value add on top like global distribution & load balancing, service discovery, TLS termination, all of which just work exactly like I'd expect it to, out of the box without any configuration on my side.

EDIT: Almost forgot to mention: their egress costs are also much more reasonable: https://fly.io/docs/about/pricing/#outbound-data-transfer

[mythz]

https://www.hetzner.com/cloud gives you 20TB bandwidth for €3.49/mo VMs, which I've essentially regarded as Hetzner gives unlimited free bandwidth for all servers.

Being lynched for egregious egress fees is only something I've experienced when using mega corp's clouds, where economies of scale suggests their vastly larger size should allow them to provide even better value.

But that's in a normal market, not the artificial lock-in mega cloud corps enjoy where they're able to distort customer behavior from artificially high pricing.

[starfallg]

It's not really that much magic. It's just a variation of EVPN-VXLAN plus smart NICs that segments and directs the traffic. Then they have normal VM hosts or nowadays devices with ASICs that handle the GW and NAT functionality.

[VectorLock]

It's weird to me how people think contrasting a raw pipe billed on 95th percentile to a service like S3 or Cloudflare is in any way a fair comparison.

[adam_arthur]

The thing about a competitive market is that it requires that cost of switching is low. The higher the cost of switching, the more leverage the company has over you.

SaaS and cloud providers in particular have a lot of leverage over their customers due to the typically high cost of switching. E.g. what does it cost to move your whole infrastructure to a new cloud? They can basically gouge you up to that cost threshold.

These cost of switching actually leads to a market that is not competitive in the traditional sense. Yes, competition on the surface, but monopoly within.

It's true that gouging can't be as egregious as a true monopoly environment, but certainly much higher than a low cost of switching environment.

The big tell is margins. Anytime a company can sustain excessively high margins, it's usually a tell that theres a lack of a competitive market. A lot of software companies get high margins either by being first to market (by many years), or leveraging high cost of switching.

Despite software being cheap to deliver, it's also cheap for your competitors to deliver, so margins should be low in a highly competitive market, regardless of marginal cost of production.

I do believe new regulation is needed to handle these business models such that we can continue to foster a competitive environment. It would have to be very carefully crafted to prevent unintended second order effects of course.

[missedthecue]

Because bandwidth isn't the only AWS product, but you can't buy bandwidth for cheap from XYZ Inc. and connect that to your AWS account. That's not technically possible, you have to buy it from AWS.

So instead of using the cheapest vendor, companies use AWS for a ton of reasons (broad support, everyone knows how to work with it so it makes recruiting easier, hundreds of features, etc...) and their high bandwidth bill is worth it.

[spoonjim]

AWS is a luxury product and the outbound charge is the delivery fee. You will also find that the delivery fee for a Louis Vuitton bag is more than for a pizza.

[tw04]

AWS spent the first decade of existence advertising to executives as a way to significantly decrease their cost of IT, not as the Louis Vuitton of IT.

[jollybean]

No, it's a very valuable tool for corporate IT.

The egress is a way to snare a lot of extra margin.

It's 100% clear that orgs. wanting to host large public platforms will have 'major concerns' over this pricing issue but that's not their target market.

I'll bet most HNeers are thinking in terms of 'Hosting my App' there vs. AWS Bread and Butter is mostly hosting corporate IT services, which is a different thing.

[butMyside]

So it is:

https://www.nytimes.com/2020/10/25/technology/apple-google-s...

I mean the politicians are in on it, and turn against them when their ability to control elections for their own purposes comes into question.

Look at the “testimony” of Robinhood guy. It’s a scripted TV show; you lose.

[bluedino]

How much margin do you think there is in coffee or soft drinks?

[rad_gruchalski]

Apparently nowhere near the 8000%. After https://www.profitableventure.com/income-soda-retailers-marg...:

> Restaurants and bars have around a 70% profit margin on a bottle of soda and soft drink, while retailers typically have between 30–50%.

[Aicy]

This is like going to a supermarket and complaining that the plastic bags for you to store your shopping in cost 10 cents which is a 8000% markup, as if plastic bags are why you're in a supermarket

[georgyo]

This is a horrible example that doesn't not apply cleanly.

To fix it, let's change it from buy per bag to per gram you want to leave the store.

So your canned soup costs at least 3 times more to leave the supermarket then in cost to buy it than to eat it in the supermarket.

This is not even an exaggeration. s3 is 0.024/GB and egress bandwidth is 0.09/GB

[jollybean]

TOC 'Total Cost of Ownership'.

Nobody is paying for egress, they are paying for everything else.

AWS is focused on corporate IT so egress was always a secondary thing for them.

It might actually be more profitable for them to have uber-markups on that egress then to serve the web space that requires cheap bandwidth.

[VectorLock]

Its not 8000% on a markup on raw metered bandwidth, they're serving data from a service they operate. Comparing acting like they're just "marking up" bandwidth is an unfair comparison.

[throwaway984393]

What do you mean, how is it possible? That's how markets work. You open a booth in the market, and you can ask any price you want. You can ask for a nickle or a million bucks. There's no rule about how much money you can ask for.

As for your assertion that there's a price fixing conspiracy, who do you imagine they're colluding with? What other company do you find charging these prices?

[adventured]

The parent comment misunderstands that most of Amazon's customers are very clearly not extremely concerned about the bandwidth costs (even if they'd all happily receive a lower bandwidth bill from AWS). They're not there for the cheap bandwidth. That's the central flaw in the parent's bafflement about markets, they did a exceptionally flat appraisal of the AWS value proposition, as though AWS isn't a gigantic business of many dozens of service offerings which act as a customer magnet and retention mechanism.

[slownews45]

1) $50/TB is not insane pricing relatively. I used to pay something like $3/GB (way back) which would be $3,000/TB. There are still plenty of folks charging high rates.

2) $50/TB may not be a big part of some customers bills. If it is on cloudfront you drop to 0.02/GB or $20/TB for larger volumes.

3) AWS doesn't charge separately for some services that wrap around networking. Their nitro instances have a pretty good networking story.

4) Especially with this new free tier on cloudfront, a fair number of users will never hit 1TB per month (free) data.

[api]

It’s absolutely insane compared with what bare metal providers charge.