AWS is full of dark patterns. You can sign up for the so-called "free" tier and then too easily, unwittingly enable something that suddenly charges you hundreds of dollars before you know it (by getting a bill at the end of the month), even if you're not doing anything with the service except looking around. AWS doesn't give any warning to free tier members that a configuration change is going to cost you, and their terms are also very confusing. For example, PostgreSQL is advertised as free, but "Aurora PostgreSQL" is quite costly.
> unwittingly enable something that suddenly charges you hundreds of dollars before you know it
The default is to have current and estimated monthly cost displayed on your root console as soon as you login. You will also get email alerts when you hit 50% and then 80% of your "free tier quota" in a month.
> even if you're not doing anything with the service except looking around.
I'm not aware of any services which will cost you money unless you actively enable them and create an object within it's class. Many services, such as S3, will attempt to force you into a "secured" configuration that avoids common traps by default.
> For example, PostgreSQL is advertised as free, but "Aurora PostgreSQL" is quite costly.
There's a rubric to the way AWS talks about it's internal services that is somewhat impenetrable at first. It's not too hard to figure out, though, if you take the time to read through their rather large set of documentation. That's the real price you must pay to successfully use the "free tier."
Anyways.. PostgreSQL is an open source project. Amazon RDS is a managed service that can run instances of it. Amazon Aurora is a different service that provides it's own engine that is _compatible_ with MySQL and PostgreSQL.
To know why you'd use one or the other, the shibboleth is "FAQ," so search for "AWS Aurora FAQ" and carefully read the whole page before you enable the service.
We have a slack channel, #aws-budget-alerts, where AWS sends a notification any time our forecasted spend reaches certain milestones or the actual spend reaches certain milestones.
I worked on a team with similar cost optimisation gurus... They abused HTTP code conventions and somehow managed to wedge in two REST frameworks into the Django app that at one point had 1m+ users...
If I understand TFA, you'd need to find a way to get S3 (which offers no server-side script execution, only basic file delivery) to emit an error code (403 specifically) alongside a response of useful data. Good luck...
> For buckets configured with website hosting, applicable request and other charges will still apply when S3 returns a custom error document or for custom redirects.
From the previous story, "S3 requests without a specified region default to us-east-1 and are redirected as needed. And the bucket’s owner pays extra for that redirected request."
So will Amazon continue charge for the redirected 403?
Can't imagine a change like this would be made without some analysis.. would love an internal view into a decision like this, I wonder if they already have log data to compute financial loss from the change, or if they have sampling instrumentation fancy enough to write/deploy custom reports like this quickly.
In any case 2 weeks seems like an impressive turnaround for such a large service, unless they'd been internally preparing to acknowledge the problem for longer
> 2 weeks seems like an impressive turnaround for such a large service
I assume they were lucky in that whatever system counts billable requests also has access to the response code, and therefore it's pretty easy to just say "if response == 403: return 0".
The fact that is the case suggests they may do the work to fulfill the request before knowing the response code and doing billing, so there might be some loophole to get them to do lots of useful work for free...
> Can't imagine a change like this would be made without some analysis.. would love an internal view into a decision like this
Sure, here you go: There was some buzz and negative press so it got picked up by the social media managers who forwarded it to executive escalations who loops in legal. Legal realizes that what they are doing is borderline fraud and sends it to the VP that oversees billing as a P0. It then gets handed down to a senior director who is responsible for fixing it within a week. Comms gets looped in to soft announce it.
At no point does anyone look at log data or give a shit about any instrumentation. It is a business decision to limit liability to a lawsuit or BCP investigation. As a publicly traded company it is also extremely risky for them to book revenue that comes from fraudulent billing.
Are you for real? Legitimately baffled by your comment.
How about the financial losses of customers that could be DDoS-ed into bankruptcy through no fault of their own? Keeping S3 bucket names secret is not always easy.
There needs to be a law that says any user needs to set any limit on any service or subscription, and then the costs can not surpass this until the budget is upped by the user. At the same time, there should be real-time cost analysis, breakdown per service and predicted costs per day.
Well, GDPR showed a bit that rather global impact is possible.
If you offer an open service on the internet you need to be prepared that users and misusers will cause costs.
However, if you block it for public access you as a customer are not offering a public service. It's the cloud provider offering a public service so it seems just a basic legal principle that it's the cloud provider who pays for misuse (attempts to access something that is not public). But of course big corporations are not known for fair contracts respecting legitimate interest of the customer before legal action is on the horizon. I wonder what made AWS wake up here.
lapcat|1 year ago
akira2501|1 year ago
The default is to have current and estimated monthly cost displayed on your root console as soon as you login. You will also get email alerts when you hit 50% and then 80% of your "free tier quota" in a month.
> even if you're not doing anything with the service except looking around.
I'm not aware of any services which will cost you money unless you actively enable them and create an object within it's class. Many services, such as S3, will attempt to force you into a "secured" configuration that avoids common traps by default.
> For example, PostgreSQL is advertised as free, but "Aurora PostgreSQL" is quite costly.
There's a rubric to the way AWS talks about it's internal services that is somewhat impenetrable at first. It's not too hard to figure out, though, if you take the time to read through their rather large set of documentation. That's the real price you must pay to successfully use the "free tier."
Anyways.. PostgreSQL is an open source project. Amazon RDS is a managed service that can run instances of it. Amazon Aurora is a different service that provides it's own engine that is _compatible_ with MySQL and PostgreSQL.
To know why you'd use one or the other, the shibboleth is "FAQ," so search for "AWS Aurora FAQ" and carefully read the whole page before you enable the service.
_adamb|1 year ago
It's a really easy to set up app!
ranger_danger|1 year ago
It does if you ask it to. You can get billing alerts if current costs are projected to go over a threshold.
alanfranz|1 year ago
Fixed price cloud offerings exist for some services, but can end up with an apparently larger sticker price.
toomuchtodo|1 year ago
Jeff Barr acknowledges S3 unauthorized request billing issue - https://news.ycombinator.com/item?id=40221108 - May 2024 (18 comments)
How an empty S3 bucket can make your AWS bill explode - https://news.ycombinator.com/item?id=40203126 - April 2024 (111 comments)
jsheard|1 year ago
https://twitter.com/cperciva/status/1785402732976992417
CSMastermind|1 year ago
The steps were to raise a big enough fuss that it would undermine customer trust if the team didn't fix it.
treve|1 year ago
cbsmith|1 year ago
hiatus|1 year ago
chadhutchins10|1 year ago
surfingdino|1 year ago
belter|1 year ago
hunter2_|1 year ago
ceejayoz|1 year ago
I was wondering about that one.
cratermoon|1 year ago
So will Amazon continue charge for the redirected 403?
dmw_ng|1 year ago
In any case 2 weeks seems like an impressive turnaround for such a large service, unless they'd been internally preparing to acknowledge the problem for longer
londons_explore|1 year ago
I assume they were lucky in that whatever system counts billable requests also has access to the response code, and therefore it's pretty easy to just say "if response == 403: return 0".
The fact that is the case suggests they may do the work to fulfill the request before knowing the response code and doing billing, so there might be some loophole to get them to do lots of useful work for free...
mike_d|1 year ago
Sure, here you go: There was some buzz and negative press so it got picked up by the social media managers who forwarded it to executive escalations who loops in legal. Legal realizes that what they are doing is borderline fraud and sends it to the VP that oversees billing as a P0. It then gets handed down to a senior director who is responsible for fixing it within a week. Comms gets looped in to soft announce it.
At no point does anyone look at log data or give a shit about any instrumentation. It is a business decision to limit liability to a lawsuit or BCP investigation. As a publicly traded company it is also extremely risky for them to book revenue that comes from fraudulent billing.
pdimitar|1 year ago
How about the financial losses of customers that could be DDoS-ed into bankruptcy through no fault of their own? Keeping S3 bucket names secret is not always easy.
moi2388|1 year ago
usr1106|1 year ago
Well, GDPR showed a bit that rather global impact is possible.
If you offer an open service on the internet you need to be prepared that users and misusers will cause costs.
However, if you block it for public access you as a customer are not offering a public service. It's the cloud provider offering a public service so it seems just a basic legal principle that it's the cloud provider who pays for misuse (attempts to access something that is not public). But of course big corporations are not known for fair contracts respecting legitimate interest of the customer before legal action is on the horizon. I wonder what made AWS wake up here.
beeeeerp|1 year ago
SushiHippie|1 year ago
https://docs.aws.amazon.com/whitepapers/latest/aws-best-prac...
I can't believe that their 'fix' is to set a wildcard dns entry, this feels somewhat like a joke.
Does this mean that a NXDOMAIN response costs more than a successful response?
mike_d|1 year ago
ranger_danger|1 year ago
paulddraper|1 year ago
dangoodmanUT|1 year ago
Joel_Mckay|1 year ago
I fail to see this as progress, YMMV =3