* PUT, COPY, POST, LIST requests are $0.005 per 1,000 requests
* GET, SELECT, and all other requests are $0.0004 per 1,000 requests
It sounds like these are getting charged at the PUT request pricing. And IIUC, those prices are set because successful write operations seek on multiple HDDs (physically moving the head) to commit the data to multiple machines, or S3 would be unable meet durability goals. Successful read operations are an order of magnitude cheaper because they likely move only one of several heads, sometimes none at all (because they're cacheable).
Authorization failures should never hit an HDD.
Other folks have mentioned that these auth failures are costing AWS something. Sure...but very very little. Far less than the cost of a successful GET, much less a successful PUT. They really shouldn't be charging at all for this, or if they do, it should be like $0.00001 per 1,000 requests (another order of magnitude cheaper than GETs). Not even that really; the actual cost for them to process this request is likely less than they're
(over)charging for the bandwidth alone.
> Other folks have mentioned that these auth failures are costing AWS something.
This should be entirely irrelevant, as those same failures were happening even before the bucket existed, they were just hitting a different failure path.
However, such logs can be enabled using AWS CloudTrail or S3 Server Access Logging. After enabling CloudTrail logs--
Aaaah! CloudTrail data events are priced in such a way that each S3 request gets logged--at a cost of 20x that of the S3 request itself. I really hope OP got THAT part of the bill waived as well.
Today I was hit by a surprise 500$ bill by using byte range request within zip files without an upper bound, but not consuming the whole stream. I tested for about 45 min at about 1gbps. My ISP meter says I downloaded 300gb which lines up with physics, but AWS says it was 6000gb.
My first thought is that can be abused to multiply the damage if you want to engage in some cost based denial of service.
About every other week AWS gets out of their way to make us remember that they are the worst cloud around.
To preface this comment, I'm not super familiar with range GETs or the library you're using. Are you saying that the library was performing range GET requests but instead of sending a byte range, it was only sending a starting byte number?
> Today I was hit by a surprise 500$ bill by using byte range request within zip files without an upper bound, but not consuming the whole stream. I tested for about 45 min at about 1gbps. My ISP meter says I downloaded 300gb which lines up with physics, but AWS says it was 6000gb.
Re the bill itself - You may be able to get the surprise bill waived by AWS with a support request, especially if you haven't had to do that before.
Re the metered billing - This isn't super clear anywhere I could find in the AWS docs, but I think from AWS's perspective your network data transfer out from S3 is the amount of data you requested + initiated, not what your app ultimately happened to consume from the stream. They really do not make this clear though, which might help your case.
FWIW, it sounds like AWS's implementation is behaving consistently about "range specifiers" with the RFC they link to in the S3 docs.
Sidebar, but: If it works for your setup, I think putting a CloudFront distribution in front of the bucket and accessing that way would mitigate the damage by caching the whole object after the first request. I've used a setup like this for streaming very large files to many viewers before.
I don't use S3, but it's kind of insane that unauthorized requests contribute to your bill... couldn't a bored/malicious actor with a botnet just spam common bucket names?
A bored or malicious actor can also just DDoS your CloudFront endpoint and run up an effectively unlimited bill on your end. That exposure is par for the course with AWS, they don't have a spending hard-stop and at this point they probably never will, you just have to hope that they have mercy and waive the bill if you get a surprise extra few zeroes on the end.
Unauthorized requests are a weird thing. They still do cost money for a web service to handle and if you (the customer) have screwed things up that's kinda on you. I've run SaaS where a customer forgot to send their key and their software would just keep retrying requests to the point that they were sending thousands of requests per second with no key. However, it's kinda crappy when it's happening through no fault of your own.
One doesn't even have to make unauthorized requests to run up a bill. One can simply make lots of spam requests to a real file in a bucket - run up the request total and bandwidth at the same time.
That's exactly it. I just spent the morning cleaning up my buckets.
I had two old buckets left over from when I was using Elasticbeanstalk 5 years ago, and they had predictable names with my account ID in them. And as we heard in February, account IDs are easily leaked and not considered private.
Besides that AWS themselves recommend you to use your domain name as bucket name when using S3 static site hosting. Thankfully I've been using Cloudfront infront of randomized bucket names for a few years already.
So now I'm forced to increase the random string I use in all my bucket names from 8 to 16 characters. Every bucket is now a password for a malicious actor who wants to attack your finances.
Obviously all accounts should have budget alerts, but there is no budget ceiling feature.
Technically AWS does offer you all the tools to create a budget ceiling feature using budget alerts and lambdas that will shutdown the affected service. I just wish someone would do it because I don't have the time.
That's why, even though during the day I work on AWS for people who pay me to do so, in the evening I work on my startups hosted on Hetzner where I know in advance what my bill will be not only this month but also this year.
This is exactly where my mind went. I assume most people, myself included, didn't realize this. This is absolutely going to turn into the economic equivalent of a ddos.
First of all, I agree this is all bananas:-) but it seems is the same with GCP and Azure. For Backblaze B2 it seems its unclear in the documentation. Willing to be corrected on this....
>kind of insane that unauthorized requests contribute to your bill
Abandoned GCP API gateway because of this...best as I can tell even with a rate limit the calls for the gateway get charged. I guess if it helps if the thing behind it is computationally expensive but still feels a bit silly
i doubt it. i would imagine any single ip address would get rate-limited/throttle before doing much. the situation the article is describing is much more like a botnet — there were hundreds or thousands (or more?) of unique instances making requests.
Well this was an unwelcome attack vector, since I wrangle a fair few S3 buckets. I hastily threw together an S3 bucket name generator in Bash under Linux [1] to create names that follow the AWS rules to start my migration to my new S3 bucket names, and welcome any help to ensure it creates names that are strongly resistant to this attack, and follow various S3-service-like vendor rules, so I can do what I can until AWS and others come out with a stronger defense.
Reminder that if you are using pre-signed urls, then that url also exposes the bucket name. So, using hard to guess bucket names isn't a solution for many people. And if you are using static websites, then amazon's own docs require you to use the website name as bucket name.
> Generally, you are not charged for operations that return 307, 4xx, or 5xx responses. The exception is 404 responses returned by buckets with Website Configuration enabled and the NotFoundPage property set to a public object in that bucket.
90% sure GCP charges for rate limited API gateway. i.e. doesn't matter if it is above rate limit. Never got 100% confirmation either way though - their documentation on pricing is too vague
There are so many things that can make your AWS bill explode, so it probably should be included into your budget. Like, $1000/mo is not actually $60K in 10 years, but $60K + some random $20K that may or may not be waived by AWS support.
I wonder why does not Amazon be responsible and lock these accounts clearly attempting abuse? Logically either they are compromised or malicious. At least after certain number of failed request. And thus should be treated as such.
For me, this is not only a cost thing, but a security issue as well. Considering the IAM auth or any kind of auth method, the goal is never to be "impossible to break-in", but "the spending of break-in is higher than the potential value it protected". Yet if the price of break-in is paid by the victim, this will be a whole different story. Someone can spam the S3 API with brute force IAM credentials can try out everything until find the right credentials, and the failure attempts are paid by the victim, which is insane.
It’s a good idea to use a reverse proxy like Nginx to handle the transfer through an internal redirect, which will hide the bucket name and authentication credentials.
1. If you are using buckets for static website hosting, amazon's own docs require you to use the website name as bucket name.
2. If you are using pre-signed urls, they expose the bucket name.
3. Bucket names must be between 3 (min) and 63 (max) characters long and have various limitations. So, it's not too hard to guess bucket names and run millions of requests without even having an account.
[+] [-] scottlamb|1 year ago|reply
* PUT, COPY, POST, LIST requests are $0.005 per 1,000 requests
* GET, SELECT, and all other requests are $0.0004 per 1,000 requests
It sounds like these are getting charged at the PUT request pricing. And IIUC, those prices are set because successful write operations seek on multiple HDDs (physically moving the head) to commit the data to multiple machines, or S3 would be unable meet durability goals. Successful read operations are an order of magnitude cheaper because they likely move only one of several heads, sometimes none at all (because they're cacheable).
Authorization failures should never hit an HDD.
Other folks have mentioned that these auth failures are costing AWS something. Sure...but very very little. Far less than the cost of a successful GET, much less a successful PUT. They really shouldn't be charging at all for this, or if they do, it should be like $0.00001 per 1,000 requests (another order of magnitude cheaper than GETs). Not even that really; the actual cost for them to process this request is likely less than they're (over)charging for the bandwidth alone.
[1] https://aws.amazon.com/s3/pricing/
[+] [-] tedmiston|1 year ago|reply
Essentially every S3 bucket, public or private, whose name is discovered can be DDoSed creating an insane bill...
That's a platform level security issue for AWS to resolve. We do not need another Bucketgate.
[1]: https://twitter.com/Lauramaywendel/status/178507498639629151...
[2]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Reques...
[+] [-] tsimionescu|1 year ago|reply
This should be entirely irrelevant, as those same failures were happening even before the bucket existed, they were just hitting a different failure path.
[+] [-] QuinnyPig|1 year ago|reply
Aaaah! CloudTrail data events are priced in such a way that each S3 request gets logged--at a cost of 20x that of the S3 request itself. I really hope OP got THAT part of the bill waived as well.
[+] [-] jpambrun|1 year ago|reply
My first thought is that can be abused to multiply the damage if you want to engage in some cost based denial of service.
About every other week AWS gets out of their way to make us remember that they are the worst cloud around.
For reference https://github.com/ZJONSSON/node-unzipper/issues/308
[+] [-] tedmiston|1 year ago|reply
> Today I was hit by a surprise 500$ bill by using byte range request within zip files without an upper bound, but not consuming the whole stream. I tested for about 45 min at about 1gbps. My ISP meter says I downloaded 300gb which lines up with physics, but AWS says it was 6000gb.
Re the bill itself - You may be able to get the surprise bill waived by AWS with a support request, especially if you haven't had to do that before.
Re the metered billing - This isn't super clear anywhere I could find in the AWS docs, but I think from AWS's perspective your network data transfer out from S3 is the amount of data you requested + initiated, not what your app ultimately happened to consume from the stream. They really do not make this clear though, which might help your case.
FWIW, it sounds like AWS's implementation is behaving consistently about "range specifiers" with the RFC they link to in the S3 docs.
https://www.rfc-editor.org/rfc/rfc9110.html#section-14.1.1-4
Sidebar, but: If it works for your setup, I think putting a CloudFront distribution in front of the bucket and accessing that way would mitigate the damage by caching the whole object after the first request. I've used a setup like this for streaming very large files to many viewers before.
https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
[+] [-] tazu|1 year ago|reply
[+] [-] jsheard|1 year ago|reply
[+] [-] mdasen|1 year ago|reply
Unauthorized requests are a weird thing. They still do cost money for a web service to handle and if you (the customer) have screwed things up that's kinda on you. I've run SaaS where a customer forgot to send their key and their software would just keep retrying requests to the point that they were sending thousands of requests per second with no key. However, it's kinda crappy when it's happening through no fault of your own.
One doesn't even have to make unauthorized requests to run up a bill. One can simply make lots of spam requests to a real file in a bucket - run up the request total and bandwidth at the same time.
[+] [-] andersa|1 year ago|reply
We really need some kind of regulation to ban predatory cloud pricing.
[+] [-] INTPenis|1 year ago|reply
I had two old buckets left over from when I was using Elasticbeanstalk 5 years ago, and they had predictable names with my account ID in them. And as we heard in February, account IDs are easily leaked and not considered private.
Besides that AWS themselves recommend you to use your domain name as bucket name when using S3 static site hosting. Thankfully I've been using Cloudfront infront of randomized bucket names for a few years already.
So now I'm forced to increase the random string I use in all my bucket names from 8 to 16 characters. Every bucket is now a password for a malicious actor who wants to attack your finances.
Obviously all accounts should have budget alerts, but there is no budget ceiling feature.
Technically AWS does offer you all the tools to create a budget ceiling feature using budget alerts and lambdas that will shutdown the affected service. I just wish someone would do it because I don't have the time.
[+] [-] dvfjsdhgfv|1 year ago|reply
[+] [-] silisili|1 year ago|reply
[+] [-] belter|1 year ago|reply
[+] [-] Havoc|1 year ago|reply
Abandoned GCP API gateway because of this...best as I can tell even with a rate limit the calls for the gateway get charged. I guess if it helps if the thing behind it is computationally expensive but still feels a bit silly
[+] [-] pid-1|1 year ago|reply
[+] [-] cjk2|1 year ago|reply
The bucket is the same name as the domain so I can imagine this could be exploited quietly very easily as PUTs are so expensive.
Literally a $5 VPS could cost someone $1k.
[+] [-] kzrdude|1 year ago|reply
[+] [-] notnmeyer|1 year ago|reply
[+] [-] yenant|1 year ago|reply
[1] https://github.com/automaticit-anthonyyen1/s3-bucket-name-ge...
[+] [-] busymom0|1 year ago|reply
[+] [-] jsiepkes|1 year ago|reply
[+] [-] belter|1 year ago|reply
Yes for GCP and Azure, also for GPT...For Backblaze I am still investigating...
Two quick examples in different contexts:
https://learn.microsoft.com/en-us/answers/questions/1064419/...
https://community.openai.com/t/cost-for-failed-gpt-requests/...
Edit: It seems for Backblaze that would be a no, see below:
https://www.reddit.com/r/backblaze/comments/16o4bed/charge_b...
[+] [-] gsf|1 year ago|reply
> Generally, you are not charged for operations that return 307, 4xx, or 5xx responses. The exception is 404 responses returned by buckets with Website Configuration enabled and the NotFoundPage property set to a public object in that bucket.
[+] [-] Havoc|1 year ago|reply
[+] [-] dilyevsky|1 year ago|reply
[+] [-] holub008|1 year ago|reply
[+] [-] belter|1 year ago|reply
[+] [-] notnmeyer|1 year ago|reply
not viable for a public bucket, but seems like it'd do the trick on private buckets.
edit: hahaha, i'm wrong https://docs.aws.amazon.com/AmazonS3/latest/userguide/Reques...
[+] [-] stackedinserter|1 year ago|reply
[+] [-] barryrandall|1 year ago|reply
[+] [-] wewtyflakes|1 year ago|reply
[+] [-] merb|1 year ago|reply
Edit: if they do that, they should make it possible to firewall buckets via ips so that they are only accessible via certain ips
[+] [-] kolanos|1 year ago|reply
[+] [-] tutunak|1 year ago|reply
[+] [-] Ekaros|1 year ago|reply
[+] [-] macok|1 year ago|reply
[+] [-] threatofrain|1 year ago|reply
[+] [-] belter|1 year ago|reply
[+] [-] cerrorism|1 year ago|reply
[+] [-] calebm|1 year ago|reply
[+] [-] victorbjorklund|1 year ago|reply
[+] [-] efge|1 year ago|reply
[+] [-] otterley|1 year ago|reply
[+] [-] fanf2|1 year ago|reply
[+] [-] busymom0|1 year ago|reply
1. If you are using buckets for static website hosting, amazon's own docs require you to use the website name as bucket name.
2. If you are using pre-signed urls, they expose the bucket name.
3. Bucket names must be between 3 (min) and 63 (max) characters long and have various limitations. So, it's not too hard to guess bucket names and run millions of requests without even having an account.