My understanding is that this is a difficult problem to solve "perfectly" due to lag between incurring a cost and recording the cost.
I believe GCP currently has the best feature for this. You can set both billing alerts as well as caps. However, I also believe that it can take up to 24 hours between incurring a cost and it showing up on your billing report. So even on GCP (which is the most forward-thinking cloud service for this feature), you can incur up to 24 hours of charges over your maximum billing threshold cutoff. I'm also not 100% sure if GCP's billing threshold is really designed to be a "hard cap" per se.
The real question is whether AWS should let perfect be the enemy of good; and/or whether providing a somewhat "broken" service like GCP's would mislead customers into feeling more protected than they actually are.
See here where someone set a Firebase billing budget of $7 but an infinite recursion generated $72,000 in charges. When the founders started seeing the charges come in, all they could do was watch as it grew and grew....because their screen was merely reflecting what had already happened in hours past.
I understand that there's an endless number of edge cases that are hard or impossible to solve here. But the unlimited potential charges eliminate AWS for certain scenarios.
I would not use AWS for private stuff because of this, it is simply not worth it to risk huge bills to me personally. Of course what I might spend on AWS is peanuts, but it could translate into using AWS with my employer because I'm familiar with certain parts of it.
And even professionally this is kind of a risk, especially if you're working for a small company. And even more if you're not that experienced with AWS, there's a lot of parts to it and it can be really hard to figure out what is costing you money exactly, if you're not an AWS expert. I once created a relatively small, but noticeable charge on AWS due to a recurring script transferring a lot more data than intended. With my limited knowledge it was pretty much impossible to figure out the exact source from AWS tools. I more or less stumbled upon the issue randomly while looking into this, which is kinda scary to me.
All this does make AWS less attractive compared to classic hosting/renting a server, if that fits your use case and the lesser flexibility isn't an issue.
As a junior developer I'm always afraid using any AWS service to make a mistake that cost me a lot of money. I'm not talking about millions, simply 2k would be really bad for me.
Anyone who is personally liable for the the bill should be terrified of AWS. It's enough to lose your business to a usage spike, much worse to lose your home and car.
I struggle to consider a scenario where setting up a limited liability organisation wouldn't make sense, even if only for interacting with Amazon.
I had been in the same position for about ten years until I set up a limited liability company. It does take some effort to set up and costs about $20/month[1], but to me the peace of mind is well worth it.
You really need to implement limits service-by-service, and with as many services as AWS has, it would make sense if some of them hadn't implemented limits yet.
Think about the details... if you are paying something for both storage and bandwidth, and get a sudden surge of bandwidth, do you really want items in storage to be deleted? You basically never want storage to be automatically deleted even if your program suddenly uses a surprising amount; limits on its maximum size and alerts are much better.
But once you realize that bill capping doesn't make sense for storage, well, many different services are essentially some type of storage. This is a feature that sounds good but in practice what people really need is something slightly different.
> But once you realize that bill capping doesn't make sense for storage
I'd think you want something like a circuit breaker.. steady costs or slowly increasing costs aren't like to be a problem, but a sudden surge in costs is what I'm concerned with.
Perhaps if it operated like a time-averaged quota.. don't let me incur _new_ costs if I slide too far out of my apparent range. Give me a knob to control that range or temporarily disable it, and maybe a way to monitor those events so I can react to them appropriately for my particular application.
I don't think you ever want storage to be deleted as part of an automatic bill cap system. Just refuse access, or allow reads but not writes until the customer reviews it. There's even an HTTP code for it: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/402 (non standard, but I know for a fact it's used by Azure).
Actually deleting things should be done after a period of time specified in writing in the contract with the customer, and I'd hope you'd try to contact them first.
But I think an automatic bill cap absolutely makes sense for storage, and I can't really see any reasons it couldn't work.
> You basically never want storage to be automatically deleted even if your program suddenly uses a surprising amount
In most use cases you're right. But never? Not true, not all storage is used the same way. And the thing is, why should Amazon decide which use case is valid or not? I might care more about being able to afford the service than to keep the data around, depending on what kind of service I offer my user. As a platform provider, they should be agnostic, but their greed for money is shining through their willingness to be a true platform provider.
Exactly. I have heard stories of bill capping that ended up with infra and data being deleted, so I can really see why they wouldn't implement it. What they could do however would be to allow usage limit based on IAM policies for example (Deny s3:Getobject Condition: monthly bandwidth >= x)
I understand that designing a real-time billing system is near impossible.
But couldn't Google/Amazon simply pretend they have real-time bill capping, and then simply swallow the costs incurred by any delays?
Doesn't seem like rocket science, probably won't cost much, and might bring in new kinds of business (a lot of businesses won't allow most employees to sign an effectively blank check)
It hasn't been implemented because it is nonsensical.
So you hit your limit. Is all your storage deleted? You can't receive an alert because that costs something (even if it's a fraction of a cent) to send. Are your domains forfeit? Audit logs destroyed? There's no reasonable way to implement this. Billing alerts are the best you can do on this problem and I think it would be a good faith move for AWS to enable some by default but a limit just doesn't make sense on any level.
EDIT: Lots of people proposing solutions that work for them. AWS has to think of everyone. And they did. That's why budget alerts exist and you can respond in the way you choose. Everyone's conflicting ideas for how to solve this can be implemented today on top of billing alerts/actions[1]. Case closed.
A limit to the cost to me. I don't care what the cost to Amazon is.
> So you hit your limit. Is all your storage deleted?
Easy. Cut access, don't delete the files, set a time limit for resolving the problem before they are deleted.
> EDIT: Lots of people proposing solutions that work for them. AWS has to think of everyone.
No they don't. A feature that solves the main problem for lots of users can be added and for the users that need a complex solution, well, they can justnot use the simple one.
As a small dev, I do not care if data is lost and they stop storing the data/turn off machines. I am more concerned about the cost. I have had two cases.
1. Misconfigured software that we accidentally pushed into production and by the time we noticed it like an hour later we were $100s out. Luckily it was not in the 1000s other that would have sunk the business.
2. An old account of ours got hacked and people spun up like 50-100 servers and it happened overnight and we had a bill of $50k-$70k. Luckily Amazon wrote that off for us. Otherwise I have no clue how we would have ever paid that.
These are usually extraordinary cases, so even if there is a data loss or your servers are shut down, it is a better option than going down the hole with 10s of 1000s in charges.
Nonsense. If you're going to bill for the alert, send it when the charge is the difference between the current bill and the limit.
Of course, it's no more realistic to charge people for the alert than it is to charge them for email receipts. Do you not get those from Amazon?
You implement the feature cap by ceasing to provide the feature once the account is over the billing limit. It is always possible to project the cost of "freeze everything until the end of the month", because there is no unknown information in that scenario. Nothing you've listed is an actual problem.
> You can't receive an alert because that costs something (even if it's a fraction of a cent) to send.
Yea, what a crazy world that would be if AWS just gave each account free stuff like free data transfer for the first gb, or free s3 storage for the first ~100mb
> but a limit just doesn't make sense on any level.
A limit very much makes sense of many levels, just not all of them at once. You're arguing against a hard limit of every single AWS service.
Nobody wants their storage deleted once their budget hits a limit, what they want, and is reasonable to have, is a limit that prevents AWS from spinning additional EC instances. Or a limit that blocks your usage of Textractor. Or many other limits an user could individually set-up for the services where a limit makes sense.
At the very least allow users to roughly define what happens when the threshold is reached: S3 will still store data but maybe not serve anything, Lambdas will still be defined but not run, EC2 instances will keep running, etc. It's not supposed to be a hard cap as in "don't spend a penny more than my $5" but more as in "do everything possible and reasonable so the customer doesn't wake up to a sudden $20k overnight AWS bill".
I used rented renderfarms a lot for VFX work. And many of those had a "if your workload stays like it is it will cost you roughly X after n frames rendered"-feature. Otheres made you transfer money first and you would get a warning before that money would run out.
The problem is that you can run into unexpectedly high cost without warning or option to decide whether it is worth it and one of the solutions would be to just tell people when the rate at which they use your stuff increases or decreases by a certain threshold. And because they can bill you, they also know (roughly) what the things that you use cost.
I never used AWS for serious things, but not being able to decide my spending will be a serious argument against it.
You do understand that this is not a binary feature? Not just on or off.
The point is not that there should be no costs, but that they should be minimised. This means that data that has to be written, such as logs, should continue to be written and may also incur costs, but everything else should be switched off to prevent you from suddenly being stuck with thousands of dollars in debt.
Resources can be divided into persistent and not. People could opt into "we'll cut off traffic and prevent spawning new resources" limit without affecting storage, domains, logs, etc.
This also aligns well with the "planned vs unexpected" costs.
This is one of the reasons I really like Vultr. They have a simple interface and their pricing has always been solid for me. I don't ever go to sleep worried I'm going to wake up with a $10k bill.
At one of the companies years ago a developer accidentally leaked an API credential on GitHub and the company woke up with a $30k bill.
That's why I don't use AWS. I am sure I forget to disable or enable something and then wake up with life ruining bill.
Since they don't want to fix it, it seems like this is still a part of their business model.
Maybe this business requires more regulation - if company cannot resolve an issue negatively impacting the consumer, then government should step in and mandate bill capping features to be implemented.
Maybe we should start writing to our representatives to come up with something, so companies like Amazon know their place.
I'm using AWS for my side-project and while I enjoy how cheap it can be, I'm also terrified of it and always have the Cost Explorer opened in a tab that I check every day.
Many comments say it cant be done, but this is disingenious. As if we were bureaucrats with no imagination:
Eg just have 2 caps: First cap triggers a red alarm prohibiting provisioning, or storing, of new data. The second one (when you are like 50% above the first cap) is dark red, closing down everything except storage.
Sure, decisions have to be made for every single service, but you don’t need a perfect solution if it is optional.
Eg, you could start ONLY with EC2 and will have fixed 50% of the problem.
I believe at the very least they could offer a more flexible way of managing Service Quotas.
Currently it's too much of a hassle to increase/decrease account-wide limits: it takes too much time, depends on AWS Support, UX is nowhere as nice as, say instance reservations etc.
As a result of this people tend to increase the limits once in a while just to make sure it doesn't become an issue at a very unfortunate time, and that's it. It never gets decreased/managed again. It's obviously not seen as a use-case by AWS, since they only have a "Request quota increase" button in the UI, and no "decrease" button. [1]
SQ on its own wouldn't cover all kinds of costs (for on-demand items like Lambda executions they offer limits on rates, not total count), but it'd be better than nothing, it'd prevent scenarios where you'd wake up to 100 GPU instances mining bitcoin, and it's within the quotas since Bob asked for an increase two years ago to try something.
[1] Just having a button there would still be an awful UX, yet I believe its complete omission is noteworthy.
Borrow what DHCP does for leases, but with payment quotas.
First layer is a burst quota, up to that maximum in any spike. This only regenerates when the (monthly?) average quota has enough slack at current that an additional burst can be harvested without the remaining average being under the initial set value. (The math would probably be different, remaining period quota less burst average over time greater than period quota average, but that's the sales description of the concept.)
The second layer is the monthly (or some other period) average for the maximum expenditure allowed.
A billing endpoint would maintain a fractional bucket of spending (divided up as makes sense) but in the case of a single quota consumer would receive an estimate for the period (ideally the burst) and allow up to half of that to be used. At that point it will 'renew' the quota (lease), and flush billing, including sending alerts. If there isn't any further quota the remaining released balance would be consumed and then requests fail.
When it reach the cap, all your service stop working? This is incredibly user hostile.
If you want some warning when utilization is high, it can be relatively easy to setup with AWS's existing feature, but you need to do some configuration/tweaking on your own.
That would be substantively less user hostile to me than a massive bill on a personal project, which is easy to see occurring for any number of reasons.
> When it reach the cap, all your service stop working? This is incredibly user hostile.
So set your cap to a very high number, much higher than any amount you expect to be billed. Surely for everyone there is some number at which they would rather their services go down, than have to pay it?
> When it reach the cap, all your service stop working?
That exactly what I would like to have. Services stop and I get time to review what's happened without any stress that my bank account will be emptied.
Yes this would be a super useful feature especially for dormant accounts. One of my old aws account was recently hacked and racked up bills in thousands of dollars.
The AWS support was super nice and took care of it for me but it gave me some sleepless nights until it was resolved.
If this was a feature I would have most definitely capped my account at 1k (my max bill being $50). They can always send alerts to increase your cap based on projections like it shows in cost explorer. Also since it's an opt-in feature I'm kinda taking responsibility that I'm okay if services are suspended due to hitting the caps.
They could easily factor natural growth and ping you when it feels like bills are getting too close to the cap. Even “bursts” could be allowed x% above the cap if you get an email within an hour.
The lack of caps is just laziness or a dark pattern by Amazon. There’s no excuse to not implement them, it’s not a hard problem.
[+] [-] runnerup|5 years ago|reply
I believe GCP currently has the best feature for this. You can set both billing alerts as well as caps. However, I also believe that it can take up to 24 hours between incurring a cost and it showing up on your billing report. So even on GCP (which is the most forward-thinking cloud service for this feature), you can incur up to 24 hours of charges over your maximum billing threshold cutoff. I'm also not 100% sure if GCP's billing threshold is really designed to be a "hard cap" per se.
The real question is whether AWS should let perfect be the enemy of good; and/or whether providing a somewhat "broken" service like GCP's would mislead customers into feeling more protected than they actually are.
See here where someone set a Firebase billing budget of $7 but an infinite recursion generated $72,000 in charges. When the founders started seeing the charges come in, all they could do was watch as it grew and grew....because their screen was merely reflecting what had already happened in hours past.
https://www.theregister.com/2020/12/10/google_cloud_over_run...
Discussed here: https://news.ycombinator.com/item?id=25398148
[+] [-] fabian2k|5 years ago|reply
I would not use AWS for private stuff because of this, it is simply not worth it to risk huge bills to me personally. Of course what I might spend on AWS is peanuts, but it could translate into using AWS with my employer because I'm familiar with certain parts of it.
And even professionally this is kind of a risk, especially if you're working for a small company. And even more if you're not that experienced with AWS, there's a lot of parts to it and it can be really hard to figure out what is costing you money exactly, if you're not an AWS expert. I once created a relatively small, but noticeable charge on AWS due to a recurring script transferring a lot more data than intended. With my limited knowledge it was pretty much impossible to figure out the exact source from AWS tools. I more or less stumbled upon the issue randomly while looking into this, which is kinda scary to me.
All this does make AWS less attractive compared to classic hosting/renting a server, if that fits your use case and the lesser flexibility isn't an issue.
[+] [-] cr1pablo|5 years ago|reply
[+] [-] oliwarner|5 years ago|reply
I struggle to consider a scenario where setting up a limited liability organisation wouldn't make sense, even if only for interacting with Amazon.
[+] [-] GrantZvolsky|5 years ago|reply
[1]: https://www.ukpostbox.com/address/business-address-service
[+] [-] lacker|5 years ago|reply
Think about the details... if you are paying something for both storage and bandwidth, and get a sudden surge of bandwidth, do you really want items in storage to be deleted? You basically never want storage to be automatically deleted even if your program suddenly uses a surprising amount; limits on its maximum size and alerts are much better.
But once you realize that bill capping doesn't make sense for storage, well, many different services are essentially some type of storage. This is a feature that sounds good but in practice what people really need is something slightly different.
[+] [-] akira2501|5 years ago|reply
I'd think you want something like a circuit breaker.. steady costs or slowly increasing costs aren't like to be a problem, but a sudden surge in costs is what I'm concerned with.
Perhaps if it operated like a time-averaged quota.. don't let me incur _new_ costs if I slide too far out of my apparent range. Give me a knob to control that range or temporarily disable it, and maybe a way to monitor those events so I can react to them appropriately for my particular application.
[+] [-] p1necone|5 years ago|reply
Actually deleting things should be done after a period of time specified in writing in the contract with the customer, and I'd hope you'd try to contact them first.
But I think an automatic bill cap absolutely makes sense for storage, and I can't really see any reasons it couldn't work.
[+] [-] capableweb|5 years ago|reply
In most use cases you're right. But never? Not true, not all storage is used the same way. And the thing is, why should Amazon decide which use case is valid or not? I might care more about being able to afford the service than to keep the data around, depending on what kind of service I offer my user. As a platform provider, they should be agnostic, but their greed for money is shining through their willingness to be a true platform provider.
[+] [-] forty|5 years ago|reply
[+] [-] Hamuko|5 years ago|reply
https://bugzilla.mozilla.org/show_bug.cgi?id=34572
[+] [-] lloeki|5 years ago|reply
[+] [-] tgv|5 years ago|reply
[+] [-] Austin_Conlon|5 years ago|reply
[+] [-] londons_explore|5 years ago|reply
But couldn't Google/Amazon simply pretend they have real-time bill capping, and then simply swallow the costs incurred by any delays?
Doesn't seem like rocket science, probably won't cost much, and might bring in new kinds of business (a lot of businesses won't allow most employees to sign an effectively blank check)
[+] [-] andrewguenther|5 years ago|reply
So you hit your limit. Is all your storage deleted? You can't receive an alert because that costs something (even if it's a fraction of a cent) to send. Are your domains forfeit? Audit logs destroyed? There's no reasonable way to implement this. Billing alerts are the best you can do on this problem and I think it would be a good faith move for AWS to enable some by default but a limit just doesn't make sense on any level.
EDIT: Lots of people proposing solutions that work for them. AWS has to think of everyone. And they did. That's why budget alerts exist and you can respond in the way you choose. Everyone's conflicting ideas for how to solve this can be implemented today on top of billing alerts/actions[1]. Case closed.
[1] https://aws.amazon.com/blogs/aws-cost-management/get-started...
[+] [-] IanCal|5 years ago|reply
> So you hit your limit. Is all your storage deleted?
Easy. Cut access, don't delete the files, set a time limit for resolving the problem before they are deleted.
> EDIT: Lots of people proposing solutions that work for them. AWS has to think of everyone.
No they don't. A feature that solves the main problem for lots of users can be added and for the users that need a complex solution, well, they can justnot use the simple one.
[+] [-] capybara_2020|5 years ago|reply
1. Misconfigured software that we accidentally pushed into production and by the time we noticed it like an hour later we were $100s out. Luckily it was not in the 1000s other that would have sunk the business. 2. An old account of ours got hacked and people spun up like 50-100 servers and it happened overnight and we had a bill of $50k-$70k. Luckily Amazon wrote that off for us. Otherwise I have no clue how we would have ever paid that.
These are usually extraordinary cases, so even if there is a data loss or your servers are shut down, it is a better option than going down the hole with 10s of 1000s in charges.
[+] [-] thaumasiotes|5 years ago|reply
Of course, it's no more realistic to charge people for the alert than it is to charge them for email receipts. Do you not get those from Amazon?
You implement the feature cap by ceasing to provide the feature once the account is over the billing limit. It is always possible to project the cost of "freeze everything until the end of the month", because there is no unknown information in that scenario. Nothing you've listed is an actual problem.
[+] [-] preommr|5 years ago|reply
Yea, what a crazy world that would be if AWS just gave each account free stuff like free data transfer for the first gb, or free s3 storage for the first ~100mb
[+] [-] monsieurbanana|5 years ago|reply
A limit very much makes sense of many levels, just not all of them at once. You're arguing against a hard limit of every single AWS service.
Nobody wants their storage deleted once their budget hits a limit, what they want, and is reasonable to have, is a limit that prevents AWS from spinning additional EC instances. Or a limit that blocks your usage of Textractor. Or many other limits an user could individually set-up for the services where a limit makes sense.
[+] [-] heipei|5 years ago|reply
[+] [-] neolog|5 years ago|reply
[+] [-] Havoc|5 years ago|reply
There is no reason why an absolute "don't ruin me financially" button can't be implemented in addition to everything you mention.
>There's no reasonable way to implement this.
Azure VS enterprise credits work this way. 150 hard cap with no credit card on the individual account.
[+] [-] atoav|5 years ago|reply
The problem is that you can run into unexpectedly high cost without warning or option to decide whether it is worth it and one of the solutions would be to just tell people when the rate at which they use your stuff increases or decreases by a certain threshold. And because they can bill you, they also know (roughly) what the things that you use cost.
I never used AWS for serious things, but not being able to decide my spending will be a serious argument against it.
[+] [-] nielsole|5 years ago|reply
> There's no reasonable way to implement this [...] on any level
That's not true (as OP proposes a reasonable way)
[+] [-] croes|5 years ago|reply
[+] [-] viraptor|5 years ago|reply
This also aligns well with the "planned vs unexpected" costs.
[+] [-] raverbashing|5 years ago|reply
Sure. That's why you make it optional. And configurable in a way
But at a very basic idea, blocking new services from being started and throttling of existing ones would go a long way in helping. A very long way.
[+] [-] ddtaylor|5 years ago|reply
At one of the companies years ago a developer accidentally leaked an API credential on GitHub and the company woke up with a $30k bill.
[+] [-] varispeed|5 years ago|reply
[+] [-] ablekh|5 years ago|reply
[1] https://www.amazon.jobs/en/principles
[+] [-] Zealotux|5 years ago|reply
[+] [-] tyrex2017|5 years ago|reply
Eg just have 2 caps: First cap triggers a red alarm prohibiting provisioning, or storing, of new data. The second one (when you are like 50% above the first cap) is dark red, closing down everything except storage.
Sure, decisions have to be made for every single service, but you don’t need a perfect solution if it is optional.
Eg, you could start ONLY with EC2 and will have fixed 50% of the problem.
[+] [-] aranelsurion|5 years ago|reply
Currently it's too much of a hassle to increase/decrease account-wide limits: it takes too much time, depends on AWS Support, UX is nowhere as nice as, say instance reservations etc.
As a result of this people tend to increase the limits once in a while just to make sure it doesn't become an issue at a very unfortunate time, and that's it. It never gets decreased/managed again. It's obviously not seen as a use-case by AWS, since they only have a "Request quota increase" button in the UI, and no "decrease" button. [1]
SQ on its own wouldn't cover all kinds of costs (for on-demand items like Lambda executions they offer limits on rates, not total count), but it'd be better than nothing, it'd prevent scenarios where you'd wake up to 100 GPU instances mining bitcoin, and it's within the quotas since Bob asked for an increase two years ago to try something.
[1] Just having a button there would still be an awful UX, yet I believe its complete omission is noteworthy.
[+] [-] mjevans|5 years ago|reply
Borrow what DHCP does for leases, but with payment quotas.
First layer is a burst quota, up to that maximum in any spike. This only regenerates when the (monthly?) average quota has enough slack at current that an additional burst can be harvested without the remaining average being under the initial set value. (The math would probably be different, remaining period quota less burst average over time greater than period quota average, but that's the sales description of the concept.)
The second layer is the monthly (or some other period) average for the maximum expenditure allowed.
A billing endpoint would maintain a fractional bucket of spending (divided up as makes sense) but in the case of a single quota consumer would receive an estimate for the period (ideally the burst) and allow up to half of that to be used. At that point it will 'renew' the quota (lease), and flush billing, including sending alerts. If there isn't any further quota the remaining released balance would be consumed and then requests fail.
[+] [-] 867-5309|5 years ago|reply
that's but a drop in the digital ocean
[+] [-] lend000|5 years ago|reply
[+] [-] karmasimida|5 years ago|reply
When it reach the cap, all your service stop working? This is incredibly user hostile.
If you want some warning when utilization is high, it can be relatively easy to setup with AWS's existing feature, but you need to do some configuration/tweaking on your own.
[+] [-] btilly|5 years ago|reply
Not nearly as user hostile as the possibility of an unexpected DOS on your bank account while you're asleep.
Which has actually happened to people.
[+] [-] technion|5 years ago|reply
[+] [-] exactlysolved|5 years ago|reply
So set your cap to a very high number, much higher than any amount you expect to be billed. Surely for everyone there is some number at which they would rather their services go down, than have to pay it?
[+] [-] varispeed|5 years ago|reply
That exactly what I would like to have. Services stop and I get time to review what's happened without any stress that my bank account will be emptied.
[+] [-] alkonaut|5 years ago|reply
[+] [-] bkovacev|5 years ago|reply
[+] [-] ghoomketu|5 years ago|reply
The AWS support was super nice and took care of it for me but it gave me some sleepless nights until it was resolved.
If this was a feature I would have most definitely capped my account at 1k (my max bill being $50). They can always send alerts to increase your cap based on projections like it shows in cost explorer. Also since it's an opt-in feature I'm kinda taking responsibility that I'm okay if services are suspended due to hitting the caps.
[+] [-] thitcanh|5 years ago|reply
The lack of caps is just laziness or a dark pattern by Amazon. There’s no excuse to not implement them, it’s not a hard problem.