Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

[huksley]

I am using awesome NextJS and serverless-nextjs and deploy my app to CloudFront and Lambda@Edge.

I made a mistake and accidentally created a serverless function that called itself. In a recursive loop, with a 30s timeout. I thought I fixed it and deployed the code to the dev environment.

I have had an AWS Billing alert (Budgets) set up to prompt me when my monthly budget goes over $300 (my usual bill is $200/month).

Imagine the terror when I woke up the next day to see the AWS Billing alert email saying I already owed $1,484! I removed a function and deployed it again in 30 minutes, but it was too late. It has already run for 24 hours, using over 70 million Gb-Second!

Only after that I've learned that AWS Billing alerts do not work this way for CloudFront. You get delayed information on charges because they collect them from all regions.

On the following day, the bill settled at a shocking $4600. This is more than we have ever spent on AWS all time.

CloudFront includes the AWS Shield Standard feature, but somehow, it was not activated for this case (Lambda@Edge calling itself via CloudFront).

Now, I understand that I should have created CloudWatch alarms, which would alert me when the number of requests exceeds the limit. The problem is, that they need to be set up per region, and I got CloudFront charges from all points of presence.

I am a big proponent of the serverless approach. It makes it easy to scale and develop things (e.g., you get PR review version branches for free, both frontend and backend code like Vercel does). But now, I am unsure because such unexpected charges can ruin a side-project or emerging startup.

Now I am waiting on a response from AWS Support on these charges; maybe they can help me waive part of that.

What is your experience with it? Would you recommend to use to build a new product if you are bootstrapped, 3-person startup?

[sirwitti]

I'm very much on the boring technology side of things with respect to hosting.

40€ / month gets you a very powerful dedicated server that can easily handle millions of requests per day and performs incredibly well and can be managed easily.

If you also use containers you even get quite a bit of flexibility and agility.

To be honest I don't really understand the sentiment that developers can get away with not knowing basic sysadmin stuff and at the same time have to spend relevant amounts of time, energy and money to get up to speed with cloud solutions, k8s and so on.

But then again, I'm not one of the cool kids...

[jjav]

> To be honest I don't really understand the sentiment that developers can get away with not knowing basic sysadmin stuff and at the same time have to spend relevant amounts of time, energy and money to get up to speed with cloud solutions, k8s and so on.

Agreed. I hear some saying it's nice to deploy on a lambda because you don't need to know anything about the runtime environment. But it's never quite true. As you say, you do have to become an expert in all the intricacies of these proprietary deployment environments to get the best out of it and avoid getting burned. So an amount of effort has to be spent, anyway.

But the drawback is that now you spend this effort on learning what is after all a proprietary product of AWS. While AWS is massively popular, the knowledge doesn't translate to anywhere else so it locks you in. If you spent similar time learning the basics of Linux deployment and administration, your knowledge is lower level and more general.

But where it really gets you is when things go wrong and you need to dig deep to diagnose. Can I attach a debugger or watch socket traffic or run bpftrace or do any kind of diagnostics at all on that lambda instance? Oh sorry no, good luck.

Unless your budget is so low you can't afford a $5 VPS and your traffic is so low that your lambda bill will never reach $5, you're really better off deploying on a VPS that you control and can debug. Keep it simple.

[jpalomaki]

Servers can also be liability. You need to document, implement and maintaing hardening, have a process for regularly patching os and apps, monitor logs, have backup and disaster recovery procedures, regularly test the procedures, figure how to implement data encryption at rest, implement high-availability and so on.

Good platform-as-service can solve many things for you and let you focus on the core thing you are providing.

Obviously not everybody needs to worry so much about the stuff mentioned above. If you are providing SaaS solution, there’s a good chance some customer will start asking these questions as part of their procurement process.

[kif]

Reading some replies here, it’s no wonder some startups go bankrupt while investing so much in infrastructure — all while not having enough users to justify more than a single dedicated server.

Scalability should be the last thing in mind if your product sucks.

[jlmorton]

I'm a fan of boring technology too, but I would like to suggest to you that Serverless _is_ kind of boring.

Essentially you just upload a ZIP of your application, and register a handler function that takes a JSON payload.

Obviously this is quite a bit more boring than a K8s cluster, with a bunch of nodes, networking, Helm charts, etc.

I would posit that even compared to something like a DO Droplet, Serverless is still kind of boring. Everything is going to fit into the model of registering a handler function to accept a JSON payload. There's no debate about whether we're going to have Nginx, or what USGI runtime we're using. It's just a function.

And with Serverless, your cost for doing a couple million, 2-second-long requests is about four cents.

[that_guy_iain]

I literally benchmarked an application I've written on a €34 euro dedicated server to 480 requests per second. Which comes out at 1,244,160,000 requests per month. If I ever reach a level anywhere near that renting a few more to provide reliability and failover would again be reasonably cheap.

This all comes while keeping the data on servers belonging to EU companies. Using Cloud servers is rather dodgy as shown by the fact Google Analytics, Google Fonts, etc are all illegal.

Everyone keeps talking about the ease of just getting a new server and not having to spend time repairing servers. This is an over sold problem. How often have we ever dealt with hardware problems 8-10 years ago? I only remember one server going down because of hardware issues that cause problems all the other times there were failover servers and the hardware issues were fixed by datacentre staff before anyone even noticed there was an issue in the first place. With tools like ansible and chef, you can provision a new server super quickly so even if the server gets a corrupt configure of whatever, you can just wipe the server and rebuild.

Most of us are not at the scale where we would realsitically see the benefits of cloud computing for hosting.

[shepardrtc]

> To be honest I don't really understand the sentiment that developers can get away with not knowing basic sysadmin stuff and at the same time have to spend relevant amounts of time, energy and money to get up to speed with cloud solutions, k8s and so on.

There was a thread here the other day on how DevOps has failed. And this should hopefully show everyone why DevOps is needed. Cloud infrastructure is complex and needs specialists, otherwise simple mistakes can be very costly. If I make a new lambda, you better believe I'm watching invocations.

[itsmemattchung]

Amen.

I've consulted for companies that went all in on AWS Lambda + AWS SQS, only to transition them to an EC2 instance that performs the same computations at a fraction of the cost.

No — you do not need K8 on day one.

[nousermane]

> easily handle millions of requests per day

Yes. People don't realize how little that is. Million requests per day is 12 requests per second, on average. Even if we factor in peak load at 20x that, it is still well within reach of exceptionally modest hardware, assuming at least vaguely sane software stack.

[Keyframe]

Yeah, but how will you scale up your multi-dozen-users-at-the-same-time app easily then?

[bilekas]

This. The company I work for use AWS and that makes sense for the level they scale at etc and I'm sure they work out a deal with Amazon.

I have a few dedicated servers monthly cost ~100$, never ran into any problems, for a brief period I needed to pay cloudflare for some dns management that was getting a bit heavy, but even that was because I didn't know enough as to how to optimize.

For any personal projects etc / POC's even startups launching to less than 100k daily users (traffic and load type dependant ofc) hold off on the AWS>

I also don't like the lambda architecture in AWS as it seems to lock you into it in a sense, you're almost tied to that infrastructure for good. Yes you can rework it but thats tech debt that may not be possible.

Love AWS but only as an enterprise solution.

[Mo3]

You're going to understand where the cool kids come from once your single dedicated server goes down or can't handle the load any more. As soon as you try to scale horizontally or become highly available and start to think about how to do it you end up falling into the same rabbit hole.

> If you also use containers you even get quite a bit of flexibility and agility.

Yeah... and then the only difference is between a single host and multiple hosts. Guess which one Kubernetes is for?

With all due respect and no offense intended, your perspective sounds a lot like "I've never attempted to scale so I can't understand the problems"

> can get away with not knowing basic sysadmin stuff

I promise you that the knowledge you need for your single dedicated server is also needed for k8s clusters, and I definitely can't imagine anyone who can maintain a k8s cluster but can't maintain a single linux host. It's more like that scaling horizontally comes with exponentially more difficult problems than what you're used to or have heard of.

Obviously the billing is an issue but that doesn't negate the whole concept. They should definitely implement better expense controls. A simple hard cap to activate if you want would fix this for good.

[throwaway894345]

If you're putting your service behind CloudFront, you're probably aiming for DDOS protection and low latency or something that you can't easily get from a VPS. Of course, you can put your VPS behind CloudFront, but you would run into the same issue (albeit the fixed capacity of your server would effectively cap your bill).

A single VPS is great if you don't need reliability or scale, but if you care about those things you probably need at least a few of them running behind load balancers, and you probably don't want humans manually poking at them (but rather you want some reproducibility). Moreover, there's a bunch of host management stuff you're taking upon yourself--configuring metrics collection and log collection and SSH and certs and stateful backups/replication/failover and application deployment and process management and a whole bunch of other things that come for free with serverless.

Implying that serverless/Kubernetes/whatever is just hype ("cool kids") while ignoring the pretty significant discrepancies in requirements is pretty silly.

[kypro]

With almost service like hosting where there could potentially be unknown costs I always select services with fixed fees and pay annually via PayPal so no card info is stored.

If you have to pay by card and you have to subscribe always use a disposal virtual card. Most of the time these companies won't pursue you for payment because it's simply not worth the effort. The most likely outcome is that they'll just suspend your account. What you don't want happening is them taking money from you before you're even aware what is happening because that way you won't get it back and there's no room for negotiation.

[p0d]

Another not cool kid here. I've been running my little Saas for 12 years on fixed price cloud vms, €18 total per month (initially shared hosting). I'm sure I will ever hit my providers 20TB monthly traffic limit.

I settled with running lxc containers. I just couldn't bring myself to be bothered about docker. Before I left my last job we started using k8s. I wouldn't use k8s unless in an enterprise environment. Even then I would consider the need. "But Mom, all the cool kids have k8s!"

[JohnHaugeland]

> To be honest I don't really understand the sentiment that developers can get away with not knowing basic sysadmin stuff

It's not that. I used to own a VPS ISP. I know how the infrastructure stuff works.

It's that I don't care. I want to write code and deploy it to a new machine.

The time savings as a developer is enormous.

[turtlebits]

Standing up a service isn't only about compute. Where is your logging, metrics, alerting? Even with a server, you need methods for access control, hardening, log rotation, updates/patches (what happens when the OS goes out of support?).

[leesec]

Where would I learn how to set this up, this 'basic sysadmin stuff'?

[justinhj]

Cloud functions are not used to look cool, nor are they used solely by people that “don’t know basic system admin”. Their use cases don’t line up with your needs and that is okay.

[thelittleone]

One benefit is compliance. If my SaaS runs on Lambda and managed DB it offloads a bunch of OS related security requirements.

[zitterbewegung]

Also , $100 a month can get you a business line with a static IP for self hosting.

[RobRivera]

I never understood people who are in such a rush not to survey whats available.

[k8sToGo]

Powerful yes, what about getting good peering and routing?

[kjhjkhf]

the cloud means you get to write sexy code and call yourself an engineer

not the cloud means you write boring configuration files and call yourself a sysadmin

[meatsauce]

Where can you get a dedicated for 40?

[impostervt]

I once committed my private AWS keys to a public github repo. A bot scooped it up nearly instantly and spun up many, many ec2 instances that were (probably) mining bitcoins.

I received an automated email from Github telling me that I had committed a private key, but it came in the middle of the night.

In the morning, when I learned what had happened, my bill was over $3k.

I fixed the issue and emailed AWS asking for some relief, and they called me and let me know they were waving all the charges.

So, perhaps you too can beg for mercy?

[causi]

The difference between his situation and yours is that you didn't create the charges. Legally you're not liable for something someone does while impersonating you, even if you walked around with your private key on a t-shirt. They may or may not be nice to him but for you they didn't have a choice.

[arlk]

Consider yourself lucky. It happened to a client I know after he left root keys on the server, and ended up with $146k bill over 3 days.

[dabinat]

I think every developer has an AWS billing horror story.

My horror story is that my site allows users to upload videos and share them to a limited number of colleagues. When a user requests a video, a CloudFront URL is created that lasts a few hours.

I had not thought much about hotlinking because the link only lasts a few hours - what would be the point? Well, those few hours make a big difference when it’s linked on a high traffic website.

Turns out someone paid for the cheapest plan ($7) and uploaded two multi-GB files. They hotlinked them on a Vietnamese porn site and ran up charges of almost $10k.

I was alerted by Cost Anomaly Detector but it had already run up most of those charges (and the totals CAD listed were much smaller and made it seem like less of a problem, thus delaying my reaction). AWS, to their credit, waived the charges.

I had WAF already setup but it wasn’t very helpful for this type of thing. I could only block sites that I already knew about. I ended up going with a Lambda@Edge solution that validates the source site before allowing access.

Lessons learned: 1. Customers may abuse things in ways you didn’t predict 2. Cost Anomaly Detector has a delay and only kicks in once charges have accrued. It can save you from an insane bill but won’t save you completely from large bills. 3. AWS can be reasonable about this but the ball is entirely in their court.

[cyberpunk]

How can two few gb files = 10k usd of data in a few hours?

How popular is that vietnamese porn site!!

[jenny91]

I want to offer two counterpoints to common sentiments here regarding AWS billing.

1. Don't be afraid of playing around with AWS (and even spending some money). AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills. Also even if you legitimately spin up big servers to try a kubernetes cluster for a couple of days, that $20 you spent is almost certainly great bang-for-buck for the benefit of learning that experience and getting your hands dirty with AWS.

2. AWS billing is actually really good for what it is. If you've ever run any non-trivial operational system (in the real world), you would know how hard it is to collate all expenses and get them tallied up. AWS collates all billing data with ~24h lag and you can slice and splice it to your heart's content. After all it's a complicated distributed system that they've managed to build that doesn't slow down your services or otherwise get in the way!

[noasaservice]

Azure supports hard stops on services with billing maximums. It does mean that stuff gets turned off if you enable that. Then again, as an individual, that's a superb way to control costs.

And since Scamazon doesn't do that and INSTEAD "gives" you a 1 month unlimited credit, there's no telling just how stratospheric your bill can be.

> AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills.

If there were hard limits, there'd be no need to beg AWS support for leniency, which they can capriciously choose you don't deserve.

[JamesSwift]

I had an ECS cron job hang one time, so instead of a 30 second compute charge it ended up being almost a full month of continuous runtime. My usual $30 bill was $800, and the estimated charges for the next month based on 2 days of use was $1300. I didn't have a billing alert setup (definitely setup a billing alert!).

It could not have been easier to get AWS to remove the charge. A quick email to support with a brief explanation and it was immediately accepted. The hardest part was that they wanted a very specific request for how much I was asking to be refunded. So I had to go back and calculate my average costs per service and compare that to the charged costs. After that it was immediately refunded.

They aren't just handing these things out though. They made me read and acknowledge I had read their service agreements and basically swear that I know what happened and it won't happen again. Really painless process overall, all things considered.

[capableweb]

Off-topic maybe, but I'm very curious as it's hard to find hard numbers:

> my usual bill is $200/month

How many req/second are you serving? What kind of things are happening?

It seems like the bills are outrageously expensive when it comes to various cloud services, as I'm personally hosting a service that does between 10-100 req/second on average during a month, and my monthly bill end up being closer to $40/month, including traffic and everything. I'm running a database on the same server, and 20% of the requests writes stuff both to disk and to the database, while the rest just reads from DB or disk.

The whole setup took around 5 hours to setup on one day, and have been running flawlessly from day one, we haven't had to migrate servers yet after ~6 months of production usage. Probably one day we're gonna have to upgrade the server to a $80/month one, but that's a one-time thing and our revenue easily covers that.

[drdrek]

AWS may promote the technologies as prototype friendly but at the end of the day its built to be enterprise grade production tool. A company will not even bother with a 4000$ mistake, its just the price of doing business so there is little incentive to address these types of problems. Playing around with AWS for side projects is like using a chainsaw, it can really accelerate your work but if you are going to make a mistake you may lose an arm and a leg :).

[jtwebman]

For small projects why do you need the scale? I feel like once you need the scale serverless is the way more expensive then even managed Kubernetes. I still think serverless is hosting services way to make far more money with the illusion that it is easier when it really isn't. Logging is normally a huge pain. Local dev is usually a huge pain. Managing versions is a pain over just git branches especially over multiple environments. It is a pain to setup different environments and full CI/CD. In then end they might be ok with prototypes but real big systems they are huge pain but that is just my real life experience.

[tlarkworthy]

I used to work for Firebase, this is a common problem. For my own developer focussed startup I have prevented functions from calling each other to an unbounded depth, exactly so this footgun is removed.

The technical details is outbound requests is given a role encoded in the user-agent, and then I can easily filter out incoming requests by user-agents [1].

[1] https://observablehq.com/@endpointservices/webcode-docs#opti... (see loop prevention flags)

[jermaustin1]

I rarely use AWS for smaller projects, and prefer to either use Digital Ocean or bare metal from a local data center (well local when I lived in NY).

After a surprise bill like this, I would re-evaluate what serverless is actually giving me.

[johnklos]

Think about it this way:

You're a small, three person group. Are you more worried about having a modest colocated server or a small VPS run out of resources because you suddenly became really, really popular, or are you more worried about who'll go without salary for a few weeks because you misconfigured something and got a bill for multiple thousands of dollars?

Amazon takes full advantage of the fact that you can get in to deep billing waters very quickly. They can't collect your billing info quickly enough? Bull - and I mean this emphatically - shit. If a company that runs some of the largest datacenters in the world can't communicate metadata in real time, they're either hopelessly inept, or they're failing on purpose to make more $.

[s1k3s]

While reading through these threads I always get a feeling that everyone's working at YouTube or stuff like that, and they need to serve millions of users per hour. Meanwhile I'm in my corner here with an old-school $20 VPS that does just fine for my 10k users.

[em1sar]

It's really difficult for AWS or any other serverless provider for that matter, to achieve a kind of "bulletproof and safe user experience" across different offerings that encompasses everything that has to do with billing/monitoring/alerting and then also cover all kinds of potential customer scenarios (like the function calling itself, as one example).

For example, it's totally understandable that the alarms can be specified per region, why shouldn't it be like this?

Also the global AWS billing $300 alert seems to have worked but you were asleep as far as I understand. If it was a call-out style alert, then you would've noticed in the middle of the night and could've stopped it.

The only thing I agree is frustrating is this: > CloudFront includes the AWS Shield Standard feature, but somehow, it was not activated for this case (Lambda@Edge calling itself via CloudFront).

Maybe you can argue that you weren't made aware of this but idk... keep us updated

[salil999]

AWS support has historically been pretty good about removing these charges. Just be careful next time.

I racked up a $8k AWS bill for my university when I was leading a club. A few emails to AWS support and it was all resolved. Although there might've been more leniency since I was a student.

[ryanSrich]

I wish cloud providers had a nuclear option. Like “if my monthly spend hits $X, then just stop everything immediately”. Often these billing issues happen on little hobby projects and things that the owner would clearly be fine taking offline to avoid thousands in fees.

[Beltiras]

This is partly the reason why I would never use anything besides VMs or baremetal I provision myself. I'd rather have scaling problems I can solve by provisioning more hardware than billing problems because I fudged the setup. Yes, AWS might be good refunding "oopsies" but when trying to bootstrap a business I have better things to do than recover from heart palpitations.

[rambambram]

What is this kind of sh1t!? Why do people put their software on a platform where you don't know what it's going to cost upfront?

Maybe I don't understand. Maybe there are legitimate use cases for AWS and other 'vague clouds'. And it's not only these kinds of bills you get, but I also heard one even has to pay to get your own stuff off of these platforms.

What's wrong with good old webhosting (in whatever shape, size or form)? I understand there are use cases - probably for big big apps - that will benefit from cloud hosting, but that can't be for every project or business. Right?

Please enlighten me.

[ralusek]

One of my biggest fears. What's to prevent trolls and competitors from just spamming your endpoints in a loop? How do people using pay-per-use infra deal with these problems?

I really want to use Lambda for public endpoints but it just scares me.

[iasay]

What’s even more worrying is the numerous accidents hiding in a $700k a month bill which is our problem.

[mfrye0]

I started off using Lambda as well and made the same sort of mistake. I can't remember exactly how much my bill was, but it was enough that it would have drained all my savings and effectively kill my startup. AWS was kind enough to write off most of it.

We now use Lambda only for simple cron / background tasks, or consuming from Kinesis. We use ECS for everything else. ECS is nice because it's relatively simple compared to K8, but still gives the full benefit of running multiple containers on one box.

I wrote a blog post last year about our migration over to ECS and experimenting with various ways to cut costs: https://blog.bigpicture.io/how-we-cut-our-aws-bill-by-70/

[mfrye0]

On a side note, I don't know if you've already acquired any AWS credits. If not, Product Hunt Founders Club is a decent deal that will give you $5k in AWS credits. Between that and the no Stripe fees for 1 year, it paid for itself in no time.

https://www.producthunt.com/founder-club

[huksley]

Thank your for ECS suggestion.

I am definitely considering it, but struggling to choose between ECS, Elastic Beanstalk or EC2.

My past experience with ECS was a bit frustrating because I was forced to use CodeCommit to deploy and I didn't liked that. I would prefer to deploy directly from CI, for example from GitHub actions.