[leesalminen]

So much this. I learned this the hard way.

We had a search function with typeahead abilities. I had intentionally removed the rate limit from that endpoint to support fast typers.

One day around 6AM, someone in Tennessee came into work and put their purse down on their keyboard. The purse depressed a single key and started hitting the API with each keystroke.

Of course after 15 minutes of this the db became very unhappy. Then a web server crashed because the db was lagging too much. Cascading failures until that whole prod cluster crashed.

Needless to say the rate limit was readded that day ;).

[o11c]

This is a reminder that "we want to support bursts" is much more common thing than "we want a higher ratelimit". Often multiple levels of bursts are reasonable (e.g. support 10 requests per minute, but only 100 requests per day; support 10 requests per user, but only 100 requests across all users).

There are several ways to track history with just a couple variables (or, if you do have the history, but only accessing a couple of variables); the key observation is that you usually don't have to be exact, only put a bound on it.

For history approximations in general, one thing I'm generally fond of is using an exponential moving average (often with λ=1/8 so it can be done with shifts `ema -= ema>>3; ema += datum>>3` and it's obvious overflow can't happen). You do have to be careful that you aren't getting hyperbolic behavior though; I'm not sure I would use this for a rate limiter in particular.