top | item 5555393 (no title) bmizerany | 13 years ago Yes. I use this for that. Query then Reset every 5 minutes. discuss order hn newest kingfishr|13 years ago Oh, that'll be great! That's not quite the same as a sliding window of last 5 minutes, but it'll definitely work for my use case.(Intuitively, a sliding window seems very hard/impossible -- how do you discard old events without keeping a complete record?) teraflop|13 years ago There are other algorithms that handle the sliding-window case, e.g. http://research.microsoft.com/pubs/77611/quantiles.pdfBut for all but the most extreme cases, it's sufficient to just keep all the values in memory until they fall out of your window. Even if you're getting 1000 requests/second, that's still only 300,000 values that you have to store. load replies (1) jcrites|13 years ago I have seen continuous exponential decay applied to this problem. unknown|13 years ago [deleted]
kingfishr|13 years ago Oh, that'll be great! That's not quite the same as a sliding window of last 5 minutes, but it'll definitely work for my use case.(Intuitively, a sliding window seems very hard/impossible -- how do you discard old events without keeping a complete record?) teraflop|13 years ago There are other algorithms that handle the sliding-window case, e.g. http://research.microsoft.com/pubs/77611/quantiles.pdfBut for all but the most extreme cases, it's sufficient to just keep all the values in memory until they fall out of your window. Even if you're getting 1000 requests/second, that's still only 300,000 values that you have to store. load replies (1) jcrites|13 years ago I have seen continuous exponential decay applied to this problem.
teraflop|13 years ago There are other algorithms that handle the sliding-window case, e.g. http://research.microsoft.com/pubs/77611/quantiles.pdfBut for all but the most extreme cases, it's sufficient to just keep all the values in memory until they fall out of your window. Even if you're getting 1000 requests/second, that's still only 300,000 values that you have to store. load replies (1)
kingfishr|13 years ago
(Intuitively, a sliding window seems very hard/impossible -- how do you discard old events without keeping a complete record?)
teraflop|13 years ago
But for all but the most extreme cases, it's sufficient to just keep all the values in memory until they fall out of your window. Even if you're getting 1000 requests/second, that's still only 300,000 values that you have to store.
jcrites|13 years ago
unknown|13 years ago
[deleted]