[bmizerany]

azmenthe's article covers the problem well: http://blog.tempo-db.com/post/42318820124/estimating-percent...

Basically, when you have more data than memory and time to sort them in order to find the percentile you're looking for, you need to employ an algorithm that trades rank selection accuracy for lower memory and CPU costs. This package does that.