[querez]

> The geometric mean of the timings improved from 218 to 12, a ca. 20× improvement.

Why do they use the geometric mean to average execution times?

[ayhanfuat]

It's a way of saying twice as fast and twice as slow have equal effect on opposite sides. If your baseline is 10 seconds, one benchmark takes 5 seconds, and another one takes 20 seconds then the geometric mean gives you 10 seconds as the result because they cancel each other. The arithmetic mean would treat it differently because in absolute terms 10 seconds slow down is bigger than 5 seconds speedup. But that is not fair for speedups because the absolute speedup you can reach is at most 10 seconds but slow down has no limits.

[zmgsabst]

But reality doesn’t care:

If half your requests are 2x as long and half are 2x as fast, you don’t take the same wall time to run — you take longer.

Let’s say you have 20 requests, 10 of type A and 10 of type B. They originally both take 10 seconds, for 200 seconds total. You halve A and double B. Now it takes 50 + 200 = 250 seconds, or 12.5 on average.

This is a case where geometric mean deceives you - because the two really are asymmetric and “twice as fast” is worth less than “twice as slow”.

[tbillington]

This is the best explain-like-im-5 I've heard for geo mean and helped it click in my head, thank you :)

[willvarfar]

Squaring is a really good way to make the common-but-small numbers have bigger representation than the outlying-but-large numbers.

I just did a quick google and first real result was this blog post with a good explanation with some good illustrations https://jlmc.medium.com/understanding-three-simple-statistic...

Its the very first illustration at the top of that blog post that 'clicks' for me. Hope it helps!

The inverse is also good: mean-square-error is the good way for comparing how similar two datasets (e.g. two images) are.

[yorwba]

The geometric mean of n numbers is the n-th root of the product of all numbers. The mean square error is the sum of the squares of all numbers, divided by n. (I.e. the arithmetic mean of the squares.) They're not the same.