[munmaek]

And then they learn how to game the benchmarks.

You just can’t win.

[skrebbel]

No, because the trick is that you're paying a knowledgeable person to run the benchmark. That person would presumably actively iterate on the benchmarks and try to detect / avoid cheating.

[kmbriedis]

People would probably find out what hardware they use for benchmarks and optimize for that, leading to performance decrease for many othes