[slaucon]

I feel like there should be some take away from the fact that we have to come up with new and interesting metrics like “Length of a Task That Can Be Automated” in order to point out that exponential growth is still happening. Fwiw, it does seem like a good metric, but it also feels like you can often find some metric that’s improving exponentially even when the base function is leveling out.

[xenobeb]

From Nassim Taleb

"Unless you have confidence in the ruler’s reliability, if you use a ruler to measure a table you may also be using the table to measure the ruler."

Seems like that is exactly what we are doing.

[jsnell]

It's the only benchmark I know of with a well-behaved scale. Benchmarks with for example a score from 0-100% get saturated quite quickly, and further improvements on the metric are literally impossible. And even excluding saturation, they just behave very oddly at the extremes. To use them to show long term exponential growth you need to chain together benchmarks, which is hard to make look credible.