john_pryan's comments

I agree, the article seems to imply the plot in the original paper will always be an incorrect thing to do, instead of something which can have some issues in cases when we have inaccurate tests.

I've gone back and updated the colab notebook to use orderings exclusively instead of values and you can see that the auto correlation plot B from the first article exists when the noise is high enough but disappears when you reduce it, definitely not a statistical law.

That's a nice spot about the decreasing CC as we increase accuracy!

My hypothesis would be that some of the DK effect in the original paper may be down to an effect like this (as suggested in the original article) but that asserting it is completely incorrect because of it is premature. We'd need access to more data to verify that the level of reliability was sufficiently acceptable.

For anyone who is interested in playing around with these charts, the various assumptions that under pin them etc. I've thrown together a colab notebook as a starting point.

Observation: if you rank via true "skill" and assume for a particular instance the predicted performance and observed performance are independent but both have the true skill as their mean you dont observe the effect. CC of 0.00332755.

If you rank via observed performance and plot observed vs predicted the effect is there. CC of -0.38085757.

This is assuming very simple gaussian noise which is not going to be accurate especially as most of these tasks have normalised scores.

Edit: fixed wrong way around

https://colab.research.google.com/drive/1Vy7JjkywxwEP8nfR6oS...