[Lxr]

The article doesn't explain what is actually going on to produce the second chart, but here is a guess. Assume people have a true ability x, an estimated ability y which is x + some unbiased noise, and a test score z which is also x + some unbiased noise. If you take a sample and collect the lowest quartile of test scores z, then the average of the estimated abilities y in that group is higher than z because the y are produced centered at x rather than centered at z. By collecting the lowest quartile of test scores, you didn't just get the dumbest people, you also got the people who happened to perform badly on the test that day. These people may be estimating their ability accurately, in which case the estimate would be higher than their randomly bad performance that day.

This seems like too obvious a mistake to not have been noticed for this long though.

[scotty79]

So it's just artifact of testing.

Lowest test scores come from low ability plus bad luck and as such are lower than just low ability and highest test scores come from high ability and great luck and as such are higher than just high ability.

[AstralStorm]

Easily verified by testing more. (Via law of big numbers.)

Unless the test is biased which means we're back to square one - discerning bias of the test from bias of respondent.

To do that you'd have to produce a specifically biased test in a different way, see how the noises add.

[SomewhatLikely]

However, from my reading of the article they were asked to estimate their performance after they had already taken the test, not their inherent ability.

[jonnycomputer]

Using quartiles like this is probably ill advised. Just show the scatter.