[sobiolite]

> Nakamura responded to Kramnik’s allegations by arguing that focusing on a particular streak while ignoring other games was cherry-picking. The researchers note that there’s a problem with this argument, too, as it violates the likelihood principle. This principle tells us the interpretation should only rely on the actual data observed, not the context in which it was collected.

I don't quite understand this objection? If I won the lottery at odds of 10 million to 1, you'd say that was a very lucky purchase. But if it turned out I bought 10 million tickets, then that context would surely be important for interpreting what happened, even if the odds of that specific ticket winning would be unchanged?

[bitshiftfaced]

I believe they're speaking within the scope of the Bayesian analysis. We could interpret games outside of the winning streak as evidence to whether he's a cheater or not. Instead, I believe they are looking at the question of "given this winning streak in particular, what's the probability of him cheating in this set of games"?

They start with a prior (very low probability), I'm assuming they use the implied probabilities from the Elo differences, and then update that prior based on the wins. That's enough to find the posterior they're interested in, without needing to look outside the winning streak.

[Archelaos]

> "given this winning streak in particular, what's the probability of him cheating in this set of games"

I think the problem lies in the antecedent. Given all chess tournaments played, how often would we observe such a winning streak on average? If the number of winning streaks is near the average, we have no indication of cheating. If it is considerably lower or higher, some people were cheating (when lower, than the opponents).

Then the question is, whether the numbers of winning streaks of one person are unusually high. If we would for example expect aprox. 10 winning streaks, but observe 100, we can conclude that aprox. 90 were cheating. The problem with this is that the more people cheat, the more likely we are to suspect an honest person of cheating as well.

Again, this would be different if the number of winning streaks for a particular person were unusually high.

[jonahx]

His performance in games outside the streak is relevant to the prior of his being a cheater, which in turn is highly relevant to how calculate p(cheater | this streak).

[nextaccountic]

The issue here is that the events are not independent. Because of that, the other games surely provide useful data

[oersted]

Indeed. I'd say that the issue is that they are misinterpreting the word "collecting". The principle is true if you are collecting or observing data live, but this data was collected long ago and with a much wider scope: when the games were recorded.

What they are doing here is sampling the data after the fact, and obviously one needs to take a uniformly random sample of a dataset for any statistical analysis done on it to be representative.

[AlecBG]

Or similarly I flip a coin a thousand times, but only tell you when it's heads and don't tell you how many flips I did.

[gus_massa]

I agree. The problem when you look only at some "interesting" data even has a name https://en.wikipedia.org/wiki/Look-elsewhere_effect

[bregma]

Or if Bob Barker has opened door number 3 and there's a goat behind it.

[sema4hacker]

Monty Hall.