[robochat]

Actually, I can't seem to get the maths to work. Isn't it just the Binomial distribution? Each question has 5 options so the probability of 'success' is 1/5 and so to get 19 questions 'specifically wrong' out of the 45 planted questions by chance is just (1/5)*19 * (1-1/5)*26 * 45!/(19!26!) = 1 in 2588 but in the article it is 1:100. What am I doing wrong ?

[probably_wrong]

Here's my best guess: 19 is the cutoff point for a binomial test [1] where the probability of at least that many answers to match those in the honeypot test goes below 0.01. But this holds only if you assume p=0.25.

Why would you use 0.25 instead of 0.2? I guess it would make sense if you only looked at wrong answers - that is, you wouldn't be asking "what's the probability of your answers matching those on the fake test" but rather "what's the probability of your wrong answers being wrong because you used the fake test". Since you are only looking at wrong answers, your probability is 1 in 4 instead of 1 in 5.

[1] At least, according to this calculator: https://www.socscistatistics.com/tests/binomial/default2.asp...