[KeepFlying]

For typefaces you check for the distribution of similarities in each group. If it has large clusters then group by 3, 3', etc then run your outlier check on each of those groups

Still would risk some weirdness, but would help a bit I'd hope.

I wonder if it would be worth running some kind of language analysis or spelling/Grammer check to verify the scan too. At least for text, you'd need another solution for number tables.