[andy_wrote]

I think the part that you have correctly included that people forget or elide is that it's the probability under a specific null hypothesis. So it's a function of what you have chosen for that - normal distributions, a certain parameter value of 0, etc. So this means that a) it's not the probability you'd see in the real world under repeated performance b) it's not the probability under other reasonable null hypotheses. Like maybe under the hypothesis parameter = 0 you get an improbably large p-value, but under parameter = 0.1, or with different assumed underlying distributions, you wouldn't see something so extreme.

I'd guess that the original writer understands this, and that Gelman is only pointing it out because casual readers sometimes don't mentally retain the full baggage that the p-value carries.