top | item 44982978

(no title)

jpcompartir | 6 months ago

^

And if we increase N enough we will be able to find these 'good measurements' and 'statistically significant differences' everywhere.

Worse still if we did not agree in advance what hypotheses we were testing, and go looking back through historical data to find 'statistically significant' correlations.

discuss

order

ants_everywhere|6 months ago

Which means that statistical significance is really a measure of whether N is big enough

kqr|6 months ago

This has been known ever since the beginning of frequentist hypothesis testing. Fisher warned us not to place too much emphasis on the p-value he asked us to calculate, specifically because it is mainly a measure of sample size, not clinical significance.

energy123|6 months ago

It's not, that would be quite the misunderstanding of statistical power.

N being big means that small real effects can plausibly be detected as being statistically significant.

It doesn't mean that a larger proportion of measurements are falsely identified as being statistically significant. That will still occur at a 5% frequency or whatever your alpha value is, unless your null is misspecified.