[ikura]

From the article:

   Personally, I think the dichotomy between hypothesis-testing and likelihood-quantification is a false one. The “P=0.05” cutoff we use to “reject” a hypothesis is an arbitrary one. When I read papers, I never “accept” or “reject” hypotheses but rather consider likelihood quantification as a measure of the weight of evidence or a distance of the data from some null hypothesis, as measured by some statistic. I encourage everyone else to consider this probabilistic worldview when viewing our paper: we aimed to quantify probabilities of this system occurring in nature, and P-values were convenient and commonly understood ways of communicating quantiles.

This paragraph does a lot of lifting. Conflating p-values and probabilities is the science equivalent of a code smell.

[zosima]

Though p-values are probabilities.

They are the probability that the data seen (or more extreme) in the experiment were generated given the null-hypothesis is true.

Now, of course to fully understand the p, you also have to understand the null hypothesis. And yeah, sometimes it is misspecified. (by e.g testing out many null-hypotheses and only showing the more interesting ones, or accidentally creating a bad unlikely null-hypothesis which may allow for many uninteresting alternative hypotheses.)

[snake_doc]

Obligatory p-value snippet from the ASA:

   P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.

Ronald L. Wasserstein & Nicole A. Lazar (2016) The ASA Statement on p-Values: Context, Process, and Purpose, The American Statistician, 70:2, 129-133, DOI: 10.1080/00031305.2016.1154108

[comte7092]

To follow that up (so people know what they actually are), what p-values represent is the likelihood we would observe our data, given the null hypothesis.

Setting a cutoff of .05 is saying “if there’s less than a 5% chance we’d see this data, assuming the null hypothesis, then we can assume that the null hypothesis is false”

[fny]

I'm becoming more and more convinced we need to multiply anything that is not strictly a probability (CIs, ML model scores, p-values) by 100.

"I have a confidence of 95" has very different ring to it than "I am 95% confident."

It would also prevent people from doing stupid things like using these values to compute expectations.

[kgwgk]

A p-value is strictly _a_ probability.