[curiousllama]

The core problem with this is:

- DON'T is very clear and specific. Don't say "Stat-Sig", don't conclude causal effect, don't conclude anything based on p>0.05.

- DO is very vague and unclear. Do be thoughtful, do accept uncertainty, do consider all relevant information.

Obviously, thoughtful consideration of all available information is ideal. But until I get another heuristic for "should I dig into this more?" - I'm just gonna live with my 5-10% FPR, thank you very much.

[truculent]

> But until I get another heuristic for "should I dig into this more?"

Why do you need a heuristic? In what areas are you doing research where you don't have any other intuition or domain knowledge to draw on?

And if you don't have that background, contextual knowledge, are you the right person to be doing the work? Are you asking the right questions?

[kidel001]

bioinformatician here. nobody has intuition or domain knowledge on all ~20,000 protein coding genes in the human body. That's just not a thing. Routinely comparing what a treatment does we do actually get 20,000 p-values. Feed that into FDR correction, filter for p < 0.01. Now I have maybe 200 genes.Then we can start applying domain knowledge. If you start trying to apply domain knowledge at the beginning, you're actually going to artificially constrict what is biologically possible. Your domain knowledge might say well there's no reason an olfactory gene should be involved in cancer, so I will exclude these (etc etc). You would be instantly wrong. People discovered that macrophages (which play a large role in cancer) can express olfactory receptors. So when I had olfactory receptors coming up in a recent analysis... the p values were onto something and I had to expand my domain knowledge to understand. This is very common. I ask for validation of targets in tissue --> then you see proof borne out that the p-value thresholding business WORKS.

[joe_the_user]

Yeah, it seems like those bullet point have the problem that they don't really contain actionable information.

Here's the way I'd put things - correlation by itself does causation at all. You need correlation plus a plausible model of the world to have a chance.

Now science, at its best, involves building up these plausible models, so a scientist creates an extra little piece of the puzzle and has to be careful also the piece is a plausible fit.

The problem you hit is that the ruthless sink-or-swim atmosphere, previous bad science and fields that have little merit make it easy to be in the "just correlation" category. And whether you're doing a p test or something else doesn't matter.

A way to put is that a scientist has to care about the truth in order to put together all the pieces of models and data in their field.

So the problem is ultimately institutional.