[davisoneee]

I think describing it as 'fake' is rather antagonistic. You could just as easily say 'poor quality' or 'unjustified'. I would say that more accurately describes the situation.

It could be...

    - unjustified and sloppy (not 'fake', but also not considered reliable evidence)
    - unjustified and malicious  (this i would consider 'fake')
    - unjustified and gamed (again, 'fake')
    - ...or just unjustified and under-specified (and would result in 'true' results if the conditions for replication were better studied and defined)

Lots of people like to think themselves smart for following 'first principles'...and then often end up falling in the same ditches. First principles + received wisdom is a bit of a contradiction... if it's 'wisdom' rather than evidence, you are skipping your principles to go with the received starting point...

[efitz]

The problem is, that as soon as a paper is published in a peer reviewed journal, it can be used to justify public policy positions. Then a large number of people will points to the study, or even worse, to press characterizations of what a person thinks the study says, to support their preference on the public policy position. Immediately, anyone who has an alternate position on the public policy will be accused of being a “science denier.”

Almost no one is interested in having an honest discussion about whether or not the original paper actually says what it’s characterized to have said, and whether it was a good study in the first place.

So nowadays, when public policy is concerned, largely I disregard any scientific study that is introduced to support any position on the policy, and just do my own cost – benefit trade-off to determine my policy position.

[Muller20]

This is a problem with the journalism and politics, it's not really about science. No scientist would trust a result that depends on a single small sample paper. Those are just stepping stones that may justify further research for more robust evidence. This fact is quite clear to scientist and it's why most would discourage the general public (including smart engineers) from reading academic articles.

But in general, I agree with you. It's ridiculous when someone pretends to shut down a complex issue by citing a random paper. However, an expert can still analyze the whole academic literature on a topic and determine what the scientific consensus is and how confident we are about it.

[adolph]

> The problem is, that as soon as a paper is published in a peer reviewed journal, it can be used to justify public policy positions.

That’s not a science problem. That’s a political problem. Making choices based on a single paper that may not be replicated is the problem.

[cynicalpeace]

Same here, and the more "the science" is used as a cudgel, the more people will resent it, whether or not they know a lot of it isn't even replicable.

[davisoneee]

You say you do a cost:benefit...but where do those costs come from? To me, that's just voluntarily doing your own ignorant, sloppy science I was mentioning above. If you only consider the blatantly obvious costs and benefits, you are completely ignorant of any 2nd or 3rd order...or even your own blind spots. You may radically under or overstate, or even calculate in the wrong direction.

I think a better position is that we should have a higher bar of what level of study or replication is required for a given situation...whether that be health, housing, infrastructure or whatever policy is coming in...what kind of monetary outlay and timeline of impact is expected. I don't think most people here would be happy with a 6-person study, unreplicated, deciding policy...so what IS the threshold?

[pvaldes]

And sometimes the opportunity window for studying a thing is just closed forever. Unique events are unique. That strange meteorite will not return until 300 years, or the species used in the experiment split in two, making impossible to duplicate it.

[cynicalpeace]

Fake is a good word. "unjustified and sloppy" is fake to me. We have different definitions of fake and shouldn't get hung up on definitions. My last word on it is I should be antagonistic against people who are deciding what goes in my son's school lunch based off of "unjustified and sloppy" studies.

First principles + received wisdom are counter balancing, not contradicting. Everything in life is a balance.

[davisoneee]

Definitions matter when it comes discussion, as what you say influences how people feel on a topic. Broad, non-specific definitions leave a lot of space for bias rather than clarity.

If you describe it as 'fake', I consider that to give the impression of 'the answer is NOT' this, and could lead to anti-policy.

If the description is 'unjustified and sloppy', that can lead to additional research to properly invalidate or potentially find something useful, so we don't throw the baby out with the bathwater.