[poomer]

Great article. This part especially rings true to me: "If researchers have an ideological bent, a meta-analytic null may just be an expression of the typical sentiments of researchers".

When I was in academia, it was increasingly the case that my peers thought of research less as a way to determine the truth, but just as a method to influence policy and public opinion. If we thought something was 80% likely to be true, there was pressure to "close ranks" and pretend as though it was 100% true, and to avoid publishing anything that contradicted it. It is also well known that papers that support certain "sides" tend to be easier to publish (and in higher ranked journals), plus can yield more media attention. See for example, this fraud in sociology - https://en.wikipedia.org/wiki/When_contact_changes_minds.

This may be better in the natural sciences, but in social science you should not trust any paper unless you read through and fully understand the methodology. Any non experimental results has so much wiggle room in the modeling methodology that it's easy to generate any result you want. The actual percentage of papers with credible results is very low, much lower than laypeople think.

[brightball]

Once, years ago, I got into a debate with somebody online who just kept dropping links to studies after making a statement as if it proved them correct.

One day, I got bored and decided to actually read every single study. The studies said nothing close to what the debater suggested and seriously opened my eyes to just how terrible some studies can be in terms of quality.

Now, I just assume that anybody who starts link bombing in a conversation has no idea what they're talking about and can't engage on a logical discussion.

[DontchaKnowit]

Yep. This is why it makes me nearly physically ill when I hear people use terms like "science denier" or say things like "<insert political party here> don't believe in science" or "our laws should be based on science".

It is invariably some of the most scientifically illiterate, ideologically entrenched, and intellectually lukewarm people who spout this garbage as a retort to any sort of argument with which they do not wish to engage.

[rahimnathwani]

When reading the first draft of the California Math Framework, I followed many of the citations. My experience was similar to yours. In particular:

- some of the cited studies did not claim what the citation said they did

- some of the cited studies did claim what the citation said they did, but the experimental results were far too weak to support these claims

- some of the cited studies had such weak experimental design, that it would be hard to conclude anything about the subject at hand

[indymike]

> One day, I got bored and decided to actually read every single study.

Nothing is more entertaining than the meeting after a business presentation where someone fact checks the reference after the power point. When we have vendor presentations, we started doing this because... so many claims, so little truth.

[svnt]

Maybe assess the argument by the first two studies. If either of them do not support the point (or as has frequently happened for me, actually oppose the point) then you can reply for posterity and no one else needs to replicate your debunking effort.

“Your first two studies not only don’t support your argument but invalidate it, and I stopped reading after that.” ends most disagreements.

You will have probably done more work than the poster who often has just discovered keyword searches and related articles in google scholar or science direct, but everyone will learn something.

[hospitalJail]

If you ever want to reject a study, just read it.

You will find something to nit pick.

[Salgat]

I'll often read through the studies in an argument but it is very unfeasible to keep up with these people because they can just keep throwing crap at you, and if you disagree with the study, they'll throw another at you until you give up. I remember reading through one 80 page paper on how a public option for healthcare would bankrupt us, but if you actually read through the thing, it never accounts for all the money saved by not being spent on private healthcare (which would fund the entire thing). My local power company had a similar situation where they posted a huge study explaining why solar was costing them a bunch of money, but if you actually read the study it explains that it considers lost income from solar as costing them money (in the same way that moving to a more efficient AC system would "cost" the power company money). It's all bullshit, and it's too easy to abuse studies to defend your position.

[sn9]

If someone replies within a few minutes with a link to a study, you can usually just assume it doesn't even support their point unless they're actually a researcher in the field.

It takes way longer even for experts to read and digest a paper thoroughly to evaluate the methodology and results.

Googling for 5 minutes and skimming the abstract for keywords that might be related to whatever you were arguing about is usually the norm online.

It takes years of training and feedback to learn to properly evaluate a paper in your own field, and that's after years of undergraduate education at least. Most people online skip even the basic textbook level background and think they understand what they're reading.

[beebmam]

What are some examples of the claims people made vs what the research actually demonstrated?

[njarboe]

It is sort of like how when some people want to prove something is true they quote the Bible. I find the behavior especially strange when they are quoting the bible to prove the bible is true.

[jfengel]

Sociology lives in an uncomfortable place at the intersection of "important" and "difficult". Forming a rigid experiment on human beings is vastly more difficult than a simple thing like an atom or a mineral, and so much worse with groups of human beings.

STEM people like to dismiss it because it doesn't produce the same kind of rigid results and is therefore useless. But sociology is nonetheless important. Like it or not, we have to make decisions about how the world will run, and the fact that we don't have perfect information doesn't let us opt out of that.

Combine that with all of the usual human failings -- pettiness, meanness, closed-mindedness, greed, etc -- it sounds impossible. But it does, slowly, gather data and formulate theories.

This isn't made better by the fact that most STEM people imagine they can read primary source material and understand it -- a mistake they wouldn't make for a "hard science" in a different discipline. Like every field, the frontiers are based on huge amounts of background material, which is even more vast for a field that's more complicated than the nice, neat laws of physics or chemistry.

None of that excuses those human failings of sociologists. They need to do better, and hold each other to account (and not just to foment their own failings in their place). But we do need to recognize that this work is important. The world is complex and difficult and we'll make better choices if we try to understand, rather than dismiss it as unknowable.

[DontchaKnowit]

While I agree that there is some work in the field that is important, there is an absolute deluge of ideologically driven garbage. There is also a lot of garbage of the "yeah, duh, of-fucking-course" variety. studies like "negative interactions with community reduce feelings of belonging" ... uh yeah, no shit sherlock. And then even the more important stuff cannot be investigated in the same rigorous manner as other scientific disciplines. I think we just need to stop calling sociology papers "scientific". Fundamentally, they are not, and should almost always be taken with a massive grain of salt, and they damn sure shouldn't be influencing public policy decisions to the degree that they currently do.

[jq-r]

I think I've learned the most about life as a citizen from the sociology lessons and a book written in late 70's or 80's. The stuff they've concluded then still stands today and I'm sure will still stand as long as there are humans. Such and eye-opening discipline and I remember it with fondness. Always surprised when people start bashing sociology for reasons unknown to me.

[getoffmycase]

It is not better in natural sciences. Alzheimer’s disease treatment based on the amyloid plaque hypothesis being the prime example. Basically researchers closed ranks and doled out grant money disproportionately to the amyloid hypothesis researchers.

[nathan_compton]

This is sort of like adding momentum to a gradient descent algorithm, though. It is rational congeal support around a plausible hypothesis to see if it pans out rather than pursue all hypotheses (whatever that would even mean) in a desultory fashion. In the case of amyloid plaques there may have been some academic misconduct involved as well.

Sometimes I feel like non-scientists have this idea that scientists should be super-rational actors and never be bamboozled or be wrong, but frankly, that isn't realistic. Scientists are fallible, have finite time, and are subject to social trends and pressures like the rest of us. Expecting that they always get it right is unrealistic and not good for science.

[whyenot]

If you are a physical anthropologist there are many questions you can not ask, and many findings that people do not want to hear. Tread carefully, because one wrong step can end your career.

[piuantiderp]

People relying on a salary are rarely free to pursuit the truth

[rqtwteye]

“ thought of research less as a way to determine the truth, but just as a method to influence policy and public opinion. If we thought something was 80% likely to be true, there was pressure to "close ranks" and pretend as though it was 100% true, and to avoid publishing anything that contradicted it. ”

That seems to be the general spirit of this time. People feel the need to take sides and then make sure their side wins. Same happens in journalism. Most journalism these days seems to be about supporting a viewpoint and less about conveying neutral or complete information.

Sad that scientists also feel that they should be activists.

[anonymouskimmer]

I'm not sure if that particular study is a good example of this as everything and its aunt was getting published about COVID back then. It is disappointing that other studies were shut down on the assumption that the mortality results from such a limited (and fatally flawed) analysis were correct, especially given previous knowledge of the side effects and dosing considerations of hydroxychloroquine.