After enjoying Sapolsky's lectures on YouTube (including this one) I began reading his book Behave. I was surprised to see that it refers to theories that fail to replicate (eg priming) as if they were solid theories. Stuart Richie (a psychologist who wrote a book on the replication crisis) enumerates other examples in his review https://www.spectator.co.uk/article/rules-of-behaviour/
Please be wary of this failure to drop unreplicated findings when reading Sapolsky's works.
I've sat this out for long time, but I've got to say that it's hilarious watching the HN community—mostly made up of engineers—constantly harp about the replication crisis in Psychology.
In some cases effect may be real even though the studies aren't replicating today. There are several factors.
Replication in psychology isn't as easy as in the physical sciences.
Let me use an example from another field to give you an idea. If I asked you to translate the sentence: "He took the train for London at eight am." into French, would you translate "London" as "London" or as "Paris?"
In other words the social context—social norms, expectations, attitudes—from when many of these studies which don't replicate today has shifted, such that re-running the study verbatim might fail to...replicate.
However say you do try to adapt the study to adjust for the shift in context, then you could also say that the replication isn't valid because you didn't re-run the study exactly as specified.
People aren't robots, so we can just re-run the unit tests on them and expect the exact same results. People aren't even self contained robots, we are social beings, and are very strongly affected by our peers. We aren't swimming in the same water that we were swimming in when these studies were done.
We need new studies for each of these effects that are designed with today's reality in mind. But. No matter how one tries to 'replicate' an old psychology result, it will leave room for skepticism. It just won't be able to account for the 'translation' required without leaving room for doubt.
I guess the most important thing we could learn from this is that it's important to replicate any current studies right now, and not wait forty years to do so.
No, the problem is a lot deeper than just "capturing the protocol in the methods section is hard". (There are very interesting articles about how word frequency shifting might be a confounding factor, which can cause old - from the 70s and 80s - papers to fail replication. But then there's no follow up with a new corpus and new replication.)
> I guess the most important thing we could learn from this is that it's important to replicate any current studies right now, and not wait forty years to do so.
Yes, that too, but what's even more important is to shift into a mindset that starts with good models, good data generation processes (ie. experiments), then we can check and compare their predictive power. Otherwise we get these statistically flawless abominations that prove ESP:
And, sure, yeah, it's hard to do this. But otherwise we'll have nothing more than just-so stories supported by random data that happened to break through some significance threshold.
The replication crisis isn't merely in psychology, old studies, or studies that didn't carefully record their methodology. Eg the review I linked mentions a study from 2006 that didnt replicate in a larger trial. The one about stories involving immorality priming people to use more antiseptic wipes. It's weird to me that someone would write a book and reference that study without mentioning this replication failure. Especially in the context of the many other such failures to replicate for priming.
Theories ought to be backed by experiments that replicate. Maybe describing methodology precisely is hard, but replication is how we know a phenomenon is actually real and not p-hacked or otherwise mistaken (eg by experimenters mis-measuring).
So what you're saying is all study designs are acceptable. All study results should be trusted. We should trust studies that can't be replicated, because "something about translating London into Paris".
What you're describing isn't science. If you want psychology to be held to a different standard, then that's fine. But then you can't call it science.
I agree on both (great books, great lectures, but not everything stands the test of time).
But the good thing is that the book is very well referenced. So whenever I wondered "Is that really true?" or simply "Why?" I was able to find the original paper(s)/source(s) and then take it from there. For me, that is the gold standard of writing a science book.
True, but Behave was published in 2017! That's ~7 years after people noticed these theories fail to replicate. So it's not that some parts of the book don't stand the test of time, it's that it doesn't accurately represent the state of the science for when it was published.
One of those citations brought me to "How to Think, Say, or Do Precisely the Worst Thing for Any Occasion", a study that show how the worst thing has a higher probability of occurring than random chance would make it. With a title like that I could just not read it.
Note that I didn't just read the word "priming" in the book and seize upon it generically. As noted in Ritchie's review, Sapolsky references specific studies which larger trials have failed to replicate (eg the Macbeth Effect where reading stories about unethical behavior made people more likely to grab antiseptic wipes).
I wouldn't give too much importance to Stuart Ritchie. He is right-wing and conservative, and conservatives hate Sapolsky.
Ritchie's political views are evident in his book.
I read Science Fictions and don't recall politics coming up much. Ironically, his most political point was his defense of the Mertonian norm of universalism from both leftist and rightist critiques. That norm is relevant in this discussion :P. That said, I haven't read his other book.
Politics aside, those theories did fail replication. Forget I mentioned Ritchie at all and the problem remains.
The replication crisis would be with us even if every right-wing pundit suddenly disappeared. Much like heliocentrism exists independent of Copernicus or the Church.
Replication crisis isn't political at its core, it is scientific.
gerbilly|4 years ago
In some cases effect may be real even though the studies aren't replicating today. There are several factors.
Replication in psychology isn't as easy as in the physical sciences.
Let me use an example from another field to give you an idea. If I asked you to translate the sentence: "He took the train for London at eight am." into French, would you translate "London" as "London" or as "Paris?"
In other words the social context—social norms, expectations, attitudes—from when many of these studies which don't replicate today has shifted, such that re-running the study verbatim might fail to...replicate.
However say you do try to adapt the study to adjust for the shift in context, then you could also say that the replication isn't valid because you didn't re-run the study exactly as specified.
People aren't robots, so we can just re-run the unit tests on them and expect the exact same results. People aren't even self contained robots, we are social beings, and are very strongly affected by our peers. We aren't swimming in the same water that we were swimming in when these studies were done.
We need new studies for each of these effects that are designed with today's reality in mind. But. No matter how one tries to 'replicate' an old psychology result, it will leave room for skepticism. It just won't be able to account for the 'translation' required without leaving room for doubt.
I guess the most important thing we could learn from this is that it's important to replicate any current studies right now, and not wait forty years to do so.
pas|4 years ago
It's not just bad (vacuous) science, at its core it's people who act in a bad (selfish, non-scientific) ways: https://statmodeling.stat.columbia.edu/2016/09/21/what-has-h...
> I guess the most important thing we could learn from this is that it's important to replicate any current studies right now, and not wait forty years to do so.
Yes, that too, but what's even more important is to shift into a mindset that starts with good models, good data generation processes (ie. experiments), then we can check and compare their predictive power. Otherwise we get these statistically flawless abominations that prove ESP:
https://statmodeling.stat.columbia.edu/2011/01/11/one_more_t...
And, sure, yeah, it's hard to do this. But otherwise we'll have nothing more than just-so stories supported by random data that happened to break through some significance threshold.
deskglass|4 years ago
Theories ought to be backed by experiments that replicate. Maybe describing methodology precisely is hard, but replication is how we know a phenomenon is actually real and not p-hacked or otherwise mistaken (eg by experimenters mis-measuring).
slingnow|4 years ago
What you're describing isn't science. If you want psychology to be held to a different standard, then that's fine. But then you can't call it science.
ChemSpider|4 years ago
But the good thing is that the book is very well referenced. So whenever I wondered "Is that really true?" or simply "Why?" I was able to find the original paper(s)/source(s) and then take it from there. For me, that is the gold standard of writing a science book.
deskglass|4 years ago
belorn|4 years ago
fulafel|4 years ago
deskglass|4 years ago
aritmo|4 years ago
deskglass|4 years ago
Politics aside, those theories did fail replication. Forget I mentioned Ritchie at all and the problem remains.
inglor_cz|4 years ago
Replication crisis isn't political at its core, it is scientific.