Scientists are human beings. I get it. I really do because – all contrary reports and demonic possessions aside – I’m a human being, too. So I have all manner of sympathy for people’s hurt feelings. It can hurt when somebody criticizes you. It may also be true that the tone of criticism isn’t always as it should be to avoid hurt.
In this post, I want to discuss ways to answer scientific criticism. I haven’t always followed this advice myself because, as I said, I’m human. But I am at least trying. The post was sparked by an as-yet unpublished editorial by a certain ex-president of the APS. I don’t want to discuss specifically the rather inflammatory statements in that article as doing so will serve no good. Since it isn’t officially published, it may still change anyway. And last time I blogged about an unpublished editorial I received a cease and desist letter forcing me to embargo my post for two full hours…
I believe most people would agree that science is an endeavor of truth seeking. It attempts to identify regularities in our chaotic observations of the world that can help us understand the underlying laws that govern it. So when multiple people are unable to replicate a previous claim, this casts doubt on the claim’s validity as a regularity of nature.
The currency of science should be evidence. Without any evidence, a claim is worthless. So if someone says “I don’t think this effect is real” but offers no evidence for that statement, be it a failed replication or a reanalysis of the same data showing the conclusions are erroneous, then you have every right to ignore them. But if they do offer evidence, this cannot be ignored. It is simply not good enough to talk about “hidden moderators” or complain about the replicators’ incompetence. Without evidence, these statements are hollow.
Whether you agree with it in principle or not, preregistration of experimental designs has become something of a standard in replication studies (and is becoming increasingly common in general). So when faced with a replication failure and the fact that people of that ilk are evidently worried about analytical flexibility and publication bias, surely it shouldn’t be very surprising that they won’t just be convinced by your rants about untested moderators or Google searches of ancient conceptual replications, let alone by your accusations of “shameless bullying” or “methodological terrorism”. Instead, what might possibly convince them is a preregistered and vetted replication attempt in which you do right all of the things that these incompetent buffoons did wrong. This proposal has already been outlined very well recently by Brent W Roberts. Speaking more generally, it is the ground-breaking, revolutionary concept that scientific debates should be fought using equivalent evidence instead of childish playground tactics and special pleading.
Granted, some might not be convinced even by that. And that’s fine, too. Skepticism is part of science. Also, some people are not convinced by any evidence you show them. It is actually not your job as a scientist to convince all your critics. It is your job to test theories and evaluate the evidence unimpassionately. If your evidence is solid, the scientific community will come around eventually. If your evidence is only shouting about hidden moderators and nightmare stories of people fearing tenure committees because someone failed to replicate your finding, then I doubt it will pass the test of time.
And maybe, just maybe, science is also about changing your mind when you realize that the evidence simply doesn’t support your previous thinking. I don’t think any single failed replication is enough to do that but a set of failed replications should certainly at least push you in that direction. As far as I can see, nobody who ever published a replication failure has even suggested that people should be refused tenure or lose their research program or whatever. I can’t speak for others, but if someone applied for a job with me and openly discussed the fact that a result of theirs failed to replicate and/or that they had to revise their theories, this would work strongly in their favor compared to the candidate with overbrimming confidence who only published Impact Factor > 30 papers, none of which have been challenged. And, in a story I think I told before, one of my scientific heroes was a man who admitted without me bringing it up that the results of his Science paper had been disproven.
Seriously, people, get a grip. I am sympathetic to the idea that criticism hurts, that we should perhaps be more mindful of just how snarky and frivolous we are with our criticism, and that there is a level of glee associated with how replication failures are publicized. But there is also a lot of glee with which high impact papers are being publicized and presented in TED talks. If you want the former to stop, perhaps we should also finally put an end to the bullshitting altogether. Anyway, I will conclude with a quote by another of my heroes and let my unbridled optimism flow, in spite of it all:
In science it often happens that scientists say, ‘You know that’s a really good argument; my position is mistaken,’ and then they would actually change their minds and you never hear that old view from them again. They really do it. It doesn’t happen as often as it should, because scientists are human and change is sometimes painful. But it happens every day. I cannot recall the last time something like that happened in politics or religion.
– Carl Sagan