All too often debates on best scientific practices descend into a chaotic mire of accusations, hurt feelings, and offended egos. As I mentioned in my previous post, I therefore decided to write a list of guidelines that I believe could help improve scientific discourse. It applies as much to replication attempts as to other scientific disagreements, say, when reanalysing someone else’s data or spotting a mistake they made.
Note that these are general points and should not be interpreted as relating to any particular case. Furthermore I know I am no angel. I have broken many of these rules before and fallible as I am I may end up doing so again. I hope that –I- learn from them but I hope they will also be useful for others. If you can think of additional rules please leave a comment!
1. Talk to the original authors
Don’t just send them a brusque email requesting data or with some basic questions about their paradigm (if you want to replicate their research). I mean actually talk and discuss with them, ideally in person at a conference or during a visit perhaps. Obviously this won’t always be possible. Either way, be open and honest about what you’re doing, why you want the data, why you want to replicate and how. Don’t be afraid to say that you find the results implausible, especially if you can name objective reasons for it.
One of my best experiences at a conference was when a man (whose name I unfortunately can’t remember) waited for me at my poster as I arrived at the beginning of my session. He said “I’m very skeptical of your results”. We had a very productive, friendly discussion and I believe it greatly improved my follow-up research.
2. Involve the original authors in your efforts
Suggest to the original authors to start a collaboration. It doesn’t need to be “adversarial” in the formal sense although it could be if you clearly disagree. But either way everyone will be better off if you work together instead of against each other. Publications are a currency in our field and I don’t see that changing anytime soon. I think you may be surprised how much more amenable many researchers will be to your replication/reanalysis effort if you both get a publication out of it.
3. Always try to be nice
If someone makes an error, point it out in an objective but friendly manner. Don’t let that manuscript you submitted/published to a journal in which you ridicule their life’s work be the first time they hear about it. Avoid emotional language or snarky comments.
I know I have a sarcastic streak so I have been no saint when it comes to that. I may have learned from a true master…
4. Don’t accuse people without hard evidence
Give people the benefit of the doubt. Don’t just blankly insinuate the original authors used questionable research practices no matter how wide-spread they apparently are. In the end you don’t know they engaged in bad practices unless you saw it yourself or some trustworthy whistle-blower told you. Statistics may indicate it but they don’t prove it.
5. Apologise even if (you think) you did no wrong
You might know this one if you’re married… We all make mistakes and slip up sometimes. We say things that come across as more offensive than intended. Sometimes we can find it very hard to understand what made the other person angry. Usually if you try you can empathise. It also doesn’t matter if you’re right. You should still say sorry because that is the right thing to do.
I am very sorry if I offended Eric-Jan Wagenmakers or any of his co-authors with my last post. Perhaps I spoke too harshly. I have the utmost respect for EJ and his colleagues and I do appreciate that they initiated the discussion we are having!
6. Admit that you could be wrong
We are all wrong, frequently. There are few things in life that I am very confident about. I believe scientists should be skeptical, including of their own beliefs. I remain somewhat surprised just with how much fervent conviction many scientists argue. I’d expect this from religious icons or political leaders. To me the best thing about being a scientist is the knowledge that we just don’t really know all that much.
I find it very hard to fathom that people can foretell the future or that their voting behaviour changes months later just because they saw a tiny flag in the corner of the screen. I just don’t think that’s plausible. But I don’t know for sure. It is possible that I’m wrong about this. It has happened before.
7. Don’t hound individuals
Replicating findings in the general field is fair enough. It is even fair enough if your replication is motivated by “I just cannot believe this!” even though this somewhat biases your expectations, which I think is problematic. But if you replicate or reanalyse some findings try not to do it only to the same person all the time. This looks petty and like a vendetta. And take a look around you. Even if you haven’t tried to replicate any of Research X’s findings, there is a chance a lot of other people already have. Don’t pester that one person and force them into a multi-front war they can’t possibly win with their physical and mental health intact.
This is one of the reasons I removed my reanalysis of Bem 2011’s precognition data from my bootstrapping manuscript.
8. Think of the children!
In the world we currently live in, researcher’s livelihoods depend on their reputation, their publication record and citations, their grant income etc. Yes, I would love for grant and hiring committees to only value trust-seekers who do creative and rigorous research. But in many cases it’s not a reality (yet?) and so it shouldn’t be surprising when some people react with panic and anger when they are criticised. It’s an instinct. Try to understand that. Saying “everyone makes mistakes” or “replication is necessary” isn’t really enough. Giving them a way to keep their job and their honour might (see rule 2).
9. Science should seek to explain
In my opinion the purpose of scientific research is to explain how the universe works (or even just that tiny part of the universe between your ears). This is what should motivate all our research, including the research that scrutinises and/or corrects previous claims. That’s why I am so skeptical of Many Labs and most direct replication projects in general. They do not explain anything. They are designed to prove the null hypothesis, which is conceptually impossible. It is fine to disbelieve big claims, in fact that’s what scientists should do. But if you don’t believe some finding, think about why you don’t believe it and then seek to disprove it by showing how simpler explanations could have produced the same finding. Showing that you, following the same recipe as the original authors, failed to reproduce the same result is pretty weak evidence – it could just mean that you are a bad cook. In general, sometimes we can’t trust our own motivations. If you really disbelieve some finding, try to think what kind of evidence could possibly convince you that it is true. Then, go and test it.