How funders could encourage replication efforts

As promised, here is a post about science stuff, finally back to a more cheerful and hopeful topic than the dreadful state the world outside science is in right now…

A Dutch research funding agency recently announced a new grant initiative that exclusively funds replication attempts. The idea is to support replication efforts of particularly momentous “cornerstone” research findings. It’s not entirely clear what this means but presumably such findings include highly cited findings, those with great media coverage and public policy impact etc. It isn’t clear who determines whether a finding falls under this.

You can read about this announcement here. In that article you can see some comments by me on how I think funders should encourage replications by requiring that new grant proposals should also contain some replication of previous work. Like most people I believe replication to be one of the pillars supporting science. Before we treat any discovery as important we must know that it is reliable and meaningful. We need to know in how far it generalizes or if it is fickle and subject to minor changes in experimental parameters. If you read anything I have written about replication, you will probably already know my view on this: most good research is built on previous findings. This is how science advances. You take some previously observed results and use it to generate new hypotheses to be tested in a new experiment. In order to do so, you should include a replication and/or sanity check condition in your new experiment. This is precisely the suggestion Richard Feynman made in his famous Cargo Cult Science lecture.

Imagine somebody published a finding that people perceive the world as darker when they listen to sad classical music (let’s ignore for the moment the inherent difficulty in actually demonstrating such an effect…). You now want to ask if they also perceive the world as darker when they listen to dark metal. If you simply run the same experiment but replace the music any result you find will be inconclusive. If you don’t find any perceptual effect, it could be that your participant sample simply isn’t affected by music. The only way to rule this out is to also include the sad classical music condition in your experiment to test whether this claim actually replicates. Importantly, even if you do find a perceptual effect of dark metal music, the same problem applies. While you could argue that this is a conceptual replication, if you don’t know that you could actually replicate the original effect of classical music, it is impossible to know that you really found the same phenomenon.

My idea is that when applying for funding we should be far more explicit about how the proposal builds on past research and, insofar this is feasible, build more replication attempts into the proposed experiments. Critically, if you fail to replicate those experiments, this would in itself be an important finding that should be added to the scientific record. The funding thus implicitly sets aside some resources for replication attempts to validate previous claims. However, this approach also supports the advance of science because every proposal is nevertheless designed to test novel hypotheses. This stands in clear contrast between pure replication efforts such as those this Dutch initiative advocates or the various large-scale replication efforts like the RPP and Many Labs project. While I think these efforts clearly have value, one major concern I have with them is that they seem to stagnate scientific progress. They highlighted a lack of replicability in the current literature and it is undoubtedly important to flag that up. But surely this cannot be the way we will continue to do science from now on. Should we have a new RPP every 10 years now? And who decides which findings should be replicated? I don’t think we should really care whether every single surprising claim is replicated. Only the ones that are in fact in need of validation because they have an impact on science and society probably need to be replicated. But determining what makes a cornerstone discovery is not really that trivial.

That is not to say that such pure replication attempts should no longer happen or that they should receive no funding at all. If anyone is happy to give you money to replicate some result, by all means do so. However, my suggestion differs from these large-scale efforts and the Dutch initiative in that it treats replication the way it should be treated, as an essential part of all research, rather than as a special effort that is somehow separate from the rest. Most research would only be funded if it is explicit about which previous findings it builds on. This inherently also answers the question which previous claims should be replicated: only those findings that are deemed important enough by other researchers to motivate new research are sufficiently important for replication attempts.

Perhaps most crucially, encouraging replication in this way will help to break down the perceived polarization between the replicators and original authors of high-impact research claims. While I doubt many scientists who published replications actually see themselves as a “replication police,” we continue to rehash these discussions. Many replication attempts are also being suspected to be motivated by mistrust in the original claim. Not that there is really anything wrong with that because surely healthy skepticism is important in science. However, whether justified or not, skepticism of previous claims can lead to the perception that the replicators were biased and the outcome of the replication was a self-fulfilling prophecy. My suggestion would mitigate this problem at least to a large degree because most grant proposals would at least seek to replicate results that have a fighting chance of being true.

In the Nature article about this Dutch initiative there are also comments from Dan Gilbert, a vocal critic of the large-scale replication efforts. He bemoans that such replication research is based on its “unoriginality” and suspects that we will learn more about the universe by spending money on “exploring important new ideas.” I think this betrays the same false dichotomy I described above. I certainly agree with Gilbert that the goal of science should be to advance our understanding of the world but originality is not really the only objective here. Scientific claims must also be valid and generalize beyond very specific experimental contexts and parameters. In my view, both are equally important for  healthy science. As such, there is not a problem with the Dutch initiative but it seems rather gimmicky to me and I am unconvinced its effects will be lasting. Instead I believe the only way to encourage active and on-going replication efforts will be to overhaul the funding structure as I outlined here.

320px-flag_of_europe-svg
52% seems barely above chance. Someone should try to replicate that stupid referendum.

2 thoughts on “How funders could encourage replication efforts

  1. “My idea is that when applying for funding we should be far more explicit about how the proposal builds on past research and, insofar this is feasible, build more replication attempts into the proposed experiments. Critically, if you fail to replicate those experiments, this would in itself be an important finding that should be added to the scientific record. The funding thus implicitly sets aside some resources for replication attempts to validate previous claims. However, this approach also supports the advance of science because every proposal is nevertheless designed to test novel hypotheses. ”

    I am not sure this works for a few reasons. Here are my thoughts and questions:

    1) It seems to me to be impossible to replicate all the experiments/papers scientists cite to make their case. How many findings are usually cited in the introduction, 10? 15? 20?. Should they all be replicated? How should a researcher decide which findings that they “build upon” should be replicated?

    2) It seems to me that it when deciding to test “new” hypotheses/come up with a new experiment, there is the assumption that the work that is used to build upon is replicable/reliable. However, this assumption is not yet tested, hence your proposal for incorparating replications in the proposal. Would it not make sense to first investigate which findings are replicable, and only then come up with a new experiment (perhaps on an entire new subject based on the possibility of the now apparent non-replicability of the previous findings)?

    Based on 1) and 2) i reason that it makes more sense to first find out which findings are replicable, and only then come up with new experiments based upon those previous findings. Combining these 2 processes into 1 grant proposal opens the possibility that the 2nd part of your proposal (coming up with new experiments) will be useless, or at best sub-optimal.

    Like

    1. Thanks for your comment. I think you’re overestimating the complications with this idea. Most good research already does this anyway. Perhaps it’s not always as explicit as what I am suggesting but it is certainly what a solid proposal should do anyway. My idea extends this somewhat because it makes it more obvious that some research money can be used to also publish any failure to replicate. The way I see it, it should be an explicit criterion in reviewing a grant proposal whether and how it seeks to replicates previous findings. This should be incorporated in any proposal but there could also be a short section in which you explicitly list the conditions that are replications.

      The way current grant proposals often work is that people present pilot or preliminary results but it is not always obvious in how far those findings are conclusive at all. I think we need to get away from that. Under my concept, we would at least base our grants more on already published results than on lofty pilot concepts.

      I also don’t think the final point you raise is really a problem. A good grant proposal includes contingency plans or makes it clear in how far the larger proposal depends on the success of the initial stages. Grants that can’t continue because experiment 1 is a null-result are unlikely to get funded. The same would apply here. Whenever you include an explicit replication in a proposal you should outline what you will do with that finding. If it succeeds you can test your new hypothesis. If it fails to replicate, this suggest the original result could have been a fluke which in turn opens up other branches of investigation.

      Finally, as I explained, even if you “first investigate which findings are replicable” you still have to replicate them in the new experiment you do. Without that any new experiment you do is inconclusive! If someone previously showed that X causes Y and you want to test if X+A causes Z you need to also confirm that X does cause Y under the same conditions. Sometimes it is perhaps more feasible to do that separately first, but it is more efficient and more scientifically valuable to show it in the same group at the same time.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s