Six flawed arguments for leaving the EU

As anyone who reads this blog probably knows, the UK will hold a referendum about its continued membership in the EU later this month, on 23rd June. I already discussed my views on this in my previous post, so I won’t go into any depth on that here. The discussion is raging, not only in the media but no doubt in many family homes and workplaces (would be curious to be a fly on the wall when Boris Johnson and his brother, science minister Jo Johnson, talk about this in private…). I do think I have said most that I can say about it already – but I keep hearing the same tired, naïve arguments over and over. So I’ll write something about it, one last time before putting my future career, my civil rights, and most likely my continued life in this country in the hands of voters. Here I address six flawed arguments for leaving the EU:

1. “It will change everything”

Actually, most likely nothing major will happen at all. By far the most likely scenario is that the UK leaves the EU, and then joins the EEA in which free movement of people remains in turn for having full access to the single market. EU citizens in the UK will retain the same rights they had previously. Parliament comprises a large number of MPs from Labour, the Lib Dems, the SNP, and one (I think?) from the Greens, plus a healthy number of Europhile Conservatives. This means that this outcome is essentially guaranteed, at least until the next general election (and even then it seems highly unlikely that this situation will change dramatically). Of course, the UK would nevertheless give up its rights to influence EU policy. Sounds like a rotten deal to me. Anyway, leaving this aside, in the remainder of this post I will pretend that a vote to leave the EU will mean also an end to freedom of movement, which is the illusory scenario the Leave campaign is  peddling.

(Update 11 June 2016: The above EEA scenario of course assumes that the UK is allowed to remain in the single market. Wolfgang Schäuble seems to think that isn’t going to happen. I don’t agree with Schäuble much about anything but then again he is also highly influential in EU politics so it’s difficult to know what to think about his argument.)

2.”We can spend the money we save on UK science”

One reason I and many scientists are vehemently opposing this nostalgic independence nonsense is that a great deal of British science funding comes from the EU and that science in the UK would suffer if that were lost. An oft-repeated counterargument to this is that by leaving the EU the UK would no longer pay contributions to European funds and could thus use those savings to spend on British science. This is based on false economy and wishful thinking. The UK brings in more science funding than it pays in, so it would have to increase its science funding. When was the last time a British government did that? Do you honestly think it is likely they will do that now? Of course this argument is not even taking into account the strain on the economy now. It also ignores the likely hit the economy will take after leaving which will reduce and quite possibly wipe out any potential savings. And it blatantly neglects the substantial cost that the UK must pay to leave the EU in the first place. None of these things suggest there will be lots of spare pennies to fund UK research and development. (For similar reasons I also don’t believe this money will be used for the NHS or building homes but that’s outside the scope of my post).

3. “We will be free of EU bureaucracy”

Science has always been collaborative and it is increasingly so in our age. We need international science projects and the EU science initiatives (which go well beyond EU member states) can facilitate this far better than any single national body could. So the UK will quite likely continue to contribute to those initiatives, just as other non-EU countries (like Switzerland) are contributing – without any say in its direction.

4. “Scientists can still collaborate”

Funding is a big factor in science and the cynics on the Leave side are probably right that it is one of the driving factors why all vice-chancellors and governing bodies of British universities want the UK to stay in the EU. But it’s not just about that. Because science is collaborative and international, universities and research centres are usually extremely multinational. This may be especially true in English-speaking countries and this ability to attract bright minds from all over the world is what boost British science output (e.g. a large proportion of research grants brought to UK universities are brought in by people who are not UK citizens). You do not help this by putting up barriers. Leave campaigners like to talk about “point-based immigration systems” that would allow the UK to hire people in professions it needs and that makes it possible for excellent students to come here. Sure, because the best thing is always to have more bureaucracy and paperwork! That will doubtless attract great applicants who could instead be free to move to Paris, Berlin – or Dublin.

5. “EU citizens already living here can stay”

Much of this referendum debate has focused on immigration. Recent years have seen unprecedented immigration of people from other EU nations (although this still only accounts for around half of overall immigration to the UK). It is not surprising that this could cause some issues and concerns. More people making demands on the health system, on housing, or on jobs may strain the country’s capacity. Stopping EU immigration dead in its tracks will perhaps relieve this strain – however, one question Leave campaigners steadfastly ignore to address is what happens to the people who are already here. Unless they all pack and leave voluntarily on 24th June they will still put a strain on the capacity for some time to come. One argument I often hear is “nobody will be kicked out”. However, non-EU citizens are being deported left and right, sometimes for ludicrous reasons and in ludicrous ways. Under the Reign of Terroresa May, neither having a doctorate nor a British spouse necessarily protect you from this. Unless some sort of special agreement is negotiated, the same rules will apply to EU citizens if the UK leaves the EU. There is a lot of conflicting information out there, the most insidious of which is blatant (but presumably lucrative?) scare-mongering by law firms pushing people to apply for citizenship. Now, I don’t think many EU citizens will be deported, especially not those who are already settled here. But Leave campaigners show an obvious disconnect: On the one hand, they seem to believe that by leaving the EU the burden on the NHS and housing is magically lifted. On the other hand, they (at least the sane ones) maintain that there won’t be any mass-deportation of the very people they blame for this burden.

6. “We will regain our sovereignty”

The UK still is, and remains to be, a sovereign nation insofar that such a thing exists in this globalised world. I wasn’t overly impressed by David Cameron’s performance in that cringe-worthy ITV townhall meeting but one compelling answer he gave is that voting to Leave the EU will give an illusion of independence from foreign powers whilst sacrificing actual influence on the world and European stage. I call this the Libertarian Fallacy because it is the same faulty logic that leads many self-declared Libertarians to oppose all sorts of policies in the name of “liberty” without achieving any individual freedom at all. It’s the reasoning that allows some to decry background checks on guns as tyranny but sees no problem with strict tests for driving licenses. It’s the cognitive dissonance in which citizen ID cards evoke the spectre of fascist dictatorship but nobody worries about the far less controlled surveillance via credit card transactions or online activities. Whatever utopian dreams you may have about a “sovereign” UK after EU exit, it will lose its seat at the table and have reduced sway in any decision-making process in Europe – and by extension also in the world. Perhaps it’s fine with many to be an isolated island in a big sea dominated by China and the US, and a new Russian empire rattling its sabres. Fine, not all nations need to be world players. Perhaps these big guys will even leave you in peace. But don’t think for a second that by leaving the EU Britannia will rule the waves again.

Uncertain times

On 23rd June 2016 the citizens of the United Kingdom (plus immigrants from Commonwealth nations and the Republic of Ireland) will vote to decide if the UK should remain a member of the European Union. Colloquially, this is known as the “Brexit” debate. I refuse to use this horrible term again, not only because it sounds like a breakfast cereal but also because it’s a misnomer: the decision is not about Britain but the whole of the UK.

Let me be straight: I am a strong and vocal supporter of the UK staying in the EU. As an immigrant (I also won’t use the offensive term “migrant”) from a EU country, a Leave vote would have direct consequences for my life in the country I called home for almost two decades. I would inevitably lose some of the rights I have enjoyed since then. The EU is what made it possible for me to study, live and work here, it allowed me to spend a year in yet another EU country during my studies, and it made my life easier in countless ways, not least of all the simplicity of crossing the borders. All of these things apply to all EU citizens so from a purely selfish perspective all of them should also support it. A lot of things we take for granted are a direct consequence of the civil liberties the EU guarantees.

The whole public debate surrounding this issue has been characterised by panic mongering and inane bickering from both sides. From deliberate obfuscations (“If we leave the EU every household will lose £4,300!”) to outright lies (“We pay £350 million a week to Brussels!”) both camps are painting nightmare scenarios of what will happen if the other side wins. Add to that all of the rubbish Boris Johnson dreams up on a typical day that is too delusional to even constitute a lie.

The truth of the matter is this: nobody has a damn clue what will happen if the UK leaves the EU. There is no precedent for a country leaving the bloc. Some European countries like Switzerland, Norway, and Liechtenstein are not EU members so they can give us an idea of what the relationship between the UK and the remainder of the EU could look like in the future. However, these are all also very different countries than the UK. They have far smaller populations, have very different economies and societies, and they have existed outside the EU whilst the UK has for decades been an integral, albeit reluctant, member. The only thing Liechtenstein has in common with the UK is its national anthem. The only thing we do know is that most of these countries have a relationship with the EU that permits free movement of people. Since one of the main arguments put forth in support of leaving the EU is “regaining control of our borders,” it actually seems very unlikely that any dramatic change in border control can be achieved this way. Nevertheless, we can’t know what will happen.

This is why it worries me and why it should worry you also. The future is very uncertain and leaving the EU will be a very big risk. The doomsday scenarios painted by either side are extreme and frankly also insulting our collective intelligence. Anyone who tells you what terrible consequences a Leave or Remain decision will have, is either lying or – at best – completely delusional. In any case, you shouldn’t listen to them. The Remain camp have one thing going for them though: they support the status quo and voting to Remain is the conservative decision. Whatever propaganda Leave proponents may spout about it, it is unlikely that any of their horrors are becoming a reality if the UK stays in the EU. Rather, things will presumably not really change much from how they are now. Voting Leave is by far the riskier and more radical thing to do. That said, I doubt it will have disastrous consequences either. The initial negotiations will be difficult and cumbersome. The process of leaving will also cost a lot of money, at least in the short run, money that won’t be saved by not paying into EU coffers (for one thing because the UK will continue to pay while these negotiations take place). For science and technology, I believe the consequences will be painful as it will make it much harder to access large European research grants (which the UK also cannot simply make up for by no longer paying in) and – what is worse – the added bureaucracy and administrative burden of turning skilled workers from EU countries into immigrants from overseas, which will stifle collaboration to some extent. So no, leaving the EU will probably not to ruin the UK. But don’t fool yourself into thinking that it won’t have bad consequences.

As far as I am concerned, I recently I started the process of naturalisation to become a British citizen. I had originally wanted to complete this before the referendum so I could vote in it. I held back on this for ages because for many years my country of origin did not allow dual citizenship (guess which supernational organisation is to thank for that being possible now?) and also because it’s damned expensive. Now I am too late to get there before 23rd June. If the UK stays in the EU, I will most likely finish the process. This place is my home and I am tired of being subject to taxation without representation. I feel it’s about time I can fully shape the future of this country with my votes.

Of course, gaining citizenship will be far more useful if the UK indeed leaves the EU. People like me would most likely lose the right to vote in other elections we could vote in until now. It is far less clear how residence rights will change. As I already said, if the UK remains in the EEA, free movement rights are unlikely to be affected at all. But if “control of the borders” is “regained” this would change. Will our automatic permanent resident status be carried over? It does seem improbable that we would suddenly be asked to apply for visas or indefinite leave as such a change would result in complete chaos. It is quite likely though that some additional bureaucratic hurdles would be erected because that’s just what governments do. But honestly, I don’t think I’ll go through with naturalisation if the UK leaves the EU. Melodramatic as it sounds (because screw it, this whole debate has been plagued with melodrama and over-emotional rhetoric) a vote to Leave is a statement that people like me are not really welcome in this country. If the UK votes to Leave, I will most likely choose to leave the UK.

But don’t get me wrong. I do think this referendum is a good idea. For one thing, it is democratic. More importantly, I don’t actually think that Britons will vote to Leave. The polls suggest a close race but the Remain camp has been steadily ahead. As the graph below shows,  the fewer people answer “Don’t know” in a poll, the farther the Remain vote is ahead of the Leave vote. It is probably simplistic to interpret too much into this because this must depend on the particular poll but it could support the interpretation that people vote conservatively. Undecided voters are unlikely to choose the radical option on referendum day.

In my view, this referendum is actually way overdue. If the Leaves have it, then yes, the British will have spoken that European integration has gone too far for them. But if the Remain votes win this will hopefully finally put to rest a common Eurosceptic assertion that the UK originally only “voted to join the Common Market.”  It is about bloody time that this discussion moved into the 21st century.

EuroPolls
Difference between Remain and Leave votes compared to the percentage of Don’t know votes. (Polls without Don’t know votes have been removed). Source: Poll of Polls

 

3 scoops of vanilla science in a low impact waffle please

A lot of young[1] researchers are worried about being “scooped”. No, this is not about something unpleasantly kinky but about when some other lab publishes an experiment that is very similar to yours before you do. Sometimes this is even more than just a worry and it actually happens. I know that this could be depressing. You’ve invested months or years of work and sleepless nights in this project and then somebody else comes along and publishes something similar and – poof – all the novelty is gone. Your science career is over. You will never publish this high impact now. You won’t ever get a grant. Immeasurable effort down the drain. Might as well give up, sell your soul to the Devil, and get a slave job in the pharmaceutical industry and get rich[2].

Except that this is total crap. There is no such thing as being scooped in this way, or at least if there is, it is not the end of your scientific career. In this post I want to briefly explain why I think so. This won’t be a lecture on the merits of open science, on replications, on how we should care more about the truth than about novelty and “sexiness”. All of these things are undoubtedly true in my mind and they are things we as a community should be actively working to change – but this is no help to young scientists who are still trying to make a name for themselves in a system that continues to reward high impact publications over substance.

No. Here I will talk about this issue with respect to the status quo. I think even in the current system, imperfect as it may be, this irrational fear is in my view unfounded. It is essential to dispel these myths about impact and novelty, about how precedence is tied to your career prospects. Early career scientists are the future of science. How can we ever hope to change science for the better if we allow this sort of madness to live on in the next generation of scientists? I say ‘live on’ for good reason – I, too, used to suffer from this madness when I was a graduate student and postdoc.

Why did I have this madness? Honestly I couldn’t say. Perhaps it’s a natural evolution of young researchers, at least in our current system. People like to point the finger at the lab PIs pressuring you into this sort of crazy behaviour. But that wasn’t it for me. For most of my postdoc I worked with Geraint Rees at UCL and perhaps the best thing he ever told me was to fucking chill[3]. He taught me – more by example than words – that while having a successful career was useful, what is much more important is to remember why you’re doing it: The point of having a (reasonably successful) science career is to be able to pay the rent/mortgage and take some enjoyment out of this life you’ve been given. The reason I do science, rather than making a mint in the pharma industry[4], is that I am genuinely curious and want to figure shit out.

Guess what? Neither of these things depend on whether somebody else publishes a similar (or even very similar) experiment while you’re still running it. We all know that novelty still matters to a lot of journals. Some have been very reluctant to publish replication attempts. I agree that publishing high impact papers does help wedge your foot in the door (that is, get you short-listed) in grant and job applications. But even if this were all that matters to be a successful scientist (and it really isn’t), here’s why you shouldn’t care too much about that anyway:

No paper was ever rejected because it was scooped

While journal editors will reject papers because they aren’t “novel,” I have never seen any paper being rejected because somebody else published something similar a few months earlier. Most editors and reviewers will not even be aware of the scooping study. You may find this hard to believe because you think your own research is so timely and important, but statistically it is true. Of course, some reviewers will know of the work. But most reviewers are not actually bad people and will not say “Something like this was published three months ago already and therefore this is not interesting.” Again, you may find this hard to believe because we’ve all heard too many stories of Reviewer 2 being an asshole. But in the end, most people aren’t that big of an asshole[5]. It happens quite frequently that I suggest in reviews that the authors cite some recently published work (usually not my own, in case you were wondering) that is very similar to theirs. In my experience this has never led to a rejection but I ask to them to put their results in the context of similar findings in the literature. You know, the way a Discussion section should be.

No two scooped studies are the same

You may think that the scooper’s experiment was very similar, but unless they actually stole your idea (a whole different story I also don’t believe but I have no time for this now…) and essentially pre-replicated (preclicated?) your design, I’d bet that there are still significant differences. Your study has not lost any of its value because of this. And it’s certainly no reason to quit and/or be depressed.

It’s actually a compliment

Not 100% sure about this one. Scientific curiosity shouldn’t have anything to do with a popularity contest if you ask me. Study whatever the hell you want to (within ethical limits, that is). But I admit, it feels reassuring to me when other people agree that the research questions I am interested in are also interesting to them. For one thing, this means that they will appreciate you working and (eventually) publishing on it, which again from a pragmatic point of view means that you can pay those rents/mortgages. And from a simple vanity perspective it is also reassuring that you’re not completely mad for pursuing a particular research question.

It has little to do with publishing high impact

Honestly, from what I can tell neither precedence nor the popularity of your topic are the critical factors in getting your work into high impact journals. The novelty of your techniques, how surprising and/or reassuringly expected your results are, and the simplicity of the narrative are actually major factors. Moreover, the place you work, the co-authors you with whom you write your papers, and the accessibility of the writing (in particular your cover letter to the editors!) definitely matter a great deal also (and these are not independent of the first points either…). It is quite possible that your “rival”[6] will publish first, but that doesn’t mean you won’t publish similar work in a higher impact journal. Journal review outcome is pretty stochastic and not really very predictable.

Actual decisions are not based on this

We all hear the horror stories of impact factors and h-indexes determining your success with grant applications and hiring decisions. Even if this were true (and I actually have my doubts that it is as black and white as this), a CV with lots of high impact publications may get your foot in the door – but it does not absolve the panel from making a hiring/funding decision. You need to do the work on that one yourself and even then luck may be against you (the odds certainly are). It also simply is not true that most people are looking for the person with the most Nature papers. Instead I bet you they are looking for people who can string together a coherent argument, communicate their ideas, and who have the drive and intellect to be a good researcher. Applicants with a long list of high impact papers may still come up with awful grant proposals or do terribly in job interviews while people with less stellar publication records can demonstrate their excellence in other ways. You may already have made a name for yourself in your field anyway, through conferences, social media, public engagement etc. This may matter far more than any high impact paper could.

There are more important things

And now we’re coming back to the work-life balance and why you’re doing this in the first place. Honestly, who the hell cares whether someone else published this a few months earlier? Is being the first to do this the reason you’re doing science? I can see the excitement of discovery but let’s face it, most of our research is neither like the work of Einstein or Newton nor are we discovering extraterrestrial life. Your discovery is no doubt exciting to you, it is hopefully exciting to some other scientists in your little bubble and it may even be exciting to some journalist who will write a distorting, simplifying article about it for the mainstream news. But seriously, it’s not as groundbreaking that it is worth sacrificing your mental and physical health over it. Live your life. Spend time with your family. Be good to your fellow creatures on this planet. By all means, don’t be complacent, ensure you make a living but don’t pressure yourself into believing that publishing ultra-high impact papers is the meaning of life.

A positive suggestion for next time…

Now if you’re really worried about this sort of thing, why not preregister your experiment? I know I said I wouldn’t talk about open science here but bear with me just this once because this is a practical point you can implement today. As I keep saying, the whole discussion about preregistration is dominated by talking about “questionable research practices”, HARKing, and all that junk. Not that these aren’t worthwhile concerns but this is a lot of negativity. There are plenty of positive reasons why preregistration can help and the (fallacious) fear of being scooped is one of them. Preregistration does not stop anyone else from publishing the same experiment before you but it does allow you to demonstrate that you had thought of the idea before they published it. With Registered Reports it becomes irrelevant if someone else published before you because your publication is guaranteed after the method has been reviewed. And I believe it will also make it far clearer to everyone how much who published what first where actually matters in the big scheme of things.

[1] Actually there are a lot of old and experienced researchers who worry about this too. And that is far worse than when early career researchers do it because they should really know better and they shouldn’t feel the same career pressures.
[2] It may sound appealing now, but thinking about it I wouldn’t trade my current professional life for anything. Except for grant admin bureaucracy perhaps. I would happily give that up at any price… :/
[3] He didn’t quite say it in those terms.
[4] This doesn’t actually happen. If you want to make a mint you need to go into scientific publishing but the whole open science movement is screwing up that opportunity now as well so you may be out of luck!
[5] Don’t bombard me with “Reviewer 2 held up my paper to publish theirs first” stories. Unless Reviewer 2 signed their review or told you specifically that it was them I don’t take such stories at face value.
[6] The sooner we stop thinking of other scientists in those terms the better for all of us.

Strawberry Ice Cream Cone

Started signing my reviews

As of this year, I started signing my reviews. This decision has been a long time coming. A lot of people sign their reviews making this not a particularly newsworthy event but I’ll tell you about it anyway, largely to have a record of when I started and also to explain my reasons.

To explain why, I first need to talk about why one might not want to sign peer reviews. The debate about whether or not to sign reviews has been raging for years. It divides people’s minds and the debate regularly sparks up again. Even the people who agree that the process of scientific research can be improved seem to often fall into two camps whose opinions are diametrically opposed: one side fervently argues that all peer reviews should be transparent and signed, whilst other people argue with equal fervour that ideally all reviews should be double-blind, so that neither reviewers nor authors’ know each other’s identities.

Whenever someone suggests double-blind reviews, people are wont to argue that this simply doesn’t work in many situations. It is possible to guess the reviewers from the research question and/or the methods used. If the authors previously presented the research at a conference it is likely that reviewers will have already seen it in a preliminary form. That said, the very few times I did review in a double-blind manner I actually didn’t guess the authors’ identities and in one case I was in fact reviewing the work of friends and collaborators without even knowing it. I’d like to think I would’ve been fair either way, but I must also admit that I was probably more sceptical and possibly less biased because I didn’t know who the authors were. Still, these cases are probably somewhat special – in many situations I would know the authors from the research or at least have a strong suspicion. The suspicion might also lead me to some erroneous assumptions, such as “These authors usually do this and that even though this isn’t mentioned here”. If my guess were actually wrong then this could skew my thought process unduly.

So I think double-blind reviewing is a bad idea. Now, many arguments have been brought forth as to why reviews should be anonymous. It can protect reviewers from the wrath of vengeful senior colleagues making unfair hiring or funding decisions because they didn’t like your review. There are a lot of arseholes in the world and this is certainly a possibility. But the truth is that anonymity doesn’t stop people from behaving in this way – and there is actually no compelling evidence that signed reviews make it worse. I have heard some harrowing tales from colleagues who were being treated unfairly by some major players in their fields because they thought that they had given their work a bad review. In one case, it was a PhD student of the assumed reviewer who received ill treatment – and the assumption was entirely incorrect.

You also frequently hear people’s guesses about who they think Reviewer 2 was on their latest rejected manuscript, often based on circumstantial or generally weak evidence. One of my favourites is the age old “He (because we know all reviewers are obviously male…) asked us to cite lots of his papers!” I am sure this happens but I wonder how often this deduction is correct. I almost never ask people to cite my papers – if I do it is because I feel they are directly relevant and citing them is the scholarly thing to do. It is far more likely that I ask people to cite the work of researchers whose work I know well when it is relevant. In many cases when people just “know” that Reviewer 2 is Professor X because they want X to be cited, it seems to me far more likely that the reviewer is one of Professor X’s postdocs or former students. In many cases, it may also be that Professor X’s work is an established part of the literature and thus in the interest of scholarship an unbiased reviewer will think it deserves being cited even though you think Professor X’s work is rubbish. In short, I find those kind of insane guessing games rather tedious and potentially quite damaging.

The first time I signed a review was when I reviewed for F1000Research where signing is mandatory. (I had already reviewed at Frontiers a few times where reviewer identities are public but I don’t think this counts: reviews aren’t signed upon submission of the review but only after publication of the paper. Moreover, the match between review and reviewer remains ambiguous). I must say reviewing this paper all in public was a rather uplifting experience. At all stages of this process I felt the communication between me and the authors was amicable and sensible in spite of the harshness of my decisions. I have also been led to believe that the authors appreciated my scepticism (although only they can tell you that for sure).

By signing I may have also been more polite than I might have been if my review were anonymous. I am not entirely convinced of this last argument because I typically try to be polite. There are a lot of dickheads out there who aren’t polite even when their identity is public :P. I also don’t buy that anonymous reviewers aren’t accountable and that thus the quality of the review suffers. Your review is still read by at least one editor – unless that editor is your close personal friend (which is still rare for me at least) then I do feel like someone is checking my review both for factual quality and politeness.

Either way, I did not perceive any adverse consequences of signing my reviews. If anything, it made me think harder about how I would write my review and to check the arguments I am making. Scientists should criticise and scrutinise each other. By this I don’t mean you should mistrust people’s intentions or question their competence. But science is fueled by scepticism and you should challenge anything that doesn’t make sense. I have certainly done so in my collaborations in the past (often to the frustration of my collaborators) and I try to encourage this in my own lab. I much rather have a student or postdoc who tells me that my idea makes no sense than someone who does everything I say. Researchers also do that at conferences where they discuss each other’s research. One of my most positive experiences from a conference was some rather intense – but very polite – discussions at a poster. Why can’t we do the same in paper reviews?

When I’m perfectly honest, the main reason I hadn’t signed reviews so far is that I was raised that way. Almost none of the reviews I ever received were signed – certainly none of the negative ones. Some reviewers (including very critical ones) revealed their identities after the manuscripts had been accepted for publication and I have done the same in some cases. But the status quo of my field was always that reviews were anonymous and that’s just how it was. Challenging this seemed to go against nature – but that really isn’t true. Whether or not reviews are signed is a question of culture, not nature. And I want to change this culture.

Signing reviews is a personal choice. I don’t think it should ever become mandatory. For one thing, I’m a libertarian (just to be clear, I’m not one of the delusional tea party types) and I don’t believe we should force people to do things that aren’t necessary. I don’t think signed reviews are necessary. I think making all review contents public would be an essential improvement to peer review, with or without signing. But signing reviews can be positive development and I believe it should be encouraged. I certainly think it is a positive development for me and this is why everyone should be free to take this step of their own accord. Signing my first reviews has been a strangely liberating experience. I don’t know if this will provoke the ire of powerful senior colleagues. In a few years’ time I may post an update about my experience. Somehow I doubt it will turn out to be a problem.

Coconut

Why Gilbert et al. are missing the point

This (hopefully brief) post is yet again related to the replicability debate I discussed in my previous post. I just read a response by Gilbert et al. to the blog comments about their reply to the reply to the reply to the (in my view, misnamed) Reproducibility Project Psychology. I won’t go over all of this again. I also won’t discuss the minutia of the statistical issues as many others have already done so and will no doubt do so again. I just want to say briefly why I believe they are missing the point:

The main argument put forth by Gilbert et al. is that there is no evidence for a replicability crisis in psychology and that the “conclusions” of the RPP are thus unfounded. I don’t think that the RPP ever claimed anything of the kind one way or the other (in fact, I was impressed by the modesty of the claims made by the RPP study when I read it) but I’ll leave that aside. I appreciate what Gilbert et al. are trying to do. I have myself frequently argued a contrarian position in these discussions (albeit not always entirely seriously). I am trying to view this whole debate the same way any scientist should: by evaluating the evidence without any investment in the answer. For that reason, the debate they have raised seems worthwhile. They tried to estimate a baseline level of replicability one could expect from psychology studies. I don’t think they’ve done it correctly (for statistical reasons) but I appreciate that they are talking about this. This is certainly what we would want to do in any other situation.

Unfortunately, it isn’t that simple. Even if there were no problems with publication bias, analytical flexibility, and lacking statistical power (and we can probably agree that this is not a tenable assumption), it wouldn’t be a straightforward thing to estimate how many psychology studies should replicate by chance. In order to know this you would need to know how many of the hypotheses are true and we usually don’t. As Einstein said – or at least the internet tells me he did: “If we knew what it was we were doing, it would not be called research, would it?”

One of the main points they brought up is that some of the replications in the RPP may have used inappropriate procedures to test the original hypotheses – I agree this is a valid concern but it also completely invalidates the argument they are trying to make. Instead of quibbling about what measure of replication rates is evidence for a “crisis” (a completely subjective judgement) let’s look at the data:

f1-large

This scatter graph from the RPP plots effect sizes in the replications against the originally reported ones. Green (referred to as “blue” by the presumably colour-blind art editors) points are replications that turned out significant, red ones are those that were not significant and thus “failed to replicate.” The separation of the two data clouds is fairly obvious. Significant replication effects have a clear linear relationship with the original ones. Non-significant ones are uncorrelated with the original effect sizes.

We can argue until the cows come home what this means. The red points are presumably at least to a large part false positives. Yes, of course some – perhaps many – may be because of methodological differences or hidden moderators etc. There is no way to quantify this reliably. And conversely, a lot of the green dots probably don’t tell us about any cosmic truths. While they are replicating wonderfully, they may just be replicating the same errors and artifacts. All of these arguments are undoubtedly valid.

But that’s not the point. When we test the reliability of something we should aim for high fidelity. Of course, perfect reliability is impossible so there must be some scatter around the identity line. We also know that there will always be false positives so there should be some data points scattering around the x-axis. But do you honestly think it should be as many as in that scatter graph? Even if these are not all false positives in the original but rather false negatives in the replication, for instance because the replicators did a poor job or there were unknown factors we don’t yet understand, this ratio of green to red dots is not very encouraging.

Replicability encompasses all of the aforementioned explanations. When I read a scientific finding I don’t expect it to be “true.” Even if the underlying effects are real, the explanation for them can be utterly wrong. But we should expect a level of replicability from a field of research that at least maximises the trustworthiness of the reported findings. Any which way you look at it, this scatter graph is unsettling: if two thirds of the dots are red because low statistical power and publication bias in the original effects, this is a major problem. But if they are red because the replications are somehow defective this isn’t exactly a great argument either. What this shows is that the way psychology studies are currently done does not permit very reliable replication. Either way, if you give me a psychology study I should probably bet against it replicating. Does anyone think that’s an acceptable state of affairs?

I am sure both of these issues play a role but the encouraging thing is that probably it is the former, false positives, that is more dominant after all. In my opinion the best way anyone has looked at the RPP data so far is Alex Etz’s Bayesian reanalysis. This suggests that one of the main reasons the replicability in the RPP is so underwhelming is that the level of evidence for the original effects was weak to begin with. This speaks for false positives (due to low power, publication bias, QRPs) and against unknown moderators being behind most of the replication failures. Believe it or not, this is actually a good thing – because it is much easier to address the former problem than the latter.

The (non-)replicability of soft science

Since last night the internet has been all atwitter about a commentary* by Dan Gilbert and colleagues about the recent and (in my view) misnamed Reproducibility Project: Psychology. In this commentary, Gilbert et al. criticise the RPP for a number of technical reasons asserting that the sampling was non-random and biased and that essentially the conclusions, in particular in the coverage by science media and blogosphere, of a replicability crisis in psychology is unfounded. Some of their points are rather questionable to say the least and some, like their interpretation of confidence intervals, are statistically simply wrong. But I won’t talk about this here.

One point they raise is the oft repeated argument that replications differed in some way from the original research. We’ve discussed this already ad nauseam in the past and there is little point going over this again. Exact replication of the methods and conditions of an original experiment can test the replicability of a finding. Indirect replications loosely testing similar hypotheses instead inform about generalisability of the idea, which in turn tells us about the robustness of the purported processes we posited. Everybody (hopefully) knows this. Both are important aspects to scientific progress.

The main problem is that most debates about replicability go down that same road with people arguing about whether the replication was of sufficient quality to yield interpretable results. One example by Gilbert and co is that one of the replications in the RPP used the same video stimuli used by the original study, even though the original study was conducted in the US while the replication was carried out in the Netherlands, and the dependent variable was related to something that had no relevance to the participants in the replication (race relations and affirmative action). Other examples like this were brought up in previous debates about replication studies. A similar argument has also been made about the differences in language context between the original Bargh social priming studies and the replications. In my view, some of these points have merit and the example raised by Gilbert et al. is certain worth a facepalm or two. It does seem mind-boggling how anyone could have thought that it is valid to replicate a result about a US-specific issue in a liberal European country whilst using the original stimuli in English.

But what this example illustrates is a much larger problem. In my mind that is actually the crux of the matter: Psychology, or at least most forms of more traditional psychology, do not lend themselves very well to replication. As I am wont to point out, I am not a psychologist but a neuroscientist. I do work in a psychology department, however, and my field obviously has considerable overlap with traditional psychology. I also think many subfields of experimental psychology work in much the same way as other so-called “harder” sciences. This is not to say that neuroscience, psychophysics, or other fields do not also have problems with replicability, publication bias, and other concerns that plague science as a whole. We know they do. But the social sciences, the more lofty sides of psychology dealing with vague concepts of the mind and psyche, in my view have an additional problem: They lack the lawful regularity of effects that scientific discovery requires.

For example, we are currently conducting an fMRI experiment in which we replicate a previous finding. We are using the approach I have long advocated that in order to try to replicate you should design experiments that do both, replicate a previous result but also seek to address a novel question. The details of the experiment are not very important. (If we ever complete this experiment and publish it you can read about it then…) What matters is that we very closely replicate the methods of a study from 2012 and this study closely replicated the methods of one from 2008. The results are pretty consistent across all three instances of the experiment. The 2012 study provided a somewhat alternative interpretation of the findings of the 2008 one. Our experiment now adds more spatially sensitive methods to yet again paint a somewhat different picture. Since we’re not finished with it I can’t tell you how interesting this difference is. It is however already blatantly obvious that the general finding is the same. Had we analysed our experiment in the same way as the 2008 study, we would have reached the same conclusions they did.

The whole idea of science is to find regularities in our complex observations of the world, to uncover lawfulness in the chaos. The entire empirical approach is based on the idea that I can perform an experiment with particular parameters and repeat it with the same results, blurred somewhat by random chance. Estimating the generalisability allows me to understand how tweaking the parameters can affect the results and thus allows me to determine what the laws are the govern the whole system.

And this right there is where much of psychology has a big problem. I agree with Gilbert et al. that repeating a social effect in US participants with identical methods in Dutch participants is not a direct replication. But what would be? They discuss how the same experiment was then repeated in the US and found results weakly consistent with the original findings. But this isn’t a direct replication either. It does not suffer from the same cultural and language differences as the replication in the Netherlands did but it has other contextual discrepancies. Even repeating exactly the same experiment in the original Stanford(?) population would not necessarily be equivalent because of the time that has passed and the way cultural factors have changed. A replication is simply not possible.

For all the failings that all fields of science have, this is a problem my research area does not suffer from (and to clarify: “my field” is not all of cognitive neuroscience, much of which is essentially straight-up psychology with the brain tagged on, and also while I don’t see myself as a psychologist, I certainly acknowledge that my research also involves psychology). Our experiment is done on people living in London. The 2012 study was presumably done mainly on Belgians in Belgium. As far as I know the 2008 study was run in the mid-western US. We are asking a question that deals with a fairly fundamental aspect of human brain function. This does not mean that there aren’t any population differences but our prior for such things affecting the results in a very substantial way are pretty small. Similarly, the methods can certainly modulate the results somewhat but I would expect the effects to be fairly robust to minor methodological changes. In fact, whenever we see that small changes in the method (say, the stimulus duration or the particular scanning sequence used) seem to obliterate a result completely, my first instinct is usually that such a finding is non-robust and thus unlikely to be meaningful.

From where I’m standing, social and other forms of traditional psychology can’t say the same. Small contextual or methodological differences can quite likely skew the results because the mind is a damn complex thing. For that reason alone, we should expect psychology to have low replicability and the effect sizes should be pretty small (i.e. smaller than what is common in the literature) because they will always be diluted by a multitude of independent factors. Perhaps more than any other field, psychology can benefit from preregistering experimental protocols to delineate the exploratory garden-path from hypothesis-driven confirmatory results.

I agree that a direct replication of a contextually dependent effect in a different country and at a different time makes little sense but that is no excuse. If you just say that the effects are so context-specific it is difficult to replicate them, you are bound to end up chasing lots of phantoms. And that isn’t science – not even a “soft” one.

purity
Then again, all fields of science are “soft”
* At first I thought the commentary was due to be published by Science on 4th March and embargoed until that date. However, it turns out to be more complicated than that because the commentary I am discussing here is not the Science article but Gilbert et al.’s reply to Nosek et al.’s reply to Gilbert et al.’s reply to the RPP (Confused yet?). It appeared on a website and then swiftly vanished again. I don’t know how I would feel posting it because the authors evidently didn’t want it to be public. I don’t think actually having that article is central to understanding my post so I feel it’s not important.

On studying precognition

Today I received an email from somebody who had read some of my discussions of Psi research. They made an interesting point that has so far been neglected in most of the debates I participated in. With their permission I post their email (without identifying information) and my response to it. I hope this clarifies my views:

I also have experienced real and significant episodes of precognition. After many experiences I researched my ancestry and found relatives who had histories of episodes of precognition. The studies I have read that claim precognition is not real all have the same error. I can’t pick a card, I can’t tell you what the next sound will be. Precognition does not work like that. I will demonstrate with this example.
I was standing at the front desk at work when I got a terrible feeling something was wrong. I didn’t know what. I called a friend and told him something is wrong.  I began a one hour drive home and continued talking to my friend. The feeling that something was wrong grew to an increasing level as a approached the river. I saw a city bus parked on the side of the road. Many vehicles were down by the river. I passed that scene and then told my friend that a child was now drowned and she was close to the bridge about 1/2 a mile down river. The next day the TV news confirmed that she was found at the bridge down river.
No one told me there was a drowning, no one told me it was a girl, no one knew she was floating by the bridge.
This type of thing happens to me regularly. I believe it results from the same thing that will stampede cattle. I think humans communicate through speech and other forms of non verbal communication. I think somehow I am able to know what the herd is thinking or saying without being there. I think the reason I got the feeling something was wrong had to do with the escalating fear and crying out of the people who were madly searching for the child who fell in the river.
So trying to study precognition by getting a person to predict the next card will never work. Look at the reality of how it happens and see if you can study it a different way.

My response to this:

Thank you for your email. I’d say we are in greater agreement than you may think. What I have written on my blog and in the scientific literature about precognition/telepathy/presentiment pertains strictly to the scientific experiments that have been done on these paranormal abilities, usually with the sole aim to prove their existence. You say you “can’t pick a card” etc – tell that to the researchers who believe that showing a very subtle difference from chance performance on such simple experiments is evidence for precognition.
Now, do I believe you have precognition? No, I don’t. The experiences you describe are not uncommon but they may be uncommonly frequent for you. Nevertheless they are anecdotal evidence and my first hunch would be to suspect cognitive biases that we know can masquerade as paranormal abilities. There may also be cognitive processes we currently simply have no understanding of. How we remember our own thoughts is still very poorly understood. The perception of causality is a fascinating topic. We know we can induce causality illusions but this line of inquiry is still in its infancy.
But I cannot be certain of this. Perhaps you do have precognition. I don’t have any intention to convince you that you don’t; I only want to clarify why I don’t believe it, certainly not based on the limited information I have. The main issue here is that your precognition is unfalsifiable. You say yourself that “Precognition does not work like that.” If it does not occur with the same regularity as other natural phenomena, it isn’t amenable to scientific study. Psi researchers believe that precognition etc have that regularity and so they think you can demonstrate it with card-picking experiments. My primary argument is about that line of thinking.
I am not one of those scientists who feel the need to tell everyone what to believe in. These people are just as irritating as religious fundamentalists who seek to convert everybody. If some belief is unfalsiable, like the existence of God or your belief in your precognition, then it falls outside the realm of science. I have no problem with you believing that you have precognition, certainly not as long as it doesn’t cause any harm to anyone. But unless we can construct a falsifiable hypothesis, science has no place in it.