Yes, science is self-correcting

If you don’t believe science self-corrects, then you probably shouldn’t believe that evolution by natural selection occurs either – it’s basically the same thing.

I have said it many times before, both under the guise of my satirical alter ego and later – more seriously – on this blog. I am getting very tired of repeating it so I wrote this final post about it that I will simply link to next time this inevitably comes up…

My latest outburst about this was triggered by this blog post by Keith Laws entitled “Science is ‘Other-Correcting‘”. I have no qualms with the actual content of this post. It gives an interesting account of the attempt to correct an error in the publication record. The people behind this effort are great researchers for whom I have the utmost respect. The story they tell is shocking and important. In particular, the email they received by accident from a journal editor is disturbing and serves as a reminder of all the things that are wrong with the way scientific research and publishing currently operates.

My issue is with the (in my view seemingly) ubiquitous doubts about the self-correcting nature of science. To quote from the first paragraph in that post:

“I have never been convinced by the ubiquitous phrase ‘Science is self-correcting’. Much evidence points to science being conservative and looking less self-correcting and more ego-protecting. It is also not clear why ‘self’ is the correct description – most change occurs because of the ‘other’ – Science is other correcting.”

In my view this and similar criticisms of self-correction completely miss the point. The suffix ‘self-‘ refers to science, not to scientists. In fact, the very same paragraph contains the key: “Science is a process.” Science is an iterative approach by which we gradually broaden our knowledge and understanding of the world. You can debate whether or not there is such a thing as the “scientific method” – perhaps it’s more of a collection of methods. However, in my view above all else science is a way of thinking.

Scientific thinking is being inquisitive, skeptical, and taking nothing for granted. Prestige, fame, success are irrelevant. Perfect theories are irrelevant. The smallest piece of contradictory evidence can refute your grand unifying theory. And science encompasses all that. It is an emergent concept. And this is what is self-correcting.

Scientists, on the other hand, are not self-correcting. Some are more so than others but none are perfect. Scientists are people and thus inherently fallible. They are subject to ego, pride, greed, and all of life’s pressures, such as the need to pay a mortgage, feed their children, and having a career. In the common vernacular “science” is often conflated with the scientific enterprise, the way scientists go about doing science. This involves all those human factors and more and, fair enough, it is anything but self-correcting. But to argue that this means science isn’t self-correcting is attacking a straw man because few people are seriously arguing that the scientific enterprise couldn’t be better.

We should always strive to improve the way we do science because due to our human failings it will never be perfect. However, in this context we also shouldn’t forget how much we have already improved it. In the times of Newton, in Europe (the hub of science then) science was largely done only by white men from a very limited socioeconomic background. Even decades later, most women or people of non-European origin didn’t even need to bother trying (although this uphill struggle makes the achievements of scientists like Marie Curie or Henrietta Swan Leavitt all the more impressive). And publishing your research findings was not subject to formal peer review but largely dependent on the ego of some society presidents and on whether they liked you. None of these problems have been wiped off the face of the Earth but I would hope most people agree that things are better than they were 100 years ago.

Like all human beings, scientists are flawed. Nevertheless I am actually optimistic about us as a group. I do believe that on the whole scientists are actually interested in learning the truth and widening their understanding of nature. Sure, there are black sheep and even the best of us will succumb to human failings. At some point or other our dogma and affinity to our pet hypotheses can blind us to the cold facts. But on average I’d like to think we do better than most of our fellow humans. (Then again, I’m probably biased…).

We will continue to make the scientific enterprise better. We will change the way we publish and evaluate scientific findings. We will improve the way we interpret evidence and we communicate scientific discoveries. The scientific enterprise will become more democratic, less dependent on publishers getting rich on our free labour. Already within the decade I have been a practicing scientist we have begun to tear down the wide-spread illusion that when a piece of research is published it must therefore be true. When I did my PhD, the only place we could critically discuss new publications was in a small journal club and the conclusions of these discussions were almost never shared with the world. Nowadays every new study is immediately discussed online by an international audience. We have taken leaps towards scientific findings, data, and materials being available to anyone, anywhere, provided they have internet access.  I am very optimistic that this is only the beginning of much more fundamental changes.

Last year I participated in a workshop called “Is Science Broken?” that was solely organised by graduate students in my department. The growing number of replication attempts in the literature and all these post-publication discussions we are having are perfect examples of science correcting itself. It seems deeply ironic to me when posts like Keith Laws’, which describes an active effort to rectify errors, argue against the self-correcting nature of the scientific process.

Of course, self-correction is not guaranteed. It can easily be stifled. There is always a danger that we drift back into the 19th century or the dark ages. But the greater academic freedom (and generous funding) scientists are given, the more science will be allowed to correct itself.

Beach
Science is like a calm lagoon in the sunset… Or whatever. There is no real reason why this picture is here.

Update (19 Jan 2016): I just read this nice post about the role of priors in Bayesian statistics. The author actually says Bayesian analysis is “self-correcting” and this epitomises my point here about science. I would say science is essentially Bayesian. We start with prior hypotheses and theories but by accumulating evidence we update our prior beliefs to posterior beliefs. It may take a long time but assuming we continue to collect data our assumptions will self-correct. It may take a reevaluation of what the evidence is (which in this analogy would be a change to the likelihood function). Thus the discussion about how we know how close to the truth we are is in my view missing the point. Self-correction describes the process.

Update (21 Jan 2016): I added a sentence from my comment in the discussion section to the top. It makes for a good summary of my post. The analogy may not be perfect – but even if not I’d say it’s close. If you disagree, please leave a comment below.

19 thoughts on “Yes, science is self-correcting

  1. great post! i totally agree with the distinction between ‘science’ and the individuals doing it.

    one (slightly tangential) addition to the ‘human failings’. from my perspective allegiances to pet hypotheses aren’t the biggest problem of the day. the major factor seems to be career pressure: the perception that sensationalism is more likely to get you the paper, the grant and the job than interpreting your data with cautious skepticism. i’m not sure to which degree this is actually true, but the perception seems widespread among junior scientists like myself.

    but in agreement with your post i do expect that whenever overhyping leads to wrong conclusions they won’t survive in the long run. the ‘other-correcting’ of scientists will kick in and assure the ‘self-correction’ of science 😉

    Like

    1. Yeah that’s really my point. Other-correcting scientists are a fundamental part of self-correcting science. Of course, I would very much like to encourage scientists to also be self-correcting. I want to live in a world where correcting your own findings is rewarded. Any suggestions how to achieve that are welcome…

      Like

  2. I think it’s possible for you and Keith Laws to be right simultaneously. Science does ultimately self-correct; it just doesn’t always do so at a pace that satisfies those who actively fight for it. In psychology, especially, where apparently very few people have a problem with (A) and (not A) being “true” simultaneously, self-correction (too) often takes the form of ideas going out of fashion. The problem is all the public policy decisions that got taken on the back of those ideas and their silly little p=.048 “demonstrations” with 47 undergraduates.

    It reminds me of Steven Pinker’s observations that everything has been getting better for the last 500 years. That’s great, and true, but I wouldn’t point it out to someone whose loved ones just got murdered.

    Like

    1. Thanks for your comment. I’d like to point out (somewhat ironically perhaps) that I don’t think Keith is wrong at all. Not about the heart of the matter anyway. I just take issue with the seemingly wide-spread delusion that the self- in self-correction refers to people. Not only does science self-correct findings and theories over time but even the process of science self-corrects because the two things are inherently linked.

      You are completely right about the pace of self-correction. Also, as Joachim Vandekerckhove pointed out on Twitter, there are now so many scientists researching different topics that there are probably a lot of dead-ends that will never self-correct but just get abandoned. I am not entirely sure this is a problem though. I certainly think we should work on speeding up the pace for self-correction but dismantling the barriers that can prevent it, like the incentive structure of our publication system or the continued focus on “novelty”.

      Interesting analogy to Pinker’s thesis. The parallel hadn’t occurred to me but you’re very right. Having read Keith’s blog post and also the actual response by Turkington to their criticism I can certainly understand why Keith and co are pissed off 😛

      Like

  3. I find this interesting to think about. I wonder if saying something like “science is self-correcting” amounts to a statement that can never be refuted, because 1) at a specific point in time you can never be sure whether or not things will be “corrected” in the future. The future is endless, so you can never really come to a conclusion about self-correction when you relate it to possible future events.

    Furthermore, even when something is “corrected” who is to say that this is correct. Merely changing a theory or interpretation of findings should not account for proper use of the word “correction” in this context i think. 2) Therefore i reason that there should be some form of evaluation that goes together with using words like “correcting”. I reason, for instance in social psychology, that it is very difficult to correctly conclusively state that something is “more true” or “contributed to gradually broaden our knowledge and understanding of the world”.

    Because of 1) and 2) i think it is not possible, or extremely difficult and impossible without some form of evaluation of the “corrected” interpretations/findings, to make clear that science is self-correcting.

    Like

    1. I disagree. I think self-correction is inherent to the process – it is the mechanism by which science itself works. In fact, I’m inclined to call it a truism. It’s just like saying “the sky is blue” or “water is wet”.

      And one of the big misconceptions in my view is to regard the undeniable fact that a lot of scientific findings and theories are false as evidence against self-correction. They are simply different things.

      Like

    2. as I understand it, ‘correction’ in this context doesn’t mean anything like ‘arriving at the final solution’. it means ‘change to accomodate conflicting data’. and, similar to what Sam said, this ability appears a defining property of science to me.

      Like

  4. “as I understand it, ‘correction’ in this context doesn’t mean anything like ‘arriving at the final solution’. it means ‘change to accomodate conflicting data’. and, similar to what Sam said, this ability appears a defining property of science to me.”

    Just “change” is not “correcting” in my opinion. That’s why i reason that some form of evaluation of some criteria needs to be there, to make clear whether the “correction” is indeed a correction. If that makes sense, the “corrections” should for instance better explain data, or lead to improvements of some criteria, etc.

    Bottom line: i reason that there needs to be evaluation with certain criteria for making clear whether science progresses and self-corrects. Concerning psychological science (that’s the field i was educated in, so i keep referring to that field) I can’t think of any criteria which are currently in (widespread) use to be able to evaluate whether the field progresses or self-corrects…

    If someone knows any, i’d be interested in hearing about them.

    Like

    1. But it’s more than mere change! As long as there are scientists who question the status quo and challenge dogma – which I’d say is what science is all about – and given sufficient time and resources, then science will eventually achieve a better understanding of the world.

      It’s not relevant whether out current corrections are actual correct. Some of them are no doubt dead ends and red herrings. Perhaps the current wave of skepticism about “social priming” is misguided and this is really how the human mind works (I doubt it but can’t rule it out). In that case, all the current talk about it is leading us astray. But provided that someone will continue doing this science, it will get closer to the “truth”.

      Now as for your question, what ways we can quantify how science progresses – in my view, asking that question is in itself part of self-correcting science…

      Like

    2. @2cents, what i meant by ‘change to accommodate conflicting data’ is that the survival of hypotheses hinges on their not being killed by data. in the long run this will yield hypotheses that are better compatible with the known data (or at least to the abandonment of hypotheses that are incompatible with these data).

      my training is in psychology (and visual neuroscience) as well. examples that come to mind are

      – Zeki’s idea of V4 being an exclusive ‘colour centre’, which hasn’t survived because of conflicting data showing all kinds of ‘form’ tuning in V4 as well as colour tuning in other areas
      – Skinner’s reinforcement theory of language acquisition crumbling away after Chomsky and others pointed out overregularization
      – The resurrection (and modification) of Hering’s opponent theory of colour vision after physiological data showed that trichromacy isn’t the whole story

      Like

  5. I have added a short addendum linking to a nice post about Bayesian statistics. Since the author also calls these self-correcting I thought it was worth mentioning that. It sheds more light on the discussion we were having.

    Like

  6. Hi Sam – thanks for the interesting discussion, although I am not quite sure where we agree and/or disagree…so I will say it how I see it

    Does the suffix ‘self-‘ in ‘self-correction’ refer to science or scientists? You suggest the former, but I cannot see how ‘science’ can be viewed separately from the people ‘doing science’…to state the obvious …there is no science without scientists.

    You mention ‘method’, hint at some self-correcting process in science and speak of falsification and ‘scientific thinking’. It may be that you hold a more ‘idealistic’ Popperian notion of Science/Psychology than I do. My view is perhaps a realistic warts-n-all view – I see psychologists often engage in what we might call ‘monster-barring’ or ‘ego-protecting’ – as described by Lakatos, their version of ‘scientific thinking’ is to actively prevent treasured ideas from rejection (correction) by building walls of auxiliary hypotheses…and I will explain why this is especially problematic for Psychology (more than other ‘sciences’)

    Some scientists may operate a falsification style rational approach – some may do… but some dont. Compared to other sciences, I would argue that Psychology is an excellent example of the “some don’ts”. The vast majority of published findings in psychology are ‘positive’ and rarely do psychologists publish null findings (compared to other sciences – see Fanelli 2010). This is itself a ‘rejection of self-correction’ in any idealistic manner such as proposed by Popper. The rate of null findings in psychology is lower than in any other ‘science’ and has never changed – the rates of positive findings in psychology has hovered around 95+% upwards for decades

    I am inclined then to say that, at its core, Psychology – as practised for decades – even conflicts with a self-correcting view of science. Hence the need for what I termed ‘other-correcting’

    Like

    1. Thanks for your reply, Keith! I also don’t think we actually disagree on a lot of things. I think you’re wrong if you single out psychology. All of science suffers from these issues although psychology maybe suffers from it a bit more than astrophysics – but that has more to do with the complications of understanding something complex like the human brain/mind compared to understanding simply things like the cosmos… 😛

      Anyway, surely the fact that there are so few null findings in psychology is worrying. As I said many times, I fully support efforts to improve the way we do science. But that speaks to how we do things in the short-term and dealing with the human factors in science.

      To answer your question, yes, I think we must separate ‘doing science’ from how science works! Scientists are human beings and that comes with baggage. But science is simply the best way to understanding the universe. It’s a process and gradually approximating the truth is the whole point of it. Sorry if that sounds harsh, but in my view, if you don’t believe that science self-corrects you should pack up and look for another job – because then you don’t believe that science works, period.

      The reason I take issue with people denying the self-correcting nature of science itself is that in my view this distracts from the real problem which is that science is done by fallible people working in an incentive structure that is an impediment to self-correction. Self-correction is often slow or may never happen if an avenue of research is abandoned. We should encourage and help speed up that process if we can.

      But to say that self-correction doesn’t happen is to me equivalent to saying that science itself doesn’t work. In my opinion that view is wrong and dangerous and gives undue support to those people who reject science as a whole.

      Like

  7. Sam – I think you are wrong when you say that its “wrong if you single out psychology.” – Psychology does operate differently from most or all other sciences. And this is especially relevant with regard to the notion of ‘correcting’ (as I mentioned above). Psychology publishes more positive findings than all other sciences e.g. Psychology is five times more likely to publish positive findings than ‘space science’ (Fanelli 2010).

    …and its not “how we do things in the short-term’ – the excess of positive findings published in psychology has documented for over 50 years…and the high rates have not changed. Rather than pointing to self-correcting in psychology, it suggests quite the opposite…

    I think we shall have to disagree about whether science and scientists can be ‘dissociated’ – Science does not exist without scientists and the operating ‘cultures’ that they create in their own topic areas. These cultures can vary considerably and greatly influence how each science operates. So, regardless of whether you think science has privileged methods or ‘way of thinking’ about the universe, psychology mostly operates outside of this self-correcting model and has done for decades…which is not to say it couldn’t be self-correcting, but that it has little or no tradition of doing so.

    Like

    1. Even if psychology has the highest rates of positive findings that doesn’t mean other fields of science don’t also have these problems. I’m not a psychologist and psychology encompasses a lot of things I know very little about. I can only comment on what I know from cognitive (which obviously overlaps with psychology) and systems neuroscience and from what I know about other fields. Replicability is also under serious doubt in biomedical research. Physics has had some serious fraud cases. The high rates of positive findings in psychology may have just as much to do with how and what things are being published and how this is being quantified. I haven’t read the Fanelli study (thanks for pointing that out) but I wouldn’t be surprised if a typical space science study addressed research questions somewhat differently than a typical psychology study. I think those differences already exist between visual psychophysics and social psychology – so they are probably even more pronounced when comparing social psychology to physics. I think it would be misguided to assume that therefore there is less publication bias in those fields. It merely manifests differently.

      More importantly, I don’t think that the rate of positive findings being published is a measure of (the lack of) self-correction. Even if psychologists published 100% positive confirmations of their hypotheses, science will still inevitably self-correct. Positive findings can be refuted even without failed replication. First of all, as I am wont to point out, a good refutation actually seeks to replicate but provides an alternative explanation. (I’ve recently tweeted two recent examples – I will probably blog about that in the future). Secondly, even if nobody ever publishes any replication (successful or failure) people will still try to replicate or scrutinise the claims and reanalyse data behind closed doors. Word will spread and unreplicable findings will gradually become known. In fact, future publications may indirectly disprove previous ones by asking a related question. Gradually, as evidence accumulates – even if only positive findings – science will thus self-correct. I completely agree with you that this way self-correction occurs more slowly, more painfully, and, what is worse, more wastefully. I do think we should change it. By sharing our data and methods, by making it more rewarding to publicly admit to yourself that your wrong, and by publishing all findings regardless of their outcome we can encourage self-correction.

      But that’s not about the fact of self-correction but about the environment that fosters self-correction. Life evolves but only as long as there is life to evolve. If I blow up this planet, life is not given a chance to evolve. This is the equivalent situation if we stopped doing science altogether. If there are only a smattering of habitable spots left on the Earth, evolution can only happen in those limited places (unless life evolves that can live outside those but let’s not overcomplicate things…). This is the equivalent of the status quo in psychology perhaps. I want to open the whole planet up for life to thrive but I’m not denying that evolution can occur even under constrained circumstance. Essentially my argument is this:

      If you don’t believe science self-corrects, then you probably shouldn’t believe that evolution by natural selection occurs either – it’s basically the same thing.

      Like

    2. This is rather tangential, but just to follow up on Sam’s point re differences on the sub-discipline level and conventions of presentation. I had a look at the Fanelli paper (thanks for pointing this out!) and started wondering about these issues.

      Fanelli reports the proportion of hypothesis-supporting findings published in various fields, ranging from ‘Space Science’ (lowest, at 70.2%) to ‘Psychology and Psychiatry’ (highest, at 91.5%). Just for fun I applied exactly the same method he used to papers published in ‘Journal of Vision’. I only made it through the first 35 of 190 or so hits (no too much time for this atm), but at this point vision science clocks in at 77% positive findings. For comparison, Fanelli finds about 85% for ‘Neuroscience & Behaviour’ at large, which is on par with ‘Physics’.

      Like Sam, I’m doubtful regarding the validity of this metric as an indicator of differential publication bias between disciplines. But if it _was_ a valid indicator of a ‘hierarchy’ (as Fanelli suggests), then vision science would be damn ‘hard’ 😉

      Like

    3. Thanks for that Ben. This is actually something I wanted to do ages ago in a different context – to quantify the prior to determine the false discovery rate, which David Colquhoun has been going on about for the past year (obviously a much older issue). Of course, the number of hypothesis-confirming findings isn’t quite the same as the base rate and the number may still be skewed by publication bias etc but it would be a good place to start.

      Like

  8. Sure science will self correct, if there are scientists around to correct it, but it may take more time than is available. False science about tobacco and cancer took a long time to correct. That was no help to the millions already dead.

    Like

    1. Yes and this is precisely the point. Just because science self-corrects doesn’t mean all scientific findings are correct. Many (most) aren’t even if we have a generous definition of what makes ‘true science’. I want science to be faster and more efficient. There are a lot of ways we can improve science. We are already better than 100 years but that’s no reason to be complacent. But we also need to face the music: there will always be false science because that is inherent to the process. We can only hope to reduce the time it sticks around for.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s