The Problem with Science
The Reproducibility Crisis and What to Do About It
R. BARKER BAUSELL
3
Oxford University Press is a department of the University of Oxford. It furthers the University’s objective of excellence in research, scholarship, and education by publishing worldwide. Oxford is a registered trade mark of Oxford University Press in the UK and certain other countries.
Published in the United States of America by Oxford University Press 198 Madison Avenue, New York, NY 10016, United States of America.
© Oxford University Press 2021
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without the prior permission in writing of Oxford University Press, or as expressly permitted by law, by license, or under terms agreed with the appropriate reproduction rights organization. Inquiries concerning reproduction outside the scope of the above should be sent to the Rights Department, Oxford University Press, at the address above.
You must not circulate this work in any other form and you must impose this same condition on any acquirer.
Library of Congress Cataloging-in-Publication Data
Names: Bausell, R. Barker, 1942– author.
Title: The problem with science : the reproducibility crisis and what to do about it / R. Barker Bausell, Ph.D.
Description: New York, NY : Oxford University Press, [2021] | Includes bibliographical references and index.
Identifiers: LCCN 2020030312 (print) | LCCN 2020030313 (ebook) | ISBN 9780197536537 (hardback) | ISBN 9780197536551 (epub) | ISBN 9780197536568
Subjects: LCSH: Science—Research—Methodology.
Classification: LCC Q126.9 .B38 2021 (print) | LCC Q126.9 (ebook) | DDC 507.2/1—dc23
LC record available at https://lccn.loc.gov/2020030312
LC ebook record available at https://lccn.loc.gov/2020030313
DOI: 10.1093/oso/9780197536537.001.0001
1 3 5 7 9 8 6 4 2
Printed by Sheridan Books, Inc., United States of America
3.
4.
5.
9. Publishing Issues and Their Impact on Reproducibility 193
10. Preregistration, Data Sharing, and Other Salutary Behaviors 222
11. A (Very) Few Concluding Thoughts 261
A Brief Note
This book was written and peer reviewed by Oxford University Press before the news concerning the “problem” in Wuhan broke, hence no mention of COVID-19 appears in the text. Relatedly, since one of my earlier books, Snake Oil Science: The Truth About Complementary and Alternative Medicine, had been published by Oxford more than a decade ago, I had seen no need to pursue this line of inquiry further since the bulk of the evidence indicated that alternative medical therapies were little more than cleverly disguised placebos, with their positive scientific results having been facilitated by substandard experimental design, insufficient scientific training, questionable research practices, or worse. So, for this book, I chose to concentrate almost exclusively on a set of problems bedeviling mainstream science and the initiative based thereupon, one that has come to be called “the reproducibility crisis.”
However, as everyone is painfully aware, in 2020, all hell broke loose. The internet lit up advocating bogus therapies; the leaders of the two most powerful countries in the world, Xi Jinping and Donald Trump, advocated traditional Chinese herbals and a household cleaner, respectively; and both disparaged or ignored actual scientific results that did not support their agendas. Both world leaders also personally employed (hence served as role models for many of their citizens) unproved, preventive remedies for COVID-19: traditional Chinese herbal compounds by Xi Jinping; hydroxychloroquine (which is accompanied by dangerous side effects) by Donald Trump.
This may actually be more understandable in Xi’s case, since, of the two countries, China is undoubtedly the more problematic from the perspective of conducting and publishing its science. As only one example, 20 years ago, Andrew Vickers’s systematic review team found that 100% of that country’s alternative medical trials (in this case acupuncture) and 99% of its conventional medical counterparts published in China were positive. And unfortunately there is credible evidence that the abysmal methodological quality of Chinese herbal medical research itself (and not coincidentally the almost universally positive results touting their efficacy) has continued to this day.
To be fair, however, science as an institution is far from blameless in democracies such as the United States. Few scientists, including research methodologists such as myself, view attempting to educate our elected officials on scientific issues as part of their civic responsibility.
So while this book was written prior to the COVID-19 pandemic, there is little in it that is not relevant to research addressing future health crises such as this (e.g., the little-appreciated and somewhat counterintuitive [but well documented] fact that early findings in a new area of inquiry often tend to be either incorrect or to report significantly greater effect sizes than follow-up studies). It is therefore my hope that one of the ultimate effects of the reproducibility crisis (which again constitutes the subject matter of this book) will be to increase the societal utility of science as well as the public’s trust therein. An aspiration that will not be realized without a substantial reduction in the prevalence of the many questionable research behaviors that permit and facilitate the inane tendency for scientists to manufacture (and publish) falsepositive results.
Introduction
This is a story about science. Not one describing great discoveries or the geniuses who make them, but one that describes the labors of scientists who are in the process of reforming the scientific enterprise itself. The impetus for this initiative involves a long-festering problem that potentially affects the usefulness and credibility of science itself.
The problem, which has come to be known as the reproducibility crisis, affects almost all of science, not one or two individual disciplines. Like its name, the problem revolves around the emerging realization that much— perhaps most—of the science being produced cannot be reproduced. And scientific findings that do not replicate are highly suspect if not worthless.
So, three of the most easily accomplished purposes of this book are
1. To present credible evidence, based on the published scientific record, that there exists (and has existed for some time) a serious reproducibility crisis that threatens many, if not most, sciences;
2. To present a menu of strategies and behaviors that, if adopted, have the potential of downgrading the problem from a crisis to a simple irritant; and
3. To serve as a resource to facilitate the teaching and acculturation of students aspiring to become scientists.
The book’s potential audience includes
1. Practicing scientists who have not had the time or the opportunity to understand the extent of this crisis or how they can personally avoid producing (and potentially embarrassing) irreproducible results;
2. Aspiring scientists, such as graduate students and postdocs, for the same reasons;
3. Academic and funding administrators who play (whether they realize it or not) a key role in perpetuating the crisis; and
The Problem with Science. R. Barker Bausell, Oxford University Press (2021). © Oxford University Press. DOI: 10.1093/oso/9780197536537.003.0001
4. Members of the general public interested in scientific issues who are barraged almost daily with media reports of outrageously counterintuitive findings or ones that contradict previously ones.
Some readers may find descriptors such as “crisis” for an institution as sacrosanct as science a bit hyperbolic, but in truth this story has two themes. One involves a plethora of wrongness and one involves a chronicling of the labors of a growing cadre of scientists who have recognized the seriousness of the problem and have accordingly introduced evidence-based strategies for its amelioration.
However, regardless of semantic preferences, this book will present overwhelming evidence that a scientific crisis does indeed exist. In so doing it will not constitute a breathless exposé of disingenuous scientific blunders or bad behavior resulting in worthless research at the public expense. Certainly some such episodes compose an important part of the story, but, in its totality, this book is intended to educate as many readers as possible to a serious but addressable societal problem.
So, in a sense, this is an optimistic story representing the belief (and hope) that the culture of science itself is in the process of being altered to usher in an era in which (a) the social and behavioral sciences (hereafter referred to simply as the social sciences) will make more substantive, reproducible contributions to society; and (b) the health sciences will become even more productive than they have been in past decades. Of course the natural and physical sciences have their own set of problems, but only a handful of reproducibility issues from these disciplines have found their way into the present story since their methodologies tend to be quite different from the experimental and correlational approaches employed in the social and health sciences.
For the record, although hardly given to giddy optimism in many things scientific, I consider this astonishing 21st-century reproducibility awakening (or, in some cases, reawakening) to be deservedly labeled as a paradigmatic shift in the Kuhnian sense (1962). Not from the perspective of an earthshattering change in scientific theories or worldviews such as ushered in by Copernicus, Newton, or Einstein, but rather in a dramatic shift (or change) in the manner in which scientific research is conducted and reported. These are behavioral and procedural changes that may also redirect scientific priorities and goals from a cultural emphasis on publishing as many professional articles as humanly possible to one of ensuring that what is published is correct, reproducible, and hence has a chance of being at least potentially useful.
However, change (whether paradigmatic or simply behavioral) cannot be fully understood or appreciated without at least a brief mention of what it replaces. So permit me the conceit of a very brief review of an important methodological initiative that occurred in the previous century.
The Age of Internal and External Validity
For the social sciences, our story is perhaps best begun in 1962, when a research methodologist (Donald T. Campbell) and a statistician (Julian C. Stanley) wrote a chapter in a handbook dealing with research on teaching of all things. The chapter garnered considerable attention at the time, and it soon became apparent that its precepts extended far beyond educational research. Accordingly, it was issued as an 84-page paperback monograph entitled Experimental and Quasi-Experimental Designs for Research (1966) and was promptly adopted as a supplemental textbook throughout the social sciences.
But while this little book’s influence arguably marked the methodological coming of age for the social sciences, it was preceded (and undoubtedly was greatly influenced) by previous methodology textbooks such as Sir Ronald Fisher’s The Design of Experiments (1935), written for agriculture researchers but influencing myriad other disciplines as well, and Sir Austin Bradford Hill’s Principles of Medical Statistics (1937), which had an equally profound effect upon medical research.
The hallmark of Campbell and Stanley’s remarkable little book involved the naming and explication of two constructs, internal and external validity, accompanied by a list of the research designs (or architecture) that addressed (or failed to address) the perceived shortcomings of research conducted in that era. Internal validity was defined in terms of whether or not an experimental outcome (generally presumed to be positive) was indeed a function of the intervention rather than extraneous events or procedural confounds. External validity addressed the question of:
To what populations, settings, treatment variables, and measurement variables can this effect [presumably positive or negative] be generalized? (p. 5)
Of the two constructs, internal validity was the more explicitly described (and certainly the more easily addressed) by a list of 12 “threats”
thereto—most of which could be largely avoided by the random assignment of participants to experimental conditions. External validity, relevant only if internal validity was ensured, was so diffuse and expansive that it was basically given only lip service for much of the remainder of the century. Ironically, however, the primary arbiter of external validity (replication) also served as the same bottom line arbiter for the reproducible–irreproducible dichotomy that constitutes the basic subject matter of this book.
Campbell and Stanley’s basic precepts, along with Jacob Cohen’s (1977, 1988) seminal (but far too often ignored work) on statistical power, were subsequently included and cited in hundreds of subsequent research methods textbooks in just about every social science discipline. And, not coincidentally, these precepts influenced much of the veritable flood of methodological work occurring during the next several decades, not only in the social sciences but in the health sciences as well.
Unfortunately, this emphasis on the avoidance of structural (i.e., experimental design) at the expense of procedural (i.e., behavioral) confounds proved to be insufficient given the tacit assumption that if the architectural design of an experiment was reasonably sound and the data were properly analyzed, then any positive results accruing therefrom could be considered correct 95% of the time (i.e., the complement of the statistical significance criterion of p ≤ 0.05). And while a vast literature did eventually accumulate around the avoidance of these procedural confounds, less attention was paid to the possibility that a veritable host of investigator-initiated questionable research practices might, purposefully or naïvely, artifactually produce falsepositive, hence irreproducible, results.
From a scientific cultural perspective, this mindset was perhaps best characterized by the writings of Robert Merton (1973), a sociologist of science whose description of this culture would be taken as Pollyannaish satire if written today. In his most famous essay (“Science and the Social Order”) he laid out “four sets of institutional imperatives—universalism, communism [sharing of information not the political designation], disinterestedness, and organized skepticism—[that] are taken to comprise the ethos of modern science” (p. 270).
Scientific ethos was further described as
[t]he ethos of science is that affectively toned complex of values and norms which is held to be binding on the man of science. The norms are expressed in the form of prescriptions, proscriptions, preferences, and permissions.
They are legitimatized in terms of institutional values. These imperatives, transmitted by precept and example and reinforced by sanctions, are in varying degrees internalized by the scientist, thus fashioning his scientific conscience or, if one prefers the latter-day phrase, his superego. (1973, p. 269, although the essay itself was first published in 1938)
While I am not fluent in Sociologese, I interpret this particular passage as describing the once popular notion that scientists’ primary motivation was to discover truth rather than to produce a publishable p-value ≤ 0.05. Or that most scientists were so firmly enculturated into the “ethos” of their calling that any irreproducible results that might accrue were of little concern given the scientific process’s “self-correcting” nature.
To be fair, Merton’s essay was actually written in the 1930s and might have been somewhat more characteristic of science then than in the latter part of the 20th and early 21st centuries. But his vision of the cultural aspect of science was prevalent (and actually taught) during the same general period as were internal and external validity. Comforting thoughts certainly, but misconceptions that may explain why early warnings regarding irreproducibility were ignored.
Also in fairness, Merton’s view of science was not patently incorrect: it was simply not sufficient. And the same can be said for Campbell and Stanley’s focus on internal validity and the sound research designs that they fostered. Theirs might even qualify as an actual methodological paradigm for some disciplines, and it was certainly not incorrect. It was in fact quite useful. It simply was not sufficient to address an as yet unrecognized (or at least unappreciated) problem with the avalanche of scientific results that were in the process of being produced.
So while we owe a professional debt of gratitude to the previous generation of methodologists and their emphasis on the necessity of randomization and the use of appropriate designs capable of negating most experimental confounds, it is now past time to move on. For this approach has proved impotent in assuring the reproducibility of research findings. And although most researchers were aware that philosophers of science from Francis Bacon to Karl Popper had argued that a quintessential prerequisite for a scientific finding to be valid resides in its reproducibility (i.e., the ability of other scientists to replicate it), this crucial tenet was largely ignored in the social sciences (but taken much more seriously by physical scientists—possibly because they weren’t required to recruit research
participants). Or perhaps it was simply due to their several millennia experiential head start.
In any event, ignoring the reproducibility of a scientific finding is a crucial failing because research that is not reproducible is worthless and, even worse, is detrimental to its parent science by (a) impeding the accumulation of knowledge, (b) squandering increasingly scarce societal resources, and (c) wasting the most precious of other scientists’ resources—their time and ability to make their contributions to science. All failings, incidentally, that the reproducibility initiative is designed to ameliorate.
Another Purpose of This Book
While this book is designed to tell a scientific story, to provide practicing scientists with a menu of strategies to adopt (and behaviors to avoid) for assuring that their research can be reproduced by other scientists, or even to serve as a resource for the teaching of reproducibility concepts and strategies to aspiring scientists, these are not the ultimate purposes of the reproducibility initiative—hence not mine either. For while knowledge and altruistic motives may have some traction with those contemplating a career in science or those desiring to change their current practices, such resolutions face powerful competition in the forms of career advancement, families, and the seductive charms of direct deposit.
The ultimate purpose of the myriad dedicated methodologists whose work is described herein involves a far more ambitious task: the introduction of a cultural change in science itself to one that demands not only the avoidance of behaviors specifically designed to produce positive results, but also the adoption of a number of strategies that require additional effort and time on the part of already stressed and overworked scientists—a culture dedicated to the production of correct inferences to the extent that John Ioannidis (2005) will someday be able to write a rebuttal to his pejorative (but probably accurate) subhead from “Most Research Findings Are False for Most Research Designs and for Most Fields” to “False Positive Results Have Largely Disappeared from the Scientific Literatures.” A culture in which the most potent personal motivations are not to produce hundreds of research publications or garner millions of dollars in research funding but to contribute knowledge to their scientific discipline that was previously unknown to anyone, anywhere. And conversely, a culture in which (short of actual fraud) the most embarrassing
professional incident that can occur for a scientist is for his or her research to be declared irreproducible due to avoidable questionable research practices when other scientists attempt to replicate it.
The
Book’s Plan for a Very Small Contribution to This Most Immodest Objective
Almost everyone is aware of what Robert Burns and John Steinbach had to say about the plans of mice and men, but planning is still necessary even if its objective is unattainable or nonexistent. So the book will begin with the past and present conditions that facilitate the troubling prevalence of irreproducible findings in the scientific literature (primarily the odd fact that many disciplines almost exclusively publish positive results in preference to negative ones). Next a very brief (and decidedly nontechnical) overview of the role that p-values and statistical power play in reproducibility/irreproducibility along with one of the most iconic modeling exercises in the history of science. The next several chapters delineate the behavioral causes (i.e., questionable research practices [QRPs]) of irreproducibility (accompanied by suggested solutions thereto) followed by a few examples of actual scientific pathology which also contribute to the problem (although hopefully not substantially). Only then will the replication process itself (the ultimate arbiter of reproducibility) be discussed in detail along with a growing number of very impressive initiatives dedicated to its widespread implementation. This will be followed by equally almost impressive initiatives for improving the publishing process (which include the enforcement of preregistration and data-sharing requirements that directly impact the reproducibility of what is published). The final chapter is basically a brief addendum positing alternate futures for the reproducibility movement, along with a few thoughts on the role of education in facilitating the production of reproducible results and the avoidance of irreproducible ones.
The Sciences Involved
While the book is designed to be multidisciplinary in nature, as mentioned previously it unavoidably concentrates on the social and health sciences. An inevitable emphasis is placed on psychological research since
that discipline’s methodologists have unquestionably been leaders in (but by no means the only contributors to) reproducibility thought and the implementation of strategies designed to ameliorate the unsettling preponderance of false-positive results. However, examples from other sciences (and contributions from their practitioners) are provided, including laboratory and preclinical research (on which some of the most crucial human experimentation is often based) with even a nod or two to the physical sciences.
But while some of the book’s content may seem irrelevant to practitioners and students of the purely biological and physical sciences, the majority of the key concepts discussed are relevant to almost all empirically based disciplines. Most sciences possess their own problems associated with publishing, the overproduction of positive results, statistical analysis, unrecognized (or hidden) questionable research practices, instrumental insensitivity, inadequate mentoring, and the sad possibility that there are just too many practicing scientists who are inadequately trained to ensure that their work is indeed reproducible.
A Few Unavoidable Irritants
Naturally, some of the content will be presented in more detail than some will prefer or require, but all adult readers have had ample practice in skipping over content they’re either conversant with or uninterested in. To facilitate that process, most chapters are relatively self-contained, with cursory warnings of their content posted at the conclusion of their immediately preceding chapter.
Also, while the story being presented almost exclusively involves the published work of others, I cannot in good consciousness avoid inserting my own opinions regarding this work and the issues involved. I have, however, attempted to clearly separate my opinions from those of others.
Otherwise, every topic discussed is supported by credible empirical evidence, and every recommendation tendered is similarly supported by either evidence or reasoned opinions by well-recognized reproducibility thinkers. This strategy has unavoidably necessitated a plethora of citations which only constitute a mere fraction of the literature reviewed. For readability purposes, this winnowing process has admittedly resulted in an unsystematic review of cited sources, although the intent was to represent an overall consensus of
those thinkers and researchers who have contributed to this crucial scientific movement.
A Very Little About My Perspective
In my long academic and statistical consulting career, I have personally witnessed examples of pretty much all of the “good,” “bad,” and “ugly” of scientific practices. Following a brief stint as an educational researcher, my writing and research has largely focused on the methodology of conducting research and the statistical analysis of its results. I even once published an annotated guide to 2,600 published methodological sources encompassing 78 topics, 224 journals, and 125 publishers (Bausell, 1991). The tome was dedicated “to the three generations of research methodologists whose work this book partially represents” (p. viii).
In the three decades that followed, the methodological literature virtually exploded with the publication of more articles, topic areas, and journals (and, of course, blogs) than in the entire history of science prior to that time. And while this work has been extremely beneficial to the scientific enterprise, its main contribution may have been the facilitation of the emergence of a new generation of methodologists studying (and advocating for) the reproducibility of scientific results.
Naturally, as a chronicler, I could hardly avoid recognizing the revolutionary importance of this latter work, not just to research methodology but also to the entire scientific enterprise. My primary motivation for telling this story is to hopefully help promulgate and explain the importance of its message to the potential audiences previously described. And, of course, I dedicate the book “to the present generation of reproducibility methodologists it partially represents.”
I must also acknowledge my debt to three virtual mentors who have guided me in interpreting and evaluating scientific evidence over the past two decades, philosophers of science from the recent and distant past whose best-known precepts I have struggled (not always successfully) to apply to the subject matter of this book as well. In chronological order these individuals are
1. William of Occam, the sternest and most fearsome of my mentors, whose most important precept was the parsimony principle, which
can be reduced to embracing the least involved explanation for the occurrence of a phenomenon that both fits the supporting data and requires the fewest assumptions (and usually constitutes the simplest explanation);
2. Yogi of Bronx, one of whose more important precepts was the principle of prediction, succinctly stated as “It’s tough to make predictions, especially about the future”; and
3. Robert Park, the title of whose most important book pretty much speaks for itself: Voodoo Science: The Road from Foolishness to Fraud (2000). And, as a disclaimer, Bob (the only one of my three mentors I ever actually met) never suggested that mainstream science had arrived at this latter destination—only that far too many scientists have blissfully traveled down that road.
And Finally an Affective Note
Some of the articles, quotations therefrom, and even some of my comments thereupon may appear overly critical. (Although for my part I have made a serious attempt to avoid doing so via the book’s many revisions and greatly facilitated by one of its anonymous peer reviewer’s very helpful comments.) It is important to remember, however, that what we are dealing with here is a paradigmatic shift (or radical change for those who prefer something less pompous) in the way in which science is conducted and published so none of us should be overly judgmental. It takes time for entrenched behaviors to change, and most practicing researchers (including myself) have violated one or more of the movement’s precepts in the past, often unthinkingly because we hadn’t been aware (or taught) that some of the questionable research behaviors discussed here were actually contraindicated.
After all, it wasn’t until sometime around 2011–2012 that the scientific community’s consciousness was bombarded with irreproducibility warnings via the work of scientists such as those discussed in this book (although warning shots had been fired earlier by scientists such as Anthony Greenwald in 1975 and John Ioannidis in 2005). However, this doesn’t mean that we, as formal or informal peer reviewers, should be all that forgiving going forward regarding obviously contradicted practices such as failures to preregister protocols or flagrantly inflating p-values. We should just be civil in doing so.
So Where to Begin?
Let’s begin with the odd phenomenon called publication bias with which everyone is familiar although many may not realize either the extent of its astonishing prevalence or its virulence as a facilitator of irreproducibility.
References
Bausell, R. B. (1991). Advanced research methodology: An annotated guide to sources. Metuchen, NJ: Scarecrow Press.
Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.
Cohen, J. (1977, 1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
Fisher, R. A. (1935). The design of experiments. London: Oliver & Boyd.
Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.
Hill, A. B. (1935). Principles of medical statistics. London: Lancet.
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2, e124.
Kuhn, T. S. (1962). The structure of scientific revolutions. Chicago: University of Chicago Press.
Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations (N. W. Storer, Ed.). Chicago: University of Chicago Press.
Park, R. (2000). Voodoo science: the road from foolishness to fraud. New York: Oxford University Press.
