The ALS Therapy Development Institute (ALS TDI) was created to address one key problem – a lack of available treatments for people with ALS. However all too often, the reproducibility crisis gets in the way of that goal. Finding treatments for ALS requires not only dedicated research, but also a concerted effort to better the research methodology.
Today, we dig into the roots of the topic of reproducibility: Is there a crisis of reproducibility? Should I know what that means? Does it really matter? And can we talk about reproducibility, guidelines, and protocol without yawning? The answer to at least three of these is yes.
Consensus on Crisis
Reproducibility is the ability of scientific studies to be repeated by another lab to answer the same research questions and come to the same conclusions (1). It is the sort of check-and-balance of the research world, a key marker of good science.
Science “moves forward by corroboration – when researchers verify others’ results,” no matter the field, and ALS research is no different (2). A single experiment provides little more than a drug lead; it is through reproducibility, through repeated experimentation and demonstrated trends, that a lead becomes a noteworthy discovery, one worth further pursuit.
Over the past 15 to 20 years, concerns have been raised over the amount of published research that is non-reproducible for one reason or another, not only in bioscience, but other fields as well (3). More than 70% of scientists surveyed by Nature in 2016 said they had been unable to reproduce published results at some point in their careers (4). A whopping 90% felt that a reproducibility crisis existed in their field (4). To many researchers, the system is “fractured, inefficient, [and] inconsistent” (5). This alarming trend is what has come to be known as the reproducibility crisis.
Numerous organizations and articles, including one written by ALS TDI CEO, Dr. Steve Perrin, have called attention to the crisis, but its causes are a topic of hot debate (4) (6). Though some element of non-reproducibility is natural, it is amplified “due to unhelpful sources, ranging from simple mistakes to methodological errors to bias and fraud” (7). Some feel the reproducibility crisis is a systemic issue born of pressure to publish, power dynamics, and insufficient mentoring. Others believe that selective reporting and peer review are to blame. The issue is complex – and is ultimately an impediment to efficient research.
Drug leads are picked up from published papers and then retested to determine potential. ALS TDI has screened over 400 of these drugs, but the work is increasingly expensive and time consuming. In 2012, ALS TDI estimated that it costs nearly $30,000 to determine the correct drug dosage in animal models (6). When dosage is perfected, another $100,000 and up to 12 months are spent testing a single dose in the model, and this must be repeated at least three times (6). Despite the investment, only a small percentage of drug results are reproducible.
The money spent determining efficacy at the preclinical stage is cost effective, though. If a drug has potential to be tested in clinical trial, then investments skyrocket – thousands of patients, several years, and hundreds of millions of dollars are required to push the drug through the clinical trial process (6). But with better rates of reproducibility the biomedical research world would have more resources available to prioritize the most effective treatments. In a space that is already underfunded, the reproducibility crisis affects ALS research directly.
Patching the Holes
It is clear that the costs of ignoring the reproducibility crisis are high, and despite the hushed conversation about change, some action has been taken. The US National Academies of Science, Engineering, and Medicine (NASEM) has recommended the creation of an independent research policy board (RPB) to best ensure rigor and transparency, serve as a central resource for research policy information and trainings, and encourage cross-institute collaboration (7). They held a plenary session on the “trustworthiness of science” at the annual NASEM meeting in April 2019 and in a follow-up seminar this September, they will continue to develop a formal review board.
Back in 2016, Nature reported that nearly 90% of scientists felt the crisis could be minimized through “more robust experimental design,” “better statistics,” and “better mentorship” (4). The simplest way to combat the reproducibility crisis was with more awareness and improved protocol – recipes, essentially - for experiments and analysis. NASEM has also noted overarching flaws in the “design, conduct and communication of a study” (7). Several journals now require step-by-step protocols to be reported alongside published results, and a number of online repositories, including BioProtocol, Protocol Online, OpenWetWare, and Protocol Exchange, aim to make procedures publicly accessible.
ALS TDI Takes the Lead
ALS TDI was at the forefront of this discussion back in 2008, after years of inefficiency and non-reproducibility in the lab. As described in the last Science Sunday post, after years of failed drug tests, ALS TDI organized a multi-year study and published a set of procedures to help standardize the SOD1 model and improve experimental design. This paper has gone on to be one of ALS TDI’s most cited published works, referenced beyond ALS research and pivotal in bioscience reproducibility discussions (8) (9).
From there, ALS TDI representatives went on to attend two international conferences and develop a detailed protocol for the SOD1 mouse model in 2010 (10). In 2012, ALS TDI joined others at the US National Institute of Neurological Disorders and Stroke to discuss the importance of rigorous study design in prioritizing resources, pipelining the best and safest treatments for clinical trials, and accelerating the development of new therapies (11). Earlier this year, ALS TDI scientists published a revamped protocol related to determining SOD1 mouse copy number (12). ALS TDI has published an additional three papers focused specifically on improving reproducibility in ALS research.
Protocols are a valuable resource. They have the power to share knowledge and neutralize technical variables to ensure the “best returns on investment” in bioscience (5). Description and publication of protocol saves “money, research time, and is more respectful of the mice”, says ALS TDI scientist Val Tassinari in the latest Endpoints Podcast. Being able to build experiments off of existing procedures means labs waste less time and money building experiments from scratch, determining minute details like which solution or centrifuge tube to use, and troubleshooting non-reproducible studies.
Instead of worrying about reproducibility, labs can focus on the important question – whether or not a drug effectively treats ALS.
Science Sunday blogs aim to make ALS research and the work of the ALS Therapy Development Institute more accessible to people with ALS, families, friends, and health care providers. ALS TDI believes in open-source science accessible to all, with the goal of empowering the public with knowledge of ALS. Comments or feedback? Email firstname.lastname@example.org.
(1) Stark P. Before reproducibility must come preproducibility. Nature 2018; 557:613.
(2) Nature. Challenges in irreproducible research. 2018. Online.
(3) Nelson N. The Reproducibility Crisis in Historical Perspective. Harvard Radcliffe Institute 2019.
(4) Baker M. 1,500 Scientists Lift the Lid on Reproducibility. Nature 2016.
(5) Gunsalus CK, McNutt M et al. Overdue: a US advisory board for research integrity. Nature 2019; 566:173-175.
(6) Perrin S. Preclinical research: Make mouse studies work. Nature 2014; 507(7493):423-5.
(7) NASEM. Reproducibility and Replicability in Science; Consensus Study Report Highlights. 2019.
(8) Harris R. Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions. Basic Books, New York. 2017.
(9) Schnabel J. Standard Model. Nature 2008.
(10) Ludolph A, et al. Guidelines for preclinical animal research in ALS/MND: A consensus meeting. Amyotroph Lateral Scler 2010;11(1-2):38-45.
(11) Landis C et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature 2012; 490(7419):187-91.
(12) Tassinari V, Vieira F. A High-throughput qPCR-based Method to Genotype the SOD1G93A Mouse Model for Relative Copy Number. Bio Protocol 2019; 9(12) : e3276.