The year, 2008. The journal, Amyotrophic Lateral Sclerosis – a small journal in the neuroscience world. The paper? One of the most important studies published in ALS drug development in the past 15 years.  

Titled “Design, power, and interpretation of studies in the standard murine model of ALS,” it was the product of several years of work by the ALS Therapy Development Institute (ALS TDI) and supported by partner statisticians at the University of California, San Francisco. The paper reviewed the quality of ALS mouse studies and retested eight treatments that had been reported to extend the lifespan of these mice (1).

The result? Not one of these eight compounds worked as reported. Not one extended the lifespan of ALS mouse models by a significant margin. It was unlikely that any of them held true promise as a potential treatment for people with ALS.

In today’s Science Sunday, we look back to ALS TDI’s first ever published study to explore the state and standardization of SOD1 mouse model studies, the importance of reproducibility in science, and how in the drug development world, everything has value – even failed tests and negative results.

 

The Background: Modeling ALS

Let’s back up and start at the beginning. Mouse, or murine, models are a common tool used to study the efficacy of different therapeutic compounds in treating ALS symptoms and prolonging lifespan. The mice used in the ALS TDI lab represent the most commonly used mouse model for ALS - the SOD1G93A mouse model.

ALS is not typically seen in mice, so these mice are genetically modified to develop ALS. The mice are engineered to have approximately 23 copies of a mutated human SOD1 gene. Mutations in the SOD1 gene are associated with about 2% of people with ALS and are one of several mutated genes identified in familial ALS (FALS) patients. When given this SOD1 transgene – “transgene” meaning any artificially introduced gene – the mice develop ALS symptoms similar to those experienced by people with ALS; they progressively lose muscle function, dexterity, and weight.

The lab tracks this progression by determining a daily NeuroScore from 0 to 4 for each mouse, where 0 is asymptomatic and 4 is full paralysis. When their symptoms are left untreated, SOD1G93A mice quickly progress to NeuroScore 4, living around 130 days. In efficacy testing at the preclinical research stage, mice are split into two groups, or cohorts – one treated group and one control group that does not receive the treatment, but is otherwise treated in exactly the same way. The lifespan of the control group will remain around 130 days, but if the lifespan of the treated mice increases or if the symptom progression slows, then the drug is earmarked and committed to further testing to determine whether or not it holds promise as an ALS treatment.

 

The Problem: Failed Tests

As of 2008, in SOD1G93A studies performed by other laboratories, eight drugs (minocycline, creatine, celecoxib, sodium phenylbutyrate, ceftriaxone, WHI-P131, thalidomide, and riluzole) had made it to this stage – they’d been seen to extend lifespan in SOD1 mouse models, in some cases by up to 49% (1). Some of these drugs had gone on to clinical trial, some were being touted as the next big thing in the ALS research world, and riluzole was already approved as a treatment for ALS by the FDA.

But despite the initially positive results reported by other researchers, treatment after promising treatment failed under further screening tests done by ALS TDI. None of these drugs extended lifespan by any significant margin - the results didn’t add up. Led by then president Sean Scott, ALS TDI developed an enormous multi-year study to try to understand why.  

 

The Study: Building a Database

The SOD1G93A model was an impressive tool, but at the time was relatively new to the ALS research world. In every research study, there are inherent variables and intricacies that make it difficult to interpret cause-effect relationships and differences across cohorts. ALS TDI had a hunch that the variables associated with the SOD1G93A model had not yet been accounted for and corrected, increasing the likelihood of faulty drug efficacy results.

Over the course of four years, ALS TDI studied 5,429 untreated SOD1G93A mice to form a massive control data set (1). By pulling groupings of data out at random, scientists could determine how often mouse symptoms and lifespan appeared to increase or decrease based on experimental variability alone. They could play around with the data to identify which variables created the most error and how this error could be minimized.

 

The Findings: Controlling for Variables

Through these studies, ALS TDI scientists were able to identify several important variables that can lead to errors in testing:

  • Littermates experienced similar symptom progression and lifespan, so cohorts were subsequently “litter-matched” by splitting up sibling pairs.
  • Female SOD1G93A mice live an average of 4 days longer than the males, so equal male/female representation within a cohort was made a priority.
  • Around 10% of the mice died earlier than expected due to non-ALS causes, so their data was excluded from the lifespan average.
  • Another 2.6% of the mice had less than 23 copies of the SOD1G93A transgene causing them to live longer lives - their data was censored as well.
  • Overwhelmingly, the largest factor contributing to false drug efficacy data was the number of mice used in the SOD1G93A studies.

Large sample sizes are important to minimize the noise created by any number of other variables. In typical published SOD1 mouse studies, an average of 5 to 10 mice were being employed in each cohort, and sometimes numbers dropped down to just 4 mice per group. With numbers this low, a single low-copy mouse in a treatment cohort could increase the average lifespan of the group by 15 days, or about 11%. This lifespan increase would only be dependent on the variables at play, completely unrelated to the drug being tested.

The solution was easy – add more mice to the cohort - but laboratories are often held back from maintaining large mouse colonies due to the high cost. ALS TDI argued that this was a necessary expense, that a cohort size of at least 24 litter-matched mice was needed to minimize the impact of all other variables. Without these standardized adjustments, no lab could rely on results of another.

 

The Take Away: Saving Time and Money

Unfortunately, unstandardized studies and an inability to reproduce results are common problems in the biomedical world. The crisis of reproducibility, as it’s called, is a constant roadblock not only for ALS drug development, but also cystic fibrosis, cancer, and other life-threatening diseases (2) (3). As recent as 2016, a Nature survey estimated that nearly 70% of biomedical scientists have, at one point, been unable to reproduce published results (4). Over 50% feel that there is a “significant crisis” of reproducibility in science, but this crisis is rarely acknowledged (4).

But in ALS research, this crisis cannot be ignored. Time and money cannot be wasted on drugs that don’t work in the lab and won’t work in the clinic.

This is what makes “Design, power, and interpretation” so important. The paper proved without a doubt that there was undeniable value and an urgent need to standardize the SOD1G93A model. By collaborating with other organizations and pushing to standardize the SOD1G93A model, ALS TDI has helped to ensure that valuable resources are not poured into therapeutics whose high efficacy rates are due to a flawed study design and nothing more.

 

References:

(1) Scott S, Kranz J, Cole J, et al (2008) “Design, power, and interpretation of studies in the standard murine model of ALS”, In Amyotrophic Lateral Sclerosis, 1-12.

(2) Harris R (2017) “Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions”, Basic Books, New York.

(3) Nelson N (2019) “The Reproducibility Crisis in Historical Perspective”, Harvard Radcliffe Institute.

(4) Baker M (2016) “1,500 scientists lift the lid on reproducibility”, In Nature, https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970.

(5) Schnabel, J (2008) “Standard Model”, In Nature, https://www.nature.com/news/2008/080807/full/454682a.html.

 

Science Sunday blogs aim to make ALS research and the work of the ALS Therapy Development Institute more accessible to people with ALS, families, friends, and health care providers. ALS TDI believes in open-source science accessible to all, with the goal of empowering the public with knowledge of ALS. Comments or feedback? Email the author at cfaulds@als.net.