Story of BrainStorm's Stem Cell Treatment for ALS - Criticality of the Statistical Analyses

This past week, the biotech company BrainStorm announced the decision to submit a BLA to the FDA for NurOwn® (a stem cell treatment) for the treatment of ALS (Amyotrophic Lateral Sclerosis). The news stirred quite some discussions. The decision to submit the BLA is driven by the reanalysis or the corrected analysis of the previously announced negative results from their pivotal study. In their news announcement, they stated the following: 

New clinical analyses strengthen the conclusions from NurOwn's® Phase 3 clinical trial

A correction was made to the Muscle and Nerve publication from December 2021 describing the results of NurOwn's® Phase 3 clinical trial in ALS following new clinical analyses which strengthen the Company's original conclusions from the trial. The correction results in a statistically significant treatment difference (p=0.050) of more than 2 points for an important secondary endpoint, average change from baseline in ALSFRS-R, in the pre-specified efficacy subgroup of participants with a baseline score of at least 35. Analyses reported in the original publication utilized an efficacy model that unintentionally deviated from the trial's pre-specified statistical analysis plan by erroneously incorporating interaction terms between the subgroup and treatment. The newly published results, which includes supporting information to the publication, employ the efficacy model as pre-specified in the trial's statistical analysis plan, correcting the analyses. The correction also relates to the other subgroup analyses published for this endpoint, demonstrating that all subgroups with ALSFRS-R baseline scores of at least 26 to 35 showed a statistically significant benefit following treatment with NurOwn® (p≤0.050) on this secondary endpoint.

The reanalysis (or as they called it 'correction') was only on the pre-specified subgroup analyses for the secondary endpoint of ALSFRS-R total score (as highlighted in yellow below from the original publication). 

An erratum was issued to present the 'corrected' results for this endpoint: 

The original publication reported results for ALSFRS-R total score subgroup endpoint using a model that unintentionally deviated from the pre-specified statistical analysis plan by erroneously incorporating interaction terms between the subgroup and treatment. The error was made by the CRO who performed the statistical analyses. Applying the correct statistical model for that outcome resulted in the average difference between NurOwn- and placebo-treated patients going from 2.01 points to 2.09 points, but importantly this difference became statistically significant with a P-value of 0.05 (from a p-value of 0.20 in the original analysis). 

While the trial did not reach statistical significance on the primary or secondary endpoints, the company believes these corrected analyses support the conclusion that NurOwn has a positive treatment effect for patients with ALS. 

A year and a half ago, FDA put out a statement (unusual) to advise the BrainStorm not to file the BLA based on the announced results after unblinding of their phase 3 study. FDA specifically stated the following: 

With the recent completion of a randomized phase 3 controlled clinical trial comparing NurOwn to placebo, it has become clear that data do not support the proposed clinical benefit of this therapy. Data indicated that none of the primary or secondary endpoints were met in the group of patients who were randomized. For the main (primary) endpoint, 27.7% of people given the placebo were scored as responding compared to 32.6% of people given NurOwn. The 4.9% absolute difference in responders was not at all statistically significant, and the small difference between the two groups was most likely due to chance. In addition, there was a modest excess in deaths in those treated with NurOwn, the significance of which is unclear at this time. If BrainStorm plans further studies of NurOwn to determine if the product can provide clinical benefit to individuals with ALS, FDA will continue to provide advice to the company on their development program.

Now, A year after FDA slammed on the breaks, BrainStorm is hitting the gas with updated data, approval plans, we will see how the FDA will react to BrainStorm's plan and if FDA will accept the BLA filing by BrainStorm. 

No matter what the fate is for BrainStorm's BLA, one thing is clear: the statistical analyses are critical to the clinical trials and to the overall drug development. It is so important to avoid errors/mistakes in the statistical analyses. This important point has been discussed in previous posts such as "Statistician's nightmare - mistakes in statistical analyses of clinical trials" and "Futility Analysis and Conditional Power When Two Phase 3 Studies are Simultaneously Conducted" where the inappropriate method for futility analysis was implemented. 

It is surprising that the p-value and the statistical significance are still playing a critical role in regulatory decision-making after all of these discussions about retiring statistical significance and p-value.