Story of BrainStorm's Stem Cell Treatment for ALS - Criticality of the Statistical Analyses

This past week, the biotech company BrainStorm announced the decision to submit a BLA to the FDA for NurOwn® (a stem cell treatment) for the treatment of ALS (Amyotrophic Lateral Sclerosis). The news stirred quite some discussions. The decision to submit the BLA is driven by the reanalysis or the corrected analysis of the previously announced negative results from their pivotal study. In their news announcement, they stated the following: 

New clinical analyses strengthen the conclusions from NurOwn's® Phase 3 clinical trial

A correction was made to the Muscle and Nerve publication from December 2021 describing the results of NurOwn's® Phase 3 clinical trial in ALS following new clinical analyses which strengthen the Company's original conclusions from the trial. The correction results in a statistically significant treatment difference (p=0.050) of more than 2 points for an important secondary endpoint, average change from baseline in ALSFRS-R, in the pre-specified efficacy subgroup of participants with a baseline score of at least 35. Analyses reported in the original publication utilized an efficacy model that unintentionally deviated from the trial's pre-specified statistical analysis plan by erroneously incorporating interaction terms between the subgroup and treatment. The newly published results, which includes supporting information to the publication, employ the efficacy model as pre-specified in the trial's statistical analysis plan, correcting the analyses. The correction also relates to the other subgroup analyses published for this endpoint, demonstrating that all subgroups with ALSFRS-R baseline scores of at least 26 to 35 showed a statistically significant benefit following treatment with NurOwn® (p≤0.050) on this secondary endpoint.

The reanalysis (or as they called it 'correction') was only on the pre-specified subgroup analyses for the secondary endpoint of ALSFRS-R total score (as highlighted in yellow below from the original publication). 

An erratum was issued to present the 'corrected' results for this endpoint: 

The original publication reported results for ALSFRS-R total score subgroup endpoint using a model that unintentionally deviated from the pre-specified statistical analysis plan by erroneously incorporating interaction terms between the subgroup and treatment. The error was made by the CRO who performed the statistical analyses. Applying the correct statistical model for that outcome resulted in the average difference between NurOwn- and placebo-treated patients going from 2.01 points to 2.09 points, but importantly this difference became statistically significant with a P-value of 0.05 (from a p-value of 0.20 in the original analysis). 

While the trial did not reach statistical significance on the primary or secondary endpoints, the company believes these corrected analyses support the conclusion that NurOwn has a positive treatment effect for patients with ALS. 

A year and a half ago, FDA put out a statement (unusual) to advise the BrainStorm not to file the BLA based on the announced results after unblinding of their phase 3 study. FDA specifically stated the following: 

With the recent completion of a randomized phase 3 controlled clinical trial comparing NurOwn to placebo, it has become clear that data do not support the proposed clinical benefit of this therapy. Data indicated that none of the primary or secondary endpoints were met in the group of patients who were randomized. For the main (primary) endpoint, 27.7% of people given the placebo were scored as responding compared to 32.6% of people given NurOwn. The 4.9% absolute difference in responders was not at all statistically significant, and the small difference between the two groups was most likely due to chance. In addition, there was a modest excess in deaths in those treated with NurOwn, the significance of which is unclear at this time. If BrainStorm plans further studies of NurOwn to determine if the product can provide clinical benefit to individuals with ALS, FDA will continue to provide advice to the company on their development program.

Now, A year after FDA slammed on the breaks, BrainStorm is hitting the gas with updated data, approval plans, we will see how the FDA will react to BrainStorm's plan and if FDA will accept the BLA filing by BrainStorm. 

No matter what the fate is for BrainStorm's BLA, one thing is clear: the statistical analyses are critical to the clinical trials and to the overall drug development. It is so important to avoid errors/mistakes in the statistical analyses. This important point has been discussed in previous posts such as "Statistician's nightmare - mistakes in statistical analyses of clinical trials" and "Futility Analysis and Conditional Power When Two Phase 3 Studies are Simultaneously Conducted" where the inappropriate method for futility analysis was implemented. 

It is surprising that the p-value and the statistical significance are still playing a critical role in regulatory decision-making after all of these discussions about retiring statistical significance and p-value. 

Mediation analysis and SAS CAUSALMED procedure

In a recent publication (Benza et al "Contemporary Risk Scores Predict Clinical Worsening in Pulmonary Arterial Hypertension - An Analysis of FREEDOM-EV"), we conducted an analysis called 'Mediation analysis'. In the statistical analysis section, the 'mediation analysis' was stated as the following: 

"To determine whether the change in Week 12 REVEAL Lite 2 risk score ‘mediated’ the treatment effect in delaying clinical worsening, we used SAS (v14.3) CAUSALMED procedure which operationalizes the work of Valeri and VanderWeele.

This analysis attempts to determine what fraction of the total treatment effect appears to be attributable to the treatment effect on the REVEAL Lite 2 score. The analysis was adjusted for baseline REVEAL Lite 2 score; we did the analysis both with and without assuming that there is a treatment and mediator (REVEAL Lite 2 score) interaction on the outcome model (clinical worsening). The definition ‘net clinical benefit’ has been previously proposed as the achievement of all three French non-invasive low risk factors without a clinical worsening event; we retrospectively used the present database to model the performance of this definition."

According to Wikipedia, the mediation model and mediation analysis are defined as the following: 

In statistics, a mediation model seeks to identify and explain the mechanism or process that underlies an observed relationship between an independent variable and a dependent variable via the inclusion of a third hypothetical variable, known as a mediator variable (also a mediating variable, intermediary variable, or intervening variable). Rather than a direct causal relationship between the independent variable and the dependent variable, a mediation model proposes that the independent variable influences the mediator variable, which in turn influences the dependent variable. Thus, the mediator variable serves to clarify the nature of the relationship between the independent and dependent variables.

Mediation analyses are employed to understand a known relationship by exploring the underlying mechanism or process by which one variable influences another variable through a mediator variable. In particular, mediation analysis can contribute to better understanding the relationship between an independent variable and a dependent variable when these variables do not have an obvious direct connection.

A mediator variable can either account for all or some of the observed relationship between two variables. 

Full Mediation

Maximum evidence for mediation, also called full mediation, would occur if the inclusion of the mediation variable drops the relationship between the independent variable and dependent variable to zero. In other words, the effect of the independent variable on the dependent variable is all through the mediator variable. 

Partial mediation

Partial mediation maintains that the mediating variable accounts for some, but not all, of the relationship between the independent variable and dependent variable. Partial mediation implies that there is no only a significant relationship between the mediator and the dependent variable, but also some direct relationship between the independent and dependent variable - the line from independent variable to dependent variable is solid and c is not equal to zero. 

 

The mediation analysis has been used in the data analysis for observational data, clinical trial data, survey data, and epidemiology study data. 

In an article by Eyre et al "Effect of Covid-19 Vaccination on Transmission of Alpha and Delta Variants", the mediation analysis was used to assess whether the effect of the vaccination status of the index patient was explained by Ct values at diagnosis.  Ct values are cycle-threshold values (indicative of viral load21) in the index patient.

In an article by Reaven et al "Intensive Glucose Control in Patients with Type 2 Diabetes — 15-Year Follow-up", mediation analyses were performed:

In prespecified mediation analyses, Cox proportional-hazards models were used to examine the effects of the glycated hemoglobin level on the primary cardiovascular disease outcome and on the observed treatment effects. Specifically, the log-linear association of the cumulative glycated hemoglobin level (modeled as a time-varying covariate) with the primary cardiovascular disease outcome was assessed during the period of separation of the glycated hemoglobin curves and after convergence. Models examined the effect of treatment group (intensive therapy or standard therapy) on the primary outcome in an unadjusted analysis (model 1) or while accounting for baseline, most recent, or cumulative mean glycated hemoglobin level (models 2, 3, and 4, respectively).

A paper by Vo et al summarized "the conduct and reporting of mediation analysis in recently published randomized controlled trials: results from a methodological systematic review"

Mediation analysis can be performed using SAS procedure CAUSALMED. CAUSALMED procedure was developed for estimating causal mediation effects from observational data, but can definitely be used for estimating mediation effects from the randomized controlled clinical trial data. Please see the references below:

• SAS documentation about CAUSALMED procedure including the application examples
• Yung et al Causal Mediation Analysis with the CAUSALMED Procedure
• Yung Introducing the CAUSALMED procedure for causal mediation analysis - Youtube video 
• Youtube video: Modeling causal effect using SAS/Stat

Mediation analysis can be performed using other software. This is very well summarized in a paper by Valente et al "Causal Mediation Programs in R, Mplus, SAS, SPSS, and Stata".

Handling of values below or above a threshold (Below the Low Limit of Quantification or Above the Upper Limit of Quantification)?

In clinical trials, the samples are often collected and sent to the central laboratory or specialty laboratory for measuring certain parameters (drug concentrations, metabolite concentrations, biomarkers,...). It is not uncommon that the results may be reported as "<xxx" or ">xxx" indicating that the measurement is below or above a threshold, outside the range, or the quality control curves. We call them below the low limit of quantification or above the upper limit of quantification.

How to handle them in the data set and in the analyses?

In the data set, the laboratory results should be reported as it is in character variable. The data listings should use the character variable so that the signs of '<' or '>' will be kept and displayed.

For the purpose of the statistical summaries and analyses, a separate numeric variable should be derived and appropriate rules will be applied to these values below the low limit of quantification or above the upper limit of quantification.

 
Most of the discussions were about the handling of the values below the low limit of quantification (BLQs). See a previous post "BLQs (below limit of quantification) and LLOQ (Lower Limit of Quantification): how to handle them in analyses?" and the researchgate.net discussion board "How should one treat data with <LOQ values during statistical analysis?".

The options for handling the BLQs are:

• Treat BLQs as missing
• Treat BLQs as 0
• Treat BLQs as 1/2 of the LLQ (lower limit of qualification). For example, if the result was reported as "<10" µg, take 5 µg as the measure - this approach is pretty common in handling pharmacokinetic concentration data. 
•  Simply remove the sign of  '<' and take the face value (i.e. LLQ value). For example, if the result was reported as "<10" µg, take 10 µg as the measure. 
• More complicated methods using statistical (regression, maximum likelihood,...) approaches 

There are fewer discussions about handling the values above the upper limit of quantification (ULQ). Usually, these values above the upper threshold will be handled by:

• Treat values above ULQ as missing
• Simply remove the sign of  '>' and take the face value (i.e. ULQ value). For example, if the result was reported as ">100" mg, take 100 mg as the numeric value

In an SAP developed by Astellas, the simple rule was specified for handling the values below or above a threshold:

"For continuous variables that are recorded as “< X” or “> X”, the value of “X” will be used in the calculation of summary statistics. The original values will be used for the listings."

In an SAP developed by Galapagos for their phase 3 study of GLPG1690 in subjects with idiopathic pulmonary fibrosis, the following rules were proposed to handle values below or above a threshold. Their approach of adding or deducting a small number from the face value is unconventional.

7.3. Handling of Values Below (or Above) a Threshold 

Values below (above) the detection limit will be imputed by the value one unit smaller or larger than the detection limit itself. In listings, the original value will be presented. Example: if the database contains the value “<0.04”, then for the descriptive statistics the value “0.03” will be used. The value “>1000” will be imputed by “1001”. 

Placebo effect and its impact on the overall treatment effect

RCTs (randomized, controlled clinical trials) are still the golden standard in clinical research. In RCTs, the most common control group is the Placebo. According to Wikipedia, a placebo is a sham substance or treatment which is designed to have no known therapeutic value. Common placebos include inert tablets (like sugar pills), inert injections (like saline), sham surgery, and other procedures. In order to maintain the blinding (masking), the placebo group may include additional excipients similar to the experimental drug so that the placebo group will have the same characteristics as the experimental drug in shape, size, color, texture, weight, taste, smell,......

Placebo is assumed to have no therapeutic effect or detrimental effect. However, the assumption may not be true especially when the composition of the placebo is in consideration. 

In previous article "Placebo Effect, Honest Placebo, Open-label Placebo", we discussed the placebo effect in diseases in the CNS and psychological area or in diseases with subjective symptom measures. In the post "Placebo effect and the choice of placebo", we discussed the composition of the placebo and some 'placebo' may actually have therapeutic effect. For example, in clinical trials to test the therapeutic effects of IGIV, the  low concentration of albumin may be selected as the placebo control - the low concentration of albumin may actually have the therapeutic effect. In both of these cases, the placebo effect or potential therapeutic effect from the 'placebo' can cause the unexpected higher response rate in Placebo arm, therefore decrease the difference between the experimental drug and the placebo groups, result in the failed trials. 

On the flip side, the placebo can have detrimental effect. There are quite some recent discussions about the placebo having the detrimental effect - consequently, the overall treatment effect observed in the clinical trials may not be due to the therapeutic effect of the experimental drug, but due to the detrimental effect of the placebo group. In other words, the overall treatment effect can be  overestimated due to the detrimental effect of the placebo.

Here are some articles discussing the potential detrimental effects of the placebo in clinical trials to study the effects of fish oil in the prevention of the cardiovascular events. The REDUCT-IT trial was published in NEJM and was the pivotal trial resulting in the FDA and EMA's approval. According to the study protocol, "the matching placebo capsule is filled with light liquid paraffin and contains 0 mg of AMR101 (icosapent ethyl [ethyl-EPA])." The detrimental effect of the placebo may come from the paraffin. 

• Effects of Randomized Treatment With Icosapent Ethyl and a Mineral Oil Comparator on Interleukin-1β, Interleukin-6, C-Reactive Protein, Oxidized Low-Density Lipoprotein Cholesterol, Homocysteine, Lipoprotein(a), and Lipoprotein-Associated Phospholipase A2: A REDUCE-IT Biomarker Substudy
• Amarin stands behind Vascepa after new analysis raises questions about its efficacy
• New analysis of Amarin’s Vascepa resurrects big questions about effectiveness
• New Biomarker Data Add to Concerns Over REDUCE-IT Trial
• Mineral oil: safety and use as placebo in REDUCE-IT and other clinical studies

When we conduct the placebo-controlled clinical trials, the composition of the placebo needs to be carefully considered and the potential therapeutic effect from the placebo needs to be minimized.