Elevidys in DMD: Approved Despite Trial Failed in Primary Efficacy Endpoint

The US Food and Drug Administration has given the green light for the first gene therapy that treats a rare form of muscular dystrophy (Duchenne muscular dystrophy, DMD) to be used in most people who have the disease and a certain genetic mutation. Last year, the drug – Elevidys, from the biotech company Sarepta Therapeutics – was approved (through accelerated approval pathway) to treat only children ages 4 and 5 with Duchenne muscular dystrophy, one of the most severe forms of inherited muscular dystrophies, who have a confirmed mutation in a gene called DMD that is associated with muscle strength. The FDA announced this past Thursday (June 20, 2024) that it had given traditional approval for Elevidys for ambulatory people 4 and older with a confirmed mutation in the DMD gene and accelerated approval for non-ambulatory people 4 and older with this mutation. There’s not enough data on safety to support its use in children under 4, the agency says.

This is another drug approved by the FDA although the pivotal study (so called EMBARK study) failed in the primary efficacy endpoint (the North Star Ambulatory Assessment (NSAA)). While secondary efficacy endpoints (such as Time to Rise From the Floor) were statistically significant in favor of the Elevidys treatment group, the secondary efficacy endpoints were supposed not to be tested if the primary efficacy endpoint was not statistically significant. 

FDA's action of giving Elevidys traditional approval is controversial. According to FDA's review documents, some of FDA internal reviewers (such as statistical reviewer) were against the approval citing there wasn't sufficient evidence, however, the head of FDA CBER division, Dr Peter Marks, overruled the functional group reviewers to approve this gene therapy.  

• Top FDA official overrules staff to approve gene therapy that failed trial
• Duchenne approval exposes FDA rift over Sarepta gene therapy
• Déjà Vu: CBER Director’s Intervention Saves Sarepta’s Elevidys Again
• How controversial was the decision by FDA’s Peter Marks to approve Sarepta’s gene therapy? Check its footnotes

Perhaps, Dr Peter Marks is right. We should not base the approval on the single p-value from the primary efficacy endpoint and we should not be a slave of the p-values. FDA's announcement says "In making this decision, the FDA considered the totality of the evidence, including the potential risks associated with the product, the life-threatening and debilitating nature of the disease and the urgent unmet medical need." In the rare disease setting, selecting a clinical scale as the primary efficacy endpoint can sometimes be a gambling decision. in DMD situation, FDA published a guidance for industry "Duchenne Muscular Dystrophy and Related Dystrophinopathies:  Developing Drugs for Treatment". The guidance did not recommend which endpoint should be used the primary efficacy endpoint. The guidance was indecisive in efficacy endpoints. 

"FDA has no defined set of required or recommended clinical outcome measures for studies in dystrophinopathies.  Although existing outcome measures developed for clinical trials and/or clinical care in dystrophinopathies or related conditions may be appropriate, FDA will also consider proposals for the use of novel outcome measures that are capable of measuring clinically meaningful effects in patients.  FDA encourages sponsors to propose and, if necessary, develop endpoints that can validly and reliably assess patients with a wide spectrum of symptoms and disease stages.  Sponsors should engage FDA early during the selection and/or development of efficacy endpoints.  The sponsor should include an assessment of multiple efficacy endpoints, when feasible, to characterize the breadth of effects on dystrophin-related pathologies, including skeletal, respiratory, and cardiac muscle function, even if the primary endpoint is only one of these measures. "

Even if the drug is effective, the statistical significance for the selected primary efficacy endpoint (NSAA total score in EMBARK study) may not be demonstrated. In the rare disease setting with the urgent unmet medical need, the strict statistical rules may need to be loosened, the sequential testing rule for controlling the overall alpha may need to be skipped, and the totality evidence from the trial needs to be considered in the decision making.

DMD, like the ALS amyotrophic lateral sclerosis, is a challenging disease for clinical trials. Majority of the late phase DMD trials failed. FDA's approval of Elevidys in DMD comes right after two recent failed trials in DMD by Pfizer and NS Pharma. Their clinical programs in DMD were stopped. 

• Pfizer’s Phase III DMD gene therapy trial misses primary endpoint
• NS Pharma's Duchenne therapy Viltepso fails confirmatory trial

Sarepta's Elevidys in DMD (EMBARK trial) and Amylyx's RELYVRIO in ALS (PHOENIX trial) faced similar situations: they received preliminary approval from the FDA but were required to conduct confirmatory studies to demonstrate clinical benefit. However, both trials failed to achieve statistical significance in their primary efficacy endpoints. The key difference lies in the EMBARK study, which showed significant differences in secondary efficacy endpoints, whereas the PHOENIX study failed to demonstrate statistical significance in any endpoints - therefore, the fate for Elevidys (traditional approval is obtained) and Relyvrio (the product was withdrawn from the market) is totally different now.  

Significant level versus p-value

Sometimes, the significant level and p-value are getting mixed up and confusing to some non-statisticians. It is not surprising to receive a question or request for statistician to design a study to obtain a p-value of 0.05 or 0.01. While the significant level and p-value are closed related, ,they are used in different stage of the trial - significant level is used in the study design stage and p-value is used in the analysis stage.

A significant level is usually set at 0.05 at the study design stage. After the study, data is analyzed and p-value is calculated. The p-value is then compared to the pre-specified significant level to determine if the study results is statistically significant. 

If the significant level is set at 0.01 at the study design stage, which is temped for avoiding doing two pivotal studies, it will set the unnecessary high bar for declaring the successful trial in the analysis stage. 

"The significance level," "alpha" (α), and "Type I error rate" are essentially referring to the same concept in the context of hypothesis testing. These terms are often used interchangeably and are closely related. Here's a brief explanation of each:

Significance Level (Alpha, α): The significance level is a pre-defined threshold (usually denoted as α) set by the researcher before conducting a statistical test. It represents the maximum acceptable probability of making a Type I error. Common choices for alpha include 0.05 (5%), 0.01 (1%), and others. It determines the level of stringency for the test, where a smaller alpha indicates a more stringent test.

Significant level is just one of the parameters in calculating the sample size during the study design stage. Other parameters include the effect size (assumed treatment difference), the standard deviation, statistical power (type 2 error), and alpha adjustment due to multiplicity issue, interim analyses,...

Alpha (α): Alpha is the symbol used to represent the significance level in statistical notation. When you see α, it's referring to the predetermined threshold for statistical significance.

Type I Error Rate: The Type I error rate is the probability of making a Type I error, which occurs when you reject the null hypothesis when it is actually true. The significance level (alpha) directly relates to the Type I error rate because the significance level sets the limit for how often you are willing to accept such an error. The Type I error rate is typically equivalent to the significance level (alpha), assuming the test is properly conducted.

P-value: The p-value is calculated as part of the statistical analysis after the data has been collected. It measures the strength of the evidence against the null hypothesis based on the collected data. A smaller p-value indicates stronger evidence against the null hypothesis, and a larger p-value suggests weaker evidence.

The p-value measures the strength of evidence against a null hypothesis. The p-value is the probability under the assumption of no effect or no difference (null hypothesis) of obtaining a result equal to or more extreme than what was actually observed. The 'p' stands for probability and measures how likely it is what any observed value between 0 and 1. Values close to 0 indicate that the observed difference is unlikely to be due to chance, whereas a p value close to 1 suggests that it is highly likely that the difference observed is due to chance. If the p-value is low, it suggests evidence against the null hypothesis, and then alternative hypothesis (assumption of the effect or difference) will be accepted. 

The p-value indicates how incompatible the data are with a specified statistical model constructed under a set of assumptions, together with a null hypothesis. The smaller the p-value, the greater the statistical incompatibility of the data with the null hypothesis. When we get a p-value that is greater than the pre-specified significant level, we fail to reject the null hypothesis - it means that there is insufficient evidence to reject. 

STAT national biotech reporter Damian Garde explains what p-value is:

Even though hypothesis testing and p-value have been criticized (see a previous post "Retire Statistical Significance and p-value?"), the p-value is still the primary indicator by the sponsor, regulator, medical community, and pretty much everybody to judge if a clinical trial is successful or not. 

,

Regulatory approval of a medicinal product depends on more than just a p-value. The approval depends on the totality of the evidence, the magnitude of the treatment difference, clinical significance or clinical meaningfulness, the confidence interval of the estimate, the safety profile, whether the benefit outweighs the risk.

We have seen the cases that the drug is approved even though the p-value was not statistically significant (i.e., did not reach the pre-specified significant level). See the previous post "Drugs Approved by FDA Despite Failed Trials or Minimal/Insufficient Data". We also see the cases that the drug was not approved even though the p-value was statistically significant. See the article "FDA blocks Alnylam's bid to expand Onpattro label" even though the study results were statistically significant and published in the NEJM "Patisiran Treatment in Patients with Transthyretin Cardiac Amyloidosis".

In the end, we can't retire the p-value. We relied on the p-value to measure how strong the evidence is. However, we should not be the slave of the p-value. 

Paired T-test and McNemar's test for paired data based on the summary data

Sometimes, it is necessary for us to calculate the p-values based on the summary (aggregate) data without the individual subject level data. In a previous post, group t-test or Chi-square test based on the summary data was discussed. Group t-test and chi-square test can be used in the setting of parallel-group comparisons. 

In single-arm clinical trials, there is no concurrent control group, and the statistical test is usually based on the pre-post comparison. For continuous measures, the pre-post comparison can be tested using paired t-test based on the change from baseline values (i.e., post-baseline measures - baseline measures):  For discrete outcomes, the pre-post comparison may be tested using McNemar's test.

Paired t-test:

A paired t-test is used when we are interested in the difference between two variables for the same subject. Suppose we have the descriptive statistics for change from baseline values: 83 subjects had the outcome measures at both baseline and week 12 (therefore, 83 pairs), the mean and standard deviation for these 83 pairs are: 10.7 (70.7); 68 subjects had the outcome measures at both baseline and week 24 (therefore 68 pairs), the mean and standard deviation for these 68 pairs are 20.2 (80.9). 

With the mean difference, the standard deviation for differences, and the sample size (# of pairs), we have all the elements to calculate the t statistics and therefore the p-value using the formula below. 

This can be implemented in SAS as the following - t-statistics and p-values can be calculated for each of weeks 12 and 24 based on the aggregate data. 
 

McNemar's Test:

McNemar's test is a statistical test used on paired nominal data. It is applied to 2 × 2 contingency tables with a dichotomous trait, with matched pairs of subjects, to determine whether the row and column marginal frequencies are equal (that is, whether there is "marginal homogeneity"). In clinical trials, the aggregate data may not be obvious as a 2 × 2 contingency table but can be converted into a 2 × 2 contingency table. 

Suppose we have the following summary data for post-baseline week 12: the number and percentage of subjects with improvement, stable (no change), and deterioration categories. 

		All subjects (n=300)
Week 12	Improved	54 (18%)
	No Change	228 (76%)
	Deteriorated	18 (  6%)

At Week 12, there are more subjects in the 'Improved' category than in the 'Deteriorated' category even though the majority of subjects are in the 'No Change' category. Are they more subjects with improvement than deterioration? 

Assuming that change from category 1 to 0 is 'Improved' and change from category 0 to 1 is 'Deteriorated', the table above can be converted into a 2 × 2 table: 

		Baseline
		0	1
Week 12	0	228	54
	1	18	0

or

		Baseline
		0	1
Week 12	0	0	54
	1	18	228

For McNemar’s test, only the numbers in the diagonal discordant cells (in our case, the # of improved and the # of deteriorated) are relevant.

The concordant cells (in our case, the # of no change) will only contribute to the sample size (therefore the degree of freedom), not have an impact on the p-value. How the # of subjects with the ‘No Change’ is split doesn’t matter with our calculation of chi-square statistics and therefore the p-value.

For the data highlighted in yellow, McNemar’s test can be performed using the SAS codes like this (weight statement indicates count variable is the frequency of the observation and agree option requests McNemar's test). How the 228 subjects in the concordance ‘No Change’ category are split has no impact on the p-value calculation. 

Retire Statistical Significance and p-value? - Revisited

In 2019, there was a public debate about the use or misuse of statistical significance and p-values. I wrote a post about it "Retire Statistical Significance and p-value?"

Obviously, the hypothesis testing, statistical significance, and p-value are still the cornerstone of our clinical trials, the basis for judging if a clinical trial is a success, and the basis for the decision-making for regulatory approvals by the FDA and other regulatory authorities. 

In the latest issue of AMSTATNews and also the Annals of Applied Statistics, there is an article "The ASA President’s Task Force Statement on Statistical Significance and Replicability". The statements confirmed that the significance and p-values are here to stay. 

"P-values are valid statistical measures that provide convenient conventions for communicating the uncertainty inherent in quantitative results. Indeed, p-values and significance tests are among the most studied and best understood statistical procedures in the statistical literature. They are important tools that have advanced science through their proper application."

"p-values and significance tests, when properly applied and interpreted, increase the rigor of the conclusions drawn from data. Analyzing data and summarizing results are often more complex than is sometimes popularly conveyed. Although all scientific methods have limitations, the proper application of statistical methods is essential for interpreting the results of data analyses and enhancing the applicability of scientific results."

I would say the following about the significant tests and p-values:

• We embrace it, not abandon it. 
• We focus on the appropriate use and interpretation
• We should not become the slave of the p-values

A quote from  Alfred Marshall, 1885:

"The most reckless and treacherous of all theorists is he who professes to let facts and figures speak for themselves, who keeps in the background the part he has played, perhaps unconsciously, in selecting and grouping them."