Futility Analysis and Conditional Power When Two Phase 3 Studies are Simultaneously Conducted

In late-phase clinical trials, an independent Data Monitoring Committee (DMC) is usually set up. If the clinical program includes multiple late-phase studies, the same DMC will be responsible for the entire program. With DMC, the interim analyses can be performed for different purposes:

• The interim analysis for safety
• with pre-specified stopping rule (for example stop the trial if the significant imbalance in # of Serious Adverse Events or in # of deaths)
• without pre-specified stopping rule (rely on DMC members to review the overall safety)
• The interim analysis for efficacy: To see if the new treatment is overwhelmingly better than the control group  - then stop the trial for efficacy
• The interim analysis for futility (futility analysis): To see if the new treatment is unlikely to be better than the control group or the study will be unlikely to achieve its objective given the data at the interim – then stop the trial for futility.

There seem to be more studies with built-in futility analysis without interim analysis for overwhelming efficacy, mainly because of the concerns about the alpha-spending for efficacy. The futility analysis will have an impact on the beta-spending and the statistical power, but not on the alpha-spending. For the decision-making, regulatory agencies are usually more concerned about the alpha level (incorrectly approves a drug that does not work) or the alpha level inflation. The sponsors are more concerned about the statistical power (incorrectly concludes a drug not working while the drug is actually working).

Futility analysis usually requires calculating the Conditional Power (CP) that is defined as the probability that the final study result will be statistically significant, given the data observed thus far at the time of the interim data cut and a specific assumption about the pattern of the data to be observed in the remainder of the study, such as assuming the original design effect (alternative hypothesis) or the effect estimated from the interim data.  

If there is one single pivotal trial, the stopping rule and the CP are relatively straightforward. However, it is uncommon that the sponsor may need to conduct two pivotal (phase 3) studies (two adequate and well-controlled (A&WC) trials in FDA's term) to demonstrate substantial evidence of effectiveness as outlined in FDA guidance for industry "Demonstrating Substantial Evidence of Effectiveness for Human Drug and Biological Products Guidance for Industry".

For a clinical program with two independent A&WC trials (usually with identical design), the futility analysis and CP calculation are a little bit more complicated. Two independent A&WC trials may have an identical design but be executed differently (i.e., may not be started at the same time; may be conducted in different geographic regions/countries; and may have different enrollment speeds,...). 

When futility analysis is performed for two A&WC trials, should the conditional powers be calculated for individual studies separately or should the conditional powers be calculated for both studies together (i.e. pooled data from both studies)? 

When there are two identical A&WC trials, the interim analysis for safety should be based on the pooled data sets from both studies because it will give a more definitive answer to the safety issues, the interim analysis for efficacy should be based on the individual study data because the decision about the overwhelming efficacy should be based on the individual study, not the integrated data from two studies; the interim analysis for futility is a little bit more complicated and the decision to use the data from an individual study or to use the data from the pooled data seems to be dependent on how close the observed results from two A&WC trials are at the time of the interim analysis. 

For futility analysis using stochastic curtailment procedure, While CPs can be calculated for each individual study assuming that the treatment effect in the remaining subjects in the same study will follow the treatment effect estimated from the data of this same study at the time of the interim data cut, 

There is an alternative way to calculate the CP, i.e., to calculate the CP for each individual study, but use the observed treatment effect from the pooled data at the interim from both studies to project the trend and pattern for the remaining subjects. 

According to the paper by Lan and Wittes (1988) "The B-Value: A Tool for Monitoring Data", the CP calculation involves the decomposition of overall critical value (B-value or B1 for example) into the sum of two statistically independent interval B-values: 

• Bt, the value of B that accumulated up through time t when interim analysis is conducted; and 
• (B1 - Bt), the incremental value of B that accumulates from time t through the end of the study. The legitimacy of the decomposition follows from the independence of distributions of the outcomes for successive study subjects

At the time t when the interim analysis is conducted, Bt is known and is estimated from the observed data up to the time t. (B1 - Bt) is a random variable that needs to be estimated. The conditional power is derived by fixing Bt and calculating the probability that Bt + (B1 - Bt) will exceed Z1-a/2.

To calculate the CPs when there are two identical A&WC studies, t, as a measure of the information fraction, will be different for different studies. At the time t, maybe 60% of subjects have been enrolled in study #1 while 50% of subjects are enrolled in study #2. In CP calculations, the Bt part will be obtained from the individual study. The (B1-Bt) part is estimated assuming the remaining data following the observed effect up to the interim time t, should the observed effect up to the interim time t be based on the data from the individual study or from the pooled data?

It turns out both approaches can be used: 

• estimate the treatment differences for each individual study and calculate the CP assuming that the reminding data follows the trend and pattern based on the observed data from individual study
• estimate the treatment difference from both studies and calculate the CP assuming that the remaining data follow the trend and pattern based on the observed data from the pooled data of two studies.           

For both of these approaches, the CPs will be calculated for each individual study (therefore one CP for each study). The difference between these two approaches is in the calculation of the (B1-Bt) part - based on the individual study itself or based on the pooled data from both studies. 

We can take a look at the famous and controversial case in Biogen's aducanumab program in Alzheimer's disease. Aducanumab program in Alzheimer's diseases consisted of two pivotal, phase 3 studies (EMERGE (study 301) and ENGAGE (study 302)), and both studies were designed the same and conducted simultaneously globally. Each study had two active arms (low dose and high dose of aducanumab) versus placebo - therefore two hypothesis tests (low dose vs. placebo and high dose vs. placebo). There was a total of four hypothesis tests (two for each study).  The protocol and SAP specified the interim analysis for futility. 

An interim analysis was performed after approximately 50% of the subjects had the opportunity to complete the Week 78 visit for both EMERGE and ENGAGE studies. An interim analysis for the futility of the primary endpoint was performed to allow early termination of the studies if it was evident that the efficacy of aducanumab was unlikely to be achieved. The futility criteria were based on conditional power, which was the chance that the primary efficacy endpoint analysis would be statistically significant in favor of aducanumab at the planned final analysis, given the data at the interim analysis. The CP was calculated assuming that the future unobserved effect was equal to the maximum likelihood estimate of what is observed in the interim data. 

For each study, two CPs were calculated. The pre-specified CP calculation was to use the pooled interim data from both EMERGE and ENGAGE studies for the (B1-Bt) part and assume that the treatment effect for the remaining of the study would follow the observed treatment effect at the interim analysis. At the interim analysis, the CPs were calculated to be 13% for low dose vs placebo and 0% for high dose vs. placebo in EMERGE study, and 11% for low dose vs placebo and 12% for high dose vs. placebo in ENGAGE study. Given all four CPs were lower than the threshold of 20% (a criterion for futility), the DMC recommended stopping both studies for futility.  Biogen followed the DMC recommendation and stopped both EMERGE and ENGAGE studies for futility. 
.

Only after two terminated studies were wrapped up, the reanalyses of the final data indicated that there were statistically significant treatment differences in one of the studies (the ENGAGE study). With the help of the FDA, Biogen was able to submit the BLA and obtain approval for aducanumab for Alzheimer's disease. Leading to the FDA approval, there was an advisory committee meeting to review the aducanumab data. In FDA's presentation, the conditional powers were retrospectively re-calculated - this time, the conditional powers were calculated for each individual study and assumed future unobserved effect would be similar to the interim data for each individual study (not the pooled interim data). FDA claimed that CPs using this approach were more appropriate and would have one of the four CPs above the threshold of 20% (CP=59% for high-dose vs placebo in ENGAGE study) - the studies would not be recommended for stopping for futility. 

Retrospectively, CPs calculated for each study independently (not using the pooled interim data to project the trend and pattern for the remaining data) seemed to be better in Biogen aducanumab program consisting of two A&WC trials. 

However, in a paper by Deng et al "Superiority of combining two independent trials in interim futility analysis", CP calculation using the observed treatment effects from the pooled interim data from two studies was considered a better approach. It concluded, "it is demonstrated that by leveraging data from the other study, the probability of making correct interim decision is increased if the treatment effects are similar between the two studies, and such benefit remains even if there is small to moderate between-study difference."

It is probably true that CP calculation using the pooled data at the interim to project the trend and pattern for the remainder data is a better approach if two studies are conducted in the same way and the results at the time of the interim analysis are similar. However, the CP calculation and the statistical analysis plan for interim analysis are usually pre-specified before seeing the unblinded data. At the time of the interim analysis, it is usually unknown whether or not the results (treatment effects) observed from two identical studies will be similar. Even though two A&WC studies are designed the same, the operation and execution of the trial can still be different: two studies may be conducted in different countries, enrollment speed may be different,... As evidenced by Biogen's EMERGE and ENGAGE trials, two identical designed studies may have different results - therefore calculating the CP entirely independently for each study may be more appropriate when two identical A&WC trials are conducted.   

Futility Analysis and Conditional Power

Adaptive design has been used to drug development programs more efficient. According to FDA's guidance Adaptive Designs for Clinical Trials of Drugs and BiologicsGuidance for Industry, an adaptive design is defined as a clinical trial design that allows for prospectively planned modifications to one or more aspects of the design based on accumulating data from subjects in the trial. The modifications to the design based on the accumulating data from an ongoing study are through 'interim analysis'. An interim analysis is any examination of data obtained from subjects in a trial while that trial is ongoing and is not restricted to cases in which there are formal between-group comparisons. The observed data used in the interim analysis can include one or more types, such as baseline data, safety outcome data, pharmacokinetic, pharmacodynamic, biomarker data, or efficacy outcome data.

when an adaptive design is proposed, which aspect(s) of the trial to be adapted will need to be pre-specified and agreed upon by the regulatory agencies such as FDA. In the list of adaptations, the most common type of adaptive design is 'group sequential design'. 

• Group sequential design
• Adaptations to the sample size
• Adaptations to the patient population (e.e., adaptive enrichment)
• Adaptations to treatment arm selection
• Adaptations to patient allocation 
• Adaptations to endpoint selection
• Adaptations to multiple design features

Group sequential design is probably the most commonly used adaptive design (even before the adaptive design concept came out). Group sequential design was once categorized as 'well-understood' adaptive design. Ironically, many studies with group sequential design may not be called 'adaptive design' and the term 'group sequential design' may not be used in the study protocol at all. 

According to FDA's Adaptive Designs for Clinical Trials of Drugs and Biologics Guidance for Industry

"Group sequential designs may include rules for stopping the trial when there is sufficient evidence of efficacy to support regulatory decision-making or when there is evidence that the trial is unlikely to demonstrate efficacy, which is often called stopping for futility."

"There are a number of additional considerations for ensuring the appropriate design, conduct, and analysis of a group sequential trial. First, for group sequential methods to be valid, it is important to adhere to the prospective analytic plan and terminate the trial for efficacy only if the stopping criteria are met. Second, guidelines for stopping the trial early for futility should be implemented appropriately. Trial designs often employ nonbinding futility rules, in that the futility stopping criteria are guidelines that may or may not be followed, depending on the totality of the available interim results. The addition of such nonbinding futility guidelines to a fixed sample trial, or to a trial with appropriate group sequential stopping rules for efficacy, does not increase the Type I error probability and is often appropriate. Alternatively, a group sequential design may include binding futility rules, in that the trial should always stop if the futility criteria are met. Binding futility rules can provide some advantages in efficacy analyses (e.g., a relaxed threshold for a determination of efficacy), but the Type I error probability is controlled only if the stopping rules are followed. Therefore, if a trial continues despite meeting prespecified binding futility rules, the Agency will likely consider that trial to have failed to provide evidence of efficacy, regardless of the outcome at the final analysis. Note also that some DMCs might prefer the flexibility of nonbinding futility guidelines."

With group sequential design, interim analyses will be performed during the study to evaluate early evidence of efficacy or early evidence of futility. To stop the study for efficacy, the most common approach is so-called 'repeat significance testing'. to stop the study for futility, the most common approach is through calculating the conditional power.

Group sequential design:

• The interim analysis for efficacy: To see if the new treatment is overwhelmingly better than control - then stop the trial for efficacy
• Repeat significance testing
• Pocock
• O'Brien-Fleming
• Alpha-spending by Lan and DeMets 

• The interim analysis for futility (futility analysis): To see if the new treatment is unlikely to be superior to the control – then stop the trial for futility - this is called ‘futility analysis’.
• Repeat significance testing
• Stochastic curtailment approach with three families of stochastic curtailment tests
• Conditional power tests (frequentist approach)
• Predictive power tests (mixed Bayesian-frequentist approach).
• Predictive probability tests (Bayesian approach)

The most common futility analysis requires the calculation of the conditional power (CP) that is the probability that the study will demonstrate statistical significance at the end of the study (i.e. final analysis to claim superiority), conditioning on the data observed in the study thus far, and an assumption about the trend of the data to be observed in the remainder of the study. 

According to the paper by Lachin "A review of methods for futility stopping based on conditional power":

“Conditional power (CP) is the probability that the final study result will be statistically significant, given the data observed thus far and a specific assumption about the pattern of the data to be observed in the remainder of the study, such as assuming the original design effect, or the effect estimated from the current data, or under the null hypothesis.”

In conditional power calculation, assumptions about the trend of the data in the remainder of the study can be as the following and the assumption of the remainder data following the observed data is probably more reasonable. The assumption of the remainder data following the alternative hypothesis can overestimate the overall treatment effect (especially if the alternative hypothesis was based on the aggressive, over-optimistic assumptions) resulting in inflated conditional power. On the other hand, the assumption of the remainder data following the null hypothesis can underestimate the overall treatment effect resulting in deflated conditional power.  

• Observed data - the effect estimated from the current data so far
• The alternative hypothesis - assuming the original design effect
• The null hypothesis - assuming no effect in the remainder of the study 

To summarize, the futility analysis is through interim analysis to determine if the trial data indicates the inability of a clinical trial to achieve its objectives. Futility analysis usually requires the calculation of the conditional power (CP) that is defined as the probability that the final study result will be statistically significant, given the data observed thus far at the time of the interim data cut and a specific assumption about the pattern of the data to be observed in the remainder of the study, such as assuming the original design effect (alternative hypothesis) or the effect estimated from the current data. It is pretty common that the threshold for futility is defined as CP less than 20% - suggesting that the probability of the final result to be statistically significant is less than 20% given the data observed at the time of interim analysis. If the CP is less than 20% at the time of the interim analysis, the Data Monitoring Committee may recommend the sponsor stop the trial (stop the trial for futility).  

in a book chapter by Tin, Ming T "Conditional Power in Clinical Trial Monitoring", the pros and cons of conditional power were discussed. 

To put things in perspective, the conditional power approach attempts to assess whether evidence for efficacy or the lack of it based on the interim data is consistent with that at the planned end of the trial by projecting forward or using conditional likelihood given the eventuality. Thus it substantially alleviates the major inconsistency in all other group sequential tests where different sequential procedures applied to the same data yield different answers. ...

The advantage of the conditional power approach for trial monitoring is its flexibility. It can be used for unplanned analysis and even analysis whose timing depends on previous data. For example, it allows inferences from overrunning or underrunning (namely, more data come in after the sequential boundary is crossed, or the trial is stopped before the stopping boundary is reached. Conditional power can be used to aid the decision for early termination of a clinical trial to complement the use of other methods or when other methods are not applicable. 

The caveat is that the conditional power can be calculated with different assumptions about the remaining data. Depending on the assumptions about the remaining data following the observed data, the alternative hypothesis (original design effect), or others, the conditional power can sometimes be quite different resulting in different conclusions about the futility assessment. 

Some examples: 

in the SAP for "Randomized, Open-Label Study of Abiraterone Acetate (JNJ-212082) plus Prednisone with or without Exemestane in Postmenopausal Women with ER+ Metastatic Breast Cancer Progressing after Letrozole or Anastrozole Therapy", conditional power was described to be calculated with both the assumption of the remaining data following the original hazard ratio (alternative hypothesis) and the assumption of the remaining data following the observed hazard ratio at the interim.

3.1.2 Conditional Power

Conditional power is the probability that the study will demonstrate statistical significance at the end of the study (i.e. final analysis to claim superiority on PFS), conditioning on the data observed in the study thus far, and an assumption about the trend of the data to be observed in the remainder of the study. Two assumptions about the trend of the data were presented below: The futility boundary corresponds to a conditional power of approximately 39% if the original hazard ratio assumption is true, while only 4% conditional power will be achieved if the observed hazard ratio at interim is true for the remainder of the study. The efficacy boundary corresponds to a conditional power of approximately 90% if the original hazard ratio assumption is true, and 92% conditional power will be achieved if the observed hazard ratio at interim is true for the remainder of the study. The conditional power of stopping boundaries was computed using method of Lan (2009).

In Gilead's trial "A Multicenter, Adaptive, Randomized Blinded Controlled Trial of the Safety and Efficacy of Investigational Therapeutics for the Treatment of COVID-19 in Hospitalized Adults", the repeat significant test procedure (the alpha spending function) was used to evaluate the potential stop for overwhelming efficacy and the stochastic curtailment approach (conditional power) was used to evaluate the potential stop for futility. 

In a trial by Incyte "GRAVITAS-301: A Randomized, Double-Blind, Placebo-Controlled Phase 3 Study of Itacitinib or Placebo in Combination With Corticosteroids for the Treatment of First-Line Acute Graft-Versus-Host Disease", interim data monitoring for the potential stop for efficacy or futility is assessed and conditional power of 20% is used as the threshold for declaring the futility: 

Further reading: 

• Lachin 2005 A review of methods for futility stopping based on conditional power
• Paul Gallo (2019) Futility Analyses in Confirmatory Clinical Trials –Methods and Procedures
• Dmitrienko and Yang Yuan (2017) Interim Data Monitoring from the book "Analysis of Clinical Trials Using SAS - a practical guide"