One-way versus two-way tipping point analysis for robustness assessment of the missing data

Tipping point analysis (TPA) is a key sensitivity analysis mandated by regulatory agencies like the FDA to assess the robustness of clinical trial results to untestable assumptions about missing data. Specifically, it explores how much the assumption about the missing not at random (MNAR) mechanism would have to change to overturn the study's primary conclusion (e.g., a statistically significant treatment effect becoming non-significant). See a previous blog post "Tipping point analysis - multiple imputation for stress test under missing not at random (MNAR)"

One-Way Tipping Point Analysis for Robustness Assessment

A one-way tipping point analysis is a sensitivity method used to evaluate the robustness of a study’s primary findings by systematically altering the missing data assumption for only one treatment group at a time—most commonly the active treatment arm. While the missing outcomes in the control group are typically handled under a standard Missing at Random (MAR) or Jump to Reference assumption, the missing outcomes in the active arm are subjected to a varying "shift parameter" (δ). This parameter progressively penalizes the imputed values (e.g., making them increasingly worse) until the statistically significant treatment effect disappears, or "tips." By identifying this specific value, researchers can present a clear, one-dimensional threshold to clinical experts and regulators, who then judge whether such a drastic deviation from the observed data is clinically plausible or an unlikely extreme.

Two-Way Tipping Point Analysis for Robustness Assessment

A two-way TPA is an advanced method to assess robustness by independently varying the missing data assumptions for both treatment groups (e.g., the active treatment arm and the control/reference arm).

Missing Data Assumptions (MAR vs. MNAR)

The two-way TPA is used to assess the robustness of the primary analysis, which is typically conducted under the assumption of Missing at Random (MAR).

• Missing at Random (MAR): Assumes that the probability of data being missing depends only on the observed data (e.g., a patient with a worse baseline condition is more likely to drop out, and we have observed the baseline data).

• Missing Not at Random (MNAR): Assumes that the probability of data being missing depends on the unobserved missing outcome data itself (e.g., a patient drops out because their unobserved outcome has worsened more than what is predicted by their observed data).

Robustness Assessment

The two-way TPA evaluates robustness to plausible MNAR scenarios. This is done by imputing the missing outcomes (often starting with an MAR method like Multiple Imputation) and then applying a systematic, independent "shift parameter" (or δ) to the imputed values in each arm.

• Process: The shift parameters (δActive and δControl) are varied systematically across a two-dimensional grid, typically in a direction that reduces the observed treatment effect.

• Tipping Point: The δActive and δControl values at which the primary conclusion (e.g., statistical significance) is "tipped" or overturned define the tipping point.

• Robustness: The larger and/or more clinically implausible the combination of shift parameters required to overturn the conclusion, the more robust the original result is considered to be under different MNAR assumptions.

Two-Way Tipping Point Result Tables

The results of a two-way TPA are typically presented as a grid or heat map table where:

• One axis represents the shift parameter applied to the missing outcomes in the Active Treatment arm (δActive).

• The other axis represents the shift parameter applied to the missing outcomes in the Control/Reference arm (δControl).

• The cells of the table contain the resulting p-value or estimated treatment difference for that specific combination of assumptions.

The goal is to find the boundary of the grid where the result crosses the significance threshold (e.g., p >= 0.05 or the lower bound of the confidence interval crosses the null value).

---

Comparison: One-Way vs. Two-Way Tipping Point Analysis

The choice between one-way and two-way TPA is a trade-off between simplicity and comprehensiveness.

Feature	One-Way Tipping Point Analysis	Two-Way Tipping Point Analysis
Missingness Assumption	The shift parameter (δ) is only applied to one arm, usually the active treatment group, while the missing data in the control arm are imputed based on the MAR assumption (e.g., Jump to Reference).	Independent shift parameters (δActive and δControl) are applied to both arms simultaneously.
Sensitivity Explored	Explores MNAR scenarios where dropouts in one arm have systematically worse/better outcomes than assumed by MAR, relative to the other arm's MAR assumption.	Explores a two-dimensional space of MNAR scenarios, allowing dropouts in both arms to vary independently.
Complexity	Simpler to calculate and interpret (one dimension).	More computationally intensive and complex to interpret (two-dimensional grid).
Plausibility	Often viewed as less comprehensive, as it does not model the possibility of simultaneous, independent MNAR mechanisms in both arms.	Considered more comprehensive as it allows for a wider range of clinically plausible and implausible MNAR scenarios.
Result Presentation	A line plot or simple table with a single 'tipping point' value.	A grid/matrix table or heat map showing the boundary of non-significance.

In essence, the two-way TPA is generally preferred by regulatory agencies for its superior ability to assess robustness because it explores a more realistic and exhaustive range of asymmetric MNAR mechanisms.

Imputation of partial dates for adverse events, concomitant medications, and disease diagnosis

Many date variables are collected in the clinical trial database. The date variables include date of birth, date of disease diagnosis, date of medical history onset, start and stop date of adverse events, start / stop date of concomitant medications, ......

It is not uncommon that the partial dates may be collected where the partial dates mean that at least one of the components (day, month, or year) is missing. 

Partial date for date of birth is not because the subjects don't remember their birth date, is because the data security law prevents the sponsors from collecting the date of birth information in certain countries (especially in Germany). 

In statistical analyses, the partial dates need to be handled or imputed for the purpose of allocating the event (adverse events, concomitant medication) into the appropriate categories (treatment-emergent adverse events, prior medications, concomitant medications added during the study,...) or calculating the duration of the events (duration from the disease diagnosis to the study start).

For clarity, the algorithm and rules for imputing the partial dates need to be specified in the statistical analysis plan (SAP). There is no regulatory guidance about which algorithm and rules will be appropriate when imputing the partial dates. Different companies may have different rules when imputing partial dates. In general, the rules will be adequate as long as it is on the conservative side, for example, if an adverse event has a partial or missing start date and can't be determined if it occurs before the first dose of the study drug, the adverse event will be classified as 'treatment-emergent adverse event'. 

Partial date imputation is always a single imputation - the missing day or missing month will be replaced with a fixed day or month based on the imputation algorithm. The candidates for replacing the missing day could be: the first day of the month, the last day of the month, the day of the first dose of the study drug. The candidates for replacing the missing day and month could be Jun 30 of the year or July 1 of the year. 

Usually, if all day, month, and year are missing, the missing date will not be imputed. The adverse events with missing onset date will be classified as 'treatment-emergent AEs' and the concomitant medication will be classified as 'on treatment medications' (i.e., to be included in summaries of concomitant medications during the study). 

Below are a list of algorithm and rules for imputing the partial dates for adverse events and concomitant medications:

In an SAP for a Pfizer phase I study, if the day of the month is missing, the 1st day of the month is used. 

In a Novartis study SAP, the rather complicated algorithm was proposed for imputing the partial dates for adverse events and concomitant medications:

In a study by Johnson & Johnson, the appended SAP specified the rules for imputing the partial dates for adverse events, concomitant medications, and for disease diagnosis as the following: 

In a study by ChemoCentryx in NEJM, the appended SAP described the imputation rules for partial dates for adverse events and concomitant medications as the following: 

 

In the paper "Partial Dates; decisions and implications of handling partially missing dates" by Bowman, the following rules were stated for imputing the partial dates for adverse events and concomitant medications. 

Missing Adverse Event Start and Stop Dates date:

There are two options available. The partial start date may be set to the first of the month or to equal the study medication start date. As previously discussed, the first option would indicate the adverse event began prior to the study medication. However, the second option, setting the start date of AE1 to the study medication start date will suggest the adverse event had a short duration, as the adverse event end date is also defined as June 2006, but began during the treatment period of the study drug. Although the second option is not ideal, as AE1 may have had a longer duration, it is more conservative to associate the adverse event start with a date during study medication. Another solution to consider is not to impute a date at all but merely to assign a study phase to the start of the adverse event. In this example, a phase of "treatment" could be allocated to the start of teh adverse event, which would ensure it was classed most conservatively, without defining an actual date to the start of the adverse event. 

Concomitant Medications:

Subject has a partial concomitant medication start date of “--Apr2006” (see figure 1). As discussed above, missing start dates may be set to the first of the month, which is shown under option 1. However, this then pushes the concomitant medication to starting before the first dose of the Study Medication (15Apr2006) and would suggest that the Study Medication had no involvement with the concomitant medication being taken. Is this really the most conservative approach? If not is there an alternative? The missing concomitant medication start date could be set to equal the first dose of Study Medication, options 2. This option allows the concomitant medication to be classed as an on-treatment medication, and is therefore the most conservative.  

Single Imputation Methods for Missing Data: LOCF, BOCF, LRCF (Last Rank Carried Forward), and NOCB (Next Observation Carried Backward)

The missing data is always an issue when analyzing the data from clinical trials. The missing data handling has been moved toward the model-based approaches (such as multiple imputation and mixed model repeated measures (MMRM)). The single imputation methods, while being heavily criticized and cast out, remain as practical approaches for handling the missing data, especially for sensitivity analyses.

Single imputation methods replace a missing data point by a single value and analyses are conducted as if all the data were observed. The single value used to fill in the missing observation is usually coming from the observed values from the same subject - Last Observation Carried Forward (LOCF), Baseline Observation Carried Forward, and Next Observation Carried Backward (NOCB, the focus of this post). The single value used to fill in the missing observation can also be derived from other sources: Last Rank Carried Forward (LRCF), Best or Worst Case Imputation (assigning the worst possible value of the outcome to dropouts for a negative reason (treatment failure) and the best possible value to positive dropouts (cures)), Mean value imputation, trimmed mean,…Single imputation approaches also include regression imputation, which imputes the predictions from a regression of the missing variables on the observed variables; and hot deck imputation, which matches the case with missing values to a case with values observed that is similar with respect to observed variables and then imputes the observed values of the respondent.

In this post, we discussed the single imputation method of LOCF, BOCF, LRCF, and NOCB (the focus of this post). 

Last Observation Carried Forward (LOCF): A single imputation technique that imputes the last measured outcome value for participants who either drop out of a clinical trial or for whom the final outcome measurement is missing. LOCF is usually used in the longitudinal study design where the outcome is measured repeatedly at pre-specified intervals. LOCF usually requires there is at least one post-baseline measure. The LOCF is the widely used single imputation method.

Baseline Observation Carried Forward (BOCF): A single imputation technique that imputes the baseline outcome value for participants who either drop out of a clinical trial or for whom the final outcome measurement is missing. BOCF is usually used in a study design with perhaps only one post-baseline measure (i.e., the outcome is only measured at the baseline and at the end of the study).

Last Rank Carried Forward (LRCF): The LRCF method carries forward the rank of the last observed value at the corresponding visit to the last visit and is the non-parametric version of LOCF. However, unlike the LOCF that is based on the observation from the same subject, for the LRCF method, the ranks come from all subjects with non-missing observations at a specific visit.  From the early visits to the later visits, the number of missing values will be different, the constant ranking, carried forward, and re-ranking will be needed. Here are some good references for LRCF:

• O’Brien et al (2005) Semi-parametric and non-parametric methods for clinical trials with incomplete data
• Peter Christenson An Alternative to Last-Observation-Carried-Forward: Last-Rank-Carried-Forward
• Fan et al (2015) Methods for Missing Data Handling in Randomized Clinical Trials With Nonnormal Endpoints With Application to a Phase III Clinical Trial

LRCF is thought to have the following features:

In a paper by Jing et al, the LRCF was used for missing data imputation: 

"...The last rank carried forward or last observation carried forward was assigned to patients who withdrew prematurely from the study or study drug for other reasons or who did not perform the 6-minute walk test for any reason not mentioned above (eg, missed visit), provided that the patient performed at least 1 postbaseline 6-minute walk test.

Next Observation Carried Backward (NOCB): NOCB is a similar approach to LOCF but works in the opposite direction by taking the first observation after the missing value and carrying it backward. NOCB may also be called Next Value Carried Backward (NVCB) or Last Observation Carried Backward (LOCB).

NOCB may be useful in handling the missing data arising from the external control group, from Real-World Data (RWD), Electronic health records (EHRs) where the outcome data collection is usually not structured and not according to the pre-specified visit schedule. 

I can foresee that the NOCB may also be an approach in handing the missing data due to the COVID-19 pandemic. Due to the COVID-19 pandemic, subjects may not be able to come to the clinic for the outcome measure at the end of the study. The outcome measure may be performed at a later time beyond the visit window allowance. Instead of having a missing observation for the end of the study visit, the NOCB approach can be applied to carry the next available outcome measure backward. 

The NOCB approach, while not popular, can be found in some publications and regulatory approval documents. Here are some examples: 

In an article by Wyles et al (2015, NEJM) Daclatasvir plus Sofosbuvir for HCV in Patients Coinfected with HIV-1, "Missing response data at post-treatment week 12 were inferred from the next available HCV RNA measurement with the use of a next-value-carried-backward approach."

In BLA 761052 of Brineura (cerliponase alfa) Injection Indication(s) for Late-Infantile Neuronal Ceroid Lipofuscinosis Type 2 (CLN2)- Batten Disease, the NOCB was used to handle the missing data for comparison to the data from a natural history study. 

Because intervals between clinical visits vary a lot in Study 901, the agency recommended performing analyses using both the last available Motor score and next observation carried backward (NOCB) for the intermediate data points although the former one is determined as the primary. 

In FDA Briefing Document for Endocrinologic and Metabolic Drugs Advisory Committee Meeting for NDA 210645, Waylivra (volanesorsen) injection for the treatment of familial chylomicronemia syndrome, NOCF was used as one of the sensitivity analyses:

Similar planned (prespecified) analyses using different variables, such as slightly different endpoint definitions (e.g. worst maximum pain intensity versus average maximum pain intensity), or imputation methods for missing data (next observation carried backward versus imputation of zero for missing values) did not demonstrate treatment differences.

 Missing values were pre-specified to be imputed using Next Observation Carried Back (NOCB); i.e., if a patient did not complete the questionnaire for several weeks, the next value entered was assumed to have occurred during all intervening (missing) weeks.

 Missing data for any post-baseline visit will be imputed by using Next Observation Carried Back (NOCB) if there is a subsequent score available. Missing data after the last available score of each patient will not be imputed.

in NDA 212157 of Celecoxib Oral Solution for Treatment of acute migraine, the NOCB was used for sensitivity analysis

Headache Pain Freedom at 2 hours - Sensitivity Analysis

To analyze the missing data for the primary endpoint, Dr. Ling performed an analysis analyzing patients who took rescue medications as nonresponders and then also imputing missing data at the 2-hour time point using the next available time point of information (Next Observation Carried Backward (NOCB)) or a worst-case type of imputation (latter not shown in table).

Single imputation methods are generally not recommended for the primary analysis because of the following disadvantages (issues): 

• Single imputation usually does no provides an unbiased estimate
• Inferences (tests and confidence intervals) based on the filled-in data can be distorted by bias if the assumptions underlying the imputation method are invalid
• Statistical precision is overstated because the imputed values are assumed to be true.
• Single imputation methods risk biasing the standard error downwards by ignoring the uncertainty of imputed values. Therefore, the confidence intervals for the treatment effect calculated using single imputation methods may be too narrow and give an artificial impression of precision that does not really exist.  
• the single imputation method such as LOCF, NOCB, and BOCF do not reflect MAR (missing at random) data mechanisms.

Further Readings:

• EMA's Guideline on Missing Data in Confirmatory Clinical Trials
• US National Research Council: The Prevention and Treatment of Missing Data in Clinical Trials 
• Missing Data Approaches in eHealth Research: Simulation Study and a Tutorial for Nonmathematically Inclined Researchers 
• John Lachin (2016) Fallacies of last observation carried forward analyses
• Missing Value Imputation by Last Observation Carried Forward - R package
• Kenward and Molenberghs (2009) Last Observation Carried Forward: A Crystal Ball? 
• Engels and Diehr (2003) Imputation of missing longitudinal data: a comparison of methods
• 
  

Multiple Imputation: Imputation Model versus Analysis Model

Multiple imputation has become more and more popular in handling the missing data in clinical trials. Multiple imputation inference involves three distinct phases:

• The missing data are filled in m times to generate m complete data sets. This step is through the imputation model and can be implemented using SAS Proc MI
• The m complete data sets are analyzed by using standard procedures. This step is through the analysis model – depending on nature of the outcome variable, the analysis model can be ANCOVA (analysis of covariance), MMRM (mixed model repeated measures), Logistic regression, GEE (generalized estimating equation), GENMOD (generalized linear model),… The analysis model is also the primary model for analyzing the corresponding outcome variable.
• The results from the m complete data sets are combined for the inference. This step is using Robin’s rule and can be implemented with SAS Proc MIANALYZE

For both the imputation model and the analysis model will need to include a list of explanatory or independent variables, but for different purposes. The list of explanatory or independent variables in the imputation model is to impute the missing values; the list of explanatory or independent variables in the analysis model are covariates as part of the standard statistical models. Here are some comparisons for the variables used in the imputation model and analysis model:

• The covariates included in the analysis model must also be included in imputation model
• The imputation model can include additional auxiliary variables including those variables that are not used as covariates in the analysis model
• The number of variables used in imputation model is greater than or equal to the number of variables in analysis model
• The imputation model can include variables measured after the randomization (such as secondary outcomes, concomitant medication use, compliance data). However, for analysis model, “variables measured after randomisation and so potentially affected by the treatment should not be included as covariates in the primary analysis.”
• For longitudinal data or repeated measures, the outcome measures at early time points will be included in the imputation model.
• If the variables used in the analysis model are transformed, the transformed variable should also be used in the imputation model
• If the interaction term is used in the analysis model, it should also be included in the imputation model - this can make the imputation model pretty complicated though. 

In many publications, multiple imputation was stated as the method for handling the missing data, however, the details about the imputation model (i.e., which variables are included in the imputation model) were not usually described. 

While there is no clear guidance about the variables included in the imputation model, it is important to pre-specify the list of variables included in the imputation model especially if the auxiliary variables or variables not included in the analysis model. 

Below are some excerpts from the literature about the imputation model and analysis model.

UCLA Seminar “MULTIPLE IMPUTATION IN STATA”

Imputation Model, Analytic Model and Compatibility :

When developing your imputation model, it is important to assess if your imputation model is “congenial” or consistent with your analytic model. Consistency means that your imputation model includes (at the very least) the same variables that are in your analytic or estimation model. This includes any transformations to variables that will be needed to assess your hypothesis of interest. This can include log transformations, interaction terms, or recodes of a continuous variable into a categorical form, if that is how it will be used in later analysis. The reason for this relates back to the earlier comments about the purpose of multiple imputation. Since we are trying to reproduce the proper variance/covariance matrix for estimation, all relationships between our analytic variables should be represented and estimated simultaneously. Otherwise, you are imputing values assuming they have a correlation of zero with the variables you did not include in your imputation model. This would result in underestimating the association between parameters of interest in your analysis and a loss of power to detect properties of your data that may be of interest such as non-linearities and statistical interactions. 

Auxiliary variables are variables in your data set that are either correlated with a missing variable(s) (the recommendation is r > 0.4) or are believed to be associated with missingness. These are factors that are not of particular interest in your analytic model , but they are added to the imputation model to increase power and/or to help make the assumption of MAR more plausible. These variables have been found to improve the quality of imputed values generate from multiple imputation. Moreover, research has demonstrated their particular importance when imputing a dependent variable and/or when you have variables with a high proportion of missing information (Johnson and Young, 2011; Young and Johnson, 2010; Enders , 2010).

You may a priori know of several variables you believe would make good auxiliary variables based on your knowledge of the data and subject matter. Additionally, a good review of the literature can often help identify them as well. However, if your not sure what variables in the data would be potential candidates (this is often the case when conducting secondary data analysis), you can uses some simple methods to help identify potential candidates.

In a presentation of “multiple imputations” by Adrienne D. Woods

Which variables should you include as predictors in the imputation model?

• Any variables you plan to use in later analyses (including controls)
• General advice: use as many as possible (could get unwieldy!)
• Although, some (i.e., Kline, 2005; Hardt, Herke, & Leonhart, 2012) believe that this introduces more imprecision, especially if the auxiliary variable explains less than 10% of the variance in missingness on Y… thoughts?
• Know your analysis model beforehand and include at least all analysis variables in imputation model (including interaction terms)

FDA’s Statistical Review for Vantrela (hydrocodone bitartrate) extended-release tablets in Management of pain severe

Analysis model:

"The primary efficacy endpoint of trial 3103 was change from baseline to week 12 in the weekly average of worst pain intensity (WPI). The primary analysis was ANCOVA model with baseline WPI, randomized treatment, opioid status, and center as covariates. The intent-to-treat analysis population, defined as all randomized patients, was used for the primary efficacy analysis."

Imputation model:

"The applicant performed multiple imputation on the week 12 missing data for the primary analysis. The imputation model included randomized treatment, opioid status, baseline and postbaseline WPI values while subjects in the active-drug treatment group who discontinued study drug because of an adverse event, were treated as if they were in the placebo group and their missing data were imputed based on the observed placebo subjects' data."

FDA's Statistical Review for EUCRISA™ (crisaborole) topical ointment, 2% for Atopic Dermatitis mentioned the imputation model for missing dichotomized outcome variable. 

The protocol specified the primary imputation method to be the multiple imputation (MI) approach. For each treatment arm separately, missing data was imputed using the Markov Chain Monte Carlo (MCMC) method. The protocol specified the following two sensitivity analyses for the handling of missing data:

· Repeated-measures logistic regression model (GEE), with dichotomized ISGA success as the dependent variable and treatment, analysis center, and visit (i.e., Days 8, 15, 22, and 29) as independent factors. In this analysis, data from all post-baseline visits will be included with no imputation for missing data.

· Model-based multiple imputation method to impute missing data for the dichotomized ISGA data. The imputation model (i.e., logistic regression) will include treatment and analysis center.

Kaifeng Lu et al (2010) Multiple Imputation Approaches for the Analysis of Dichotomized Responses in Longitudinal Studies with Missing Data pointed out the issue if the analysis model is different from the imputation model. 

Despite its conceptual simplicity and flexibility, the above MI procedure is not valid for the analysis of dichotomized responses because Rubin’s variance estimator is biased when the analysis model is different from the imputation model (Meng, 1994; Robins and Wang, 2000). This is true even when the imputation and analysis models are compatible, e.g. when the treatment is the only effect in the logistic regression model.

Ian R. White  et al (2012) Including all individuals is not enough: lessons for intention-to-treat analysis

In some cases, an MI procedure can be improved by including in the imputation model ‘auxiliary variables’ that are not in the analysis model [36, Chapter 4]: auxiliary variables in a randomised trial might be secondary outcomes or compliance summaries. MI then produces estimates of the treatment effect that are genuinely different from a likelihoodbased analysis, by incorporating information on individuals with missing outcome but observed values of auxiliary variables. However, in our experience, the contribution to such an analysis of individuals missing the outcome of interest is moderate unless correlations between the outcome and one or more auxiliary variables are substantial [37].

Michael Spratt et al (2010) Strategies for Multiple Imputation in Longitudinal Studies

Where there are nontrivial amounts of missing data in covariates, both preliminary analyses and imputation models will become more complex. An MAR assumption may often become more plausible after the inclusion in the imputation model of additional variables that are not in our analysis model (because they are on the causal pathway, for example). Thus, multiple imputation models should typically be more complex than the analysis model. Including variables that are not related to the variable being imputed in the imputation models may slightly decrease efficiency but should not cause bias (29, 31). Model diagnostics should be used to highlight any implausibility in the imputed values. For example, the distributions of observed and imputed data should be compared and the plausibility of any differences examined. Imputation models should also preserve the structure of the analysis model (32). For example, where the substantive analysis exploits the hierarchical nature of longitudinal data (e.g., using a multilevel model), the imputation model should be similarly structured. Here, the longitudinal nature of the data allowed us to include variables (previous wheezing) that predicted the values of the variable with the most missing data (wheeze at 81 months) in imputation models.

Jochen Hard et al (2012) Auxiliary variables in multiple imputation in regression with missing X: a warning against including too many in small sample research

• An additional advantage of MI over CC (complete-case analysis) is the possibility of including information from auxiliary variables into the imputation model. Auxiliary variables are variables within the original data that are not included in the analysis, but are correlated to the variables of interest or help to keep the missing process random [MAR: 1]. Little [6] has calculated the amount of decrease in variance of a regression coefficient Y on X1 when a covariate X2 is added that has no missing data. White and Carlin [7] have extended this proof to more than one covariate. In practice however, it is likely that auxiliary variables themselves will have missing data.

EMA Guideline on Missing Data in Confirmatory Clinical Trials mentioned the multiple imputation as an approach to handle the missing data with MAR assumption, however, it did not mention anything about the imputation model.   

Panel on Handling Missing Data in Clinical Trials; National Research Council  (2010) The Prevention and Treatment of Missing Data in Clinical Trials

Multiple imputation methods address concerns about (b) “simple imputation is generally not true because the methods do not always yield conservative effect estimators, and standard errors and confidence interval widths can be underestimated when uncertainty about the imputation process is neglected.”  and enable the use of large amounts of auxiliary information.

An important advantage of multiple imputation in the clinical trial setting is that auxiliary variables that are not included in the final analysis model can be used in the imputation model. For example, consider a longitudinal study of HIV, for which the primary outcome Y is longitudinal CD4 count and that some CD4 counts are missing. Further, assume the presence of auxiliary information V in the form of longitudinal viral load. If V is not included in the model, the MAR condition requires the analysis to assume that, conditional on observed CD4 history, missing outcome data are unrelated to the CD4 count that would have been measured; this assumption may be unrealistic. However, if the investigator can confidently specify the relationship between CD4 count and viral load (e.g., based on knowledge of disease progression dynamics) and if viral load values are observed for all cases, then MAR implies that the predictive distribution of missing CD4 counts given the observed CD4 counts and viral load values is the same for cases with CD4 missing as for cases with CD4 observed, which may be a much more acceptable assumption.

Meyer et al (2020) Statistical Issues and Recommendations for Clinical Trials Conducted During the COVID-19 Pandemic

Multiple imputation (MI) methodology (Rubin, 1987) may be helpful in this respect as it allows inclusion of auxiliary variables (both pre- and post-randomization) in the imputation model while utilizing the previously planned analysis model. Multiple imputation with auxiliary variables may be used for various types of endpoints, including continuous, binary, count, and time-to-event and coupled with various inferential methods in the analysis step.

Thomas R Sullivan et al (2018) Should multiple imputation be the method of choice for handling missing data in randomized trials?

In the first stage of MI, multiple values (m > 1) for each missing observation are independently simulated from an imputation model. For missing data restricted to the outcome, the imputation model would typically regress observed values of Y on X and T. Additional auxiliary variables that are not in the analysis model can also be added to the imputation model to improve the prediction of missing values.

In applying MI, the repeated measurements of the outcome are usually treated as distinct variables in the imputation model. Where interest lies in the treatment effect at the final time point, the analysis model need not include the intermediate outcome measures; following imputation a comparison of final time point results is sufficient. In this case, the intermediate measures operate as auxiliary variables, assisting with the prediction of missing values at the final time point and making the MAR assumption more plausible. Other auxiliary variables, for instance measures of compliance or related outcomes, can also be added to the imputation model as required. If data are collected but more likely to be missing following treatment discontinuation, an indicator variable for discontinuation may also be valuable as an auxiliary variable. The ability to incorporate auxiliary variables, both for univariate and multivariate outcomes, is considered one of the key strengths of MI.

Thus in settings where MI is adopted, we recommend imputing by randomized group; compared to MI overall, this approach offers greater robustness at little cost. The approach is also consistent with general recommendations for over- rather than under-specifying imputation models. It should be noted that imputing by group only protects against bias in estimating the ATE if effect modifiers are included in the imputation model.

One of the strengths of MI is its ability to easily incorporate variables of different types (e.g. continuous, binary) in the imputation model, whether for univariate or multivariate data. An added benefit of including all outcomes in a single imputation model is that associations between related outcomes can aid imputation. Another appealing feature of MI is its ability to be implemented under an assumption that data are MNAR. This property makes MI well suited to undertaking sensitivity analyses around a primary assumption that data are MAR, and as a primary method of analysis in settings where data are believed to be MNAR. One such setting is RCTs where participants cannot followed up after discontinuing treatment. If all observed data are ‘on-treatment’, a MAR assumption entails estimating the effect of treatment had all participants remained on their assigned treatment.27 However, for a de facto type estimand (such as ITT), it may be more appropriate to assume that data are MNAR. In this situation, reference based sensitivity analyses have been proposed, which at present require the use of MI.2

Interaction terms are not suggested.

Although the bias of MI overall could be eliminated by including the interaction term in the imputation model (results not shown), this may not be an obvious strategy if subgroup analyses are not of interest.

Simon Grund et al (2018) Multiple Imputation of Missing Data for Multilevel Models: Simulations and Recommendations

A crucial point in the application of MI to multilevel data is that the imputation model not only includes all relevant variables, but also that it “matches” the model of interest (i.e., the substantive analysis model; see Meng, 1994; Schafer, 2003). In other words, the imputation model must capture the relevant aspects of the analysis model, making the imputation model at least as general as (or more general than) the analysis model. If the imputation model is more restrictive than the analysis
model, then imputations are generated under a simplified set of assumptions, and the results of subsequent analyses may be misleading.

Protocol for: Hatemi G, Mahr A, Ishigatsubo Y, et al. Trial of apremilast for oral ulcers in Behçet’s syndrome. N Engl J Med 2019;381:1918-28. DOI: 10.1056/NEJMoa1816594

 
REFERENCES:

•  von Hippel, 2009 “HOW TO IMPUTE INTERACTIONS, SQUARES, AND OTHER TRANSFORMED VARIABLES“
• von Hippel, 2013 “Should a Normal Imputation Model be Modified to Impute Skewed Variables?”
• White et al., 2010 “Avoiding bias due to perfect prediction in multiple imputation of incomplete categorical variables”
• Yang Yuan "Multiple Imputation for Missing Data: Concepts and NewDevelopment (Version 9.0)"
• Cro et al (2020) "Sensitivity analysis for clinical trials with missing continuous outcome data using controlled multiple imputation: A practical guide."
• SAS/Stat 15.2 User's Guide: Proc MI and Proc MIANALYZE

Handling of Missing Data: Comparison of MMRM (mixed model repeated measures) versus MI (multiple imputation)

Longitudinal study has become one of the most commonly adopted designs in clinical trials. Since the outcome measures are performed at various visits, it is usually the case that for some subjects in the study, the outcome measures will not be available at some visits (for example after subjects drop out from the study or lost-to-follow-up) - this is where the missing data issue arises. If the outcome measure is a continuous variable, the missing data issues can be handled implicitly through using the mixed-effects repeated measure (MMRM) models or explicitly through multiple imputations (MI).

Both MMRM and MI methods are based on the assumption of missing at random (MAR) and are model-based approaches suggested by EMA's Guideline on Missing Data in Confirmatory Clinical Trials and US National Research Council: The Prevention and Treatment of Missing Data in Clinical Trials. US FDA has not issued any guidance on handling the missing data in clinical trials, but generally follows the guidelines from the National Research Council. 

In terms of MMRM and MI, which one should be the primary method for handling the missing data? For a long time, it seems that in the US, the MMRM is the preferred method in handling the missing data and analyzing the longitudinal data with continuous outcome measures. The MI methods are generally used as sensitivity analyses to check the robustness of the primary analyses against the deviation from the MAR assumption. This can be observed by the article by Dr. Siddiqui in FDA "MMRM versus MI in Dealing with Missing Data - a Comparison Based on 25 NDA data sets" and many NDA / BLA reviews (listed below). 

FDA Statistical Review for NDA 210655 in the indication of Schizophrenia:

"The primary analysis was conducted on the change from baseline in the total PANSS score at Day 57 (primary time point) based on the ITT population. A mixed-effects model for repeated measures (MMRM) was used with treatment, visit, interaction of treatment and visit as fixed effects and the baseline total PANSS score as a covariate. Data from Days 15, 29, 43, and 57 were used. The unstructured covariance matrix was be used to model the within-subject variance-covariance errors."

"In addition to the model-based missing data approach of the MMRM model, the primary efficacy analysis was also analyzed using a pattern mixture model (PMM) and a multiple imputation approach as sensitivity analyses. "

FDA BLA 761037 Kevzara (sarilumab) in Treatment of rheumatoid arthritis

"The continuous HAQ-DI change from baseline at Week 16 was analyzed with a mixed model for repeated measures (MMRM). The repeated-measures analysis was based on the restricted maximum likelihood method assuming an unstructured covariance structure to model the within-subject errors. The model, including treatment, region, prior biologic use, visit (all visits from week 2 to week 16), and treatment-by-visit interaction as fixed effects and baseline as a covariate, was used to test the difference between each active treatment group versus placebo in the change from baseline in HAQ-DI at Week 16. The data collected after treatment discontinuation or rescue were set to missing. Therefore, the MMRM analysis assumed a missing-at-random (MAR) mechanism for missing data due to dropout and post-rescue data."

FDA NDA 203313/203314S-2 /S-3Tresiba;Ryzodeg 70/30Glycemic Control in Patients with Diabetes

The applicant used a mixed effect model for repeated measure (MMRM) to assess the efficacy of IDegAsp compared with IDet. The MMRM model included treatment, sex, region, age group and visits as factors and baseline as covariate, and interactions between visits and all factors and covariate. An unstructured covariance matrix was utilized for model fitting.

Multiple imputation was performed as sensitivity analysis

SNDA for Merck's Dulera in the treatment of asthma (2019)

"Missing Data Handling and Sensitivity Analyses The primary analysis incorporated a control-based multiple imputation of missing data. Missing data for subjects who discontinued treatment early were estimated using the MF group; that is, the change from baseline AM post-dose ppFEV1 in patients who discontinued treatment and missed study visits was assumed to be similar to the change from baseline in patients who continued study visits through Week 12 in the MF treatment group. The dataset was first multiply imputed to have monotone missing patterns, then for each visit, a regression method was used to impute for missing data on both study drug arm and the control arm based on trend from the control arm. After applying the control-based multiple imputation, the cLDA analysis was performed. MF/F 100/10 mcg BID was considered superior to MF 100 mcg BID with a p-value less than 0.05. "

EMA seems to have a different opinion about missing data handling using MMRM or MI. On several occasions, we have heard that EMA prefers the MI approach in handling the missing data especially the reference-baseline multiple imputation. They are moving towards developing the reference-based multiple imputation into the new standard missing data approach. 

Here is a table summarizing some comparisons between the MMRM and MI in handling the missing data. 

	MMRM	MI
Missing data mechanism	MAR (missing at random)
Missing data imputation	Not imputed for individual missing values But missing data is implicitly imputed	Individual missing values are explicitly imputed
# of steps for calculations	One step	At least three steps: Imputation model to create multiple data sets with missing values filled in Analysis model to analyze each imputed data set Using Robin’s rule to combine results for inference
Analysis Model	Mixed model with Maximum likelihood-based method	Analysis of Covariance or Mixed model using maximum likelihood-based method
Data points used in analyses	Utilized all observed data points from all visits	Usually, with ANCOVA, only the data points for the corresponding visits (with imputed values) are used.
SAS procedure(s)	Proc Mixed	Imputation model: Proc MI Analysis model: Proc Mixed, Proc GLM, Proc Genmod,… Robin’s rule: Proc MIANALYZE
Results	The two approaches will be approximately equivalent, provided the variables used in the imputation model are the same as those included in the analysis model, and conditionals are accommodated by a single joint model. In such settings, MI essentially provides an approximation to the observed likelihood analysis. If an infinite number of imputations could be performed, then the two approaches would be equivalent. In practice, the level of equivalence will depend on the number of imputations due to the Monte Carlo (simulation) sampling variability of the imputation process (described in more detail below), thus will be stronger for a larger number of imputations.
Auxiliary variables	Can not be used	Auxiliary variables can be used in the imputation model to improve the accuracy of the missing data prediction
Information observed post-randomization	Can not be included in the MMRM model	Can be included in the imputation model to improve the accuracy of the missing data prediction and can’t be included in the analysis model (MI approach allows the differences in the covariates used in the imputation model and in analysis model
Justification of MAR assumption	Not available through MMRM model	Justification of MAR assumption can be performed through the tipping point approach or delta-based imputation
Handling the MNAR (missing not at random)	Not directly available through MMRM	Can be performed through PMM (pattern mixed model), reference-based or control-based multiple imputation
For studies with only one post-baseline measure	Not appropriate	Appropriate to use MI to impute the missing data and then run analysis of covariance model as the analysis model
For outcome measures that are not continuous variables	Like MMRM, there are statistical approaches that handle missing data without employing explicit imputation. As mentioned in the EMA guideline “For categorical responses and count data, the so-called marginal (e.g. generalized estimating equations (GEE)) and random-effects (e.g. generalized linear mixed models (GLMM)) approaches are in use. Likelihood-based methods (MMRM and GLMM) and some extended GEE (i.e. weighted GEE) models are applicable under MCAR and MAR assumptions.”	MI approach can be easily applied to the outcome measures that are categorical responses or count data with missing data. The analysis model may need to be PROC Logistics; PROC GLIMMIX, PROC NLMIXED, or PROC GENMOD
Preferred by regulatory agencies	US FDA but with multiple imputation approaches as sensitivity analyses (for example, reference-based MI, PMM, tipping point)	EMA

REFERENCES:

• CRO et al (2020) Sensitivity analysis for clinical trials with missing continuous outcome data using controlled multiple imputation: A practical guide
• Michael Kenward (2013) The handling of missing data in clinical trials
• Lisa M. LaVange Thomas Permutt (statistics in Medicine, 2015) A regulatory perspective on missing data in the aftermath of the NRC report
• Little et al (NEJM 2012) The Prevention and Treatment of Missing Data in Clinical Trials
• Ware et al (NEJM 2012) Missing Data
• Permutt (Statistics in Medicine, 2015) Sensitivity analysis for missing data in regulatory submissions
• LaVange and Permutt (Statistics in Medicine, 2015) A regulatory perspective on missing data in the aftermath of the NRC report
• Liu and Pang (Statistics in Biopharmaceutical Research, 2017) Control-Based Imputation and Delta-Adjustment Stress Test for Missing Data Analysis in Longitudinal Clinical Trials
• Tang (Statistics in Biopharmaceutical Research, 2017) An Efficient Multiple Imputation Algorithm for Control-Based and Delta-Adjusted Pattern Mixture Models using SAS
• Berglund and Heeringa (2014) Multiple Imputation of Missing Data Using SAS Chapter 7 gives examples of missing data imputation for dichotomous outcome variable and count data.