In a previous post, different imputation methods were summarized by
the different missingness assumptions. One method, the tipping point approach, has
gained popularity recently as an approach for performing the sensitivity
analysis under the missing not at random (MNAR) assumption. In other words, the
tipping point approach is like a progressive stress-testing to assess how
severe departures from missing at random (MAR) must be in order to overturn
conclusions from the primary analysis. If implausible departures from MAR in
order to change the results from statistically significant (p<=0.05) to
statistically insignificant (p>0.05), the results will be said to be robust
to the departure from MAR assumption. We will then be more confident in the
results obtained based on statistical methods with the MAR assumptions (such as
multiple imputation, mixed model repeated measurements – MMRM). Tipping point
approach is not intended for the primary analysis method and is only used for
the sensitivity analysis.
The tipping point approach can be seen as a special application of the
multiple imputation. It can also be considered as a special case of the controlled
imputation method (i.e., applying the shift parameter only to the active
treatment group, not to the placebo group).
Implementing the tipping point approach includes the following
steps with the first three steps being the standard multiple imputation (MI)
steps:
- The missing data are filled in m times to generate m
complete data sets.
- The m complete data sets are analyzed by using
standard procedures.
- The results from the m complete data sets are
combined for the inference.
- Repeat the steps #1 to generate multiple imputed data sets, with a specified shift parameter that adjusts the imputed values for observations in the treatment group, not the placebo group).
- Repeat step 2 for the imputed data sets with shift parameter applied.
- Repeat step 3 to obtain the p-value to see if the p-value is still <=0.05.
- Repeat steps 4-6 with a more stringent shift parameter applied until the p-value >0.05.
The tipping point approach can be easily implemented using SAS
procedures MI and MIANALYZE. A SAS example “Sensitivity
Analysis with Tipping-Point Approach” provides step-by-step instructions on how to implement the tipping point
approach.
The following papers are also helpful in understanding and
implementing the tipping point approach.
- Ratitch and O’kelly (2011) Implementation of Pattern-Mixture Models Using Standard SAS/STAT Procedures
- Ratitch and O’kelly Placebo MI including Tipping Point Analysis with Delta-Adjusting ImputationPlacebo MI including Tipping Point Analysis with Delta-Adjusting Imputation
- Yuan (2014) Sensitivity Analysis in Multiple Imputation for Missing Data
- PhD Dissertation by Smuk (2015)Missing Data Methodology:- Sensitivity analysis after multiple imputation (with tipping point analysis, we select a shift parameter that moves the data further and further away from MAR in a particular direction, until key inferences from the substantive model change. Experts can then discuss whether the degree of departure from MAR needed to change the substantive results is plausible)
The tipping point approach has been discussed in several drug trials:
In Dry
Powder Mannitol (DPM) Pharmaxis Pulmonary and Allergy Drugs Advisory Committee
(slides are here)
January 30, 2013, tipping point approach was used for a stress test to see how the primary analysis method is robust to the departure of the MAR
assumption.
They explored the tipping point in the ITT population at which DPM would no longer show a significant effect. To do this, the penalty at each missing time point is increased up to the point that statistical significance is lost. They showed what happens when they stress tested the data even more. They increased the size of penalty for each missing visit in the pattern mixture model up until the point where significance is lost. The penalty would need to be more than 450 mLs at each missing time point before the effect estimate is reduced to 55 mLs and is no longer significant. This means that each patient leaving before week six could be penalized by 1,350 mLs. A tipping point requiring such a large volume does not seem plausible. They challenged the robustness even further, again using the same pattern mixture model, but this time identifying a tipping point when only penalizing the DPM arm but not control. Even applying this extreme method, the tipping point needed to reach 150 mLs before significance was lost. Now, this means that even patients withdrawing before week six in the control arm carry no penalty at all, but, similarly, DPM withdrawals being penalized by 450 mLs.
In FDA’s
Statistical Review for NDA 204168 Drug Name: FETZIMA (Levomilnacipran) extended-release
capsules 20, 40, 80, and 120 mg Indication: Major Depressive Disorder
Applicant: Forest Laboratories, Inc.
A “tipping point” analysis was conducted by increasing the shift parameter beyond the maximum value of 8 considered by the sponsor. The mean difference in MADRS change scores between drug and placebo would loose statistical significance at alpha = 0.05 at a shift parameter of 16 (see Table 16). The value of 16 appears to be rather large and unlikely to be a realistic mean difference at yt+1 between patients that drop-out after the tth visit and patients that continue. The PMM model results are consistent with the primary MMRM model results at the more realistic values of the shift parameter (i.e., 2, 4, …, 14).
Slide presentation “Missing
Data Sensitivity Analysis of a Continuous Endpoint – An Example from a Recent
Submission” by Arno
Fritsch indicated that the tipping point approach was explored as sensitivity
analysis for 6MWT endpoint in Riociguat in Pulmonary Arterial Hypertension.
- Need to increase penalty for riociguat to -71m per visit after drop-out until statistical significance is lost
- Would imply a very steep decline after drop-out, even giving negative 6MWD values for many patients (Mean 6MWD at baseline 364m, some patients in the 200’s)
- So positive treatment effect seems unquestionable
In EMA’s
assessment report for PALIPERIDONE (2015), various sensitivity analyses including tipping point
analysis were performed. This was also indicated in the paper “Paliperidone
Palmitate Once-Monthly Reduce Risk of Relapse of Psychotic, Depressive, and
Manic Symptoms and Maintains Functioning in a Double-Blind, Randomized Study of
Schizoaffective Disorder”.
Pattern Mixture Model: This analysis allows missing data to be missing not at random (MNAR). A repeated measures ANCOVA model for change in PSP included time as categorical factor, and a factor for completers versus early dropouts, as well as the interaction of completion status by treatment and time.
Tipping-point Analysis: Analysis of PSP score using an iterative process of worsening last observation carried forward (LOCF) values for only the active treatment group (paliperidone monthly) were implemented.
7 comments:
Dear Dr. Deng,
In my research, I assumed that data were MNAR. So I decided to conduct tipping point analysis using stata. Is it possible to conduct this analysis in stata? Would you please share an example of stata codes that are used for tipping point analysis?
Thanks.
Sonia Sharmin
Sorry, I don't use STATA.
Since the tipping point analysis is closely related to MI (multiple imputation), you may check the STATA codes for multiple imputation. For example, https://stats.idre.ucla.edu/stata/seminars/mi_in_stata_pt1_new/
Dear Dr. Deng, Thank the discussion you posted. Here my question is how to we select the tipping point that reverses the study conclusion (smin and smax).
I think that you can use the trial and error method for the tipping point selection.
Dr. Deng,
You had very comprehensive coverage on the Tipping point analysis. Do you an example for Binary outcome tipping analysis example to share?
Thanks,
Huibin
Hi Huibin,
you may want to take a look at this video "Tipping point analysis in multiple imputation for binary missing data" at:
https://www.quanticate.com/blog/tipping-point-analysis-in-multiple-imputation-for-missing-data
also check the paper by Paul Allison "Imputation of Categorical Variables with PROC MI" at
https://support.sas.com/resources/papers/proceedings/proceedings/sugi30/113-30.pdf
Hi,Dr. Deng,
your share is very useful for me. can you share more content about Tipping point analysis,J2R,CR for time to event data? thanks vary much
Post a Comment