Saturday, August 15, 2015

Tipping point analysis - multiple imputation for stress test under missing not at random (MNAR)

In a previous post, different imputation methods were summarized by the different missingness assumptions. One method, tipping point approach, has gained the popularity recently as an approach for performing the sensitivity analysis under the missing at not random (MNAR) assumption. In other words, the tipping point approach is like a progressive stress-testing to assess how severe departures from missing at random (MAR) must be in order to overturn conclusions from the primary analysis. If implausible departures from MAR in order to change the results from statistically significance (p<=0.05) to statistically insignificance (p>0.05), the results will be said to be robust to the departure from MAR assumption. We will then be more confident in the results obtained based on statistical methods with the MAR assumptions (such as multiple imputation, mixed model repeated measurements – MMRM). Tipping point approach is not intended for the primary analysis method and is only used for the sensitivity analysis.

Tipping point approach can be seen as a special application of the multiple imputation. It can also be considered as a special case of controlled imputation method (i.e., applying the shift parameter only to the active treatment group, not to the placebo group).

Implementing the tipping point approach include the following steps with the first three steps being the standard multiple imputation (MI) steps:
  1. The missing data are filled in m times to generate m complete data sets.
  2. The m complete data sets are analyzed by using standard procedures.
  3. The results from the m complete data sets are combined for the inference.
  4. Repeat the steps #1 to generate multiple imputed data sets, with a specified shift parameter that adjust the imputed values for observations in the treatment group, not the placebo group).
  5. Repeat the step 2 for the imputed data sets with shift parameter applied.
  6. Repeat the step 3 to obtain the p-value to see if the p-value is still <=0.05.
  7. Repeat the steps 4-6 with more stringent shift parameter applied until the p-value >0.05.
The tipping point approach can be easily implemented using SAS procedures MI and MIANALYZE. A SAS example “Sensitivity Analysis with Tipping-Point Approachprovides the step-by-step instructions how to implement the tipping point approach.

The following papers are also helpful in understanding and implementing the tipping point approach.

Tipping point approach has been discussed in several drug trials:

In Dry Powder Mannitol (DPM) Pharmaxis Pulmonary and Allergy Drugs Advisory Committee (slides are here) January 30, 2013, tipping point approach was used for stress test to see how robust primary analysis method is robust to the departure of the MAR assumption.
They explored the tipping point in the ITT population at which DPM would no longer show a significant effect. To do this, the penalty at each missing time point is increased up to the point that statistical significance is lost. They showed what happens when they stress tested the data even more. They increased the size of penalty for each missing visit in the pattern mixture model up until the point where significance is lost. The penalty would need to be more than 450 mLs at each missing time point before the effect estimate is reduced to 55 mLs and is no longer significant. This means that each patient leaving before week six could be penalized by 1,350 mLs. A tipping point requiring such a large volume does not seem plausible. They challenged the robustness even further, again using the same pattern mixture model, but this time identifying a tipping point when only penalizing the DPM arm but not control. Even applying this extreme method, the tipping point needed to reach 150 mLs before significance was lost. Now, this means that even patients withdrawing before week six in the control arm carry no penalty at all, but, similarly, DPM withdrawals being penalized by 450 mLs.
In FDA’s Statistical Review for NDA 204168 Drug Name: FETZIMA (Levomilnacipran) extended-release capsules 20, 40, 80, and 120 mg Indication: Major Depressive Disorder Applicant: Forest Laboratories, Inc.
A “tipping point” analysis was conducted by increasing the shift parameter beyond the maximum value of 8 considered by the sponsor. The mean difference in MADRS change scores between drug and placebo would loose statistical significance at alpha = 0.05 at a shift parameter of 16 (see Table 16). The value of 16 appears to be rather large and unlikely to be a realistic mean difference at yt+1 between patients that drop-out after the tth visit and patients that continue. The PMM model results are consistent with the primary MMRM model results at the more realistic values of the shift parameter (i.e., 2, 4, …, 14).
Slide presentation “Missing Data Sensitivity Analysis of a Continuous Endpoint – An Example from a Recent Submission” by Arno Fritsch indicated that the tipping point approach was explored as sensitivity analysis for 6MWT endpoint in Riociguat in Pulmonary Arterial Hypertension.
  • Need to increase penalty for riociguat to -71m per visit after drop-out until statistical significance is lost
  • Would imply a very steep decline after drop-out, even giving negative 6MWD values for many patients (Mean 6MWD at baseline 364m, some patients in the 200’s)
  • So positive treatment effect seems unquestionable
Pattern Mixture Model: This analysis allows missing data to be missing not at random (MNAR). A repeated measures ANCOVA model for change in PSP included time as categorical factor, and a factor for completers versus early dropouts, as well as the interaction of completion status by treatment and time.
 Tipping-point Analysis: Analysis of PSP score using an iterative process of worsening last observation carried forward (LOCF) values for only the active treatment group (paliperidone monthly) were implemented.

Friday, August 07, 2015


In this weekend's NPR on the media, "a skeptic's guide to health news and diet fads" was discussed. It mentioned a story that Johannes Bohannon deliberately designed a bad bogus study to test how the bogus study results was published and cited in the news. See the blog article "I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here's How" and the news "Study showing that chocolate can help with weight loss was a trick to show how easily shoddy science can make headlines". Just today, I read an article titled "Could Too Many Refined Carbs Make You Depressed?", which is most like a bogus study.

Here is the story from Wikipedia:
Publishing under the name Johannes Bohannon, he produced a deliberately bad study to see how the media would pick up their findings. He worked with a film-maker Peter Onneken who was making a film about junk science in the diet industry with fad diets becoming headline news despite terrible study design and almost no evidence.

Bohannon designed a deliberately bad study with a small sample size, many variables that naturally fluctuate in participants, and a statistician told to deliberately "massage the data" using overfitting and p-hacking. The study's sample size was tiny, measuring 18 different measurements from only 15 participants, who were split into three groups. The purported finding of the study was that eating chocolate could assist weight loss. The GP running the study sums up his dislike of food pseudoscience as a "religion" that teaches “Bitter chocolate tastes bad, therefore it must be good for you.” Two thirds of the participants were female, and natural weight changes due to menstrual cycles were greater than the observed difference between chocolate and low-carb groups. The group who were assigned to the "control" were not asked what their diet contained.

He submitted the manuscript to 20 open access publishers well known for their predatory journals; the article ended up published in the International Archives of Medicine. He invented a fake "diet institute" that lacks even a website, and used the pen name, "Johannes Bohannon," a name that does not have any publications or appear on any website. Bohannon fabricated a press release which was picked up on the front cover of German tabloid Bild, as well as "the Daily Star, the Irish Examiner, Cosmopolitan’s German website, the Times of India, both the German and Indian site of the Huffington Post, and even television news in Texas and an Australian morning talk show."

The few journalists who contacted the scientist asked puff piece questions and no reporter published how many subjects were tested, or quoted independent researchers. Most outlets sought to maximise page views by including "vaguely pornographic images of women eating chocolate." He argues that diet fads are covered like gossip columnists "echoing whatever they find in press releases" rather than evaluating the accuracy of scientific papers.

Bohannon argues that because of the large number of factors in diet and lifestyle, large scale studies are frequently inconclusive, even when billions of dollars have been spent on well-designed studies by government agencies that label obesity an epidemic.

The original paper "chocolate with high cocoa content as a weight-loss accelerator" can be read at: The statistics section of the paper is below. Looks real, right?
A t-test for independent samples was used to assess differences in baseline variables between the groups. The analysis was a repeated-measures analysis of variance in which the baseline value was carried forward in the case of missing data. One subject (low-carbohydrate) had to be excluded from the analysis, because of a weight measure is-sue within the trial

Unfortunately, in today's world, a lot of published studies were based on the bad science. The bogus studies can be written and published as if it is a real study. The news media would pick it up, disseminate and broadcast it like the great news.

The podcast from PBS is available below.

Saturday, August 01, 2015

Missing Data Mechanisms/Assumptions and the Corresponding Imputation Methods

Missing data is one of the classical issues in clinical trials and biostatistics. Since the National Research Council's report on missing data is issued in 2010, the paradigm has been shifted to the prevention of the missing data. Even the prevention has been given the great emphasis, the missing data is still inevitable in pretty much any clinical trial. When analyzing a clinical trial with the missing data, it is common that various sensitivity analyses need to be performed to see how the study result is robust to the handling of the missing data. Handling of the missing data depends on the assumptions. 

Missing Data Assumptions and the Corresponding Imputation Methods 

No assumption

Missing Complete at Random
Missing at Random – ignorability assumption
Missing Not at Random

The missingness is independent of both unobserved and observed data.

The probability of missingness is the same for all units.
Conditional on the observed data, the missingness is independent of the unobserved measurements.

The probability a variable is missing depends only on available information.
Not MCAR or MAR.

Missingness that depends on unobserved predictors.

Missingness is no longer at random if it depends on information that has not been recorded and this information also predicts the missing values.

Missingness that depends on the missing value itself

LOCF (last observation carried forward)

BOCF (baseline value carried forward)

WOCF (worst observation carried forward)

Imputation based on logical rules
CC (Complete-case Analysis) - listwise deletion

Pairwise Deletion

Available Case analysis

Single-value Imputation (for example, mean replacement, regression prediction (conditional mean imputation), regression prediction plus error (stochastic regression imputation )

– under MCAR, throwing out cases with missing data does not bias your inferences. However, there are many drawbacks
Maximum Likelihood using the EM algorithm – FIML (full information maximum likelihood)

MMRM (mixed model repeated measurement) – REML (restricted maximum likelihood)

Multiple Imputation

Two assumptions: the joint distribution of the data is multivariate normal and the missing data mechanism is ignorable

Under MAR, it is acceptable to exclude the missing cases, as long as the regression controls for all the variables that affect the probability of missingness
PMM (Pattern-mixture modeling)
Jump to Reference
Last Mean Carried Forward.
Copy Differences in Reference
Copy Reference

Tipping Point Approach

Selection model (Heckman)

Web resources are available in discussing the missing data and the handling of the missing data. Some of the recent materials are listed below. For people who are using SAS, SAS procedures MI and MIANALYZE are handy for use in performing the multiple imputation and pattern mixture model: