Sample Size Estimation Based on Precision for Survey and Clinical Studies such as Immunogenicity Studies

Sometimes, we may need to calculate the sample size to estimate a population proportion or a population mean with a precision or margin of error. Here we use the terms ‘precision’ and ‘margin of error’ interchangeably. The precision may also be referred as “half of the confidence interval”, “half of the width of CI”, and “Distance from mean to limit” depending on the sample size calculation software.

Statistician may need to estimate the sample sizes for the following situations:

Example 1: A survey estimated that 20% of all Americans aged 16 to 20 drove under the influence of drugs or alcohol. A similar survey is planned for New Zealand. The researchers want to estimate a sample size for the survey and they want a 95% confidence interval to have a margin of error of 0.04.

Example 2: an immunogenicity study is planned to investigate the occurrence of the antibody to a therapeutic protein. There is no prior information about the percentage patients who may develop the antibody to the therapeutic protein. How many patients are needed for the study with a 95% confidence interval and a precision of 10%?

Example 3: A tax assessor wants to assess the mean property tax bill for all homeowners in Madison, Wisconsin. A survey ten years ago got a sample mean and standard deviation of $1400 and $1000. How many tax records should be sampled for a 95% confidence interval to have a margin of error of $100?

These are set of situations where the sample size estimation is based on the confidence interval and the margin of error. The examples #1 and #2 are dealing with the one-sample proportion where we would like to estimate the sample size in order to obtain an estimate for population proportion with certain precision. The example #3 is dealing with one-sample mean where we would like to estimate the sample size in order to obtain an estimate for population mean with certain precision.

Sample Size to Estimate A Proportion With a Precision

The usually formula is:

N = z^2 p(1-q) / d^2

where p is the proportion (may be obtained from the previous study or and d is the precision or margin of error. Z is the Z-score e.g. 1.645 for a 90% confidence interval, 1.96 for a 95% confidence interval, 2.58 for a 99% confidence interval

For example #1, the sample size will be calculated as:
          N = 1.96^2 x 0.2 x 0.8/0.04^2 = 384.2 round up to 385

Similarly, if we use PASS, the input parameters will be

         Confidence Interval:  Simple Asynptotic

         Interval Type: Two-sided

         Confidence level (1-alpha): 0.95

         Confidence Interval Width (two-sided): 0.08      (note: 0.04 x 2)

         P (Proportion): 0.2

For example #2, since there is no prior information about the proportion, the practical way is that if no estimate of p is available, assume p = 0.50 to obtain a sample that is big enough to ensure precision.

If we use formula, the sample size will be calculated as:

          N = 1.96^2 x 0.5 x 0.5 / 0.1^2 = 96

Similarly, if we use PASS, the input parameters will be

          Confidence Interval:  Simple Asymptotic

          Interval Type: Two-sided

          Confidence level (1-alpha): 0.95

          Confidence Interval Width (two-sided): 0.2    (note: 0.1 x 2)

          P (Proportion): 0.5

Sample Size to Estimate A Proportion With a Precision

The usually formula for is:

          N = (s t/d)^2

Where s is the standard deviation, t is the t-score (approximate to Z-score if assuming normal) and d is the precision or margin of error.

For example #3:

N=(1000 x 1.96/100)^2 = 385   

Similarly, if we use PASS, the input parameters will be:
                    Solved for: Sample size
                    Interval type: two-sided
                    Population size: infinite
                    Confidence Interval (1-alpha): 0.95
                    Distance from mean to limits: 100
                    S (standard deviation): 1000

The sample size calculation based on the precision is population in survey in epidemiology studies and polling in political science. In clinical trials, it seems to be common in immunogenicity studies. In immunogenicity studies, it is not just for one sample situation, it may also be used in the two sample situation. In a book “Biosimilars: Design and Analysis of Follow-on Biologics” by Dr Chow, sample size section mentioned the calculation based on precision:

In immunogenicity studies, the incidence rate of immune response is expected to be low. In this case, the usual pre-study power analysis for sample size calculation for detecting a clinically meaningful difference may not be feasible. Alternatively, we may consider selecting an appropriate sample size based on precision analysis rather than power analysis to provide some statistical inference.

The half of the width of the CI by w=Z(1-alpha)/2*sigma hat which is usually referred to as the maximum error margin allowed for a given sample size n. In practice, the maximum error margin allowed represents the precision that one would expect for the selected sample size. The precision analysis for sample size determination is to consider the maximum error margin allowed. In other words, we are confident that the true difference signma=pR-pr would fall within the margin of w=Z(1-alpha)/2*sigma for a given sample size of n. Thus, the sample size required for achieving the desired precision can be chosen.

This approach, based on the interest in only the type I error, is to specify precision while estimating the true delta for selecting n.
Under a fixed power and significance level, the sample size based on power analysis is much larger than the sample size based on precision analysis with extremely low infection rate difference or large allowed error margin.

SAS Proc Power can also calculate the sample size. The exact method is used for sample size calculation in SAS. The obtained sample size is usually greater that the ones calculated by hand (formula) or using PASS.

For confidence interval for one-sample proportion situation, the SAS codes will be something like this:

      proc power;

           onesamplefreq ci=wilson

           halfwidth = 0.1

           proportion = 0.3

           ntotal = 70

           probwidth = .;

   run;

For confidence interval for one-sample mean, refer to an example provided in SAS online document:  SAS 9.22 User’s Guide Example 68.7 Confidence Interval PrecisionExample

References:

• Determine the Sample Size Class Note by Dr Richard Smith
• Confidence Intervals and Sample Size
• PASS Confidence Intervals for One Proportion
• PASS Confidence Intervals for one proportion from a finite population

How Low in Sample Size Can We Go? FDA approves ultra-orphan drug on a 4-patient trial

The development of the orphan drugs has been red hot in recent years due to the favorable regulatory environment and the government incentives. It is well know that it is challenging to conduct the clinical trials in orphan diseases. The rigorous statistical rules and principles may not be feasible to be followed in drug trials in rare diseases. The sample size in rare diseases are usually small, but how low in sample size can we go? A recent FDA approval gives an answer.

Recently, FDA approved a new orphan drug to treat rare autosomal recessive disorder. The drug called Xuriden was approved for treating patients with Hereditary orotic aciduria – an ultral orphan disease. According to FDA’s announcement:

“The safety and effectiveness of Xuriden were evaluated in a single arm, six-week, open-label trial in four patients with hereditary orotic aciduria, ranging in age from three to 19 years of age, and in a six-month extension phase of the trial. The study assessed changes in the patients’ pre-specified hematologic parameters during the trial period. At both the six-week and six-month assessments, Xuriden treatment resulted in stability of the hematologic parameters in all four clinical trial patients.”

It looks like that the company (Wellstat Therapeutics) initially planned to enroll 10 subjects. I guess that due to the enrollment challenge, only 4 subjects were enrolled. Wellstat presented the data from 4 subjects and got the green light from FDA. According to the clinicaltrials.gov, the study protocol is titled "Open-Label Study of Uridine Triacetate in Pediatric Patients With Hereditary Orotic Aciduria" and the primary and secondary outcome measures are biomarkers or PD markers:

Primary Outcome Measures:

•  Stability of predetermined principal hematologic parameters

Secondary Outcome Measures:

• Levels of orotic acid and orotidine in urine 
• Levels of uridine in the plasma

With the results from four patients, it is not possible to perform any inferential statistical analysis. The approval is essentially based on the summary results or data listings from the outcome measures in 4 patients. See Clinical Studies section in Xuriden package insert.

As part of the approval, the manufacturer of Xuriden (Wellstat therapeutics) was also granted a rare pediatric disease priority review voucher. The priority review voucher itself worthies millions dollars and probably will be more than enough to recoup all the investment costs that the company has invested in the development of Xuriden.

With the emphasis on precision medicine and personalized medicine, we will see a trend of conducting more smaller trials. In the future, every drug is so specifically targeted that it would apply to only a few people worldwide. If we can really achieve the precision medicine, more and more medicines will be classified as orphan drugs.

Some backgrounds about the orphan drug is summarized in one of my previous articles "Drug for Treating Rare Diseases: Orphan Drug, Orphan Disease, Orphan Subset". Similarly a presentation by Dr Lawrence J. Lesko from FDA gave a good introduction about the rare diseases and orphan drugs. Rare and ultra-rare diseases, often referred to as orphan and ultra-orphan diseases, affect very small numbers of patients. In the United States, a disease is defined as rare if it affects fewer than 650 patients per million of population, and the European definition of a rare disease is one that affects fewer than 500 patients per million of population. In contrast, a disease is generally considered to be ultra rare if it affects fewer than 20 patients per million of population (or, one patient per 50,000 people)—and most ultra-rare diseases affect far fewer than this—as few as one per million or less.

Time to Event End Points

Time to Event analysis has expanded beyond the traditional term of survival analysis even though the term of survival analysis may still be used in oncology trials and time to event analysis remain as the primary tool in oncology clinical trials. In oncology area, depending on the type of the cancer and the regulatory requirement, the primary efficacy endpoint may be one of the following 'time to event' variables: overall survival (time to death), progression free survival (time to progression), disease free survival (time to disease occurrence), event free survival (time to event occurrence),… In the non-oncology area, we also often need to perform 'time to event' analysis. Except in oncology area where the terms 'overall survival', 'progression free survival' are used, all 'time to event' endpoints will be labelled with ‘time to something’. Here are some of the examples of 'time to event' variable in non-oncology areas: 

Endpoint	Indication
Time to hemostasis (time to stopping the bleeding)	Adjunct to Hemostasis after various types of surgeries using fibrin sealant or thrombin COPD, Bronchiectasis
Time to first pulmonary exacerbation
Time to healing of the primary lesion complex (loss of crust from vesicular [classical] lesions)	Recurrent Herpes Labialis
Time to clinical worsening	Pulmonary arterial hypertension
Time to parasite clearance Time to recurrent infection	Malaria

For all 'time to event' variables, the EVENT of interest must be clearly pre-defined. Sometimes, defining an EVENT is not easy. In many situations, the EVENT of interest may need to be adjudicated by an event adjudication committee (EAC) or by central reader (if an Event of interest is determined by imaging).

Depending on the definition of the ‘event’, the 'time to event' variable could be a hard end point or a surrogate (soft) endpoint.

Overall survival (time to death) is usually considered as a hard end point because death is a definite event.

Time to first hospitalization, time to lung transplantation,… may be considered as a hard endpoint. However, the hospitalization and lung transplantation could be impacted by the health care resources and might be different depending on the countries/regions.

If there is a composite endpoint that includes the 'time to MI', 'time to Stroke', you might think that the event MI and Stroke are definite endpoints. You might think that myocardial infarction (MI) is an event that can be easily identified/diagnosed until you realized that the myocardial infarction could be determined based on cardiac markers such as tropolin level. Whenever we deal with a laboratory test, we will run into the issue of the assay sensitivity, the measurement error, and the cut point for defining the event. Stroke might also be considered as identifiable event, but if the stroke event includes the transient ischemic attack (TIA), TIA may not be easily identifiable.

'Time to event' variables may depend on the pre-specified schedules for checking the EVENT of interest. For example, in malaria studies, ‘time to parasite clearance’ was assessed by taking blood samples and examining it by light microscopy prior (0 hour) and during treatment at 4, 8, 12 hours and then 6 hourly until two consecutive negative blood slides. If the pre-specified time points for taking blood samples are different, the ‘time to parasite clearance’ variable will also be different.

For Progression Free Survival or Time to Progression variables, the disease progression will need to be clearly defined. For the solid tumor, the progression is usually defined using Response Evaluation Criteria In Solid Tumors (RECIST) criteria that is usually based on the radiological and imaging (CT, MRI) results to measure the changes in tumor size (or target lesions).

Many women with early-stage breast cancer undergo breast-conserving surgery followed by whole breast irradiation, which reduces the rate of local recurrence. Radiotherapy to the chest wall and regional lymph nodes, termed regional nodal irradiation, which is commonly used after mastectomy in women with node-positive breast cancer who are treated with adjuvant systemic therapy, reduces locoregional and distant recurrence and improves overall survival. In early stage breast cancer studies, Disease Free Survival (DSF) measures the length of time after primary treatment to the re-occurrence of any signs or symptoms of the cancer. DFS may also be called relapse-free survival or RFS. It will be a surrogate endpoint since the determination of the re-occurrence of any signs or symptoms may not be accurately detected. 

In oncology studies, Event-Free Survival (EFS) measures the length of time after primary treatment to the re-occurrence of certain complications or events that the treatment was intended to prevent or delay. These events may include the return of the cancer or the onset of certain symptoms, such as bone pain from cancer that has spread to the bone. For example, in neoroblastoma studies, Event-Free Survival was used and the EFS was defined as the time from study enrollment (which occurred after transplantation) until the first occurrence of relapse, progressive disease, secondary cancer, or death or, if none of these events occurred, until the last contact with the patient. It will also be a surrogate end point since the determination of relapse, progressive disease, and secondary cancer may depend on when the patients are examined and how the imaging results are examined.

In summary, for time to event variables, it is critical to have criteria to determine the Event of Interest. If the determination of the Event of Interest is soft, it may require a third party (independent of the sponsor and the investigator) to determine the Event of Interest. If the radiological and imaging techniques are used, a central reader is usually needed. For other 'time to event' variables where the determination of the event is soft, an independent Event Adjudication Committee (EAC) is usually needed. For the studies in the same indication, it is ideal to have standardized criteria to determine the Event of interest so that the study results across different sponsors may be compared. For example, in breast cancer area, people are trying to use a STEEP system to standardize the criteria for clinical trial end points. 

References:

• End Points and United States Food and Drug Administration Approval of Oncology Drugs
• NCI: Imaging Response Criteria
• R. Sridhara: Statistical Considerations in Designing Randomized Clinical Trials
• R A Kyle and S V Rajkumar Definitions of time to event end points in multiple myeloma

Understanding the endpoints in oncology: overall survival, progression free survival, hazard ratio, censored value

For clinical trials in oncology area, various terms related to the study endpoints are confusing to non-statisticians. The commonly used terms such as overall survival, progression free survival, censor, and hazard ratio are not straightforward to non-statisticians.

FDA guidance “Guidance for Industry Clinical Trail Endpoints for the Approval of Cancer Drugs and Biologics” delineated various endpoint measures and provided guidance on which endpoint should be used in which situation.

Overall Survival versus Survival Rate: The term overall survival can be easily confused with the survival rate or can be easily thought as the survival rate. While these two terms are related, they measure different things. Overall Survival is a measure of time to event and Survival rate measures the percentage of subjects who survived (at the end of the study, after 3 years, 5 years,…). 

Perhaps, it is clearer to understand the differences if we put these terms side-by-side for a comparison.

Table 1: Comparison of Overall Survival and Survival Rate

Overall Survival (OS)	Survival Rate
Measuring how long a patient can survive	Measuring how many patients survive during a given time (3 year, 5 years)
The time from the randomization or the start of the study treatment to the death	the percentage of subjects who are alive at the end of the study
Hard endpoint since both are based on the death event
Commonly used in clinical trials	Commonly used in epidemiology
Recommended study endpoint	Not usually used as the study endpoint
Analyzed using survival analysis or time to event analysis methods (Kaplan-Meier estimate, log-rank test, proportional hazard model,…)	Analyzed as proportion/rate/ratio (Chi-square test, CMH test,..) or dichotomous variable using logistic regression.

  

Table 2: Comparison of Overall Survival and Progression Free Survival

Overall Survival (OS)	Progression Free Survival (PFS)
Measuring how long a patient can survive	Measuring how long a patient can live without disease progression
The time from the randomization or the start of the study treatment to the death (all causes)	the time from the randomization or the start of the study treatment to the disease progression including death
Commonly used in clinical trials
Hard endpoint	Surrogate endpoint
Analyzed using survival analysis or time to event analysis methods (Kaplan-Meier estimate, log-rank test, proportional hazard model,…)
Study with OS as the primary efficacy endpoint requires relatively larger sample size and longer follow-up duration	Study with PFS as the primary efficacy endpoint requires relatively smaller sample size and shorter follow-up duration
	PFS may or may not predict the OS.
	How to determine the disease progression may get tricky. Disease progression often relies on the imaging (for example, using imaging to determine if there is any change in tumor size)

Disease free survival (DFS), event free survival (EFS), time to progression (TTP) et al have the similar features to the Progression Free Survival (PFS). 

• Disease free survival: time to disease reoccurring. Measuring the length of time after treatment during which no disease is found. 
• Event free survival: Time from randomization* to disease progression, death, or discontinuation of treatment for any reason (eg, toxicity, patient preference, or initiation, of a new treatment without documented progression). may be useful in evaluation of highly toxic therapies
• Time to progression: Time from randomization* until objective tumor progression; does not include deaths

Table 3: Comparison of Hazard Ratio and Risk Ratio

Hazard Ratio (HR)	Risk Ratio (RR)
For time to event variables for example OS and PFS	For binary variables (such as live/death, success/not success)
The Hazard ratio is the ratio of the probability of an event (death or progression) in the experimental arm to the probability in the comparator arm. A hazard is the rate at which events happen	The risk ratio (or relative risk) is the ratio of the risk of an event in the two groups.
The hazard ratio is a related measure that weights the risk change according to when events occur over time	The relative risk is a measure of the relative change in the risk of a preventable event.
Calculated using (Cox) proportional hazard model (for example using SAS PROC PHREG)	Calculated by dividing two proportions (for example proportion of subjects who survived in active group / proportion of subjects who survived in placebo group). In SAS, it can be obtained using PROC FREQ or PROC Logistic
Assumption that the data follows the proportional hazard	No such assumption is needed
A hazard ratio of 2 means the event will occur twice as often at each time point (at any given instantaneous time point) given a one-unit increase in the predictor.	A risk ratio of 2 means that the event is 2 time more probable given a one-unit increase in the predictor

Risk ratio and relative risk are two terms that can be used interchangeably. Risk ratio and odds ratio are similar and have the same features, but with different formula for calculation. For clinical trials, both Risk Ratio and Odds Ratio are used. For epidemiology studies especially the case-control studies, Odds Ratio is usually used. See table below for comparison in calculating the Risk Ratio and Odds Ratio.

Table 4: Calculation of risk ratio (RR), odds ratio (OR) and risk difference (RD) from a 2×2 table

The results of a clinical trial can be displayed as a 2×2 table: Event (‘Success’) No event (‘Fail’) Total Experimental intervention SE FE NE Control intervention SC FC NC where SE, SC, FE and FC are the numbers of participants with each outcome (‘S’ or ‘F’) in each group (‘E’ or ‘C’). The following summary statistics can be calculated:

The term ‘censored value’ is used to describe an incomplete measure. For example, for a biomarker or laboratory measures, if the value is too low or too high that exceeding the quantification limit (for example exceeding the lower limit of quantification – LLQ), we would indicate the value is less than LLQ, but the exact value is unknown.

The “censored value” is especially common in cancer clinical trials. For patients who did not experience the event, the time to event will be censored. Suppose patients are followed in a study for 20 weeks. A patient who does not experience the event of interest for the duration of the study is said to be censored (exactly right censored). The interpretation is that even though the subject does not experience the event and we cannot calculate the time to event, we set up the time to event as a censored value for time to event analyses.  

For cancer clinical trials, it is essential to understand whether or not the study endpoint is to count the events or to measure the time to events. Measuring the time to events is more commonly accepted endpoint in cancer clinical trials. 

References: 

• Clinical Endpoints: Advantages and Limitations
• FDA Guidance: BiologicsClinical Trial Endpoints for the Approval of Cancer Drugs and Biologics 
• FDA Guidance: Clinical Trial Endpoints for the Approval of Non-Small Cell Lung Cancer Drugs and Biologics
• EMA Guidance: Guideline on the evaluation of anticancer medicinal products in man
• Robert Kane FDA Oncology Drug Approval: endpoint, effectiveness, and approval
• Hotte et al (2011) Progression-free survival as a clinical trial endpoint in advanced renal cell carcinoma
• Genetech: The Ongoing Evolution Of Endpoints in Oncology

Tipping point analysis - multiple imputation for stress test under missing not at random (MNAR)

In a previous post, different imputation methods were summarized by the different missingness assumptions. One method, the tipping point approach, has gained popularity recently as an approach for performing the sensitivity analysis under the missing not at random (MNAR) assumption. In other words, the tipping point approach is like a progressive stress-testing to assess how severe departures from missing at random (MAR) must be in order to overturn conclusions from the primary analysis. If implausible departures from MAR in order to change the results from statistically significant (p<=0.05) to statistically insignificant (p>0.05), the results will be said to be robust to the departure from MAR assumption. We will then be more confident in the results obtained based on statistical methods with the MAR assumptions (such as multiple imputation, mixed model repeated measurements – MMRM). Tipping point approach is not intended for the primary analysis method and is only used for the sensitivity analysis.

The tipping point approach can be seen as a special application of the multiple imputation. It can also be considered as a special case of the controlled imputation method (i.e., applying the shift parameter only to the active treatment group, not to the placebo group).

Implementing the tipping point approach includes the following steps with the first three steps being the standard multiple imputation (MI) steps:

1. The missing data are filled in m times to generate m complete data sets.
2. The m complete data sets are analyzed by using standard procedures.
3. The results from the m complete data sets are combined for the inference.
4. Repeat the steps #1 to generate multiple imputed data sets, with a specified shift parameter that adjusts the imputed values for observations in the treatment group, not the placebo group).
5. Repeat step 2 for the imputed data sets with shift parameter applied.
6. Repeat step 3 to obtain the p-value to see if the p-value is still <=0.05.
7. Repeat steps 4-6 with a more stringent shift parameter applied until the p-value >0.05.

The tipping point approach can be easily implemented using SAS procedures MI and MIANALYZE. A SAS example “Sensitivity Analysis with Tipping-Point Approach” provides step-by-step instructions on how to implement the tipping point approach.

The following papers are also helpful in understanding and implementing the tipping point approach.

• Ratitch and O’kelly (2011) Implementation of Pattern-Mixture Models Using Standard SAS/STAT Procedures
• Ratitch and O’kelly  Placebo MI including Tipping Point Analysis with Delta-Adjusting ImputationPlacebo MI including Tipping Point Analysis with Delta-Adjusting Imputation
• Yuan (2014) Sensitivity Analysis in Multiple Imputation for Missing Data
• PhD Dissertation by Smuk (2015)Missing Data Methodology:- Sensitivity analysis after multiple imputation (with tipping point analysis, we select a shift parameter that moves the data further and further away from MAR in a particular direction, until key inferences from the substantive model change. Experts can then discuss whether the degree of departure from MAR needed to change the substantive results is plausible)

The tipping point approach has been discussed in several drug trials:

In Dry Powder Mannitol (DPM) Pharmaxis Pulmonary and Allergy Drugs Advisory Committee (slides are here) January 30, 2013, tipping point approach was used for a stress test to see how the primary analysis method is robust to the departure of the MAR assumption.

They explored the tipping point in the ITT population at which DPM would no longer show a significant effect. To do this, the penalty at each missing time point is increased up to the point that statistical significance is lost. They showed what happens when they stress tested the data even more. They increased the size of penalty for each missing visit in the pattern mixture model up until the point where significance is lost. The penalty would need to be more than 450 mLs at each missing time point before the effect estimate is reduced to 55 mLs and is no longer significant. This means that each patient leaving before week six could be penalized by 1,350 mLs. A tipping point requiring such a large volume does not seem plausible. They challenged the robustness even further, again using the same pattern mixture model, but this time identifying a tipping point when only penalizing the DPM arm but not control. Even applying this extreme method, the tipping point needed to reach 150 mLs before significance was lost. Now, this means that even patients withdrawing before week six in the control arm carry no penalty at all, but, similarly, DPM withdrawals being penalized by 450 mLs.

In FDA’s Statistical Review for NDA 204168 Drug Name: FETZIMA (Levomilnacipran) extended-release capsules 20, 40, 80, and 120 mg Indication: Major Depressive Disorder Applicant: Forest Laboratories, Inc.

A “tipping point” analysis was conducted by increasing the shift parameter beyond the maximum value of 8 considered by the sponsor. The mean difference in MADRS change scores between drug and placebo would loose statistical significance at alpha = 0.05 at a shift parameter of 16 (see Table 16). The value of 16 appears to be rather large and unlikely to be a realistic mean difference at yt+1 between patients that drop-out after the tth visit and patients that continue. The PMM model results are consistent with the primary MMRM model results at the more realistic values of the shift parameter (i.e., 2, 4, …, 14).

Slide presentation “Missing Data Sensitivity Analysis of a Continuous Endpoint – An Example from a Recent Submission” by Arno Fritsch indicated that the tipping point approach was explored as sensitivity analysis for 6MWT endpoint in Riociguat in Pulmonary Arterial Hypertension.

• Need to increase penalty for riociguat to -71m per visit after drop-out until statistical significance is lost
• Would imply a very steep decline after drop-out, even giving negative 6MWD values for many patients (Mean 6MWD at baseline 364m, some patients in the 200’s)
• So positive treatment effect seems unquestionable

In EMA’s assessment report for PALIPERIDONE (2015), various sensitivity analyses including tipping point analysis were performed. This was also indicated in the paper “Paliperidone Palmitate Once-Monthly Reduce Risk of Relapse of Psychotic, Depressive, and Manic Symptoms and Maintains Functioning in a Double-Blind, Randomized Study of Schizoaffective Disorder”.

Pattern Mixture Model: This analysis allows missing data to be missing not at random (MNAR). A repeated measures ANCOVA model for change in PSP included time as categorical factor, and a factor for completers versus early dropouts, as well as the interaction of completion status by treatment and time.
 Tipping-point Analysis: Analysis of PSP score using an iterative process of worsening last observation carried forward (LOCF) values for only the active treatment group (paliperidone monthly) were implemented.

A SKEPTIC'S GUIDE TO HEALTH NEWS AND DIET FADS; A BOGUS STUDY TREATED AS REAL

In this weekend's NPR on the media, "a skeptic's guide to health news and diet fads" was discussed. It mentioned a story that Johannes Bohannon deliberately designed a bad bogus study to test how the bogus study results was published and cited in the news. See the blog article "I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here's How" and the news "Study showing that chocolate can help with weight loss was a trick to show how easily shoddy science can make headlines". Just today, I read an article titled "Could Too Many Refined Carbs Make You Depressed?", which is most like a bogus study.

Here is the story from Wikipedia:

Publishing under the name Johannes Bohannon, he produced a deliberately bad study to see how the media would pick up their findings. He worked with a film-maker Peter Onneken who was making a film about junk science in the diet industry with fad diets becoming headline news despite terrible study design and almost no evidence.

Bohannon designed a deliberately bad study with a small sample size, many variables that naturally fluctuate in participants, and a statistician told to deliberately "massage the data" using overfitting and p-hacking. The study's sample size was tiny, measuring 18 different measurements from only 15 participants, who were split into three groups. The purported finding of the study was that eating chocolate could assist weight loss. The GP running the study sums up his dislike of food pseudoscience as a "religion" that teaches “Bitter chocolate tastes bad, therefore it must be good for you.” Two thirds of the participants were female, and natural weight changes due to menstrual cycles were greater than the observed difference between chocolate and low-carb groups. The group who were assigned to the "control" were not asked what their diet contained.

He submitted the manuscript to 20 open access publishers well known for their predatory journals; the article ended up published in the International Archives of Medicine. He invented a fake "diet institute" that lacks even a website, and used the pen name, "Johannes Bohannon," a name that does not have any publications or appear on any website. Bohannon fabricated a press release which was picked up on the front cover of German tabloid Bild, as well as "the Daily Star, the Irish Examiner, Cosmopolitan’s German website, the Times of India, both the German and Indian site of the Huffington Post, and even television news in Texas and an Australian morning talk show."

The few journalists who contacted the scientist asked puff piece questions and no reporter published how many subjects were tested, or quoted independent researchers. Most outlets sought to maximise page views by including "vaguely pornographic images of women eating chocolate." He argues that diet fads are covered like gossip columnists "echoing whatever they find in press releases" rather than evaluating the accuracy of scientific papers.

Bohannon argues that because of the large number of factors in diet and lifestyle, large scale studies are frequently inconclusive, even when billions of dollars have been spent on well-designed studies by government agencies that label obesity an epidemic.

The original paper "chocolate with high cocoa content as a weight-loss accelerator" can be read at: http://www.scribd.com/doc/266969860/Chocolate-causes-weight-loss. The statistics section of the paper is below. Looks real, right?

A t-test for independent samples was used to assess differences in baseline variables between the groups. The analysis was a repeated-measures analysis of variance in which the baseline value was carried forward in the case of missing data. One subject (low-carbohydrate) had to be excluded from the analysis, because of a weight measure is-sue within the trial

Unfortunately, in today's world, a lot of published studies were based on the bad science. The bogus studies can be written and published as if it is a real study. The news media would pick it up, disseminate and broadcast it like the great news.

The podcast from PBS is available below.