Futility Analysis in Clinical Trials - Stop the trial for futility

A colleague of mine asked me to explain the concept of “futility analysis” using plain languages. The question is triggered by the recent news such as “Solanezumab, Gammagard Trials Survive Futility Analysis” In an Alzheimer trial, it even says from the futility analysis, “there is greater than a 20% statistical probability of success in achieving the primary outcome measure of cognitive function preservation.

During a clinical trial, we can perform interim analysis (or DMC, DSMB review) for three different reasons:

1. The interim analysis for safety

1)       with pre-specified stopping rule (for example stop the trial if we see # of cases of Serious Adverse Events)

2)       without pre-specified stopping rule (rely on DMC members to review the overall safety)

2. The interim analysis for efficacy: To see if the new treatment is overwhelmingly better than control  - then stop the trial for efficacy
3. The interim analysis for futility:  To see if the new treatment is unlikely to beat the control – then stop the trial for futility  - this is called ‘futility analysis’.

In situations 2 and 3, the criteria for stopping rule for efficacy could be different from the stopping rule for futility, but need to be pre-specified.

In situation #2 (stopping the study for efficacy), there will be a penalty for alpha spending - if the overall alpha is 0.05, a portion of the alpha will be allocated for tee interim analysis and the alpha for final analysis will be less than 0.05.   In situation #3, there is no penalty for alpha spending. 

An example for futility analysis: at the beginning of the trial, we assumed 65% successful rate for new treatment group and 50% successful rate for the control group. We would like to establish superiority. In the middle of the study, we did an interim analysis. The interim analysis showed 55% successful rate for new treatment group and 50% successful rate for the control group. Based on the results from the interim analysis, we can calculate the probability and conditional power: if we continue to finish the study, what is the probability of the new treatment group better than control? If this probability is too small and meets the pre-specified criteria, we would stop the trial for futility. If this probability is reasonable, we can continue the trial as pre-planned or we can continue the trial with the sample size adjustment (typically increase due to the smaller effect size).

In a paper by Miller et al “Paclitaxel plus Bevacizumab versus Paclitaxel Alone for Metastatic Breast Cancer”, one pre-planned interim analysis and two additional interim analyses were performed and three stopping rules (for safety, for efficacy, and for futility) were pre-specified and evaluated. It is reasonable to assume that none of these stopping rules was triggered since the study was not stopped.  

Futility analysis or stopping the trial for futility is not without controversy. An article by Schoenfeld and Meade discussed this issue. See “Pro/con clinical debate: It is acceptable to stop large multicentre randomized controlled trials at interim analysis for futility”.

Further readings:

• Snapinn et al (2006) Assessment of futility in clinical trials
• Lachin (2005) A review of methods for futility stopping based on conditional power (or free access from here)

How to interpret odds ratios that are smaller than 1?

Here is a question posted on the web about the interpretation of odds ratios that are less than 1.

"I know that OR estimates= 1 mean that both groups/categories have the same odds. I also know that if OR estimates are greater than 1, e.g, 1.24 for Young vs. Old persons, then I can say: Young people have 24% increase in the odds of living in an apartment than older people. Or, I also know I can say, for example, for an OR of 0.322 Non-White vs. White, that the odds of Whites are 1/.322 = about 3 times higher than those of Non-Whites, to live in a house they own. Now, how would I say the odds are of a NON-White person in the example above, to live in a house they own? Is is 1-.322=.678 less likely, with respect to odds, to live in a house they own? Or, similarly, they have 67.8% lower odds to live in a house they own? "

If we have to say the odds for a Non-White person, we may say "Non Whites have odds .322 times as great as those of Whites".

In an article "When can odds ratio misled?", Davies et al stated:

"the odds of an event is the number of those who experience the event divided by the number of those who do not. It is expressed as a number from zero (event will never happen) to infinity (event is certain to happen). Odds are fairly easy to visualise when they are greater than one, but are less easily grasped when the value is less than one. Thus odds of six (that is, six to one) mean that six people will experience the event for every one that does not (a risk of six out of seven or 86%). An odds of 0.2 however seems less intuitive: 0.2 people will experience the event for every one that does not. This translates to one event for every five non-events (a risk of one in six or 17%). "

Another webblog described the issue in interpreting the odds ratio that is less than one.

"When you are interpreting an odds ratio (or any ratio for that matter), it is often helpful to look at how much it deviates from 1. So, for example, an odds ratio of 0.75 means that in one group the outcome is 25% less likely. An odds ratio of 1.33 means that in one group the outcome is 33% more likely."

In an article "The odds ratio: calculation, usage, and interpretation" in Biochemia Medica, the author clear suggest converting the odds ratio to be greater than 1 by arranging the higher odds of the evnet to avoid the difficulties in interpreting the odds ratio that is less than 1.

“An OR of less than 1 means that the first group was less likely to experience the event. However, an OR value below 1.00 is not directly interpretable. The degree to which the first group is less likely to experience the event is not the OR result. It is important to put the group expected to have higher odds of the event in the first column. It is not valid to try to determine how much less the first group’s odds of the event was than the second group’s. When the odds of the first group experiencing the event is less than the odds of the second group, one must reverse the two columns so that the second group becomes the first and the first group becomes the second. Then it will be possible to interpret the difference because that reversal will calculate how many more times the second group experienced the event than the first. If we reverse the columns in the example above, the odds ratio is: (5/22)/(45/28) = (0.2273/1.607) = 0.14 and as can be seen, that does not tell us that the new drug group died 0.14 times less than the standard treatment group. In fact, this arrangement produces a result that can only be interpreted as “the odds of the first group experiencing the event is less than the odds of the second group experiencing the event”. The degree to which the first group’s odds are lower than that of the second group is not known.”

In practice, when dealing with the odds ratio less than 1, when possible, I almost always try to reverse the column or recode the response variable to get the odds ratio larger than 1 before I do an interpretation. It is easier for people (especially non-statisticians) to understand the odds ratio with the value greater than 1.

In an example below, the treatment group is actually less effective in terms of the response.

Treatment	Failure (0)	Success (1)
No (0)	21	30
Yes (1)	32	17

The following SAS code can be easily used to calculate the odds ratio:
Data test; input Trt resp count; datalines;
1 1 17
1 0 32
0 1 30
0 0 21
;
proc logistic data=test descending; weight count; model resp=trt;
run;

From the SAS outputs, we get the odds ratio of 0.372, which indicates that the treatment group has odds 0.372 times lower compared to the non-treatmetn group in terms of the success. The interpretation is somewhat difficult to understand.

The program can be easily revised to calculate the odds ratio of failure rate, which gives an odds ratio of 1/0.372 = 2.689. The odds ratio can be intepretated as "the odds of achieve the success in non-treatment group is 2.689 times higher than that in treatment group".

proc logistic data=test; weight count; model resp=trt;
run;

In SAS PROC Logistic, with descending option, probability modeled is response=1 (success); without descending option, probability modeled is response=0 (failure);

Design and Analysis of Bioequivalence Studies for Highly Variable Drugs (HVD) or Highly Variable Drug Products (HVDP)

For bioequivalence studies, it is often for us to show the average bioequivalence by declaring the bioequivalence if the 90% confidence interval of the geometric least squares mean ratio is within 80-125%. The associated study design is typically 2x2x2 cross over design with reasonable sample size (for example, 12 subjects, 24 subjects,…) if the within subject variable is not so big. This approach has been outlined in several FDA’s guidelines:

-         Statistical Approaches to Establishing Bioequivalence

-         Bioavailability and Bioequivalence Studies for Orally Administered Drug Products — General Considerations

-         Food-Effect Bioavailability and Fed Bioequivalence Studies

Recently, there are a lot of discussions about the bioequivalence studies for a product with high variability (high variable drugs). Highly Variable Drugs refer to the type of drugs with higher within subject variability and  is Defined as one for which the root mean square error (RMSE) from the ANOVA bioequivalence analysis > 0.3 for either AUC or Cmax.

For highly variable drugs, if we employ the common study design, the required sample size will be very large, which will cause the ethic concerns to implement such studies.

FDA had several advisory committee meeting in discussing this issue. The most recent meetings were in 2004 and 2009. In 2009 meeting, the slide presentation  by Dr Conner from FDA summarized the development in dealing with this issue and FDA’s position (see slide presentation “Bioequivalence Methods for Highly Variable Drugs and Drug Products”).

Among various approaches to address the bioequivalence issue for highly variable drugs, reference-scaled average BE approach has been suggested. This approach requires less subjects in the study, but with replicated treatment design such as three-period, reference- replicated, crossover design with sequences of TRR, RTR, & RRT or four-period design with sequences of TRTR and RTRT. The replicated crossover designs were also discussed in FDA guidance “Statistical Approaches to Establish Bioequivalance”, but was for dealing with the carryover effects. Here, the replicated crossover designs are for dealing with highly variable drugs.

The implementation of the reference-scaled average BE approaches have been detailed and discussed in FDA guidance (draft) many publications. The most relevant ones are:

-         FDA Guidance on Progesterone (2011)

-         Sample Sizes for Designing Bioequivalence Studies for Highly Variable Drugs by Endrenyi and Tothfalusi (2012)

-         Bioequivalence of Highly Variable Drugs: A Comparison of the Newly Proposed Regulatory Approaches by FDA and EMA by Karalis et al (2011)

The European Medicines Agency also recognizes certain drugs as highly variable drug products (HVDP) and  is willing to accept a wider difference (i.e., a wider 90% confidence interval) in Cmax for bioequivalence evaluation. In its guidance "Guideline on the Investigation of Bioequivalence", section 4.1.10 specifically discussed the HVDP:

Other Readings:

-         Bioequivalence Approaches for Highly Variable Drugs and Drug Products by Haidar et al

-         Generic Drug Bioequivalence by Dr Aaron Sigler

-         An Example of How to Write the Statistical Section of a Bioequivalence Study Protocol for FDA Review

-         Global Bioequivalence Guidelines

-         Using SAS Proc Power to Perform Model-based Power Analysis for Clinical Pharmacology Studies

Recording the outcome for AE/SAE when multiple events contribute to Death

In clinical trials, death event may occur. According to CDISC CDASH standards, death should not be recorded as an adverse event (AE) or serious AE, but should be recorded as the outcome of the event. The condition that resulted in the death should be recorded as the AE/SAE.

In the case of multiple AE/SAEs contributing to the fatal (death) outcome, there seem to be two different ways in recording the AE outcome. There is no clear regulatory guideline in detail about this situation. The most closely related guideline may be from the ICH E2B where it states:

B.2.i.8 Outcome of reaction/event at the time of last observation

•  recovered/resolved
•  recovering/resolving
•  not recovered/not resolved
•  recovered/resolved with sequelae
•  fatal
•  unknown

User Guidance:
In case of irreversible congenital anomalies the choice, not recovered/not resolved should be used.
Fatal should be used when death is possibly related to the reaction/event. Considering the difficulty of deciding between "reaction/event caused death" and "reaction/event contributed significantly to death", both were grouped in a single category. Where the death is unrelated, according to both the reporter and the sender, to the reaction/event, death should not be selected here, but should be reported only under section B.1.9.

In practice, one way is to record multiple SAEs and record 'fatal' for each of these SAEs. The drawback of this approach is to have multiple SAEs with 'fatal' outcome for the same subject while subject can only die once.

Another way is to identify one SAE as the principal cause of the death. In this case, only will one SAE have the outcome recorded as Fatal. The subject can only die once so it makes sense to record ‘fatal’ as the outcome for the principal event  The question is that what the appropriate outcome should be for other SAEs that may also contribute to the death event. If there is an 'Ongoing' option in the list of AE outcomes, the appropriate choice may be 'ongoing' indicating that the SAE is ongoing during the time of death. If there is no choice of 'ongoing' in the AE outcome list (as specified in E2B above), the most appropriate choice seems to be "not recovered/not resolved" indicating that the SAE is still not resolved during the time of death. For AE stop time, the principal SAE with fatal outcome should be the time of death. For other SAEs contributing to death, the stop time may be appropriately recorded as 'ongoing' instead of recording the death time as the SAE stop time.

Serious Adverse Events (SAE) vs Severe Adverse Events

Professionals who are new to the clinical trial field are often confused with the concept of 'Serious Adverse Events (SAEs)' and 'Severe Adverse Events". Severity is not synonymous with seriousness. SAE is based on patient/event outcome or action criteria usually associated with events that pose a threat to a patient's life or functioning. Seriousness (not severity) serves as a guide for defining regulatory reporting obligations. In other words, the SAEs need to be filfill additional reporting process (reported to corporate global drug safety group or pharmacovigilence group, regulatory authorities, EC/IRBs). Severe AE is one class of AEs with severity (old term intensity) classified as 'severe'. Severe AE is one of the AE classifications – AE severity (other classifications are relationships/causality).

The FDA defines a serious adverse event (SAE) as one when the patient outcome is one of the following:

• Death
• Life-threatening
• Hospitalization (initial or prolonged)
• Disability - significant, persistent, or permanent change, impairment, damage or disruption in the patient's body function/structure, physical activities or quality of life.
• Congenital anomaly
• Requires intervention to prevent permanent impairment or damage

On the other hand, Severity of an AE is a point on an arbitrary scale of intensity of the adverse event in question. The terms "severe" and "serious" when applied to adverse events are technically very different. They are easily confused but can not be used interchangeably, require care in usage.

 

A headache is severe, if it causes intense pain. There are scales like "visual analog scale" that help us assess the severity. On the other hand, a headache is not usually serious (but may be in case of subarachnoid haemorrhage, subdural bleed, even a migraine may temporally fit criteria), unless it also satisfies the criteria for seriousness listed above. Similarly, a severe rash is not likely to be an SAE. However, mild chest pain may result in a day’s hospitalization and thus is an SAE.

 

Classifications of the AE sevirity often include the following:

• Mild: Awareness of signs or symptoms, but easily tolerated and are of minor irritant type causing no loss of time from normal activities. Symptoms do not require therapy or a medical evaluation; signs and symptoms are transient.
• Moderate: Events introduce a low level of inconvenience or concern to the participant and may interfere with daily activities, but are usually improved by simple therapeutic measures; moderate experiences may cause some interference with functioning
• Severe: Events interrupt the participant’s normal daily activities and generally require systemic drug therapy or other treatment; they are usually incapacitating

The guidelines for AE severity assessment is based on:

• Common Terminology Criteria for Adverse Events (CTCAE) CTC criteria, 4.03)
• It is also available as iPhone App. Here is the link.
• Guidance for Industry: Toxicity Grading Scale for Healthy Adult and Adolescent Volunteers Enrolled in Preventive Vaccine Clinical Trials

Other sources about this topic may be useful:

• Wikipedia
• HHS The Office for Human Research Protections (OHRP): Guidance on Reviewing and Reporting Unanticipated Problems Involving Risks to Subjects or Others and Adverse Events
• NIH National Institute of Aging: Adverse Event and Serious Adverse Event Guidelines

Reporting/recording the Serious Adverse Events (SAE) vs. Adverse Event (AE) Outcomes

In clinical trials, the serious adverse event reporting is critical to the safety assessment and to fulfill the regulatory requirements. The criteria for defining an SAE have been documented in many regulatory guidelines. However, in clinical trial implementation, the confusion could arise whether or not an event should be reported as an SAE or outcome of an SAE. Misinterpretation of the regulatory guidelines could cause in the inappropriate reporting of SAEs.

Acording to ICH E2A “CLINICAL SAFETY DATA MANAGEMENT: DEFINITIONS AND STANDARDS FOR EXPEDITED REPORTING”

                  A serious adverse event (experience) or reaction is any untoward medical occurrence that at any dose:                          * results in death,
                         * is life-threatening,
                             NOTE: The term "life-threatening" in the definition of "serious" refers to an event in which the patient was at 
                             risk of death at the time of the event; it does not refer to an event which hypothetically might have caused death
                             if it were more severe.
                         * requires inpatient hospitalisation or prolongation of existing hospitalisation,
                         * results in persistent or significant disability/incapacity, or
                         * is a congenital anomaly/birth defect.
FDA website has provided a little bit more detail descriptions on SAE

"What is a Serious Adverse Event?

An adverse event is any undesirable experience associated with the use of a medical product in a patient. The event is serious and should be reported to FDA when the patient outcome is:
Death

Report if you suspect that the death was an outcome of the adverse event, and include the date if known.
Life-threatening

Report if suspected that the patient was at substantial risk of dying at the time of the adverse event, or use or continued use of the device or other medical product might have resulted in the death of the patient.
Hospitalization (initial or prolonged)

Report if admission to the hospital or prolongation of hospitalization was a result of the adverse event.

Emergency room visits that do not result in admission to the hospital should be evaluated for one of the other serious outcomes (e.g., life-threatening; required intervention to prevent permanent impairment or damage; other serious medically important event).
Disability or Permanent Damage

Report if the adverse event resulted in a substantial disruption of a person's ability to conduct normal life functions, i.e., the adverse event resulted in a significant, persistent or permanent change, impairment, damage or disruption in the patient's body function/structure, physical activities and/or quality of life.
Congenital Anomaly/Birth Defect

Report if you suspect that exposure to a medical product prior to conception or during pregnancy may have resulted in an adverse outcome in the child.
Required Intervention to Prevent Permanent Impairment or Damage (Devices)

Report if you believe that medical or surgical intervention was necessary to preclude permanent impairment of a body function, or prevent permanent damage to a body structure, either situation suspected to be due to the use of a medical product.
Other Serious (Important Medical Events)

Report when the event does not fit the other outcomes, but the event may jeopardize the patient and may require medical or surgical intervention (treatment) to prevent one of the other outcomes. Examples include allergic brochospasm (a serious problem with breathing) requiring treatment in an emergency room, serious blood dyscrasias (blood disorders) or seizures/convulsions that do not result in hospitalization. The development of drug dependence or drug abuse would also be examples of important medical events."

The standard coding dictionary for adverse events is MedDRA (Medical Dictionary for Regulatory Activities). The guidance document MedDRA® TERM SELECTION: POINTS TO CONSIDER gives clear explanation how death and other patient outcomes should be handled.

3.2 – Death and Other Patient Outcomes

Death, disability, and hospitalization are considered outcomes in the context of safety reporting and not usually considered ARs/AEs. Outcomes are typically recorded in a separate manner (data field) from AR/AE information. A term for the outcome should be selected if it is the only information reported or provides significant clinical information.

(For reports of suicide and self-harm, see Section 3.3).

3.2.1 Death with ARs/AEs

Death is an outcome and not usually considered an AR/AE. If ARs/AEs are reported along with death, select terms for the ARs/AEs. Record the fatal outcome in an appropriate data field.


3.2.4 Other patient outcomes (non-fatal)

Hospitalization, disability and other patient outcomes are not generally considered ARs/AEs.

There are many other examples in terms of recording the outcome instead of AE/SAE. Adverse events represent the untoward medical event, not the intervention to treat that event. For example, if a subject has appendectomy, the AE is appendicitis not the surgical procedure; if a subject has an limb amputation, the AE is the cause for amputation (perhaps, the worsening of the ischemia in the peripheral artery) and limb amputation should be reported as the outcome of the AE/SAE; If a patient is hospitalized due to congestive heart failure, congestive heart failure should be reported SAE and hospitalization should be reported as an outcome for congestive heart failure.  

We should also be aware that not every hospitalization will have an associated SAE to be reported. Any AE leading to hospitalization or prolongation of hospitalization meets ONE of the followings should not be reported as SAE.

• A hospitalization admission is pre-planned (ie, elective or scheduled surgery arranged prior to the start of the study). European Commission’s guidelines on medical devices “CLINICAL INVESTIGATIONS: SERIOUS ADVERSE EVENT REPORTING “ indicated that a planned hospitalization for pre-existing condition, or a procedure required by the Clinical Investigation Plan, without a serious deterioration in health, is not considered to be a serious adverse event.
• A hospitalization admission is clearly not associated with an AE (eg, social hospitalization for purposes of respite care). If a patient wants to be stay in the hospital during the drug treatment because of the fear that something bad could happen, this should not be reported as SAE just because of the hospital stay if nothing else happens

According to these definitions, the events with outcome of death, hospitalization, disability or permanent damage, congenital anomaly/birth defect, … should be reported as SAE while death, hospitalization, disability or permanent damage, congenital anomaly/birth defect…should be reported as the outcome of the corresponding SAE. To be crystal clear, the Death, Hospitalization should not be reported as SAE and the causes leading to the death and hospitalization should be reported as SAE.

Studentized residual for detecting outliers

Last time, I discussed the outliers and a simple approach of Dixon’s Q test for detecting a single outlier. When there are multiple outliers, we can detect the outliers using the standard deviation (for data that is normal distributed) or using percentiles (for the skewed data). A box plot may be useful to visually check the data for potential outliers.

In regression setting, there are several approaches in detecting the outliers. One of the approaches is to utilize the ‘standardized residual’ or ‘studentized resitual’. In linear regression, an outlier is an observation with large residual. In other words, it is an observation whose dependent-variable value is unusual given its values on the predictor variables.

The studentized residual is the quotient resulting from division of a residual by an estimate of its standard deviation. Just like the standard deviation, the studentized residual is very useful in detecting the outliers. For values outside the 3, 4, or 5 times standard deviation, we may have reasonable doubt that the values are outliers. In regression setting, observed values outside 3, 4, or 5 times the studentized residual are the targets for outliers.

In SAS, two regression procedures can be easily utilized to compute the studendized residual for detecting outliers. PROC REG and PROC GLM. The studentized residual is labelled as RSTUDENT in Output statement. Other regression procedure (such as PROC MIXED) also compute studentized residual as part of Influence test. 

               output out=newdata rstudent=xxx;
Further readings:

• Regression with SAS - Regression Diagnostics

• SAS version 9.3 PROC REG

• SAS version 9.3 PROC GLM

Outliers in clinical trial, Dixon's Q test for a single outlier

In clinical trials, we deal with the outlier issue differently from other fields. During the clinical trial, for the suspected ‘outliers’, every effort should be taken to query the investigator sites, to repeat measures, or to re-test the samples in order to get the correct information. Typically those suspected ‘outliers’ can be clarified during the data cleaning process. It is just not very common to throw away the data (even it is suspected to be ‘outlier’) in clinical trials. In one of pharmacokinetics studies, I did have to deal with the suspected outliers (we used the term ‘exceptional value’ instead of ‘outliers’). After the sample re-test, we still had one value very high. Instead of throwing away this exceptional value, we had to perform the analysis with and without this exceptional value. 

In one of the presentations by a FDA officer, the term ‘outliers’ vs anomalous are used.

• Outlier subjects may be “real” results and are therefore very valuable in making a correct BE conclusion
• Anomalous results are data that are not correct due to some flaw in study conduct or analysis

In many situations, it is very difficult to know for sure whether or not an exceptional value is a outlier or an anomalous result.

In ICH E9 "Statistical Principles for Clinical Trials", the handling of outliers was discussed in the section of "missing values and outliers".      

5.3 Missing Values and Outliers

Missing values represent a potential source of bias in a clinical trial. Hence, every effort should be undertaken to fulfil all the requirements of the protocol concerning the collection and management of data. In reality, however, there will almost always be some missing data. A trial may be regarded as valid, nonetheless, provided the methods of dealing with missing values are sensible, and particularly if those methods are pre-defined in the protocol. Definition of methods may be refined by updating this aspect in the statistical analysis plan during the blind review. Unfortunately, no universally applicable methods of handling missing values can be recommended. An investigation should be made concerning the sensitivity of the results of analysis to the method of handling missing values, especially if the number of missing values is substantial.

A similar approach should be adopted to exploring the influence of outliers, the statistical definition of which is, to some extent, arbitrary. Clear identification of a particular value as an outlier is most convincing when justified medically as well as statistically, and the medical context will then often define the appropriate action. Any outlier procedure set out in the protocol or the statistical analysis plan should be such as not to favour any treatment group a priori. Once again, this aspect of the analysis can be usefully updated during blind review. If no procedure for dealing with outliers was foreseen in the trial protocol, one analysis with the actual values and at least one other analysis eliminating or reducing the outlier effect should be performed and differences between their results discussed.

I was recently asked for help to test an outlier for the data from a lab experiment (not a clinical trial).

The titer for the same sample was measured for 20 times.  The titer is 25 for 7 times, 125 for 12 times. However, for one time, the title is 625.  Is there any way to test (statistically) whether the titer of 625 is an outlier?

Titer	25	125	625
N	7	12	1

There is a simple test for outlier called Dixon's Q-test. Dixon’s Q-test calculates the Q value that is the ratio of the Gap (the difference between the extreme value and the immediately adjacent value) and the Range (the difference between the extreme value and the maximal or minimal value)

In the case above, the titer value needs to be log-transferred first, therefore, with Log10 data transfer, data will be listed as the following (in order):

1.39794 1.39794 1.39794 1.39794 1.39794 1.39794 1.39794 2.09691 2.09691 2.09691
2.09691 2.09691 2.09691 2.09691 2.09691 2.09691 2.09691 2.09691 2.09691 2.79588

The gap = 2.79588 - 2.09691 = 0.69897
The range = 2.79588 - 1.39794 = 1.39794
The Q value = 0.69897 / 1.39794 = 0.5

The Q value will then be compared with the critical value. The critical value can be found at difference web sources or from the original paper. The critical value for N=(7+12+1) = 20 is 0.342.
Since Q value is larger than 0.342, we can reject 2.79588 and conclude that the original value 625 (log-transferred value of 2.79588) is a outlier.

If we use a Log5 data transfer, the calculation will be easier and conclusion is the same. 

This approach can only be used for detecting a single outlier. If there are more than one values in 625 titer group, Dixon's Q test will not be an appropriate approach.

Typically, identifying of the outliers is against a continuous variable (ie, the data is continuous). The data above contains many ties (due to the design). Therefore, the results from the Dixon’s Q-test needs to be interpreted in caution. The determination of the outliers should always be based on the understanding of the experimental data.

For further reading about the outlier issues: 

• Dealing with 'Outliers': Maintain Your Data's Integrity 
• Outlier detection

Story of Xigris (Protein C) for Sepsis

Xigris, also called Drotrecogin Alfa (Activated) or Protein C, was the only approved drug for severe sepsis indication and it was withdrawn from the market last week (Oct 25, 2011). In a recently completed clinical trial (PROWESS-SHOCK trial), Xigris failed to show a survival benefit. Due to the early controversies over Xigris’s approval and the continuous debate on Xigris’s risk benefit, PROWESS-SHOCK trial has been under watch since its start. The study design, statistical analysis plan, and unblinding plan have all been published way before the completion of the trial.

A decade ago, prior to the approval of Xigris for sepsis indication, the risk-benefit had been debated quite a bit. Xigris was know to be linked to the increased risk of serious bleeding in patients. there was "controversy surrounds both the drug study itself and the FDA approval," wrote NEJM editor-at-large Richard P. Wenzel, MD in 2002. FDA held the anti-infective advisory committee meeting for Xigris in treating sepsis. The FDA approved the drug despite the advisory committee's split vote (10 to 10) due to concerns about the validity of the claimed efficacy and safety findings on the basis of a single trial. At that time, Xigris was approved based on a single pivotal trial (PROWESS trial) that was also stopped early for efficacy. At that time, the FDA reviewers certainly believed that Xigris was beneficial and could save a lot of lives.

PROWESS trial has been the model for other clinical trials in Sepsis even though the PROWESS trial itself has been criticized for changes in the protocol during the trial. According to the NEJM article by H. Shaw Warren, MD, from Massachusetts General Hospital in Boston, and fellow consultants to the FDA, the study protocol changed during the PROWESS trial, shifting the study population composition toward patients with less severe underlying disease and more acute infectious illnesses. Other changes included use of a different placebo and elimination of protein C deficiency status as a primary variable. Around the same time, Lilly began producing the drug using a new master cell bank. Cumulative mortality curves suggest an improvement in protective efficacy of Xigris after these changes were made.

Subsequent trials have now shown that Xigris has no benefit and has unfavorable risk-benefit profiles. The ADDRESS trial (published in 2005) showed the absence of a beneficial treatment effect, coupled with an increased incidence of serious bleeding complications. The result indicates that Xigris should not be used in patients with severe sepsis who are at low risk for death, such as those with single-organ failure or an APACHE II score less than 25. Now the PROWESS-SHOCK trial further confirmed that the risk of bleeding outweigh the benefit in reducing the mortality – unfortunately it is a decade after Xigris has been on the market.

The market practice for Xigris has also been criticized. Several years ago, there were a lot of talks about Lilly’s influence on a committee in defining the sepsis treatment guidelines which was in favor of using Xigris.

Retrospectively, we can have something to learn from the Xigris story: 1) a single pivotal trial may be insufficient in confirming the treatment benefit; 2) change the protocol during the trial could have bias to the trial results; 3) stop a trial for efficacy may be risky.

New drugs for life-threatening disease such as sepsis are desperately needed, however, to demonstrate the benefit of any drug in the complicated sepsis treatment is a challenging task. The diversities in sepsis treatment in various institutes make the clinical trials in sepsis very difficult and the sample size for sepsis trials need to be sufficiently large to show the benefit.

A medical joke to share

Not sure where the origin is. It is circulated quite a bit. 

Best friends graduated from medical school at the same time and decided that, in spite of two different specialties, they would open a practice together to share office space and personnel.

Dr. Smith was the psychiatrist and Dr. Jones was the proctologist; they put up a sign reading: "Dr. Smith and Dr. Jones: Hysterias and Posteriors". The town council was livid and insisted they change it.

So, the docs changed it to read: "Schizoids and Hemorrhoids". This was also not acceptable, so they again changed the sign. "Catatonics and High Colonics" - No go.

Next, they tried "Manic Depressives and Anal Retentives" - thumbs down again. Then came "Minds and Behinds" - still no good. Another attempt resulted in "Lost Souls and Butt Holes" - unacceptable again! So they tried "Analysis and Anal Cysts" - not a chance. "Nuts and Butts" - no way. "Freaks and Cheeks" - still no good. "Loons and Moons" - forget it.

Almost at their wit's end, the docs finally came up with: "Dr. Smith and Dr. Jones - Specializing in Odds and Ends". Everyone loved it.

Will Electronic Data Capture be always better than Paper-CRF?

The traditional way to do the clinical trial data management is to use the paper based case report forms (CRFs). The blank paper CRFs are distributed to the investigator sites. The investigator or study coordinator fills out the CRFs. CRFs will then be monitored and collected from the investigator sites. CRFs will subsequently be handled by a centralized group - data management group where the activities include the clinical database building, data entry, data cleaning, data clarification,...

The industry trend has been gradually moving away from the paper-based CRFs and moving toward to the electronic data capture (EDC). In EDC world, the database was built prior to the study start (significant longer leading time prior to the study study is needed) . The data will be directly entered into the database by the investigator site (investigator or study coordinator). EDC has been touted by many vendors as the preferred way for conducting clinical trials: getting the data fast, saving timeline, saving cost, minimizing data transcription errors... While this is generally true, it is not universal.


In some situations, the trial using the traditional paper-based CRFs is a better way than EDC. For example, in a clinical trial for a rare disease, there are many investigator sites and each site may only enroll very few subjects or not enroll any subject. The EDC will not be an efficient way in data collection. Many site staff will be trained on EDC and never have chance to enroll any patient into the study and never have a chance to use EDC. When a site finally has a chance to enroll a subject, the initial training on using EDC may be a distant memory.

The EDC trial is not always cheap. With EDC trial, significant cost could be spent on the EDC system hosting and EDC system help desk support. Imagining a slow enrollment trial running for 7-8 years, the cost for hosting EDC system and providing the help desk support will be too much comparing to a paper-based study.

While EDC is a trend, the adoption of EDC is not universal. In some situations, the traditional paper CRFs may be better. 

Confidence Interval for Difference in Two Proportions

In many clinical trials, the outcome is binomial and a 2 x 2 table can be constructed. The analysis can be based on the difference in two proportions (treatment group vs. control group). SAS Proc Freq can be used to obtain the difference between the proportions and the asymptotic confidence interval can be calculated for the difference between two proportions. The formula is (p1-p2) +/- Z(alpha/2)*sqrt((p1*q1/n1)+p2*q2/n2)).
However, the asymptotic confidence interval produced by PROC FREQ requires a somewhat large sample size (say cell counts of at least 12) - this is the case at least for SAS version up to 9.2. For moderately small sample size, it is better to use the formula provided in Fleiss (1981, page 29) Stokes (2000, page 29-30) where the confidence interval is adjusted by 0.5*(1/n1 + 1/n2) - therefore a little wider.  The confidence interval directly from SAS Proc FREQ is a little narrower than those using the formula. In practice, the statistician needs to make the choice which one to use in calculating the confidence interval for difference in proportions depending on the sample size situation.

Fleiss, JL (1981) Statistical Methods for Rates and Proportions. New York: John Wiley & Sons, Inc.
Stokes, Davis, and Kock (2000) Categorical Data Analysis using the SAS System, 2nd edition
FDA Draft Guidance on Tazarotene detailed the calculation of the 90% confidence interval for establishing the bioequivalence for the clinical endpoint using the second approach mentioned above.
 
The example from Stocks book can be implemented in SAS using the following SAS codes:

data respire2;
  input treat $ outcome $ count @@;
  datalines;
test    f 40
test    u 20
placebo f 16
placebo u 48
;

*** the confidence interval directly from SAS PROC FREQ;
proc freq order=data;
  weight count;
  tables treat*outcome / riskdiff;
run;

*** the confidence interval calculated from the formula (See section 2.4 Difference in Proportions
     in Stokes et al 'Categorical Data Analysis Using the SAS System' 2nd edition;
proc freq data=respire2 order=data;
    weight count;
    tables treat/noprint out=tots (drop=percent rename=(count=bign));
  run;
 
proc freq data=respire2;
    weight count;
    tables treat*outcome/noprint out=outcome (drop=percent);
    run;
 
proc sort data=tots;
  by treat;
  run;
 
proc sort data=outcome;
    by treat;
run;
 
data prop;
    merge outcome tots;
    by treat;
    if treat='test' then p1=count/bign;
    if treat='placebo' then p2=count/bign;
run;

data prop1(rename=(count=count1 bign=bign1)) prop2(rename=(count=count2 bign=bign2));
     set prop;
     if treat='test' then output prop1 ;
     if treat='placebo' then output prop2;
run;

data proportion;
  merge prop1(drop= p2 treat) prop2(drop = p1 treat);
run;

***Calculate the difference in proportions, SE, and 95% confidence interval using formula by Fleiss;
data cal;
  set proportion;
    variance=(p1*(1-p1)/(bign1)) + (p2*(1-p2)/(bign2));
    diff=(p1-p2);
    lower=(diff - ((1.96*(sqrt(variance)) + .5*(1/bign1 + 1/bign2))));
    upper=(diff + ((1.96*(sqrt(variance)) + .5*(1/bign1 + 1/bign2))));
    se=(sqrt(variance));

run;

proc print;
  format p1 p2 variance diff lower upper se 5.3;
run;

Is it time to change the clinical monitoring practice in clinical trials?

In industry, the current monitoring practice relies on ‘on-site monitoring’ and 100% source data verification (on all data fields). This process is very costly and is one of the main reasons that the clinical trials now become so expensive. This process is really the most conservative interpretation of ICH E-6 guidance on
Guideline for Good Clinical Practice and the 1988 FDA’s “Guidance for the Monitoring of Clinical Investigations”. These guidance only require “the sponsor should ensure that the trials are adequately monitored” and leave the door open in terms of the frequency of the monitoring and the approaches of the clinical monitoring. In industry, the conduct of the clinical trials are highly regulated. Sponsors are usually take the most conservative approaches no matter how costly these approaches are.

Will ‘on-site monitoring’ be really effective? Will 100% source data verification really be needed? Should we identify the new ways to conduct the cost-effective clinical monitoring?

Last month, FDA withdrew its 1988 guidance on “Guidance for the Monitoring of Clinical Investigations” and issued its draft guidance “Oversight of Clinical Investigations - A Risk-based Approach to Monitoring”. The newly issued guidance suggested it is acceptable to use alternative approaches (such as remote monitoring, centralized monitoring, risk-based monitoring). The guidance also suggested that the source data verification should be focused on critical fields (key efficacy and safety variables) and less than 100% source data verification on less important fields may be acceptable. The guidance gives a clear signal that the Sponsors are encouraged to explore the cost-effective ways to conduct the clinical monitoring instead of solely relying on the on-site monitoring.

If this guidance gets implemented, we may expect the increasing role of statisticians in clinical monitoring, especially the centralized monitoring. Currently, statisticians will identify the issues in the late stage of the clinical trials when statisticians or statistical programmers start to perform the data analyses. The new guidance says:

“…notably, the advancement in EDC systems enabling centralized access to both trial and source data and the growing appreciation of the ability of statistical assessments to identify clinical sites that require additional training and/or monitoring.”

“Centralized monitoring is a remote evaluation carried out by sponsor personnel or representatives (e.g., data management personnel, statisticians, or clinical monitors) at a location other than the site(s) at which the clinical investigation is being conducted.”

Odds ratio and risk ratio in clinical trials #2

In my previous article, I discussed the odds ratio and risk ratio (or relative risk ratio). In clinical trials with binary outcome, both odds ratio and relative risk ratio are used. Since the clinical trials are similar to the cohort studies in epidemiology field, it seems to be more reasonable to use relative risk ratio in clinical trials. However, the odds ratio may be more commonly used in practice. This may be due to the fact that the odds ratio can be easily modeled using logistic regression. This could also be due to the fact that the odds ratio is typically larger than relative risk ratio that may be desired by the researcher.

For a non-inferiority or equivalence trials with binary outcome, one may desire to have a smaller standard error, therefore a narrower confidence interval – in this case, the relative risk ratio may be better than odds ratio.

Wikipedia gives an excellent comparison between relative risk ratio and odds ratio.

In an article “How can I estimate relative risk in SAS using proc genmod for common outcomes in cohort studies?”, the calculation of odds ratio, relative risk ratio, and their confidence intervals are illustrated.

Using SAS Proc Genmod, both odds ratio, relative risk ratio, and their confidence intervals can be easily calculated:

For odds ratio:

Proc genmod data = xxx descending;
    class treatment;
    model  outcomevariable = treatment
                                                  / dist = binomial link = logit;
    estimate 'Beta' treatment 1 -1/ exp;
run;

Here, the “link=logit” can be omitted since the logit link function is default when distribution is binomial.

For relative risk ratio, 
proc genmod data = xxx descending;
    class treatment;
    model outcome = treatment
                              / dist = binomial link = log;
    estimate 'Beta' treatment 1 -1/ exp;
run;

Here, the “link=log” can NOT be omitted since the log link function is NOT default when distribution is binomial.

Relative risk ratio can also be estimated using poisson regression especially when the event ratio is small.
 
Proc genmod data = eyestudy;
    class id;
    model outcome = treatment
                          / dist = poisson link = log;
    repeated subject = id/ type = unstr;
    estimate 'Beta' treatment 1 -1/ exp;
run;

Here, the “link=log” can be omitted since the log link function is default when distribution is poisson.

There are several advantages of using Proc Genmod to calculate the odds ratio and risk ratio. Adjusted odds ratio and adjusted relative risk ratio can be easily calculated when there are continuous or categorical covariates. The model can be easily modified to fit the longitudinal data.

Proc Logistic can be used for calculating the odds ratio (and the confidence interval) and can adjust for continuous or categorical covariates. However, Proc Logistic can not be used for calculating the relative risk ratio.

proc logistic;
     model outcome = treatment;
run;

Proc FREQ can be used for calculating the odds ratio and relative risk ratio (and asymptotic confidence interval) using /cmh option. For adjusted odds ratio or risk ratio, only the categorical covariate can be used. 

proc freq order=data;
    tables covariate*treatment*response / CMH;
run;

In the output, the odds ratio will be explicitly indicated while relative risk ratio will be labeled as “col1 risk” or “col2 risk”.

There is no regulatory guidance forcing the use of odds ratio or relative risk ratio. However, in clinical trials, if we compare the ratio of two proportions (eg the proportion of success in treated group vs. the proportion of success in control group), relative risk ratio seems to be better. Relative risk ratio resemble the hazard ratio in may aspects.

In FDA’s guidance “Diabetes Mellitus — Evaluating Cardiovascular Risk in New Antidiabetic Therapies to
Treat Type 2 Diabetes” The calculation of risk ratio is suggested. The guidance indicated
"Sponsors should compare the incidence of important cardiovascular events occurring with the investigational agent to the incidence of the same types of events occurring with the control group to show that the upper bound of the two-sided 95 percent confidence interval for the estimated risk ratio is less than 1.8. This can be accomplished in several ways. The integrated analysis (meta-analysis) of the phase 2 and phase 3 clinical trials described above can be used. Or, if the data from all the studies that are part of the meta-analysis will not by itself be able to show that the upper bound of the two-sided 95 percent confidence interval for the estimated risk ratio is less than 1.8, then an additional single, large safety trial should be conducted that alone, or added to other trials, would be able to satisfy this upper bound before NDA/BLA submission. Regardless of the method used, sponsors should consider the entire range of possible increased risk consistent with the confidence interval and the point estimate of the risk increase. For example, it would not be reassuring to find a point estimate of 1.5 (a nominally significant increase) even if the 95 percent upper bound was less than 1.8.”

In a presentation by Dr Bob O’Neill “Non-Inferiority Clinical Trials Some key statistical issues and concepts” he suggested that Log (Hazard ratio) or Log(relative risk) is preferred when determining the non-inferiority margin

In statistical review for Maxipime (cefepime hydrochloride) NDA, the risk ratio and 95% confidence interval are used. In an article “Relative risks of reported serious injury and death associated with hemostasis devices by gender”, the risk ratio were reported.

However, there are also many cases of using odds ratio instead of risk ratio in clinical trials. Some examples are:

• Antiepileptic drugs and suicidality NDA review
• Tenecteplase, recombinant (TNK-tPA) for Treatment of acute myocardial infarction (AMI) BLA
• In vitro polymerase chain reaction (PCR) based assay for HBV viral load detection PMA review

In summary, while both odds ratio and risk ratio can be used in clinical trials, risk ratio should be given the adequate emphasis in comparing the ratio of two proportions (between two treatment groups). In non-inferiority clinical trials, the risk ratio and its confidence interval are preferred.

FDA's new guidance on device approval process and device clinical trial

For a long time, device makers have been complaining about the FDA's device approval process. I personally heard a lot of talks that the device approval process is easier than the approval of the drug and the biological product. The requirements for clinical trials in device approval has lower standard comparing to  clinical trials in drugs and biological products. FDA device division (CDRH) has the loose criteria for the product approval. In response to the critics, FDA now releases two new draft guidance on August 15, 2011 and is seeking to educate industry on device approval.

The first guidance “Factors to Consider when Making Benefit-Risk Determinations in Medical Device Premarket Review” explains the agency's approval process for diagnostic and therapeutic devices, specifically:

• How the agency weighs the benefits and risks of a device
• How the agency assess the seriousness of a disease or condition
• How many people would use the device if approved
• The availability of other devices approved to treat the same condition

It is interesting enough that in examples used in this guidance, several clinical trials were mentioned as flawed or with unreliable data, however, FDA would approve the device anyway. This is just another reflection that CDRH indeed has lower data quality standard comparing to CDER for drugs and CBER for biological products.

In the second draft document “Design Considerations for Pivotal Clinical Investigations for Medical Devices”, the FDA laid out its expectations for clinical trials for medical devices. The agency stated that it looks for a study to provide reasonable assurance that the device is safe and effective.

Back in July 2011, FDA issued a guidance on “in Vitro Companion Diagnostic Devices” where the In Vitro Companion diagnostic device (IVD companion diagnostic device) is defined as an in vitro diagnostic device that provides information that is essential for the safe and effective use of a corresponding therapeutic product. The guidance intended to accomplish the following:

• Define in vitro companion diagnostic device
• Explain the need for FDA oversight of IVD companion diagnostic devices
• Clarify that, in most circumstances, if use of an IVD companion diagnostic device is essential for the safe and effective use of a therapeutic product, the IVD companion diagnostic device and therapeutic product should be approved or cleared contemporaneously by FDA for the use indicated in the therapeutic product labeling
• Provide guidance for industry and FDA staff on possible premarket regulatory pathways and FDA’s regulatory enforcement policy
• Describe certain statutory and regulatory approval requirements relevant to therapeutic product labeling that stipulates concomitant use of an IVD companion diagnostic device to ensure safety and effectiveness of the therapeutic product

No matter what the device is, if clinical trials are required, the statistical analyses can be based on the FDA guidance “Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests” that was issued in 2007.

In Device field, not all approval requires clincial trials. If clinical trials are required, they do not have to be interventional. If clinical trials are interventional, they do not have to be randomized, controlled. If clinical trials are randomized, controled, they do not have to be double-blinded. Some clinical trials in device field may be using the samples (for example blood samples) from patients with no intervention performed. The blood samples could be historical retains or prospectively collected. The use of the blood samples for device clinical trial still needs the informed consent from the patient.

Equipoise and Lack of Equipoise in Randomized Clinical Trials

According to Merriam-Webster dictionary, the word "Equipoise" means a state of equilibrium. In clinical trial, the concept of 'clinical equipoise' means that there is genuine uncertainty over whether a treatment will be beneficial. In other words, in randomized controlled clinical trials, there should be substantial uncertainty or there is no clear evidence that one treatment arm is particularlly better or worse. Clinical equipoise provides the ethical basis for medical research involving patients assigned to different treatment arms of a clinical trial - it is unethical to assign a subject to an inferior arm if the lack of equipoise exists and if there is substantial evidence that one treatment is better or worse than another treatment.

In the real word, when we plan a randomized, controlled clinical trial, 'lack of equipoise' may often exist. This is especially true in late stage clinical trials. In late stage clinical trials, there typically be some evidences about the benefit and treatment effect of the experimental drug from early phase I or phase II clinical trials. Our sample size calculation is based on the assumed treatment effect of the experimental drug.


We recently published a paper in the Journal of Neurology "Challenges of clinical trial design when there is lack of clinical equipoise: use of a response-conditional crossover design" where we discussed a situation of 'lack of equipoise' and the use of a response-conditional crossover design to ease the concern about the lack of equipoise in clinical trial design. Several small trials had suggested that IVIg is beneficial in treating the disease CIDP - lack of equipoise. However, in the absence of an approved treatment for this indication,  gaining regulatory approval for the use of IVIg in this indication required the conduct of large-scale, placebo-controlled confirmatory trials. Using the response-conditional crossover design, we eased the concern about subjects exposed to the 'perceived' inferior arm (Placebo in this case). The results indicated that we minimized subject's exposure to the inferior treatment arm.

Interpretation of 'clinical equipoise' may be different among clinicians, investigators, patients, clinical trial sponsors, and regulatory agencies. Evidences of treatment effects from small-scale clinical trials may be thought as real evidence for clinician and patients, but not for regulatory agencies.

When we bring the overall benefit/risk into the picture for assessing the 'clinical equipoise', it may be difficult to determine whether or not a clinical equipoise exist or not. A new treatment may have been demonstrated beneficial in efficacy, but with great uncertainty in safety.