Multiplicity Adjustments: Gatekeeping, fixed-sequence, and fallback procedures

The multiplicity issue has evolved in last several years and a lot of new procedures have been proposed mainly in handing the issues encountered in the clinical trial and the drug development area.

 

EMA/CHMP recently released a "Concept paper on the need for a guideline on multiplicity issues in clinical trials" for seeking the public comments. In the introduction of this concept paper, it mentioned some new procedures

“The guideline is not to give advice on technical questions related to a new methodology. However, the increasing complexity of hypothesis frameworks and methods used may result in new issues and pose questions on general principles that haven’t been considered before. These include consistency problems, the construction of simultaneous confidence intervals and the usefulness of newly developed methods e.g. gatekeeping and fallback procedures as well as graphical solutions in the regulatory context.”

It is necessary to differentiate the differences among three new procedures for multiplicity adjustment:

• Gatekeeping procedure
• Fixed sequence procedure
• Fallback procedure

All these three procedures are mainly designed to deal with the issue with multiple endpoints (including the primary endpoints and the secondary endpoints).

Gatekeeping Procedure is used in the situation when there are multiple endpoints and these multiple endpoints are grouped into different families. For example, a clinical trial will typically have one or more primary endpoints (family for primary endpoints) and have multiple secondary endpoints (family for secondary endpoints). If there are many secondary endpoints, the secondary endpoints can be further divided into multiple secondary different families. With gatekeeping procedure, the families are tested in a sequential manner and the tests for subsequent families will be performed only if the tests for the previous family is significant. In other words, the families of hypotheses examined earlier serve as gatekeepers. While the term ‘gatekeeping procedure’ may not used, this approach has been implemented in many clinical trials, especially in the regulatory setting. It is very typical that the secondary endpoints will only be tested only if the primary endpoint is tested significantly. In this way, the alpha-level for primary efficacy endpoints will be tested at alpha=0.05 level and not be compromised due to the consideration of the secondary endpoints.

• The website http://multxpert.com/wiki/Gatekeeping_Procedures maintained by Alex Dmitrienko et al contains a lot of useful information about the gatekeeping procedures. 
•  A slide presentation by Branching tests in clinical trials with multiple objectives is helpful in understanding the gatekeeping procedure.
•  The gatekeeping strategy is used in NDA 22-554 GI Drugs Advisory Committee Meeting NDA 22-554 Xifaxan (Rifaximin) where the secondary endpoints were grouped as “Key Secondary Endpoints” and “Other Secondary Endpoints”. Key secondary endpoints are those designated as most clinically important with pre-specified order for their analysis. P-values and confidence intervals for all other analyses are presented with NO adjustment for multiplicity. Nominal p-values and confidence intervals are consequently exploratory and cannot be used as a basis for efficacy claims in the product label if approved.

Fixed Sequence Procedure is a stepwise multiple testing procedure that is constructed using a pre-specified sequence of hypotheses. When there are multiple endpoints, these endpoints can be ordered according to their importance. All tests will be performed at the 0.05 level following the pre-specified order. Once one hypothesis is tested not significantly, all subsequent tests will not be performed.  The advantage and disadvantage of this testing procedure are obvious: power will be maximized as long as previous hypotheses are rejected, but minimized if a previous hypothesis is not rejected.  Another drawback for this procedure is that the ordering of multiple hypotheses based on the clinical importance is subjective in nature.

• Fixed Sequence Procedure could be used under the umbrella of gatekeeping procedure for one specific family. In previous example of Xifaxan NDA, the gatekeeping procedure is used in general with considering of both primary and secondary endpoints, however the fixed sequence procedure is used in testing the key secondary endpoints.
• While it is not explicitly stated, fixed sequence procedure is actually mentioned in the EMA’s   “Points to consider on multiplicity issues in clinical trials” that is issued in 2002. In the case of “two or more primary variables ranked according to clinical relevance, no formal adjustment is necessary. However, no confirmatory claims can be based on variables that have a rank lower than or equal to that variable whose null hypothesis was the first that could not be rejected.”
• A   g
  ood reference paper is “A flexible fixed-sequence testing method for hierarchically ordered correlated multiple endpoints in clinical trials” by Huque and Alosh

The Fallback Procedure is concepturely similar to a fixed sequence test, in which hypotheses are tested in an a priori order at the full alpha level. The difference of the fallback procedure from the fixed sequence test is that the full alpha of 0.05 is split for endpoints in a pre-specified order (based on the clinical relevance) and the hypotheses in late order can still be tested (but with different alpha levels) if the previous hypothesis is not rejected. To explain how the fallback procedure differs from the fixed-sequence procedure, we can use an example from a paper “the Fallback procedure for evaluating a single family of hypotheses” by Wiens and Dmitrienko. There are five endpoints with actual p-values of 0.010, 0.060, 0.0002, 0.0004, and 0.0268. With the fixed-sequence procedure, the endpoints #3, #4, and #5 will never be tested since the endpoint #2 is not significant. However, with the fallback procedure, the endpoints #3, #4, and #5 can still be tested (just at different alpha levels).

In order	With Fixed-sequence procedure	With fallback procedure*
Endpoint #1	0.010 comparing to alpha=0.05	0.010 comparing to alpha=0.04
Endpoint #2	0.06 comparing to alpha=0.05	0.060 comparing to alpha=0.04 + 0.005 and result is not significant
Endpoint #3	Not tested due to the endpoint #2 is not significant	0.0002 comparing to alpha=0.002 and result is significant
Endpoint #4	Not tested	0.0004 comparing to alpha=0.002 + 0.002 and result is significant
Endpoint #5	Not tested	0.0268 comparing to 0.002+0.002+0.001 and result is not significant

* five endpoints are given weights for their importance and alpha levels are assigned as 0.04, 0.005, 0.002, 0.002, and 0.001 (corresponding to 0.80, 0.10, 0.04, 0.04, and 0.02 of total alpha of 0.05)

 

Additional References:

•  Free online conference on multiplicity issues in confirmatory clinical trials
• Tree-structured gatekeeping tests in clinical trials

SAS tips: converting the data between SAS data sets and Excel

Converting SAS data sets to Excel Book

 In clinical trials, the database may be stored in SAS data set format. Sometimes, there is a need to convert multiple SAS data sets into an excel book. The following program can be easily modified to serve this purpose. With the small macro using Proc Export, SAS data sets can be converted into Excel book with multiple tabs (each tab is corresponding to a SAS data set). 

libname aa "c:\Data\CRF Data\Final Data\";

%macro export(dst=);
PROC EXPORT DATA= aa.&dst

            OUTFILE= "c:\Data\CRF Data\Final Data\excel\ExcelData.xls"

            LABEL DBMS=xls REPLACE;  

   SHEET="&dst";
RUN;
%mend;

%export(dst=IE);
%export(dst=DM);
%export(dst=MH);
%export(dst=VS);
%export(dst=PE);
%export(dst=CLAB);
%export(dst=PREG);
%export(dst=DRUG);
%export(dst=AE);
%export(dst=CM);
%export(dst=COM);

In the macro above, the keyword 'LABEL' is important. With 'LABEL', the SAS variable label will be used as the column header. Without 'LABEL', the SAS variable name will be used as the column header. The keyword 'LABEL' must be placed before the keyword 'DBMS' in order to be effective.
DBMS=xls indicates that the output data file will be an excel book (no version number is needed). other options for DBMS are csv,  dlm, tab, jmp. Check SAS manual for detail. 

If you run into an error due to the Excel version issue, you may try to use xlsx engine.

%macro export(dst=);
PROC EXPORT DATA= aa.&dst

            OUTFILE= "c:\Data\CRF Data\Final Data\excel\ExcelData.xlsx"

            LABEL DBMS=xlsx REPLACE; 

   SHEET="&dst";
RUN;
%mend;

Converting Excel Book to SAS data sets

The opposite way is to convert the Excel book (with multiple table) into different SAS data sets. The program below can be modified to fulfill this task.

libname aa "c:\temp\";

%macro import(dst=);
PROC IMPORT DATAFILE= "C:\Dengc\ExcelData.xls"  OUT= aa.&dst
            DBMS=xls REPLACE;
     SHEET="&dst";
     GETNAMES=YES;
RUN;
%mend;

%import(dst=primary);
%import(dst=secondary);
%import(dst= tertiary);

In the macro above, GETNAMES=YES indicates that the first row from excel spreadsheet will be used as the SAS variable name.


Other References:

• How do I read/write Excel files in SAS?
• SAS Proc Import
• SAS Proc Export

Switching from non-inferiority to superiority - is multiplicity adjustment needed?

For a non-inferiority trial, after the non-inferiority is shown, one will typically try to show the non-inferiority. People may argue that the multiplicity adjustment arise in this situation. This can be seen in a presentation of "Branching tests in clinical trials with multiple objectives" by Alex Dmitrienko and Brian Wiens. In their presentation, the multiplicity adjustment is considered for switching from non-inferiority test to superiority test as part of the gatekeeping methods. 

However, the regulatory guidelines clearly stated that no multiplicity adjustment is needed when interpreting a non-inferiority trial as a superiority trial. In EMA's guidance "Point to consider on switching between superiority and non-inferiority", the following statement is stated:

"if the 95% confidence interval for the treatment effect no only lies entirely above -delta but also above zero then there is evidence of superiority in terms of statistical significance at the 5% level (p<0.05). In this case it is acceptable to calculate the p-value associated with a test of superiority and to evaluate whether this is sufficiently small to reject convincingly the hypothesis of no difference. There is no multiplicity argument that affects this interpretation because, in statistical terms, it corresponds to a simple closed test procedure. Usually this demonstration of a benefit is sufficient on its own, provided the safety profiles of the new agent and the comparator are similar...."

In FDA's guidance "Non-inferiority clinical trials", similar statements are included:

"In some cases, a study planned as an NI study may show superiority to the active control. ICH E-9 and FDA policy has been that such a superiority finding arising in an NI study can be interpreted without adjustment for multiplicity. Showing superiority to an active control is very persuasive with respect to the effectiveness of the test drug, because demonstrating superiority to an active drug is much more difficult than showing superiority to placebo. Similarly, a finding of less than superiority, but with a 95% CI upper bound for C-T considerably smaller than M2, is also statistically persuasive."

 The multiplicity adjustment is now everywhere. It is good to know that there is no need to do the multiplicity adjustment in the situation of interpreting a non-inferiority study as a superiority. 

SAS Resources on the Web

To be a good statistician, mastering the programming (SAS, SPSS, R, ...) is very important. This is especially true for statisticians who are working in the pharmaceutical/biotech/CRO industry. While the other statistical software may be popular in other area, SAS is still dominant in the pharmaceutical/biotech/CRO industry.

Fortunately, there are a lot of resources for SAS programming. The following links are worth being bookmarked.

• SAS support from SAS.com
• Resources to help you learn and use SAS provided by UCLA Academic Technology Services
• SAS library provided by UCLA Academic Technology Services
• CTSpedia: a knowledge base for clinical and translational research
• CTSpedia: SAS codes for graphic presentation of the safety data
• Biopharmaceutical network: containing presentation slides and SAS codes
• http://www.sas-resources.com/

Log(x+1) Data Transformation

When performing the data analysis, sometimes the data is skewed and not normal-distributed, and the data transformation is needed. We are very familiar with the typically data transformation approaches such as log transformation, square root transformation. As a special case of logarithm transformation, log(x+1) or log(1+x) can also be used.

The first time I had to use log(x+1) transformation is for a dose-response data set where the dose is in exponential scale with a control group dose concentration of zero. The data set is from a so-called Whole Effluent Toxicity Test. The Whole Effluent Toxicity test, one of the aquatic toxicological experiments, has been used by the US Environmental Protection Agency (USEPA) to identify effluents and receiving waters containing toxic materials, and to estimate the toxicity of waster water. In the Whole Effluent Toxicity testing, many different species and several endpoints are used to measure the aggregate toxic effect of an effluent. For many of these biological endpoints, toxicity is manifested as a reduction in the response relative to the control group. The whole Effluent toxicity testing is often designed as multi-concentrations, and includes a minimum of five concentrations of effluent and one control group. Therefore, from a dose-response analysis standpoint, the control group dose is considered as zero and the various concentrations are designed in exponential scale. Prior to the analysis, the log transformation for the dose, log(x), is usually applied. Since the control group dose is considered zero and log(x) does not exist, an easy solution is to use log(x+1). For the control group, the log(0+1) = 0, which seems to be a perfect approach in this case.

However, in clinical trials, I have seen many applications of the log-transformation, but not the log(x+1) transformation. From the FDA website, I could only find one study where the log(1+x) transformation was used. In advisory committee meeting document for AZ’s Drug Esomeprazole, the statistical analysis for the primary endpoint was stated as:

"The primary endpoint, change from baseline in signs and symptoms of GERD observed from video and cardiorespiratory monitoring, was analyzed by ANCOVA. Prior to the analysis, the number of events at baseline and final visit were normalized (to correspond to 8 hours observation time) then log-transformed via a log(1+x) transformation. The ANCOVA of change from baseline on the log-scale was adjusted for treatment and baseline. The least square means (lsmeans) for each treatment group were transformed and expressed as estimated percentage changes from baseline, and the lsmean for the esomeprazole treatment effect was transformed similarly, and expressed as a percentage difference from placebo, which was presented with the associated 2-sided 95% CI and p-value. "

Recently I read an article by Lachin et al “Sample size requirements for studies of treatment effects on Beta-cell function in newly diagnosed type 1 diabetes”, where various data transformation techniques were compared and the log(x+1) and sqrt(x) (square root of x) were suggested for the primary endpoint of C-peptide AUC mean. According to the paper “Most C-peptide values will fall between 0 and 1 and the distribution is positively skewed. Thus, scale-contracting transformations were considered. However, the log transformation could introduce negative skewness because log(x) approaches negative infinity as the value x approaches zero. This can be corrected by using log(x+1)"

When discussing with my friend, Dr Song, from CDC, we came up with the following Q&A regarding the use of  log(x+1) transformation:

Q: Is log(x+1) a fine approach for data transformation?
A: it’s fine to use ln(x+1) as long as this transformation makes data normal and variance relatively constant.

Q: Since the reason for using log(x+1) transformation is to avoid the log(x) approaching negative infinity as the x approaches zero. Could we change the measurement unit from pmol/mL to pmol/dL (1 pmol/dL = 100 pmol/mL)?
A: Log(100x) = log(100) + log(x). It only makes transformed value positive and it does not change the normality and variability. From statistical point of view, it is the same or equivalent to the transformation of log(x).

Q: With log(x), square root of x,…transformation, we can essentially transfer the calculated values or estimates back to the originally scale. With log(x+1), do we have a problem to convert the calculated values or estimates back to the original scale?
A: According to the paper by Lachin, “For each transformation y=f(x), the mean values and confidence limits are presented using the inverse transformation applied to the mean of the transformed values, and the corresponding confidence limits. Thus, for an analysis using y=log(x), the inverse mean is the geometric mean exp(mean y). For an analysis using y=log(x+1), the inverse mean is the geometric-like mean exp(mean y) - 1. For an analysis using y=sqrt(x), the inverse mean is (mean y)**2.”

Q: Whether or not one transformation approach is better than another depending on the range of the values?
A: log(x+1) transformation is often used for transforming data that are right-skewed, but also include zero values. The shape of the resulting distribution will depend on how big x is compared to the constant 1. Therefore the shape of the resulting distribution depends on the units in which x was measured. In the C-peptide AUC mean situation, all transformations are similar at the higher level of x (mean = 0.04 at month 24), but at the lower value level of x (mean = 0.01 at month 12), sqrt(x) is better than log(x+1) and log(x+1) is better than ln(x). This can be easily understood from the curvature of transformations shown in the following graph.

Some additional notes about the use of log(x+1) transformation:

• Any base for the logarithm can be used, but base 10 is often used because of interpretability
• In addition to log(x+1), log(2x+1) or log(x+3/8) transformation may also be used
• Remember to re-inspect the data after transformation to confirm its suitability. This will also be true no matter which data transformation approach is used.

Is Last Observation Carried Forward (LOCF) a dead approach to be used?

“To cope with situations where data collection is interrupted before the predetermined last evaluation timepoint, one widely used single imputation method is Last Observation Carried Forward (LOCF). This analysis imputes the last measured value of the endpoint to all subsequent, scheduled, but missing, evaluations. “ For a study with clinical outcomes measured at multiple timepoints (repeated measures), if the endpoint analysis approach is used for the primary efficacy variable, the most convenient and easy-to-understand imputation method is LOCF. In endpoint analysis, the change from the baseline to the last measurement (at a fixed timepoint such as at one year, at two year) is the dependent variable.

the LOCF is the easiest imputation approach for missing data to be understood by the non-statisticians. However, the LOCF approach has been the target for criticisms from the statisticians for  its lack of  a sound statistical foundation and for its biases in either direction (i.e., it is not necessarily conservative). After the National Academies published its draft report "The prevention and treatment of missing data in clinical trials”, using LOCF approach seemed to be out-dated and markedly out of step with modern statistical thinking.

Is the LOCF dead? Can we still use this approach in some situations in some clinical trials?

In reviewing some of the regulatory guidance, I believe that the LOCF is not totally dead. While we acknowledge that the LOCF is not a perfect approach, the LOCF approach should not be totally abandoned. In some situations, the LOCF approach is commonly agreed to be a more conservative approach and may be appropriate to be used.

In EMEA’s guidance "Guideline on missing data in confirmatory clinical trials", opinions and examples about the use of LOCF was explained:

Only under certain restrictive assumptions does LOCF produce an unbiased estimate of the treatment effect. Moreover, in some situations, LOCF does not produce conservative estimates. However, this approach can still provide a conservative estimate of the treatment effect in some circumstances.

To give some particular examples, if the patient’s condition is expected to deteriorate over time (for example in Alzheimer’s disease) an LOCF analysis is very likely to give overly optimistic results for both treatment groups, and if the withdrawals on the active group are earlier (e.g. because of adverse events) the treatment comparison will clearly provide an inappropriate estimate of the treatment effect and may be biased in favour of the test product. Hence in this situation an LOCF analysis is not considered appropriate. Indeed in Alzheimer’s disease, and other indications for diseases that deteriorate over time, finding a method that gives an appropriate estimate of the treatment effect will usually be difficult and multiple sensitivity analyses will frequently be required.

However, in other clinical situations (e.g. depression), where the condition is expected to improve spontaneously over time, LOCF (even though it has some sub-optimal statistical properties) might be conservative in the situations where patients in the experimental group tend to withdraw earlier and more frequently. Establishing a treatment effect based on a primary analysis which is clearly conservative represents compelling evidence of efficacy from a regulatory perspective.

Some of the FDA’s guidance gave clear instructions on the use of the LOCF approach in handling the missing data, which is more surprising to me.

In FDA’s guidance on “Diabetes Mellitus: DevelopingDrugs and Therapeutic Biologics for Treatment and Prevention”, I am surprised to see that the LOCF approach is actually suggested even though the LOCF approach may not be a conservative approach as indicated in the statements below since the HbA1c is expected to increase if the experimental drug is effective.

Although every reasonable attempt should be made to obtain complete HbA1c data on all subjects, dropouts are often unavoidable in diabetes clinical trials. The resulting missing data problems do not have a single general analytical solution. Statistical analysis using last observation carried forward (LOCF) is easy to apply and transparent in the context of diabetes trials. Assuming an effective investigational therapy, it is often the case that more placebo patients will drop out early because of a lack of efficacy, and as such, LOCF will tend to underestimate the true effect of the drug relative to placebo providing a conservative estimate of the drug’s effect. The primary method the sponsor chooses for handling incomplete data should be robust to the expected missing data structure and the time-course of HbA1c changes, and whose results can be supported by alternative analyses. We also suggest that additional analyses be conducted in studies with missing data from patients who receive rescue medication for lack of adequate glycemic control. These sensitivity analyses should take account of the effects of rescue medication on the outcome.

In FDA’s guidance “Developing Products for Weight Management”, the LOCF is also suggested even though it also says ‘repeated measures analyses can be used to analyze longitudinal weight measurements but should estimate the treatment effect at the final time point.

The analysis of (percentage) weight change from baseline should use ANOVA or ANCOVA with baseline weight as a covariate in the model. The analysis should be applied to the last observation carried forward on treatment in the modified ITT population defined as subjects who received at least one dose of study drug and have at least one post-baseline assessment of body weight. Sensitivity analyses employing other imputation strategies should assess the effect of dropouts on the results. The imputation strategy should always be prespecified and should consider the expected dropout patterns and the time-course of weight changes in the treatment groups. No imputation strategy will work for all situations, particularly when the dropout rate is high, so a primary study objective should be to keep missing values to a minimum. Repeated measures analyses can be used to analyze longitudinal weight measurements but should estimate the treatment effect at the final time point. Statistical models should incorporate as factors any variables used to stratify the randomization. As important as assessing statistical significance is estimating the size of the treatment effect. If statistical significance is achieved on the co-primary endpoints, type 1 error should be controlled across all clinically relevant secondary efficacy endpoints intended for product labeling.

This guidance was criticized by academics for several issues including the use of LOCF approach for the primary efficacy analyses. In comments submitted by UAB and Duke, there were the following statements:

While we strongly agree with the use of ITT approaches, we believe that the use of last observation carried forward (LOCF) is markedly out of step with modern statistical thinking. This perhaps reflects the fact that the 2004 FDA advisory meeting addressing this topic did not include a statistician with clinical trial expertise. A review of the video tapes referred to above will show that several leading statisticians all eschewed LOCF and suggested alternatives. These alternatives are now well established2 and available in major statistical packages. We have a paper nearing completion that compares the performance of these various approaches in multiple real obesity trials and will be glad to share a copy with FDA upon request. LOCF does not have a sound statistical foundation and can be biased in either direction (i.e., it is not necessarily conservative). Our own work suggests that multiple imputation may be the best method for conducting ITT analyses in obesity trials and that standard mixed models also work quite well in reasonably sized studies.

However, An advisory committee meet material for NDA 022580 for QNEXA in 2012 indicated that the LOCF approach is used for the primary efficacy analyses for weight management product developments.

When the outcome variable is dichotomous (success/failure, responder/non-responder,…), the LOCF is more acceptable if any subject who withdraw from the study early is considered as treatment failure or non-responder. This approach may also be called ‘the treatment failure imputation’, which is the most conservative approach. This approach is suggested in FDA Draft Guidance on Tacrolimus. In a recent NDA submission ( 202-736/N0001 Sklice (Ivermectin), topical cream, 0.5%, augmented Treatment of head lice infestations), this approach is also used in handling the missing data for primary efficacy analysis.

In the end, there is no perfect imputation approach if the missing data occurs too often. During the clinical trial from the study design to protocol compliance, to the data collection, every effort should be made to minimize the missing data. I think that the statements about the handling of missing data is pretty clear and reasonable in FDA’s Draft Guidance for Industry and Food and Drug Administration Staff - The Content of Investigational Device Exemption (IDE) and Premarket Approval (PMA) Applications for Low Glucose Suspend (LGS) Device Systems

Handling of Missing Data
Starting at the study design stage and throughout the clinical trial, every effort should be made to minimize patient withdrawals and lost to follow-ups. Premature discontinuation should be summarized by reason for discontinuation and treatment group. For an ITT population, an appropriate imputation method should be specified to impute missing HbA1c and other primary endpoints in the primary analysis. It is recommended that the Sponsor/Applicant plan a sensitivity analysis in the protocol to evaluate the impact of missing data using different methods, which may include but is not limited to per protocol, Last Observation Carry Forward (LOCF), multiple imputation, all missing as failures or success, worst case scenario, best case scenario, tipping point, etc.

Cookbook SAS Codes for Bioequivalence Test in 2x2x2 Crossover Design

In Clinical Pharmacology, inferential statistics is performed to show the bioequivalence in terms of the Area Under the Curve (AUC) and the Maximum Concentration (Cmax) that are obtained from the time-concentration data. The typical clinical trial design is 2x2x2 crossover design contains two treatment sequences (Test followed by Reference vs. Reference followed by Test), two treatment periods (period 1 vs period 2), and two treatment groups (Test vs. Reference).

According to FDA’s guidance “Statistical Approaches to Establishing Bioequivalence”, the following assumptions can be made for the test of bioequivalence:

1. AUC and Cmax follow log-normal distribution

2. Bioequivalence is shown if the 90% confidence interval for the geometric least square mean ratio of Test/Reference is fall within 0.8 and 1.25

The statistical tests will follow so called two one-sided tests procedure (TOST) which can be implemented using the following cookbook SAS codes.

* Preparing the data and log-transfer the AUC and Cmax data;

data pkparm;
    set pdkparm;
    keep subno seqence treat period AUC CMAX;
    lauc=log(auc);
    lcmax=log(cmax);
run;

*** Fit the ANOVA model;
ods output LSMeans=lsmean;
ods output estimates=est;
proc mixed data=pkparm;
      class sequence period treat subno;
      model LAUC or Cmax=sequence period treat;
      random subno(sequence);
      lsmeans treat/pdiff cl alpha=0.1;
      estimate 'T/R' treat -1 1 / cl alpha=0.1;
     * make 'LSMEANS' out=lsmean; *used in old SAS versions;
     * make 'estimate' out=est; *used in old SAS versions;
run;

* Anti-log transformation to obtain the Geometric Means;
data lsmean;
      set lsmean;
      gmean=exp(estimate); *Geometric means;
run;

proc print data=lsmean;
run;

* Anti-log transformation to obtain the ratio of Geometric Means (point estimate) and its 90% confidence interval (lower and upper bounds);
data diffs;
     set EST;
     ratio=exp(estimate); ** Ratio of geometric mean;
     lower=exp(lower); ** 90% CI lower bound;
     upper=exp(upper); ** 90% CI upper bound;
run;

proc print data=diffs;
run;

Some Notes:

1. p-value for Treatment/Reference comparison can also be obtained from the model (in above EST data set). However, p-value is not the criteria for declaring the bioequivalence and must be interpreted appropriately. We could have a significant p-value (p<0.05) and still show bioequivalence as long as the 90% confidence interval of the geometric mean ratio is fall within 0.8 and 1.25 range. If we have a 90% confidence interval like [0.85, 0.95] or [1.05, 1.15], the bioequivalence will be shown even though the p-values are significant.

2. In SAS Proc Mixed model, the subject within sequence is coded as subno(seqence), not sequence(subno). However, if you use sequence(subno), the results will be the same.

3. For log transformation, it does not matter which base (base 10, 5, or e (natural log)) as long as the final results from the model are correctly an-log transferred back.

4. while we typically say ‘Ratio of geometric mean’, it is actually the ‘ratio of geometric least square mean’ from the model.

5. FDA guidance "Statistical Approaches to Establishing Bioequivalence" appendix E "SAS Program Statements for Average BE Analysis of Replicated Crossover Studies" provided the detail SAS codes with Proc Mixed. While it is stated for the 'replicated crossover studies', however, 2x2x2 crossover design is a simplest case of the replicated crossover studies.

The following illustrates an example of program statements to run the average BE analysis using
PROC MIXED in SAS version 6.12, with SEQ, SUBJ, PER, and TRT identifying sequence,
subject, period, and treatment variables, respectively, and Y denoting the response measure (e.g., log(AUC), log(Cmax)) being analyzed:

PROC MIXED;
CLASSES SEQ SUBJ PER TRT;
MODEL Y = SEQ PER TRT/ DDFM=SATTERTH;
RANDOM TRT/TYPE=FA0(2) SUB=SUBJ G;
REPEATED/GRP=TRT SUB=SUBJ;
ESTIMATE 'T vs. R' TRT 1 -1/CL ALPHA=0.1;

The Estimate statement assumes that the code for the T formulation precedes the code for the R formulation in sort order (this would be the case, for example, if T were coded as 1 and R were coded as 2). If the R code precedes the T code in sort order, the coefficients in the Estimate statement would be changed to -1 1.

In the random statement, TYPE=FA0(2) could possibly be replaced by TYPE=CSH. This guidance recommends that TYPE=UN not be used, as it could result in an invalid (i.e., not non-negative definite) estimated covariance matrix.
Additions and modifications to these statements can be made if the study is carried out in more than one groups of subjects

The "PATIENTS' FDA" Act - Sens. Richard Burr and Tom Coburn Introduce a New Plan to Reform the FDA

In my previous article "Should the design and conduct of clinical trials be simplified? ", I discussed several FDA guidance that suggested that in several areas, the dada collections may be reduced and the clinical trial monitoring may need to switch to the risk-based approach instead of the current frequent on-site visits and 100% source data verification.

Coincidently, yesterday, Sens. Richard Burr and Tom Coburn introduced a new plan to reform the FDA - The "PATIENTS' FDA" Act . The patient' FDA act (if approved) will force FDA to be further transparent and to be mindful in requesting too much data from the pharmaceutical companies. For last several years, after several high-profile drug withdrawals (Vioxx, Avandia for example), FDA has swung to another extreme and become very conservative, which subsequently made the clinical trials more difficult to execute and drug development  more costly. Perhaps, it is really not the FDA's intension, however, many of its staff/reviewers become too conservative. Instead of working with the industry to bring the new medications to the patients with the reduced cost and within the reasonable timeframe, some reviewers request the sponsors to collect data with no real justification and ask the sponsors to implement something that may just be for reviewer's own interest or opinion.   

Forbes published a good article as a companion to this bill. Here are some of the paragraphs from this article.

More accountability for meeting drug-review deadlines. The FDA has been increasingly failing to meet its PDUFA-mandated deadlines for giving companies approval decisions on new drug applications. The PATIENTS’ FDA Act would require the FDA to “report [to Congress] on a deeper level detail with respect to the performance goals agreed to in the prescription drug, generic drug, and biosimilar user fee agreements,” and hold individual reviewers accountable for their speed in reviewing applications.

stop forcing companies to do unnecessary and expensive busywork. The bill’s summary notes that “some FDA reviewers request reams of additional information about a drug or device that is beyond the scope of data needed to meet the FDA’s approval standard.” The FDA will be required, under the bill, to “document the scientific and regulatory rationale” for such decisions, and review within one year “the costs and adoption of the least burdensome approaches to regulation.” The bill would also codify the FDA’s “commitment to improve on patient risk-benefit considerations…to ensure accountability for fulfilling…the user fee agreements.”

 Take more advantage of clinical trials in other countries. The bill would require FDA to work with “other specific regulatory authorities of similar standing” to encourage uniform standards for clinical trials. (The Geneva-based International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use, or ICH, performs many of these functions.) FDA will also be instructed to help sponsors “minimize the need for duplication of clinical studies, preclinical studies or non-clinical studies.”

 

Should the design and conduct of clinical trials be simplified?

As mentioned in one of FDA’s guidance, “in the past two decades, the number and complexity of clinical trials have grown dramatically. These changes create new challenges in clinical trial oversight such as increased variability in investigator experience, ethical oversight, site infrastructure, treatment choices, standards of health care, and geographic dispersion.”. According to an article by Mr Getz titled “The Heavy Burden of Protocol Design More complex and demanding protocols are hurting clinical trial performance and success”, companies sponsoring clinical research have openly acknowledged that protocol design negatively impacts clinical trial performance and may well be the single largest source of delays in getting studies completed. When designing a clinical trial, sponsors often try to include too many endpoints and too many measures hoping that all of these endpoints (if results are good) will contribute to the overall evidence of the clinical efficacy. The sponsors attempt to collect a lot of information that is not must-to-have, but nice-to-have (for future data dredging, marketing, publications,…). We want to do it all in one clinical study. This obviously increases the complexity of the study protocol that subsequently increases the length of the clinical trial, the cost of the clinical trial, and the quality (more protocol incompliance) of clinical trial data.

In terms of the conduct of the clinical trials, emphasis on the compliance of good clinical practice has resulted in perceptions that the clinical trial data must be 100% monitored and source-verified, all data programming and analysis must be independently validated, over-reporting adverse events must be requirement of the GCP compliance…

Over the last two years, FDA has issues several guidance in an attempt to change these perceptions.

In FDA’s new draft guidance "Determining the extent of safety data collection needed in late stage premarket and post-approval clinical investigations", it states that its intention is “to assist clinical trial sponsors in determining the amount and types of safety data to collect in late-stage premarket and post-market clinical investigations for drugs or biological products, based on existing information about a product’s safety profile.” This new guidance addresses the circumstances in which it may be acceptable to acquire a reduced amount of safety information during clinical trials. In some situations, excessive data collection may be unnecessary and not helpful.

• FDA conducted a webinar on this topic and the webinar is free to the general public at: https://collaboration.fda.gov/p42291115/
• The presentation slides can be accessed at: http://www.fda.gov/downloads/Drugs/UCM297332.pdf

In Guidance for Industry “Oversight of Clinical Investigations — A Risk-Based Approach to Monitoring”, FDA encourages the sponsors to implement the alternative clinical monitoring instead of the only way of on-site monitoring. The webinar and presentations slides can be found at http://www.fda.gov/Training/GuidanceWebinars/ucm277044.htm

“For major efficacy trials, companies typically conduct on-site monitoring visits at approximately four- to eight-week intervals,8 at least partly because of the perception that the frequent on-site monitoring visit model, with 100% verification of all data, is FDA’s preferred way for sponsors to meet their monitoring obligations. In contrast, academic coordinating centers, cooperative groups, and government organizations use on-site monitoring less extensively. For example, some government agencies and oncology cooperative groups typically visit sites only once every two or three years to qualify/certify clinical study sites to ensure they have the resources, training, and safeguards to conduct clinical trials. FDA also recognizes that data from critical outcome studies (e.g., many National Institutes of Health-sponsored trials, Medical Research Council-sponsored trials in the United Kingdom, International Study of Infarct Survival, and GISSI), which had no regular on-site monitoring and relied largely on centralized and other alternative monitoring methods, have been relied on by regulators and practitioners. These examples demonstrate that use of alternative monitoring approaches should be considered by all sponsors, including commercial sponsors when developing risk-based monitoring strategies and plans”

Many sponsors may be reluctant to adopt this guidance and stick with the status quo approach of frequent on-site visits with 100% verification of all data. They may be worried that loosing the clinical monitoring could incur the incompliance.

In Guidance for Clinical Investigators, Sponsors, and IRBs Adverse Event Reporting to IRBs — Improving Human Subject Protection, FDA advised the sponsors to report to IRB (and FDA presumably) the AE only if it were unexpected, serious, and would have implications for the conduct of the study, not all unanticipated AEs. The sponsor should analyze the unanticipated AEs before reporting.

“the practice of local investigators reporting individual, unanalyzed events to IRBs, including reports of events from other study sites that the investigator receives from the sponsor of a multi-center study—often with limited information and no explanation of how the event represents an unanticipated problem—has led to the submission of large numbers of reports to IRBs that are uninformative. IRBs have expressed concern that the way in which investigators and sponsors of IND studies typically interpret the regulatory requirement to inform IRBs of all "unanticipated problems" does not yield information about adverse events that is useful to IRBs and thus hinders their ability to ensure the protection of human subjects.”

 

There are many other areas in clinical trial practices that can and should be simplified. For example, some protocols instruct investigators to record and report all untoward events that occur during a study as AEs/SAEs, which could include common symptoms of the disease under study and/or other expected clinical outcomes that are not study endpoints. Over-reporting of AE/SAEs can incur additional burdens and can dilute or obscure signal identification. Another example is to spend too much efforts on the screening failure subjects. It is true that the recording of the adverse events starts once the informed consent form is signed. However it is unnecessary to write a full-blown SAE narrative for a screening failure subject that has nothing to do with the assessement of the safety of the experimental product.

Standardized study data for electronic submission - CDISC compliance

FDA has recently issued a series of draft guidance on “providing regulatory submission in electronic format”. The most recent one is about “providing regulatory submissions in electronic format – standardized study data”. We have heard a lot of discussions about CDISC, SDTM, ADaM,…While these data standards are not mandated yet, FDA is encouraging the submission of the electronic data in CDISC compliant format. The draft guidance about ‘standardized study data’ is another sign that the industry should move toward the compliance of the CDISC standards for the submission.

 

In terms of what kinds of data standards to be used in electronic submissions, please see FDA’s web pages on study data standards.

CBER may have a little bit different requirements from other divisions. CBER has its own resource page for submission of data in CDISC format to CBER

 

While CDISC standards may be good for FDA reviewers and may accelerate the review process, it will indeed add burdens to the industry. For the original clinical datasets, there will be additional mapping to be implemented in order to prepare the datasets in SDTM compliant formats. SDTM format is not user-friendly and sometimes it is several steps to link back to the CRF/eCRFs.

 

Other readings:

• Providing Regulatory Submissions in Electronic Format — General Considerations Regulatory Submissions in Electronic Format for Biologic Products
• CDER Common Data Standards Issues Document issued last month (Version 1.1/December 2011)
• CBER Process for Planning and Accepting CDISC SDTM and ADaM Formatted Submissions in CBER
• Assessment of the Impact of the Electronic Submission and Review Environment on the Efficiency and Effectiveness of the Review of Human Drugs – Final Report
• Assessment of the Impact of the Electronic Submission and Review Environment on the Efficiency and Effectiveness of the Review of Human Drugs – Final Report 

 

Futility Analysis in Clinical Trials - Stop the trial for futility

A colleague of mine asked me to explain the concept of “futility analysis” using plain languages. The question is triggered by the recent news such as “Solanezumab, Gammagard Trials Survive Futility Analysis” In an Alzheimer trial, it even says from the futility analysis, “there is greater than a 20% statistical probability of success in achieving the primary outcome measure of cognitive function preservation.

During a clinical trial, we can perform interim analysis (or DMC, DSMB review) for three different reasons:

1. The interim analysis for safety

1)       with pre-specified stopping rule (for example stop the trial if we see # of cases of Serious Adverse Events)

2)       without pre-specified stopping rule (rely on DMC members to review the overall safety)

2. The interim analysis for efficacy: To see if the new treatment is overwhelmingly better than control  - then stop the trial for efficacy
3. The interim analysis for futility:  To see if the new treatment is unlikely to beat the control – then stop the trial for futility  - this is called ‘futility analysis’.

In situations 2 and 3, the criteria for stopping rule for efficacy could be different from the stopping rule for futility, but need to be pre-specified.

In situation #2 (stopping the study for efficacy), there will be a penalty for alpha spending - if the overall alpha is 0.05, a portion of the alpha will be allocated for tee interim analysis and the alpha for final analysis will be less than 0.05.   In situation #3, there is no penalty for alpha spending. 

An example for futility analysis: at the beginning of the trial, we assumed 65% successful rate for new treatment group and 50% successful rate for the control group. We would like to establish superiority. In the middle of the study, we did an interim analysis. The interim analysis showed 55% successful rate for new treatment group and 50% successful rate for the control group. Based on the results from the interim analysis, we can calculate the probability and conditional power: if we continue to finish the study, what is the probability of the new treatment group better than control? If this probability is too small and meets the pre-specified criteria, we would stop the trial for futility. If this probability is reasonable, we can continue the trial as pre-planned or we can continue the trial with the sample size adjustment (typically increase due to the smaller effect size).

In a paper by Miller et al “Paclitaxel plus Bevacizumab versus Paclitaxel Alone for Metastatic Breast Cancer”, one pre-planned interim analysis and two additional interim analyses were performed and three stopping rules (for safety, for efficacy, and for futility) were pre-specified and evaluated. It is reasonable to assume that none of these stopping rules was triggered since the study was not stopped.  

Futility analysis or stopping the trial for futility is not without controversy. An article by Schoenfeld and Meade discussed this issue. See “Pro/con clinical debate: It is acceptable to stop large multicentre randomized controlled trials at interim analysis for futility”.

Further readings:

• Snapinn et al (2006) Assessment of futility in clinical trials
• Lachin (2005) A review of methods for futility stopping based on conditional power (or free access from here)

How to interpret odds ratios that are smaller than 1?

Here is a question posted on the web about the interpretation of odds ratios that are less than 1.

"I know that OR estimates= 1 mean that both groups/categories have the same odds. I also know that if OR estimates are greater than 1, e.g, 1.24 for Young vs. Old persons, then I can say: Young people have 24% increase in the odds of living in an apartment than older people. Or, I also know I can say, for example, for an OR of 0.322 Non-White vs. White, that the odds of Whites are 1/.322 = about 3 times higher than those of Non-Whites, to live in a house they own. Now, how would I say the odds are of a NON-White person in the example above, to live in a house they own? Is is 1-.322=.678 less likely, with respect to odds, to live in a house they own? Or, similarly, they have 67.8% lower odds to live in a house they own? "

If we have to say the odds for a Non-White person, we may say "Non Whites have odds .322 times as great as those of Whites".

In an article "When can odds ratio misled?", Davies et al stated:

"the odds of an event is the number of those who experience the event divided by the number of those who do not. It is expressed as a number from zero (event will never happen) to infinity (event is certain to happen). Odds are fairly easy to visualise when they are greater than one, but are less easily grasped when the value is less than one. Thus odds of six (that is, six to one) mean that six people will experience the event for every one that does not (a risk of six out of seven or 86%). An odds of 0.2 however seems less intuitive: 0.2 people will experience the event for every one that does not. This translates to one event for every five non-events (a risk of one in six or 17%). "

Another webblog described the issue in interpreting the odds ratio that is less than one.

"When you are interpreting an odds ratio (or any ratio for that matter), it is often helpful to look at how much it deviates from 1. So, for example, an odds ratio of 0.75 means that in one group the outcome is 25% less likely. An odds ratio of 1.33 means that in one group the outcome is 33% more likely."

In an article "The odds ratio: calculation, usage, and interpretation" in Biochemia Medica, the author clear suggest converting the odds ratio to be greater than 1 by arranging the higher odds of the evnet to avoid the difficulties in interpreting the odds ratio that is less than 1.

“An OR of less than 1 means that the first group was less likely to experience the event. However, an OR value below 1.00 is not directly interpretable. The degree to which the first group is less likely to experience the event is not the OR result. It is important to put the group expected to have higher odds of the event in the first column. It is not valid to try to determine how much less the first group’s odds of the event was than the second group’s. When the odds of the first group experiencing the event is less than the odds of the second group, one must reverse the two columns so that the second group becomes the first and the first group becomes the second. Then it will be possible to interpret the difference because that reversal will calculate how many more times the second group experienced the event than the first. If we reverse the columns in the example above, the odds ratio is: (5/22)/(45/28) = (0.2273/1.607) = 0.14 and as can be seen, that does not tell us that the new drug group died 0.14 times less than the standard treatment group. In fact, this arrangement produces a result that can only be interpreted as “the odds of the first group experiencing the event is less than the odds of the second group experiencing the event”. The degree to which the first group’s odds are lower than that of the second group is not known.”

In practice, when dealing with the odds ratio less than 1, when possible, I almost always try to reverse the column or recode the response variable to get the odds ratio larger than 1 before I do an interpretation. It is easier for people (especially non-statisticians) to understand the odds ratio with the value greater than 1.

In an example below, the treatment group is actually less effective in terms of the response.

Treatment	Failure (0)	Success (1)
No (0)	21	30
Yes (1)	32	17

The following SAS code can be easily used to calculate the odds ratio:
Data test; input Trt resp count; datalines;
1 1 17
1 0 32
0 1 30
0 0 21
;
proc logistic data=test descending; weight count; model resp=trt;
run;

From the SAS outputs, we get the odds ratio of 0.372, which indicates that the treatment group has odds 0.372 times lower compared to the non-treatmetn group in terms of the success. The interpretation is somewhat difficult to understand.

The program can be easily revised to calculate the odds ratio of failure rate, which gives an odds ratio of 1/0.372 = 2.689. The odds ratio can be intepretated as "the odds of achieve the success in non-treatment group is 2.689 times higher than that in treatment group".

proc logistic data=test; weight count; model resp=trt;
run;

In SAS PROC Logistic, with descending option, probability modeled is response=1 (success); without descending option, probability modeled is response=0 (failure);

Design and Analysis of Bioequivalence Studies for Highly Variable Drugs (HVD) or Highly Variable Drug Products (HVDP)

For bioequivalence studies, it is often for us to show the average bioequivalence by declaring the bioequivalence if the 90% confidence interval of the geometric least squares mean ratio is within 80-125%. The associated study design is typically 2x2x2 cross over design with reasonable sample size (for example, 12 subjects, 24 subjects,…) if the within subject variable is not so big. This approach has been outlined in several FDA’s guidelines:

-         Statistical Approaches to Establishing Bioequivalence

-         Bioavailability and Bioequivalence Studies for Orally Administered Drug Products — General Considerations

-         Food-Effect Bioavailability and Fed Bioequivalence Studies

Recently, there are a lot of discussions about the bioequivalence studies for a product with high variability (high variable drugs). Highly Variable Drugs refer to the type of drugs with higher within subject variability and  is Defined as one for which the root mean square error (RMSE) from the ANOVA bioequivalence analysis > 0.3 for either AUC or Cmax.

For highly variable drugs, if we employ the common study design, the required sample size will be very large, which will cause the ethic concerns to implement such studies.

FDA had several advisory committee meeting in discussing this issue. The most recent meetings were in 2004 and 2009. In 2009 meeting, the slide presentation  by Dr Conner from FDA summarized the development in dealing with this issue and FDA’s position (see slide presentation “Bioequivalence Methods for Highly Variable Drugs and Drug Products”).

Among various approaches to address the bioequivalence issue for highly variable drugs, reference-scaled average BE approach has been suggested. This approach requires less subjects in the study, but with replicated treatment design such as three-period, reference- replicated, crossover design with sequences of TRR, RTR, & RRT or four-period design with sequences of TRTR and RTRT. The replicated crossover designs were also discussed in FDA guidance “Statistical Approaches to Establish Bioequivalance”, but was for dealing with the carryover effects. Here, the replicated crossover designs are for dealing with highly variable drugs.

The implementation of the reference-scaled average BE approaches have been detailed and discussed in FDA guidance (draft) many publications. The most relevant ones are:

-         FDA Guidance on Progesterone (2011)

-         Sample Sizes for Designing Bioequivalence Studies for Highly Variable Drugs by Endrenyi and Tothfalusi (2012)

-         Bioequivalence of Highly Variable Drugs: A Comparison of the Newly Proposed Regulatory Approaches by FDA and EMA by Karalis et al (2011)

The European Medicines Agency also recognizes certain drugs as highly variable drug products (HVDP) and  is willing to accept a wider difference (i.e., a wider 90% confidence interval) in Cmax for bioequivalence evaluation. In its guidance "Guideline on the Investigation of Bioequivalence", section 4.1.10 specifically discussed the HVDP:

Other Readings:

-         Bioequivalence Approaches for Highly Variable Drugs and Drug Products by Haidar et al

-         Generic Drug Bioequivalence by Dr Aaron Sigler

-         An Example of How to Write the Statistical Section of a Bioequivalence Study Protocol for FDA Review

-         Global Bioequivalence Guidelines

-         Using SAS Proc Power to Perform Model-based Power Analysis for Clinical Pharmacology Studies