Pre-specification and statistical analysis plan

For industry-sponsored clinical trials especially those trials for the regulatory submission purpose (i.e., adequate and well-controlled studies), a prespecified analysis plan is essential. In a previous post, When to Finalize the Statistical Analysis Plan (SAP)? the timing of the SAP development/finalization was discussed. 

According to the current guidelines (such as ICH E9 STATISTICAL PRINCIPLES FOR CLINICAL TRIALS", the SAP may be written as a separate document (as we usually do) and should be finalized before breaking the blind.

"5.1 Prespecification of the Analysis When designing a clinical trial the principal features of the eventual statistical analysis of the data should be described in the statistical section of the protocol. This section should include all the principal features of the proposed confirmatory analysis of the primary variable(s) and the way in which anticipated analysis problems will be handled. In case of exploratory trials this section could describe more general principles and directions. The statistical analysis plan may be written as a separate document to be completed after finalising the protocol. In this document, a more technical and detailed elaboration of the principal features stated in the protocol may be included. The plan may include detailed procedures for executing the statistical analysis of the primary and secondary variables and other data. The plan should be reviewed and possibly updated as a result of the blind review of the data and should be finalised before breaking the blind. Formal records should be kept of when the statistical analysis plan was finalised as well as when the blind was subsequently broken. If the blind review suggests changes to the principal features stated in the protocol, these should be documented in a protocol amendment. Otherwise, it will suffice to update the statistical analysis plan with the considerations suggested from the blind review. Only results from analyses envisaged in the protocol (including amendments) can be regarded as confirmatory.

In the statistical section of the clinical study report the statistical methodology should be clearly described including when in the clinical trial process methodology decisions were made (see ICH E3). "

Similar languages are in ICH E8 "GENERAL CONSIDERATIONS FOR CLINICAL TRIALS":

3.2.4 Analysis

The study protocol should have a specified analysis plan that is appropriate for the objectives and design of the study, taking into account the method of subject allocation, the measurement methods of response variables, specific hypotheses to be tested, and analytical approaches to common problems including early study withdrawal and protocol violations. A description of the statistical methods to be employed, including timing of any planned interim analysis(es) should be included in the protocol (see ICH E3, ICH E6 and ICH E9).  The results of a clinical trial should be analysed in accordance with the plan prospectively stated in the protocol and all deviations from the plan should be indicated in the study report. Detailed guidance is available in other ICH guidelines on planning of the protocol (ICH E6), on the analysis plan and statistical analysis of results (ICH E9) and on study reports (ICH E3).

However, there are randomized controlled studies being single-blind or no blinding (open-label study). According to the ICH E14 revision 1 "GENERAL CONSIDERATIONS FOR CLINICAL STUDIESE8(R1)", the statistical analysis plan should be finalized before the unblinding of study data (for blinded studies) and before the conduct of the study (for open-label studies):

5.1.6 Statistical Analysis

The statistical analysis of a study encompasses important elements necessary to achieving the study objectives. The study protocol should include a statistical methods section that is appropriate for the objectives and study design (ICH E6 and E9). A separate statistical analysis plan may be used to provide the necessary details for implementation. The protocol should be finalised before the conduct of the study, and the statistical analysis plan should be finalised before the unblinding of study data, or in the case of an open-label study, before the conduct of the study. These steps will increase confidence that important aspects of analysis planning were not based on accumulating data in the study or inappropriate use of external data, both of which can negatively impact the reliability of study results. For example, the choice of analysis methods in a randomised clinical trial should not change after examining unblinded study data, and external control subjects should not be selected based on outcomes to be used in comparative analyses with treated study subjects.

It is commonly accepted that the pre-specified analysis plan needs to be included in the protocol for primary efficacy endpoint and perhaps also the secondary efficacy endpoints. A separate statistical analysis plan will be prepared (usually after the study protocol has been implemented and some patient data (in a blinded fashion) is available).

FDA has several guidelines for FDA reviewers for their review of the SAP.  We can see what the FDA's expectations are for the SAP.

Good Review Practice: Statistical Review Template

Data and Analysis Quality

Review the quality and integrity of the submitted data. Examples of relevant issues include the following: 

• Whether it is possible to reproduce the primary analysis dataset, and in particular the primary endpoint, from the original data source 

• Whether it is possible to verify the randomized treatment assignments 

• Findings from the Division of Scientific Investigation or other source(s) that question the usability of the data 

• Whether the applicant submitted documentation of data quality control/assurance procedures (see ICH E3,1 section 9.6; also ICH E6,2 section 5.1) 

• Whether the blinding/unblinding procedures were well documented (see ICH E3, section 9.4.6)

• Whether a final statistical analysis plan (SAP) was submitted and relevant analysis decisions (e.g., pooling of sites, analysis population membership, etc.) were made prior to unblinding.

Applicants are expected to submit data of high quality and make it possible for the FDA to reproduce their results. In turn, FDA reviewers should provide adequate documentation so that the applicant or another data user could reproduce their independent findings. The level of documentation needed will depend on the complexity and novelty of the analysis. If an ordinary ANOVA or ANCOVA is used, for example, it would suffice to identify the dependent and independent variables. If a more unusual analysis is performed, then it may be necessary to provide code. The code should be either included in the report or put in an appropriate digital archive. 

Good Review Practice: Clinical Review of Investigational New DrugApplications

8. STATISTICAL ANALYSIS PLANS
Sponsors should be encouraged to include the SAP as part of the protocol, rather than providing it in a separate document, even if the SAP has not been finalized. If the SAP is changed late in the trial, particularly after the data may be available, it is critical for the sponsor to assure the FDA that anyone making such changes has been unaware of the results. Sponsors should be encouraged to describe the methods used to ensure compliance. Additional information on the principles of statistical analyses of clinical trials is available in ICH E9. 75 The review of the SAP requires close collaboration with the biostatistical reviewer. 

8.1 Planned Analyses Analyses intended to support a marketing application (generally analyses for the phase 3 efficacy trials) should be prospectively identified in the protocol and described in adequate detail. An incomplete description of the proposed analyses in the protocol can leave ambiguity after trial completion in how the trial will be analyzed. 

Nonprospectively defined analyses pose problems because they leave the possibility that various statistical methods were tried and only the most favorable analysis was reported. In such cases, the estimates of drug effect may be biased by the selection of the analysis, and the proper correction for such bias can be impossible to determine. Preplanning of analyses reduces the potential for bias and often reduces disputes between sponsors and the FDA on the interpretation of results. The same principles apply to supportive and/or sensitivity analyses. These analyses should be prospectively specified, despite the fact that the results of such analyses cannot be used as a substitute for the primary analysis. If the protocol pertains to a multinational trial, it is important that an analysis of the regional  differences be prespecified. Clinical reviewers should review these considerations for planned analyses in collaboration with statistical reviewers. 

Although detailed prespecification is essential for the primary efficacy analysis, the ability to interpret findings on other outcomes, such as important secondary efficacy endpoints for which a claim might be sought, is also dependent on the presence of a prospectively described analysis plan. Observations of potential interest, termed descriptive endpoints because the trial will almost always be underpowered in their respect, may be considered in a trial that is successful on its primary endpoint to further explore consistency in demographic subgroups (e.g., sex, age, and race) or evaluate regional differences in multinational trials. Safety outcomes are also important and should be specified prospectively. They will often not be part of the primary analysis unless the trial was designed to assess such an endpoint. Analyses not prospectively defined will in most cases be considered exploratory; see section 8.2.2.1, Descriptive Analysis, for potential use of such descriptive analyses. 

Interim analyses may play an important role in trial design. They present complex issues, including preservation of overall Type I error (alpha spending function), re-estimation of sample size, and stopping guidelines. Plans for interim analyses should be prospectively determined and reviewers should discuss these plans with the statistician. See section 8.1.3, Interim Analysis Plans, for further discussion of these plans. 

8.1.1 Adequacy of the Statistical Analysis Plan
When reviewing the SAP, it is critical to consider whether there is ambiguity about the planned analyses. Particular attention should be paid to the primary endpoint and how it will be analyzed. If there are multiple primary endpoints or analyses, the Type 1 error rate should be controlled appropriately. If there is a single primary endpoint, details of the analysis are important. For example, an SAP that defines the primary analysis as a comparison of the time to event between treatment arms leaves open many possibilities, such as the specific analytical approach (e.g., Cox regression, log rank test), whether the analyses will be adjusted for covariates (and which covariates would be included), and the method for this adjustment. Censoring for subjects who drop out of the trial or who are lost to follow-up should be discussed, particularly since dropout may not be random. Post dropout follow-up may have different implications for superiority and noninferiority trials. 

Consideration also should be paid to other preplanned analyses, such as secondary endpoint analysis, population subset analysis, regional analysis, and interim analysis. Both clinical and statistical reviewers should collaborate in order to make appropriate recommendations. 

When there are possible secondary efficacy endpoints (e.g., different time points, population subsets, different statistical tests, different outcome measures), it is critical to determine how they will be analyzed and their role in the efficacy assessment. In general, secondary analyses are not considered in regulatory decision-making unless there is an effect on the primary endpoint, so that no Type 1 error adjustment is needed for the primary endpoint. A secondary endpoint intended to represent a trial finding (and thus a possible claim) after success on the primary endpoint should be considered as part of the overall SAP and, if there is more than one of these, a multiplicity adjustment or gatekeeper approach may be necessary to protect the Type 1 error rate at a desired level (alpha = 0.05) for such analyses. Positive results in a secondary analysis when the primary endpoint did not demonstrate a statistically significant difference generally will not be considered evidence of effectiveness. 

Protection of the overall (family-wide) Type 1 error rate at a desired level (alpha = 0.05) is essential when the protocol has designated multiple hypotheses testing. Examples include efficacy comparisons among multiple doses with respect to primary and secondary endpoints, subpopulation analysis, and regional analysis. Various commonly used statistical procedures can be used for this multiplicity adjustment (e.g., Bonferroni, Dunnett, Hochberg, Holm, Hommel, and gatekeeping procedures), and these procedures will be considered in a multiplicity guidance under development. The proper use of each procedure depends on the priority of the hypotheses to be tested and the definition of a successful trial outcome. The following two examples are illustrative:

Example 1. A placebo-controlled trial with one primary endpoint and three treatment doses (low, medium, and high) is planned. To assess the efficacy of the three doses as compared to placebo, a commonly used hierarchical procedure tests sequentially from high dose to low against placebo, each at alpha = 0.05, until a pvalue ≤ 0.05 is not attained for a dose. Significance is then declared for all doses that achieved a p-value ≤ 0.05. 

The Bonferroni correction approach also can be used to share alpha = 0.05 among the three doses and test each one at alpha = 0.05/3 = 0.017. This method will be less efficient than the sequential method, if the effect is likely to be positively associated with dose. The primary analysis could also evaluate all three doses pooled versus placebo (less efficient if the low doses are not effective) or of the two highest doses versus placebo. 

Example 2. A placebo-controlled trial of two endpoints, A and B, and three treatment doses (low, medium, and high) is planned. Suppose endpoint A is thought to be more indicative of the true effect than B and so is placed higher in the hierarchy than B. Also, suppose the medium and high doses are hypothesized to be equally effective while the low dose is considered less likely to exhibit significance. Multiple clinical decision rules are designated in hierarchical order to demonstrate the efficacy: 

• Show benefit for each of two higher doses individually compared to placebo with respect to endpoint A and endpoint B 

• Show benefit for the two higher doses pooled compared to placebo with respect to endpoint A or B 

• Show benefit for the low dose with respect to endpoint A 

• Show benefit for the low dose with respect to endpoint B 

Although the Bonferroni, Holm, or Hommel procedure can be applied to test these four hypotheses, a gatekeeper procedure that sequentially tests the four sets of hypotheses in hierarchical order, each at alpha = 0.05, is likely to be more efficient. 

How missing data (particularly from dropouts) are handled can profoundly affect trial outcomes. Sponsors should be encouraged to detail how they plan to minimize dropouts and to specify particular methods for assessing data from dropouts. It is not credible to design these analyses once unblinded data are available. Reviewers should consider the best approach for the particular situation, recognizing that such classic methods as last observation carried forward (known as LOCF) can bias a trial for or against a drug, depending on the reasons for attrition, the time course of the disease, and response to treatment. Dropout may not be random, as subjects may drop out of either the new drug or the control therapy for toxicity or for lack of efficacy. Depending on the cause of the dropout, the use of modeling approaches might have advantages. 

8.1.2 Reviewing Changes to the Statistical Analysis Plan

Changes to critical elements of the analysis (e.g., the primary endpoint, handling of dropouts) during a trial can raise concerns regarding bias, specifically whether the changes could reflect knowledge of unblinded data. Concerns are inevitably greatest when the change is made late and has an important effect on outcome. In theory, if such changes are unequivocally made blindly (e.g., because of data from other trials or careful reconsideration) they should not pose problems, but the assurance of blinding can be hard to provide. For obvious reasons, changes made with data in hand (but purportedly still blinded) pose the greatest difficulties and are hard to support. 

When changes to the original SAP are proposed during the course of conducting the trial, it is critical to determine exactly what information, if any, regarding trial outcomes was available to those involved in proposing the change. Changes made with knowledge of results can introduce bias that can be substantial and impossible to measure. Note that such biases can occur subtly (e.g., the likelihood of adoption of a proposal made by an individual with no knowledge of data can be influenced by the comments or nonverbal communication of an individual who does have such knowledge). Therefore, major protocol changes are not credible if knowledge of interim outcome data is available to any individual who is involved with those planning the change. If there is any potential for such changes, sponsors should be encouraged to describe fully who has had access to data and how the firewalls were maintained, among other information. 

After trial data collection is completed, and before unblinding, there is often a blinded data cleanup phase. During that phase, previously unaddressed specific concerns about the data may be identified (e.g., types and amounts of missing data, concomitant therapies), and decisions are often made by the sponsor as to how to address those concerns. Typically, any changes made during this data cleanup phase should be minor clarifications of the SAP. If more than minor clarifications are made to the SAP, sponsors should be encouraged to submit these changes to the FDA for review as protocol amendments.

Good Review Practice: Good Review Management Principles and Practices for Effective IND Development and Review

Statistical analysis plan. Submission of a detailed statistical analysis plan (SAP) in the initial protocol submission for phase 3 protocols is not required by CDER regulations. However, review staff should strongly encourage sponsors to include the SAP in the initial protocol submission, because phase 3 protocols generally include a detailed section devoted to statistical methods that are closely linked to trial design.

Good Review Practice: Clinical Review Template

"Reviewers are expected to use the study/clinical trial protocol for discussions on study/clinical trial design and planned efficacy analyses and not the final report itself, because documentation of the study/clinical trial design and the statistical analysis plan within the final report are occasionally incomplete or inaccurate."
"With respect to adequate and well-controlled clinical trials, the reviewer should consider: • Minimization of bias (adequacy of blinding, randomization, endpoint committees, prospective statistical analysis plan, and identification of endpoints) • Choice of control group and the limitations of various choices, especially for historical controls or noninferiority clinical trials, including adequacy of documented effect size for the control drug"

"6.1.5 Analysis of Secondary Endpoint(s) Reviewers should describe the secondary endpoints and their potential supportive role. Was an analysis plan prespecified? Were the secondary endpoints considered for analysis as a hierarchical structure? Should any secondary endpoint be assessed if the primary endpoint fails to achieve statistical significance? "

Pre-specified analyses plan should also describe how the safety data should be analyzed even though the pre-specification for safety analyses are not as critical as the efficacy. Unlike the efficacy analyses where statistical analysis methods vary depending on the study design, the endpoint measure, and other issues, the analysis of safety data is relatively standardized. 

• Reviewer GuidanceConducting a Clinical Safety Review of a New Product Application and Preparing a Report on the Review
• Reviewer Guidance Conducting a Clinical Safety Review of a New Product Application and Preparing a Report on the Review