Thursday, December 22, 2016

Control for Type I Error (or Adjustment for Multiplicity) for Secondary Endpoints

In clinical trial protocols, we usually specify one or more primary efficacy endpoints, then a list of secondary efficacy endpoints, and then more tertiary endpoints or exploratory endpoints. It is pretty standard that the primary efficacy endpoints are those the hypothesis testing or inferential statistics and sample size estimation are based on. If we have more than one primary efficacy endpoints, we will need to adjust for multiple tests or multiplicity. It is also clear that the tertiary or exploratory endpoints are for hypothesis generating or more bluntly for fishing expedition.  

Questions we are asked very often are:
  • For secondary efficacy endpoints, do we need to do formal hypothesis testing?
  • If so, do we need to adjust for multiple tests and multiplicity?
  • What is the purpose of controlling the type I error for secondary efficacy endpoints?  

While controlling Family Wise Error Rate (FWER) for primary efficacy endpoints (if multiple test situation exists) is necessary, controlling FWER for secondary efficacy endpoints is often questioned.

Controlling the FWER for secondary efficacy endpoints is valuable or is necessary if we are planning to include the secondary efficacy endpoints in the product label if the product is approved. In other words, if we would like to make the claim of the benefits based on the secondary efficacy endpoints. It is advisable to perform the formal hypothesis testing with controlling for FWER.

In FDA’s guidance for industry “Clinical Studies Section of Labeling for Human Prescription Drug and Biological Products — Content and Format”, there is a statement about the primary and secondary endpoints:
§   Primary and Secondary Endpoints: The terms primary endpoint and secondary endpoint are used so variably that they are rarely helpful. The appropriate inquiry is whether there is a well-documented, statistically and clinically meaningful effect on a prospectively defined endpoint, not whether the endpoint was identified as primary or secondary.
FDA does not care whether or not a study endpoint is called primary or secondary. However, if the information from these endpoints are used for supporting evidence and for product label, the endpoints need to be predefined and tests for these endpoints need to be controlled for the overall alpha or overall type I error.

Even though FDA does not care about the terminology of primary/secondary endpoints, it is still very common in practice that the clinical trials (especially the industry-sponsored clinical trials) specify primary, secondary, and exploratory endpoints.

In a presentation by Kathleen Fritsch from FDA “Multiplicity Issues in FDA-Reviewed Clinical Trials”, the difference between primary, secondary, and exploratory efficacy endpoints are clearly specified and it is stated that the secondary efficacy endpoints may be included in the product label if multiplicity issue is addressed.



It is not needed or rarely needed to control for FWER for exploratory efficacy endpoints.

In EMA’s guidance “Points to Consider on Multiplicity Issues in Clinical Trials”, adjustment for multiplicity is explicitly required if the secondary variables are used for additional claims.





A slide presentation regarding EMA guidance further explained the secondary efficacy endpoints for claims.

In FDA’s guidance on “Clinical Investigations of Devices Indicated for the Treatment of Urinary Incontinence”, the secondary endpoints were called out:

                Secondary Endpoints
      FDA believes secondary endpoint measures, by themselves, are not sufficient to characterize fully the treatment benefit. However, these measures may provide additional characterization of the treatment effect. Specifically, secondary endpoints can:

  • supply background and understanding of the primary endpoints, in terms of overall direction and strength of the treatment effect;
  • be the individual components of a composite primary endpoint, if used;
  • include variables for which the study is underpowered to definitively assess;
  • aid in the understanding of the treatment’s mechanism of action;
  • be associated with relevant sub-hypotheses (separate from the major objective of the treatment); or
  • be used to perform exploratory analyses.

       Assuming that the primary safety and effectiveness endpoints of the study are successfully met, we recommend you analyze the secondary endpoints to provide supportive evidence concerning the safety and effectiveness of the device, as well as to support descriptions of device performance in the labeling. To minimize bias, your protocol should prospectively identify all secondary endpoints, indicating how the data will be analyzed and what success criteria will be applied.
 Secondary Endpoint Analyses
We recommend your protocol prospectively define the statistical plan for performing secondary endpoint analyses in the event that the primary endpoint analysis has been successfully met. If the secondary endpoint analyses are intended purely as exploratory analyses, or are not intended to support the indication for use or device performance, we recommend you submit only simple descriptions of the analyses. If, on the other hand, any of the secondary endpoint analyses are intended to support the indication for use or the performance of your device in the labeling (e.g., comparing treatment and control groups using p-values or confidence intervals), we recommend you pre-specify this intention in your study protocol and describe in detail the statistical methods you plan to follow.
In summary, if we don’t perform the hypothesis testing for these secondary endpoints or not having appropriate control for overall type I error, we will lose the chance to claim the benefit on the secondary endpoints and to list secondary endpoints into the product label. Therefore, it is always a wise decision to perform the hypothesis testing for secondary endpoints with appropriate control for type I error. 

3 comments:

John Judge said...

Enjoy your commentary. I have one question. If a completed Phase 3 Clinical Trial fails but one or more outliers is identified after unblinding that may have affected the primary outcome, could a further statistical review and modified analysis that showed significant differentiation between protocol and placebo change the outcome of the study? Would an FDA submission still be possible with full analysis of the data-set that showed outliers influenced or impacted the primary conclusion?

Web blog from Dr. Deng said...

In clinical trials, outliers may not really be the outliers. it may be the data recording error that can be corrected through the data management.

If they are indeed the outliers, they should be identified before the database lock and before the unblinding of the treatment assignments. if you propose to exclude the 'outliers' from the analysis in the statistical analysis with the statistical analysis plan finalized prior to the study unblinding, it is still possible for FDA to accept. If you exclude the 'outliers' after the study unblinding and after you see the results, it is unlikely for FDA to accept.

John Judge said...

Thank-You for the timely response Dr. Deng.