Protocol Deviations - How is the Protocol Deviation Data Analyzed and Used?

For any clinical trial, the study protocol is the blueprint of how the clinical trial should be conducted. Clinical study protocols must be conducted according to the International Council for Harmonization (ICH) guidance on good clinical practice (GCP), which, among other things, helps safeguard the rights, safety, and well-being of study participants. If conducted as designed, the associated data should be reliable and reproducible and support clear interpretation of the results, while maintaining the participants’ protection. In light of this, one might reasonably assume that deviations from this protocol could be harmful to the participant or the accuracy of the data and should therefore be avoided.

However, the reality is that within clinical trials, protocol deviations do happen, despite best efforts in designing and conducting the clinical trial. According to the ICH E3 Q&A R1, the protocol deviation is defined as: 

"A protocol deviation is any change, divergence, or departure from the study design or procedures defined in the protocol.

Important protocol deviations are a subset of protocol deviations that may significantly impact the completeness, accuracy, and/or reliability of the study data or that may significantly affect a subject's rights, safety, or well-being."

In the early days, the term 'protocol violation' may be used and is still used in some clinical trial documents. According to the ICH E3 Q&A R1, the term 'protocol violation' should be replaced with 'protocol deviation'

"To avoid confusion over terminology, sponsors are encouraged to replace the phrase “protocol violation” in Annex IVa with “protocol deviation”,

There is considerable variability regarding the interpretation, and classification of what an important protocol deviation is, which creates challenges in the identification, collection, and reporting of deviations - resulting in over-reporting PDs that could potentially delay the identification of important patient safety information by increasing the noise in the system or under-reporting PDs that could influence the reliability of the study results and patient safety signals.

In order to reduce the variability in what, when, how the PDs should be collected, it might be helpful to take a look at how the data about the PDs are analyzed and used. 

After the protocol deviation data is collected, the data will be mapped into the CDISC data sets (DV data set in SDTM and ADDV data set in ADaM). Then a data listing will be generated to list all protocol deviations by subjects including the date the protocol deviation occurred, the description of the protocol deviation, the category of the protocol deviation (out of visit window, informed consent issue, SAE safety reporting issue, ......), and the classification of the protocol deviations (minor, major, critical, or important).  A statistical summary table will then be generated for all protocol deviations and for major protocol deviations. 

The protocol deviation data will then be used in the following aspects: 

For the clinical study report (CSR):

CSR must include a section to describe the protocol deviations that occurred during the conduct of the study. According to  ICH E3 (Structure and Content of Clinical Study Reports), the protocol deviations section is an essential part of the CSR. 

10.2 Protocol Deviations 

All important deviations related to study inclusion or exclusion criteria, conduct of the trial, patient managements or patient assessment should be described. 

In the body of the text, protocol deviations should be appropriately summarized by center and grouped into different categories, such as: 

• Those who entered the study even though they did not satisfy the entry criteria. 
• Those who developed withdrawal criteria during the study but were not withdrawn. 
• Those who received the wrong treatment or incorrect dose. 
• Those who received an excluded concomitant treatment. 

In Appendix 16.2.2, individual patients with these protocol deviations should be listed, broken down by center for multicenter studies. 

For FDA BIMO of NDA/BLA submission under Center for Drug Evaluation and Research (CDER):

The FDA Office of Scientific Investigations (OSI) requests that the by-site data and listing are provided for Bioresearch Monitoring (BIMO). The by-site data is used by OSI to select the investigational sites for inspections for CDER submissions.

The FDA Office of Scientific Investigations (OSI) requests that the items described in the draft guidance for industry Standardized Format for Electronic Submission of NDA and BLA Content for the Planning of Bioresearch Monitoring (BIMO) Inspections for CDER Submissions (February 2018) and the associated Bioresearch Monitoring Technical Conformance Guide Containing Technical Specifications be provided to facilitate development of clinical investigator and sponsor/monitor/CRO inspection assignments, and the background packages that are sent with those assignments to the FDA ORA investigators who conduct those inspections. This information is requested for all major
trials used to support safety and efficacy in the application (i.e., phase 2/3 pivotal trials).

One critical piece of the BIMO data is the protocol deviation data by site. The number of protocol deviations or important protocol deviations can be an indicator of protocol compliance and the study quality for specific investigational sites. According to BIORESEARCH MONITORINGTECHNICAL CONFORMANCE GUIDE Technical Specifications Document", the following protocol deviation data should be provided:

"Subject-level data line listings, by clinical site, should include consented subjects, treatment assignment, discontinuations, study population, inclusion and exclusion criteria, adverse events, important protocol deviations, efficacy endpoints, concomitant medications, and safety monitoring, as further described below"

7. Important Protocol Deviations Contains Nonbinding Recommendations 

This by-subject, by-clinical site listing should include all protocol deviations. The listing should include a description of the deviation and identify whether the sponsor considered the deviation to be an important or non-important protocol deviation.  

Clinical Site Data Elements Summary Listing

"Total number of important protocol deviations at a given site by treatment arm for subjects in the SAFPOP. A protocol deviation is any change, divergence, or departure from the study design or procedures defined in the protocol or associated investigational plans that is not implemented or intended as a systematic change. This value should include multiple deviations per subject and all major deviation types. Important deviations are those deviations that might significantly affect the completeness, accuracy, and/or reliability of the study data or that might significantly affect a subject's rights, safety, or well-being"

"Total number of protocol deviations, excluding important protocol deviations, at a given site by treatment arm for subjects in the SAFPOP. A protocol deviation is any change, divergence, or departure from the study design or procedures defined in the protocol or associated investigational plans that is not implemented or intended as a systematic change."

FDA's Good Review Practice: Clinical Review Template listed the protocol violation information as part of the review items for assessing the GCP compliance and the impact of the protocol violations on the data quality and the safety and efficacy results. 

3.2 Compliance With Good Clinical Practices

This section should include comments on compliance with good clinical practices, including informed consent, protocol violations, site-specific issues, and whether the clinical trials were conducted in accordance with acceptable ethical standards. 

If a DSI audit process and report is not requested, provide a brief summary on the quality and nature of other methods used to audit or check the applicant’s data and/or analyses. 

For DSI-requested audits, include a brief summary of the rationale for DSI audits and site selection such as: 

• A specific safety concern at a particular site based on review of adverse events, serious adverse events, deaths, or discontinuation 

• A specific efficacy concern based on review of site-specific efficacy data

 • A specific concern for scientific misconduct at one or more particular sites based on review of financial disclosures, protocol violations, clinical trial discontinuations, or safety and efficacy results  

For defining the per-protocol population for Sensitivity or Supplemental Analyses

Efficacy analyses are usually performed in full analysis set or intention-to-treat population. However, to test the robustness of the statistical results and to assess the impact of the protocol deviations on the statistical results, additional sensitivity or supplemental analyses are usually performed on the per-protocol population. 

According to ICH E9 (Statistical Principles for Clinical Trials), the per-protocol population is defined and sensitivity analyses based on per-protocol population are suggested:  

Per protocol set (valid cases, efficacy sample, evaluable subjects sample): The set of data generated by the subset of subjects who complied with the protocol sufficiently to ensure that these data would be likely to exhibit the effects of treatment according to the underlying scientific model. Compliance covers such considerations as exposure to treatment, availability of measurements, and absence of major protocol violations.

In practice, the per-protocol population is usually defined as "all randomized subjects who have no major protocol deviations" or "all randomized subjects who have no major protocol deviations that may have an impact on the efficacy assessment". The decision on the inclusion of subjects in per-protocol population needs to be decided before the database lock and study unblinding. The protocol deviation data may be reviewed at the blinded data review meeting. Subjects who are excluded from the per-protocol population need to be documented in the blinded data review report.

ICH E9 Addendum also discussed the per-protocol set (PPS) and analyses based on PPS are included as supplemental analysis: 

The meaning and role of an analysis of the per protocol set is also re-visited in this addendum; in particular whether the need to explore the impact of protocol violations and deviations can be addressed in a way that is less biased and more interpretable than naïve analysis of the per protocol set

Section 5.2.3. indicates that it is usually appropriate to plan for analyses based on both the FAS and the Per Protocol Set (PPS) so that differences between them can be the subject of explicit discussion and interpretation.

Additional Reference: Transcelerate Protocol Deviations

Recommendations for DMC: How Many Statisticians Need to be Involved in Successful DMC Process?

Independent data monitoring committee (iDMC) is more commonly used in clinical trials than before for various reasons. For innovative and complex trials, an iDMC is usually a must-to-have external committee. For clinical trials containing the adaptations, interim analyses, iDMC can provide a way to review the unblinded data to recommend some critical suggestions about the ongoing clinical trial while maintaining the study integrity. In some cases, the sponsor just wants to have an independent body to oversee the study for extra credibility. 

There are plenty of regulatory guidelines for the general principles for establishing and operating a DMC. There are also recommendations from the industry/academic for successful DMC implementation. 

• FDA guidance: Establishment and Operation of Clinical Trial Data Monitoring Committees Guidance for Clinical Trial Sponsors
• EMA: GUIDELINE ON DATA MONITORING COMMITTEES
• EMA: Questions and answers on Data Monitoring Committees issues
• RAPs (2020): EMA offers new Q&A on data monitoring committees
• CTTI: Recommendations for data monitoring committees from the Clinical Trials Transformation Initiative
• Annals of Internal Medicine: Presenting Risks and Benefits: Helping the Data Monitoring Committee Do Its Job
• Tips For A Successful Organizational DMC Meeting (wcgclinical.com)

DMC is also mentioned in the recent guidance on Conduct of Clinical Trials of Medical Products During the COVID-19 Public Health Emergency (updated August 2021), COVID-19: Developing Drugs and Biological Products for Treatment or Prevention, and Development and Licensure of Vaccines to Prevent COVID-19. 

"What are some of the key factors that a sponsor should consider when deciding whether to suspend or continue an ongoing study or to initiate a new study during the COVID-19 public health emergency?

Involvement of a study’s DMC, if one has been established, can provide support for the assessments discussed above. Since a primary responsibility of the DMC is assuring the safety of participating trial participants, the DMC’s assessment of the impact of modifications of trial conduct due to COVID-19 on patient safety is important to consider."

When planning for a DMC, multiple statisticians need to be involved - at least three statisticians (blinded and unblinded statisticians) need to play a role in the successful conduct of a DMC. The diagram below lists three different statisticians that are needed for a successful DMC.  

 

DMC Statistician: 

• One of the DMC members
• the voting member
• with biostatistical expertise, applying the statistical methods in the monitoring process. 
• understands the complex clinical trial designs (group sequential design, adaptive design, ...), familiar with the application of the stopping rules, Type I error issue, multiplicity issue. 
• According to FDA's guidance on DMC: "Most DMCs are composed of clinicians with expertise in relevant clinical specialties and at least one biostatistician knowledgeable about statistical methods for clinical trials and sequential analysis of trial data."
• usually unblinded and have access to the unblinded data 

Trial Statistician: 

• clinical trial study team member and can not be the DMC member
• can attend the open session of the DMC meetings; can't attend the closed session of the DMC meetings
• only have access to the blinded or pooled study data during the study
• responsible for the statistical section of the study protocol including the statistical methods for group sequential design, adaptive design, stopping rules... 
• responsible for simulations (if needed) to address questions about the alpha I error spending, multiplicity issue
• maybe the author to develop the statistical analysis plan for DMC and for interim analysis
• provide the blinded data sets to the SDAC (reporting) statistician
• may work with SDAC statistician and review the blinded statistical outputs from the SDAC statistician to ensure the correctness of the statistical package (blinded version)

SDAC Statistician:

• statistician on the Statistical Data Analysis Center (SDAC) - in industry, this is called reporting statistician or independent statistician. 
• non-voting member
• primary responsibility is to prepare the statistical outputs (tables, listings, and figures) for DMC. perform the analyses on unblinded interim data and provide the results of comparative interim analyses.
• Reporting statistician will receive the data sets from the data management group and receive the randomization information (unblinded information) directly from the IRT (interactive response technology) group
• Reporting statistician sends the statistical package (unblinded) to DMC for review. 
• Reporting statistician is usually unblinded.
• reporting statistician may have a programming team to help with generating the data sets, tables, listings, and figures. Anybody who is working with the reporting statistician needs to follow the same rules that are applicable to the reporting statistician. 
• attend the DMC meeting (open and close sessions) and provide explanations for any issues raised by DMC members related to the DMC statistical outputs  

As indicated in the diagram above, the trial statistician and reporting statistician may have close discussions and interactions while working on the blinded data. The reporting statistician and the DMC statistician may also have close interactions while working on the unblinded data. There should be no direct interaction or very little interaction between the trial statistician and the DMC statistician (usually the interaction may be limited to the initial administrative meeting and the open session at the scheduled DMC meetings).

FDA guidance on DMC contains a lengthy section to discuss the reporting statistician's roles in the DMC process. FDA is very concerned about the potential risks that statisticians who conduct the interim analyses may comprise the integrity of the clinical trial. The primary trial statistician should not be the one who conduct the interim analysis. The statistician who conducts the interim analysis should not be in the sponsor's organization.  

Serial Blood Sample Timepoints for Comparing Pharmacokinetics Profiles Between Two Different Dose Frequencies

It is very common in the drug development process that the dosing frequency needs to be studied. The dosing frequency is usually based on the pharmacokinetic profiles. In multiple-dose studies, the dose frequency decides the dosing interval: QD for once daily, BID for twice daily, and TID for three times daily. 

Usually, to compare the pharmacokinetic profiles, serial blood samples will be taken over the period of the dosing interval (between the previous dose and the next dose).  The area under the curve (AUC) will then be calculated over the dosing interval (commonly denoted as tau).

• For QD dose, Tau = 24 hours, AUCtau is AUC[0-24 hours]
• For BID dose, Tau = 12 hours, AUCtau is AUC[0-12 hours]
• For TID dose, Tau = 8 hours, AUCtau is AUC[0-8 hours]

It will be straightforward to select the Pharmacokinetics (PK) sampling time points if two drugs/formulations to be compared have the same dose interval. However, in practice, we often need to compare the PK profiles for two drugs/formulations with different dose intervals, for example, between QD versus BID, or between QD versus TID.  

For comparison of PK profiles between QD dose, BID dose, and TID dose, one will need to compare AUC[0-24 hours] with 2 x AUC[0-12 hours] for BID and with 3 x AUC[0-8 hours] for TID.

For QD dosing, serial PK samples will be collected over 24 hours. For BID dosing and TID dosing, there are two different ways to decide the serial PK samples:

For BID dosing,   

1. serial PK samples can also be collected over 24 hours, the calculated AUC[0-24 hours] can directly be compared with AUC[0-24 hours] from QD dosing even though there will be an extra dose at the middle of the 24 hours period. 
2. serial PK samples can be collected over 12 hours, then the calculated AUC[0-12 hours] needs to be multiplied by 2 in order to be compared with AUC[0-24 hours]

For TID dosing, 

1. serial PK samples can also be collected over 24 hours, the calculated AUC[0-24 hours] can directly be compared with AUC[0-24 hours] from QD dosing even though there will be two extra doses during the 24 hours period. 
2. serial PK samples can be collected over 8 hours, then the calculated AUC[0-8 hours] needs to be multiplied by 3 in order to be compared with AUC[0-24 hours]

In a study by Dawra et al, "A PK/PD study comparing twice-daily to once-daily dosing regimens of ertugliflozin in healthy subjects", the blood samples were collected for QD and BID as the followings:

For each period, blood samples for PK analysis were collected for QD dosing as follows: on days 4, 5, and 6 before administration of the morning dose, and at 0.5, 1, 2, 3, 4, 8, 12, and 24 hours after the morning dose on day 6. For BID dosing, blood samples were collected at 0.5, 1, 2, 3, 4, 8, 12 (preevening dose), 12.5, 13, 14, 15, 16, 20, and 24 hours after the morning dose on day 6.            

Notice that the blood samples for BID were collected at 0.5, 1, 2, 3, 4, 8, 12 post the morning dose and then 0.5, 1, 2, 3, 4, 8, 12 post the evening dose. 

The PK profiles for OD and BID doses are displayed in the figures below: 

This approach of the PK blood sampling schema will require an extensive number of blood draws. For the BID dose, two serial PK samples need to be drawn; for the TID dose, three serial PK samples need to be drawn. The advantage is to capture the potential impact of the circadian and diurnal cycles. 

In some situations, too many blood sample draws may not be practical, and an alternative approach needs to be taken. For example, in pediatric PK studies, the number of blood sample draws may be limited due to the restriction in the total blood volume. 

One of the alternative approaches is to draw the serial PK samples only for one dose interval, not draw additional serial PK samples after the next dose. For example, for the BID dose, the serial PK samples are drawn over 12 hours period assuming that the PK profiles after the evening dose will be the same as the PK profiles over 12 hours post the morning dose. The AUC[0-12 hours] needs to be multiplied by 2 before comparing it to the AUC[0-24 hours] for QD dose. 

In a study to compare the IVIG (every four weeks dosing schedule) and SCIG (weekly dose schedule), the serial PK samples for IVIG were drawn over 4 week period, and the serial PK samples for SCIG were drawn over 1 week period (instead of 4 week period). To compare the AUCs between IVIG and SCIG, the AUC[0-7 days] was multiplied by 4 before comparing it to the AUC[0-28 days]. With this approach, the next three weekly intervals are assumed to have the same PK profiles as the first weekly interval. We can also say that the PK profiles for the next three intervals (dotted lines) are projected or simulated.  This approach was accepted by the FDA and PK results were included in the product label. The PK profiles for IVIG and SCIG are displayed below (the dotted portion for SCIG was projected).