Should Clinical Trial Protocol be Made Public While the Trial is still ongoing?

Drug companies are under pressure for more and more transparency in clinical trials. It has been a common practice now to post the clinical trial in clinicaltrials.gov before the first patient is enrolled in the study. However, the registration in clinicaltrials.gov includes not all details about the clinical trial. For example, the planned statistical analysis, the interim analysis, stopping rule,... are usually not part of the items to be published. 

Transparency is also implemented after the clinical trial is completed. The clinical trial results need to be published in clinicaltrials.gov within one year of the last efficacy assessment. FDA is implementing a pilot program to encourage drug companies to post the clinical study report (CSR) and the statistical analysis plan (SAP). Some journal such as the New England Journal of Medicine (see an example) requires the study protocol and the SAP to be available as part of the supplemental material to the main publication. 

It is rare to see the study protocol or SAP to be published while the clinical trial is still ongoing. But this happens now for COVID-19 vaccine clinical trials. 

According to StatNews.com:

In an unprecedented show of transparency, the two frontrunners in the race to develop a Covid-19 vaccine released detailed protocols describing the studies testing their vaccines. The unusual step is part of an effort to proactively increase the public's trust in an eventual vaccine.

"This is an example of the comprehensive trial description that industry should be sharing," said Harlan Krumholz, a Yale cardiologist who has been a long-time advocate for increased transparency from drug companies.

The protocols are dense, and researchers will be poring over them in coming days. In both cases, there are plans for potentially stopping the studies early if the vaccines prove more effective than planned. Both protocols require that decision to be suggested by an independent Data Monitoring Committee, not by the companies. And in both cases, this committee reports directly to the company, not to academics who are running the study, something some experts say is disappointing.

The Moderna protocol can be found here "A Phase 3, Randomized, Stratified, Observer-Blind,  Placebo-Controlled Study to Evaluate the Efficacy, Safety, and Immunogenicity of mRNA-1273 SARS-CoV-2 Vaccine in Adults Aged 18 Years and Older"

The Pfizer protocol can be found here "A Phase 1/2/3, Placebo-Controlled, Randomized, Observer-Blind, Dose-Finding Study to Evaluate the Safety, Tolerability, Immunogenicity, and Efficacy of SARS-COV-2 RNA Vaccine Candidates Against COVID-19 in Healthy Individuals"

AstraZeneca followed the suit and had just published its phase III study protocol. "A Phase III Randomized, Double-blind, Placebo-controlled Multicenter Study in Adults to Determine the Safety, Efficacy, and Immunogenicity of AZD1222, a Non-replicating ChAdOx1 VectorVaccine, for the Prevention of COVID-19". See the article "AstraZeneca, Under Fire for Vaccine Safety, Releases Trial Blueprints"

The unprecedented move by three leading COVID-19 vaccine development companies is mainly driven by what happened in their phase III clinical trials. AstraZeneca's trial was paused for data monitoring committee review and then resumed; Pfizer's trial had its sample size increased from the planned 30,000 to 44,000 volunteers; Moderna's trial was slowed down in enrollment in order to ensure minority representation.

These phase III clinical trials for the COVID-19 vaccine are under the public's scrutiny for every move. It is understandable that the public demands transparency to see clinical trial protocols. For regular clinical trials under normal circumstances, it is still not the time for drug companies to make the study protocols public.

WHO - Draft landscape of COVID-19 candidate vaccines - tracking COVID-19 vaccines

World Health Organization keeps tracking the progress of the vaccine candidates against COVID-19 and just published a report called "draft landscape of COVID-19 candidate vaccines". It collected all vaccine candidates already in the clinical trial stage (with links to clinicaltrials.gov registries) and in the pre-clinical stage. 

"These landscape documents have been prepared by the World Health Organization (WHO) for information purposes only concerning the 2019-2020 global of the novel coronavirus. Inclusion of any particular product or entity in any of these landscape documents does not constitute, and shall not be deemed or construed as, any approval or endorsement by WHO of such product or entity (or any of its businesses or activities). While WHO takes reasonable steps to verify the accuracy of the information presented in these landscape documents, WHO does not make any (and hereby disclaims all) representations and warranties regarding the accuracy, completeness, fitness for a particular purpose (including any of the aforementioned purposes), quality, safety, efficacy, merchantability and/or non-infringement of any information provided in these landscape documents and/or of any of the products referenced therein. WHO also disclaims any and all liability or responsibility whatsoever for any death, disability, injury, suffering, loss, damage or other prejudice of any kind that may arise from or in connection with the procurement, distribution or use of any product included in any of these landscape documents."

We anticipate that this draft landscape will be updated periodically to reflect the latest development in COVID-19 vaccine front. The current report can be downloaded at the web link here. It showed that 9 COVID-19 vaccine candidates are in the phase III trial stage now.  

Finkelstein-Schoenfeld Method, Win Ratio, and Hodges-Lehman Estimates - Statistical Methods Based on All Paired Comparisons

Finkelstein-Schoenfeld methods can be used in analyzing the data with a composite endpoint where different components for the composite endpoint have different levels of importance. Hodges-Lehmann estimate is used to estimate the magnitude of treatment difference in a non-parametric statistical test such as the Wilcoxon Rank Test. What is in common between these two methods? Well, both methods are based on the pairwise comparisons - the value/outcome from each subject in treatment group A is compared to each of all subjects in treatment group B - in other words, both methods are based on n (# of subjects in treatment group A) time m (# of subjects in treatment group B) comparisons. 

In clinical trials for serious conditions, but not deadly enough, a composite endpoint is often used as the primary efficacy endpoint. The composite endpoint usually consists of several categories (or components) with different degrees of importance because there will not be enough events for a single category for a feasible clinical trial. The examples of composite endpoints are: 

• a composite endpoint in heart failure may include death, hospitalization, and clinical status
• a composite endpoint in pulmonary arterial hypertension may include death, hospitalization, and disease progression; 
• a composite endpoint in cardiovascular outcome study may be the major adverse cardiovascular events (MACE) consisting of death; MI; stroke, hospitalization.

Usually, these different components are not weighted and treated as equally important and the statistical analyses are based on the time to first event (no matter if the first event is death, hospitalization, or others) - this approach of no weighting is the focal point being criticized. 

Finkelstein-Schoenfeld method is a non-parametric method aiming to bring the weighting into the analysis of the composite endpoints. Finkelstein-Schoenfeld's method was named after their paper in 1999 in Statistics in Medicine "Combining Mortality and Longitudinal Measures in Clinical Trials".  The method was a generalization of the Gehan‐Wilcoxon test based on pairwise comparison of patients on a primary outcome when possible but otherwise on a secondary outcome. The Finkelstein-Schoenfeld method was originally proposed for "analyzing the impact of treatment which combines a (possibly censored) event with a longitudinal measure of clinical effect", not explicitly for analyzing the composite endpoint. 

Based on the Finkelstein-Schoenfeld method, Pocock and colleagues suggested an estimate, the Win Ratio, which summarized the ratio of the number of patients who fared better versus worse on the experimental arm. The Win-Ratio method was proposed explicitly for analyzing the composite endpoint (Pocock et al 2012) "The win ratio: a new approach to the analysis of composite endpoints in clinical trials based on clinical priorities".

With Finkelstein-Schoenfeld or Win-ratio method, pairwise comparisons are performed and the scores are calculated based on the comparison of the importance of the outcome. For example, for a study with composite endpoint including death and hospitalization, all patients had multiple pairwise comparisons performed, first with respect to time to death and to hospitalization, if the latter occurred. 

Below are some additional references discussing the Finkelstein-Schoenfeld or Win-Ratio method and their applications.  

• Finkelstein (2014) "Introduction: Challenges in Analyzing and Interpreting Multiple (Censored) Outcomes in Chronic Disease Trials"
• A novel test to compare two treatments based on endpoints involving both nonfatal and fatal events
• Exact Bootstrap and Permutation Distribution of Wins and Losses in a Hierarchical Trial∗
• Paul Brown's Ph.D. Dissertation (2017) Multiple Outcomes in Heart Failure Research: Composite Endpoints and Multivariate Modelling
• Sun et al (2012) Evaluating Treatment Efficacy by Multiple End Points in Phase II Acute Heart Failure Clinical Trials Analyzing Data Using a Global Method
• Svensson et al "A comprehensive review of the PARTNER trial"
• Graphing the Win Ratio and its components over time

There are several pivotal studies where the Finkelstein-Schoenfeld method is used to analyze the primary efficacy endpoint. The study protocol and statistical analysis plan posted online contain the detail descriptions about the application of the Finkelstein-Schoenfeld method. 

• EXCEL Trial ""Clinical Trial Evaluation of XIENCE versus Coronary Artery Bypass Surgery for Effectiveness of Left Main Revascularization" - Study Protocol;  Statistical Analysis Plan; 
• ATTR-ACT Trial "A Multicenter, International, Phase 3, Double-blind, Placebo-Controlled, Randomized Study to Evaluate the Efficacy, Safety, and Tolerability of Daily Oral Dosing of Tafamidis Meglumine 20 mg or 80 mg in Comparison to Placebo in Subjects Diagnosed with Transthyretin Cardiomyopathy (TTR-CM)" - Study Protocol; Statistical Analysis Plan
• THE PARTNER TRIAL: Placement of AoRTic TraNscathetER Valve Trial Edwards SAPIEN Transcatheter Heart Valve. -  Article (Makkar 2012);  Study Protocol from FDA; Study Protocol/SAP

In the protocol / statistical analysis plan for the Partner trial, there are the following descriptions for the Finkelstein-Schoenfeld method: 

In PARTNER Trial was the basis for FDA approval of Vyndaqel and Vyndamax and Finkelstein and Schoenfeld's method was mentioned in the product label. 

"The primary analysis used a hierarchical combination applying the method of Finkelstein-Schoenfeld (F-S) to all-cause mortality and frequency of cardiovascular-related hospitalizations, which was defined as the number of times a subject was hospitalized (i.e., admitted to a hospital) for cardiovascular-related morbidity. The method compared each patient to every other patient within each stratum in a pair-wise manner that proceeded in a hierarchical fashion using all-cause mortality followed by frequency of cardiovascular-related hospitalizations when patients could not be differentiated based on mortality."

Hodges-Lehmann estimate is used in totally different situations, but similar to the Finkelstein-Schoenfeld method, the estimate relies on the pairwise comparison. While the Finkelstein-Schoenfeld method is primarily used in the analysis of composite endpoint,  Hodges-Lehmann estimate is mainly used to obtain the treatment difference for a continuous variable with normality assumption violation and non-parametric method being used.

 

With the Hodges-Lehmann method, the treatment difference is calculated for each pair for total n x m pairs (where n and m are the # of subjects in each treatment group). The Hodges-Lehmann estimate is the median of differences from all pairs.  

• "Hodges-Lehmann estimator of location shift: Median of Differences versus Difference in Medians or Median Difference"
• Splitting p-value and estimate of the treatment difference
• Hodges-Lehmann Estimator

Hodges-Lehmann estimate has been used in many clinical trials that result in FDA approval of the products. For example, Hodges-Lehmann estimate was the method used in the SIROCCO trial in Asthma. The FDA statistical review document stated the primary analysis method of the study: 

"The primary analysis for the OCS percent reduction endpoint used the Wilcoxon rank-sum test approach. The primary analyses were performed in the FAS population. For each of the two Benralizumab dose regimen groups, the median difference in the OCS percent reduction between Benralizumab dose regimen and placebo was derived using asymptotic Hodges-Lehmann estimation, together with associated 95% CI and p-value. The same analyses were also performed for the EHS without multiplicity control."