The fate of confirmatory clinical trials for Remdesivir for treatment of COVID-19

Remdesivir, as the first highly touted drug to treat COVID-19 patients, has now been approved for emergency use authorization in several countries. The focus of fighting COVID-19 seems to shift to the safe and effective vaccine development. Unfortunately, the efficacy of Remdesivir has not been confirmed due to the flaws in the study design (for example, no placebo control) or due to the issues in the study conduct (for example, early discontinuation of the study resulted in underpowered studies) .

• Despite the Hype, Gilead’s Remdesivir Will Do Nothing to End the Coronavirus Pandemic
• Remdesivir alone is not enough, researchers conclude in first major Covid-19 trial of the drug
• Data Monitoring for the Chinese Clinical Trials of Remdesivir in Treating Patients with COVID-19 During the Pandemic Crisis
• Gilead to end coronavirus drug trials, adding to access worry: researchers

In a previous post, six pivotal studies were listed for Remdesivir. These six studies were also listed in the article by Singh et al Remdesivir in COVID-19: A critical review of pharmacology, pre-clinical and clinical studies.

The table below listed the status (fate) of these studies.

Protocol Title	Study Features	Fate of Studies
A Phase 3 Randomized Study to Evaluate the Safety and Antiviral Activity of Remdesivir (GS-5734™) in Participants With Severe COVID-19	Gilead Sciences Phase III, initially planned for 400 subjects, then increased to 2400, and then 6000 subjects Two  arms: Standard of care + Remdesivir for 5 days, Standard of care + Remdesivir for 10 days	Enrolment was stopped early after 397 subjects were randomized.   Results were published in NEJM (Goldman et al “Remdesivir for 5 or 10 Days in Patients with Severe Covid-19“ Conclusion: “In patients with severe Covid-19 not requiring mechanical ventilation, our trial did not show a significant difference between a 5-day course and a 10-day course of remdesivir. With no placebo control, however, the magnitude of benefit cannot be determined.” It is noted that the study design was flawed and should have included a third arm with Standard of Care without Remdesivir. The data from this study was further compared to the external control group and results were announced in a recent press release "Comparative Analysis of Phase 3 SIMPLE-Severe Study and Real-World Retrospective Cohort of Patients Diagnosed with Severe COVID-19 Receiving Standard of Care" to show the statistically significant reduction in mortality in Remdesivir group.
A Phase 3 Randomized Study to Evaluate the Safety and Antiviral Activity of Remdesivir (GS-5734™) in Participants With Moderate COVID-19 Compared to Standard of Care Treatment	Gilead Sciences Phase III, initially planned for 600 subjects, then increased to 1600 subjects Three arms: Remdesivir for 5 days, Remdesivir for 10 days, Standard of care	The study is active, but not recruiting new patients. The enrolment has stopped. The results have not been published yet. Gilead announces results in a press release “Gilead Announces Results From Phase 3 Trial of Remdesivir in Patients With Moderate COVID-19” Study Demonstrates 5-Day Treatment Course of Remdesivir Resulted in Significantly Greater Clinical Improvement Versus Treatment with Standard of Care Alone Data Add to Body of Evidence from Prior Studies Demonstrating Benefit of Remdesivir in Hospitalized Patients with COVID-19 The results were later published in JAMA "Effect of Remdesivir vs Standard Care on Clinical Status at 11 Days in Patients with Moderate Covid-19" Also see the articles “More evidence remdesivir helps some coronavirus patients”  “Gilead's remdesivir shows modest improvement in moderate COVID-19 patients” “A slice of half-baked remde­sivir da­ta takes a dark turn — but we're still wait­ing for a de­fin­i­tive study”
A Phase 3 Randomized, Double-blind, Placebo-controlled Multicenter Study to Evaluate the Efficacy and Safety of Remdesivir in Hospitalized Adult Patients With Mild and Moderate 2019-nCoV Respiratory Disease.	Capital Medical University/Chinese Academy of Medical Sciences Phase III, 308 Subjects Two arms: Remdesivir, placebo Mainland China only	The study was suspended (The epidemic of COVID-19 has been controlled well at present, no eligible patients can be recruited).  The results have not been published yet.  See “Second trial testing Gilead's remdesivir in COVID-19 suspended in China”
A Phase 3 Randomized, Double-blind, Placebo-controlled, Multicenter Study to Evaluate the Efficacy and Safety of Remdesivir in Hospitalized Adult Patients With Severe 2019-nCoV Respiratory Disease.	Capital Medical University Phase III, 453 Subjects Two arms: Remdesivir, placebo Mainland China only	The study was terminated after 237 patients were enrolled and randomly (158 in remdesivir and 79 in placebo) (The epidemic of COVID-19 has been controlled well in China, no eligible patients can be enrolled at present.) Results were published at Lancet Wang et al “Remdesivir in adults with severe COVID-19: a randomised, double-blind, placebo-controlled, multicentre trial” Conclusion: “In this study of adult patients admitted to hospital for severe COVID-19, remdesivir was not associated with statistically significant clinical benefits. However, the numerical reduction in time to clinical improvement in those treated earlier requires confirmation in larger studies.”
A Multicenter, Adaptive, Randomized Blinded Controlled Trial of the Safety and Efficacy of Investigational Therapeutics for the Treatment of COVID-19 in Hospitalized Adults	National Institute of Allergy and Infectious Diseases (NIAID) Phase II, planned 440 Subjects (protocol specified 394 subjects), actual enrolment: 1063 subjects at the time of DMC review) Two arms: Placebo, Remdesivir with additional arms to be added Multi-National: US, Japan, South Korea, Singapore	The study was stopped after the interim analyses.  Preliminary results were published in NEJM by Beigel et al. Remdesivir for the Treatment of Covid-19 - Preliminary Report Conclusion: “Remdesivir was superior to placebo in shortening the time to recovery in adults hospitalized with Covid-19 and evidence of lower respiratory tract infection.” The results from this study were the basis for FDA to issue Emergency Use Authorization for Remdesivir. Subsequently, several other countries followed suit. It is disappointing that the study was stopped after inconclusive or not convincing results from the interim analyses. There was no mention if there was a pre-specified stopping rule and whether the boundaries for stopping the study had been crossed.  Also see: Inside the NIH’s controversial decision to stop its big remdesivir study The final report was later published in NEJM (on Oct 8). Results were better than those reported in the preliminary report (median recovery time was shorten by 5 days). The final report concluded "Our data show that remdesivir was superior to placebo in shortening the time to recovery in adults who were hospitalized with Covid-19 and had evidence of lower respiratory tract infection."
Multi-centre, Adaptive, Randomized Trial of the Safety and Efficacy of Treatments of COVID-19 in Hospitalized Adults	Institut National de la Santé Et de la Recherche Médicale, France Phase III, 3100 Subjects Four arms: Remdesivir, Lopinavir/ritonavir, Interferon Beta-1A, Hydroxychloroquine, Standard of care France Only	This study is funded by WHO and is called DIsCoVeRy in clinicaltrials.gov and SOLIDARITY trial in ISRCTN registration. The interim results were published in a paper "Repurposed antiviral drugs for COVID-19; interim WHO SOLIDARITY trial results". It concludes "These Remdesivir, Hydroxychloroquine, Lopinavir and Interferon regimens appeared to have little or no effect on hospitalized COVID-19, as indicated by overall mortality, initiation of ventilation and duration of hospital stay. The mortality findings contain most of the randomized evidence on Remdesivir and Interferon, and are consistent with meta-analyses of mortality in all major trials." The results were disputed by the manufacturer of Remdesirvir Gilead.

Randomized Controlled Trial in Sheep and Methodological Rigor in Preclinical Studies

Miller et al published a paper in blue journal (AJRCCM) "Combined Mesenchymal Stromal Cell Therapy and Extracorporeal Membrane Oxygenation in Acute Respiratory Distress Syndrom: A Randomized Controlled Trial in Sheep." At the first glance, I thought it was a randomized controlled clinical trial. Then I realized it was a randomized controlled trial in sheep and the word 'clinical' was not in the title. Whether or not the study is conducted in humans or in sheep, it can still be called 'randomized controlled trial' or RCT in short. 

It is great to see that the pre-clinical studies are conducted in a way with scientific rigor. The results from RCT in animals will be more reliable and provide more definitive evidence for us to decide if additional RCT in humans should be warranted. 

The presentation of the paper by Miller et al followed exactly the same way how a randomized controlled clinical trial will be presented. 

Study Design

Ethical approvals were obtained from University Animal Ethics Committees of Queensland University of Technology and the University of Queensland and authorization for in vivo use of hMSCs was granted by the Australian Department of Agriculture. The study was conducted in accordance with Australian Code for the Care and Use of Animals for Scientific Purposes and is reported in compliance with Animal Research: Reporting of In Vivo Experiments guidelines. ......

Statistical Analysis

An a prior sample size calculations, based on the primary outcome of PaO2/FiO2, ratio at 24 hours, is detailed in the online supplement. Data are expressed as mean (+/-SD) or median (interquartile range [IQR]) if nonnormally distributed. Analysis was undertaken in Graphpad Prism. Longitudinal data were analyzed by fitting a mixed model. This model uses a compound symmetry covariance matrix and is fit using restricted maximum likelihood. Where a significant interaction was observed, post hoc comparisons were undertaken. Correction for multiple comparisons was made using the Benjamini-Hochberg method (false discovery rate restricted to 5%). ......

Last July, I attended a symposium in Paris and Dr. Sébastien Bonnet from Université Laval in Canada from gave a presentation titled "Improving the rigour of preclinical studies to identify promising therapies". His presentation resonated with me.

Preclinical studies are usually small in sample size, conducted at a single institute, not randomized, and not reproducible. For years, there has been a push for improving the methodological rigor in preclinical studies. The methods used in the design and analysis of clinical trials can also be used in pre-clinical studies. We hope to see the terms (such as randomized, controlled, multi-center, meta-analysis) applied in more pre-clinical studies.   

• Methodological Rigor in Preclinical Cardiovascular Studies - Targets to Enhance Reproducibility and Promote Research Translation
• Standards and Methodological Rigor in Pulmonary Arterial Hypertension Preclinical and Translational Research
• Principles and Guidelines for Reporting Preclinical Research
• A call for transparent reporting to optimize the predictive value of preclinical research
• Limitations and Flaws of Preclinical Pulmonary Hypertension Studies

Flawed preclinical studies can produce misleading results that may be used as the basis for clinical trials. A lot of efforts and money be spent on clinical trials that are based on the results from shady preclinical studies. 

Covid-19 Vaccine: Is 50% Vaccine Efficacy (VE) Too Low?

Ever since FDA issued its guidance “Development and Licensure of Vaccines to Prevent COVID-19,” to help facilitate timely development of safe, effective COVID-19 Vaccines, the question arose whether the threshold of 50% vaccine efficacy (VE) was set too low.

• FDA issues COVID-19 vaccine guidance, setting 50% effectiveness threshold
• A COVID-19 Vaccine May Be Only 50% Effective. Is That Good Enough?
• COVID-19 vaccine trials should seek worthwhile efficacy

As cited in an article “FDA to Require 50 Percent Efficacy for COVID-19 Vaccines”:

Gregory Poland, the director of the Mayo Vaccine Research Group, tells Reuters the efficacy guidelines are standard compared to other vaccines. “They look pretty much like influenza vaccine guidelines,” Poland says. “I don’t think that’s a high bar. I think that’s a low to . . . appropriate bar for a first-generation COVID-19 vaccine.” The effectiveness of the annual flu shot, for example, generally ranges between 40 percent and 60 percent, according to The Washington Post.

Peter Hotez, a vaccine expert at the Baylor College of Medicine, tells the Post the 50 percent threshold is low, a sign that the FDA recognizes “our first vaccine won’t be our best.” Ultimately, he says, vaccine developers should aim for 70–75 percent efficacy. 

If we just look at the face meaning of the 50% VE, it does look like the bar is low. Some people may interpret 50% VE as that the COVID-19 vaccine will only need to be effective in 50% of people – which is not true. Even in the FDA’s announcement about the issuance of its guidance, the statement about the requirement of 50% VE was incorrectly stated:

“The guidance also discusses the importance of ensuring that the sizes of clinical trials are large enough to demonstrate the safety and effectiveness of a vaccine. It conveys that the FDA would expect that a COVID-19 vaccine would prevent disease or decrease its severity in at least 50% of people who are vaccinated.”

Let’s see how the vaccine efficacy is calculated and what the 50% VE means.

According to Wikipedia, Vaccine efficacy (VE) is the percentage reduction of disease in a vaccinated group of people compared to an unvaccinated group, using the most favorable conditions.

The outcome data (vaccine efficacy) generally are expressed as a proportionate reduction in disease attack rate (AR) between the unvaccinated (ARU) and vaccinated (ARV), or can be calculated from the relative risk (RR) of disease among the vaccinated group.

The basic formula is written as:

VE=(ARU-ARV) / ARU * 100%

with

VE = Vaccine efficacy,

ARU = Attack rate of unvaccinated people,

ARV = Attack rate of vaccinated people.

An alternative, equivalent formulation of vaccine efficacy

VE=1-RR

where RR is the relative risk of developing the disease for vaccinated people compared to unvaccinated people.

In the actual calculation of VE, we will need to consider the total exposure time (usually measured by the total person-time). One person observed for one year = 1 person-year; one person observed for 3 months = 0.25 person year. the total person-time in year (or total person-years) will be the summation of person-years across all participants in the vaccine group and similarly across all participants in the placebo group. 

The point estimate of the VE can be written as:

If the clinical trial has a 1:1 randomization ratio (all participants are randomized equally into the vaccine group and the placebo group), the 'total person-time' in the vaccine group will be approximately equal to the 'total person-time' in the placebo group, the point estimate of VE can then be easily calculated as:

 

If we know the number of cases (here COVID-19 cases) in the vaccine group and in the placebo group, we can easily calculate the VE.  For example, if the total number of cases is 150 (50 cases observed in the vaccine group and 100 cases observed in the placebo group), the VE will be 1 - (50/100) = 0.5 = 50%.

If an interim analysis is performed after a total of 75 cases are observed, VE will be 50% if 25 cases are observed in the vaccine group and 50 cases are observed in the placebo group. 

Here is a comparison of VE calculations from three Phase III protocols of COVID-19 vaccines:

 

	Moderna	Pfizer	AstraZeneca
Primary efficacy endpoint	VE will be estimated with 1 - HR (mRNA-1273 vs placebo) using a Cox proportional hazard regression model with treatment group as a fixed effect and adjusting for stratification factor	VE will be estimated by 100 × (1 – IRR), where IRR is the  calculated ratio of confirmed COVID-19 illness per 1000 person-years follow-up in the active vaccine group to the corresponding illness rate in the placebo group 7 days after the last dose.	VE is calculated as RRR = 100*(1-relative risk), which RRR is the incidence of infection in the vaccine group relative to the incidence of infection in the control group expressed as a percentage.
Statistical model for calculating the VE and its 95% confidence interval	Cox proportional hazard model	Beta-binomial model	Modified Poisson regression model with robust variance
Sample size (number of volunteers to be recruited)	30,000	43,998	33,000
Number of cases needed to be observed	151	164	150

With the COVID-19 pandemic is still not under control in the US and a large number of volunteers participating in these phase III clinical trials, we hope that the total number of COVID-19 cases can be easily reached so that we can have a readout about the vaccine's efficacy. All three studies have included at least one interim analysis to have a possible readout much earlier. 

In addition to the requirement of at least 50% VE, FDA guidance also requires that the lower bound of the 95% confidence interval of the VE must be greater than 30%. 

The sample size (the number of COVID-19 infection cases) is largely dictated by this criterion of 30% for the lower bound of 95% CI. Otherwise, with 3 cases (1 case in the vaccine group and 2 cases in the placebo group), we would have a point estimate of VE = 50% to meet the requirement. 

50% VE implies that the vaccine can decrease the risk of COVID-19 cases by 50%. Comparing with other clinical trials, the 50% reduction is substantial and meaningful. In Moderna's trial, the VE will be estimated using the Cox proportional hazard model where the time to the first case of COVID-19 infection is also considered. In order to meet the criteria of at least 50% VE, the estimated hazard ratio (HR) needs to be equal to or less than 0.5. In oncology trials or in other clinical trials with time to event variables, if we can have an HR of 0.5 or lower, we will claim that the experimental treatment can reduce the risk of death or event by at least 50% - a result to die for. 

I agree with the statement about the COVID-19 vaccine from a Lancet paper:

“A vaccine that has 50% efficacy could appreciably reduce incidence of COVID-19 in vaccinated individuals, and might provide useful herd immunity. Hence, although efficacy far greater than 50% would be better, efficacy of about 50% would represent substantial progress.”

SARS-CoV-2 (the virus causing COVID-19) has a very high R0 (2.5 according to the table below) which estimates the speed at which a disease is capable of spreading in a population. We hope that we will have a vaccine that will meet the efficacy requirements of at least 50% VE in point estimate and at least 30% VE in the lower bound of 95% confidence interval. With an effective vaccine and the majority of people being vaccinated, we may be able to drop the transmissibility R0 below 1 to prevent the spread of the SARS-CoV-2.

     Comparing SARS-CoV-2 with SARS-CoV and influenza pandemics

Should We Follow ICH E9 Addendum to Include the Estimands in all Clinical Trial Protocols?

ICH E9 "Statistical Principles for Clinical Trials" was issued in 1998 - more than 20 years ago. While the principles specified in ICH E9 are still being followed, a call for a revision or addendum has been there for many years. In 2017, the draft version of ICH E9 (R1) “Addendum on Estimands and Sensitivity Analysis in Clinical Trials to the Guideline on Statistical Principles for Clinical Trials” was released and at the end of 2019, ICH E9 (R1) was finalized. The E9 (R1) guidelines are now gradually been adopted by various regulatory agencies. In terms of the implementation, EMA seems to be ahead of the US requiring the sponsors to include the concept of Estimands in the regulatory submissions. 

Purpose and scope of the addendum to ICH E9:

• Provides a framework for describing with precision a treatment effect of interest
• Precision in describing a treatment effect of interest is facilitated by constructing the “estimand”
• Estimand: A precise description of the treatment effect reflecting the clinical question posed by the trial objective. It summarises at a population-level what the outcomes would be in the same patients under different treatment conditions being compared
• Clarity requires a thoughtful envisioning of “intercurrent events” such as discontinuation of assigned treatment, use of additional or alternative treatment, and terminal events such as death
• Intercurrent Events: Events occurring after treatment initiation that affect either the interpretation of the existence of the measurements associated with the clinical question of interest
• It is necessary to address intercurrent events when describing the clinical question of interest in order to precisely define the treatment effect that is to be estimated
• Addendum introduces strategies to reflect different questions of interest that might be posed
• Attributes used to construct the estimand are also introduced in the addendum
• Addendum clarifies the definition and the role of sensitivity analysis
• Sensitivity Analysis: A series of analyses conducted with the intent to explore the robustness of inferences from the main estimator to deviations from its underlying modeling assumptions and limitations in the data

Estimand attributes:

• Treatment: The treatment condition of interest and, as appropriate, the alternative treatment condition to which comparison will be made
• Population: Patients targeted by the clinical question
• Variable (or endpoint): Obtained for each patient and required to address the clinical question
• Population-level summary: Provides a basis for comparison between treatment conditions for the variable
• Handling of intercurrent events

While FDA has not mandated the implementation of the ICH E9 (R1), there seems to be a trend in the industry that ICH E9 (R1) is gradually being adopted and the concept ‘estimands’ is being mentioned in the study protocols and statistical analysis plans (SAPs). 

Looking at the clinical trial protocol and SAP templates developed by TransCelerate Biopharma, both contained a section about estimands and the estimands was listed together with endpoints.

In my previous post "Should Clinical Trial Protocol be Made Public While the Trial is still ongoing?", I mentioned that the protocols for three phase III studies for Covid-19 vaccine were all made public because of the demand for transparency. I examed all three protocols and they had described the 'estimands'. The concept of 'intercurrent events', 'principal stratum strategy', 'treatment policy' was also mentioned. 

In Phase III study protocol by Moderna "A Phase 3, Randomized, Stratified, Observer-Blind, Placebo-Controlled Study to Evaluate the Efficacy, Safety, and Immunogenicity of mRNA-1273 SARS-CoV-2 Vaccine in Adults Aged 18 Years and Older":

In the estimand of the primary analysis on the primary endpoint, a treatment policy strategy will be used to address the intercurrent events of 1) withdrawal from the study or death unrelated to COVID-19, where the time to COVID-19 will be censored at the date of withdrawal from the study or death; 2) early COVID-19, where the time to COVID-19 will be censored at the time of early infection. Principal stratum strategy will be used to address the other 2 types of intercurrent events in the primary analysis based on the PP Set. The details of intercurrent event description and estimand strategies are presented in Section 11.4.1.

In Phase III study protocol by AstraZeneca / Oxford "A Phase III Randomized, Double-blind, Placebo-controlled Multicenter Study in Adults to Determine the Safety, Efficacy, and Immunogenicity of AZD1222, a Non-replicating ChAdOx1 VectorVaccine, for the Prevention of COVID-19":

The primary estimand will be used for the analysis of the primary efficacy endpoint. It will be based on participants in the full analysis set, defined as all randomized participants who received at least 1 dose of study intervention excluding those participants who are seropositive at baseline, analyzed according to their randomized treatment. For participants with multiple events, only the first occurrence will be used for the primary efficacy endpoint analysis. The set of intercurrent events for this estimand consists of participants who withdraw from the study prior to having met the primary efficacy endpoint. The intercurrent events will be handled using the treatment policy strategy and the absence of data following these participants’ withdrawal will be treated as missing (ie, counted as not having met the criteria). Participants who withdraw before 15 days post second dose or who have a case prior to 15 days post second dose will be excluded from primary endpoint analysis.
Additional estimands will be specified for the primary efficacy endpoint to carry out sensitivity analyses for assessing the robustness of results. These sensitivity analyses will explore different methods for handling intercurrent events and different assumptions for missing data. Estimands will also be specified for the analysis of secondary endpoints. Full details will be provided in the SAP.

In Phase III study protocol by Pfizer/BioNtech "A Phase 1/2/3, Placebo-Controlled, Randomized, Observer-Blind, Dose-Finding Study to Evaluate the Safety, Tolerability, Immunogenicity, and Efficacy of SARS-COV-2 RNA Vaccine Candidates Against COVID-19 in Healthy Individuals": 

It looks like that the concept of 'estimands' has not been widely accepted by the medical community - it is indeed a new concept and a new term for clinical trialists to digest. Using a most recent paper in the New England Journal of Medicine (Rabe et al 2020 "Triple Inhaled Therapy at Two Glucocorticoid
Doses in Moderate-to-Very-Severe COPD"), the main body of the article had no mention of the concept of 'estimands' even though the attached protocol and SAP contained a section about 'estimands':

In the phase III study by J&J (Janssen) "A Randomized, Double-blind, Placebo-controlled Phase 3 Study to Assess the Efficacy and Safety of Ad26.COV2.S for the Prevention of SARS-CoV-2-mediated COVID-19 in Adults Aged 18 Years and Older", there was no mention of estimands. 

Additional References: 

• US FDA and Health Canada Joint Regional Consultation on the International Council for Harmonisation (ICH)
• University of Pennsylvania 11th annual conference on statistical issues in clinical trials: Estimands, missing data and sensitivity analysis (morning panel session)
• Estimand Framework: What it is and Why You Need it
• Estimands in clinical trials – broadening the perspective
• PSI, EFSPI & ASA BIOP Webinar: Estimands
• Estimand Framework in Oncology Drug Development: Challenges and Opportunities

Should Clinical Trial Protocol be Made Public While the Trial is still ongoing?

Drug companies are under pressure for more and more transparency in clinical trials. It has been a common practice now to post the clinical trial in clinicaltrials.gov before the first patient is enrolled in the study. However, the registration in clinicaltrials.gov includes not all details about the clinical trial. For example, the planned statistical analysis, the interim analysis, stopping rule,... are usually not part of the items to be published. 

Transparency is also implemented after the clinical trial is completed. The clinical trial results need to be published in clinicaltrials.gov within one year of the last efficacy assessment. FDA is implementing a pilot program to encourage drug companies to post the clinical study report (CSR) and the statistical analysis plan (SAP). Some journal such as the New England Journal of Medicine (see an example) requires the study protocol and the SAP to be available as part of the supplemental material to the main publication. 

It is rare to see the study protocol or SAP to be published while the clinical trial is still ongoing. But this happens now for COVID-19 vaccine clinical trials. 

According to StatNews.com:

In an unprecedented show of transparency, the two frontrunners in the race to develop a Covid-19 vaccine released detailed protocols describing the studies testing their vaccines. The unusual step is part of an effort to proactively increase the public's trust in an eventual vaccine.

"This is an example of the comprehensive trial description that industry should be sharing," said Harlan Krumholz, a Yale cardiologist who has been a long-time advocate for increased transparency from drug companies.

The protocols are dense, and researchers will be poring over them in coming days. In both cases, there are plans for potentially stopping the studies early if the vaccines prove more effective than planned. Both protocols require that decision to be suggested by an independent Data Monitoring Committee, not by the companies. And in both cases, this committee reports directly to the company, not to academics who are running the study, something some experts say is disappointing.

The Moderna protocol can be found here "A Phase 3, Randomized, Stratified, Observer-Blind,  Placebo-Controlled Study to Evaluate the Efficacy, Safety, and Immunogenicity of mRNA-1273 SARS-CoV-2 Vaccine in Adults Aged 18 Years and Older"

The Pfizer protocol can be found here "A Phase 1/2/3, Placebo-Controlled, Randomized, Observer-Blind, Dose-Finding Study to Evaluate the Safety, Tolerability, Immunogenicity, and Efficacy of SARS-COV-2 RNA Vaccine Candidates Against COVID-19 in Healthy Individuals"

AstraZeneca followed the suit and had just published its phase III study protocol. "A Phase III Randomized, Double-blind, Placebo-controlled Multicenter Study in Adults to Determine the Safety, Efficacy, and Immunogenicity of AZD1222, a Non-replicating ChAdOx1 VectorVaccine, for the Prevention of COVID-19". See the article "AstraZeneca, Under Fire for Vaccine Safety, Releases Trial Blueprints"

The unprecedented move by three leading COVID-19 vaccine development companies is mainly driven by what happened in their phase III clinical trials. AstraZeneca's trial was paused for data monitoring committee review and then resumed; Pfizer's trial had its sample size increased from the planned 30,000 to 44,000 volunteers; Moderna's trial was slowed down in enrollment in order to ensure minority representation.

These phase III clinical trials for the COVID-19 vaccine are under the public's scrutiny for every move. It is understandable that the public demands transparency to see clinical trial protocols. For regular clinical trials under normal circumstances, it is still not the time for drug companies to make the study protocols public.

WHO - Draft landscape of COVID-19 candidate vaccines - tracking COVID-19 vaccines

World Health Organization keeps tracking the progress of the vaccine candidates against COVID-19 and just published a report called "draft landscape of COVID-19 candidate vaccines". It collected all vaccine candidates already in the clinical trial stage (with links to clinicaltrials.gov registries) and in the pre-clinical stage. 

"These landscape documents have been prepared by the World Health Organization (WHO) for information purposes only concerning the 2019-2020 global of the novel coronavirus. Inclusion of any particular product or entity in any of these landscape documents does not constitute, and shall not be deemed or construed as, any approval or endorsement by WHO of such product or entity (or any of its businesses or activities). While WHO takes reasonable steps to verify the accuracy of the information presented in these landscape documents, WHO does not make any (and hereby disclaims all) representations and warranties regarding the accuracy, completeness, fitness for a particular purpose (including any of the aforementioned purposes), quality, safety, efficacy, merchantability and/or non-infringement of any information provided in these landscape documents and/or of any of the products referenced therein. WHO also disclaims any and all liability or responsibility whatsoever for any death, disability, injury, suffering, loss, damage or other prejudice of any kind that may arise from or in connection with the procurement, distribution or use of any product included in any of these landscape documents."

We anticipate that this draft landscape will be updated periodically to reflect the latest development in COVID-19 vaccine front. The current report can be downloaded at the web link here. It showed that 9 COVID-19 vaccine candidates are in the phase III trial stage now.  

Finkelstein-Schoenfeld Method, Win Ratio, and Hodges-Lehman Estimates - Statistical Methods Based on All Paired Comparisons

Finkelstein-Schoenfeld methods can be used in analyzing the data with a composite endpoint where different components for the composite endpoint have different levels of importance. Hodges-Lehmann estimate is used to estimate the magnitude of treatment difference in a non-parametric statistical test such as the Wilcoxon Rank Test. What is in common between these two methods? Well, both methods are based on the pairwise comparisons - the value/outcome from each subject in treatment group A is compared to each of all subjects in treatment group B - in other words, both methods are based on n (# of subjects in treatment group A) time m (# of subjects in treatment group B) comparisons. 

In clinical trials for serious conditions, but not deadly enough, a composite endpoint is often used as the primary efficacy endpoint. The composite endpoint usually consists of several categories (or components) with different degrees of importance because there will not be enough events for a single category for a feasible clinical trial. The examples of composite endpoints are: 

• a composite endpoint in heart failure may include death, hospitalization, and clinical status
• a composite endpoint in pulmonary arterial hypertension may include death, hospitalization, and disease progression; 
• a composite endpoint in cardiovascular outcome study may be the major adverse cardiovascular events (MACE) consisting of death; MI; stroke, hospitalization.

Usually, these different components are not weighted and treated as equally important and the statistical analyses are based on the time to first event (no matter if the first event is death, hospitalization, or others) - this approach of no weighting is the focal point being criticized. 

Finkelstein-Schoenfeld method is a non-parametric method aiming to bring the weighting into the analysis of the composite endpoints. Finkelstein-Schoenfeld's method was named after their paper in 1999 in Statistics in Medicine "Combining Mortality and Longitudinal Measures in Clinical Trials".  The method was a generalization of the Gehan‐Wilcoxon test based on pairwise comparison of patients on a primary outcome when possible but otherwise on a secondary outcome. The Finkelstein-Schoenfeld method was originally proposed for "analyzing the impact of treatment which combines a (possibly censored) event with a longitudinal measure of clinical effect", not explicitly for analyzing the composite endpoint. 

Based on the Finkelstein-Schoenfeld method, Pocock and colleagues suggested an estimate, the Win Ratio, which summarized the ratio of the number of patients who fared better versus worse on the experimental arm. The Win-Ratio method was proposed explicitly for analyzing the composite endpoint (Pocock et al 2012) "The win ratio: a new approach to the analysis of composite endpoints in clinical trials based on clinical priorities".

With Finkelstein-Schoenfeld or Win-ratio method, pairwise comparisons are performed and the scores are calculated based on the comparison of the importance of the outcome. For example, for a study with composite endpoint including death and hospitalization, all patients had multiple pairwise comparisons performed, first with respect to time to death and to hospitalization, if the latter occurred. 

Below are some additional references discussing the Finkelstein-Schoenfeld or Win-Ratio method and their applications.  

• Finkelstein (2014) "Introduction: Challenges in Analyzing and Interpreting Multiple (Censored) Outcomes in Chronic Disease Trials"
• A novel test to compare two treatments based on endpoints involving both nonfatal and fatal events
• Exact Bootstrap and Permutation Distribution of Wins and Losses in a Hierarchical Trial∗
• Paul Brown's Ph.D. Dissertation (2017) Multiple Outcomes in Heart Failure Research: Composite Endpoints and Multivariate Modelling
• Sun et al (2012) Evaluating Treatment Efficacy by Multiple End Points in Phase II Acute Heart Failure Clinical Trials Analyzing Data Using a Global Method
• Svensson et al "A comprehensive review of the PARTNER trial"
• Graphing the Win Ratio and its components over time

There are several pivotal studies where the Finkelstein-Schoenfeld method is used to analyze the primary efficacy endpoint. The study protocol and statistical analysis plan posted online contain the detail descriptions about the application of the Finkelstein-Schoenfeld method. 

• EXCEL Trial ""Clinical Trial Evaluation of XIENCE versus Coronary Artery Bypass Surgery for Effectiveness of Left Main Revascularization" - Study Protocol;  Statistical Analysis Plan; 
• ATTR-ACT Trial "A Multicenter, International, Phase 3, Double-blind, Placebo-Controlled, Randomized Study to Evaluate the Efficacy, Safety, and Tolerability of Daily Oral Dosing of Tafamidis Meglumine 20 mg or 80 mg in Comparison to Placebo in Subjects Diagnosed with Transthyretin Cardiomyopathy (TTR-CM)" - Study Protocol; Statistical Analysis Plan
• THE PARTNER TRIAL: Placement of AoRTic TraNscathetER Valve Trial Edwards SAPIEN Transcatheter Heart Valve. -  Article (Makkar 2012);  Study Protocol from FDA; Study Protocol/SAP

In the protocol / statistical analysis plan for the Partner trial, there are the following descriptions for the Finkelstein-Schoenfeld method: 

In PARTNER Trial was the basis for FDA approval of Vyndaqel and Vyndamax and Finkelstein and Schoenfeld's method was mentioned in the product label. 

"The primary analysis used a hierarchical combination applying the method of Finkelstein-Schoenfeld (F-S) to all-cause mortality and frequency of cardiovascular-related hospitalizations, which was defined as the number of times a subject was hospitalized (i.e., admitted to a hospital) for cardiovascular-related morbidity. The method compared each patient to every other patient within each stratum in a pair-wise manner that proceeded in a hierarchical fashion using all-cause mortality followed by frequency of cardiovascular-related hospitalizations when patients could not be differentiated based on mortality."

Hodges-Lehmann estimate is used in totally different situations, but similar to the Finkelstein-Schoenfeld method, the estimate relies on the pairwise comparison. While the Finkelstein-Schoenfeld method is primarily used in the analysis of composite endpoint,  Hodges-Lehmann estimate is mainly used to obtain the treatment difference for a continuous variable with normality assumption violation and non-parametric method being used.

 

With the Hodges-Lehmann method, the treatment difference is calculated for each pair for total n x m pairs (where n and m are the # of subjects in each treatment group). The Hodges-Lehmann estimate is the median of differences from all pairs.  

• "Hodges-Lehmann estimator of location shift: Median of Differences versus Difference in Medians or Median Difference"
• Splitting p-value and estimate of the treatment difference
• Hodges-Lehmann Estimator

Hodges-Lehmann estimate has been used in many clinical trials that result in FDA approval of the products. For example, Hodges-Lehmann estimate was the method used in the SIROCCO trial in Asthma. The FDA statistical review document stated the primary analysis method of the study: 

"The primary analysis for the OCS percent reduction endpoint used the Wilcoxon rank-sum test approach. The primary analyses were performed in the FAS population. For each of the two Benralizumab dose regimen groups, the median difference in the OCS percent reduction between Benralizumab dose regimen and placebo was derived using asymptotic Hodges-Lehmann estimation, together with associated 95% CI and p-value. The same analyses were also performed for the EHS without multiplicity control."