Covid-19 Vaccine: Is 50% Vaccine Efficacy (VE) Too Low?

Ever since FDA issued its guidance “Development and Licensure of Vaccines to Prevent COVID-19,” to help facilitate timely development of safe, effective COVID-19 Vaccines, the question arose whether the threshold of 50% vaccine efficacy (VE) was set too low.

• FDA issues COVID-19 vaccine guidance, setting 50% effectiveness threshold
• A COVID-19 Vaccine May Be Only 50% Effective. Is That Good Enough?
• COVID-19 vaccine trials should seek worthwhile efficacy

As cited in an article “FDA to Require 50 Percent Efficacy for COVID-19 Vaccines”:

Gregory Poland, the director of the Mayo Vaccine Research Group, tells Reuters the efficacy guidelines are standard compared to other vaccines. “They look pretty much like influenza vaccine guidelines,” Poland says. “I don’t think that’s a high bar. I think that’s a low to . . . appropriate bar for a first-generation COVID-19 vaccine.” The effectiveness of the annual flu shot, for example, generally ranges between 40 percent and 60 percent, according to The Washington Post.

Peter Hotez, a vaccine expert at the Baylor College of Medicine, tells the Post the 50 percent threshold is low, a sign that the FDA recognizes “our first vaccine won’t be our best.” Ultimately, he says, vaccine developers should aim for 70–75 percent efficacy. 

If we just look at the face meaning of the 50% VE, it does look like the bar is low. Some people may interpret 50% VE as that the COVID-19 vaccine will only need to be effective in 50% of people – which is not true. Even in the FDA’s announcement about the issuance of its guidance, the statement about the requirement of 50% VE was incorrectly stated:

“The guidance also discusses the importance of ensuring that the sizes of clinical trials are large enough to demonstrate the safety and effectiveness of a vaccine. It conveys that the FDA would expect that a COVID-19 vaccine would prevent disease or decrease its severity in at least 50% of people who are vaccinated.”

Let’s see how the vaccine efficacy is calculated and what the 50% VE means.

According to Wikipedia, Vaccine efficacy (VE) is the percentage reduction of disease in a vaccinated group of people compared to an unvaccinated group, using the most favorable conditions.

The outcome data (vaccine efficacy) generally are expressed as a proportionate reduction in disease attack rate (AR) between the unvaccinated (ARU) and vaccinated (ARV), or can be calculated from the relative risk (RR) of disease among the vaccinated group.

The basic formula is written as:

VE=(ARU-ARV) / ARU * 100%

with

VE = Vaccine efficacy,

ARU = Attack rate of unvaccinated people,

ARV = Attack rate of vaccinated people.

An alternative, equivalent formulation of vaccine efficacy

VE=1-RR

where RR is the relative risk of developing the disease for vaccinated people compared to unvaccinated people.

In the actual calculation of VE, we will need to consider the total exposure time (usually measured by the total person-time). One person observed for one year = 1 person-year; one person observed for 3 months = 0.25 person year. the total person-time in year (or total person-years) will be the summation of person-years across all participants in the vaccine group and similarly across all participants in the placebo group. 

The point estimate of the VE can be written as:

If the clinical trial has a 1:1 randomization ratio (all participants are randomized equally into the vaccine group and the placebo group), the 'total person-time' in the vaccine group will be approximately equal to the 'total person-time' in the placebo group, the point estimate of VE can then be easily calculated as:

 

If we know the number of cases (here COVID-19 cases) in the vaccine group and in the placebo group, we can easily calculate the VE.  For example, if the total number of cases is 150 (50 cases observed in the vaccine group and 100 cases observed in the placebo group), the VE will be 1 - (50/100) = 0.5 = 50%.

If an interim analysis is performed after a total of 75 cases are observed, VE will be 50% if 25 cases are observed in the vaccine group and 50 cases are observed in the placebo group. 

Here is a comparison of VE calculations from three Phase III protocols of COVID-19 vaccines:

 

	Moderna	Pfizer	AstraZeneca
Primary efficacy endpoint	VE will be estimated with 1 - HR (mRNA-1273 vs placebo) using a Cox proportional hazard regression model with treatment group as a fixed effect and adjusting for stratification factor	VE will be estimated by 100 × (1 – IRR), where IRR is the  calculated ratio of confirmed COVID-19 illness per 1000 person-years follow-up in the active vaccine group to the corresponding illness rate in the placebo group 7 days after the last dose.	VE is calculated as RRR = 100*(1-relative risk), which RRR is the incidence of infection in the vaccine group relative to the incidence of infection in the control group expressed as a percentage.
Statistical model for calculating the VE and its 95% confidence interval	Cox proportional hazard model	Beta-binomial model	Modified Poisson regression model with robust variance
Sample size (number of volunteers to be recruited)	30,000	43,998	33,000
Number of cases needed to be observed	151	164	150

With the COVID-19 pandemic is still not under control in the US and a large number of volunteers participating in these phase III clinical trials, we hope that the total number of COVID-19 cases can be easily reached so that we can have a readout about the vaccine's efficacy. All three studies have included at least one interim analysis to have a possible readout much earlier. 

In addition to the requirement of at least 50% VE, FDA guidance also requires that the lower bound of the 95% confidence interval of the VE must be greater than 30%. 

The sample size (the number of COVID-19 infection cases) is largely dictated by this criterion of 30% for the lower bound of 95% CI. Otherwise, with 3 cases (1 case in the vaccine group and 2 cases in the placebo group), we would have a point estimate of VE = 50% to meet the requirement. 

50% VE implies that the vaccine can decrease the risk of COVID-19 cases by 50%. Comparing with other clinical trials, the 50% reduction is substantial and meaningful. In Moderna's trial, the VE will be estimated using the Cox proportional hazard model where the time to the first case of COVID-19 infection is also considered. In order to meet the criteria of at least 50% VE, the estimated hazard ratio (HR) needs to be equal to or less than 0.5. In oncology trials or in other clinical trials with time to event variables, if we can have an HR of 0.5 or lower, we will claim that the experimental treatment can reduce the risk of death or event by at least 50% - a result to die for. 

I agree with the statement about the COVID-19 vaccine from a Lancet paper:

“A vaccine that has 50% efficacy could appreciably reduce incidence of COVID-19 in vaccinated individuals, and might provide useful herd immunity. Hence, although efficacy far greater than 50% would be better, efficacy of about 50% would represent substantial progress.”

SARS-CoV-2 (the virus causing COVID-19) has a very high R0 (2.5 according to the table below) which estimates the speed at which a disease is capable of spreading in a population. We hope that we will have a vaccine that will meet the efficacy requirements of at least 50% VE in point estimate and at least 30% VE in the lower bound of 95% confidence interval. With an effective vaccine and the majority of people being vaccinated, we may be able to drop the transmissibility R0 below 1 to prevent the spread of the SARS-CoV-2.

     Comparing SARS-CoV-2 with SARS-CoV and influenza pandemics

Should We Follow ICH E9 Addendum to Include the Estimands in all Clinical Trial Protocols?

ICH E9 "Statistical Principles for Clinical Trials" was issued in 1998 - more than 20 years ago. While the principles specified in ICH E9 are still being followed, a call for a revision or addendum has been there for many years. In 2017, the draft version of ICH E9 (R1) “Addendum on Estimands and Sensitivity Analysis in Clinical Trials to the Guideline on Statistical Principles for Clinical Trials” was released and at the end of 2019, ICH E9 (R1) was finalized. The E9 (R1) guidelines are now gradually been adopted by various regulatory agencies. In terms of the implementation, EMA seems to be ahead of the US requiring the sponsors to include the concept of Estimands in the regulatory submissions. 

Purpose and scope of the addendum to ICH E9:

• Provides a framework for describing with precision a treatment effect of interest
• Precision in describing a treatment effect of interest is facilitated by constructing the “estimand”
• Estimand: A precise description of the treatment effect reflecting the clinical question posed by the trial objective. It summarises at a population-level what the outcomes would be in the same patients under different treatment conditions being compared
• Clarity requires a thoughtful envisioning of “intercurrent events” such as discontinuation of assigned treatment, use of additional or alternative treatment, and terminal events such as death
• Intercurrent Events: Events occurring after treatment initiation that affect either the interpretation of the existence of the measurements associated with the clinical question of interest
• It is necessary to address intercurrent events when describing the clinical question of interest in order to precisely define the treatment effect that is to be estimated
• Addendum introduces strategies to reflect different questions of interest that might be posed
• Attributes used to construct the estimand are also introduced in the addendum
• Addendum clarifies the definition and the role of sensitivity analysis
• Sensitivity Analysis: A series of analyses conducted with the intent to explore the robustness of inferences from the main estimator to deviations from its underlying modeling assumptions and limitations in the data

Estimand attributes:

• Treatment: The treatment condition of interest and, as appropriate, the alternative treatment condition to which comparison will be made
• Population: Patients targeted by the clinical question
• Variable (or endpoint): Obtained for each patient and required to address the clinical question
• Population-level summary: Provides a basis for comparison between treatment conditions for the variable
• Handling of intercurrent events

While FDA has not mandated the implementation of the ICH E9 (R1), there seems to be a trend in the industry that ICH E9 (R1) is gradually being adopted and the concept ‘estimands’ is being mentioned in the study protocols and statistical analysis plans (SAPs). 

Looking at the clinical trial protocol and SAP templates developed by TransCelerate Biopharma, both contained a section about estimands and the estimands was listed together with endpoints.

In my previous post "Should Clinical Trial Protocol be Made Public While the Trial is still ongoing?", I mentioned that the protocols for three phase III studies for Covid-19 vaccine were all made public because of the demand for transparency. I examed all three protocols and they had described the 'estimands'. The concept of 'intercurrent events', 'principal stratum strategy', 'treatment policy' was also mentioned. 

In Phase III study protocol by Moderna "A Phase 3, Randomized, Stratified, Observer-Blind, Placebo-Controlled Study to Evaluate the Efficacy, Safety, and Immunogenicity of mRNA-1273 SARS-CoV-2 Vaccine in Adults Aged 18 Years and Older":

In the estimand of the primary analysis on the primary endpoint, a treatment policy strategy will be used to address the intercurrent events of 1) withdrawal from the study or death unrelated to COVID-19, where the time to COVID-19 will be censored at the date of withdrawal from the study or death; 2) early COVID-19, where the time to COVID-19 will be censored at the time of early infection. Principal stratum strategy will be used to address the other 2 types of intercurrent events in the primary analysis based on the PP Set. The details of intercurrent event description and estimand strategies are presented in Section 11.4.1.

In Phase III study protocol by AstraZeneca / Oxford "A Phase III Randomized, Double-blind, Placebo-controlled Multicenter Study in Adults to Determine the Safety, Efficacy, and Immunogenicity of AZD1222, a Non-replicating ChAdOx1 VectorVaccine, for the Prevention of COVID-19":

The primary estimand will be used for the analysis of the primary efficacy endpoint. It will be based on participants in the full analysis set, defined as all randomized participants who received at least 1 dose of study intervention excluding those participants who are seropositive at baseline, analyzed according to their randomized treatment. For participants with multiple events, only the first occurrence will be used for the primary efficacy endpoint analysis. The set of intercurrent events for this estimand consists of participants who withdraw from the study prior to having met the primary efficacy endpoint. The intercurrent events will be handled using the treatment policy strategy and the absence of data following these participants’ withdrawal will be treated as missing (ie, counted as not having met the criteria). Participants who withdraw before 15 days post second dose or who have a case prior to 15 days post second dose will be excluded from primary endpoint analysis.
Additional estimands will be specified for the primary efficacy endpoint to carry out sensitivity analyses for assessing the robustness of results. These sensitivity analyses will explore different methods for handling intercurrent events and different assumptions for missing data. Estimands will also be specified for the analysis of secondary endpoints. Full details will be provided in the SAP.

In Phase III study protocol by Pfizer/BioNtech "A Phase 1/2/3, Placebo-Controlled, Randomized, Observer-Blind, Dose-Finding Study to Evaluate the Safety, Tolerability, Immunogenicity, and Efficacy of SARS-COV-2 RNA Vaccine Candidates Against COVID-19 in Healthy Individuals": 

It looks like that the concept of 'estimands' has not been widely accepted by the medical community - it is indeed a new concept and a new term for clinical trialists to digest. Using a most recent paper in the New England Journal of Medicine (Rabe et al 2020 "Triple Inhaled Therapy at Two Glucocorticoid
Doses in Moderate-to-Very-Severe COPD"), the main body of the article had no mention of the concept of 'estimands' even though the attached protocol and SAP contained a section about 'estimands':

In the phase III study by J&J (Janssen) "A Randomized, Double-blind, Placebo-controlled Phase 3 Study to Assess the Efficacy and Safety of Ad26.COV2.S for the Prevention of SARS-CoV-2-mediated COVID-19 in Adults Aged 18 Years and Older", there was no mention of estimands. 

Additional References: 

• US FDA and Health Canada Joint Regional Consultation on the International Council for Harmonisation (ICH)
• University of Pennsylvania 11th annual conference on statistical issues in clinical trials: Estimands, missing data and sensitivity analysis (morning panel session)
• Estimand Framework: What it is and Why You Need it
• Estimands in clinical trials – broadening the perspective
• PSI, EFSPI & ASA BIOP Webinar: Estimands
• Estimand Framework in Oncology Drug Development: Challenges and Opportunities

Should Clinical Trial Protocol be Made Public While the Trial is still ongoing?

Drug companies are under pressure for more and more transparency in clinical trials. It has been a common practice now to post the clinical trial in clinicaltrials.gov before the first patient is enrolled in the study. However, the registration in clinicaltrials.gov includes not all details about the clinical trial. For example, the planned statistical analysis, the interim analysis, stopping rule,... are usually not part of the items to be published. 

Transparency is also implemented after the clinical trial is completed. The clinical trial results need to be published in clinicaltrials.gov within one year of the last efficacy assessment. FDA is implementing a pilot program to encourage drug companies to post the clinical study report (CSR) and the statistical analysis plan (SAP). Some journal such as the New England Journal of Medicine (see an example) requires the study protocol and the SAP to be available as part of the supplemental material to the main publication. 

It is rare to see the study protocol or SAP to be published while the clinical trial is still ongoing. But this happens now for COVID-19 vaccine clinical trials. 

According to StatNews.com:

In an unprecedented show of transparency, the two frontrunners in the race to develop a Covid-19 vaccine released detailed protocols describing the studies testing their vaccines. The unusual step is part of an effort to proactively increase the public's trust in an eventual vaccine.

"This is an example of the comprehensive trial description that industry should be sharing," said Harlan Krumholz, a Yale cardiologist who has been a long-time advocate for increased transparency from drug companies.

The protocols are dense, and researchers will be poring over them in coming days. In both cases, there are plans for potentially stopping the studies early if the vaccines prove more effective than planned. Both protocols require that decision to be suggested by an independent Data Monitoring Committee, not by the companies. And in both cases, this committee reports directly to the company, not to academics who are running the study, something some experts say is disappointing.

The Moderna protocol can be found here "A Phase 3, Randomized, Stratified, Observer-Blind,  Placebo-Controlled Study to Evaluate the Efficacy, Safety, and Immunogenicity of mRNA-1273 SARS-CoV-2 Vaccine in Adults Aged 18 Years and Older"

The Pfizer protocol can be found here "A Phase 1/2/3, Placebo-Controlled, Randomized, Observer-Blind, Dose-Finding Study to Evaluate the Safety, Tolerability, Immunogenicity, and Efficacy of SARS-COV-2 RNA Vaccine Candidates Against COVID-19 in Healthy Individuals"

AstraZeneca followed the suit and had just published its phase III study protocol. "A Phase III Randomized, Double-blind, Placebo-controlled Multicenter Study in Adults to Determine the Safety, Efficacy, and Immunogenicity of AZD1222, a Non-replicating ChAdOx1 VectorVaccine, for the Prevention of COVID-19". See the article "AstraZeneca, Under Fire for Vaccine Safety, Releases Trial Blueprints"

The unprecedented move by three leading COVID-19 vaccine development companies is mainly driven by what happened in their phase III clinical trials. AstraZeneca's trial was paused for data monitoring committee review and then resumed; Pfizer's trial had its sample size increased from the planned 30,000 to 44,000 volunteers; Moderna's trial was slowed down in enrollment in order to ensure minority representation.

These phase III clinical trials for the COVID-19 vaccine are under the public's scrutiny for every move. It is understandable that the public demands transparency to see clinical trial protocols. For regular clinical trials under normal circumstances, it is still not the time for drug companies to make the study protocols public.

WHO - Draft landscape of COVID-19 candidate vaccines - tracking COVID-19 vaccines

World Health Organization keeps tracking the progress of the vaccine candidates against COVID-19 and just published a report called "draft landscape of COVID-19 candidate vaccines". It collected all vaccine candidates already in the clinical trial stage (with links to clinicaltrials.gov registries) and in the pre-clinical stage. 

"These landscape documents have been prepared by the World Health Organization (WHO) for information purposes only concerning the 2019-2020 global of the novel coronavirus. Inclusion of any particular product or entity in any of these landscape documents does not constitute, and shall not be deemed or construed as, any approval or endorsement by WHO of such product or entity (or any of its businesses or activities). While WHO takes reasonable steps to verify the accuracy of the information presented in these landscape documents, WHO does not make any (and hereby disclaims all) representations and warranties regarding the accuracy, completeness, fitness for a particular purpose (including any of the aforementioned purposes), quality, safety, efficacy, merchantability and/or non-infringement of any information provided in these landscape documents and/or of any of the products referenced therein. WHO also disclaims any and all liability or responsibility whatsoever for any death, disability, injury, suffering, loss, damage or other prejudice of any kind that may arise from or in connection with the procurement, distribution or use of any product included in any of these landscape documents."

We anticipate that this draft landscape will be updated periodically to reflect the latest development in COVID-19 vaccine front. The current report can be downloaded at the web link here. It showed that 9 COVID-19 vaccine candidates are in the phase III trial stage now.  

Finkelstein-Schoenfeld Method, Win Ratio, and Hodges-Lehman Estimates - Statistical Methods Based on All Paired Comparisons

Finkelstein-Schoenfeld methods can be used in analyzing the data with a composite endpoint where different components for the composite endpoint have different levels of importance. Hodges-Lehmann estimate is used to estimate the magnitude of treatment difference in a non-parametric statistical test such as the Wilcoxon Rank Test. What is in common between these two methods? Well, both methods are based on the pairwise comparisons - the value/outcome from each subject in treatment group A is compared to each of all subjects in treatment group B - in other words, both methods are based on n (# of subjects in treatment group A) time m (# of subjects in treatment group B) comparisons. 

In clinical trials for serious conditions, but not deadly enough, a composite endpoint is often used as the primary efficacy endpoint. The composite endpoint usually consists of several categories (or components) with different degrees of importance because there will not be enough events for a single category for a feasible clinical trial. The examples of composite endpoints are: 

• a composite endpoint in heart failure may include death, hospitalization, and clinical status
• a composite endpoint in pulmonary arterial hypertension may include death, hospitalization, and disease progression; 
• a composite endpoint in cardiovascular outcome study may be the major adverse cardiovascular events (MACE) consisting of death; MI; stroke, hospitalization.

Usually, these different components are not weighted and treated as equally important and the statistical analyses are based on the time to first event (no matter if the first event is death, hospitalization, or others) - this approach of no weighting is the focal point being criticized. 

Finkelstein-Schoenfeld method is a non-parametric method aiming to bring the weighting into the analysis of the composite endpoints. Finkelstein-Schoenfeld's method was named after their paper in 1999 in Statistics in Medicine "Combining Mortality and Longitudinal Measures in Clinical Trials".  The method was a generalization of the Gehan‐Wilcoxon test based on pairwise comparison of patients on a primary outcome when possible but otherwise on a secondary outcome. The Finkelstein-Schoenfeld method was originally proposed for "analyzing the impact of treatment which combines a (possibly censored) event with a longitudinal measure of clinical effect", not explicitly for analyzing the composite endpoint. 

Based on the Finkelstein-Schoenfeld method, Pocock and colleagues suggested an estimate, the Win Ratio, which summarized the ratio of the number of patients who fared better versus worse on the experimental arm. The Win-Ratio method was proposed explicitly for analyzing the composite endpoint (Pocock et al 2012) "The win ratio: a new approach to the analysis of composite endpoints in clinical trials based on clinical priorities".

With Finkelstein-Schoenfeld or Win-ratio method, pairwise comparisons are performed and the scores are calculated based on the comparison of the importance of the outcome. For example, for a study with composite endpoint including death and hospitalization, all patients had multiple pairwise comparisons performed, first with respect to time to death and to hospitalization, if the latter occurred. 

Below are some additional references discussing the Finkelstein-Schoenfeld or Win-Ratio method and their applications.  

• Finkelstein (2014) "Introduction: Challenges in Analyzing and Interpreting Multiple (Censored) Outcomes in Chronic Disease Trials"
• A novel test to compare two treatments based on endpoints involving both nonfatal and fatal events
• Exact Bootstrap and Permutation Distribution of Wins and Losses in a Hierarchical Trial∗
• Paul Brown's Ph.D. Dissertation (2017) Multiple Outcomes in Heart Failure Research: Composite Endpoints and Multivariate Modelling
• Sun et al (2012) Evaluating Treatment Efficacy by Multiple End Points in Phase II Acute Heart Failure Clinical Trials Analyzing Data Using a Global Method
• Svensson et al "A comprehensive review of the PARTNER trial"
• Graphing the Win Ratio and its components over time

There are several pivotal studies where the Finkelstein-Schoenfeld method is used to analyze the primary efficacy endpoint. The study protocol and statistical analysis plan posted online contain the detail descriptions about the application of the Finkelstein-Schoenfeld method. 

• EXCEL Trial ""Clinical Trial Evaluation of XIENCE versus Coronary Artery Bypass Surgery for Effectiveness of Left Main Revascularization" - Study Protocol;  Statistical Analysis Plan; 
• ATTR-ACT Trial "A Multicenter, International, Phase 3, Double-blind, Placebo-Controlled, Randomized Study to Evaluate the Efficacy, Safety, and Tolerability of Daily Oral Dosing of Tafamidis Meglumine 20 mg or 80 mg in Comparison to Placebo in Subjects Diagnosed with Transthyretin Cardiomyopathy (TTR-CM)" - Study Protocol; Statistical Analysis Plan
• THE PARTNER TRIAL: Placement of AoRTic TraNscathetER Valve Trial Edwards SAPIEN Transcatheter Heart Valve. -  Article (Makkar 2012);  Study Protocol from FDA; Study Protocol/SAP

In the protocol / statistical analysis plan for the Partner trial, there are the following descriptions for the Finkelstein-Schoenfeld method: 

In PARTNER Trial was the basis for FDA approval of Vyndaqel and Vyndamax and Finkelstein and Schoenfeld's method was mentioned in the product label. 

"The primary analysis used a hierarchical combination applying the method of Finkelstein-Schoenfeld (F-S) to all-cause mortality and frequency of cardiovascular-related hospitalizations, which was defined as the number of times a subject was hospitalized (i.e., admitted to a hospital) for cardiovascular-related morbidity. The method compared each patient to every other patient within each stratum in a pair-wise manner that proceeded in a hierarchical fashion using all-cause mortality followed by frequency of cardiovascular-related hospitalizations when patients could not be differentiated based on mortality."

Hodges-Lehmann estimate is used in totally different situations, but similar to the Finkelstein-Schoenfeld method, the estimate relies on the pairwise comparison. While the Finkelstein-Schoenfeld method is primarily used in the analysis of composite endpoint,  Hodges-Lehmann estimate is mainly used to obtain the treatment difference for a continuous variable with normality assumption violation and non-parametric method being used.

 

With the Hodges-Lehmann method, the treatment difference is calculated for each pair for total n x m pairs (where n and m are the # of subjects in each treatment group). The Hodges-Lehmann estimate is the median of differences from all pairs.  

• "Hodges-Lehmann estimator of location shift: Median of Differences versus Difference in Medians or Median Difference"
• Splitting p-value and estimate of the treatment difference
• Hodges-Lehmann Estimator

Hodges-Lehmann estimate has been used in many clinical trials that result in FDA approval of the products. For example, Hodges-Lehmann estimate was the method used in the SIROCCO trial in Asthma. The FDA statistical review document stated the primary analysis method of the study: 

"The primary analysis for the OCS percent reduction endpoint used the Wilcoxon rank-sum test approach. The primary analyses were performed in the FAS population. For each of the two Benralizumab dose regimen groups, the median difference in the OCS percent reduction between Benralizumab dose regimen and placebo was derived using asymptotic Hodges-Lehmann estimation, together with associated 95% CI and p-value. The same analyses were also performed for the EHS without multiplicity control." 

How to interpret the risk ratio? - A controversy related to FDA's EUA announcement of convalescent plasma in treatment of Covid-19

This past Sunday, FDA Issues Emergency Use Authorization for Convalescent Plasma as Potential Promising COVID–19 Treatment, Another Achievement in Administration’s Fight Against Pandemic. The EUA (emergency use authorization) was mainly based on a study conducted by Mayo Clinic and the study results were described in the paper (not yet peer-reviewed yet) uploaded to Medrxiv.

Joyner 2020 “Effect of Convalescent Plasma on Mortality among Hospitalized Patients with COVID-19: Initial Three-Month Experience”.

 Issuance of EUA for convalescent plasma immediately draw some criticize about not having strong evidence to support the EUA approval and about the timing of the EUA announcement (right before the Republic National Convention) – We will leave this to others to debate.

One thing related to the statistics is how to present or interpret the results from Mayo Clinic’s study. In the paper, the author stated the hypotheses for analyzing the data from this study that has no concurrent control group.  

“We hypothesized, based on historical data that earlier administration of convalescent plasma with high antibody levels would be associated with reduced mortality. To address this hypothesis, we evaluated seven and 30-day mortality in 35,322 hospitalized adults transfused with COVID-19 convalescent plasma by asking two questions. First, was earlier treatment of patients with convalescent plasma after diagnosis of COVID-19 associated with reduced mortality compared to later treatment in the course of disease? Second, were higher antibody levels in the transfused convalescent plasma associated with reduced mortality?”

For the statistical analyses, the study cohort "was stratified into categories based on the days from COVID-19 diagnosis to plasma transfusion, including: 0, 1-3, 4-10, and 11 or more days and for some graphical presentations and analyses, dichotomized into 0-3 vs. 4 or more days". The relative risk was calculated for each sub-group for mortality for patients who received convalescent plasma with IgG S/co greater than 18.45 (high antibody level group) vs. patients that received less than 4.62 S/Co (low antibody level group). The pooled, or common, relative risk for 7-day and 30-day mortality were then calculated. The paper concluded, "the pooled relative risk of mortality among patients transfused with high antibody level plasma units was 0.65 [0.47-0.92] for 7 days and 0.77 [0.63-0.94] for 30 days compared to low antibody level plasma units."  For 7-day mortality, the forest plot was depicted below:

 

How to interpret the pooled relative risk of 0.65? FDA commissioner, Dr. Hahn, was criticized for overstating the efficacy results because of his interpretation of this relative risk. 

“In the optimal patients … treated with convalescent plasma at the highest titers, there was a 35% improvement in survival, which is a significant clinical benefit,” Hahn said during the press conference, noting that, “This clearly meets the criteria that we’ve established for emergency use authorization.”
 
Hahn went on to say that, “A 35% improvement in survival is a pretty substantial clinical benefit. What that means is—and if the data continue to pan out—100 people who are sick with COVID-19, 35 would have been saved because of the administration of plasma.”

Dr. Hahn's statement immediately drew a lot of criticizes and he had to come up to apologize for overstating the treatment effect of convalescent plasma in the treatment of COVID-19. 

• Lawmakers, experts raise questions after convalescent plasma EUA
• FDA chief apologizes for overstating plasma effect on virus
• FDA's Hahn Apologizes For Overselling Plasma's Benefits As A COVID-19 Treatment
• Covid-19 Live Updates: 2 F.D.A. Public Relations Experts Are Fired After Plasma Fiasco
• Is convalescent plasma safe and effective? We answer the major questions about the Covid-19 treatment

What is the right way to state the relative risk of 0.65? Well, relative risk is relative and can't be stated as an absolute benefit. CDC’s website “An Introduction to Applied Epidemiology and Biostatistics” has a chapter about the relative risk. It explains how the relative risk is calculated and how to interpret the results. 

In general, a risk ratio greater than 1.0 indicates an increased risk for the group in the numerator, usually the exposed group. A risk ratio of less than 1.0 indicates a decreased risk for the exposed group, indicating that perhaps exposure actually protects against disease occurrence.

With the convalescent plasma study by Mayo Clinic, the relative risk is calculated as the ratio of "the risk of 7-day mortality in patients receiving convalescent plasma with high antibody level" divided by "the risk of 7-day mortality in patients receiving convalescent plasma with low antibody level". A relative ratio of 0.65 (less than 1.0) indicates the benefit of receiving convalescent plasma with high antibody level - an indication of dose-response. 

It would be correct to state:

"The risk ratio of 0.65 indicates that COVID-19 patients who received convalescent plasma with high antibody level were only 65% as likely to die in 7 days as were patients who received convalescent plasma with low antibody level"

or

"The risk ratio of 0.65 indicates that the convalescent plasma with high antibody level reduced the risk of 7-day mortality by 35%"

Participants, Patients, Subjects, Volunteers, What to Use?

For people who participate in clinical trials, what should we call them? Subject, volunteer (healthy volunteer), participant, or patient? This seems like an easy question, but there are actually a lot of disagreements. Here are some of the articles and blogs discussing this:

Subject, Volunteer, Participant, or Patient?	The author clearly disliked the use of ‘subject’ for clinical trial participants. “In spite of its official sanctioned use, I’ve always objected to the term ‘subject’ to describe a person who donates time, effort and bodily fluids to further clinical research.  It has a negative connotation for me, conjuring up the image of a cold scientific investigation.  I wonder if people considering participating in clinical research studies are dissuaded by this term?  Do they feel they will be ‘subject’ ed to tests and procedures?   Acted upon, rather than participating in their health care?”
Revisiting What to Call People Who Participate in Clinical Research	The author prefers the term ‘participants’ to be used and even used the survey data to support the use of ‘participants’. “The New England Journal of Medicine, American Journal of Public Health, and International Committee of Medical Journal Editors all use the term participant exclusively. NIH Director Francis Collins is quoted as saying, “Medical advances would not be possible without participants in clinical trials.”
People are “participants” in research: Further suggestions for other terms to describe “participants” are needed	As the title suggested, “People are participants in research, not subjects”. Suggesting that the word “subject” should be banned from reports of research on humans. The word “subject” is demeaning.
When to use subjects, participants or volunteers to describe your research subjects	The author seems to be open to all of these terms that may be best used in different situations. Also provided are the definitions for each of these terms: Participants A participant is a person that voluntarily participates in a study. This is perhaps the most accepted terms and is generally recommended when in doubt (provided the subject of the investigation is human). Subjects The term subject describes the person or thing that is the topic of study.  Patients A patient is a participant with a medical condition which is the interest of the investigation.  Volunteers A volunteer is someone that freely offers to participate in a study. A volunteer is very similar to the participant and implies that the person as a whole is actively involved in the study. It also infers that they are free from any particular factor of interest, such as a medical condition.

The term ‘patient’ cannot cover all people who participate in clinical trials. While the clinical trials will ultimately be conducted in the patients who have a medical condition which is the interest of the investigation, clinical trial participants can also be volunteers or healthy volunteers, not patients. Here are some situations that the clinical trial participants are volunteers or healthy volunteers:

• Phase I clinical trials including first-in-human trials. The majority of phase I studies are conducted in healthy volunteers at a dedicated clinical research unit. Only in rare situations (such as oncology studies, studies using human-plasma derived products), are phase I studies conducted in patients.
• Vaccine clinical trials. In vaccine clinical trials, volunteers (or the participants who do not have the disease) are recruited.  The purpose of these clinical trials is to test if the vaccine will safe and effective against developing a disease.
• Preventive clinical trials. If the purpose of a clinical trial is to test if any therapeutic agent (not necessarily the vaccine) can prevent a disease, the participants will be volunteers and should not be called ‘patients’.
• Clinical trials in pregnant women. Pregnancy women are not patients.

People can get confused about using the term ‘volunteers’. In a co-ed by professors Emanuel and Offit, “Could Trump Turn a Vaccine Into a Campaign Stunt?”, the authors mistakenly used the term ‘patients’ for the planned vaccine clinical trials. For Covid-19 vaccine clinical trials, volunteers, healthy volunteers, or healthy volunteers with potential exposure to Covid-19 are recruited to participate in clinical trials. The term 'patient' should not be used. The term 'patient' can be used in therapeutical clinical trials where the purpose is to test if a drug/therapy (such as remdesivir, antibody cocktail,...) is effective in treating Covid-19 infected patients.

"Pfizer is planning to give its vaccine to approximately 8,000 patients. The N.I.H. is planning to enroll 30,000 participants — 20,000 getting a candidate vaccine and 10,000 research controls."

"Researchers are expecting that it will be likely to take at least another eight to 12 months to determine whether these coronavirus vaccines are effective. Scientists have to wait until a sufficient number of patients are exposed to coronavirus to see if the vaccine really reduces the infection rate, as well as how many people develop uncommon side effects. For comparison, the effectiveness trial for the rotavirus vaccines took about four years and the human papillomavirus vaccine studies to prevent cervical cancer took seven years."

“Scientists have to wait until a sufficient number of patients are exposed to coronavirus to see if the vaccine really reduces the infection rate, as well as how many people develop uncommon side effects. “

For years, we have been using the term ‘subject’ to describe clinical trial participants. The term ‘subject’ can cover all different types of clinical trial participants including patients and healthy volunteers. The term ‘subject’ reflects the facts (whether we like it or not) that the participants are subjects in clinical trials.

Many regulatory guidelines used the term ‘subject’ or ‘human subject’. As mentioned in an article “Comparison of FDA and HHS Human Subject Protection Regulations”:

"Human subject" means an individual who is or becomes a participant in research, either as a recipient of the test article or as a control. A subject may be either a healthy individual or a patient.

Human subject" means a living individual about whom an investigator (whether professional or student) conducting research obtains (1) data through intervention or interaction with the individual, or (2) identifiable private information.

There is a trend that the term ‘participant’ is commonly used to replace the term ‘subject’. For example, in Transcelerate Biopharma’s Common Protocol Template and NIH/FDA’s “Final “Phase 2 and 3 Clinical Trial” Template Documents”, the term ‘participant’ or ‘participants’ is used and the term ‘subject’ or ‘subjects’ is only used in special places. ‘Number of subjects’ is now called the ‘number of participants’.  

The term ‘patient’ may be preferred by doctors and study coordinators in investigational sites but is not usually used in clinical trial protocols. The term ‘patient’ may still be used in some special cases such as patient-reported outcome (PRO) and patient medical record. The term 'patient' is commonly used in publications for clinical trials conducted in patients. 

Even though the term ‘participant’ may be preferred in clinical trial protocols, the term ‘subject’ may be used in other clinical trial documents. For example, in case report form and database set up, the term 'subject' will continue to be used. The term ‘subject’ is the primary term used in CDISC documents such as CDASH (Clinical Data Acquisition Standards Harmonization) and SDTM (Study Data Tabulation Model) that we are following as the data standards. The variable Subjid is used for the subject identifier; the variable Usubjid is used for the unique subject identifier, …

In journal publications, we continue to see that different terms are used depending on the nature of the participants in a specific clinical trial. In the New England Journal of Medicine, the term ‘patient’ or ‘patients’ is used in clinical trials conducted in patients and the ‘participant’ or ‘participants’ is used in clinical trials conducted in volunteers or healthy volunteers. The term ‘subject’ is gradually phased out in publication. 

• in Jackson et al "An mRNA Vaccine against SARS-CoV-2 —Preliminary Report", the term 'participant' or 'participants' is used to describe the volunteers who participated in this vaccine trial. 
• in Beigel et al "Remdesivir for the Treatment of Covid-19 — Preliminary Report", the term 'patient' or 'patients' is used to describe the participants who are patients with Covid-19 infections. 

Unit of Analysis in Clinical Trials

The Unit of Analysis is the entity that frames what is being analyzed in a study or clinical trial. It is the entity being studied as a whole, within which most factors of causality and change exist. The unit of analysis is the “who” or the “what” that are being analyzed for a study or clinical trial. The Unit of Analysis is based on the experimental unit defined as "the smallest division of experimental material such that any two units may receive different treatments in the actual experiment" (Cox, 1992). Usually, the Unit of Analysis is on the same level as the unit for randomization.

We rarely talk about the Unit of Analysis, but actually, deal with it every time when we analyze the data. In clinical trials, we don’t explicitly talk about the Unit of Analysis because the Unit of Analysis is almost always the subject (or maybe called patient, health volunteer, or participant). Once the Unit of Analysis is established, all statistical analyses will be based on the Unit of Analysis – it means we count the number, perform the statistical model, include the explanatory variables all on the Unit of Analysis level. Given that the Unit of Analysis is ‘subject’ in clinical trials (in general), the subject level information or subject level variables will be used in analysis – that is why in CDISC ADaM data set, an ADSL (subject-level analysis dataset) will always be created.

The Unit of Analysis doesn’t have to be always the ‘subject’.

• For meta-analysis that is based on the summary information from multiple studies, the Unit of Analysis is ‘study’, not ‘subject’.

In a paper by Wong (2020) Estimation of clinical trial success rates and related parameters, the unit of analysis is 'study' or individual 'clinical trial'.  

• In analyses of Covid-19 data, all models are based on county-level or hospital-level data. Due to the concern about privacy, the data on the individuals is not available. See https://covidseverity.com/ website for county-level Covid-19 related data. Here the unit of analysis is 'county' or 'hospital'. 
• For studies using cluster randomization, the Unit of Analysis may be the cluster (township, city, household), not ‘subject’ even though the subject may be the observation unit.

In FDA’s guidance “Influenza: Developing Drugs for Treatment and/or Prophylaxis”, it specified that the Unit of Analysis could be the household.

"In household trials, the entire household is both the randomized unit and the unit of analysis. The primary efficacy analysis should compare the treatment groups for the percentage of households in which at least one randomized contact case developed symptomatic, laboratory-confirmed influenza. In other words, if one contact case in the household becomes symptomatically infected, the household is counted as infected. If none of the contact cases becomes infected, the household is considered not infected. Secondary analyses also can compare the percentage of contact cases that had symptomatic, laboratory-confirmed influenza in the active and placebo treatment groups.

Designs in which different contact cases in the same household receive different regimens raise concerns of drug sharing and intrahousehold correlation. Analysis using individual contact cases as the unit of analysis also may cause similar problems. Stratification on the size of household can be used, but is not expected to produce any consequential increase in power. "

• In some clinical trials, the Unit of Analysis may be smaller than the ‘subject’ level, for example, the tumor lesion in oncology studies, target bleeding site in studies for hemostasis agents.

In FDA Statistical Review for Lumason NDA, the Unit of Analysis using the lesion was performed

“The unit of analysis was the lesion; each subject had a single lesion that was to be characterized Sensitivity and Specificity are in percent (%) and n is the denominator for percentage calculation”

The Unit of Analysis may be different from the unit of observation. Within each unit of analysis, there may be multiple observations, for example, each subject with multiple events of hospitalization, exacerbation, adverse events. In this situation, we usually still analyze the data on the subject level and multiple events within a subject can be converted into the subject level data (time to first exacerbation, time to bleeding stoppage for the targeted bleeding site, best overall response based on the aggregated information from multiple lesions)

In FDA’s Statistical Review for Zerviate NDA, "The unit of analysis for all ocular variables was the average of both eyes of each subject."

In FDA’s review of Extended-Release and Long-Acting opioid analgesic (ER/LA) products, the unit of analysis is zip code (spatial) and quarter (time).

In both of the models proposed in the RADARS data analysis section, the unit of analysis is zip code (spatial) and quarter (time). Thus, testing for change between pre and post period for each outcome is investigating whether the average rate of events over time for the average zip code has changed from the pre-REMS period to the post-REMS period.

In clinicaltrials.gov, when clinical trial results are posted, the unit of analysis needs to be specified if the unit of analysis is not the subject.

"Type of Units Analyzed 

Definition: If the analysis is based on a unit other than participants, a description of the unit of analysis (for example, eyes, lesions, implants). "

In a handbook from Cochrane.org, there was a section to discuss the Unit of Analysis:

9.3.1 Unit-of-analysis issues

An important principle in clinical trials is that the analysis must take into account the level at which randomization occurred. In most circumstances the number of observations in the analysis should match the number of ‘units’ that were randomized. In a simple parallel group design for a clinical trial, participants are individually randomized to one of two intervention groups, and a single measurement for each outcome from each participant is collected and analysed. However, there are numerous variations on this design. Authors should consider whether in each study:

groups of individuals were randomized together to the same intervention (i.e. cluster-randomized trials);

individuals undergo more than one intervention (e.g. in a cross-over trial, or simultaneous treatment of multiple sites on each individual); or

there are multiple observations for the same outcome (e.g. repeated measurements, recurring events, measurements on different body parts).

There follows a more detailed list of situations in which unit-of-analysis issues commonly arise, together with directions to relevant discussions elsewhere in the Handbook.

Sometimes, the Unit of Analysis can be misused. In a paper by A. Vail and E. Gardener “Common statistical errors in the design and analysis of subfertility trials”, it said that “Most trials (82%) included at least one ‘unit of analysis’ error”. 

The most common error I can see is in the analysis of adverse events (AE). People can be confused with the different use of the Unit of Analysis. On the subject level, the adverse event should be analyzed to compare the incidence of AEs which is calculated as “the number of subjects with at least one specific AE divided by the number of subjects”. On the AE level, if we count the number of AEs, we can calculate the AE rate (number of AEs per subject; number of AEs per unit of exposure (person-year)) or AE density (number of AEs per drug infusion) – the meaning and interpretation are totally different than the incidence of AE.

In clinical trials with longitudinal design and crossover design, while the analyses will include the multiple measures for each individual subject, the unit of analysis is still the subject, but the more sophisticated statistical models (mixed model repeat measures, random coefficient model, multi-level or hierarchical linear models) will be needed.