Covid-19 Vaccine: Is 50% Vaccine Efficacy (VE) Too Low?

Ever since FDA issued its guidance “Development and Licensure of Vaccines to Prevent COVID-19,” to help facilitate timely development of safe, effective COVID-19 Vaccines, the question arose whether the threshold of 50% vaccine efficacy (VE) was set too low.

• FDA issues COVID-19 vaccine guidance, setting 50% effectiveness threshold
• A COVID-19 Vaccine May Be Only 50% Effective. Is That Good Enough?
• COVID-19 vaccine trials should seek worthwhile efficacy

As cited in an article “FDA to Require 50 Percent Efficacy for COVID-19 Vaccines”:

Gregory Poland, the director of the Mayo Vaccine Research Group, tells Reuters the efficacy guidelines are standard compared to other vaccines. “They look pretty much like influenza vaccine guidelines,” Poland says. “I don’t think that’s a high bar. I think that’s a low to . . . appropriate bar for a first-generation COVID-19 vaccine.” The effectiveness of the annual flu shot, for example, generally ranges between 40 percent and 60 percent, according to The Washington Post.

Peter Hotez, a vaccine expert at the Baylor College of Medicine, tells the Post the 50 percent threshold is low, a sign that the FDA recognizes “our first vaccine won’t be our best.” Ultimately, he says, vaccine developers should aim for 70–75 percent efficacy. 

If we just look at the face meaning of the 50% VE, it does look like the bar is low. Some people may interpret 50% VE as that the COVID-19 vaccine will only need to be effective in 50% of people – which is not true. Even in the FDA’s announcement about the issuance of its guidance, the statement about the requirement of 50% VE was incorrectly stated:

“The guidance also discusses the importance of ensuring that the sizes of clinical trials are large enough to demonstrate the safety and effectiveness of a vaccine. It conveys that the FDA would expect that a COVID-19 vaccine would prevent disease or decrease its severity in at least 50% of people who are vaccinated.”

Let’s see how the vaccine efficacy is calculated and what the 50% VE means.

According to Wikipedia, Vaccine efficacy (VE) is the percentage reduction of disease in a vaccinated group of people compared to an unvaccinated group, using the most favorable conditions.

The outcome data (vaccine efficacy) generally are expressed as a proportionate reduction in disease attack rate (AR) between the unvaccinated (ARU) and vaccinated (ARV), or can be calculated from the relative risk (RR) of disease among the vaccinated group.

The basic formula is written as:

VE=(ARU-ARV) / ARU * 100%

with

VE = Vaccine efficacy,

ARU = Attack rate of unvaccinated people,

ARV = Attack rate of vaccinated people.

An alternative, equivalent formulation of vaccine efficacy

VE=1-RR

where RR is the relative risk of developing the disease for vaccinated people compared to unvaccinated people.

In the actual calculation of VE, we will need to consider the total exposure time (usually measured by the total person-time). One person observed for one year = 1 person-year; one person observed for 3 months = 0.25 person year. the total person-time in year (or total person-years) will be the summation of person-years across all participants in the vaccine group and similarly across all participants in the placebo group. 

The point estimate of the VE can be written as:

If the clinical trial has a 1:1 randomization ratio (all participants are randomized equally into the vaccine group and the placebo group), the 'total person-time' in the vaccine group will be approximately equal to the 'total person-time' in the placebo group, the point estimate of VE can then be easily calculated as:

 

If we know the number of cases (here COVID-19 cases) in the vaccine group and in the placebo group, we can easily calculate the VE.  For example, if the total number of cases is 150 (50 cases observed in the vaccine group and 100 cases observed in the placebo group), the VE will be 1 - (50/100) = 0.5 = 50%.

If an interim analysis is performed after a total of 75 cases are observed, VE will be 50% if 25 cases are observed in the vaccine group and 50 cases are observed in the placebo group. 

Here is a comparison of VE calculations from three Phase III protocols of COVID-19 vaccines:

 

	Moderna	Pfizer	AstraZeneca
Primary efficacy endpoint	VE will be estimated with 1 - HR (mRNA-1273 vs placebo) using a Cox proportional hazard regression model with treatment group as a fixed effect and adjusting for stratification factor	VE will be estimated by 100 × (1 – IRR), where IRR is the  calculated ratio of confirmed COVID-19 illness per 1000 person-years follow-up in the active vaccine group to the corresponding illness rate in the placebo group 7 days after the last dose.	VE is calculated as RRR = 100*(1-relative risk), which RRR is the incidence of infection in the vaccine group relative to the incidence of infection in the control group expressed as a percentage.
Statistical model for calculating the VE and its 95% confidence interval	Cox proportional hazard model	Beta-binomial model	Modified Poisson regression model with robust variance
Sample size (number of volunteers to be recruited)	30,000	43,998	33,000
Number of cases needed to be observed	151	164	150

With the COVID-19 pandemic is still not under control in the US and a large number of volunteers participating in these phase III clinical trials, we hope that the total number of COVID-19 cases can be easily reached so that we can have a readout about the vaccine's efficacy. All three studies have included at least one interim analysis to have a possible readout much earlier. 

In addition to the requirement of at least 50% VE, FDA guidance also requires that the lower bound of the 95% confidence interval of the VE must be greater than 30%. 

The sample size (the number of COVID-19 infection cases) is largely dictated by this criterion of 30% for the lower bound of 95% CI. Otherwise, with 3 cases (1 case in the vaccine group and 2 cases in the placebo group), we would have a point estimate of VE = 50% to meet the requirement. 

50% VE implies that the vaccine can decrease the risk of COVID-19 cases by 50%. Comparing with other clinical trials, the 50% reduction is substantial and meaningful. In Moderna's trial, the VE will be estimated using the Cox proportional hazard model where the time to the first case of COVID-19 infection is also considered. In order to meet the criteria of at least 50% VE, the estimated hazard ratio (HR) needs to be equal to or less than 0.5. In oncology trials or in other clinical trials with time to event variables, if we can have an HR of 0.5 or lower, we will claim that the experimental treatment can reduce the risk of death or event by at least 50% - a result to die for. 

I agree with the statement about the COVID-19 vaccine from a Lancet paper:

“A vaccine that has 50% efficacy could appreciably reduce incidence of COVID-19 in vaccinated individuals, and might provide useful herd immunity. Hence, although efficacy far greater than 50% would be better, efficacy of about 50% would represent substantial progress.”

SARS-CoV-2 (the virus causing COVID-19) has a very high R0 (2.5 according to the table below) which estimates the speed at which a disease is capable of spreading in a population. We hope that we will have a vaccine that will meet the efficacy requirements of at least 50% VE in point estimate and at least 30% VE in the lower bound of 95% confidence interval. With an effective vaccine and the majority of people being vaccinated, we may be able to drop the transmissibility R0 below 1 to prevent the spread of the SARS-CoV-2.

     Comparing SARS-CoV-2 with SARS-CoV and influenza pandemics