Blinding and Masking Issue in Covid-19 Vaccine Clinical Trials

Clinical trials for Covid-19 vaccine development are moving into the critical late phase stage. The front runners right now are Moderna (in collaboration with NIAIH), Oxford University (in collaboration with AstraZeneca), and BioNTech (in collaboration with Pfizer). All three had published the positive results from their phase 1/2 studies to demonstrate that the Covid-19 vaccines can generate utilizing antibodies against SARS-COV-2 virus and vaccines are tolerable and generally safe in healthy volunteers. 

• Moderna Phase 1 results: Jackson et al (2020) An mRNA Vaccine against SARS-CoV-2 — Preliminary Report NEJM 
• Oxford/AstraZeneca Phase 1/2 results:  Folegatti et al (2020) Safety and immunogenicity of the ChAdOx1 nCoV-19 vaccine against SARS-CoV-2: a preliminary report of a phase 1/2, single-blind, randomised controlled trial. Lancet
• BioNTech/Pfizer Phase 1/2 results: Mulligan et al (2020) Phase 1/2 Study to Describe the Safety and Immunogenicity of a COVID-19 RNA Vaccine Candidate (BNT162b1) in Adults 18 to 55 Years of Age: Interim Report. medRxiV

The confirmatory studies are about to begin to demonstrate the efficacy and safety of the Covid-19 vaccines. The requirements for study design, efficacy endpoint, and safety endpoints are laid out in FDA's guidance "Development and Licensure of Vaccines to Prevent COVID-19"

Moderna is supposed to announce the start of phase 3 study next week. The other two will follow. The phase 3 studies from these three companies have already been registered in clinicaltrials.gov. The table below lists key parameters from these three studies. BioNTch/Pfizer had phase 1/2/3 studies combined in the same study protocol where the results from the phase 1 portion of the study have been published (see above Mulligan et al) 

	Moderna/NIAIH	Oxford/AstraZeneca	BioNTech/Pfizer
Protocol Title	A Phase 3, Randomized, Stratified, Observer-Blind, Placebo-Controlled Study to Evaluate the Efficacy, Safety, and Immunogenicity of mRNA-1273 SARS-CoV-2 Vaccine in Adults Aged 18 Years and Older	A Phase 2/3 Study to Determine the Efficacy, Safety and Immunogenicity of the Candidate Coronavirus Disease (COVID-19) Vaccine ChAdOx1 nCoV-19	A Phase 1/2/3, Placebo-Controlled, Randomized, Observer-Blind, Dose-Finding Study to Evaluate the Safety, Tolerability, Immunogenicity, and Efficacy of SARS-COV-2 RNA Vaccine Candidates Against COVID-19 in Healthy Adults
Phase	Phase 3	Phase 2/3	Phase 1/2/3
Sample Size	30,000	10,260	32,000
Treatment Groups	mRNA-1273 Placebo	ChAdOx1 nCoV-19 (Abs 260) MenACWY vaccine ChAdOx1 nCoV-19 (Abs 260) + 2.2x10^10vp (qPCR) boost Two dose MenACWY vaccine ChAdox1 n-CoV-19 (Abs 260) vaccine low dose ChAdOx1 nCoV-19 (qPCR) ChAdOx1 nCoV-19 plus 5x10^10vp boost (qPCR)	BNT162b1 BNT162b2 BNT162b3 Placebo
Age Groups	18 years and older	18 years or older 18-55 years 70 years and older 5-12 years inclusive	18-55 years of age 65-85 years of age 18-85 years of age
Number of Doses	100 microgram 2 doses (on day 1 and day 29)	1 or 2 doses	Low, low-mid, mid, or high doses 1 or 2 doses
Randomization	Randomized	Randomized	Randomized
Control Group	Placebo [0.9% sodium chloride (normal saline) injection]	MenACWY vaccine (also named Menveo or Nimenrix)	Placebo [a sterile saline solution for injection (0.9% sodium chloride injection, in a 0.5-mL dose)]
Blinding/Masking	Quadruple (Participant, Care Provider, Investigator, Outcome Assessor)	Single (Participant)	Triple (Participant, Care Provider, Investigator)

With the side-by-side comparison, we can see the clear difference in selecting the control group and how the blinding/masking is handled. In studies by Moderna and BioNTech, the control group is a placebo consisting of only the normal saline. But the quadruple and the triple masking (beyond the double-blind) are used to prevent the potential unblinding. 

 In the study by Oxford, the control group is another vaccine, MenACWY vaccine that is approved for protecting against meningococcal disease (meningitis and blood poisoning (septicaemia)) caused by serogroups A, C, W, and Y. The single blinding is used and the participants (volunteers) will not know whether they receive Covid-19 vaccine or MenACWY vaccine. 

In order to prevent potential unblinding - the participants become knowing which treatment they have received, using an active vaccine such as MenACWY that have been approved to be safe seems to be better and more adequate. In the publication of their phase 1 study results, they explained why it's necessary to use MenACWT vaccine as control. 

"MenACWY was used as a comparator vaccine to maintain blinding of participants who experienced local or systemic reactions, since these reactions are a known association with viral vector vaccinations. Use of saline as a placebo would risk unblinding participants as those who had notable reactions would know they were in the ChAdOx1 nCoV-19 vaccine group."

Placebo with saline as the control group is acceptable to FDA. In FDA's guidance "Development and Licensure of Vaccines to Prevent COVID-19", it says "Later phase trials, including efficacy trials, should be randomized, double-blinded, and placebo controlled" even though there is no mention about the requirement for the component of the placebo. 

With placebo (saline) as the control group, no matter whether the triple or quadruple blinding is used, there is still a potential unblinding by the participants because the participants can guess which treatment (Covid-19 vaccine or placebo) they have received based on the adverse events they may experience.

The published early phase results indicate that participants receiving Covid-19 vaccine experience more frequent adverse events in local injection site reactions and systemic reactions. BioNTech/Pfizer study says: 

"pain at the injection site was the most frequent prompted local reaction, reported after Dose 1 by 58.3% (7/12) in the 10 μg, 100.0% (12/12 each) in the 30 μg and 100 μg BNT162b1 groups, and by 22.2% (2/9) of placebo recipients. After Dose 2, pain was reported by 83.3% and 100.0% of BNT162b1 recipients at the 10 μg and 30 μg dose levels, respectively, and by 16.7 % of placebo recipients."

"Reports of fatigue and headache were more common in the BNT162b1 groups compared to placebo. Additionally, chills, muscle pain, and joint pain were reported among BNT162b1 recipients and not in placebo recipients."

After vaccination, participants may be able to guess they have received Covid-19 if they experience adverse events such as local injection site pain and systemic side effects such as fatigue, headache, chills, muscle pain,... They will be able to guess (pretty accurately) that they have received Placebo (saline) if they don't experience any local reactions or systemic side effects. 

If participants become aware of the treatment they have received, will it have an impact on their behavior? Will participants knowing to receive Covid-19 vaccine feel they have some protection, therefore maybe let loose their guard against Covid-19? I hope this will not be the case, otherwise, the biases induced by the behavior change because of the potential unblinding will have an impact on the efficacy results (most likely toward the null hypothesis of no difference).

Waterfall plot(s) to display the results in oncology and non-oncology clinical trials

The waterfall plot(s) started as a visualization tool in oncology studies to display the results of tumor burden, tumor size (or change in tumor size), the tumor responses have gained popularity and appeared in many publications. The application of the waterfall plot has gone beyond the oncology clinical trials. 

According to a paper by Gillespie (2012) Understanding Waterfall Plots,

Waterfall plots are graphic illustrations of data that can vary from audio frequencies to clinical trial patient information and results. In oncology, for example, a waterfall plot may be used to present each individual patient’s response to a particular drug based on a parameter, such as tumor burden. The horizontal (x) axis across the plot may serve as a baseline measure; vertical bars are drawn for each patient, either above or below the baseline. The vertical (y) axis may be used to measure maximum percent change from baseline, e.g., percent growth or reduction of the tumor by radiologic measurement. Those vertical bars that are above the line represent nonresponders or progressive disease. Vertical bars below the baseline (x) axis are drawn for each patient that has achieved some degree of tumor reduction, often depicted as negative percent.

In general, waterfall plots go from the worst value, such as greatest progression of disease, on the left side of the plot, to the best value, i.e., most reduction of tumor, on the right side of the plot; this can also be shown by shifting the graph to a similar presentation, moving from the worst outcomes on the bottom to the best outcomes on the top. The length of each vertical bar hanging below the horizontal axis increases as the plot moves to the right side of the graph, thus resembling a waterfall and giving the graph its name. Thus, the data are not presented randomly, or in order of when a patient first enrolled in a trial, but are organized in order to provide a clear picture of the study population’s results: from worst to best, based on the parameters included. 

The waterfall plot(s) has the following features: 

• It’s basically a bar graph, where each bar typically represents a patient; they are usually ordered from worst results to best.
• The horizontal axis is generally chosen to be a baseline measure, and the bars may go either above or below the baseline. 
• The x-axis is generally the subject number. If the x-axis is not labeled, it defaults to be the subject number. The subjects are listed according to the rank from worst results (on the left) to best results (on the right)
• The y-axis is generally used to quantify response to treatment; for instance, it might represent the percent of growth or reduction in a tumor while a patient is undergoing radiology. Negative bars would show reduction; positive bars would be patients whose cancer is still progressing or non-responders.
• For a study with multiple arms, each arm will have its own waterfall plot. For a study with three treatment arms, there will be three waterfall plots. The difference can be seen by comparing the patterns from different waterfall plots. 

In Advani (2018) CD47 Blockade by Hu5F9-G4 and Rituximab in Non-Hodgkin’s Lymphoma, a waterfall plot was used to display the change in tumor-lesion size with treatments of 5F9 and Rituximab. The waterfall plot showed the best overall change in the size of tumor target lesions among patients with diffuse large B-cell lymphoma (DLBCL; indicated by an asterisk) or follicular lymphoma, according to the maintenance dose received. The y-axis is the percentage changes in the tumor burden of target lesions and the x-axis is the patient number.

In Kopetz et al (2019) Encorafenib, Binimetinib, and Cetuximabin BRAF V600E–Mutated Colorectal Cancer, three waterfall plots were used to display the differences in patterns in best percentage change in the size of target lesions among three treatment groups (triple-therapy, double-therapy, and control groups). Notice that each treatment group has its own waterfall plot. Y-axis is the best percentage change from baseline in tumor size of target lesion. The X-axis is the subject number (even though it is not labeled). 

Waterfall plot(s) has been used in studies beyond the oncology studies. Here are some examples:

In Vichinsky et al (2019) A Phase 3 Randomized Trial of Voxelotor in Sickle Cell Disease, three waterfall plots were used to display the treatment effect in change in hemoglobin level of Vexelotor comparing to Placebo. The y-axis is the change in hemoglobin level from baseline to week 24 (g/dL) and the x-axis is the subject number for each treatment group (even though it is not labeled). 

In Nathan et al (2020) Efficacy of Pirfenidone in the Context of Multiple Disease Progression Events in Patients With Idiopathic Pulmonary Fibrosis, two colorful waterfall plots (one for pirfenidone group and one for the placebo group) were used to display the pattern and distribution of frequency and type of adverse outcome (or disease progression) events including the decline in 6MWD, the decline in %FVC, respiratory-related hospitalization, death, and combination of them. The y-axis is the number of events and the x-axis is the patient number for each treatment group. 

In a retrospective pretest-posttest study with no controls by Sanchez et al (2019) Multiple lifestyle interventions reverses hypertension, two waterfall plots (one for SBP and one for DBP) were used to display the pre-post change in systolic and diastolic blood pressure to indicate the NEWSTART Lifestyle intervention was an effective and rapid means to decrease SBP and DBP.

While waterfall plots can visually show the treatment effects either change from baseline or between treatment groups, there are drawbacks as well. 

According to Kim et al (2019) Assessment of Accuracy of Waterfall Plot Representations of Response Rates in Cancer Treatment Published in Medical Journals, the article assessed 126 studies published in 6 journals where waterfall plots were used to show visual response rates. The author concludes that that waterfall plots are used more frequently over time and exaggerate the visual estimate of the response rate.

In a paper by Shao et al Use and Misuse of Waterfall Plots, the authors concluded that "there was substantial variability in criteria used to generate published waterfall plots. Waterfall plots are subject to substantial variability in criteria used to define them and are influenced by measurement errors; they should be generated by trained radiologists. Caution should be exercised when interpreting the results of waterfall plots in the context of clinical trials."

Waterfall plots can be generated in SAS. There are quite some papers discussing the tips and tricks in generating waterfall plots: 

• Create a waterfall plot in SAS
• Waterfall plot: two different approaches, one beautiful graph 
• Graphical Results in Clinical Oncology Studies
• Useful SAS techniques in Efficacy Analysis for Oncology studies
• Plotting Against Cancer: Creating Oncology Plots Using SAS®

FDA Guidance "Development and Licensure of Vaccines to Prevent COVID-19" - sample size situation for phase 3 studies

Last week, FDA issued its guidance for industry "Development and Licensure of Vaccines to Prevent COVID-19". Unlike the usual FDA guidance where FDA issues the draft guidance with a comment period, this guidance is immediately effective as the final version upon its issuance.

The guidance sets its expectations for the development and licensure of vaccines to prevent coronavirus disease (COVID-19), including considerations for manufacturing, nonclinical and clinical studies, and post-licensure requirements. Also, Dr. Peter Marks, director of the Center for Biologics Evaluation and Research, shed light on the reasoning behind the agency’s 50% efficacy threshold and where the agency stands on challenge trials and emergency use authorizations (EUAs). See the article "Marks on COVID-19 vaccine efficacy, EUAs and challenge trials"

The guidance provided details about the pivotal (phase 3) efficacy and safety study. For phase 3 study, the primary efficacy endpoint should be "the incidence of laboratory-confirmed symptomatic COVID-19" specified in the guidance as the following: 

Section E of the guidance 'Statistical Considerations' provided the specific requirements for the statistical success criteria (i.e., point estimate of vaccine efficacy at least 50% and the lower bound of the alpha-adjusted confidence interval at least 30%). 

What does the vaccine efficacy of 50% mean?

Vaccine Efficacy (VE) = [(COVID-19 attack rate in the unvaccinated group - COVID-19 attack rate in the vaccinated group) / COVID-19 attack rate in the unvaccinated group] * 100%

where the attack rate is equivalent to the incidence rate. 

Using the relative risk (RR) or risk ratio [= (incidence of COVID-19 cases in the vaccinated group) / (incidence of COVID-19 cases in the unvaccinated group)], VE = 1 - RR.

Suppose after 3-6 months observation period post-vaccination, there are 100 cases of laboratory-confirmed symptomatic COVID-19 patients in the unvaccinated group and 50 cases in the vaccinated group, assuming the total follow-up time (person-time) are similar between the unvaccinated and vaccinated groups, the VE will be calculated as (100-50)/100 *100%= 50%.

In practice, the person time (PT) in the vaccinated group and unvaccinated group will be included in the calculation of attack rate or incidence rate where the attack rate = the number of cases observed in the vaccination group or unvaccinated group / Person Time (PT) in the vaccination or vaccination group. If the Poisson regression method is used, the person time will be used in the model as an offset variable. Person time (PT) is the same concept as person-year or patient-year and can be calculated in the same way as the person year with a perhaps different unit. 

VE at least 50% is a point estimate - not dependent on the sample size. For a much smaller trial, if we have 5 cases in the vaccination group and 10 cases in the vaccination group, the VE will still be 50%.

For sample size calculation, we will also need to know the confidence interval. As indicated in the FDA's guidance,  the lower bound of the appropriately alpha-adjusted confidence interval around the primary efficacy endpoint point estimate needs to be greater than 30%.

How to calculate the sample size? Which parameters do we need to calculate the sample size? 

The commercial software (such as EAST, SAS Proc Power, PASS, NQuery Advisor) can all be used to calculate the sample size. In a hypothetic example in the previous post, the sample size was calculated using EAST module for Ratio of Poisson Rates. 

Matthew M. Loiacono had a thesis on "Sample Size Estimation and Power Calculations for Vaccine Efficacy Trials for Exceedingly Rare Diseases" where two SAS macros were developed to estimate the sample sizes: one based on the Exact Conditional Test and one based on the Z Test Normal Approximation.

The sample size calculation will need the following five parameters:

• Incidence of laboratory-confirmed symptomatic COVID-19 cases in the unvaccinated group 
• True efficacy of test vaccine under the alternative hypothesis (according to FDA guidance, this is 50%) 
• Minimum efficacy of test vaccine, under the null hypothesis (according to FDA guidance, this is 30%)
• Power: pre-specified statistical power desired to achieve (usually 80% or 90%) 
• Alpha: pre-specified maximum one-sided level of the test (usually 0.05 for experimental level)

The most critical parameter is the incidence of laboratory-confirmed symptomatic COVID-19 cases - it is difficult to predict; it shifts with geographic location and time; it is impacted by the COVID-19 prevention strategies and policies. In general, the lower the incidence of COVID-19 cases, the larger the sample size is needed for phase 3 study to demonstrate the vaccine efficacy. 

Assuming the incidence of COVID-19 is 0.01 in unvaccinated (placebo) group, with 80% statistical power and alpha = 0.05, using the SAS macro based on Exact Conditional Test method, 33868 volunteers need to be randomized (estimated 254 COVID-19 cases observed) to detect the vaccine efficacy with point estimate at least 50% and lower bound of 95% confidence interval greater than 30%.