How to interpret the risk ratio? - A controversy related to FDA's EUA announcement of convalescent plasma in treatment of Covid-19

This past Sunday, FDA Issues Emergency Use Authorization for Convalescent Plasma as Potential Promising COVID–19 Treatment, Another Achievement in Administration’s Fight Against Pandemic. The EUA (emergency use authorization) was mainly based on a study conducted by Mayo Clinic and the study results were described in the paper (not yet peer-reviewed yet) uploaded to Medrxiv.

Joyner 2020 “Effect of Convalescent Plasma on Mortality among Hospitalized Patients with COVID-19: Initial Three-Month Experience”.

 Issuance of EUA for convalescent plasma immediately draw some criticize about not having strong evidence to support the EUA approval and about the timing of the EUA announcement (right before the Republic National Convention) – We will leave this to others to debate.

One thing related to the statistics is how to present or interpret the results from Mayo Clinic’s study. In the paper, the author stated the hypotheses for analyzing the data from this study that has no concurrent control group.  

“We hypothesized, based on historical data that earlier administration of convalescent plasma with high antibody levels would be associated with reduced mortality. To address this hypothesis, we evaluated seven and 30-day mortality in 35,322 hospitalized adults transfused with COVID-19 convalescent plasma by asking two questions. First, was earlier treatment of patients with convalescent plasma after diagnosis of COVID-19 associated with reduced mortality compared to later treatment in the course of disease? Second, were higher antibody levels in the transfused convalescent plasma associated with reduced mortality?”

For the statistical analyses, the study cohort "was stratified into categories based on the days from COVID-19 diagnosis to plasma transfusion, including: 0, 1-3, 4-10, and 11 or more days and for some graphical presentations and analyses, dichotomized into 0-3 vs. 4 or more days". The relative risk was calculated for each sub-group for mortality for patients who received convalescent plasma with IgG S/co greater than 18.45 (high antibody level group) vs. patients that received less than 4.62 S/Co (low antibody level group). The pooled, or common, relative risk for 7-day and 30-day mortality were then calculated. The paper concluded, "the pooled relative risk of mortality among patients transfused with high antibody level plasma units was 0.65 [0.47-0.92] for 7 days and 0.77 [0.63-0.94] for 30 days compared to low antibody level plasma units."  For 7-day mortality, the forest plot was depicted below:

 

How to interpret the pooled relative risk of 0.65? FDA commissioner, Dr. Hahn, was criticized for overstating the efficacy results because of his interpretation of this relative risk. 

“In the optimal patients … treated with convalescent plasma at the highest titers, there was a 35% improvement in survival, which is a significant clinical benefit,” Hahn said during the press conference, noting that, “This clearly meets the criteria that we’ve established for emergency use authorization.”
 
Hahn went on to say that, “A 35% improvement in survival is a pretty substantial clinical benefit. What that means is—and if the data continue to pan out—100 people who are sick with COVID-19, 35 would have been saved because of the administration of plasma.”

Dr. Hahn's statement immediately drew a lot of criticizes and he had to come up to apologize for overstating the treatment effect of convalescent plasma in the treatment of COVID-19. 

• Lawmakers, experts raise questions after convalescent plasma EUA
• FDA chief apologizes for overstating plasma effect on virus
• FDA's Hahn Apologizes For Overselling Plasma's Benefits As A COVID-19 Treatment
• Covid-19 Live Updates: 2 F.D.A. Public Relations Experts Are Fired After Plasma Fiasco
• Is convalescent plasma safe and effective? We answer the major questions about the Covid-19 treatment

What is the right way to state the relative risk of 0.65? Well, relative risk is relative and can't be stated as an absolute benefit. CDC’s website “An Introduction to Applied Epidemiology and Biostatistics” has a chapter about the relative risk. It explains how the relative risk is calculated and how to interpret the results. 

In general, a risk ratio greater than 1.0 indicates an increased risk for the group in the numerator, usually the exposed group. A risk ratio of less than 1.0 indicates a decreased risk for the exposed group, indicating that perhaps exposure actually protects against disease occurrence.

With the convalescent plasma study by Mayo Clinic, the relative risk is calculated as the ratio of "the risk of 7-day mortality in patients receiving convalescent plasma with high antibody level" divided by "the risk of 7-day mortality in patients receiving convalescent plasma with low antibody level". A relative ratio of 0.65 (less than 1.0) indicates the benefit of receiving convalescent plasma with high antibody level - an indication of dose-response. 

It would be correct to state:

"The risk ratio of 0.65 indicates that COVID-19 patients who received convalescent plasma with high antibody level were only 65% as likely to die in 7 days as were patients who received convalescent plasma with low antibody level"

or

"The risk ratio of 0.65 indicates that the convalescent plasma with high antibody level reduced the risk of 7-day mortality by 35%"

Participants, Patients, Subjects, Volunteers, What to Use?

For people who participate in clinical trials, what should we call them? Subject, volunteer (healthy volunteer), participant, or patient? This seems like an easy question, but there are actually a lot of disagreements. Here are some of the articles and blogs discussing this:

Subject, Volunteer, Participant, or Patient?	The author clearly disliked the use of ‘subject’ for clinical trial participants. “In spite of its official sanctioned use, I’ve always objected to the term ‘subject’ to describe a person who donates time, effort and bodily fluids to further clinical research.  It has a negative connotation for me, conjuring up the image of a cold scientific investigation.  I wonder if people considering participating in clinical research studies are dissuaded by this term?  Do they feel they will be ‘subject’ ed to tests and procedures?   Acted upon, rather than participating in their health care?”
Revisiting What to Call People Who Participate in Clinical Research	The author prefers the term ‘participants’ to be used and even used the survey data to support the use of ‘participants’. “The New England Journal of Medicine, American Journal of Public Health, and International Committee of Medical Journal Editors all use the term participant exclusively. NIH Director Francis Collins is quoted as saying, “Medical advances would not be possible without participants in clinical trials.”
People are “participants” in research: Further suggestions for other terms to describe “participants” are needed	As the title suggested, “People are participants in research, not subjects”. Suggesting that the word “subject” should be banned from reports of research on humans. The word “subject” is demeaning.
When to use subjects, participants or volunteers to describe your research subjects	The author seems to be open to all of these terms that may be best used in different situations. Also provided are the definitions for each of these terms: Participants A participant is a person that voluntarily participates in a study. This is perhaps the most accepted terms and is generally recommended when in doubt (provided the subject of the investigation is human). Subjects The term subject describes the person or thing that is the topic of study.  Patients A patient is a participant with a medical condition which is the interest of the investigation.  Volunteers A volunteer is someone that freely offers to participate in a study. A volunteer is very similar to the participant and implies that the person as a whole is actively involved in the study. It also infers that they are free from any particular factor of interest, such as a medical condition.

The term ‘patient’ cannot cover all people who participate in clinical trials. While the clinical trials will ultimately be conducted in the patients who have a medical condition which is the interest of the investigation, clinical trial participants can also be volunteers or healthy volunteers, not patients. Here are some situations that the clinical trial participants are volunteers or healthy volunteers:

• Phase I clinical trials including first-in-human trials. The majority of phase I studies are conducted in healthy volunteers at a dedicated clinical research unit. Only in rare situations (such as oncology studies, studies using human-plasma derived products), are phase I studies conducted in patients.
• Vaccine clinical trials. In vaccine clinical trials, volunteers (or the participants who do not have the disease) are recruited.  The purpose of these clinical trials is to test if the vaccine will safe and effective against developing a disease.
• Preventive clinical trials. If the purpose of a clinical trial is to test if any therapeutic agent (not necessarily the vaccine) can prevent a disease, the participants will be volunteers and should not be called ‘patients’.
• Clinical trials in pregnant women. Pregnancy women are not patients.

People can get confused about using the term ‘volunteers’. In a co-ed by professors Emanuel and Offit, “Could Trump Turn a Vaccine Into a Campaign Stunt?”, the authors mistakenly used the term ‘patients’ for the planned vaccine clinical trials. For Covid-19 vaccine clinical trials, volunteers, healthy volunteers, or healthy volunteers with potential exposure to Covid-19 are recruited to participate in clinical trials. The term 'patient' should not be used. The term 'patient' can be used in therapeutical clinical trials where the purpose is to test if a drug/therapy (such as remdesivir, antibody cocktail,...) is effective in treating Covid-19 infected patients.

"Pfizer is planning to give its vaccine to approximately 8,000 patients. The N.I.H. is planning to enroll 30,000 participants — 20,000 getting a candidate vaccine and 10,000 research controls."

"Researchers are expecting that it will be likely to take at least another eight to 12 months to determine whether these coronavirus vaccines are effective. Scientists have to wait until a sufficient number of patients are exposed to coronavirus to see if the vaccine really reduces the infection rate, as well as how many people develop uncommon side effects. For comparison, the effectiveness trial for the rotavirus vaccines took about four years and the human papillomavirus vaccine studies to prevent cervical cancer took seven years."

“Scientists have to wait until a sufficient number of patients are exposed to coronavirus to see if the vaccine really reduces the infection rate, as well as how many people develop uncommon side effects. “

For years, we have been using the term ‘subject’ to describe clinical trial participants. The term ‘subject’ can cover all different types of clinical trial participants including patients and healthy volunteers. The term ‘subject’ reflects the facts (whether we like it or not) that the participants are subjects in clinical trials.

Many regulatory guidelines used the term ‘subject’ or ‘human subject’. As mentioned in an article “Comparison of FDA and HHS Human Subject Protection Regulations”:

"Human subject" means an individual who is or becomes a participant in research, either as a recipient of the test article or as a control. A subject may be either a healthy individual or a patient.

Human subject" means a living individual about whom an investigator (whether professional or student) conducting research obtains (1) data through intervention or interaction with the individual, or (2) identifiable private information.

There is a trend that the term ‘participant’ is commonly used to replace the term ‘subject’. For example, in Transcelerate Biopharma’s Common Protocol Template and NIH/FDA’s “Final “Phase 2 and 3 Clinical Trial” Template Documents”, the term ‘participant’ or ‘participants’ is used and the term ‘subject’ or ‘subjects’ is only used in special places. ‘Number of subjects’ is now called the ‘number of participants’.  

The term ‘patient’ may be preferred by doctors and study coordinators in investigational sites but is not usually used in clinical trial protocols. The term ‘patient’ may still be used in some special cases such as patient-reported outcome (PRO) and patient medical record. The term 'patient' is commonly used in publications for clinical trials conducted in patients. 

Even though the term ‘participant’ may be preferred in clinical trial protocols, the term ‘subject’ may be used in other clinical trial documents. For example, in case report form and database set up, the term 'subject' will continue to be used. The term ‘subject’ is the primary term used in CDISC documents such as CDASH (Clinical Data Acquisition Standards Harmonization) and SDTM (Study Data Tabulation Model) that we are following as the data standards. The variable Subjid is used for the subject identifier; the variable Usubjid is used for the unique subject identifier, …

In journal publications, we continue to see that different terms are used depending on the nature of the participants in a specific clinical trial. In the New England Journal of Medicine, the term ‘patient’ or ‘patients’ is used in clinical trials conducted in patients and the ‘participant’ or ‘participants’ is used in clinical trials conducted in volunteers or healthy volunteers. The term ‘subject’ is gradually phased out in publication. 

• in Jackson et al "An mRNA Vaccine against SARS-CoV-2 —Preliminary Report", the term 'participant' or 'participants' is used to describe the volunteers who participated in this vaccine trial. 
• in Beigel et al "Remdesivir for the Treatment of Covid-19 — Preliminary Report", the term 'patient' or 'patients' is used to describe the participants who are patients with Covid-19 infections. 

Unit of Analysis in Clinical Trials

The Unit of Analysis is the entity that frames what is being analyzed in a study or clinical trial. It is the entity being studied as a whole, within which most factors of causality and change exist. The unit of analysis is the “who” or the “what” that are being analyzed for a study or clinical trial. The Unit of Analysis is based on the experimental unit defined as "the smallest division of experimental material such that any two units may receive different treatments in the actual experiment" (Cox, 1992). Usually, the Unit of Analysis is on the same level as the unit for randomization.

We rarely talk about the Unit of Analysis, but actually, deal with it every time when we analyze the data. In clinical trials, we don’t explicitly talk about the Unit of Analysis because the Unit of Analysis is almost always the subject (or maybe called patient, health volunteer, or participant). Once the Unit of Analysis is established, all statistical analyses will be based on the Unit of Analysis – it means we count the number, perform the statistical model, include the explanatory variables all on the Unit of Analysis level. Given that the Unit of Analysis is ‘subject’ in clinical trials (in general), the subject level information or subject level variables will be used in analysis – that is why in CDISC ADaM data set, an ADSL (subject-level analysis dataset) will always be created.

The Unit of Analysis doesn’t have to be always the ‘subject’.

• For meta-analysis that is based on the summary information from multiple studies, the Unit of Analysis is ‘study’, not ‘subject’.

In a paper by Wong (2020) Estimation of clinical trial success rates and related parameters, the unit of analysis is 'study' or individual 'clinical trial'.  

• In analyses of Covid-19 data, all models are based on county-level or hospital-level data. Due to the concern about privacy, the data on the individuals is not available. See https://covidseverity.com/ website for county-level Covid-19 related data. Here the unit of analysis is 'county' or 'hospital'. 
• For studies using cluster randomization, the Unit of Analysis may be the cluster (township, city, household), not ‘subject’ even though the subject may be the observation unit.

In FDA’s guidance “Influenza: Developing Drugs for Treatment and/or Prophylaxis”, it specified that the Unit of Analysis could be the household.

"In household trials, the entire household is both the randomized unit and the unit of analysis. The primary efficacy analysis should compare the treatment groups for the percentage of households in which at least one randomized contact case developed symptomatic, laboratory-confirmed influenza. In other words, if one contact case in the household becomes symptomatically infected, the household is counted as infected. If none of the contact cases becomes infected, the household is considered not infected. Secondary analyses also can compare the percentage of contact cases that had symptomatic, laboratory-confirmed influenza in the active and placebo treatment groups.

Designs in which different contact cases in the same household receive different regimens raise concerns of drug sharing and intrahousehold correlation. Analysis using individual contact cases as the unit of analysis also may cause similar problems. Stratification on the size of household can be used, but is not expected to produce any consequential increase in power. "

• In some clinical trials, the Unit of Analysis may be smaller than the ‘subject’ level, for example, the tumor lesion in oncology studies, target bleeding site in studies for hemostasis agents.

In FDA Statistical Review for Lumason NDA, the Unit of Analysis using the lesion was performed

“The unit of analysis was the lesion; each subject had a single lesion that was to be characterized Sensitivity and Specificity are in percent (%) and n is the denominator for percentage calculation”

The Unit of Analysis may be different from the unit of observation. Within each unit of analysis, there may be multiple observations, for example, each subject with multiple events of hospitalization, exacerbation, adverse events. In this situation, we usually still analyze the data on the subject level and multiple events within a subject can be converted into the subject level data (time to first exacerbation, time to bleeding stoppage for the targeted bleeding site, best overall response based on the aggregated information from multiple lesions)

In FDA’s Statistical Review for Zerviate NDA, "The unit of analysis for all ocular variables was the average of both eyes of each subject."

In FDA’s review of Extended-Release and Long-Acting opioid analgesic (ER/LA) products, the unit of analysis is zip code (spatial) and quarter (time).

In both of the models proposed in the RADARS data analysis section, the unit of analysis is zip code (spatial) and quarter (time). Thus, testing for change between pre and post period for each outcome is investigating whether the average rate of events over time for the average zip code has changed from the pre-REMS period to the post-REMS period.

In clinicaltrials.gov, when clinical trial results are posted, the unit of analysis needs to be specified if the unit of analysis is not the subject.

"Type of Units Analyzed 

Definition: If the analysis is based on a unit other than participants, a description of the unit of analysis (for example, eyes, lesions, implants). "

In a handbook from Cochrane.org, there was a section to discuss the Unit of Analysis:

9.3.1 Unit-of-analysis issues

An important principle in clinical trials is that the analysis must take into account the level at which randomization occurred. In most circumstances the number of observations in the analysis should match the number of ‘units’ that were randomized. In a simple parallel group design for a clinical trial, participants are individually randomized to one of two intervention groups, and a single measurement for each outcome from each participant is collected and analysed. However, there are numerous variations on this design. Authors should consider whether in each study:

groups of individuals were randomized together to the same intervention (i.e. cluster-randomized trials);

individuals undergo more than one intervention (e.g. in a cross-over trial, or simultaneous treatment of multiple sites on each individual); or

there are multiple observations for the same outcome (e.g. repeated measurements, recurring events, measurements on different body parts).

There follows a more detailed list of situations in which unit-of-analysis issues commonly arise, together with directions to relevant discussions elsewhere in the Handbook.

Sometimes, the Unit of Analysis can be misused. In a paper by A. Vail and E. Gardener “Common statistical errors in the design and analysis of subfertility trials”, it said that “Most trials (82%) included at least one ‘unit of analysis’ error”. 

The most common error I can see is in the analysis of adverse events (AE). People can be confused with the different use of the Unit of Analysis. On the subject level, the adverse event should be analyzed to compare the incidence of AEs which is calculated as “the number of subjects with at least one specific AE divided by the number of subjects”. On the AE level, if we count the number of AEs, we can calculate the AE rate (number of AEs per subject; number of AEs per unit of exposure (person-year)) or AE density (number of AEs per drug infusion) – the meaning and interpretation are totally different than the incidence of AE.

In clinical trials with longitudinal design and crossover design, while the analyses will include the multiple measures for each individual subject, the unit of analysis is still the subject, but the more sophisticated statistical models (mixed model repeat measures, random coefficient model, multi-level or hierarchical linear models) will be needed. 

Time to Event Data: What to Present? Hazard Ratio, Median Time, Survival Rate?

One of the common endpoints in clinical trials is time to event as calculated as the duration from the time of randomization to the time of occurrence of the specific event (either the good or bad event). In oncology studies, the time to event variable can be overall survival (OS) as calculated from the time of randomization to the time of death or progression-free survival (PFS) as calculated from the time of randomization to the time of disease progression or death (whichever occurs first). In non-oncology studies, the time to event variable is everywhere:

• Time to first exacerbation in COPD, bronchiectasis 
• Time to first clinical worsening event in pulmonary hypertension
• Time to clinical recovery in COVID-19 therapeutical trials
• Time to healing of all non-aborted genital herpes lesions in recurrent genital herpes infection treatment studies 

While time to event may not be related to the death (survival), the time to event analysis is still commonly called 'survival analysis'. 

The statistical analyses for time to event variable include mainly the Kaplan-Meier estimate along with the log-rank test for different survival curves and Cox proportional hazard regression model (or Cox regression in short). 

The statistics can include survival rate (or rate of subjects without an event), median survival time (median time to event), hazard ratio, and their 95% confidence intervals. 

Survival rate (or rate of subjects without an event) is the percentage of subjects in a study or treatment group who are still alive for a certain period of time after they were randomized and started treatment for a disease, such as cancer. It may be called a milestone survival rate. A five-year survival rate will be the percentage of people in a study or treatment group who are alive five years after their randomization or the start of treatment. For clinical trials with short durations, usually, a short survival rate (for example, 6-month survival rate, 1-year survival rate, 3-year survival rate) will be more commonly used.  

Median survival is a statistic that refers to how long subjects survive with a disease in general or after the randomization or initiation of the treatment. It is the time — expressed in weeks, months, or years — when half the subjects are expected to be alive. It means that the chance of surviving beyond that time is 50 percent. similarly, median time to event is a statistic that refers to how long subjects have no specific event after the randomization or initiation of the treatment. It is the time — expressed in weeks, months or years — when half the subjects are expected to be event free. It means that the chance of having an event beyond that time is 50 percent.

Hazard ratio is the ratio of hazards and equals to the hazard rate in the treatment group ÷ the hazard rate in the control group. Hazard rate represents the instantaneous event rate, which means the probability that an individual would experience an event at a particular given point in time after the intervention. 

To present the analysis results for time to event variable, all different statistics can be displayed in the same table. The summary table can be designed as the following: 

Table 14.2.x 

Summary and Analysis of Time to XXX Event

ITT Analysis Set

	Test Drug (N=xx)	Control (N=xx)	p-value
Number of Subjects with Event (n, %)	xx (xx.x)	xx (xx.x)	0.xxx [1]
Number of Subjects Censored (n, %)	xx (xx.x)	xx (xx.x)
Time to XXX Event (time unit)
Kaplan-Meier Estimate			0.xxx [2]
25th Quartile (95% CI)	xx.x (xx.x, xx.x)	xx.x (xx.x, xx.x)
Median (95% CI)	xx.x (xx.x, xx.x)	xx.x (xx.x, xx.x)
75th Quartile (95% CI)	xx.x (xx.x, xx.x)	xx.x (xx.x, xx.x)
Rate (%) of Subjects without an    Event for at Least
1 time unit (95% CI)	xx.x (xx.x, xx.x)	xx.x (xx.x, xx.x)
2 time unit (95% CI)	xx.x (xx.x, xx.x)	xx.x (xx.x, xx.x)
3 time unit (95% CI)	xx.x (xx.x, xx.x)	xx.x (xx.x, xx.x)
4 time unit (95% CI)	xx.x (xx.x, xx.x)	xx.x (xx.x, xx.x)
Etc.
Hazard Ratio (95% CI) (Test Drug     vs Control) [3]	x.xx (x.xx, x.xx)		0.xxx [3]

[1] p-value is calculated with Fisher’s exact test.
[2] p-value is calculated with Logrank test stratified by strata1 and strata2.
[3] Hazard ratio, 95% CI, and p-value are calculated with Cox proportional hazard model with treatment, strata1, strata2 as explanatory variables.

Notice that all three statistics are included: median time to event (or median survival time), rate of subjects without an event (or survival rate), and hazard ratio. Three p-values are calculated: a p-value from Fisher's exact test (or Chi-square test) to compare the event rates between two groups - time was not factored in the calculation; a p-value from log-rank test to compare two survival curves; and p-value from Cox regression model.   

Survival rate is mostly used in oncology studies and rate of subjects with no event is not very commonly used in non-oncology studies. We still see some publications in oncology area where only survival rate is reported and neither the median time nor the hazard ratio is reported - seems to be a little bit obsolete practice. For example, almost all studies from the Children's Oncology Group would only report the survival rate, not the median survival time, not the hazard ratio. 

Median survival time is a good measure if there are enough events that occurred during the study period. If not too many events are observed in the treatment group during the study, the median survival time can not be calculated. 

Hazard ratio is a good measure for the treatment effect when comparing two treatment groups or two sub-groups. see a previous post "Interpreting Hazard Ratio: Can we say "percent reduction in risk"?"

Blinding and Masking Issue in Covid-19 Vaccine Clinical Trials

Clinical trials for Covid-19 vaccine development are moving into the critical late phase stage. The front runners right now are Moderna (in collaboration with NIAIH), Oxford University (in collaboration with AstraZeneca), and BioNTech (in collaboration with Pfizer). All three had published the positive results from their phase 1/2 studies to demonstrate that the Covid-19 vaccines can generate utilizing antibodies against SARS-COV-2 virus and vaccines are tolerable and generally safe in healthy volunteers. 

• Moderna Phase 1 results: Jackson et al (2020) An mRNA Vaccine against SARS-CoV-2 — Preliminary Report NEJM 
• Oxford/AstraZeneca Phase 1/2 results:  Folegatti et al (2020) Safety and immunogenicity of the ChAdOx1 nCoV-19 vaccine against SARS-CoV-2: a preliminary report of a phase 1/2, single-blind, randomised controlled trial. Lancet
• BioNTech/Pfizer Phase 1/2 results: Mulligan et al (2020) Phase 1/2 Study to Describe the Safety and Immunogenicity of a COVID-19 RNA Vaccine Candidate (BNT162b1) in Adults 18 to 55 Years of Age: Interim Report. medRxiV

The confirmatory studies are about to begin to demonstrate the efficacy and safety of the Covid-19 vaccines. The requirements for study design, efficacy endpoint, and safety endpoints are laid out in FDA's guidance "Development and Licensure of Vaccines to Prevent COVID-19"

Moderna is supposed to announce the start of phase 3 study next week. The other two will follow. The phase 3 studies from these three companies have already been registered in clinicaltrials.gov. The table below lists key parameters from these three studies. BioNTch/Pfizer had phase 1/2/3 studies combined in the same study protocol where the results from the phase 1 portion of the study have been published (see above Mulligan et al) 

	Moderna/NIAIH	Oxford/AstraZeneca	BioNTech/Pfizer
Protocol Title	A Phase 3, Randomized, Stratified, Observer-Blind, Placebo-Controlled Study to Evaluate the Efficacy, Safety, and Immunogenicity of mRNA-1273 SARS-CoV-2 Vaccine in Adults Aged 18 Years and Older	A Phase 2/3 Study to Determine the Efficacy, Safety and Immunogenicity of the Candidate Coronavirus Disease (COVID-19) Vaccine ChAdOx1 nCoV-19	A Phase 1/2/3, Placebo-Controlled, Randomized, Observer-Blind, Dose-Finding Study to Evaluate the Safety, Tolerability, Immunogenicity, and Efficacy of SARS-COV-2 RNA Vaccine Candidates Against COVID-19 in Healthy Adults
Phase	Phase 3	Phase 2/3	Phase 1/2/3
Sample Size	30,000	10,260	32,000
Treatment Groups	mRNA-1273 Placebo	ChAdOx1 nCoV-19 (Abs 260) MenACWY vaccine ChAdOx1 nCoV-19 (Abs 260) + 2.2x10^10vp (qPCR) boost Two dose MenACWY vaccine ChAdox1 n-CoV-19 (Abs 260) vaccine low dose ChAdOx1 nCoV-19 (qPCR) ChAdOx1 nCoV-19 plus 5x10^10vp boost (qPCR)	BNT162b1 BNT162b2 BNT162b3 Placebo
Age Groups	18 years and older	18 years or older 18-55 years 70 years and older 5-12 years inclusive	18-55 years of age 65-85 years of age 18-85 years of age
Number of Doses	100 microgram 2 doses (on day 1 and day 29)	1 or 2 doses	Low, low-mid, mid, or high doses 1 or 2 doses
Randomization	Randomized	Randomized	Randomized
Control Group	Placebo [0.9% sodium chloride (normal saline) injection]	MenACWY vaccine (also named Menveo or Nimenrix)	Placebo [a sterile saline solution for injection (0.9% sodium chloride injection, in a 0.5-mL dose)]
Blinding/Masking	Quadruple (Participant, Care Provider, Investigator, Outcome Assessor)	Single (Participant)	Triple (Participant, Care Provider, Investigator)

With the side-by-side comparison, we can see the clear difference in selecting the control group and how the blinding/masking is handled. In studies by Moderna and BioNTech, the control group is a placebo consisting of only the normal saline. But the quadruple and the triple masking (beyond the double-blind) are used to prevent the potential unblinding. 

 In the study by Oxford, the control group is another vaccine, MenACWY vaccine that is approved for protecting against meningococcal disease (meningitis and blood poisoning (septicaemia)) caused by serogroups A, C, W, and Y. The single blinding is used and the participants (volunteers) will not know whether they receive Covid-19 vaccine or MenACWY vaccine. 

In order to prevent potential unblinding - the participants become knowing which treatment they have received, using an active vaccine such as MenACWY that have been approved to be safe seems to be better and more adequate. In the publication of their phase 1 study results, they explained why it's necessary to use MenACWT vaccine as control. 

"MenACWY was used as a comparator vaccine to maintain blinding of participants who experienced local or systemic reactions, since these reactions are a known association with viral vector vaccinations. Use of saline as a placebo would risk unblinding participants as those who had notable reactions would know they were in the ChAdOx1 nCoV-19 vaccine group."

Placebo with saline as the control group is acceptable to FDA. In FDA's guidance "Development and Licensure of Vaccines to Prevent COVID-19", it says "Later phase trials, including efficacy trials, should be randomized, double-blinded, and placebo controlled" even though there is no mention about the requirement for the component of the placebo. 

With placebo (saline) as the control group, no matter whether the triple or quadruple blinding is used, there is still a potential unblinding by the participants because the participants can guess which treatment (Covid-19 vaccine or placebo) they have received based on the adverse events they may experience.

The published early phase results indicate that participants receiving Covid-19 vaccine experience more frequent adverse events in local injection site reactions and systemic reactions. BioNTech/Pfizer study says: 

"pain at the injection site was the most frequent prompted local reaction, reported after Dose 1 by 58.3% (7/12) in the 10 μg, 100.0% (12/12 each) in the 30 μg and 100 μg BNT162b1 groups, and by 22.2% (2/9) of placebo recipients. After Dose 2, pain was reported by 83.3% and 100.0% of BNT162b1 recipients at the 10 μg and 30 μg dose levels, respectively, and by 16.7 % of placebo recipients."

"Reports of fatigue and headache were more common in the BNT162b1 groups compared to placebo. Additionally, chills, muscle pain, and joint pain were reported among BNT162b1 recipients and not in placebo recipients."

After vaccination, participants may be able to guess they have received Covid-19 if they experience adverse events such as local injection site pain and systemic side effects such as fatigue, headache, chills, muscle pain,... They will be able to guess (pretty accurately) that they have received Placebo (saline) if they don't experience any local reactions or systemic side effects. 

If participants become aware of the treatment they have received, will it have an impact on their behavior? Will participants knowing to receive Covid-19 vaccine feel they have some protection, therefore maybe let loose their guard against Covid-19? I hope this will not be the case, otherwise, the biases induced by the behavior change because of the potential unblinding will have an impact on the efficacy results (most likely toward the null hypothesis of no difference).

Waterfall plot(s) to display the results in oncology and non-oncology clinical trials

The waterfall plot(s) started as a visualization tool in oncology studies to display the results of tumor burden, tumor size (or change in tumor size), the tumor responses have gained popularity and appeared in many publications. The application of the waterfall plot has gone beyond the oncology clinical trials. 

According to a paper by Gillespie (2012) Understanding Waterfall Plots,

Waterfall plots are graphic illustrations of data that can vary from audio frequencies to clinical trial patient information and results. In oncology, for example, a waterfall plot may be used to present each individual patient’s response to a particular drug based on a parameter, such as tumor burden. The horizontal (x) axis across the plot may serve as a baseline measure; vertical bars are drawn for each patient, either above or below the baseline. The vertical (y) axis may be used to measure maximum percent change from baseline, e.g., percent growth or reduction of the tumor by radiologic measurement. Those vertical bars that are above the line represent nonresponders or progressive disease. Vertical bars below the baseline (x) axis are drawn for each patient that has achieved some degree of tumor reduction, often depicted as negative percent.

In general, waterfall plots go from the worst value, such as greatest progression of disease, on the left side of the plot, to the best value, i.e., most reduction of tumor, on the right side of the plot; this can also be shown by shifting the graph to a similar presentation, moving from the worst outcomes on the bottom to the best outcomes on the top. The length of each vertical bar hanging below the horizontal axis increases as the plot moves to the right side of the graph, thus resembling a waterfall and giving the graph its name. Thus, the data are not presented randomly, or in order of when a patient first enrolled in a trial, but are organized in order to provide a clear picture of the study population’s results: from worst to best, based on the parameters included. 

The waterfall plot(s) has the following features: 

• It’s basically a bar graph, where each bar typically represents a patient; they are usually ordered from worst results to best.
• The horizontal axis is generally chosen to be a baseline measure, and the bars may go either above or below the baseline. 
• The x-axis is generally the subject number. If the x-axis is not labeled, it defaults to be the subject number. The subjects are listed according to the rank from worst results (on the left) to best results (on the right)
• The y-axis is generally used to quantify response to treatment; for instance, it might represent the percent of growth or reduction in a tumor while a patient is undergoing radiology. Negative bars would show reduction; positive bars would be patients whose cancer is still progressing or non-responders.
• For a study with multiple arms, each arm will have its own waterfall plot. For a study with three treatment arms, there will be three waterfall plots. The difference can be seen by comparing the patterns from different waterfall plots. 

In Advani (2018) CD47 Blockade by Hu5F9-G4 and Rituximab in Non-Hodgkin’s Lymphoma, a waterfall plot was used to display the change in tumor-lesion size with treatments of 5F9 and Rituximab. The waterfall plot showed the best overall change in the size of tumor target lesions among patients with diffuse large B-cell lymphoma (DLBCL; indicated by an asterisk) or follicular lymphoma, according to the maintenance dose received. The y-axis is the percentage changes in the tumor burden of target lesions and the x-axis is the patient number.

In Kopetz et al (2019) Encorafenib, Binimetinib, and Cetuximabin BRAF V600E–Mutated Colorectal Cancer, three waterfall plots were used to display the differences in patterns in best percentage change in the size of target lesions among three treatment groups (triple-therapy, double-therapy, and control groups). Notice that each treatment group has its own waterfall plot. Y-axis is the best percentage change from baseline in tumor size of target lesion. The X-axis is the subject number (even though it is not labeled). 

Waterfall plot(s) has been used in studies beyond the oncology studies. Here are some examples:

In Vichinsky et al (2019) A Phase 3 Randomized Trial of Voxelotor in Sickle Cell Disease, three waterfall plots were used to display the treatment effect in change in hemoglobin level of Vexelotor comparing to Placebo. The y-axis is the change in hemoglobin level from baseline to week 24 (g/dL) and the x-axis is the subject number for each treatment group (even though it is not labeled). 

In Nathan et al (2020) Efficacy of Pirfenidone in the Context of Multiple Disease Progression Events in Patients With Idiopathic Pulmonary Fibrosis, two colorful waterfall plots (one for pirfenidone group and one for the placebo group) were used to display the pattern and distribution of frequency and type of adverse outcome (or disease progression) events including the decline in 6MWD, the decline in %FVC, respiratory-related hospitalization, death, and combination of them. The y-axis is the number of events and the x-axis is the patient number for each treatment group. 

In a retrospective pretest-posttest study with no controls by Sanchez et al (2019) Multiple lifestyle interventions reverses hypertension, two waterfall plots (one for SBP and one for DBP) were used to display the pre-post change in systolic and diastolic blood pressure to indicate the NEWSTART Lifestyle intervention was an effective and rapid means to decrease SBP and DBP.

While waterfall plots can visually show the treatment effects either change from baseline or between treatment groups, there are drawbacks as well. 

According to Kim et al (2019) Assessment of Accuracy of Waterfall Plot Representations of Response Rates in Cancer Treatment Published in Medical Journals, the article assessed 126 studies published in 6 journals where waterfall plots were used to show visual response rates. The author concludes that that waterfall plots are used more frequently over time and exaggerate the visual estimate of the response rate.

In a paper by Shao et al Use and Misuse of Waterfall Plots, the authors concluded that "there was substantial variability in criteria used to generate published waterfall plots. Waterfall plots are subject to substantial variability in criteria used to define them and are influenced by measurement errors; they should be generated by trained radiologists. Caution should be exercised when interpreting the results of waterfall plots in the context of clinical trials."

Waterfall plots can be generated in SAS. There are quite some papers discussing the tips and tricks in generating waterfall plots: 

• Create a waterfall plot in SAS
• Waterfall plot: two different approaches, one beautiful graph 
• Graphical Results in Clinical Oncology Studies
• Useful SAS techniques in Efficacy Analysis for Oncology studies
• Plotting Against Cancer: Creating Oncology Plots Using SAS®