How to interpret the risk ratio? - A controversy related to FDA's EUA announcement of convalescent plasma in treatment of Covid-19

This past Sunday, FDA Issues Emergency Use Authorization for Convalescent Plasma as Potential Promising COVID–19 Treatment, Another Achievement in Administration’s Fight Against Pandemic. The EUA (emergency use authorization) was mainly based on a study conducted by Mayo Clinic and the study results were described in the paper (not yet peer-reviewed yet) uploaded to Medrxiv.

Joyner 2020 “Effect of Convalescent Plasma on Mortality among Hospitalized Patients with COVID-19: Initial Three-Month Experience”.

 Issuance of EUA for convalescent plasma immediately draw some criticize about not having strong evidence to support the EUA approval and about the timing of the EUA announcement (right before the Republic National Convention) – We will leave this to others to debate.

One thing related to the statistics is how to present or interpret the results from Mayo Clinic’s study. In the paper, the author stated the hypotheses for analyzing the data from this study that has no concurrent control group.  

“We hypothesized, based on historical data that earlier administration of convalescent plasma with high antibody levels would be associated with reduced mortality. To address this hypothesis, we evaluated seven and 30-day mortality in 35,322 hospitalized adults transfused with COVID-19 convalescent plasma by asking two questions. First, was earlier treatment of patients with convalescent plasma after diagnosis of COVID-19 associated with reduced mortality compared to later treatment in the course of disease? Second, were higher antibody levels in the transfused convalescent plasma associated with reduced mortality?”

For the statistical analyses, the study cohort "was stratified into categories based on the days from COVID-19 diagnosis to plasma transfusion, including: 0, 1-3, 4-10, and 11 or more days and for some graphical presentations and analyses, dichotomized into 0-3 vs. 4 or more days". The relative risk was calculated for each sub-group for mortality for patients who received convalescent plasma with IgG S/co greater than 18.45 (high antibody level group) vs. patients that received less than 4.62 S/Co (low antibody level group). The pooled, or common, relative risk for 7-day and 30-day mortality were then calculated. The paper concluded, "the pooled relative risk of mortality among patients transfused with high antibody level plasma units was 0.65 [0.47-0.92] for 7 days and 0.77 [0.63-0.94] for 30 days compared to low antibody level plasma units."  For 7-day mortality, the forest plot was depicted below:

 

How to interpret the pooled relative risk of 0.65? FDA commissioner, Dr. Hahn, was criticized for overstating the efficacy results because of his interpretation of this relative risk. 

“In the optimal patients … treated with convalescent plasma at the highest titers, there was a 35% improvement in survival, which is a significant clinical benefit,” Hahn said during the press conference, noting that, “This clearly meets the criteria that we’ve established for emergency use authorization.”
 
Hahn went on to say that, “A 35% improvement in survival is a pretty substantial clinical benefit. What that means is—and if the data continue to pan out—100 people who are sick with COVID-19, 35 would have been saved because of the administration of plasma.”

Dr. Hahn's statement immediately drew a lot of criticizes and he had to come up to apologize for overstating the treatment effect of convalescent plasma in the treatment of COVID-19. 

• Lawmakers, experts raise questions after convalescent plasma EUA
• FDA chief apologizes for overstating plasma effect on virus
• FDA's Hahn Apologizes For Overselling Plasma's Benefits As A COVID-19 Treatment
• Covid-19 Live Updates: 2 F.D.A. Public Relations Experts Are Fired After Plasma Fiasco
• Is convalescent plasma safe and effective? We answer the major questions about the Covid-19 treatment

What is the right way to state the relative risk of 0.65? Well, relative risk is relative and can't be stated as an absolute benefit. CDC’s website “An Introduction to Applied Epidemiology and Biostatistics” has a chapter about the relative risk. It explains how the relative risk is calculated and how to interpret the results. 

In general, a risk ratio greater than 1.0 indicates an increased risk for the group in the numerator, usually the exposed group. A risk ratio of less than 1.0 indicates a decreased risk for the exposed group, indicating that perhaps exposure actually protects against disease occurrence.

With the convalescent plasma study by Mayo Clinic, the relative risk is calculated as the ratio of "the risk of 7-day mortality in patients receiving convalescent plasma with high antibody level" divided by "the risk of 7-day mortality in patients receiving convalescent plasma with low antibody level". A relative ratio of 0.65 (less than 1.0) indicates the benefit of receiving convalescent plasma with high antibody level - an indication of dose-response. 

It would be correct to state:

"The risk ratio of 0.65 indicates that COVID-19 patients who received convalescent plasma with high antibody level were only 65% as likely to die in 7 days as were patients who received convalescent plasma with low antibody level"

or

"The risk ratio of 0.65 indicates that the convalescent plasma with high antibody level reduced the risk of 7-day mortality by 35%"

Participants, Patients, Subjects, Volunteers, What to Use?

For people who participate in clinical trials, what should we call them? Subject, volunteer (healthy volunteer), participant, or patient? This seems like an easy question, but there are actually a lot of disagreements. Here are some of the articles and blogs discussing this:

Subject, Volunteer, Participant, or Patient?	The author clearly disliked the use of ‘subject’ for clinical trial participants. “In spite of its official sanctioned use, I’ve always objected to the term ‘subject’ to describe a person who donates time, effort and bodily fluids to further clinical research.  It has a negative connotation for me, conjuring up the image of a cold scientific investigation.  I wonder if people considering participating in clinical research studies are dissuaded by this term?  Do they feel they will be ‘subject’ ed to tests and procedures?   Acted upon, rather than participating in their health care?”
Revisiting What to Call People Who Participate in Clinical Research	The author prefers the term ‘participants’ to be used and even used the survey data to support the use of ‘participants’. “The New England Journal of Medicine, American Journal of Public Health, and International Committee of Medical Journal Editors all use the term participant exclusively. NIH Director Francis Collins is quoted as saying, “Medical advances would not be possible without participants in clinical trials.”
People are “participants” in research: Further suggestions for other terms to describe “participants” are needed	As the title suggested, “People are participants in research, not subjects”. Suggesting that the word “subject” should be banned from reports of research on humans. The word “subject” is demeaning.
When to use subjects, participants or volunteers to describe your research subjects	The author seems to be open to all of these terms that may be best used in different situations. Also provided are the definitions for each of these terms: Participants A participant is a person that voluntarily participates in a study. This is perhaps the most accepted terms and is generally recommended when in doubt (provided the subject of the investigation is human). Subjects The term subject describes the person or thing that is the topic of study.  Patients A patient is a participant with a medical condition which is the interest of the investigation.  Volunteers A volunteer is someone that freely offers to participate in a study. A volunteer is very similar to the participant and implies that the person as a whole is actively involved in the study. It also infers that they are free from any particular factor of interest, such as a medical condition.

The term ‘patient’ cannot cover all people who participate in clinical trials. While the clinical trials will ultimately be conducted in the patients who have a medical condition which is the interest of the investigation, clinical trial participants can also be volunteers or healthy volunteers, not patients. Here are some situations that the clinical trial participants are volunteers or healthy volunteers:

• Phase I clinical trials including first-in-human trials. The majority of phase I studies are conducted in healthy volunteers at a dedicated clinical research unit. Only in rare situations (such as oncology studies, studies using human-plasma derived products), are phase I studies conducted in patients.
• Vaccine clinical trials. In vaccine clinical trials, volunteers (or the participants who do not have the disease) are recruited.  The purpose of these clinical trials is to test if the vaccine will safe and effective against developing a disease.
• Preventive clinical trials. If the purpose of a clinical trial is to test if any therapeutic agent (not necessarily the vaccine) can prevent a disease, the participants will be volunteers and should not be called ‘patients’.
• Clinical trials in pregnant women. Pregnancy women are not patients.

People can get confused about using the term ‘volunteers’. In a co-ed by professors Emanuel and Offit, “Could Trump Turn a Vaccine Into a Campaign Stunt?”, the authors mistakenly used the term ‘patients’ for the planned vaccine clinical trials. For Covid-19 vaccine clinical trials, volunteers, healthy volunteers, or healthy volunteers with potential exposure to Covid-19 are recruited to participate in clinical trials. The term 'patient' should not be used. The term 'patient' can be used in therapeutical clinical trials where the purpose is to test if a drug/therapy (such as remdesivir, antibody cocktail,...) is effective in treating Covid-19 infected patients.

"Pfizer is planning to give its vaccine to approximately 8,000 patients. The N.I.H. is planning to enroll 30,000 participants — 20,000 getting a candidate vaccine and 10,000 research controls."

"Researchers are expecting that it will be likely to take at least another eight to 12 months to determine whether these coronavirus vaccines are effective. Scientists have to wait until a sufficient number of patients are exposed to coronavirus to see if the vaccine really reduces the infection rate, as well as how many people develop uncommon side effects. For comparison, the effectiveness trial for the rotavirus vaccines took about four years and the human papillomavirus vaccine studies to prevent cervical cancer took seven years."

“Scientists have to wait until a sufficient number of patients are exposed to coronavirus to see if the vaccine really reduces the infection rate, as well as how many people develop uncommon side effects. “

For years, we have been using the term ‘subject’ to describe clinical trial participants. The term ‘subject’ can cover all different types of clinical trial participants including patients and healthy volunteers. The term ‘subject’ reflects the facts (whether we like it or not) that the participants are subjects in clinical trials.

Many regulatory guidelines used the term ‘subject’ or ‘human subject’. As mentioned in an article “Comparison of FDA and HHS Human Subject Protection Regulations”:

"Human subject" means an individual who is or becomes a participant in research, either as a recipient of the test article or as a control. A subject may be either a healthy individual or a patient.

Human subject" means a living individual about whom an investigator (whether professional or student) conducting research obtains (1) data through intervention or interaction with the individual, or (2) identifiable private information.

There is a trend that the term ‘participant’ is commonly used to replace the term ‘subject’. For example, in Transcelerate Biopharma’s Common Protocol Template and NIH/FDA’s “Final “Phase 2 and 3 Clinical Trial” Template Documents”, the term ‘participant’ or ‘participants’ is used and the term ‘subject’ or ‘subjects’ is only used in special places. ‘Number of subjects’ is now called the ‘number of participants’.  

The term ‘patient’ may be preferred by doctors and study coordinators in investigational sites but is not usually used in clinical trial protocols. The term ‘patient’ may still be used in some special cases such as patient-reported outcome (PRO) and patient medical record. The term 'patient' is commonly used in publications for clinical trials conducted in patients. 

Even though the term ‘participant’ may be preferred in clinical trial protocols, the term ‘subject’ may be used in other clinical trial documents. For example, in case report form and database set up, the term 'subject' will continue to be used. The term ‘subject’ is the primary term used in CDISC documents such as CDASH (Clinical Data Acquisition Standards Harmonization) and SDTM (Study Data Tabulation Model) that we are following as the data standards. The variable Subjid is used for the subject identifier; the variable Usubjid is used for the unique subject identifier, …

In journal publications, we continue to see that different terms are used depending on the nature of the participants in a specific clinical trial. In the New England Journal of Medicine, the term ‘patient’ or ‘patients’ is used in clinical trials conducted in patients and the ‘participant’ or ‘participants’ is used in clinical trials conducted in volunteers or healthy volunteers. The term ‘subject’ is gradually phased out in publication. 

• in Jackson et al "An mRNA Vaccine against SARS-CoV-2 —Preliminary Report", the term 'participant' or 'participants' is used to describe the volunteers who participated in this vaccine trial. 
• in Beigel et al "Remdesivir for the Treatment of Covid-19 — Preliminary Report", the term 'patient' or 'patients' is used to describe the participants who are patients with Covid-19 infections. 

Unit of Analysis in Clinical Trials

The Unit of Analysis is the entity that frames what is being analyzed in a study or clinical trial. It is the entity being studied as a whole, within which most factors of causality and change exist. The unit of analysis is the “who” or the “what” that are being analyzed for a study or clinical trial. The Unit of Analysis is based on the experimental unit defined as "the smallest division of experimental material such that any two units may receive different treatments in the actual experiment" (Cox, 1992). Usually, the Unit of Analysis is on the same level as the unit for randomization.

We rarely talk about the Unit of Analysis, but actually, deal with it every time when we analyze the data. In clinical trials, we don’t explicitly talk about the Unit of Analysis because the Unit of Analysis is almost always the subject (or maybe called patient, health volunteer, or participant). Once the Unit of Analysis is established, all statistical analyses will be based on the Unit of Analysis – it means we count the number, perform the statistical model, include the explanatory variables all on the Unit of Analysis level. Given that the Unit of Analysis is ‘subject’ in clinical trials (in general), the subject level information or subject level variables will be used in analysis – that is why in CDISC ADaM data set, an ADSL (subject-level analysis dataset) will always be created.

The Unit of Analysis doesn’t have to be always the ‘subject’.

• For meta-analysis that is based on the summary information from multiple studies, the Unit of Analysis is ‘study’, not ‘subject’.

In a paper by Wong (2020) Estimation of clinical trial success rates and related parameters, the unit of analysis is 'study' or individual 'clinical trial'.  

• In analyses of Covid-19 data, all models are based on county-level or hospital-level data. Due to the concern about privacy, the data on the individuals is not available. See https://covidseverity.com/ website for county-level Covid-19 related data. Here the unit of analysis is 'county' or 'hospital'. 
• For studies using cluster randomization, the Unit of Analysis may be the cluster (township, city, household), not ‘subject’ even though the subject may be the observation unit.

In FDA’s guidance “Influenza: Developing Drugs for Treatment and/or Prophylaxis”, it specified that the Unit of Analysis could be the household.

"In household trials, the entire household is both the randomized unit and the unit of analysis. The primary efficacy analysis should compare the treatment groups for the percentage of households in which at least one randomized contact case developed symptomatic, laboratory-confirmed influenza. In other words, if one contact case in the household becomes symptomatically infected, the household is counted as infected. If none of the contact cases becomes infected, the household is considered not infected. Secondary analyses also can compare the percentage of contact cases that had symptomatic, laboratory-confirmed influenza in the active and placebo treatment groups.

Designs in which different contact cases in the same household receive different regimens raise concerns of drug sharing and intrahousehold correlation. Analysis using individual contact cases as the unit of analysis also may cause similar problems. Stratification on the size of household can be used, but is not expected to produce any consequential increase in power. "

• In some clinical trials, the Unit of Analysis may be smaller than the ‘subject’ level, for example, the tumor lesion in oncology studies, target bleeding site in studies for hemostasis agents.

In FDA Statistical Review for Lumason NDA, the Unit of Analysis using the lesion was performed

“The unit of analysis was the lesion; each subject had a single lesion that was to be characterized Sensitivity and Specificity are in percent (%) and n is the denominator for percentage calculation”

The Unit of Analysis may be different from the unit of observation. Within each unit of analysis, there may be multiple observations, for example, each subject with multiple events of hospitalization, exacerbation, adverse events. In this situation, we usually still analyze the data on the subject level and multiple events within a subject can be converted into the subject level data (time to first exacerbation, time to bleeding stoppage for the targeted bleeding site, best overall response based on the aggregated information from multiple lesions)

In FDA’s Statistical Review for Zerviate NDA, "The unit of analysis for all ocular variables was the average of both eyes of each subject."

In FDA’s review of Extended-Release and Long-Acting opioid analgesic (ER/LA) products, the unit of analysis is zip code (spatial) and quarter (time).

In both of the models proposed in the RADARS data analysis section, the unit of analysis is zip code (spatial) and quarter (time). Thus, testing for change between pre and post period for each outcome is investigating whether the average rate of events over time for the average zip code has changed from the pre-REMS period to the post-REMS period.

In clinicaltrials.gov, when clinical trial results are posted, the unit of analysis needs to be specified if the unit of analysis is not the subject.

"Type of Units Analyzed 

Definition: If the analysis is based on a unit other than participants, a description of the unit of analysis (for example, eyes, lesions, implants). "

In a handbook from Cochrane.org, there was a section to discuss the Unit of Analysis:

9.3.1 Unit-of-analysis issues

An important principle in clinical trials is that the analysis must take into account the level at which randomization occurred. In most circumstances the number of observations in the analysis should match the number of ‘units’ that were randomized. In a simple parallel group design for a clinical trial, participants are individually randomized to one of two intervention groups, and a single measurement for each outcome from each participant is collected and analysed. However, there are numerous variations on this design. Authors should consider whether in each study:

groups of individuals were randomized together to the same intervention (i.e. cluster-randomized trials);

individuals undergo more than one intervention (e.g. in a cross-over trial, or simultaneous treatment of multiple sites on each individual); or

there are multiple observations for the same outcome (e.g. repeated measurements, recurring events, measurements on different body parts).

There follows a more detailed list of situations in which unit-of-analysis issues commonly arise, together with directions to relevant discussions elsewhere in the Handbook.

Sometimes, the Unit of Analysis can be misused. In a paper by A. Vail and E. Gardener “Common statistical errors in the design and analysis of subfertility trials”, it said that “Most trials (82%) included at least one ‘unit of analysis’ error”. 

The most common error I can see is in the analysis of adverse events (AE). People can be confused with the different use of the Unit of Analysis. On the subject level, the adverse event should be analyzed to compare the incidence of AEs which is calculated as “the number of subjects with at least one specific AE divided by the number of subjects”. On the AE level, if we count the number of AEs, we can calculate the AE rate (number of AEs per subject; number of AEs per unit of exposure (person-year)) or AE density (number of AEs per drug infusion) – the meaning and interpretation are totally different than the incidence of AE.

In clinical trials with longitudinal design and crossover design, while the analyses will include the multiple measures for each individual subject, the unit of analysis is still the subject, but the more sophisticated statistical models (mixed model repeat measures, random coefficient model, multi-level or hierarchical linear models) will be needed. 

Time to Event Data: What to Present? Hazard Ratio, Median Time, Survival Rate?

One of the common endpoints in clinical trials is time to event as calculated as the duration from the time of randomization to the time of occurrence of the specific event (either the good or bad event). In oncology studies, the time to event variable can be overall survival (OS) as calculated from the time of randomization to the time of death or progression-free survival (PFS) as calculated from the time of randomization to the time of disease progression or death (whichever occurs first). In non-oncology studies, the time to event variable is everywhere:

• Time to first exacerbation in COPD, bronchiectasis 
• Time to first clinical worsening event in pulmonary hypertension
• Time to clinical recovery in COVID-19 therapeutical trials
• Time to healing of all non-aborted genital herpes lesions in recurrent genital herpes infection treatment studies 

While time to event may not be related to the death (survival), the time to event analysis is still commonly called 'survival analysis'. 

The statistical analyses for time to event variable include mainly the Kaplan-Meier estimate along with the log-rank test for different survival curves and Cox proportional hazard regression model (or Cox regression in short). 

The statistics can include survival rate (or rate of subjects without an event), median survival time (median time to event), hazard ratio, and their 95% confidence intervals. 

Survival rate (or rate of subjects without an event) is the percentage of subjects in a study or treatment group who are still alive for a certain period of time after they were randomized and started treatment for a disease, such as cancer. It may be called a milestone survival rate. A five-year survival rate will be the percentage of people in a study or treatment group who are alive five years after their randomization or the start of treatment. For clinical trials with short durations, usually, a short survival rate (for example, 6-month survival rate, 1-year survival rate, 3-year survival rate) will be more commonly used.  

Median survival is a statistic that refers to how long subjects survive with a disease in general or after the randomization or initiation of the treatment. It is the time — expressed in weeks, months, or years — when half the subjects are expected to be alive. It means that the chance of surviving beyond that time is 50 percent. similarly, median time to event is a statistic that refers to how long subjects have no specific event after the randomization or initiation of the treatment. It is the time — expressed in weeks, months or years — when half the subjects are expected to be event free. It means that the chance of having an event beyond that time is 50 percent.

Hazard ratio is the ratio of hazards and equals to the hazard rate in the treatment group ÷ the hazard rate in the control group. Hazard rate represents the instantaneous event rate, which means the probability that an individual would experience an event at a particular given point in time after the intervention. 

To present the analysis results for time to event variable, all different statistics can be displayed in the same table. The summary table can be designed as the following: 

Table 14.2.x 

Summary and Analysis of Time to XXX Event

ITT Analysis Set

	Test Drug (N=xx)	Control (N=xx)	p-value
Number of Subjects with Event (n, %)	xx (xx.x)	xx (xx.x)	0.xxx [1]
Number of Subjects Censored (n, %)	xx (xx.x)	xx (xx.x)
Time to XXX Event (time unit)
Kaplan-Meier Estimate			0.xxx [2]
25th Quartile (95% CI)	xx.x (xx.x, xx.x)	xx.x (xx.x, xx.x)
Median (95% CI)	xx.x (xx.x, xx.x)	xx.x (xx.x, xx.x)
75th Quartile (95% CI)	xx.x (xx.x, xx.x)	xx.x (xx.x, xx.x)
Rate (%) of Subjects without an    Event for at Least
1 time unit (95% CI)	xx.x (xx.x, xx.x)	xx.x (xx.x, xx.x)
2 time unit (95% CI)	xx.x (xx.x, xx.x)	xx.x (xx.x, xx.x)
3 time unit (95% CI)	xx.x (xx.x, xx.x)	xx.x (xx.x, xx.x)
4 time unit (95% CI)	xx.x (xx.x, xx.x)	xx.x (xx.x, xx.x)
Etc.
Hazard Ratio (95% CI) (Test Drug     vs Control) [3]	x.xx (x.xx, x.xx)		0.xxx [3]

[1] p-value is calculated with Fisher’s exact test.
[2] p-value is calculated with Logrank test stratified by strata1 and strata2.
[3] Hazard ratio, 95% CI, and p-value are calculated with Cox proportional hazard model with treatment, strata1, strata2 as explanatory variables.

Notice that all three statistics are included: median time to event (or median survival time), rate of subjects without an event (or survival rate), and hazard ratio. Three p-values are calculated: a p-value from Fisher's exact test (or Chi-square test) to compare the event rates between two groups - time was not factored in the calculation; a p-value from log-rank test to compare two survival curves; and p-value from Cox regression model.   

Survival rate is mostly used in oncology studies and rate of subjects with no event is not very commonly used in non-oncology studies. We still see some publications in oncology area where only survival rate is reported and neither the median time nor the hazard ratio is reported - seems to be a little bit obsolete practice. For example, almost all studies from the Children's Oncology Group would only report the survival rate, not the median survival time, not the hazard ratio. 

Median survival time is a good measure if there are enough events that occurred during the study period. If not too many events are observed in the treatment group during the study, the median survival time can not be calculated. 

Hazard ratio is a good measure for the treatment effect when comparing two treatment groups or two sub-groups. see a previous post "Interpreting Hazard Ratio: Can we say "percent reduction in risk"?"