In our daily life, benefit-risk evaluation is something we always do whether we realize it or not. Benefit-risk evaluation is especially critical in drug development and in the regulator's decision process. We often hear that a drug is approved because the benefits outweigh the risks. In the recent decision of resuming the J&J Covid-19 vaccine, the CDC and the FDA cited that the benefits of rolling out the J&J Covid vaccine outweigh the risks of developing the rare blood clot (so-called CVST Cerebral Venous Sinus Thrombosis) in some young women who received the J&J Covid vaccine.
In a recent New York Times article "Irrational Covid Fears", the benefit and risk of the Covid-19 vaccine are compared to a fable of our times and automobiles.
A fable for our times
Guido Calabresi, a federal judge and Yale law professor, invented a little fable that he has been telling law students for more than three decades.
He tells the students to imagine a god coming forth to offer society a wondrous invention that would improve everyday life in almost every way. It would allow people to spend more time with friends and family, see new places and do jobs they otherwise could not do. But it would also come with a high cost. In exchange for bestowing this invention on society, the god would choose 1,000 young men and women and strike them dead.
Calabresi then asks: Would you take the deal? Almost invariably, the students say no. The professor then delivers the fable’s lesson: “What’s the difference between this and the automobile?”
In truth, automobiles kill many more than 1,000 young Americans each year; the total U.S. death toll hovers at about 40,000 annually. We accept this toll, almost unthinkingly, because vehicle crashes have always been part of our lives. We can’t fathom a world without them.
It’s a classic example of human irrationality about risk. We often underestimate large, chronic dangers, like car crashes or chemical pollution, and fixate on tiny but salient risks, like plane crashes or shark attacks.
One way for a risk to become salient is for it to be new. That’s a core idea behind Calabresi’s fable. He asks students to consider whether they would accept the cost of vehicle travel if it did not already exist. That they say no underscores the very different ways we treat new risks and enduring ones.
I have been thinking about the fable recently because of Covid-19. Covid certainly presents a salient risk: It’s a global pandemic that has upended daily life for more than a year. It has changed how we live, where we work, even what we wear on our faces. Covid feels ubiquitous.
Fortunately, it is also curable. The vaccines have nearly eliminated death, hospitalization and other serious Covid illness among people who have received shots. The vaccines have also radically reduced the chances that people contract even a mild version of Covid or can pass it on to others.
Yet many vaccinated people continue to obsess over the risks from Covid — because they are so new and salient.
This article reminds me of the seminars presented by Scott Evans. In his seminars, for example, the one posted on youtube, he started with a hypothetical question:
If you are given a choice to choose drug A or drug B, Drug A increases your intelligence, but decreases your good looks; Drug B increases your good looks, but decreases your intelligence; which drug will you choose?
This is a typical question about the benefit-risk evaluation or benefit-risk tradeoff. With this question, he brought up a topic about an alternative (supposed to be optimal) way to perform the benefit-risk evaluation (i.e., the benefit-risk assessment on each individual patient level before aggregating the data on the group level).
- Using Outcomes to Analyze Patients Rather than Patients to Analyze Outcomes: A Step toward Pragmatism in Benefit:risk Evaluation
- Methodologies for pragmatic and efficient assessment of benefits and harms: Application to the SOCRATES trial
- Pragmatic Benefit:Risk Evaluation: Healthy Disruption for Clinical Trials and Diagnostic Studies
Currently, in clinical trials, the benefit (efficacy) evaluation and risk (safety) evaluation are performed independently. The study protocol was designed for showing the benefit (efficacy) - selecting the sensitive and clinically meaningful efficacy endpoint, ensuring sufficient large sample size for statistical power, sound statistical analysis methods are all for ensuring that the efficacy results can be used to demonstrate the benefit of the new drug. FDA has issued specific guidance only for efficacy "Demonstrating Substantial Evidence of Effectiveness for Human Drug and Biological Products".
Risk (safety) evaluation is usually assessed separately from the efficacy. While we collect the data for risk (safety) analysis (adverse events, serious adverse events, death, clinical laboratory results, ECG results, vital signs,...), the analyses of safety data are usually based on the summaries (no hypothesis testing) to assess the nature/pattern of the serious adverse events, related to the investigational new drug, if there is elevated levels in certain laboratory parameters,... Safety analyses contain a lot of subjective judgment. Different reviewers may come to different conclusions.
There is no separate guidance from FDA specifically about the risk (safety assessment). Instead, the safety assessment is included in FDA's Good Review Practice: Clinical Review Template - a checklist for FDA reviewers in evaluating the safety.
Only after the efficacy and safety are separately analyzed and evaluated, are a benefit-risk section written as a formal evaluation of the benefit-risk - this is usually in CTD module 1 and 2.
This approach of assessing the efficacy and safety separately evaluates the average effect (efficacy or safety) in the entire study population. The benefit or risk can not be easily translated into the individual patient level. In clinical trials, it is almost impossible to decide if a drug is good (the benefit outweighs the risk) for a specific patient. We have to wait for the aggregate data to determine the benefit and risk on a group level.
With advances in precision medicine and pharmacogenomics, we hope that in the future, within-patient benefit-risk evaluation can be performed. In the present days (perhaps the foreseeable future), the benefit-risk evaluation (or efficacy-safety evaluation) will still be primarily based on the population level to assess the average group effect.
- Average effect (Using Patients to Analyze Outcomes)
- Subgroup analyses to identify the prognostic factors (phenotypes) to help identify the patients who will more likely to respond to the therapy with fewer side effects
- Targeted therapies, Precision Medicine to identify the genetic biomarkers (genes) to help identify the subgroup of patients who will more likely to respond to the therapy with few side effects
- Individual effect - within patient benefit-risk evaluation
Even with targeted therapy, it is still not possible to be certain if a therapy will be good (the benefit outweighs the risks) for a specific patient.
For the J&J Covid-19 vaccine issue, it seems to be clear that the vaccine does appear to increase the risk of the rare blood clot - CVST. Since the CVST is so rare, the benefit of receiving the Covid-19 vaccine outweighs the risk of the rare blood clot - this assessment is on the population as a whole. When it comes to the individual person, it will be his/her own choice - the risk is small, but maybe there.
