Simpson's paradox is a statistical paradox wherein the successes of groups seem reversed when the groups are combined. Simpson's paradox occurs when we combine data or when we perform the sub-group analyses.
According to Wikipedia, the Simpson's paradox is defined as the following:
Simpson's paradox is a phenomenon in probability and statistics in which a trend appears in several groups of data but disappears or reverses when the groups are combined. This result is often encountered in social-science and medical-science statistics, and is particularly problematic when frequency data are unduly given causal interpretations. The paradox can be resolved when confounding variables and causal relations are appropriately addressed in the statistical modeling. Simpson's paradox has been used to illustrate the kind of misleading results that the misuse of statistics can generate
It gave several examples of the Simpson's paradox including the kidney stone treatment example:
Kidney stone treatment
Another example comes from a real-life medical study comparing the success rates of two treatments for kidney stones. The table below shows the success rates (the term success rate here actually means the success proportion) and numbers of treatments for treatments involving both small and large kidney stones, where Treatment A includes open surgical procedures and Treatment B includes closed surgical procedures. The numbers in parentheses indicate the number of success cases over the total size of the group.
The paradoxical conclusion is that treatment A is more effective when used on small stones, and also when used on large stones, yet treatment B appears to be more effective when considering both sizes at the same time. In this example, the "lurking" variable (or confounding variable) causing the paradox is the size of the stones, which was not previously known to researchers to be important until its effects were included.
Here are some additional links to explain the Simpson's paradox:
- Bonovas & Piovani (2023) Simpson’s Paradox in Clinical Research: A Cautionary Tale
- 李永乐老师 (辛普森悖论)
- Stanford Encyclopedia of Philosophy (2021) Simpson’s Paradox
- Wagner (1982) Simpson's paradox in real life
- Fenton et al (2019) Simpson's paradox and the implication for medical trials
- Kievit et al (2013) Simpson's paradox in psychological science: a practical guide
- Shi and Chan (2020) Simpson's paradox in meta-analysis - choice of studies and summary statistics
- Simpson's Paradox (and How to Avoid Its Effects)
For example, consider a clinical trial evaluating the effectiveness of two treatments (Drug A and Drug B) for a certain medical condition. The overall analysis of the trial data might suggest that Drug A is more effective than Drug B in improving patient outcomes. However, when the data is stratified by important demographic or clinical variables, such as age, gender, or disease severity, a different picture may emerge.
In this scenario, Simpson's paradox occurs because the distribution of confounding variables (in this case, age) differs between the treatment groups, leading to a reversal of the observed treatment effects when the data is aggregated. To avoid drawing misleading conclusions from clinical trial data affected by Simpson's paradox, it's crucial to conduct subgroup analyses and consider potential confounding variables that may influence treatment outcomes. Additionally, techniques such as propensity score matching or regression adjustment can help mitigate the impact of confounding variables and provide more accurate estimates of treatment effects.
Simpson's paradox can occur in pooling the data from multiple clinical trials, in meta analysis, in integrated summary of effectiveness (ISE), and in integrated summary of safety (ISS).
FDA summary of review REPATHA for the treatment of Primary Hyperlipidemia and Mixed Dyslipidemia discussed the safety review:
... crude pooling of trials with varying allocation ratios can also lead to confounding by trial (i.e., Simpson’s paradox).
This same issue was mentioned in a paper by SUH (2009) The use of atypical antipsychotics in dementia: rethinking Simpson’s paradox"
for safety analysis, pooling the data from multiple studies may obscure the finding of the safety signals.
In 2009, for example, an FDA panel weighed whether the agency should approve AstraZeneca's widely prescribed antipsychotic drug quetiapine (Seroquel) for a wider range of conditions. The panel heard from health policy expert Wayne Ray of Vanderbilt University in Nashville, who described his research linking the drug to sudden cardiac death when used with certain other medications. Ray recalls "an FDA staff member who gave a very negative presentation on our paper." And according to the meeting transcript, the agency's then-Director of Psychiatric Products Thomas Laughren, who was instrumental in shepherding Seroquel and similar drugs through the review process and personally signed their FDA approvals, also challenged Ray's results and defended AstraZeneca's clinical trial findings in the discussion that followed. The company's "analysis should have been able to pick up a difference in sudden cardiac death, and they didn't find any difference between drug and placebo," he said.I consulted ChatGPT about strategies for addressing potential Simpson's paradox, and I wholeheartedly endorse the following responses.
Ray told Laughren and the panel that AstraZeneca had pooled data from all its trials as though the data were one data set, causing a well-known statistical error called Simpson's paradox. To take the company's conclusion "as definitive" would be "very dangerous," Ray said, according to the transcript. Laughren responded by calling sudden death "a pretty definitive event."
Ultimately, the committee voted overwhelmingly to advise approval of the drug for new indications and made no recommendation on labeling it to warn about sudden cardiac death. Later evidence showed that the cardiac problems Ray described are real, and in 2011, FDA required adding a warning on Seroquel's label.
To minimize the potential biases caused by Simpson's paradox, you can employ several strategies during data analysis:
Awareness: Be aware of the possibility of Simpson's paradox and actively look for it, especially when dealing with aggregated data or subgroup analyses.
Visualization: Visualize the data in various ways to explore relationships between variables. Use scatter plots, histograms, box plots, and other visualization techniques to identify patterns and potential confounding variables.
Stratification: Stratify the data by relevant variables to examine relationships within homogeneous subgroups. This allows you to identify any instances where the direction of association changes across subgroups.
Control Variables: Include potential confounding variables as control variables in your analysis. By controlling for these variables, you can assess the relationship between the variables of interest while holding other factors constant.
Causal Inference Techniques: Use causal inference techniques, such as causal mediation analysis or propensity score matching, to account for potential confounding variables and identify true causal relationships.
Sensitivity Analysis: Conduct sensitivity analyses to assess the robustness of your findings to different assumptions and model specifications. This helps you understand the potential impact of confounding variables on your results.
Expert Consultation: Consult with domain experts to ensure that you're considering all relevant variables and potential sources of bias in your analysis.
By incorporating these strategies into your data analysis, you can minimize the potential biases caused by Simpson's paradox and obtain more accurate and reliable results.