Simpson's Paradox - in integrated analyses or in sub-group analyses

 Simpson's paradox is a statistical paradox wherein the successes of groups seem reversed when the groups are combined. Simpson's paradox occurs when we combine data or when we perform the sub-group analyses.

According to Wikipedia, the Simpson's paradox is defined as the following:

Simpson's paradox is a phenomenon in probability and statistics in which a trend appears in several groups of data but disappears or reverses when the groups are combined. This result is often encountered in social-science and medical-science statistics, and is particularly problematic when frequency data are unduly given causal interpretations. The paradox can be resolved when confounding variables and causal relations are appropriately addressed in the statistical modeling. Simpson's paradox has been used to illustrate the kind of misleading results that the misuse of statistics can generate

It gave several examples of the Simpson's paradox including the kidney stone treatment example:

Kidney stone treatment

Another example comes from a real-life medical study comparing the success rates of two treatments for kidney stones. The table below shows the success rates (the term success rate here actually means the success proportion) and numbers of treatments for treatments involving both small and large kidney stones, where Treatment A includes open surgical procedures and Treatment B includes closed surgical procedures. The numbers in parentheses indicate the number of success cases over the total size of the group.

The paradoxical conclusion is that treatment A is more effective when used on small stones, and also when used on large stones, yet treatment B appears to be more effective when considering both sizes at the same time. In this example, the "lurking" variable (or confounding variable) causing the paradox is the size of the stones, which was not previously known to researchers to be important until its effects were included.

Here are some additional links to explain the Simpson's paradox:

• Bonovas & Piovani (2023) Simpson’s Paradox in Clinical Research: A Cautionary Tale
• 李永乐老师 (辛普森悖论)
• Stanford Encyclopedia of Philosophy (2021) Simpson’s Paradox
• Wagner (1982) Simpson's paradox in real life
• Fenton et al (2019) Simpson's paradox and the implication for medical trials
• Kievit et al (2013) Simpson's paradox in psychological science: a practical guide
• Shi and Chan (2020) Simpson's paradox in meta-analysis - choice of studies and summary statistics
• Simpson's Paradox (and How to Avoid Its Effects)

In clinical trials, Simpson's paradox can occur when the overall treatment effect observed in the entire study population is different from the treatment effect observed within subgroups of the population. This phenomenon can lead to misleading conclusions about the effectiveness of a treatment if not properly addressed.

For example, consider a clinical trial evaluating the effectiveness of two treatments (Drug A and Drug B) for a certain medical condition. The overall analysis of the trial data might suggest that Drug A is more effective than Drug B in improving patient outcomes. However, when the data is stratified by important demographic or clinical variables, such as age, gender, or disease severity, a different picture may emerge.

Let's say that within each age group, Drug B appears to be more effective than Drug A. However, the older age group constitutes a larger proportion of the study population, and within this group, patients tend to have poorer outcomes regardless of the treatment they receive. As a result, the overall analysis might erroneously indicate that Drug A is more effective, when in fact Drug B is more effective within each age group.

In this scenario, Simpson's paradox occurs because the distribution of confounding variables (in this case, age) differs between the treatment groups, leading to a reversal of the observed treatment effects when the data is aggregated. To avoid drawing misleading conclusions from clinical trial data affected by Simpson's paradox, it's crucial to conduct subgroup analyses and consider potential confounding variables that may influence treatment outcomes. Additionally, techniques such as propensity score matching or regression adjustment can help mitigate the impact of confounding variables and provide more accurate estimates of treatment effects.

Simpson's paradox can occur in pooling the data from multiple clinical trials, in meta analysis, in integrated summary of effectiveness (ISE), and in integrated summary of safety (ISS). 

FDA guidance for industry "Integrated Summary of Effectiveness" discussed 'pooled analyses of data from more than one study' and specifically mentioned  

Simpson's paradox can be a subject of the regulatory review comments when pooling the data or combining the data from multiple studies. 

Statistical review on BI's NDA of Empagliflozin for the treatment of type 2 diabetes:

FDA Briefing Document Cardiovascular and Renal Drugs Advisory Committee Meeting July 15, 2021 Roxadustat for the treatment of anemia due to chronic kidney disease (CKD)

FDA summary of review REPATHA for the treatment of Primary Hyperlipidemia and Mixed Dyslipidemia discussed the safety review:

... crude pooling of trials with varying allocation ratios can also lead to confounding by trial (i.e., Simpson’s paradox). 

In a DIA webinar presentation (2024) Integrated Safety Analyses in Drug Marketing Applications: Avoiding Common Mistakes, Mary Nilsson provided an example of Simpson's paradox when pooling the safety data from three randomized controlled clinical trials - Simpson's paradox can obscure the safety signals in the pooled analyses. She suggested that the stratification as an approach to address the Simpson's paradox issue. For the integrated safety analyses, the analytical methods should always be stratified by study - take differences within studies first, then takes the average of those differences.

This same issue was mentioned in a paper by SUH (2009) The use of atypical antipsychotics in dementia: rethinking Simpson’s paradox"

for safety analysis, pooling the data from multiple studies may obscure the finding of the safety signals. 

Science.org had an article in 2018 FDA's revolving door: Companies often hire agency staffers who managed their successful drug reviews. It described a case of pooled data analyses obscuring the signal of sudden cardiac death due to Simpson's paradox.  raise conflict of interest questions

In 2009, for example, an FDA panel weighed whether the agency should approve AstraZeneca's widely prescribed antipsychotic drug quetiapine (Seroquel) for a wider range of conditions. The panel heard from health policy expert Wayne Ray of Vanderbilt University in Nashville, who described his research linking the drug to sudden cardiac death when used with certain other medications. Ray recalls "an FDA staff member who gave a very negative presentation on our paper." And according to the meeting transcript, the agency's then-Director of Psychiatric Products Thomas Laughren, who was instrumental in shepherding Seroquel and similar drugs through the review process and personally signed their FDA approvals, also challenged Ray's results and defended AstraZeneca's clinical trial findings in the discussion that followed. The company's "analysis should have been able to pick up a difference in sudden cardiac death, and they didn't find any difference between drug and placebo," he said.

Ray told Laughren and the panel that AstraZeneca had pooled data from all its trials as though the data were one data set, causing a well-known statistical error called Simpson's paradox. To take the company's conclusion "as definitive" would be "very dangerous," Ray said, according to the transcript. Laughren responded by calling sudden death "a pretty definitive event."

Ultimately, the committee voted overwhelmingly to advise approval of the drug for new indications and made no recommendation on labeling it to warn about sudden cardiac death. Later evidence showed that the cardiac problems Ray described are real, and in 2011, FDA required adding a warning on Seroquel's label.

I consulted ChatGPT about strategies for addressing potential Simpson's paradox, and I wholeheartedly endorse the following responses.

To minimize the potential biases caused by Simpson's paradox, you can employ several strategies during data analysis:

Awareness: Be aware of the possibility of Simpson's paradox and actively look for it, especially when dealing with aggregated data or subgroup analyses.

Visualization: Visualize the data in various ways to explore relationships between variables. Use scatter plots, histograms, box plots, and other visualization techniques to identify patterns and potential confounding variables.

Stratification: Stratify the data by relevant variables to examine relationships within homogeneous subgroups. This allows you to identify any instances where the direction of association changes across subgroups.

Control Variables: Include potential confounding variables as control variables in your analysis. By controlling for these variables, you can assess the relationship between the variables of interest while holding other factors constant.

Causal Inference Techniques: Use causal inference techniques, such as causal mediation analysis or propensity score matching, to account for potential confounding variables and identify true causal relationships.

Sensitivity Analysis: Conduct sensitivity analyses to assess the robustness of your findings to different assumptions and model specifications. This helps you understand the potential impact of confounding variables on your results.

Expert Consultation: Consult with domain experts to ensure that you're considering all relevant variables and potential sources of bias in your analysis.

By incorporating these strategies into your data analysis, you can minimize the potential biases caused by Simpson's paradox and obtain more accurate and reliable results.