Multiple Comparisons

In statistics or biostatistics, the multiple comparisons problem occurs when one considers a set, or family, of statistical inferences simultaneously. Errors in inference, including confidence intervals that fail to include their corresponding population parameters, or hypothesis tests that incorrectly reject the null hypothesis, are more likely when one considers the family as a whole.

Multiple comparison issues were nicely summarized in EMEA's guidance titled "Points to consider on multiplicity issues in clinical trials". This guidance also discussed the situations where the adjustment for multiplicity is not needed.

Adjustment for multiplicity is also mentioned in many regulatory guidance, for example, FDA guidance on ISE and its importance has been recognized in may medical journal review process.

SAMSI held a workshop in 2005 to discuss teh multiplicity issues which included the issue in Multiple Testing, Reproducibility, and Subgroup analysis.

For an introduction about multiple comparisons, refer to Wikipedia "http://en.wikipedia.org/wiki/Multiple_comparisons"

SAS Proc Multitest can be an easy tool to compute the adjusted p-values (with different methods) if the raw p-values from multiple tests are provided. For example, with the following program, we would be able to obtain a set of adjusted p-values.

data integrated;
input Method$ Raw_P;
datalines;
method1 .331
method2 .090
method3 .105
method4 .xxx
;
proc multtest pdata=integrated holm hoc fdr bon;
run;