Hochberg procedure for adjustment for multiplicity - an illustration

In a May article, I discussed several practical procedures for multiple testing issue. One of the procedures is Hockberg's procedure. The original paper is pretty short and published in Biometrika.

Hochberg (1988) A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75(4):800-802

Hochberg's procedure is a step-up procedure and its comparison with other procedures are discussed in a paper by Huang & Hsu.

To help the non-statisticians to understand the application of Hochberg's procedure, we can use the hypothetical examples (three situations with three pairs of p-values).

Suppose we have k=2 t-tests
Assume target alpha(T)=0.05

Unadjusted p-values are ordered from the largest to the smallest

Situation #1:
P1=0.074
P2=0.013

For the jth test, calculate alpha(j) = alpha(T)/(k – j +1)

For test j = 2,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(2 – 2 + 1)
= 0.05

P1=0.074 is greater than 0.05, we can not reject the null hypothesis. Proceed to the next test

For test j = 1,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(2 – 1 + 1)
= 0.025

P2=0.013 is less than 0.025, reject the null hypothesis.

Situation #2:
P1=0.074
P2=0.030
For the jth test, calculate alpha(j) = alpha(T)/(k – j +1)

For test j = 2,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(2 – 2 + 1)
= 0.05

P1=0.074 is greater than 0.05, we can not reject the null hypothesis. Proceed to the next test

For test j = 1,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(2 – 1 + 1)
= 0.025

P2=0.030 is greater than 0.025, we can not reject the null hypothesis.

Situation #3:
P1=0.013
P2=0.001
For the jth test, calculate alpha(j) = alpha(T)/(k – j +1)

For test j = 2,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(2 – 2 + 1)
= 0.05

P1=0.013 is less than 0.05, we reject the null hypothesis.
Since the all p-values are less than 0.05, we reject all null hypothesis at 0,05.

More than two comparisons
If we have more than two comparisons, we can still use the same logic

For the jth test, calculate alpha(j) = alpha(T)/(k – j +1)

For example, if there are three comparisons with p-values as:
p1=0.074
p2=0.013
p3=0.010

For test j = 3,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(3 – 3 + 1)
= 0.05

For test j=3, the observed p1 = 0.074 is less than alpha(j) = 0.05, so we can not reject the null hypothesis. We proceed to the next test.


For test j = 2,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(3 – 2 + 1)
= 0.05 / 2
= 0.025

For test j=2, the observed p2 = 0.013 is less than alpha(j) = 0.025, so we reject all remaining null hypothesis.



For example, if there are three comparisons with p-values as:
p1=0.074
p2=0.030
p3=0.010

For test j = 3,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(3 – 3 + 1)
= 0.05

For test j=3, the observed p1 = 0.074 is less than alpha(j) = 0.05, so we can not reject the null hypothesis. We proceed to the next test.


For test j = 2,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(3 – 2 + 1)
= 0.05 / 2
= 0.025

For test j=2, the observed p2 = 0.030 is greater than alpha(j) = 0.025, so we can not reject the null hypothesis. We proceed to the next test.

For test j = 1,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(3 – 1 + 1)
= 0.05 / 3
= 0.017

For test j=2, the observed p2 = 0.010 is less than alpha(j) = 0.017, so we can reject the null hypothesis.
.