In a May article, I discussed several practical procedures for multiple testing issue. One of the procedures is Hockberg's procedure. The original paper is pretty short and published in Biometrika.
Hochberg (1988) A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75(4):800-802
Hochberg's procedure is a step-up procedure and its comparison with other procedures are discussed in a paper by Huang & Hsu.
To help the non-statisticians to understand the application of Hochberg's procedure, we can use the hypothetical examples (three situations with three pairs of p-values).
Suppose we have k=2 t-tests
Assume target alpha(T)=0.05
Unadjusted p-values are ordered from the largest to the smallest
Situation #1:
P1=0.074
P2=0.013
For the jth test, calculate alpha(j) = alpha(T)/(k – j +1)
For test j = 2,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(2 – 2 + 1)
= 0.05
P1=0.074 is greater than 0.05, we can not reject the null hypothesis. Proceed to the next test
For test j = 1,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(2 – 1 + 1)
= 0.025
P2=0.013 is less than 0.025, reject the null hypothesis.
Situation #2:
P1=0.074
P2=0.030
For the jth test, calculate alpha(j) = alpha(T)/(k – j +1)
For test j = 2,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(2 – 2 + 1)
= 0.05
P1=0.074 is greater than 0.05, we can not reject the null hypothesis. Proceed to the next test
For test j = 1,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(2 – 1 + 1)
= 0.025
P2=0.030 is greater than 0.025, we can not reject the null hypothesis.
Situation #3:
P1=0.013
P2=0.001
For the jth test, calculate alpha(j) = alpha(T)/(k – j +1)
For test j = 2,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(2 – 2 + 1)
= 0.05
P1=0.013 is less than 0.05, we reject the null hypothesis.
Since the all p-values are less than 0.05, we reject all null hypothesis at 0,05.
More than two comparisons
If we have more than two comparisons, we can still use the same logic
For the jth test, calculate alpha(j) = alpha(T)/(k – j +1)
For example, if there are three comparisons with p-values as:
p1=0.074
p2=0.013
p3=0.010
For test j = 3,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(3 – 3 + 1)
= 0.05
For test j=3, the observed p1 = 0.074 is less than alpha(j) = 0.05, so we can not reject the null hypothesis. We proceed to the next test.
For test j = 2,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(3 – 2 + 1)
= 0.05 / 2
= 0.025
For test j=2, the observed p2 = 0.013 is less than alpha(j) = 0.025, so we reject all remaining null hypothesis.
For example, if there are three comparisons with p-values as:
p1=0.074
p2=0.030
p3=0.010
For test j = 3,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(3 – 3 + 1)
= 0.05
For test j=3, the observed p1 = 0.074 is less than alpha(j) = 0.05, so we can not reject the null hypothesis. We proceed to the next test.
For test j = 2,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(3 – 2 + 1)
= 0.05 / 2
= 0.025
For test j=2, the observed p2 = 0.030 is greater than alpha(j) = 0.025, so we can not reject the null hypothesis. We proceed to the next test.
For test j = 1,
alpha(j) = alpha(T)/(k – j +1)
= 0.05/(3 – 1 + 1)
= 0.05 / 3
= 0.017
For test j=2, the observed p2 = 0.010 is less than alpha(j) = 0.017, so we can reject the null hypothesis.
.
5 comments:
For #1, the test should be stopped at first step because insiginicance. The 2nd test should be insignicant regardless the p value.
"For #1, the test should be stopped at first step because insiginicance. The 2nd test should be insignicant regardless the p value"
^^ Not so. That would be true if a step down procedure was being used (i.e., Holm), but Hochberg is a step up procedure. In Hochberg, you start with the largest p value. If it is retained, you move to the next.
If you use a step down (like Holm) then you would start with the smallest p-value. If you retain the null for that smallest p-value, then you know the null for all other pairwise comparisons will also be retained, so you stop at the first step.
If I have two test for two doses comparing with Placebo and I want to calculate sample size for these tests -can anybody point to me how I can calculate sample size for this kind of test?
I would calculate the sample size using alpha level = 0.025 to be conservative to account for two doses versus placebo.
Post a Comment