Seminar notes: Adjustments for multiple comparisons (July 17, 2006). Category: Analysis of variance

One of the talks at the 18th Annual Applied Statistics in Agriculture Conference, sponsored by Kansas State University was "A Comparison of Multiple Tests Procedures: Spinosad as a Treatment for Lice on Cattle" by Zhanglin Cui, Eli Lilly and Company. Daniel H. Mowrey, Alan G. Zimmermann, and Douglas E. Hutchens, also of Eli Lilly and Company were co-authors.

Adjustments for multiple comparisons should be considered whenever you have a family of tests. Some examples of families of tests include dose response contrasts, pairwise comparisons of treatment groups, pairwise comparisons with the control, and pairwise comparisons with the "best" treatment.

Multiplicity can inflate the overall Type I error. For efficacy evaluations, this might mean declaring effectiveness when it is not true. For safety evaluations, this might mean identifying a safety issue when none existed.

When do you need adjustments for multiple testing? This is an area of controversy in the Statistics community, but adjustments are called for, according to Dr. Cui when it is plausible that many of the effects studied might truly be null and when you are prepared to perform much data manipulation in order to achieve a significant result.

He reviewed five p-value adjustment methods available in SAS: Bonferroni, Sidak, Holm step down adjustment, Hochberg step up adjustment, and Benjamini and Hochberg's False Discovery Rate and compared those methods in a study of a drug, spinosad, that is used to treat lice on cattle. I have a web page that discusses many of these same issues:

• Stats: Step-down procedures for multiple comparisons (June 16, 2005)

This webpage was written by Steve Simon and was last modified on 07/08/2008.