In Reply:

Judging a Plethora of p-Values: How to Contend With the Problem of Multiple Testing – Part 10 of a Series on Evaluation of Scientific Publications by Dr. rer. physiol. Anja Victor, Dipl.-Stat. Amelie Elsäßer, Prof. Dr. rer. nat. Gerhard Hommel, Prof. Dr. rer. nat. Maria Blettner in volume 4/2010

We thank the commentators for their supporting letters and for indicating other important points related to multiple testing.

Unfortunately, it is not always possible to recognize selective reporting of secondary outcomes with “good” p-values. We can only appeal to the authors to be honest for the sake of research and encourage reviewers, editors and readers to ask critically whether hypotheses

other than the original ones were reported and to inquire about the number of tests performed.

Quite aside from multiple testing, the problem with p-values for correlation coefficients is that the test is for deviation from a correlation coefficient of zero (null hypothesis: r = 0). As one commentator remarked, even a very weak correlation can give a small p-value (if the sample size is adequate). This is often ignored when interpreting the p-values of correlation coefficients.

The results of statistical tests must always be evaluated with respect to their clinical relevance. With respect to clinical relevance a p-value is not a measure, but only serves to estimate whether the probability of error lies under a prescribed limit. The effect must accordingly be assessed as a value – such as a median, mean, relative frequency or similar. It is unfortunately the case that multiple testing is accompanied by an overestimate of effects. The overestimate is not eliminated by correcting the level or the p-values for multiple testing.

DOI: 10.3238/arztebl.2010.0417c

Dr. rer. physiol. Anja Victor

Institut für Medizinische Biometrie, Epidemiologie und Informatik

Universitätsmedizin der Johannes-Gutenberg-Universität Mainz

Obere Zahlbacher Str. 69

55131 Mainz, Germany

Conflict of interest statement
The authors of all contributions declare that no conflict of interest exists according to the guidelines of the International Committee of Medical Journal Editors.

References

1. Victor A, Elsäßer A, Hommel G, Blettner M: Judging a plethora of p-values: How to contend with the problem of multiple testing — Part 10 of a series on evaluation of scientific publications [Wie bewertet man die p-Wert-Flut? Hinweise zum Umgang mit dem multiplen Testen – Teil 10 der Serie zur Bewertung wissenschaftlicher Publikationen]. Dtsch Arztebl Int 2010; 107(4): 50–6.