It is axiomatic to argue that sometimes scientific researchers get carried away when identifying random variables where more than two groups especially in health outcomes are involved. They tend to misuse various statistical tools which might compromise the study when dealing with multiple comparisons. This has caused a great dilemma for many observers and the world at large. Public health for instance requires a lot of accuracy and any level of data fishing should be discouraged. (Sullivan, 2012)
The study of ‘low back pain in the Ullensaker’ is one of the public health examples. The purpose of this study was to show whether musculoskeletal pain sites correlate with low back pain. The study identified the respondents with low back pain from adults who were enrolled in an epidemiological surgery for musculoskeletal pain within Ullensaker municipality in Norway in 1990. 4050 participants were sent questionnaires to participate in the study and of these 67% responded.
Individuals who registered low back pain were excluded leaving a total sample population of 1283 in 1990. A similar study was carried out 14 years later (2004) and 763 which was 59% of the population responded and formed part of the sample. After a 14 year follows up the results demonstrated that musculoskeletal pain sites significantly predict a low back pain. (Dahl, Grotle, Benth, and Natvig, 2008).
There are a number of problems that are involved in multiple comparisons; the main problem is that, as one adds each additional test, the probability that the researcher will conclude that there is a statistically significant effect across tests even when there are no such effect increases.
In Sullivan, L. M. (2012) the use of multiple comparisons brings a statistical problem, because there is the likelihood of an uncontrolled rate of rejecting null hypothesis or failing to reject a false null hypothesis; even in scenarios when it should not or should be rejected if the subsequent hypothesis is performed on the outcome of the same data. This raises the biggest dilemma of reducing the risk or rejecting or accepting the null hypothesis and maintaining the likelihood that there is, or there is, no correlation. ANOVA which is one of the Multiple Comparison Procedures can be used to correct this.
Finally, researchers using multiple comparisons may erroneously identify additional statistically significant effects in scenarios where there is the existence of nonzero true effects. When there is complex data, there is a high likelihood of, overinterpretation of patterns.
Rouse, (2010), data fishing which is also referred to as data dredging is a practice whereby large volumes of data are analyzed at once to get any correlation between the data. Data fishing can also be described as seeking to get more information than the data can really provide. Data fishing on many occasions’ results in relationships between variables which might be deduced as significant when in real situations it is not because many variables may be related by chance while others may be related by some unknown factors.
In the study ‘low back pain in the Ullensaker’ the conclusion that musculoskeletal pain sites significantly predicts a low back pain could be misleading and is insignificant when factors like age, types of meals and gene inheritance could be the immediate direct cause of back pain. Although there has been constant misuse of data fishing, it can be useful when finding unexpected relationships that might have not been discovered. This will require further analysis because the concurrence of data might be coincidental.
References
Dahl, F.A., Grotle, M., Benth, J. S & Natvig,B. (2008). Data splitting as a counter measure against hypothesis fishing: with a case study of predictors for low back pain. European Journal of Epidemiology, 23(4): 237–242. Web.
Rouse M. (2010). Data dredging (data fishing) Data Management. Web.
Sullivan, L. M. (2012). Essentials of biostatistics in public health (2nd Ed.)Sudbury, MA: Jones & Bartlett Learning.