When assessing the safety of chemicals, epidemiological researchers face many challenges associated with finding a sufficiently large, truly random sample of the population to study.
To assess the value of many small studies, sometimes researchers will run new analyses on data pooled from a number of small studies, hoping that this larger data set will provide a stronger analysis and conclusion. These so-called meta-analyses may add value to a field of science, but have their own pitfalls. Selection of the studies for which data are pooled raises a host of issues from researcher bias when selecting studies, varied data quality, difficulty finding truly heterogeneous data worthy of pooling, biases toward using only studies with positive findings, and more. As Mark Battersby notes: “So meta analyses, which tend not to include negative data, may be biased towards finding a correlation or finding a greater degree of correlation than a total aggregation of the research would justify.”(1)
Meta-analyses also provide opportunities for scientific abuse, as some researchers may try to validate weak research by pooling it with a selection of other poorly designed studies. If the data and designs of the studies going into the analysis are not sound, a meta-analysis will not fix those problems.
Researchers face a number of other challenges to establishing the validity of their findings. These include challenges associated with confounding factors, recall bias, and, researcher bias.
Browse the terms on the sidebar of this webpage for more information and/or download a copy of A Consumer’s Guide to Chemical Risk: Deciphering the “Science” Behind Chemical Scares.
(1) Mark Battersby, Is it a Fact?: A Field Guide to Statistical and Scientific Information, revised edition (Ontario, CA: Broadview Press, 2010),164.
