Pros and cons of meta-analyses

I have been considering doing some meta-analysis for a particular field of study in evolution, but before I go any further I would like to know; what are the positives and negatives of the process? For example, no need for a practical experiment is an advantage (time & money) but there will be a publication bias (more exciting results get published) which would be a disadvantage.

What papers in statistics journals discuss the pros and cons of meta-analysis?


Introduction to Meta-Analysis by Borenstein, Hedges, Higgins and Rothstein provides a detailed discussion of the pros and cons of meta-analysis. See for example the chapter “Criticism of meta-analysis” where the authors respond to various criticisms of meta-analysis. I note the section headings for that chapter and then make a few observations from memory that relate to that point:

  • “one number cannot summarise a research field”: A good meta analysis will model variability in true effect sizes and model the uncertainty of estimates.
  • “the file drawer problem invalidates meta-analysis”: Funnel plots and related tools allows you to see whether sample size is related to effect size in order to check for publication bias. Good meta-analyses endeavour to obtain unpublished studies. This issue is shared with narrative studies.
  • “Mixing apples and oranges”: Good meta-analyses provide a rigorous coding system for categorising included studies and justifying the inclusion and exclusion of studies in the meta-analysis. After studies have been classified, moderator analysis can be performed to see whether effect sizes vary across study type.
  • “Important studies are ignored”: You can code for the evaluated quality of the studies. Large samples can be given greater weighting.
  • “meta analysis can disagree with randomised trials”:
  • “meta-analyses are performed poorly”: This is merely an argument for improving the standards of meta-analytic methods.
  • “Is a narrative review better?”: Many of the critiques of meta-analysis (e.g., publication bias) are shared by narrative reviews. It is just that the methods of inference are less explicit and less rigorous in narrative reviews.

Source : Link , Question Author : rg255 , Answer Author : Harvey Motulsky

Leave a Comment