“Doing a meta-analysis is easy. Doing one well is hard.” –Ingram Olkin
Biology and ecology (like most sciences) are comparative; we document and investigate patterns, and draw conclusions about the underlying processes that produced those patterns. To do this, we conduct observational studies or manipulative experiments to test specific hypotheses that could explain the patterns we find. Other researchers conduct similar studies or experiments to test similar hypotheses, often in different systems or under slightly different conditions, to determine if the results obtained previously hold under new contexts. In many cases, they do not, at least not to the same extent as before. Faced with a collection of similar studies that may yield different results, we often seek to synthesize those results in a single comprehensive review.
Synthesis allows us to summarize what is known about a topic, but it also identifies what we don’t know; in other words, synthesis reveals the redundancies and holes in our knowledge. For example, did all the studies on a topic yield similar results? If not, was the variation in results due to sampling error, or was it due to fundamental differences in the dynamics of the study systems? What attributes of these systems most likely explain the disparity in results? What additional studies would be most effective at resolving the remaining ambiguities? The syntheses that we conduct can be either qualitative, such as in a narrative review, or they can be quantitative, such as in a meta-analysis. These two synthesis types both have their advantages and disadvantages. For example, qualitative syntheses are highly flexible, however they can incorporate varying degrees of subjectivity, such as decisions that affect which studies to include, how dissimilar studies are evaluated relative to one another, etc. Quantitative syntheses can reduce subjectivity, and ideally provide more transparent way of evaluating a research question, but might restrict the scope of studies that can be directly compared to one another.
What is meta-analysis?
Meta-analysis is a quantitative approach for systematically combining results from previous research to arrive at conclusions about that body of research. It allows us to combine data from multiple studies to draw broader conclusions, and it also allows us to explore drivers of variation across studies. These efforts help us test existing hypotheses, but they also help us formulate new hypotheses. For example, a primary study can measure the strength of a trophic cascade in a single lake, while a meta-analysis can compare the strength of trophic cascades in lakes vs. streams vs. old-fields (e.g., Shurin 2002).
Goals of meta-analysis:
- General goal:
- Guide & inform theory and future empirical studies
- Specific goals:
- Provide more powerful test of null hypothesis
- Estimate the magnitude (i.e. size) of an effect
- Estimate the heterogeneity (i.e. variability) in effects
- Determine the methodological and biological factors that explain the variation in effects
Performing a meta-analysis requires a series of steps:
- Define the problem/question
- Collect and process data
- Evaluate data: calculate effect sizes and determine their weights
- Test overall effect(s)
- Explore heterogeneity in effect sizes
- Test for effects of moderating variables in causing that heterogeneity
- Test for publication bias
- Present results and draw conclusions
Each step requires one or more decisions regarding the execution of that step. Sometimes these decisions must be made explicitly by the meta-analyst, and sometimes they are made implicitly by the default settings in the analysis software used by the researcher (unless the researcher changes those settings). These decisions can have large effects on the results and conclusions that are drawn from the meta-analysis. For example, a series of meta-analyses that came out within a relatively short time frame that examine the magnitude of change in soil carbon to elevated CO2 (Jastrow et al. 2005, van Groenigen et al. 2006, Luo et al. 2006, de Graaff et al. 2006). Although these papers drew from the same literature, they reached different conclusions about the magnitude of soil carbon response to CO2 addition, and effect of soil nitrogen on that response. These differences arose from different decisions made by the authors at various steps within each meta-analysis (Hungate et al. 2009); e.g., the papers differed in the criteria used to exclude studies from the meta-analysis, the choice of an effect size metric, the ways in which individual effects were weighted, and the methods used to deal with non-independence.
It is concerning that these decisions can lead to diverging conclusions in a meta-analysis. Because the results of meta-analyses are often highly influential within their discipline, and sometimes have significant implications for policy and management, it is critical for researchers to make decisions at each step of their analysis that reduce bias, and ensure robust results. As part of this tutorial we will point out areas where these decisions are particularly important. We also will give the best-practice recommendations when conducting a meta-analysis. While we emphasize applications of meta-analysis in the field of ecology, the issues we discuss have relevance to applications in any discipline.
In the following modules, we will cover the theory behind of each of these steps and their practical execution. We also will discuss the decisions that need to be made within each step and the implications of each decision. These modules are designed to parallel the steps in a meta-analysis, although we recommend starting with conceptual topics (modules 1-2, 5-8, and 10; highlighted in bold) and then moving on to cover practical aspects of meta-analysis, such as literature search methods, data extraction, and visualizing results.
This online tutorial contains the following 10 modules, including this introduction:
- Introduction & meta-analysis overview
- Defining a question for meta-analysis
- Formal literature search methods
- Data extraction and processing
- Effect Sizes
- Statistical models
- Weighting schemes
- Exploring heterogeneity & testing moderators
- Visualizing results
- Diagnostics (e.g., publication bias)
Click here to go to the next section on Defining a question for meta-analysis
* Note: This website is meant to facilitate researchers trying to learn meta-analysis. However, it is not meant to be an exhaustive resource. Fortunately, many other useful papers, books, and online resources on meta-analysis are available. We recommend the following texts as good general resources:
Borenstein, Michael, Larry V Hedges, Julian PT Higgins, and Hannah R Rothstein. 2009. Introduction to Meta-Analysis. John Wiley & Sons, Ltd.
Koricheva, Julia, Jessica Gurevitch, and Kerrie Mengersen. 2013. Handbook of Meta-Analysis in Ecology and Evolution. Princeton University Press. https://muse.jhu.edu/book/41629.
de Graaff, M. A., K. J. van Groenigen, J. Six, B. HUNGATE, and C. van Kessel. 2006. Interactions between plant growth and soil nutrient cycling under elevated CO 2: a meta-analysis. Global Change Biology 12:2077–2091.
Hungate, B. A., K.-J. van GROENIGEN, J. Six, J. D. Jastrow, Y. Luo, M.-A. de GRAAFF, C. van KESSEL, and C. W. Osenberg. 2009. Assessing the effect of elevated carbon dioxide on soil carbon: a comparison of four meta-analyses. Global Change Biology 15:2020–2034.
Jastrow, J. D., R. M. Miller, R. Matamala, R. J. Norby, T. W. Boutton, C. W. Rice, and C. E. Owensby. 2005. Elevated atmospheric carbon dioxide increases soil carbon. Global Change Biology 11:2057–2064.
Luo, Y., D. Hui, and D. Zhang. 2006. Elevated CO2 stimulates net accumulations of carbon and nitrogen in land ecosystems: a meta-analysis. Ecology 87:53–63.
Shurin, J. B. 2002. A cross-ecosystem comparison of the strength of trophic cascades. Ecology Letters 5:785–791.
van Groenigen, K. J., M. A. de Graaff, J. Six, D. Harris, P. Kuikman, and C. van Kessel. 2006. The Impact of Elevated Atmospheric [CO2] on Soil C and N Dynamics: A Meta-Analysis. Pages 373–391 in J. Nösberger, S. P. Long, R. J. Norby, M. Stitt, G. R. Hendrey, and H. Blum, editors. Managed Ecosystems and CO2: Case Studies, Processes, and Perspectives. Springer Berlin Heidelberg, Berlin, Heidelberg.