Previously, we gave an overview of meta-analysis. Now we are going to go more in depth into the individual steps comprising one, starting with the first step, defining the research question. Although this seems like a straightforward step, it can be difficult to define a specific question that is clearly quantifiable. However, putting thought into this question at the beginning of a meta-analysis is critical, because it will influence the keyword search terms, criteria for including studies and/or data in your analysis, which effect size metrics are appropriate, etc. Additionally, if the research question is vague at the beginning of your meta-analysis and has to be re-defined later on, it can add a lot of time and extra work to the data extraction and processing steps.
Types of questions used in meta-analysis
At the broadest level, there are three main types of meta-analysis questions:
- Does it have an effect?
- How big is that effect?
- What influences how big the effect is?
Each of these categories have different goals, and exist along a gradient of increasing complexity of the question itself, as well as the amount of information gained by answering it.
The first type of question is the simplest, essentially just a test of the null hypothesis. This type of question is most commonly used when the meta-analysist is only interested in a narrow set of studies conducted under similar conditions or experimental designs. This type of question is common in medical studies. For example, does Drug X reduce Symptom Y of Disease Z in randomized clinical trials on men? However, although the knowing the answer to this question might be useful, it would be more informative to ask, “How big is the effect of Drug X on reducing Symptom Y?” In other words, what is the effect size of X? Knowing this additional information about the magnitude of the effect of the drug, rather than simply whether it has an effect at all, allows clinicians and researchers to determine whether it is worth the cost, effort, side effects, etc. as a treatment. This type of question is also commonly used in meta-analyses that have the goal of estimating the value of parameters that can be incorporated into dynamical models, such as a population growth rates (lambda) or competition coefficients (alpha).
The third type of question adds additional nuance to the second, seeking to determine whether certain observed patterns in the size of an effect are robust to background heterogeneity in studies, systems, etc. It also attempts to determine what factors explain any variation in how large observed effects are. This last category of meta-analysis question is common within the field of ecology (as well as in many other areas of biology), where we are interested in making comparisons across systems. For example, a researcher could ask, “does patch size affect the population density of a species living in that patch, and is this result consistent across different species and across different ecosystem types?” (sensu Bender et al. 1998). In this case, researchers could include studies done in Western Hemisphere forest habitats, temperate lakes, islets in tropical Pacific atolls, etc., with various species as the focal organism of those studies. After collecting data from all these studies and systems types, the meta-analyst could then explore whether predators show different responses than herbivores to patch size, or if patterns differ among geographic regions.
It is important to note that these types of questions don’t have to exist independently; they can often be complementary parts a single meta-analysis, starting with the first question regarding the significance of an effect of interest, and proceeding to the last question about heterogeneity in effects. However, sometimes the available data, the effect size metric, or the statistical test might not be appropriate for answering all of them within the same meta-analysis. We will discuss this more in later sections on effect sizes and heterogeneity.
How to define your research question
So, now that you know about the general types of questions that can be addressed with meta-analysis, how do you first tackle trying to define a good research question for meta-analysis?
Step 1: Set your objectives
- What is your general question?
- What knowledge gap does it fill?
- Why is it important?
- What level of generality are you interested in?
The first three points apply to most typical research questions and are fairly straightforward. The last point however, is critical in meta-analysis, and will help refine the research topic into a more specific question, which can affect the type of studies, effect size metrics, and statistical models you will use later on in your meta-analysis.
Step 2: Review the field
- About how many studies are on your topic?
- Ideally 10 +
- What types of studies/data are there?
- What models (conceptual or analytical) exist for the process(es) you are interested in?
After setting the objectives of your meta-analysis, you can begin reviewing the published literature, in order to determine 1) whether you can reasonably answer your research question with the available science, and 2) what types of effect sizes can you calculate from the available data. Ideally, there should be about ten or more studies with the appropriate type of data, that way the mean effect size and its variance can be reasonably estimated. Additionally, by having an idea of the type of data and models that are used to describe your general process or relationship of interest, you can further narrow the precise research question, and begin thinking about how you will extract and process data, and which effect size metric(s) you will use. More details on this in the Effect sizes module.
Other important considerations:
Along the same lines as considering what model describes the process that you are interested in, it is important to consider what time scale the process of interest operates on, and over what time frame the effect sizes should be measured. (The effects of spatial scale should be considered similarly to time.) For example, are you interested in equilibrial or transient dynamics?
Be specific, because different meta-analyses can have the same general research question, but vary in the specifics, which influence the data they include, their analyses, and eventually can influence the results and the conclusions that they draw.
We’ll address how to consider these issues in greater detail in the Effect sizes module, so we recommend reading that section next, before the Formal literature search, and Data extraction sections.
Click here to proceed to the Effect sizes.
References
Bender, D. J., T. A. Contreras, and L. Fahrig. 1998. Habitat loss and population decline: a meta-analysis of the patch size effect. Ecology 79:517–533.