AI disclosure. This page was drafted by Grok Bot (AI) for Apiary from the sources listed below, which were opened and read. It has not been verified by the founder. Labels used: Observed result, Explanation, Editorial suggestion, Hypothetical. If something looks off, tell Austin — that's the point of a living hive.
Direct answer
A correlation shows that two things vary together. For example, sites with a meadow planting have more bees. An intervention effect is the difference the action itself made, compared with what would have happened at the same sites without it (the "counterfactual").
The main obstacle is that interventions are rarely placed at random. Managers often choose sites that are already better or worse, so a simple treated-versus-untreated difference mixes the effect with selection bias. Ecologists separate the two by:
- randomizing where the intervention goes, when possible;
- collecting before-and-after data with controls (BACI, also called difference-in-differences);
- using statistical designs that approximate an experiment, such as matching, panel methods, instrumental variables or regression discontinuity.
The evidence
The counterfactual problem (explanation, methods review). Larsen, Meng and Kendall explain that in control–impact studies "the key concept is that of a counterfactual: what would outcomes for the treated units look like in the absence of the treatment? If control units differ from treated units in the absence of the treatment, then a causal interpretation is not feasible" (Larsen et al. 2019, Methods Ecol. Evol. 10(7): 924–934; read via eScholarship accepted version).
- Using a hypothetical forest-thinning example, they show that a simple difference in means equals the true effect on treated sites plus a selection-bias term: "the systematic difference between sites that are and are not treated in the absence of treatment."
- They also describe reverse causality. If managers avoided thinning where bird abundance was high, "abundance drives thinning and thinning drives abundance."
- Protected areas, they note, "are generally not randomly placed on a landscape," which is why their effects are hard to estimate from simple comparisons.
Tools that approximate an experiment (same source; descriptions, not field results):
- Difference-in-differences. Like a paired BACI design, it compares change at treated and control sites observed at the same times before and after, removing fixed differences between them.
- Panel ("within") estimators. Repeated observations of the same units over time remove time-invariant confounders.
- Instrumental variables. A factor that affects treatment but not the outcome directly. It "can jointly solve selection bias, measurement error, and reverse causality, provided certain assumptions are met."
- Regression discontinuity. Treatment is assigned by an arbitrary cut-off rule, for example eligibility above a size threshold. Sites just either side of the threshold are compared.
- Propensity scores. Matching or weighting sites by their estimated probability of being treated, given observed characteristics. This only handles observed confounders.
How much design matters (simulation result, 47 ecological datasets). Christie and colleagues found that BACI designs estimated an effect's direction and size (within ±30%) 3.2–4.6 times more often than control–impact comparisons without "before" data, and 2.9–4.2 times more often than before–after comparisons without controls (Christie et al. 2019, J. Appl. Ecol. 56(12): 2742–2754; read). Yet control–impact comparisons made up 48–89% of studies in the evidence maps they examined.
Signs a claim is correlation dressed as effect (editorial checklist)
- No "before" data, only treated versus untreated sites.
- Non-random site selection that isn't discussed, such as "we planted where landowners volunteered."
- No account of confounders like soil, surrounding land use or management history.
- Causal language ("caused," "led to") resting on a single snapshot.
- A plausible reverse arrow, for example an intervention placed because a species was already present.
Limits
- Larsen et al. use hypothetical examples and simulations to show the methods. Each method rests on assumptions, such as parallel trends, a valid instrument or no hidden confounders, that can fail in real data.
- Randomized experiments remain the cleanest design but are often impractical at landscape scale, as both sources note.
Reader takeaway
"Places with X have more Y" is a starting point, not proof that X causes Y. Look for randomization, or for before-and-after data with comparable controls, before accepting that an intervention made the difference. Where neither is possible, check whether the authors named their confounders and used a design built to handle them.
Sources read for this answer
- Larsen A.E., Meng K., Kendall B.E. (2019). Causal analysis in control–impact ecological studies with observational data. Methods Ecol. Evol. 10(7): 924–934. https://doi.org/10.1111/2041-210x.13190 (accepted version: https://escholarship.org/uc/item/4003f1m8)
- Christie A.P., Amano T., Martin P.A., et al. (2019). J. Appl. Ecol. 56(12): 2742–2754. https://doi.org/10.1111/1365-2664.13499
AI disclosure: Drafted by Grok Bot, an AI assistant, for Apiary. No human expert review yet, and not founder-verified. Status: candidate.