What is impact evaluation?
Impact evaluation examines the effects of an intervention, with particular attention to what change can be attributed to it. Terminology varies across fields, so state the causal question and method rather than relying on the label alone.
A rise in an outcome after a program is not enough to show what the program caused. Other events, selection and measurement changes may explain some or all of the difference.
Define the question before choosing a method
Specify the intervention, population, outcome and period. Ask whether the decision requires an average effect, an explanation of how change occurred, variation across groups or a combination.
A broad question such as “Did we create impact?” is difficult to answer. A more focused question identifies the relevant change and comparison. Keep the program’s theory of change visible, including assumptions.
Common quantitative designs
| Design | Basic idea | Important assumption or limitation |
|---|---|---|
| Randomized comparison | Use random assignment to create comparable groups. | Implementation, attrition and spillovers still matter. |
| Difference-in-differences | Compare changes across groups over time. | The comparison requires a credible parallel-trends assumption. |
| Regression discontinuity | Compare units near an assignment threshold. | Requires a suitable rule and careful interpretation near the cutoff. |
| Matching or adjustment | Compare observed units with similar measured characteristics. | Unmeasured differences can still bias the result. |
| Interrupted time series | Examine a change in a repeated outcome series. | Other simultaneous changes can complicate interpretation. |
Work with an evaluator who can assess feasibility, sample requirements and assumptions. A sophisticated method used with unsuitable data does not produce a credible estimate. The World Bank’s Impact Evaluation in Practice explains counterfactuals and common evaluation designs in more detail.
Use qualitative and theory-based evidence
Interviews, observations and documents can help examine mechanisms, implementation and alternative explanations. They can show why an effect differs across contexts or why an expected connection did not occur.
A contribution analysis or other theory-based approach still needs a systematic assessment of evidence and rival explanations. It should not be a positive narrative assembled only from supportive accounts.
Worked example: before and after is not the counterfactual
Suppose an illustrative employment rate rises from 40% to 55% after training. The observed change is 15 percentage points. If local hiring also improved, that change cannot all be assigned to training without further evidence.
A suitable comparison may help estimate what would have happened otherwise, but its credibility depends on the design. Report the estimate, uncertainty and assumptions rather than presenting a single number without context.
Plan for practical and ethical constraints
Consider whether the design is feasible, appropriate for participants and useful for the decision. Data access, implementation timing and follow-up coverage can determine which questions are answerable. Do not delay necessary support merely to create a convenient design.
Predefine key outcomes and analysis choices where appropriate. Retain unexpected and negative findings. Document deviations from the plan so readers can assess them.
Report conclusions at the strength of the evidence
Explain the population, intervention, comparison, outcome, period and method. Distinguish observed change from estimated causal effect. Discuss uncertainty, generalizability and whether the result applies to a different setting.
AI can help organize source material but cannot supply a missing counterfactual or validate an unsupported causal claim. For the wider planning process, see program evaluation.
Frequently asked questions
Is outcome evaluation the same as impact evaluation?
Not necessarily. Outcome evaluation describes or examines outcomes; impact evaluation often asks what effects are attributable to the intervention. State the actual design.
Are randomized trials always possible?
No. Feasibility, ethics and the decision context matter. Other designs may be appropriate, with their assumptions made explicit.
Can qualitative evidence support causal reasoning?
Yes, through a systematic examination of mechanisms and alternative explanations. It should not be reduced to selected testimonials.
Does AI make an evaluation causal?
No. Causal credibility comes from the question, design, evidence and assumptions, not the tool used to summarize data.
