Impact evaluation is the process where we study the results of social interventions. We ascertain the value of the overall initiative and use the findings as learnings for future endeavors. This is a critical process for both the investors to understand the rationale behind continuing the program and for impact makers to see if their efforts are bringing positive change. This article underlines the steps you need to take to define an effective impact evaluation plan.
1. Establish Program Theory of the Social Impact Initiative:
Program theory is the step where you document the assumptions and logical arguments that define the rationale behind your program. If you would like to take a more sophisticated approach we recommend to go a level higher and do as below:
Define Theory of Change:
Theory of change is the program theory where you not only describe the assumptions and logical arguments in favor of your initiative but also lay down all the possible scenarios that can result from your impact actions. This gives a clearer picture and helps you understand the evaluation reports from a wider perspective.
Here is all you need to know to define Theory of Change for your program.
Setting up the Counterfactual:
In its essence counterfactual analysis is a “with versus without” analysis. We study the impact of a program by comparing the results from a control group where no artificial factors were stimulated. You need to set up this control group in the early stage so that the evaluation can be done parallelly and results are comparable. In some cases, this can be avoided where there is no other factor that can bring any observed change in outcomes (e.g. reductions in time spent fetching water after the installation of water pumps).
2. Addressing Selection Bias in Impact Evaluation:
The presence of selection bias can skew the results of evaluation hence, it is important that we take whatever steps possible to eliminate all bias. Below are the preconditions to identify the extent of selection bias and the remedial step that can be taken to eradicate it:
- If the evaluation metrics are determined prior to the event then we need to see if randomization is possible. To further explain if the treatment group is chosen randomly then another set of the random counterfactual is a valid test. It is possible to target a subgroup of the random subject and still remain unbiased. For example, if an initiative was designed for below minimum wage workers then the counterfactual group can also be from the same subset to maintain relevance.
- If the above is not the case we see whether the selection determinants are observed. There are a number of regression techniques that can help eliminate bias in this case.
- In the case of unobserved selection determinants, we need them to be time-invariant so that panel data can be used to remove bias. For this case baseline (or some means of substituting baseline) is critical.
- In this case, the panel is not possible. Since the selection determinants are unobserved, we need to identify ways of observing the determinants.
- If that fails we can go for the pipeline approach provided there are some untreated beneficiaries.
We cannot address the problem of selection bias if none of the above are possible. Thus, we are left to rely on program theory and triangulation to build an argument. Thus, setting up a Theory of Change (TOC) Model helps make plausible association easy.
3. Designing the Baseline Survey:
A baseline survey is performed when the project is being initiated i.e. at the beginning of the project but implementation. It helps prioritize between different objectives of an initiative and works as a benchmark to identify the success or failure of it.
Guiding principles for an effective baseline survey in Impact Evaluation:
- It should be in line with the program theory and data must be collected across the results chain, not just on outcomes.
- The counterfactual should be presented in the same questionnaire. Intervention-specific questions should be replaced with similar questions of a more general nature can help test for any influence of initiative.
- Allocate enough time to double check on instruments before initiating the survey. It should be convenient to form a relational database to ease analysis and data entry. This process can easily take 4-6 months.
- Include PII in the survey so you can refer to the same respondent for later rounds of the survey as needed.
- Avoid changes in survey design mid-way of the process as this can result in inconsistent results.
What to do if the Baseline Survey is Ruled-Out?
If you are reading an article on Impact Evaluation procedure chances are that you are in the end of the program phase. Thus, if the baseline survey is missed in the beginning you cannot go back to collect that data. Here are a few things we can do:
- Find another dataset to serve as a baseline, this can be a secondary data collected by a different agency on similar parameters.
- If no such study can be referred you can use publicly available a national survey data and create a counterfactual group using propensity score matching. If you are evaluating a national or sector-wide intervention that this is a completely reasonable approach.
- A survey can be performed by asking respondents of an interest group to recall on the variables in focus. It is practical if we expect a major life change resulting from the initiative. For example, farmers are expected to remember what it was like in the absence of irrigation 5 years back. If you choose to do this make note of below:
- People often consider past events more recent than they actually were. Use some time and a historic benchmark to avoid this psychological conundrum.
- Don’t expect your respondents to remember exact figures like dates, time, prices, etc. Give range values to keep them comfortable.
- You can go back to your Theory of Change model and analyze if there are events that could have resulted in the outcome apart from your initiative and if the cause-effect relations established in the beginning were in fact true.
Triangulation means using different types of samples and methods of data collection. Thus, we can ensure the validity of results by comparison. This step becomes all the more necessary when we cannot eliminate selection bias or establish an authentic baseline. Triangulation helps build confidence in findings and fills in gaps in statistical studies. Be sure to allocate part of your budget and time to this step.
5. Adding Context to Evaluation Findings:
The purpose of the evaluation is not just to measure but evaluate results. In this step, we need to weigh in the qualitative data alongside the numbers gathered to make a deeper study on the impact actions. The feedback from field data collectors comes handy here. They can give perspective on the situation, status and people lives as well the authenticity of the experiment. The second type of qualitative data would be the inferences are drawn and cause-effect relationships established at the beginning of the experiment. Do they still go in-line with the numbers? If not, where did you miss?
These are some of the steps you can take to effectively conduct an impact evaluation. Be sure to check out our complete guide to Actionable Impact Management to further understand the process of social impact monitoring, evaluation and assessment.