<img height="1" width="1" style="display:none" src="https://q.quora.com/_/ad/0609320ad13c4183af7978b1c1744b71/pixel?tag=ViewContent&amp;noscript=1">
Impact Evaluation

Best Practices

Impact Evaluation

For Social Purpose Organizations

Impact Evaluation

Impact Evaluation

What is impact evaluation

Learn what change you are making

 

Impact evaluation provides a deeper understanding of social impact outcomes based on carefully designed scoring, outcome tracking, or stakeholder feedback.

 

Are you a large nonprofit organization collecting results from multiple countries or communities? Or maybe you are an impact investor, accelerator, or family office aiming to create an ecosystem of impact entrepreneurs helping their local communities. If you would like to understand if your interventions are really generating a positive change in people or the planet, you are in the right place.

Watch and Learn

Chris Gaines, SoPact's Lead Trainer explains good outcome tracking can be achieved through a better social impact measurement aligned with common outcome management. Learn detailed tips for outcome evaluation in monitoring and evaluation and social impact measurement

 

How do you start with the best impact evaluation process?

Best Practices

Learning the effectiveness of programs focused on complex issues like poverty, hunger and job creation can be complicated. These programs usually involve an ecosystem of players and multiple entrepreneurs or nonprofits working on their communities. 

In previous videos we highlighted the importance of defining the right outcomes as part of your Theory of Change or your Five Dimensions of Impact. Now, if you are directly working with the target communities, you design a survey to collect data and track progress. 

But how do you know if your survey is really collecting data relevant to your outcomes? How can you tell if your intervention is having positive, negative or neutral effects? 

Tips:

  • Better survey design
  • Applying learnings to other geographies
  • Applying observation to collect data.
  • Scoring survey responses
  • Correlating results to find causality
  • Outcome stars
  • Investor's contribution - Collective Impact

Designing the right survey design

Ask the right questions

As mentioned before, your data will help you track positive and negative outcomes only if your survey is designed properly. 

There are many impact consultants out there that can help you design a good survey, appropriate to your context, and relevant to your outcomes. For example, if your organization has an intervention to reduce poverty through job creation, your survey should collect enough information about how the new job has improved the household’s conditions. It’s not enough to learn that the household income is higher, you need to understand what is that income being used for. Education? Healthcare? Food?

 

 

 

Applying learnings to other geographies.

If your organization is working on multiple countries, you can start by grouping the countries in regions based on context similarities, poverty line, language, etc. It’s important to analyze if the regions are similar enough for you to use the same survey or if you need to make some adjustments depending on the region.

country groups

 

In any case, we recommend starting by collecting data in one country only. This will help you analyze what kind of results you are getting, what questions or options within your question might not be relevant to your target population, etc. If you then apply those learnings to your next country or region, you are more likely to end up with relevant outcome data.

Applying observation to collect data

Observe more and ask less

Some organizations are concerned about having huge surveys that will overwhelm their respondents. To avoid cutting out questions that might provide important information, you can identify the survey responses that you can gather using observation. For example, if you visit the household to apply your survey, you don’t need to ask about the material of the roof, you can observe it. Short surveys save resources and increase result collection.

Screen Shot 2020-03-27 at 3.25.08 PM

 

Scoring survey responses.

Once you start collecting relevant data, you will be able to observe how the outcome results change from one collection period to the other. For example, you will be able to see that last year, the household sent 2 kids to school, but this year they sent 4. 

But to be able to link the results back to the outcome metrics, you might need to apply a scoring mechanism to the survey answers. For example, if the respondents are sending 2 kids to school they get a score of 1 point, but if they are sending 4 kids to school, they get a score of 2 points. 

This allows you to:

  1.  Have an easier way to track if the trends are getting better or not over time
  2. Aggregate the results across multiple communities or countries
  3. Compare similar communities
  4. Report your results in an easier way to stakeholders that are not familiar with your full process.

Correlating results to find causality.

Causality

Learn from your data and correct the root cause

Even if you decide to aggregate your data for reporting purposes, make sure that you are also analyzing the survey results using correlation. For example, if one of your hypotheses is that increasing the income will also increase the consumption of more nutritious food, put those two variables together. Are they really correlated? If not, it’s time to go back to your communities and understand what’s preventing them from consuming nutritious food. Maybe there’s a lack of access to such food. How can your organization resolve this issue? One of the ways is working with a  partner making nutritious food available in your communities. 

Outcome Stars

Progress

Outcomes Stars are evidence-based tools designed to support positive change and greater wellbeing, with scales presented in a star shape and measured on a clearly defined ‘Journey of Change’. The Outcomes Star is completed as part of conversations between individuals and support practitioners such as key workers. There are many outcome stars defined such as areas Adult Care, Community, Criminal Justice, Domestic Violence. Each set or star consists of a series of questions that can eventually be ranked from one for low to five for high.  Based on response, individuals or cohort can be tracked over a time to see overall progress and design appropriate intervention.

outcomestar

 

Investors contribution.

Collaborative impact

Traditionally, investors and accelerators have limited their involvement with the portfolio companies to simply requesting results data. But increasingly, we see a shift to a deeper level of involvement, from providing the monetary or technology resources to manage the companies’ impact to providing advisory in the definition of outcomes, metrics, and surveys. 

Here at sopact, we understand that organizations just like yours face challenges with impact measurement and management on a daily basis. So we’ve developed a platform that allows you to collect and manage any kind of stakeholder data and link it back to your outcome metrics to demonstrate progress over time.  

Impact Evaluation Plan

 

Impact evaluation is the process where we study the results of social interventions. We ascertain the value of the overall initiative and use the findings as learnings for future endeavors. This is a critical process for both the investors to understand the rationale behind continuing the program and for impact makers to see if their efforts are bringing positive change. This article underlines the steps you need to take to define an effective impact evaluation plan.

1. Establish Program Theory of the Social Impact Initiative

Program theory is the step where you document the assumptions and logical arguments that define the rationale behind your program. If you would like to take a more sophisticated approach we recommend to go a level higher and do as below:

Define Theory of Change:

Theory of change is the program theory where you not only describe the assumptions and logical arguments in favor of your initiative but also lay down all the possible scenarios that can result from your impact actions. This gives a clearer picture and helps you understand the evaluation reports from a wider perspective.

Here is all you need to know to define Theory of Change for your program.

Setting up the Counterfactual:

In its essence counterfactual analysis is a “with versus without” analysis. We study the impact of a program by comparing the results from a control group where no artificial factors were stimulated. You need to set up this control group in the early stage so that the evaluation can be done parallelly and results are comparable. In some cases, this can be avoided where there is no other factor that can bring any observed change in outcomes (e.g. reductions in time spent fetching water after the installation of water pumps).  

Read More: The best tools you should know for Monitoring and Evaluation

 

2. Addressing Selection Bias in Impact Evaluation

 

selection bias in Impact data collection and ways to eliminate it during assessment

The presence of selection bias can skew the results of evaluation hence, it is important that we take whatever steps possible to eliminate all bias. Below are the preconditions to identify the extent of selection bias and the remedial step that can be taken to eradicate it:

 

Ex-ante Procedure:

  1. If the evaluation metrics are determined prior to the event then we need to see if randomization is possible. To further explain if the treatment group is chosen randomly then another set of the random counterfactual is a valid test. It is possible to target a subgroup of the random subject and still remain unbiased. For example, if an initiative was designed for below minimum wage workers then the counterfactual group can also be from the same subset to maintain relevance.
  2. If the above is not the case we see whether the selection determinants are observed. There are a number of regression techniques that can help eliminate bias in this case.
  3. In the case of unobserved selection determinants, we need them to be time-invariant so that panel data can be used to remove bias. For this case baseline (or some means of substituting baseline) is critical.

 

Ex-post Procedure:

  1. In this case, the panel is not possible. Since the selection determinants are unobserved, we need to identify ways of observing the determinants.
  2. If that fails we can go for the pipeline approach provided there are some untreated beneficiaries.

 

We cannot address the problem of selection bias if none of the above are possible. Thus, we are left to rely on program theory and triangulation to build an argument. Thus, setting up a Theory of Change (TOC) Model helps make plausible association easy.

Read More: How to Evaluate Impact of Social Innovation Program

 

3. Designing the Stakeholder Survey

 

A baseline survey is performed when the project is being initiated i.e. at the beginning of the project but implementation. It helps prioritize between different objectives of an initiative and works as a benchmark to identify the success or failure of it.

Guiding principles for an effective baseline survey in Impact Evaluation:

  1. It should be in line with the program theory and data must be collected across the results chain, not just on outcomes.
  2. The counterfactual should be presented in the same questionnaire. Intervention-specific questions should be replaced with similar questions of a more general nature can help test for any influence of initiative.
  3. Allocate enough time to double check on instruments before initiating the survey. It should be convenient to form a relational database to ease analysis and data entry. This process can easily take 4-6 months.
  4. Include PII in the survey so you can refer to the same respondent for later rounds of the survey as needed.
  5. Avoid changes in survey design mid-way of the process as this can result in inconsistent results.

 

Read More: 8 Key Challenges in Monitoring and Evaluation and How to Overcome them?

 

 

What to do if the Baseline Survey is Ruled-Out?

If you are reading an article on Impact Evaluation procedure chances are that you are in the end of the program phase. Thus, if the baseline survey is missed in the beginning you cannot go back to collect that data. Here are a few things we can do:

 

  1. Find another dataset to serve as a baseline, this can be a secondary data collected by a different agency on similar parameters.
  2. If no such study can be referred you can use publicly available a national survey data and create a counterfactual group using propensity score matching. If you are evaluating a national or sector-wide intervention that this is a completely reasonable approach.
  3. A survey can be performed by asking respondents of an interest group to recall on the variables in focus. It is practical if we expect a major life change resulting from the initiative. For example, farmers are expected to remember what it was like in the absence of irrigation 5 years back. If you choose to do this make note of below:
    1. People often consider past events more recent than they actually were. Use some time and a historic benchmark to avoid this psychological conundrum.
    2. Don’t expect your respondents to remember exact figures like dates, time, prices, etc. Give range values to keep them comfortable.
  4. You can go back to your Theory of Change model and analyze if there are events that could have resulted in the outcome apart from your initiative and if the cause-effect relations established in the beginning were in fact true.

Read More: How to use Impact Reporting for Storytelling Impact Learnings

 

4. Triangulation

data and team work to use triangulation to establish conformity on impact assessmentTriangulation means using different types of samples and methods of data collection. Thus, we can ensure the validity of results by comparison. This step becomes all the more necessary when we cannot eliminate selection bias or establish an authentic baseline. Triangulation helps build confidence in findings and fills in gaps in statistical studies. Be sure to allocate part of your budget and time to this step.

Read More: The best software to use for Monitoring, Evaluation and Impact Learning

 

5. Qualitative vs quantitative analysis

The purpose of the evaluation is not just to measure but evaluate results. In this step, we need to weigh in the qualitative data alongside the numbers gathered to make a deeper study on the impact actions. The feedback from field data collectors comes handy here. They can give perspective on the situation, status and people lives as well the authenticity of the experiment. The second type of qualitative data would be the inferences are drawn and cause-effect relationships established at the beginning of the experiment. Do they still go in-line with the numbers? If not, where did you miss?

Read More:  5 Worst Mistakes to Avoid in M&E Software Selection

 

These are some of the steps you can take to effectively conduct an impact evaluation. Be sure to check out our complete guide to Actionable Impact Management  to further understand the process of social impact monitoring, evaluation and assessment.

 

Read More: Social Impact Assessment

More Impact Reporting Resources

Sopact Academy
New call-to-action
Jump To Section:
All About Outcomes
impact evaluation

Stay connected with latest in impact strategy!