Unmesh Sheth 8/27/19 3:28 PM 31 min read

8 Effective Phases of Impact Data Management

Are you a mission-driven organization looking to build an impact data strategy? Where do you start? This article gives a practical and step-by-step process that you can follow based on your organization's stage. The purpose of this article is not to show you a finish line but to demonstrate how to take a step-by-step journey. The truth is that most organizations remain at an early stage; rather, mission-driven organizations should build a long-term iterative strategy to improve and come closer to their mission and vision continuously. 

The key benefits of taking this approach are: 

  •  Improved understanding of your impact roadmap to meet the purpose of your organization
  • Real knowledge about the effectiveness of your programs, products, or services
  • Access to impact capital based on the most effective use of impact data
  • Making your role even more meaningful 

1. Impact management starts with impact strategy

Let's say that your organization is just starting with the first impact report; where do you start? Starting from the data is not the right way. The foundation of high-quality impact reporting always begins with an impact strategy. So the next question is, how do I define my strategy? Depending on your impact perspective, you can start with the theory of change, logic model, logical framework, or the Impact Management Project dimensions. And based on that, design the indicators to measure performance. 

  • Create an impact strategy per project, program, country, or portfolio
  • Improve the understanding of the context by defining risks and assumptions
  • Build on feedback loops by relating outcomes to the activities for a more efficient impact management
  • Develop a due diligence and application process aligned with the impact

Build Strategy-03

2. You have some program and operational data; now what?

Let's say you are an executive director of a CDFI, healthcare, or education organization. Chances are you are already collecting some form of demographic, program, or service data. Are they useful in the impact report? The probability is high that this data is limited to activity or output data. While you can undoubtedly build impact reports out of them (and that's what an overwhelming number of organizations do), unfortunately, it will not be sufficient for an impact or outcome-oriented grant or impact evidence proposal. While there are many steps in improving impact data maturity, it is essential to streamline the process so that data aggregation is an ongoing activity and not a year-end scrambling exercise.
 

Start with clear documentation of all AS-IS data  

The chances are that your first inventory will reveal that you already have some of the following data:  
    • Financial Results:  Grants, revenue, expenses, other accounting results
    • Operational or Activity Results: Number of students, training, etc
    • Stakeholder or Beneficiary Demographics:  race, income, age, etc
    • Fundraising reporting:  Donor profile, donations results

The first step here is to build a clear inventory and examine data. As soon as you have created the inventory, you are likely to see incompleteness or duplication of data. You must allocate sufficient time to cleanse data once and for all. While this data can serve as a baseline for you, you will need to prepare a strategy to avoid frequent data aggregation pain in every impact reporting cycle.   Next, do this

  • Remove any duplicated results.
  • Do not report anything that can not be verified.
  • Unique count:  It is common that beneficiaries/stakeholders attend multiple activities such as training classes, events, etc. In such a situation, it is better to report unique and total counts both.
  •  

Automate your impact data aggregation for continuous feedback 

Data collection and analysis are not a one-time affair. Mission-driven organizations must build a robust data aggregation strategy so that after every activity you can automatically summarize and analyze the latest results. Every time your field manager goes to the field and syncs your data, your system should automatically build reports, scorecards, and dashboards.

You are here.

excel-imm

While 95% of social sector organizations use Excel or Google docs as data management tools along with other third-party reporting dashboards such as Google Data Studio or Tableau,  this strategy is likely to create a broken process. First, you must design a data aggregation strategy. Tools like Excel or Google docs are easier to use but they create many data management challenges such as:

  • Data loss when the file manager leaves
  • Formula update every time you bring new data 
  • Tracking original sources of data
  • Continuously changing impact metrics and data strategy

To avoid year-end data scrambling, you want to be here.

Average social sector organizations have anywhere from 1 to 50+ data sources internally. Each of them may have different types of stakeholders with varying requirements of reporting, such as:

  • Report to funders on regular frequency like monthly, quarterly, semi-annual, or annual period
  • Narrative oriented report with emphasis on the robust storytelling
  • Individual and personalized reports to the large donors
  • Grantee or investment effectiveness reports
  • Dashboard for internal operations and other KPIs

Management should strive to build a robust data warehousing-like application to improve high-quality impact data evidence and reporting. Most Excel or Google docs-like systems will lose data integrity over time. Instead, a system that can provide high-quality reporting is needed.

So how do we do that?
  • Build a simple data warehouse-like system that simplifies data aggregation from most common data sources that are important for impact reporting
  • Ideally, that system should allow you to build formulas at the database level so that user does not have to update them every-time
  • Provide simple data analytics to find answers to the most frequent output, outcome, and unique data reporting with the ability to visualize results based on reporting type

14

Define an impact data roadmap

  1. There are many objectives of your social impact journey. Depending on our internal management goals and funder requirements, you must define priorities. Based on each priority, now you will continue to add metrics to your impact strategy (theory of change, logic model, or Impact Management Project dimensions). While it is not necessary to do all, depending on your internal goals you will need to develop a systematic plan to scale your program. Your plan should include:
  • Stakeholder voice - immediate feedback
  • Stakeholder voice - qualitative feedback
  • Performance tracking
  • Outcome tracking
  • Stakeholder voice - impact management approach

undefined-2

3. Start with stakeholder's voice

A stakeholder is perhaps the central focal point for assessing impact. Many programs can get much better insights by having a smarter analysis of qualitative feedback collected from clients (beneficiaries or participants). The process of measuring the beneficiary voice must go well beyond simple data collection. The essential requirement is, "can we find out something that you are not looking for?". If you are looking for something specific, you can always ask a close-ended question. 
 

It starts with a well-defined survey and a better engagement process

A well-designed stakeholder survey is a foundation to start building an impact management journey. The key to successful stakeholder listening requires:

  • Well-designed surveys with more precise questions and a mix of open-ended and closed-ended questions
  • Various dimensions of feedback specific to product or services provided by the organization
  • Cost-effective surveys with a clear collection method (door to door, email, SMS, call center), etc.
  • A decent size of a sample to survey (typically around 5% to 15% depending on the size of beneficiary or stakeholder pool)

The following customer impact report helps you understand the methodology, questions, and results for a publicly traded company & community development institution.

undefined-2


4. Integrate qualitative and quantitative analysis 

While the first generation of the survey can be quantitative, a well-designed approach may include qualitative data. An excellent example of a qualitative approach in the interview. Specific well-designed open-ended question will offer a real insight into what stakeholder thinks. A closed-ended question often fails to capture the stakeholder sentiment as they focus on what the organization wants to learn and not what the stakeholder wants to say.

There are many different ways to learn what the stakeholder says. A few examples are:

  1. A poll question followed by opened-ended and well-defined follow-up questions can be captured through social media, interviews, or survey questions.
  2. Analyzing emotions and tones in what people write online, like social media feeds, impact reports, or reviews. By processing qualitative text analysis through Artificial Intelligence can give you tremendous insights into what your stakeholders are saying. 

A well-designed approach can capture real sentiments at the individual stakeholder and group level. The following example gives the overall summary of critical emotions:

  • Predict whether they are happy, sad, confident, and more
  • Key challenges facing product and services
  • What are the other issues they would like for your organization to address in the future?

By linking qualitative response with quantitative or closed-ended question results, as seen in the scorecard below, you can quickly understand causality or correlation between crucial indicators.

Simple emotional analysis & Integrated Scorecard

While qualitative sentiment analysis is valuable in understanding beneficiary voice, the real value comes after analyzing and correlating qualitative and quantitative analysis.  The following example impact scorecard gives an in-depth understanding of how you can analyze and communicate results effectively:

Artboard – 1@2x-1

5. Start with performance management

Mission-driven organizations should clearly define critical indicators that are key to measuring the overall success of the program. These indicators may be activity and/or output-oriented, but they must refer to the overall progress to evaluate internal performance.  

 

Compare cohort results over a period of time.

Analysis can often be derived from existing results tracking systems such as the educational scores of students. The key to such analysis is to conduct the study under similar conditions, with a non-biased approach.

 

density-of-baseline-and

In this example, we can gather learnings like:

  • The average score of participant grew from 4 to 6
  • 35% of the cohort had a score greater than 6 at the end of the program
  • 96% of participants have seen the positive  change in the positive average score
 

Compare individual stakeholder results over a period of time

Target audience: Impact Analyst, Evaluation Team

In this specific case, it may be necessary to look at the patterns of improvement between multiple stakeholders to see if a particular approach works better for individual students. 

monthly-education-score (1)

 

Compare program results with key constraints.

Target Audience: Impact Analyst, Evaluation Team

Funders may be looking to learn more about social, economic, and financial results. Before starting, funders must define a clear hypothesis of the improvement that they are seeking, such as: 

  • Total monthly salary improvement over a period of time
  • Overall monthly salary improvement over a period by Gender

This requires comparing multiple metrics and filtering/grouping by specific criteria such as gender, district, or education level. If they do not see necessary improvements, funders can initiate an action or strategic plan using the impact scorecard. 

 

6. Build a good outcome tracking strategy

 
Outcome tracking should be designed based on a combination of subjective analysis and an evaluative approach. The fundamental criteria here are to review the holistic improvement or observe a specific limitation over time. This analysis not only confirms the overall breadth of improvement but also points out the deficiencies in the program approach.
 
cohort-analysis
 

7. Impact management project stakeholder

The Impact Management Project has made significant progress by bringing different impact practitioners together to agree on a standard definition of impact and the dimensions of performance that matter for impact measurement, management, and reporting across the value chain. While this approach is likely to be well-adapted in the impact investment industry, we believe it could and should be adopted by philanthropic organizations. On the other hand, businesses and ESG investments are already paying attention to this approach.

How to design an IMP-based stakeholder survey

Enterprise or mission-driven organizations might want to start by understanding their stakeholder by collecting data on their demographics. This information is often available in their own internal systems such as student enrollment, loan management systems, etc. Generally, demographic information should be collected based on appropriate data collection for each stakeholder (WHO) and not based on a selected survey.  

Enterprises need further data on which impacts are most important to the stakeholder. They also need to assess the degree of change (How Much) that has occurred for each stakeholder.

What
Impact Category Commentary
   
Outcome in period The outcome experienced by the<stakeholder>when engaging with the enterprise.
The threshold for a positive outcome The level of outcome that the<stakeholder>considers being positive/good enough. Anything below this level is considered negative/not good enough.
Capital type The type of capital that the outcome relates to.
SDG The Sustainable Development Goal that the outcome relates to, along with the specific target and indicator. An outcome may relate to more than one SDG.
SDG target and indicator  
Importance of<outcome>to stakeholder Stakeholder’s view of whether the outcome they experience is important. Where possible, the people experiencing the outcome provide this data, e.g., through direct surveying, although third-party research may also be included. For the environment, scientific research provides this view, such as the work of the Stockholm Resilience Centre.
 
Listen to this  in-depth webinar about the history of stakeholder-oriented impact management and the evolution of impact management:
 

8. Social return on investments

Social Return on Investment (SROI), defined by Social Value International, is a pioneering approach to engage in stakeholder-based social value accountability. Social return on investment (SROI) is a principles-based method for calculating a social return valuation. This method allows to evaluate the impact on stakeholders and create a better performance measurement for investments.

SROI accounts for the stakeholder's view of impact, similar to the Impact Management Project (IMP). However, unlike IMP, it requires financial 'proxy' values on all those impacts identified by stakeholders, which do not typically have market values. The challenge, however, is that the 'proxy' is often context-sensitive, and finding standard proxies may not be easy and may require additional cost. On the other hand, SROI does give a great amount of guidance to an investor if they can define high-quality proxies considering the stakeholder's views.  

It is important to understand that SROI is one mechanism to demonstrate impact, but it should be compared and valued against other alternatives described in this article.

Conclusion

There are many ways to demonstrate the social impact of your mission.  Depending on the maturity of your organization, you can start your journey and build a step-by-step strategy to learn and demonstrate impact.  We demonstrated the following commonly used techniques that will surely put you on the path to raise impact capital from your mission.

  1. Define Impact Strategy
  2. Programmatic Reporting: Financial, Demographics, Operation, Donors
  3. Stakeholder's voice
  4. Performance Improvement
  5. Outcome Improvement
  6. Impact Management Project based Stakeholder Analysis
  7. Social Return On Investment (SROI)
  8. Aggregating results from partners

Learn More:

Impact Measurement and Management

 
avatar

Unmesh Sheth

32 years of track record In technology companies, innovation, leadership. Deep understanding of bottom-up and top-down data trust challenges in high impact philanthropy and impact investments.