8 Effective Phases of Impact Data Management

Are you a mission-driven organization looking to build impact data strategy, where do you start? This article gives a practical and step-by-step process that you can follow based on your organization's stage. The purpose of this article is not to show you a finish line, but to demonstrate how to take a step-by-step journey. The truth is that most organizations remain at an early stage, rather mission-driven organization should build a long-term iterative strategy to continuously improve and come closer to your mission and vision.

The key benefits of taking this approach are:

Improved understanding of your impact roadmap to meet the purpose of your organization
Real knowledge about the effectiveness of your programs, products or services
Access to impact capital based on the most effective use of impact data
Making your role even more meaningful

1. Impact management starts with impact strategy

Let's say that your organization is just starting with the first impact report, where do you start? Starting from the data is not the right way. The foundation of high-quality impact reporting always begins with an impact strategy. So the next question is, how do I define my strategy? Depending on your impact perspective, you can start with the theory of change, logic model, logical framework or the Impact Management Project dimensions. And based on that, design the indicators to measure performance.

Create a impact strategy per project, program, country, or portfolio
Improve the understanding of the context by defining risks and assumptions
Build on feedback-loops by relating outcomes to the activities for a more efficient impact management
Develop a due-diligence and application process aligned with the impact

Build Strategy-03

2. You have some program and operational data, now what?

Let's say you are an executive director of a CDFI, healthcare, or education organization. Chances are you are already collecting some form of demographic, program, or service data. Are they useful in impact report? The probability is high that this data is limited to activity or output data. While you can undoubtedly build impact reports out of them (and that's what an overwhelming number of organizations do), unfortunately it will not be sufficient for an impact or outcome-oriented grant or impact evidence proposal. While there are many steps in improving impact data maturity, it is essential to streamline the process so that data aggregation is an ongoing activity and not a year-end scrambling exercise.

Start with a clear documentation of all AS-IS data

Chances are that your first inventory will reveal that you already have a some of the following data:

- Financial Results: Grants, revenue, expenses, other accounting results
- Operational or Activity Results: Number of students, trainings etc
- Stakeholder or Beneficiary Demographics: race, income, age etc
- Fundraising reporting: Donor profile, donations results

The first step here is to build a clear inventory and examine data. As soon as you have a created the inventory, you are likely to see incompleteness or duplication of data. You must allocate sufficient time to cleanse data once and for all. While this data can serve as a baseline for you, you will need to prepare a strategy to avoid a frequent data aggregation pain in every impact reporting cycle.

Next, do this

- Remove any duplicated results
- Do not report anything that can not be verified
- Unique count: It is common that beneficiary/stakeholders attend multiple activities such as training classes, events etc. In such a situation, it is better to report unique and total counts both.

Automate your impact data aggregation for continuous feedback

Data collection and analysis is not a one-time affair. The mission-driven organizations must build a robust data aggregation strategy, so that after every activity you can automatically summarize and analyze the latest results. Every time your field manager goes to the field and syncs your data, your system should automatically build reports, scorecards and dashboards.

You are here.

excel-imm

While 95% of social sector organizations use Excel or Google docs as data management tools along with other third party reporting dashboards such as Google Data Studio or Tableau, this strategy is likely to create a broken process. First you must design a data aggregation strategy. Tools like Excel or Google docs are easier to use but they create many data management challenges such as:

Data loss when the file manager leaves
Formula update every-time you bring new data
Tracking original sources of data
Continuously changing impact metrics and data strategy

To avoid year-end data scrambling, you want to be here.

Average social sector organizations have anywhere from 1 to 50+ data sources internally. Each of them may have different types of stakeholders with varying requirements of reporting, such as:

Report to funders on regular frequency like monthly, quarterly, semi-annual or annual period
Narrative oriented report with emphasis on the robust storytelling
Individual and personalized reports to the large donors
Grantee or investment effectiveness reports
Dashboard for internal operations and other KPIs

Management should strive to build a robust data warehousing-like application, to improve high-quality impact data evidence and reporting. Most Excel or Google docs-like systems will lose data integrity over time. Instead, a system that can provide high-quality reporting is needed.

So how do we do that?

Build a simple data warehouse-like system that simplifies data aggregation from most common data sources that are important for impact reporting
Ideally, that system should allow you to build formulas at the database level, so that user do not have to update them every-time
Provide simple data analytics to find answers to most frequent output, outcome and unique data reporting with the ability to visualize results based on reporting type

Define an impact data roadmap

There are many objectives of your social impact journey. Depending on our internal management goals and funder requirements, you must define priorities. Based on each priority, now you will continue to add metrics to your impact strategy (theory of change, logic model or Impact Management Project dimensions). While it is not necessary to do all, depending on your internal goals you will need to develop a systematic plan to scale your program. Your plan should include:

Stakeholder voice - immediate feedback
Stakeholder voice - qualitative feedback
Performance tracking
Outcome tracking
Stakeholder voice - impact management approach

3. Start with stakeholder voice

A stakeholder is perhaps the central focal point for assessing impact. Many programs can get much better insights by having a smarter analysis into qualitative feedback collected from clients (beneficiaries or participants). The process of measuring the beneficiary voice must go well beyond simple data collection. The essential requirement is, "can we find out something that you are not looking for?". If you are looking for something specific, you can always ask a close-ended question.

It starts with a well-defined survey and a better engagement process

A well-designed stakeholder survey is the foundation to start building an impact management journey. The key to successful stakeholder listening requires:

Well-designed surveys with more precise questions and a mix of open-ended and closed-ended questions
Various dimensions of feedback specific to product or services provided by the organization
Cost-effective surveys with a clear collection method (door to door, email, SMS, call center), etc.
A decent size of a sample to survey (typically around 5% to 15% depending on the size of beneficiary or stakeholder pool)

The following customer impact report helps you understand the methodology, questions, and results for a publicly-traded company & community development institution.

4. Integrate qualitative and quantitative analysis

While the first generation of the survey can be quantitative, a well-designed approach may include qualitative data. An excellent example of a qualitative approach is the interview. Specific well-designed open-ended question will offer a real insight into what stakeholder thinks. A closed-ended question often fails to capture the stakeholder sentiment as they focus on what the organization wants to learn and not what the stakeholder wants to say.

There are many different ways to learn what the stakeholder says. A few examples are:

A poll question followed by opened-ended and well-defined follow-up questions that can be captured through social media, interviews, or survey questions.
Analyzing emotions and tones in what people write online, like social media feeds, impact reports, or reviews. By processing qualitative text analysis through Artificial Intelligence can give you tremendous insights into what your stakeholders are saying.

A well-designed approach can capture real sentiments at the individual stakeholder and group level. The following example gives the overall summary of critical emotions:

Predict whether they are happy, sad, confident, and more
Key challenges facing product and services
What are the other issues they would like for your organization to address in the future

By linking qualitative response with quantitative or closed-ended question results as seen in the scorecard below, you can quickly understand causality or correlation between crucial indicators.

Simple emotional analysis & Integrated Scorecard

While qualitative sentiment analysis is valuable in understanding beneficiary voice. The real value comes after analyzing and correlating qualitative and quantitative analysis. Following example impact scorecard gives an in-depth understanding of how you can analyze and communicate results effectively:

5. Start with performance management

The mission-driven organizations should clearly define critical indicators that are key to measuring the overall success of the program. These indicators may be activity and/or output-oriented, but they must refer to the overall progress to evaluate internal performance.

Compare cohort results over a period of time

Analysis can often be derived from existing results tracking systems such as educational scores of students. Key to such analysis is to conduct the study under similar conditions, with a non-biased approach.

density-of-baseline-and

In this example, we are able to gather learnings like:

The average score of participant grew from 4 to 6
35% of the cohort had a score greater than 6 at the end of the program
96% of participants have seen the positive change in the positive average score

Compare individual stakeholder results over a period of time

Target audience: Impact Analyst, Evaluation Team

In this specific case, it may be necessary to look at the patterns of improvement between multiple stakeholders, to see if a particular approach is working better for individual students.

monthly-education-score (1)

Compare program results with key constraints

Target Audience: Impact Analyst, Evaluation Team

Funders may be looking to learn more about social, economic, and financial results. Before starting, funders must define a clear hypothesis of the improvement that they are seeking, such as:

Total monthly salary improvement over a period of time
Overall monthly salary improvement over a period by Gender

Often this requires comparing multiple metrics and filtering/grouping by specific criteria such as gender, district or education level. If they do not see necessary improvements, funders can initiate an action or strategic plan using the impact scorecard.

6. Build a good outcome tracking strategy

Outcome tracking should be designed based on a combination of subjective analysis and evaluative approach.The fundamental criteria here is to review the holistic improvement or observe a specific limitation over a period of time. This analysis not only confirms the overall breadth of improvement, but also points out the deficiencies in the program approach.

7. Impact management project stakeholder

The Impact Management Project has made significant progress by bringing different impact practitioners together to agree on a standard definition of impact, and the dimensions of performance that matter for impact measurement, management and reporting across the value chain. While this approach is likely to be well-adapted in the impact investment industry, we believe that it could and should be adopted by the philanthropic organizations. On the other hand, businesses and ESG investments are already paying attention to this approach.

How to design an IMP-based stakeholder survey

Enterprise or mission-driven organizations might want to start by understanding who is their stakeholder by collecting data on their demographics. Often time, this information is already available in their own internal systems such as student enrollment, loan management systems, etc. Generally, demographic information should be collected based on appropriate data collection for each of the stakeholders (WHO) and not based on a selected survey.

Enterprises need further data on which impacts are most important to the stakeholder. They also need to assess the degree of change (How Much) that impact has occurred for each of the stakeholders.

What
Impact Category	Commentary

Outcome in period	The outcome experienced by the<stakeholder>when engaging with the enterprise.
Threshold for positive outcome	The level of outcome that the<stakeholder>considers to be positive/good enough. Anything below this level is considered negative/not good enough.
Capital type	The type of capital that the outcome relates to.
SDG	The Sustainable Development Goal that the outcome relates to, along with the specific target and indicator. An outcome may relate to more than one SDG.
SDG target and indicator
Importance of<outcome>to stakeholder	Stakeholder’s view of whether the outcome they experience is important. Where possible, the people experiencing the outcome provides this data, e.g. through direct surveying, although third party research may also be included. For the environment, scientific research provides this view, such as the work of the Stockholm Resilience Centre.

Listen to the this in-depth webinar about the history of the stakeholder-oriented impact management and the evolution of impact management:

8. Social return on investments

Social Return on Investment (SROI) defined by Social Value International is a pioneering approach to engage in stakeholder-based social value accountability. Social return on investment (SROI) is a principles-based method for calculating a social return valuation. This method allows to evaluate the impact on stakeholders and create a better performance measurement for investments.

SROI accounts for the stakeholders view of impact, similar to Impact Management Project (IMP). However, unlike IMP, it requires financial 'proxy' values on all those impacts identified by stakeholders which do not typically have market values. The challenge, however, is that the 'proxy' is often context-sensitive and finding standard proxies may not be easy and may require additional cost. On the other hand, SROI does give a great amount of guidance to an investor if they are able to define high-quality proxies considering the stakeholders views.

It is important to understand that SROI is one of the mechanisms to demonstrate impact but it should be compared and valued against other alternatives described in this article.

Conclusion

There are many ways to demonstrate social impact of your mission. Depending on the maturity of your organization you can start your journey and build a step by step strategy to learn and demonstrate impact. We demonstrated following commonly used techniques that will surely put you on path to raise impact capital from your mission.

Define Impact Strategy
Programmatic Reporting: Financial, Demographics, Operation, Donors
Stakeholder voice
Performance Improvement
Outcome Improvement
Impact Management Project based Stakeholder Analysis
Social Return On Investment (SROI)
Aggregating results from partners

Learn More:

Impact Measurement and Management

Impact Perspective