Social impact assessment is a rapidly evolving practice. As a result, no matter if your organization is just starting out or if it's been around for over 30 years, chances are your impact assessment process (if you have one) lacks in consistency of both data and messaging.
While some members of your staff may know the names of every beneficiary you serve, they might not know the Vision or Mission. Challenges like this make the concept of a formidable impact report feel like a possibility only in the distant future. Indeed, the process of establishing the framework, the metrics, the data tracking tools, the waiting, collection of data, analysis of data - all before it's possible to publish your impact report. What a process! However, it's not just work and waiting - you can plunge into the waters of social impact assessment with the assurance that as the process evolves over time, starting week one new elements of your reporting needs will be unearthed as you go. This article will talk you through navigating the limbo world of social impact reporting in the time before the indicators' data is aggregated for analysis. What follows is how to make this period productive and enable you to communicate elements of your evolving impact story as the process unfolds...
Typically impact reports cover three areas: (1) Impact Framework; (2) Assessment Results; (3) Performance Evaluation.
You'll notice, that while the last two are necessary for completed evaluative report, the first offers great insights into the impact hypothesis of an organization. Effectively demonstrating this hypothesis alongside a roadmap for data collection for analysis, is impact communication in its own right.
Here's a quick tale of two cities: the first was dreamt up with a clear vision that was well communicated. This allowed for strategic planning. The infrastructure and city development, built from the ground-up, evolved into a beautifully efficient community. The second, a city built organically over time - subject to the whims of a diverse array of mayors and stakeholders, grew into a large and clumpy conglomerate of traffic and inequity.
When it comes to building your impact framework, starting your assessment journey, you can follow the design of the first city - even if your organization is no where near new. However, it is just as powerful to have a well established impact framework that nods to the organization's history (no matter how inconsistent a story it is), while outlining a definitive impact hypothesis alongside a way forward. You can make the data waiting period a productive one - building out the documentation for your organization's historical context and your impact framework. This clarity in vision will be a bold cornerstone of supporting narrative once the data does roll in for assessment and demonstration.
Of the three areas of an impact report identified above, one of those sections (Impact Framework) does not require data analysis. Nonetheless, it is a fundamental piece of your impact story. In some cases, the ability to speak to your approach toward impact moving forward can open up unexpected doors for engagement with potential funders. That was the case with one of our asset manager and social businesses (investment). To embark on the journey of developing your own impact framework, take a look at Actionable Impact Management (AIM) a series of guidebooks discovering and documenting your framework.
So, you've got your framework in place. Now what? Knowing where to put it starts by understanding your framework's value and why people are interested. By explaining your impact framework you are engaging your stakeholders (donors, beneficiaries, board, staff, public) in a transparent way - looping them in to your practices and processes. It is a credibility-building exercise.
What if you've been around for 50 years? How do you introduce a new way forward while giving due credit to where you've been? As we mentioned in the opening of this article - impact assessment is a RAPIDLY evolving field. Few - if any - can boast a consistent and comprehensive impact measurement strategy over their history. So be a little forgiving and embrace your historical assessments.
Anything with recurring structures catches the eye of technology enthusiasts as an opportunity. At SoPact we caught on to the recurrence of impact storytelling and its intrinsic coupling with data management. That is how the Impact Cloud was born. The software platform manages the data collection, aggregation, and analysis for impact storytelling. We noticed the importance of demonstrating all of the primary components of impact reports mentioned earlier: (1) Impact Framework; (2) Assessment Results; (3) Performance Evaluation. The recurring structures embedded within each have evolved into layouts of what we call 'impact cards.' In the system, each impact card contains a layout demonstrative of the information it is conveying (from Theory of Change to data assessment results).
These cards are automatically updated as new data comes in and can be easily downloaded and placed on anything from postcards for donors to websites, emails, reports - and the list goes on. As we move out of Phase One's Impact Framework, and the data does start streaming in, it's helpful to have an enduring process already in place. If you want to check out Impact Cloud, we're always happy to chat.
There's a common Catch-22 in the social impact assessment or social impact measurement field.
Funders often choose metrics based on the varying capacity of grantees. Paired with the limited data-capacity faced by the funders themselves, this results in assessments which only capture the low hanging fruit of the data world.
In other words, it translates into the selection of easy-to-measure activity- or output- focused metrics.
In this article, we'll further explore why this tendency is so pervasive, how it works against the goals of a social impact assessment, and what we can do about it.
Whether you find yourself on the funder side, the asset side (social enterprises, etc.), or somewhere in between, by the end of this article you'll better understand how to avoid the primary social impact assessment pitfall, and will be armed with the tools necessary to do so successfully.
A social impact assessment ostensibly tells us how well a program or intervention is performing in terms of achieving impact outcomes.
And yet, our view is that most assessments fail because they too often focus on outputs that have occurred as opposed to getting at outcomes.
Outputs are, of course, easily measured. Your questionnaire for program managers or beneficiary liaisons might include questions like: How many hours of training were delivered; How many people were served; How many products are being used by beneficiaries. These imply impact, but do not demonstrate it.
And these metrics hold little strategic value for the grantees, especially if they are only measured for a short period.
And so, a stigma has evolved around the trustworthiness of the data received as well as how helpful what we're calling 'measurement and evaluation' really is to discern the impactful results of our interventions. With such little regard for the practice, there is little incentive to invest in improving the process, and so, here we are in a Catch-22 of activity-driven (i.e. output-driven) measurement.
Read More - Is collective impact evaluation the future of social impact assessment?
Something wonderful happens when an organization decides to have an outcome-driven social impact measurement process. They begin to reassess their impact framework more critically than ever before. That's because, to build an enduring practice for impact measurement, an organization must have a well-developed impact foundation to guide it through the metrics selection. This starts with their Vision statement and moves through their Theory of Change.
What we've found is that some of our clients were missing elements of this impact framework. The ability (for the first time) to measure for results facilitates an internal conversation around the broader impact hypothesis, and the organization is left with more established impact messaging.
The benefits of results-driven evaluation go beyond the structuring of an organization's impact framework. It includes the strategic insights gained from a better understanding of the outcomes of your interventions. Armed with such insights, your organization is better able to iterate the process. And when built upon a solid foundation that includes a thoughtful Theory of Change, an organization is more aware of its assumptions and areas of potential risk. With an outcome-oriented approach, measurement and evaluation become a pivotal element of your organization's strategic decisions. With the ability to more confidently speak to the specific results of your interventions, your marketing and donor relations will also benefit.
Read More - The potential social and economic returns of collective impact
One phrase that pops up again and again in the unfortunate cycle is "data capacity". If we might be able to enhance our own capacity and that of our partners, perhaps we will be able to take on more challenging metrics that offer more strategic insights. Once the value of the data becomes apparent, its positive impact on programming, marketing, donor relations, and grant seeking lead to better staff engagement with the data reporting. Especially if reporting becomes more user-friendly.
Impact Cloud™ was designed to be accessible and thorough, flexible enough to meet the data needs of various socially minded organizations. And it starts with a well-developed impact framework. Actionable Impact Management begins with the introspective work of defining a Vision, Mission, and Goals; Program Structure; and Theory of Change before moving on to the selection of metrics and your data strategy.
The transformation from activity or output metrics to outcome metrics can mean all the difference in buy-in. By strengthening data capacity, funders can make that transformation a reality.
Read More: How can we improve social impact accounting?
With better data, we can direct capital to grow the social enterprises that are making the most difference.
We will go over commonly used social impact approaches today and point out SoPact’s focus area and approach to improve a better outcome.
When social impact evaluators communicate findings to funders, there is a fundamental lack of common approach. Let’s start by understanding the four most common approaches in this domain from bottom-up.
Compare Social Impact Measurement Methods Randomized Control Trial at the bottom is primarily used by impact researchers who lead evidence-based research approaches. Though these methods can be justified in many long-term studies, often they are useful in proving or disproving – if treatment is effective for a selected program. Unfortunately, this approach can be costly, often error-prone and time-consuming. More importantly, they do not provide immediate feedback.
is more useful when the beneficiary group requires agile feedback and regular interventions such as disability employment, HIV/AIDS outcome, Patient outcome, etc. Unlike the survey-based approach, these systems manage the beneficiary lifecycle and manage program outcomes.
Often this area is looked as a traditional M & E from a funder perspective. The challenge with current approaches is that there is a limited agreement between funder and investee/grantee on measurement language and limited sharing reduces impact envisioned by funders. Later in the section, I will demonstrate SoPact’s collaborative approach allows to create shared measurement – improving a long-lasting effect on intervention.
Social impact measurement practices can be seen as a pyramid, where B-Analytics/GIIN works with IRIS 3.0 at the top. Though they are leading impact measurement services for social impact funds, their approach has major transparency issues. For Example- IRIS does not include any metrics about immunization rates - or even about health. One of the many IRIS Partner Metrics systems being developed with the Center for Health Market Interventions does include health metrics,….. one of which is “Health Intervention Completion Rate” But imagine being the manager of this impressive immunization program and having to enter 33% in an investor form that included that metric - it just doesn’t capture the impact that you have been working so hard for.
Stakeholder Impact Analysis is thoughts, beliefs, needs, feedback, etc. communicated by individuals defined as stakeholders for any given impact area.
Principally, social sector stakeholders are the target beneficiaries of an intervention, although in general terms a stakeholder is any individual or entity that stands to be affected (positively or negatively) by the activities undertaken by an organization. The following content focuses on those beneficiary stakeholders who are essential to any impact-creating ambitions.
The success of an impact-driven initiative is nearly always bound by the extent to which those that are designing and implementing the initiative take into account the perspective of key stakeholders, especially those beneficiaries who are the reason for the initiative. Here are a few reasons why raising those voices is so important:
Natural Language Processing has come a long way in recent years and is now being applied to capture impact insights from beneficiaries.
Qualitative analysis, for example asking beneficiaries questions and recording their long-form responses for later analysis (using surveys, interviews, etc.), is time-intensive and not quite a hard science. And yet, stories can be a powerful tool to understand the depths of impacts perceived by a beneficiary group. The key is to be able to transform those narratives into quantitative impact evidence. This is where the marriage between artificial intelligence and natural language processing comes into play.
Sopact’s Impact Cloud, an end-to-end cloud-based impact data management tool, gives users the capacity to harness such technology.
Uploading qualitative responses to the tool enables impact practitioners to easily conduct keyword sentiment and emotion analysis and transform qualitative data into quantitative.
In this way, from impact narratives full of stakeholder sentiment practitioners can glean important impact insights.
Shared insights are those impact learnings which are determined in collaboration with target beneficiaries. These insights go beyond traditional outputs such as number of beneficiaries affected to revealing how those beneficiaries were affected, taking into account their own perceptions of impacts generated.
Shared insight can also refer to the open-sourcing of impact data and impact learnings across impact organizations and across sectors. Taking a more collaborative approach to how we wield impact data can ultimately help impact initiatives globally, if impact players share analyses, insights, and outcomes achieved. If you are interested in such collaboration, take a look at the Fund for Shared Insight.
As with any activity your organization performs, it should be done with a clear strategic purpose, not just because it seems like something you should do. With that strategic perspective in mind, here are a few ways to make sure you leverage those interactions with constituents.
Keep it periodic, but not constant.
For most beneficiaries, you don’t want to completely invade their life with requests for data. Doing so can lead to overwhelm and have an adverse affect on their willingness to be a part of that process. Ask yourself, given the impact outcomes we are targeting and our program design, when are key moments we should be “checking in” to get feedback from our beneficiaries? Of course, if it is unclear, ask the beneficiaries themselves!
Have an analysis plan in place.
You conduct dozens of interviews and even more online surveys. Great, but what are you going to do with those data? Do you have the expertise to analyze it properly? The time? Program design shouldn’t just be about creating an effective intervention, it should also include how you will hold yourself accountable to making that intervention an effective one. Because if you are ultimately unable to utilize constituent feedback to assess and improve, you run the risk of wasting not just your time and energy, but that of your beneficiaries as well.
Tell stakeholders how it helped you!
A great way to achieve buy-in from stakeholders in terms of their willingness to be a part of the data collective process is to explain clearly how such data will be used. And most importantly, follow up with those communities after analyses have been conducted to show how constituent voice contributed to understanding the impacts created and potential design improvements moving forward.
A stakeholder impact analysis can be conducted at a variety of times throughout the life of an intervention. At the outset, such an analysis serves to support the feasibility testing and initial design. During and post-intervention these analyses serve to assess impacts created and eventually how to improve the intervention design.
According to the Impact Management Project, there are five key dimensions of impact which should be taken into account in any stakeholder-centered analysis (seen below).
Source: Impact Management Project
It is important to distinguish between the two main types of analyses available to impact practitioners. A quantitative analysis takes raw numbers and calculates outputs, determines probabilities or percentages, and enables practitioners to make inferences based on those results. A qualitative analysis primarily takes data from subjective self-reporting of stakeholders and uses various methods to extract insights.
A breakdown of the differing data types can be seen below, taken from a recent report on using self-reported data:
Source: Using Self-reported Data
A quantitative approach is best used when there is specific, precise, and easily obtainable information that can be collected from a stakeholder group. For analyses that seek to compare between time periods (for example, before and after), or between groups (for example, an experimental and control), quantitative approaches are often the most useful. In addition, in evaluating outputs and frequency of activities, a quantitative approach is usually the most feasible.
A qualitative evaluation assesses contextual factors, usually from the subjective view of primary stakeholders themselves. Qualitative approaches take into account how such individuals evaluate their own experiences and potential impacts perceived. This approach is best used when the objective of the evaluation is to assess needs, desires, and expectations of a group, as well as how they feel about impacts generated.
While there is no one way to conduct a stakeholder analysis, there are, of course, best practices to be followed and common tools which can be used to get the most out of the time invested.
For a useful introduction to the processes involved in raising stakeholder voice and how to do so effectively, we recommend reviewing “Using IRIS+ to Incorporate Stakeholder Voice,” a report published by the Global Impact Investing Network (which takes the investor engagement perspective). But it will also help general impact practitioners in any role, especially those wishing to dive further into the world of standardized metrics (IRIS+), which can be essential in various types of analyses of stakeholders as well.
Here are a few of the key takeaways of the report to get you started effectively incorporating stakeholder voice:
If you are looking to dive deeper into the topic for your own research or are looking for expert consulting or to explore collaboration opportunities, the following organizations maintain stakeholder voice as a key principle of their work.
Essential to raising stakeholder voice is making sure that in addition to opening up that conversation with them, organizations also analyze those data and put insights to work to create better interventions and ultimately better outcomes. That process is exactly what the non-profit organization Feedback Labs seeks to foster across the social sector.
Founded in 2013, Feedback Labs works to create standardized metrics which assess how organizations are doing in conducting feedback-related work. They also promote the development of tools, trainings, and events to further strengthen the field.
Social Value International brings together a global community (45 countries) of impact practitioners to promote better practices in social value measurement and analysis and to influence policy. Its driving goal is to improve the way we account for social value. As part of their knowledge-sharing approach, they developed seven guiding principles of social value, the first of which is defined as “Involving Stakeholders.”
This as the first principle underscores the importance of stakeholder engagement in the social value measurement process. As discussed above, it is essential that primary stakeholders are involved in, or at the very least informed about, what is getting measured and how.
The Keystone team are experts in everything to do with managing feedback and getting the most out ofthose interactions with constituents. In addition to consulting services, they offer analyses of your existing measurement systems, diversity programs, and more. Their Constituent Voice tool can help organizations get the most out of those feedback data and the Feedback Commons provides tutorials to help practitioners improve how they manage those processes. You can find their full page of resources and guides here.
A social enterprise created by Acumen and based in its Lean Data approach, 60 Decibels aims to both improve adoption of Lean Data approaches across sectors and also introduce what they hope will be the new standard for impact measurement across sectors as well. This standard puts stakeholder voice at the forefront of the measurement process, emphasizing the importance of a beneficiary’s “lived experience.” The enterprise seeks to solve impact measurement issues (comparability, for example) faced by the impact investing sector and to ultimately improve outcomes through better measurement that is stakeholder centric.