Social Impact Assessment

Social impact assessment is a rapidly evolving practice. As a result, no matter if your organization is just starting out or if it's been around for over 30 years, chances are your impact assessment process (if you have one) lacks in consistency of both data and messaging.

While some members of your staff may know the names of every beneficiary you serve, they might not know the Vision or Mission. Challenges like this make the concept of a formidable impact report feel like a possibility only in the distant future. Indeed, the process of establishing the framework, the metrics, the data tracking tools, the waiting, collection of data, analysis of data - all before it's possible to publish your impact report. What a process! However, it's not just work and waiting - you can plunge into the waters of social impact assessment with the assurance that as the process evolves over time, starting week one new elements of your reporting needs will be unearthed as you go. This article will talk you through navigating the limbo world of social impact reporting in the time before the indicators' data is aggregated for analysis. What follows is how to make this period productive and enable you to communicate elements of your evolving impact story as the process unfolds...

What is the Social Impact Assessment?

Typically impact reports cover three areas: (1) Impact Framework; (2) Assessment Results; (3) Performance Evaluation.

1. Impact Framework: An outlined method for social impact assessment that explains:
   • the organization's reasons for measuring impact
   • the documentation of the impact hypothesis (AIM Volume 1: Groundwork)
     • Mission, Vision, Goals
     • Program Design
     • Theory of Change
   • and how they measure impact
     • The metrics used - custom or standard (IRIS, Guidestar, etc.)
     • the tools for forms/surveys and data management (TurboMetrics Impact Cloud; Salesforce)
     • Who they are collecting the data from
     • the data reporting frequency, etc. 
2. Assessment Results
   • data visualization and demonstration for quick insights
3. Performance Evaluation
   • causality and other contextual support narratives

You'll notice, that while the last two are necessary for completed evaluative report, the first offers great insights into the impact hypothesis of an organization. Effectively demonstrating this hypothesis alongside a roadmap for data collection for analysis, is impact communication in its own right. 

Start with Social Impact Measurement Framework

Here's a quick tale of two cities: the first was dreamt up with a clear vision that was well communicated. This allowed for strategic planning. The infrastructure and city development, built from the ground-up, evolved into a beautifully efficient community. The second, a city built organically over time - subject to the whims of a diverse array of mayors and stakeholders, grew into a large and clumpy conglomerate of traffic and inequity.

When it comes to building your impact framework, starting your assessment journey, you can follow the design of the first city - even if your organization is no where near new.  However, it is just as powerful to have a well established impact framework that nods to the organization's history (no matter how inconsistent a story it is), while outlining a definitive impact hypothesis alongside a way forward. You can make the data waiting period a productive one - building out the documentation for your organization's historical context and your impact framework. This clarity in vision will be a bold cornerstone of supporting narrative once the data does roll in for assessment and demonstration. 

Of the three areas of an impact report identified above, one of those sections (Impact Framework) does not require data analysis. Nonetheless, it is a fundamental piece of your impact story. In some cases, the ability to speak to your approach toward impact moving forward can open up unexpected doors for engagement with potential funders. That was the case with one of our asset manager and social businesses (investment). To embark on the journey of developing your own impact framework, take a look at Actionable Impact Management (AIM) a series of guidebooks discovering and documenting your framework.

Communicating Part I: Your Impact Framework

So, you've got your framework in place. Now what? Knowing where to put it starts by understanding your framework's value and why people are interested. By explaining your impact framework you are engaging your stakeholders (donors, beneficiaries, board, staff, public) in a transparent way - looping them in to your practices and processes. It is a credibility-building exercise.

• Website 
  • Methodology Example: Samasource
  • Theory of Change Example: breakthrough; PropelNext
• Social Media : LinkedIn, Facebook, Twitter etc. 
  • Images of your Theory of Change
  • Links to your impact framework webpage
  • your Vision and Mission statements
• In-bound Marketing Approach : Blogs 
  
  • Example: Worldwide Cancer Research

Communicating Part II: Your History

What if you've been around for 50 years? How do you introduce a new way forward while giving due credit to where you've been? As we mentioned in the opening of this article - impact assessment is a RAPIDLY evolving field. Few - if any - can boast a consistent and comprehensive impact measurement strategy over their history. So be a little forgiving and embrace your historical assessments. 

• Website 
  • Story Highlights
  • Timeline Review
  • "Our Story" - the origins of your organization
• Social Media : LinkedIn, Facebook, Twitter etc. 
  • Photos with story highlights
  • Links to your timeline and story highlights webpages
  • Trivia-like messages about your organization's history
• Inbound Marketing Approach : Blogs 
  
  •  The story of your organization's founding
  • Highlighting stories from the archives of beneficiaries, donors, previous programs/activities
  •  

Leveraging Technology

Anything with recurring structures catches the eye of technology enthusiasts as an opportunity. At SoPact we caught on to the recurrence of impact storytelling and its intrinsic coupling with data management. That is how the Impact Cloud was born. The software platform manages the data collection, aggregation, and analysis for impact storytelling. We noticed the importance of demonstrating all of the primary components of impact reports mentioned earlier: (1) Impact Framework; (2) Assessment Results; (3) Performance Evaluation. The recurring structures embedded within each have evolved into layouts of what we call 'impact cards.' In the system, each impact card contains a layout demonstrative of the information it is conveying (from Theory of Change to data assessment results). 

These cards are automatically updated as new data comes in and can be easily downloaded and placed on anything from postcards for donors to websites, emails, reports - and the list goes on. As we move out of Phase One's Impact Framework, and the data does start streaming in, it's helpful to have an enduring process already in place. If you want to check out Impact Cloud, we're always happy to chat. 

There's a common Catch-22 in the social impact assessment or social impact measurement field.

Funders often choose metrics based on the varying capacity of grantees.  Paired with the limited data-capacity faced by the funders themselves, this results in assessments which only capture the low hanging fruit of the data world.

In other words, it translates into the selection of easy-to-measure activity- or output- focused metrics.

In this article, we'll further explore why this tendency is so pervasive, how it works against the goals of a social impact assessment, and what we can do about it.

Whether you find yourself on the funder side, the asset side (social enterprises, etc.), or somewhere in between, by the end of this article you'll better understand how to avoid the primary social impact assessment pitfall, and will be armed with the tools necessary to do so successfully.

Why Your Social Impact Assessment Questionnaire is Failing

A social impact assessment ostensibly tells us how well a program or intervention is performing in terms of achieving impact outcomes.

And yet, our view is that most assessments fail because they too often focus on outputs that have occurred as opposed to getting at outcomes.

Outputs are, of course, easily measured. Your questionnaire for program managers or beneficiary liaisons might include questions like: How many hours of training were delivered; How many people were served; How many products are being used by beneficiaries. These imply impact, but do not demonstrate it.

And these metrics hold little strategic value for the grantees, especially if they are only measured for a short period.

And so, a stigma has evolved around the trustworthiness of the data received as well as how helpful what we're calling 'measurement and evaluation' really is to discern the impactful results of our interventions. With such little regard for the practice, there is little incentive to invest in improving the process, and so, here we are in a Catch-22 of activity-driven (i.e. output-driven) measurement. 

Read More - Is collective impact evaluation the future of social impact assessment?

Is Outcome-Driven Social Impact Assessment Worth the Effort?

Something wonderful happens when an organization decides to have an outcome-driven social impact measurement process. They begin to reassess their impact framework more critically than ever before. That's because, to build an enduring practice for impact measurement, an organization must have a well-developed impact foundation to guide it through the metrics selection. This starts with their Vision statement and moves through their Theory of Change.

What we've found is that some of our clients were missing elements of this impact framework. The ability (for the first time) to measure for results facilitates an internal conversation around the broader impact hypothesis, and the organization is left with more established impact messaging.

The benefits of results-driven evaluation go beyond the structuring of an organization's impact framework. It includes the strategic insights gained from a better understanding of the outcomes of your interventions. Armed with such insights, your organization is better able to iterate the process. And when built upon a solid foundation that includes a thoughtful Theory of Change, an organization is more aware of its assumptions and areas of potential risk. With an outcome-oriented approach, measurement and evaluation become a pivotal element of your organization's strategic decisions. With the ability to more confidently speak to the specific results of your interventions, your marketing and donor relations will also benefit. 

Read More - The potential social and economic returns of collective impact

Improved Social Impact Measurement Processes

One phrase that pops up again and again in the unfortunate cycle is "data capacity". If we might be able to enhance our own capacity and that of our partners, perhaps we will be able to take on more challenging metrics that offer more strategic insights. Once the value of the data becomes apparent, its positive impact on programming, marketing, donor relations, and grant seeking lead to better staff engagement with the data reporting. Especially if reporting becomes more user-friendly.

Impact Cloud™ was designed to be accessible and thorough, flexible enough to meet the data needs of various socially minded organizations. And it starts with a well-developed impact framework. Actionable Impact Management begins with the introspective work of defining a Vision, Mission, and Goals; Program Structure; and Theory of Change before moving on to the selection of metrics and your data strategy. 

 The transformation from activity or output metrics to outcome metrics can mean all the difference in buy-in. By strengthening data capacity, funders can make that transformation a reality.

Read More: How can we improve social impact accounting?

Social Impact Evaluation

Social impact For Different Organizations

With better data, we can direct capital to grow the social enterprises that are making the most difference.

We will go over commonly used social impact approaches today and point out SoPact’s focus area and approach to improve a better outcome.

When social impact evaluators communicate findings to funders, there is a fundamental lack of common approach. Let’s start by understanding the four most common approaches in this domain from bottom-up.

Compare Social Impact Measurement Methods Randomized Control Trial at the bottom is primarily used by impact researchers who lead evidence-based research approaches.  Though these methods can be justified in many long-term studies, often they are useful in proving or disproving – if treatment is effective for a selected program. Unfortunately, this approach can be costly, often error-prone and time-consuming. More importantly, they do not provide immediate feedback.

Program Outcome Improvement

is more useful when the beneficiary group requires agile feedback and regular interventions such as disability employment, HIV/AIDS outcome, Patient outcome, etc. Unlike the survey-based approach, these systems manage the beneficiary lifecycle and manage program outcomes.

Funder Impact Measurement

Often this area is looked as a traditional M & E from a funder perspective.  The challenge with current approaches is that there is a limited agreement between funder and investee/grantee on measurement language and limited sharing reduces impact envisioned by funders. Later in the section, I will demonstrate SoPact’s collaborative approach allows to create shared measurement – improving a long-lasting effect on intervention.

Standards-Based Benchmarking

Social impact measurement practices can be seen as a pyramid, where B-Analytics/GIIN works with IRIS 3.0 at the top.  Though they are leading impact measurement services for social impact funds, their approach has major transparency issues.  For Example- IRIS does not include any metrics about immunization rates - or even about health. One of the many IRIS Partner Metrics systems being developed with the Center for Health Market Interventions does include health metrics,….. one of which is “Health Intervention Completion Rate”  But imagine being the manager of this impressive immunization program and having to enter 33% in an investor form that included that metric - it just doesn’t capture the impact that you have been working so hard for.

Stakeholder Impact Analysis is thoughts, beliefs, needs, feedback, etc. communicated by individuals defined as stakeholders for any given impact area. 

Principally, social sector stakeholders are the target beneficiaries of an intervention, although in general terms a stakeholder is any individual or entity that stands to be affected (positively or negatively) by the activities undertaken by an organization. The following content focuses on those beneficiary stakeholders who are essential to any impact-creating ambitions.

The success of an impact-driven initiative is nearly always bound by the extent to which those that are designing and implementing the initiative take into account the perspective of key stakeholders, especially those beneficiaries who are the reason for the initiative. Here are a few reasons why raising those voices is so important:

• Program Design: Before execution of an intervention there must be a design process which ensures there is a need for the activity, and that it is honed to the needs, resources, schedules, etc. of beneficiaries. In short, it must pass the, “Is it feasible?” and, “Will it be effective?” tests. While success is never a certainty, involving beneficiary stakeholders in the design process can ensure a greater probability of positive outcomes. Because local knowledge will often reveal contextual nuances which will be key in the creation of a program or other intervention.
• Managing Risk: Understanding how local beneficiaries live, the particularities of their culture, and what difficulties they encounter in their day-to-day not only serves to improve program design, but also serves to illuminate potential negative externalities an intervention could cause or exacerbate. Defining those possibilities early on and planning for any eventuality can help ensure they do not come to pass or, if they do, ensure that those effects can be mitigated. Demonstrating commitment to this area can also help build credibility with local communities.
• Assessing Impact: Understanding whether impact is indeed being created is almost impossible without taking into account what stakeholders are telling you. Sure, you can measure how many hours of activities you performed or how many products were delivered, but without bringing beneficiaries into the impact conversation, it will be difficult to determine the extent of the impact created (positive or otherwise).
• Relationship Building: Rarely do we create an intervention, implement it with our target community, and then leave. Because driving impact is usually a long-term endeavor, it is essential that long-term relationship-building efforts are included in the project scope. Buy-in from local leaders and communities as a whole will also increase the chance of program success, while also simply being the caring, human thing to do!

Natural Language Processing has come a long way in recent years and is now being applied to capture impact insights from beneficiaries.

Qualitative analysis, for example asking beneficiaries questions and recording their long-form responses for later analysis (using surveys, interviews, etc.), is time-intensive and not quite a hard science. And yet, stories can be a powerful tool to understand the depths of impacts perceived by a beneficiary group. The key is to be able to transform those narratives into quantitative impact evidence. This is where the marriage between artificial intelligence and natural language processing comes into play.

Sopact’s Impact Cloud, an end-to-end cloud-based impact data management tool, gives users the capacity to harness such technology.

Uploading qualitative responses to the tool enables impact practitioners to easily conduct keyword sentiment and emotion analysis and transform qualitative data into quantitative. 

In this way, from impact narratives full of stakeholder sentiment practitioners can glean important impact insights.

Shared insights are those impact learnings which are determined in collaboration with target beneficiaries. These insights go beyond traditional outputs such as number of beneficiaries affected to revealing how those beneficiaries were affected, taking into account their own perceptions of impacts generated.

Shared insight can also refer to the open-sourcing of impact data and impact learnings across impact organizations and across sectors. Taking a more collaborative approach to how we wield impact data can ultimately help impact initiatives globally, if impact players share analyses, insights, and outcomes achieved. If you are interested in such collaboration, take a look at the Fund for Shared Insight.

As with any activity your organization performs, it should be done with a clear strategic purpose, not just because it seems like something you should do. With that strategic perspective in mind, here are a few ways to make sure you leverage those interactions with constituents.

• Keep it periodic, but not constant. 

  For most beneficiaries, you don’t want to completely invade their life with requests for data. Doing so can lead to overwhelm and have an adverse affect on their willingness to be a part of that process. Ask yourself, given the impact outcomes we are targeting and our program design, when are key moments we should be “checking in” to get feedback from our beneficiaries? Of course, if it is unclear, ask the beneficiaries themselves!

• Have an analysis plan in place.

  You conduct dozens of interviews and even more online surveys. Great, but what are you going to do with those data? Do you have the expertise to analyze it properly? The time? Program design shouldn’t just be about creating an effective intervention, it should also include how you will hold yourself accountable to making that intervention an effective one. Because if you are ultimately unable to utilize constituent feedback to assess and improve, you run the risk of wasting not just your time and energy, but that of your beneficiaries as well.

• Tell stakeholders how it helped you!

  A great way to achieve buy-in from stakeholders in terms of their willingness to be a part of the data collective process is to explain clearly how such data will be used. And most importantly, follow up with those communities after analyses have been conducted to show how constituent voice contributed to understanding the impacts created and potential design improvements moving forward.

A stakeholder impact analysis can be conducted at a variety of times throughout the life of an intervention. At the outset, such an analysis serves to support the feasibility testing and initial design. During and post-intervention these analyses serve to assess impacts created and eventually how to improve the intervention design.

According to the Impact Management Project, there are five key dimensions of impact which should be taken into account in any stakeholder-centered analysis (seen below).

Source: Impact Management Project

It is important to distinguish between the two main types of analyses available to impact practitioners. A quantitative analysis takes raw numbers and calculates outputs, determines probabilities or percentages, and enables practitioners to make inferences based on those results. A qualitative analysis primarily takes data from subjective self-reporting of stakeholders and uses various methods to extract insights.

A breakdown of the differing data types can be seen below, taken from a recent report on using self-reported data:

Source: Using Self-reported Data

A quantitative approach is best used when there is specific, precise, and easily obtainable information that can be collected from a stakeholder group. For analyses that seek to compare between time periods (for example, before and after), or between groups (for example, an experimental and control), quantitative approaches are often the most useful. In addition, in evaluating outputs and frequency of activities, a quantitative approach is usually the most feasible.

A qualitative evaluation assesses contextual factors, usually from the subjective view of primary stakeholders themselves. Qualitative approaches take into account how such individuals evaluate their own experiences and potential impacts perceived. This approach is best used when the objective of the evaluation is to assess needs, desires, and expectations of a group, as well as how they feel about impacts generated.

While there is no one way to conduct a stakeholder analysis, there are, of course, best practices to be followed and common tools which can be used to get the most out of the time invested.

For a useful introduction to the processes involved in raising stakeholder voice and how to do so effectively, we recommend reviewing “Using IRIS+ to Incorporate Stakeholder Voice,” a report published by the Global Impact Investing Network (which takes the investor engagement perspective). But it will also help general impact practitioners in any role, especially those wishing to dive further into the world of standardized metrics (IRIS+), which can be essential in various types of analyses of stakeholders as well.

Here are a few of the key takeaways of the report to get you started effectively incorporating stakeholder voice:

• Transparency: Be clear about why you are doing things the way you are doing them (or the way you intend to) and ask stakeholders for feedback in a way that is authentic -- in other words, build trust by actually using that feedback and reporting back. 
• There is no one-size-fits-all approach: Context. Context. Context. Be aware of just how much beneficiaries are willing and able to be a part of your analyses. Take those contextual cues directly from them and plan your engagement accordingly.
• Embrace an open-source mentality. The learnings you glean from those hard-earned moments with stakeholders, and subsequent analyses, may help other organizations make better decisions from the outset of an initiative and enable the social sector at large to operate more effectively and efficiently.

If you are looking to dive deeper into the topic for your own research or are looking for expert consulting or to explore collaboration opportunities, the following organizations maintain stakeholder voice as a key principle of their work.

Essential to raising stakeholder voice is making sure that in addition to opening up that conversation with them, organizations also analyze those data and put insights to work to create better interventions and ultimately better outcomes. That process is exactly what the non-profit organization Feedback Labs seeks to foster across the social sector.

Founded in 2013, Feedback Labs works to create standardized metrics which assess how organizations are doing in conducting feedback-related work. They also promote the development of tools, trainings, and events to further strengthen the field.

Social Value International brings together a global community (45 countries) of impact practitioners to promote better practices in social value measurement and analysis and to influence policy. Its driving goal is to improve the way we account for social value. As part of their knowledge-sharing approach, they developed seven guiding principles of social value, the first of which is defined as “Involving Stakeholders.”

This as the first principle underscores the importance of stakeholder engagement in the social value measurement process. As discussed above, it is essential that primary stakeholders are involved in, or at the very least informed about, what is getting measured and how.

The Keystone team are experts in everything to do with managing feedback and getting the most out ofthose interactions with constituents. In addition to consulting services, they offer analyses of your existing measurement systems, diversity programs, and more. Their Constituent Voice tool can help organizations get the most out of those feedback data and the Feedback Commons provides tutorials to help practitioners improve how they manage those processes. You can find their full page of resources and guides here.

A social enterprise created by Acumen and based in its Lean Data approach, 60 Decibels aims to both improve adoption of Lean Data approaches across sectors and also introduce what they hope will be the new standard for impact measurement across sectors as well. This standard puts stakeholder voice at the forefront of the measurement process, emphasizing the importance of a beneficiary’s “lived experience.” The enterprise seeks to solve impact measurement issues (comparability, for example) faced by the impact investing sector and to ultimately improve outcomes through better measurement that is stakeholder centric.

• Impact Manager
  
  For impact investors, grantmakers, social accelerators/incubators, Development Finance, international development & public agencies, this powerful cloud-based tool is not just an impact data management resource. It could also be called a comprehensive stakeholder management tool because of its numerous integrations and accessible cloud-based interface which allows for seamless integrations across teams, no matter where they are or how technical (or not) their training.

• 3 Reasons Why Listening to Every Beneficiary Sustains Social Impact Investment

• Using self-reported data for impact measurement (report)
• Impact Management Project
• IRIS+ to Incorporate Stakeholder Voice
• The Case for Stakeholder Engagement 
• Engaging All Affected Stakeholders: Guidance for investors, funders, and organizations (World Economic Forum)
• Listening to Those Who Matter Most, the Beneficiaries
• Can Stakeholder Reviews by Beneficiaries Bring New Perspectives to Philanthropy?
• The Surprising Truth Behind Beneficiary Feedback
• Systems Change Should Lift Up Beneficiary Voices
• Listening to the Voice of the Beneficiary