The online depth of opinion articles and how-to pieces about impact metrics and measuring social impact grows with each passing month. We need to measure impact! Nobody in the impact sector dares to doubt this call to action. And for good reason -- it is essential to track how we’re doing. Yet, we mustn’t measure simply for measuring.
Superficial impact metrics or simply copying and pasting IRIS metrics over your activities might be cost effective and make us feel good about the “impact” we seem to be generating. But they do a disservice to the impact sector as a whole, and most importantly, to those stakeholders passionate about the impact we are seeking to create.
Pitfalls of Standards Like the IRIS Indicators
It may seem trivial, but it’s worth taking a look at how we define impact metrics to understand how we often fall short in establishing the right ones for our programs.
Impact metrics form a defined system or standard of measurement to track progress of change by your organization. In the impact space there are standard metrics and custom metrics. Standards are written by research and evaluation organizations and generally exist around focus areas or organization type.
For example, the IRIS Metrics Catalogue is a vast database of metrics used most often by impact investors and their investees. A pitfall of impact investors' frequent use of these metrics and demanding that investees do the same is that those metrics will not always provide a relevant assessment of the outcomes sought by that change-making organization.
Sometimes, custom metrics -- created by an organization and designed around their use case -- are necessary.
Mistaking impact metrics for change
Put simply, social impact metrics fail when we mistake the metrics themselves for the change we are seeking to create. Let’s take a look at a famous example from the social impact sector, PlayPump. The idea was apparently genius. A merry-go-round apparatus which pumped underground water the more it spun around, the more local children played on it.
Hailed as an ingenious solution to the lack of access to quality drinking water, it received millions in backing and enjoyed public praise from leaders around the globe. That praise was short lived. The failures of the PlayPump have been well-documented (what if there aren’t enough children to play or they don’t want to?) – as have the organization’s laudable efforts to learn from their mistakes.
Let’s look at the possible impact metrics which could have led to the continued implementation of the PlayPump even if it was not improving the lives of its targeted marginalized communities.
Disclaimer: This is a hypothetical break-down of metrics and is not meant to imply that PlayPump used these metrics and/or failed because of it.
Metric 1: # of communities where PlayPumps are installed
Metric 2: % of children using PlayPumps
Metric 3: Liters of water pumped
At first glance, these seem like positive things to track the impactful progress of the PlayPump in the rural communities where it aimed to do good. In fact, PlayPump should probably track all of these metrics. But they do not communicate impact.
They are outputs which communicate implementation of a product, not whether that product is having a positive effect on the lives of the target beneficiaries.
However enticing it may be to say that 500 pumps have been installed and a million liters of water pumped, as social impact metrics, these three would fail.
Many IRIS metrics fall into this same pitfall, encouraging us to measure indicators, while overlooking ways of measuring which can help us assess "progress of change".
Outcome metrics more accurately tell us whether we are having an impact or not as they describe the intended medium term consequences of a program. They are the second level of results associated with a project (after outputs like those mentioned above) and more closely relate to the project goal or aim. Here are some impact metrics which might be more relevant to the social impact PlayPump seeks to generate:
Outcome metric #1: Water-borne illness rate
Outcome metric #2: School attendance rate
Outcome metric #3: Mortality rate
These metrics hone in on the change that is potentially occurring in the communities where PlayPump has a presence and might be more readily attributed to the introduction of that pump. Furthermore, they can (and should be) measured against a baseline in order to understand the comparative value of the pump over a period of time.
Getting it Right: Start with Theory Of Change
It’s worth reiterating that the initial failures of PlayPump were more design based (they could have benefited from deeper engagement with beneficiaries from the outset), but the point here is that if they had focused on measuring the desired outcomes of the project, they might have discovered sooner that the pumps were not having the desired positive effect on the lives of the beneficiaries. A course correction at an earlier stage might have been possible.
So how might those of us in the impact sector establish the right metrics from the outset? What we can do first is go through a step-by-step process to clearly connect our mission, vision, and program structure to a set of metrics relevant to a certain program.
And in the process, we should clearly demarcate those metrics which are outputs vs. outcomes (a Theory of Change model would help). By holding ourselves accountable to the various kinds of metrics, we can better understand where our true impact lies.
Integrating IRIS Indicators Isn't Always the Answer
If approached by a funder demanding we use IRIS metrics, an effective use of IRIS indicators must still include assessments using the outcome-oriented approach.
With the above discussion in mind we can negotiate how to either combine IRIS indicators which custom metrics/other standard metrics, or define our own social impact metrics framework.