The online depth of opinion articles and how-to pieces about impact metrics and measuring social impact grows with each passing month. We need to measure impact! Nobody in the impact sector dares to doubt this call to action. And for good reason -- it is essential to track how we’re doing. Yet, we mustn’t measure simply for measuring’s sake.
Superficial impact metrics might be cost effective and make us feel good about the “impact” we seem to be generating. But they do a disservice to the impact sector as a whole, and most importantly, to those stakeholders passionate about the impact we are seeking to create.
What are impact metrics?
It may seem trivial, but it’s worth taking a look at how we define impact metrics to understand how we often fall short in establishing the right ones for our programs.
Impact metrics form a defined system or standard of measurement to track progress of change by your organization. In the impact space there are standard metrics and custom metrics. Standards are written by research and evaluation organizations and generally exist around focus areas or organization type, for example IRIS Metrics Catalogue. Custom metrics are created by an organization and are designed around their use case.
The key phrase in this definition is “progress of change.” Impact metrics shouldn’t just tell us what is happening in a certain moment. They need to be leveraged to understand how much change has occurred over a period of time because of our program activities. Change that wouldn’t have happened otherwise.
Mistaking impact metrics for change
Put simply, social impact metrics fail when we mistake the metrics themselves for the change we are seeking to create. Let’s take a look at a famous example from the social impact sector, PlayPump. The idea was apparently genius. A merry-go-round apparatus which pumped underground water the more it spun around, the more local children played on it.
Hailed as an ingenious solution to the lack of access to quality drinking water, it received millions in backing and enjoyed public praise from leaders around the globe. That praise was short lived. The failures of the PlayPump have been well-documented (what if there aren’t enough children to play or they don’t want to?) – as have the organization’s laudable efforts to learn from their mistakes.
Let’s look at the possible impact metrics which could have led to the continued implementation of the PlayPump even if it was not improving the lives of its targeted marginalized communities.
Disclaimer: This is a hypothetical break-down of metrics and is not meant to imply that PlayPump used these metrics and/or failed because of it.
Metric 1: # of communities where PlayPumps are installed
Metric 2: % of children using PlayPumps
Metric 3: Liters of water pumped
At first glance, these seem like positive things to track the impactful progress of the PlayPump in the rural communities where it aimed to do good. In fact, PlayPump should probably track all of these metrics. But they do not communicate impact.
They are outputs which communicate implementation of a product, not whether that product is having a positive effect on the lives of the target beneficiaries.
However enticing it may be to say that 500 pumps have been installed and a million liters of water pumped, as social impact metrics, these three would fail.
True Social Impact Metrics
Outcome metrics more accurately tell us whether we are having an impact or not as they describe the intended medium term consequences of a program. They are the second level of results associated with a project (after outputs like those mentioned above) and more closely relate to the project goal or aim. Here are some impact metrics which might be more relevant to the social impact PlayPump seeks to generate:
Outcome metric #1: Water-borne illness rate
Outcome metric #2: School attendance rate
Outcome metric #3: Mortality rate
These metrics hone in on the change that is potentially occurring in the communities where PlayPump has a presence and might be more readily attributed to the introduction of that pump. Furthermore, they can (and should be) measured against a baseline in order to understand the comparative value of the pump over a period of time.
Getting Metrics Right the First Time
It’s worth reiterating that the initial failures of PlayPump were more design based (they could have benefited from deeper engagement with beneficiaries from the outset), but the point here is that if they had focused on measuring the desired outcomes of the project, they might have discovered sooner that the pumps were not having the desired positive effect on the lives of the beneficiaries. A course correction at an earlier stage might have been possible.
So how might those of us in the impact sector establish the right metrics from the outset? What we can do first is go through a step-by-step process to clearly connect our mission, vision, and program structure to a set of metrics relevant to a certain program.
And in the process, we should clearly demarcate those metrics which are outputs vs. outcomes (a Theory of Change model would help). By holding ourselves accountable to the various kinds of metrics, we can better understand where our true impact lies.
For a primer on how to ensure that your impact metrics are truly serving you and your beneficiaries, check out Volumes I & II of our Actionable Impact Management guides!