<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=255921171522003&amp;ev=PageView&amp;noscript=1">

When Impact Metrics Fail (And What to do About it)

Perspectives on Metrics | impact data | Metrics | monitoring & evaluation

Reading time: 7 minutes

Alan is a social sector consultant and one of the founding directors of Quantica Education, a school of social entrepreneurship in Colombia. Alan has published more than 100 blogs and articles, managed global research projects, & facilitated dozens of workshops for entrepreneurs in Latin and South America.

‹ Back to the listing

Posted on 2018-06-26

The online depth of opinion articles and how-to pieces about impact metrics and measuring social impact grows with each passing month. We need to measure impact! Nobody in the impact sector dares to doubt this call to action. And for good reason -- it is essential to track how we’re doing. Yet, we mustn’t measure simply for measuring’s sake.

Superficial impact metrics might be cost effective and make us feel good about the “impact” we seem to be generating. But they do a disservice to the impact sector as a whole, and most importantly, to those stakeholders passionate about the impact we are seeking to create.

What are impact metrics?

It may seem trivial, but it’s worth taking a look at how we define impact metrics to understand how we often fall short in establishing the right ones for our programs.

Impact metrics form a defined system or standard of measurement to track progress of change by your organization. In the impact space there are standard metrics and custom metrics. Standards are written by research and evaluation organizations and generally exist around focus areas or organization type, for example IRIS Metrics Catalogue. Custom metrics are created by an organization and are designed around their use case.

The key phrase in this definition is “progress of change.” Impact metrics shouldn’t just tell us what is happening in a certain moment. They need to be leveraged to understand how much change has occurred over a period of time because of our program activities. Change that wouldn’t have happened otherwise.

Mistaking impact metrics for change

Put simply, social impact metrics fail when we mistake the metrics themselves for the change we are seeking to create. Let’s take a look at a famous example from the social impact sector, PlayPump. The idea was apparently genius. A merry-go-round apparatus which pumped underground water the more it spun around, the more local children played on it.impact metrics playpump

Hailed as an ingenious solution to the lack of access to quality drinking water, it received millions in backing and enjoyed public praise from leaders around the globe. That praise was short lived. The failures of the PlayPump have been well-documented (what if there aren’t enough children to play or they don’t want to?) – as have the organization’s laudable efforts to learn from their mistakes.

Let’s look at the possible impact metrics which could have led to the continued implementation of the PlayPump even if it was not improving the lives of its targeted marginalized communities.

Disclaimer: This is a hypothetical break-down of metrics and is not meant to imply that PlayPump used these metrics and/or failed because of it.

Metric 1: # of communities where PlayPumps are installed

Metric 2: % of children using PlayPumps

Metric 3: Liters of water pumped

At first glance, these seem like positive things to track the impactful progress of the PlayPump in the rural communities where it aimed to do good. In fact, PlayPump should probably track all of these metrics. But they do not communicate impact.

They are outputs which communicate implementation of a product, not whether that product is having a positive effect on the lives of the target beneficiaries.

However enticing it may be to say that 500 pumps have been installed and a million liters of water pumped, as social impact metrics, these three would fail.

True Social Impact Metrics

Outcome metrics more accurately tell us whether we are having an impact or not as they describe the intended medium term consequences of a program. They are the second level of results associated with a project (after outputs like those mentioned above) and more closely relate to the project goal or aim. Here are some impact metrics which might be more relevant to the social impact PlayPump seeks to generate:

Outcome metric #1: Water-borne illness rate

Outcome metric #2: School attendance rate

Outcome metric #3: Mortality rate

These metrics hone in on the change that is potentially occurring in the communities where PlayPump has a presence and might be more readily attributed to the introduction of that pump. Furthermore, they can (and should be) measured against a baseline in order to understand the comparative value of the pump over a period of time.

Getting Metrics Right the First Time

It’s worth reiterating that the initial failures of PlayPump were more design based (they could have benefited from deeper engagement with beneficiaries from the outset), but the point here is that if they had focused on measuring the desired outcomes of the project, they might have discovered sooner that the pumps were not having the desired positive effect on the lives of the beneficiaries. A course correction at an earlier stage might have been possible.

So how might those of us in the impact sector establish the right metrics from the outset? What we can do first is go through a step-by-step process to clearly connect our mission, vision, and program structure to a set of metrics relevant to a certain program.

And in the process, we should clearly demarcate those metrics which are outputs vs. outcomes (a Theory of Change model would help). By holding ourselves accountable to the various kinds of metrics, we can better understand where our true impact lies.

For a primer on how to ensure that your impact metrics are truly serving you and your beneficiaries, check out Volumes I & II of our Actionable Impact Management guides!

Learn More About Metrics