<img height="1" width="1" style="display:none" src="https://q.quora.com/_/ad/0609320ad13c4183af7978b1c1744b71/pixel?tag=ViewContent&amp;noscript=1">

Common Pitfalls When Using IRIS Indicators (And How to Avoid Them)

Posted by Alan Pierce on Jun 25, 2018 11:21:33 PM

Superficial impact metrics  or simply copying and pasting IRIS metrics over your activities might be cost effective and make you feel good about the “impact” you seem to be generating.

But such "shortcut" approaches do a disservice to the impact sector as a whole, and most importantly, to those stakeholders passionate about the impact we are seeking to create.

In this blog, we'll share common mistakes of IRIS Indicator adoption and how you can avoid them, so that you can use the IRIS system to amplify your impact, instead of selling it short.

With this overview you'll be able to take a hard look at your own IRIS metrics, or go about defining them in a more efficient and effective way, preventing your organization from falling into common IRIS indicator implementation mistakes.

Read More: The new way of choosing Impact Metrics beyond IRIS metrics

 

Pitfall 1: Misalignment of outcome goals

It may seem trivial, but it’s worth taking a look at how we define impact metrics to understand how we often fall short in establishing the right ones for our programs.

Impact metrics form a defined system or standard of measurement to track progress of change by your organization. In the impact space there are standard metrics and custom metrics. Standards are written by research and evaluation organizations and generally exist around focus areas or organization type.

For example, the IRIS Metrics Catalogue is a vast database of metrics used most often by impact investors and their investees.

The first common pitfall in the social sector regarding IRIS indicators is that investors frequently demand use of these metrics in their investees, even if the indicators do not provide a relevant assessment of the outcomes sought by that change-making organization.

Sometimes, custom metrics -- created by an organization and designed around their use case -- are necessary.

Read More: Pros and Cons of Standardized Social Impact Metrics

 

Pitfall 2: Mistaking IRIS Indicators for change

Put simply, social impact metrics fail when we mistake the metrics themselves for the change we are seeking to create. Let’s take a look at a famous example from the social impact sector, PlayPump. The idea was apparently genius. A merry-go-round apparatus which pumped underground water the more it spun around, the more local children played on it.playpump example to demonstrate using IRIS Metrics to measure impact

Hailed as an ingenious solution to the lack of access to quality drinking water, it received millions in backing and enjoyed public praise from leaders around the globe. That praise was short lived. The failures of the PlayPump have been well-documented (what if there aren’t enough children to play or they don’t want to?) – as have the organization’s laudable efforts to learn from their mistakes.

Let’s look at the possible indicators which could have led to the continued implementation of the PlayPump even if it was not improving the lives of its targeted marginalized communities.

Disclaimer: This is a hypothetical break-down of metrics and is not meant to imply that PlayPump used these metrics and/or failed because of it.

Metric 1: # of communities where PlayPumps are installed

Metric 2: % of children using PlayPumps

Metric 3: Liters of water pumped

At first glance, these seem like positive things to track the impactful progress of the PlayPump in the rural communities where it aimed to do good. In fact, PlayPump should probably track all of these metrics. But they do not communicate impact.

They are outputs which communicate implementation of a product, not whether that product is having a positive effect on the lives of the target beneficiaries.

However enticing it may be to say that 500 pumps have been installed and a million liters of water pumped, as social impact metrics, these three would fail.

Many IRIS metrics fall into this same pitfall, encouraging us to measure indicators, while overlooking ways of measuring which can help us assess "progress of change". 

 Read More: Democratizing Impact Metrics for Bringing Collective Impact

Outcome Metrics

Outcome metrics more accurately tell us whether we are having an impact or not as they describe the intended medium term consequences of a program. They are the second level of results associated with a project (after outputs like those mentioned above) and more closely relate to the project goal or aim. Here are some impact metrics which might be more relevant to the social impact PlayPump seeks to generate:

Outcome metric #1: Water-borne illness rate

Outcome metric #2: School attendance rate

Outcome metric #3: Mortality rate

These metrics hone in on the change that is potentially occurring in the communities where PlayPump has a presence and might be more readily attributed to the introduction of that pump. Furthermore, they can (and should be) measured against a baseline in order to understand the comparative value of the pump over a period of time.

Read More: Choosing Impact Metrics for Gender Lens Investing

Getting Started with IRIS: Begin with Theory Of Change

It’s worth reiterating that the initial failures of PlayPump were more design based (they could have benefited from deeper engagement with beneficiaries from the outset), but the point here is that if they had focused on measuring the desired outcomes of the project, they might have discovered sooner that the pumps were not having the desired positive effect on the lives of the beneficiaries. A course correction at an earlier stage might have been possible.

Read More: How to choose right impact indicators to demonstrate impact?

So how might those of us in the impact sector establish the right metrics from the outset? What we can do first is go through a step-by-step process to clearly connect our mission, vision, and program structure to a set of metrics relevant to a certain program.

And in the process, we should clearly demarcate those metrics which are outputs vs. outcomes (a Theory of Change model would help). By holding ourselves accountable to the various kinds of metrics, we can better understand where our true impact lies.

Finding the right balance with IRIS Indicators 

If approached by a funder demanding we use IRIS metrics, an effective use of IRIS indicators must still include assessments using the outcome-oriented approach.

With the above discussion in mind we can negotiate how to either combine IRIS indicators which custom metrics/other standard metrics, or define our own social impact metrics framework.

For a primer on how to ensure that your impact metrics are truly serving you and your beneficiaries, check out Volumes I & II of our Actionable Impact Management guides!

Learn More About Metrics

 

Topics: monitoring & evaluation, impact metrics

Alan Pierce

Written by Alan Pierce

Alan is a social sector consultant and one of the founding directors of Quantica Education, a school of social entrepreneurship in Colombia.