Monitoring and Evaluation Tools in 2020

• Is your monitoring and evaluation system working for you? 

• Are you able to provide impact reports or grant reports to your funders on time?  

• Do you have data sitting across multiple systems that increase your time to build impact reports? 

The  focus on activity based reporting is doing more harm than good. While activity and output reporting are necessary, by no means this can be end goal. Mission driven organization must demonstrate if they are meeting their mission. Only way to demonstrate is to continuously learn from stakeholders, keeping theory of change throughout life-cycle, detect intervention from data driven approach that changes curve and improve outcome.

Monitoring and Evaluation Today

Challanges

While there are many tools available for monitoring and evaluation, data collection, and reporting, without a well-designed impact strategy, funder reporting often becomes chaos. Let us look at the challenges of current systems and discuss potential solutions. If you are a nonprofit or social enterprise struggling to compete against other grantees or investees to raise much-needed grants or impact capital, you’re at the right place!

Today, we’re diving into the biggest challenges of collecting data and impact reports. Our goal is to turn this challenge into the most critical opportunity that most organizations ignore - raising grants or impact capital. While many nonprofits and social enterprises collect data, most of it isn't utilized, let alone actionable. A well-designed data strategy can yield proper accountability, making it easier for continuous learning, improvement, and reporting specific to each funder. Keep in mind, most monitoring and evaluation systems fail to provide scalable solutions to aggregate results regularly. If they do, it may require significant customization and are expensive.

Inconsistent Data Collection

Mission-driven organizations often grow data collection and reporting organically as their programs and funders grow. In most organizations, data collection and reporting evolves in ad hoc fashion, difficult to streamline funder specific reporting. Hence, most of them manually export data on Excel or google sheet, clean up data, and prepare unique funder reports. However, there are several challenges with this approach. 

• Lack of consistency in reporting due to multiple funder requirements.
• Data collection is inconsistent and not aligned to funders
• Data cleaning and aggregation for reporting is time-consuming
• Relying on external data analytics tools like Tableau and Power BI can increase the time significantly and also require specialized skills.
• Often theory of change, data collection, and reporting are not aligned, which makes continuous learning and improvement a pipe dream.

Activity & Output as Impact Reporting?

A successful grant and impact capital depends on the demonstration of impact reporting. But, effective reporting requires demonstration of impact evidence or what is changing? Often today’s monitoring and evaluation systems focus on activity and output based reporting and not real change.

Many organizations start with a long household survey; often, this survey fails to collect key outcomes aligned with the theory of change. We need a system that focuses on core aspects of impact management and outcomes aligned towards program goals. A theory of change driven and outcome-aligned stakeholder survey with well balanced qualitative and quantitative stakeholder feedback can achieve desired impact management.

Impact Data Collection, Cleaning & Aggregation Costs $$$

As an organization adds new programs, their reporting system becomes challenging and isolated. For example, for homelessness, a government funder may insist on using the HMIS system, social services organizations may utilize case management systems, customize Salesforce-based program management, or Effort To Outcome system. While these systems might manage program data, often, their reporting tends to relatively limited and often does not align well with funders reporting requirements. Hence, an organization that often hire monitoring and evaluation analyst who usually export data from different systems, and manually cleanse data with complex, tedious, and often error-prone spreadsheet. This process will keep an organization from good insight for continuous learning and improvement required for impact management.

Low-Quality Impact Data Collection  

A large number of organizations still today collect results on excel or google spreadsheet. Many in the field often receive results in the form of paper and later move to excel, google spreadsheets, or proprietary systems. Low-quality impact data and errors bring no insight needed for an organization to change the program direction. It costs more in the long term by losing an opportunity cost as well as resource bleeding. 

Mobile Data Collection 

While there are hundreds of online and offline data collection tools, their built-in data analytics gives a summary at the question or field level. The vital analysis requires an organization to compare multiple variations such as villages, racial, gender, performance over a period of time. 

Excel or Google Spreadsheet

As different programs may use more than one system, most organizations try to bring data into MS-Excel or Google Docs. They carefully reformat data regularly so that they can calculate the results required by funders.

• Data gets unmanageable
• Data integrity not preserved
• Recreate formula every time with new data collection
• Staff turnover can be vulnerable 

Salesforce & Similar Program Data Management Tools

Many systems such as Salesforce, Case Management, or Program data may be collecting data; often, building reports may be challenging. Because it is,

• Challenging to combine data from multiple tables
• Report building is too complex
• Reporting for all agreed-upon metrics come from different external sources
• Requires frequent reporting such as monthly, quarterly and annually

While systems such as Salesforce can be robust, it may be challenging to summarize results. Often funders may require qualitative reporting that may not be part of the Salesforce like systems. It could increase the reporting time. 

When building a theory of change model, much of the effort involved focuses on identifying activities, outputs, outcomes, etc. that will hopefully deliver the impact you seek to create. Outcome tracking and impact management are critical; not all monitoring and evaluation software are equal.

But how do you plan on managing all those data you’ll need to make that theory of change come alive through proper monitoring and evaluation software?

Unfortunately, for many impact-driven organizations, the tech know-how that is necessary for proper impact data management is scarce. And as a result, we default to traditional Excel-based processes (which can turn ugly real quick).

So let’s take a look at selected few software tools that are trending in the monitoring and evaluation space. Beware of M &E branding, as it is essential to define focus on core requirements and value.

Based on detailed research, following review provides an in-depth and easy to use guide.

Program Evaluation:

Determines whether program activities have been implemented as intended and resulted in certain outputs. You may conduct process evaluation periodically throughout the life of your program and start by reviewing the activities and output components of the theory of change or logic model.

The common approach to collect program evaluation is based on key metrics defined for the output metrics. Output data can be collected in a broad range of systems.

Outcome Evaluation:

measures program effects in the target population by assessing the progress in the outcomes that the program is to address. To design an outcome evaluation, begin with a review of the outcome components of your theory of change or logic model.  The survey is most common for outcome evaluation.  Depending on where the target population or stakeholder resides, you can determine offline or online-based data collection.

Types of Evaluation Tools

Offline Data Collection & Evaluation

There are over 50+ offline data collection mobile data collection tools. 

• ODK based Ecosystem
• Dimagi
• KoboToolbox
• SurveyCTO
• Magpi
• SocialCops
• Dharma.io
• TaroWorks
• and many more!

Luthern World Relief has done an extensive review of many of these products and here is a quick summary of their review

• Best for Basic Mobile Data Collection: KoBoToolbox, Magpi, Fulcrum
• Best for Mapping: ArcGIS Collector, Smap
• Best for Broadcast Messaging: engageSPARK, Magpi
• Best for Recurring Collection: TaroWorks, Akvo Flow
• Detailed product review

Online Survey Tools & Evaluation

• Google Forms
• Survey Monkey
• Qualtrics
  And many more! Complete comparison is beyond the scope as there are too many choices.

Qualitative Evaluation

Core technique: Qualitative Analysis

Appropriate for Organization: Any 

Implementation: Relatively easy but survey design and collection may require substantial design 

Both qualitative and quantitative methods can be used in the monitoring and evaluation of an impact initiative. Some tools can handle both, while others are more suitable for one or the other.

For each of these tools, we’ll indicate the types of data it manages best and identifies some of the most robust complementary features. It is worth noting that this is not an exhaustive list, but some of the most robust and currently used M&E data management tools on the market.

Nvivo

This software boasts that it is number one for qualitative data crunching. While hard to quantify where it ranks, it comes highly recommended by academics and researchers, especially in public health fields. One of the most significant benefits of this software is the ability to manage diverse types of data, from text and audio to video and online surveys. An automated transcription feature adds to the efficiency-boosting nature it provides for qualitative data analysts.

Strength

• Designed for qualitative research

Weakness

• Limited use case

Monitoring Single Organization (Not for hierarchical organization)

Core technique: Aggregate result for a single enterprise for an internal purpose

Appropriate for Organization: Medium size nonprofits

Implementation: Relatively easy to setup 

TolaData

An impact-focused tool (they declare you can align with SDGs), TolaData offers real-time data management, combining built-in data collection features as well as import capabilities to manage data collected offline. A results-framework feature complements the analysis tools so that users can track how they are doing in reaching their impact goals.

TolaData does have an ability to aggregate results from external sources; it focuses on integration with few well known offline data tools.

Level

• Small & Midsize NGO (Department Level) and foundation 

Strength

• Easy to setup
• Integrates data feed from a few offline tools

Limitations

• Reliance on results framework
• Cannot scale with data governance required at different department or country with different level of data culture
• Aggregate and roll-up of the department, country data not feasible
• Impact Framework: Custom & Limited
• Dashboard: Basic
• Scorecard & Professional Impact Reporting: None

LogAlto

A cloud-based software solution, LogAlto heavily emphasizes the collaboration-boosting benefits of its offering. It purports to be easy to use, which invites stakeholders of all levels of tech-savviness into the M&E journey. Its mobile app can be used to manage data in real-time while in the field or to take advantage of offline data collection functions. LogAlto is mostly used by NGOs and foundations, or by government-led initiatives.

LogAlto is a simple activity and indicator management tracking platform designed for relatively simple M & E programs. LogAlto does not have a sophisticated indicator framework, hence building effective outcome-oriented reporting is somewhat challenging. Data aggregation capabilities are relatively basic, which can cause challenges for M & E team especially if you are going to be frequently collecting results from different sources.

Level

• Small & Midsize NGO (Department Level)

Strength

• Easy to setup

Limitations

• Reliance on results framework
• Cannot scale with data governance required at different department or country with different level of data culture
• External data integration limited
• Aggregate and roll-up of the department, country data not feasible
• Impact Framework: Custom & Limited
• Dashboard: Basic
• Scorecard & Professional Impact Reporting: None

ActivityInfo

If you need to work together across many different organizational partners, ActivityInfo offers a powerful solution for collaborative M&E. Used in a diverse number of sectors such as education, health, and water and sanitation, it is built for impact leaders working at the front-lines of “humanitarian operations.” Its database builder ensures easier adoption even in less tech-savvy teams. See its full list of features here.

Monitoring Sector-Specific Multi Programs

Core technique: Aggregate sector-specific results such as Health or WASH

Appropriate for Organization: Multiple Program Initiative

Implementation: Medium to complex 

Example Software: DHIS2 for Healthcare and Akvo for WASH

Monitoring Aid Related Multiple Programs

Core technique: Designed for large M & E programs funded by public funds to monitor activity, output through Results Framework

Appropriate for Organization: Designed for large M & E programs financed by federal funds to monitor activity, output through Results Framework

Implementation: Relatively complex

DevResults

Led by its activity mapping tool, which enables practitioners to visualize the geographic landscape of their interventions, DevResults is undoubtedly a leader in M&E software solutions. It complements this unique feature with a framework built to help teams align and track key impact indicators during the life of a project. It rounds out these features with a built-in work plan platform to keep organizations accountable to project timelines and goals.

Level

• Large Hierarchical Organizations

Strength

• Comprehensive
• Highly customizable

Limitations

• While DevResults is a relatively robust and complete platform, you should seriously consider the following risks: 
• 5 Worst Mistakes to Avoid in Monitoring & Evaluation Tools Selection
• More features than an organization can adopt successfully
• Cost
• Time to Implement
• Salesforce Based

• Vera solutions
• Social Suite
  
  

Impact Monitoring

Core technique: Designed for INGO,  Foundations, Nonprofits with multiple programs, Hierarchical Structure to Aggregate Results, and Raise Grant or Impact Capital

Appropriate for Organization: Multiple or Single Organization

Implementation: Comprehensive, Flexible, Relatively Fast

Unique Value Proposition

• Impact Framework: impact knowledge graph simplifies consistent and standardized impact theme, impact metrics, survey and due diligence implementation across different organizations
• Data Repository: Manage organization, partners, grantee and investee output, outcome, performance, story and document in a single location
• Designed for Fundraising: Align funders with program results
•  Data Warehouse, Aggregate from many sources: Aggregate results from many sources and auto calculate metrics in seconds
• Seamless Offline Data: Seamless bidirectional integration with KoboToolbox
• Collaboration between funders & partner:  Collaborate with impact results, field notes, stories, and program evaluation notes
• Document Management: Simple and Seamless document search for all stakeholders (based on access)
• Data Mapping & Multi-Table Calculation: Complex calculation and multi-table alignment
• Automated Scorecard with SDG Alignment:  Streamline Integrate stakeholder, investors, SDG and IMP based dashboards
• Qualitative AI-Based Analysis:  Stakeholder Voice

While a newer player in the Monitoring & Evaluation field, Sopact’s Impact Cloud incorporates many of the features mentioned above and houses them all in one place on its cloud-based platform.

It’s a virtual data warehouse that thrives at simplifying the often complex data aggregation process from multiple sources of data. It will help teams do away with the file after file of Excel spreadsheets by offering one place for data to be housed, managed, analyzed, and reported.

In short, it helps practitioners transform impact data into impact insights.

A theory of change model is only as good as the tools you have to make it a reality in your organization’s implementation and in the lives of the beneficiaries you hope to impact!

All of the above legacy M&E software platform have stated based on results framework.  During the mid-1990s, USAID started a new approach to monitor programs throughout the international development agencies, known as Performance Monitoring Plans (PMP). Central to PMP is the results framework -- planning, communications, and management tool.

The results framework includes

• Strategic objectives
• All intermediate results

Limitations of Results Framework

Key Limitations:

• Cause-and-effect Linkages: The framework describes the ways that program interventions contribute to the results through cause-and-effect linkages. The framework cannot do robust traceability of influence or attributions
• Not easy to quantify many interventions that require policy changes etc.
• Framework forces unnecessary fear of not reaching specific outcome causing program managers in limiting or selecting limited outputs

Limitations of Legacy M&E Software

Many M & E Software suffer from

• Rigid framework does not allow to define appropriate social impact context required for each unique initiative
• Long and tedious implementation time. Most software implementation can take a long time to implement.
• Cost is significantly high. See example
• By the time implementation takes, the requirements may change.  This creates a tremendous change control risk
• Project management can now be done easily by so many online SAAS based products at the faction of the cost.

Monitoring and Evaluation For NGOs

Many International NGOs are different from traditional international development agencies.  Unlike development agencies, INGO typically has office headquartered in western countries such as the US, Canada or Europe. They often act a funder to the program organizations in different countries. These county offices often act as decentralized units, who often have different types of processes.  The challenge with that approach is headquartered often needs much faster visibility of data

• Financial
• Donor
• Operational
• Social Impact

Traditional M & E software does not provide any support similar to the data warehouse for the country level offices. These offices often collect data in the form of paper, offline mobile data collection or local IT systems. However, aligning data requirements between different offices becomes challenging. Mid and Large size development finance institutes often are faced with inherent limitations of results framework based design. This approach seems to make the process of impact data aggregation time consuming.

Impact Learning

Traditional M&E Software and SoPact Impact Cloud® both have similar goals. However their approaches are completely different.  Remember SoPact Impact Cloud  is 

• A flexible M & E platform that allows to build a flexible impact framework unique to each organizations unique need
  
  • Traditional M & E on the other hand is is tightly coupled with results framework
• Start your impact journey with theory of change 
  • Logframe based approach focuses on activity and not outcome  
• Save time with global impact indicators
  • Traditional M&E software
    
    • Only custom indicators
    • Limited reporting format
    • Limited qualitative and quantitative alignment
•  Aligns with sustainable development goals (SDG) and global indicators
  • No concept or scope of SDG alignment
• SoPact Impact Cloud simplifies top level portfolio data aggregation.  Impact Cloud focuses on an easier alignment with the downstream, decentralized organizations.  Impact Cloud's flexible approach allows rapid and flexible alignment necessary for the decentralized organization structure.
  • Traditional M&E software  is based on rigid reporting framework approach
•  SoPact Impact Cloud automatically integrates program data.  Its unique data warehouse based approach simplifies data aggregation from different data sources such as paper, online survey, offline survey and custom database
  
  • Traditional M & E focuses on results aggregation and does not provide any support for data aggregation at the downstream level.  This is a major flaw in their approach as downstream impact organization often do not lean social impact through lean data approach.
• SoPact Impact Cloud is integrated with powerful state of art qualitative and quantitative analytics

Quantitative Impact Data Collection Approaches

Every M&E process should have a strong quantitative data foundation. Such data helps set baseline comparisons, gauge program relevancy, determine efficiency (or inefficiencies, etc.). It is usually less cost-intensive to acquire quantitative data compared with qualitative.

Here are some examples of quantitative data tools, approaches, and resources:

• Survey instruments/questionnaires

  • Using tools like SurveyMonkey or even simpler SMS-based applications can be a relatively quick, cost-effective approach to acquiring beneficiary data.

• Existing articles, media, etc.

  • It is not always necessary to do the work yourself -- secondary sources such as research articles in magazines, academic papers, books, etc. may already have data points relevant to an organization’s context. Make sure the data is current and aligned with the impact objectives of the program.

• Similar, partner or competitor organizations’ reports

  • Are there other organizations working on similar issues? Most likely, there are. And it is likely that that they have done some impact reporting, which could include relevant data.

• Government

  • State-run websites include statistics and other reporting that could provide important contextual data for a program’s impact theme or region. However, just because it is a governmental source, doesn’t mean it automatically is credible data. Organization’s must make their own judgements on the quality of such a source.

Qualitative Impact Data Approaches

To acquire data that is more nuanced, and perhaps with deeper insights, it can be necessary to take a qualitative approach. This tends to be more time- and resource-intensive, as the approaches are usually highly participatory, but it is important to make such an investment in most programs in order to complement the quantitative data that has been acquired.

• Focus groups/interviews

  • Getting beneficiaries together and having face-to-face conversations (either 1-on-1 or in groups) can be a great way to understand their perception of the impacts generated by the organization’s activities.

• Field immersion/observations

  • Qualitative observations of beneficiary context can not only help understand the extent of impacts generated, but can also help in uncovering ways to improve program/intervention design to be more context-relevant.

• Use photo/video

  • Encourage beneficiaries to document their journey (or collaborate with them to do this) using photographic media as the medium.

• Journaling

  • Encourage beneficiaries to reflect in written form periodically (daily, weekly, etc.) about their experience. Organization’s could also send periodic long-form questions to capture such data.
    
    

Read More: One must have tool for Lean Data Social Impact Analysis

Monitoring & evaluation systems were designed to manage projects for results management.  However, results management should not be confused in social impact measurement.  

Common challenges of most M&E Suites include:

• Expensive to acquire and use
• They take a long time to deploy
• High execution risk (e.g. because of a lack of organizational skills)
• Data silo-ing (importing, exporting, and sharing data across different programs can lead to data corruption, data loss, and/or reduction in data reliability).  

Making the Monitoring and Evaluation Successful

Most M&E Systems were designed 10-20 years ago.  Organizations continuing to use these outdated systems will continue to face mounting challenges because of their cumbersome nature and the lack of nimble functionality/adaptability of those systems.

Key limitations include:

Impact Learning

It is one thing to have data to analyze, it is another to know how to analyze it and extract important impact insights from those data. Those insights can be program design-related, outcomes-related, etc. Built into our M&E tools needs to be the functionality to streamline that insight generation.

Data Warehouse & Program Data Aggregation

In the information age we have the capacity to acquire so much more data, more quickly, more often, and from more sources. Managing those data across different software programs and between teams is one of the biggest headaches faced by organizations across sectors.

What is needed are technological solutions that are end-to-end, empowering practitioners to house and manage data all in place (accessible to on the ground workers, and to directors and funders).

Social Impact & Outcome Management

Managing impacts isn't just about determining what was generated by a program, what wasn't, etc. It is about providing the necessary tools (and ensuring proper adoption/implementation)  so that impact strategies can be created, executed, tracked/measured, analyzed, improved, and reported on. Moving towards cloud-based applications that can do all of those things in one place will allow practitioners to manage impact all in one place, improving both efficiency and, ultimately, outcomes as well.

Distributed, Decentralized and Hierarchical

Many international development organizations employ a distributed, decentralized and hierarchical approach.  This creates many challenges for stakeholders, both internal (teams), and external (partners, funders, etc.).

Challenges:

• Inconsistent reporting to top
  • Funders, for example, can find it difficult to get a clear, transparent picture of the impacts being generated, how they were generated, and how such insights were acquired. This is made even more difficult if different standards are used at each level of data collection, reporting, etc.
• Inconsistent data collection processes at different locations
  • If team members are "plugging in" to beneficiaries in different ways (using different tools, or different data management software) this can create huge reliability issues within data sets. 
• Long reporting cycle
  
  • Often because of resource limitations, organizations may not be employing M&E processes often enough (e.g. during implementation executing enough monitoring). This could reduce the visibility of impacts generated, while also making it near impossible to adapt in an agile manner to dynamic contextual factors.