Close
SEARCH
Talk to us
Talk to us

MONITORING & EVALUATION TOOLS

Monitoring and Evaluation Tools - Post COVID19 Guide đź“š
Download Guide

MONITORING AND EVALUATION TOOLS

The best monitoring and evaluation tool provides an integrated approach that is easy to adopt and takes you to continuous learning and improvement goals. Selecting monitoring and evaluation tools can be daunting.  Some provide a point-to-point feature and have to use multiple tools to meet donor reporting requirements. Whereas some provide a suite of services which can cost a fortune combined with high risk of implementation failure and a longer customization time. This even when implemented may not meet requirements due to poor user experience.

Evidence and data are keys to long term social impact. The process of M&E tool selection must start with a well-defined goal. Many still focus on activity tracking and outcome harvesting, which might be sufficient for some donors who want impact justification (reporting), but it doesn't advance you towards evidence based approach. At the same time, a mission-oriented organization understands the purpose of monitoring and evaluation as listening to the stakeholder and improving the community's outcome. 

- Which is a better approach?  Which side are you on?

 

 

TOUCH
M&E PITFALLS
M&E PITFALLS Read More
TOUCH
M&E TOOLS CHECKLIST
M&E TOOLS CHECKLIST

M&E Tools Checklist

 Read More
TOUCH
POST PANDEMIC TRENDS
POST PANDEMIC TRENDS Read More
TOUCH
M&E TOOLS LANDSCAPE
M&E TOOLS LANDSCAPE

M&E Tools Landscape

 Read More
TOUCH
TYPES OF EVALUATION TOOLS
TYPES OF EVALUATION TOOLS

Types of Evaluation Tools

 Read More
TOUCH
MONITORING TOOLS
MONITORING TOOLS

Monitoring Tools

 Read More
TOUCH
M&E SUITE
M&E SUITE

M&E Suite

 Read More
TOUCH
CONCLUSION
CONCLUSION Read More
CHALLENGES

MONITORING AND EVALUATION - COMMON PITFALLS

Many tools are readily available to manage monitoring and evaluation, data collection, data analysis and impact reporting. These tools do offer some distinct advantages to small-scale organizations but does not scale well when the data is collected from different sources for multiple programs. As most organizations need a common data warehouse to store all of their information so seamless continuous data learning and reporting to happen.
While many nonprofits and social enterprises collect data, most organizations trip up to derive insights from the collected data, which gets worse when data is collected continuously. There is no well-established process to turn the data from raw data to insightful information. 
 
When evaluating pricing of a solution, it is important to think about the different factors that affect the cost.
  • Cost for the tool
  • Cost to train people to use the tools
  • Hiring people with special skills on specific products you may be using
  • The time it takes to set up the entire process  - for continuous learning and improvement.
  • Customization to suit to specific needs

 
The perfect tool selection integrates with a well-designed data strategy and can effectively improve your data's accountability and insights. This makes continuous learning easier and drastically simplifying the process of reporting direct impact specific to each funder with whom you engage. 

Managing end to end monitoring and evaluation or social impact measurement require four major steps.
  • Impact Strategy
  • Data Collection
  • Data Analysis
  • Impact Reporting
 
Organizations commonly use different tools for monitoring and evaluation or impact measurement. Using different tools can be effective in smaller-scale operations but it becomes quite cumbersome for an organization running multiple projects in multiple locations. So, juggling with these different tools(such as online AND offline data collection, excel for aggregation, and Tableau for reporting) can be extremely challenging  to establish a robust learning and improvement process. In most cases, these tools are way more sophisticated than one may need, resulting in unnecessary complexity and learning curves when applying them to your specific use-case.
Now we can move on to Impact Strategy. Often most organizations work with impact practitioners to define the theory of change or logic model. Theory of change or logical framework is not just diagrams that they can agree upon. Still, it needs to be a living and breathing process throughout the impact management lifecycle - from strategy, data collection, data analysis, and reporting. This is a process that they use for continuous learning and improvement.  Having identified measurable impact metrics is critical, organizations can start with activities and output with a clear goal to align them with measuring core outcomes.
These metrics are merely measurement elements, but a well-defined strategy that first identifies WHO, i.e., the kind of community or the stakeholder the organization is reaching out to, WHAT impact or outcome will they focus on, HOW much outcome, i.e., depth and scale of impact, and what CONTRIBUTION of organizations and impact RISK if intervention is not carried forward. 
Identifying measurable impact metrics is just one of the steps. Organizations, especially with a larger scale of operations, need to develop a systematic process to make the strategy actionable through systematic data collection and impact data analytics. 
Next up,  Mobile Data Collection: This may sound obvious, but organizations that serve people in parts of the world with choppy internet connection need to use mobile offline data collection tools. It is quite surprising to see well-established organizations using paper-based data collection practice resulting into a massive waste of time and resources. This can be solved very easily using tools such as KoboToolBox. 
Offline mobile data collection reduces data entry pain and improves accuracy; however, organizations should have well-established processes to learn from the collected data after the data collection. In absence of such processes, monitoring and evaluation staff painstakingly go through complex data aggregation for different programs, locations, and sources, limiting their ability to learn and improve their stakeholders' programs.
Processes that involves multiple tools by engaging in manual processing of data are erroneous, time-consuming, and inefficient. This is where an easy to use configuration-driven platform can make a big difference in improving its data governance.
Lastly, let's talk about Data analytics and visualizations.
Alternative: A large number of organizations use bespoke visualization applications to learn from their collected data. These tools are great at what they do. However, to effectively establish monitoring and evaluation practice, one needs a tightly integrated analytics application that can derive insights from the collected data. This ensures that the data flow is seamless from stakeholders to the visualizations that make their way into a dashboard or reports. This improves operational efficiency and helps the organization focus on what they need to focus on: serve the stakeholders!
 
Download Guide & Requirement Gathering Template
WHAT YOU SHOULD LOOK FOR IN SELECTING

CHECKLIST MONITORING AND EVALUATION TOOLS

  • DO NOT become ideological about frameworks such as Log Frame, Results-Based Accounting (RBA), Logic model, or "Theory of Change (TOC)."  This is the fundamental point here. 
  • Offline data collection can improve data accuracy.
  • Not all offline data collection is the same.  Make sure the goals are clear: 
    • Faster data collection or even real-time data collection to improve longitudinal data
    • Ease of use for field worker
    • Beneficiary understanding: Language and education barriers
    • Fieldworker trust (avoid fudging data)
    • Training barriers
    • Ease of analyzing that allows you closer to faster data. 
  • Stay away from tools that require extensive customization. Well designed modern SAAS tools provide a configuration-based approach.
  • DO NOT overload M&E tool selection with a requirement such as a project management.  Modern SAAS tools are the best of the breed (i.e., they provide the best user experience and functionality and cost significantly less)
 
  • Consider end-to-end monitoring and evaluation tool(s) that is easy to configure to reduce all the core monitoring and evaluation functionalities.
  • Data visualization tools should not be confused with impact reporting.  Different data collection and reporting tools can keep you away from continuous learning and improvement goals. If you are using it to create many charts to demonstrate activities and outcomes then where is the learning?
  • While many non-profits encourage the RFP process, this process may often increase your timeline, cost, bring favoritism and often not serve true M&E requirements.   The best approach is to work with a vendor with a strong team to demonstrate lifetime customer success commitment and fully understand all the goals, design gap analysis. Ask them to build one program, FastTrack (30-Days with working exercise), and align the entire team together. This allows you to reduce implementation risk and reduce the total cost of ownership.

COVID-19 HAS SHIFTED DATA COLLECTION

More data collection efficiency is needed especially after COVID-19
WHERE TO START WITH

TWO MAJOR DATA COLLECTION TRENDS POST PANDEMIC

COVID-19 Pandemic has already put pressure on data enumerators as household data collection has become difficult for many. Now as we get ready for the post-Pandemic world.

(HINT: MORE REAL-TIME)

FUTURE OF REMOTE DATA COLLECTION

  • What is the future of the Remote Data Collection in Monitoring, Evaluation, and Learning?

  • How will Social Impact Measurement focus on real-time impact learning?
Post-COVID 19, there is a reawakening about how monitoring and evaluation is going to change? The use of offline data collection with household door to door surveys will be a thing of a past. Instead more people will shift to monitoring and evaluation learning or evaluation based on remote data collection such as WhatsApp enabled mobile data collection approach.
Data Collection Apps like Kobotoolbox are useful in collecting offline data. However, they can be used to collect remote data using social media such as Whatsapps or Facebook. The benefit of this data collection tools and technique is that you can collect data from stakeholders who do not own email address but uses social media tools. Impact Cloud and Kobotoolbox provide modern data collection and analysis tools. New data collection techniques and data collection tools now form the  foundations of monitoring, evaluation and learning, and social impact measurement. Learn how funders will have to keep investing in social outcome data.
 
 

INTEGRATED OFFLINE DATA COLLECTION AND ANALYTICS

Do you use mobile data collection based on an open data kit (ODK) or mobile apps like Kobotoolobox or surveyCTO?
While these platforms are perfect for offline data collection - moving from paper to digital but they provide limited data insight as these platforms focus on one form or survey at a time. Post-COVID the need for monitoring evaluation and learning is even more important. Real-time monitoring is critical for continuous impact learning. Offline data tools like Kobotoolbox and surveyCTO can get real-time impact measurement and impact management using Impact Cloud.
 
Challenges

MONITORING AND EVALUATION TOOLS LANDSCAPE

With multiple tools available in the market for monitoring and evaluation, data collection, and reporting, funder reporting often becomes chaotic without a well-designed impact strategy. Let us look at the challenges of current systems and discuss potential solutions. If you are a nonprofit or social enterprise struggling to compete against other grantees or investees to raise much-needed grants or impact capital, you’re at the right place!
Today, we’re diving into the biggest challenges of collecting data and impact reports. Our goal is to turn this challenge into the most critical opportunity that most organizations ignore - raising grants or impact capital. While many nonprofits and social enterprises collect data, most of it isn't utilized, let alone actionable. A well-designed data strategy can yield proper accountability, making it easier for continuous learning, improvement, and reporting specific to each funder. Keep in mind, most monitoring and evaluation systems fail to provide scalable solutions to aggregate results regularly. If they do, it may require significant customization and are expensive too.
 
Some of the challenges are listed below :
  • Inconsistent Data Collection
  • Activity & Output as Impact Reporting
  • Impact Data Collection, Cleaning & Aggregation Costs
  • Low-Quality Impact Data Collection
  • Mobile Data Collection 
  • Excel or Google Spreadsheet
  • Salesforce & Similar Program Data Management Tools
Mission-driven organizations often grow data collection and reporting organically as their programs and funders grow. While in many organizations, data collection and reporting evolves in an ad hoc fashion, difficult to streamline funder specific reporting. Hence, most of them manually export data on Excel or google sheet, clean up data, and prepare unique funder reports. However, there are several challenges with this approach. 
  • Lack of consistency in reporting due to multiple funder requirements.
  • Data collection is inconsistent and not aligned with funders.
  • Data cleaning and aggregation for reporting are time-consuming.
  • Relying on external data analytics tools like Tableau and Power BI can significantly increase time and require specialized skills.
  • Often Theory of Change, data collection, and reporting are not aligned, making continuous learning and improvement a pipe dream.
 
A successful grant and impact capital depends on the demonstration of impact reporting. But, effective reporting requires demonstration of impact evidence or what is changing? Often today’s monitoring and evaluation systems focus on activity and output based reporting and not real change.

Many organizations start with a long household survey; often, this survey fails to collect key outcomes aligned with change theory. We need a system that focuses on core aspects of impact management and outcomes aligned towards program goals. A Theory of Change driven and outcome-aligned stakeholder survey with well balanced qualitative and quantitative stakeholder feedback can achieve desired impact management.

As organization adds new programs, its reporting system becomes challenging and isolated. For example, for homelessness, a government funder may insist on using the HMIS system whereas Social services organizations may utilize case management systems, customize Salesforce-based program management, or Effort To Outcome system. While these systems might manage program data, often, their reporting tends to be relatively limited and does not even align well with funders reporting requirements. Hence, such organization often hires monitoring and evaluation analysts who usually export data from different systems and manually cleanse data with complex, tedious, and often error-prone spreadsheets. This process will keep an organization from good insight for continuous learning and improvement required for impact management.

 

Use of Excel or Google Spreadsheets to collect results is prevalent in large number of organization. Many in the field often receive results in the form of paper and later move to excel, google spreadsheets, or proprietary systems. Low-quality impact data and errors results in no insight needed for an organization to change the program direction. It costs more in the long term by losing an opportunity cost as well as resource bleeding.
There are innumerable online and offline data collection tools available but their built-in data analytics generally summarizes at the question or field level. The vital analysis requires an organization to compare multiple variations such as villages, racial, gender, and performance over time.
As different programs may use more than one system, most organizations try to bring data into MS-Excel or Google Docs. They carefully reformat data regularly so that they can calculate the results required by funders.
  • Data gets unmanageable
  • Data integrity not preserved
  • Recreate formula every time with new data collection
  • Staff turnover can be vulnerable. 
Many systems such as Salesforce, Case Management, or Program data may be  robust in collecting data; often, building reports may be challenging. Because it is,
  • Challenging to combine data from multiple tables
  • Report building is too complex.
  • Reporting for all agreed-upon metrics come from different external sources.
  • Requires frequent reporting such as monthly, quarterly, and annually
Systems such as Salesforce are robust while it may be challenging to summarize results. Often funders may require qualitative reporting that may not be part of the Salesforce like systems. It could increase the reporting time. 
CURRENT STATE

TYPES OF EVALUATION TOOLS

  • 01 Offline Data Collection 
  • 02 Online Survey Tools & Evaluation
  • 03 Online Data Management Tools
  • 04 Qualitative Evaluation
  • 05 Nvivo
There are over 50+ offline data collection mobile data collection tools. 
Luthern World Relief has done an extensive review of many of these products, and here is a quick summary of their review
  • Best for Basic Mobile Data Collection: KoBoToolbox, Magpi, Fulcrum
  • Best for Mapping: ArcGIS Collector, Smap
  • Best for Broadcast Messaging: engageSPARK, Magpi
  • Best for Recurring Collection: TaroWorks, Akvo Flow
  • Detailed product review
 
  • Google Forms
  • Survey Monkey
  • Qualtrics
    And many more! Complete comparison is beyond the scope as there are too many choices.
Core technique: Qualitative Analysis
Appropriate for Organization: Any 
Implementation: Relatively easy but survey design and collection may require substantial design 
Both qualitative and quantitative methods can be used in the monitoring and evaluation of an impact initiative. Some tools can handle both, while others are more suitable for one or the other.
We’ll indicate the types of data it manages best and identifies some of the most robust complementary features for each of these tools. It is worth noting that this is not an exhaustive list, but some of the most robust and currently used M&E data management tools on the market.

This software boasts that it is number one for qualitative data crunching. While hard to quantify where it ranks, it comes highly recommended by academics and researchers, especially in public health fields. One of the most significant benefits of this software is managing diverse data types, from text and audio to video and online surveys. An automated transcription feature adds to the efficiency-boosting nature it provides for qualitative data analysts.

Strength
  • Designed for qualitative research
Weakness
  • Limited use case
  • More AI-based approach allows better sentiment, emotion, and keyword analysis.  This approach is easier to analyze and understand.
CURRENT STATE

MONITORING TOOLS

  • 01 Monitoring Single Organization
  • 02 TolaData
  • 03 LogAlto
  • 04 ActivityInfo
  • Core technique: Aggregate result for a single enterprise for an internal purpose
  • Appropriate for Organization: Medium size nonprofits
  • Implementation: Relatively easy to set up  and cost-effective
CONS
  • Focuses on activity and output reporting
  • Often rely on third-party visualization tools for analysis.
  • Not appropriate for continuous learning and improvement

An impact-focused tool (they declare you can align with SDGs), TolaData offers real-time data management, combining built-in data collection features as well as import capabilities to manage data collected offline. A results-framework feature complements the analysis tools so that users can track how they are doing in reaching their impact goals.

TolaData can aggregate results from external sources; it focuses on integration with a few well-known offline data tools.

Level
Small & Midsize NGO (Department Level) and foundation 
Strengths
  • Easy to setup
  • Integrates data feed from a few offline tools
Limitations
  • Reliance on results framework
  • Cannot scale with data governance required at different department or country with different level of data culture
  • Aggregate and roll-up of the department, country data not feasible
  • Impact Framework: Custom & Limited
  • Dashboard: Basic
  • Scorecard & Professional Impact Reporting: None

A cloud-based software solution, LogAlto heavily emphasizes the collaboration-boosting benefits of its offering. It purports to be easy to use, which invites stakeholders of all tech-savviness levels into the M&E journey. Its mobile app can be used to manage data in real-time while in the field or to take advantage of offline data collection functions. LogAlto is mostly used by NGOs and foundations or by government-led initiatives.

LogAlto is a simple activity and indicator management tracking platform designed for relatively simple M & E programs. LogAlto does not have a sophisticated indicator framework; hence building effective outcome-oriented reporting is somewhat challenging. Data aggregation capabilities are relatively basic, which can cause challenges for the M & E team, especially if you will frequently be collecting results from different sources.

Level
Small & Midsize NGO (Department Level)
  •  
Strengths
  • Easy to setup
Limitations
  • Reliance on results framework
  • Cannot scale with data governance required at different department or country with different level of data culture
  • External data integration limited
  • Aggregate and roll-up of the department, country data not feasible
  • Impact Framework: Custom & Limited
  • Dashboard: Basic
  • Scorecard & Professional Impact Reporting: None

ActivityInfo

If you need to work together across many different organizational partners, ActivityInfo offers a powerful solution for collaborative M&E. Used in a diverse number of sectors such as education, health, and water and sanitation; it is built for impact leaders working at the front-lines of “humanitarian operations.” Its database builder ensures easier adoption even in less tech-savvy teams. See its full list of features here.

CURRENT STATE

MONITORING AND EVALUATION SUITE

  • 01 Monitoring Sector-Specific Multi Programs
  • 02 Monitoring Aid Related Multiple Programs
  • 03 DevResults
  • 04 Salesforce Based
  • 05 Continuous Learning and Improvement
  • Core technique: Aggregate sector-specific results such as Health or WASH
  • Appropriate for Organization: Multiple Program Initiative
  • Implementation: Medium to complex 

Example Software: DHIS2 for Healthcare and Akvo for WASH

 

Core technique: Designed for large M & E programs funded by public funds to monitor activity, output through Results Framework

Appropriate for Organization: Designed for large M & E programs financed by federal funds to monitor activity, output through Results Framework

Implementation: Relatively complex

Led by its activity mapping tool, which enables practitioners to visualize their interventions' geographic landscape, DevResults is undoubtedly a leader in M&E software solutions. It complements this unique feature with a framework built to help teams align and track key impact indicators during a project's life. It rounds out these features with a built-in work plan platform to keep organizations accountable to project timelines and goals.

Level

  • Large Hierarchical Organizations

Strengths

  • Comprehensive
  • Highly customizable

Limitations

Salesforce is one of the most robust and popular cloud based platform.  Salesforce freemium pricing has invited many nonprofits to use this platform.  If you are building a data management solution, Salesforce may be a good robust solution.  However, impact measurement is quite dynamic and specific to every organization's internal framework, changing over time. While vendors may develop the solution, they often may require significant customization.  Changing future requirements for the different processes can add additional challanges.

  • Vera solutions
  • Social Suite

Core technique: Designed for INGO,  Foundations, Nonprofits with multiple programs, Hierarchical Structure to Aggregate Results, and Raise Grant or Impact Capital

Appropriate for Organization: Multiple or Single Organization

Implementation: 

  • Comprehensive
  • Flexible
  • Relatively Fast
  • Compatible with popular data collection tools
  • Built-in sophisticated analytics designed for continuous learning and improvement

Unique Value Proposition

  • Impact Framework: impact knowledge graph simplifies consistent and standardized impact theme, impact metrics, survey and due diligence implementation across different organizations
  • Data Repository: Manage organization, partners, grantee and investee output, outcome, performance, story, and document in a single location
  • Designed for Fundraising: Align funders with program results.
  •  Data Warehouse, Aggregate from many sources: Aggregate results from many sources and auto calculate metrics in seconds
  • Seamless Offline Data: Seamless bidirectional integration with KoboToolbox, SurveyCTO and others (in progress)
  • Collaboration between funders & partner:  Collaborate with impact results, field notes, stories, and program evaluation notes
  • Document Management: Simple and Seamless document search for all stakeholders (based on access)
  • Data Mapping & Multi-Table Calculation: Complex calculation and multi-table alignment
  • Automated Scorecard with SDG Alignment:  Streamline Integrate stakeholder, investors, SDG and IMP based dashboards
  • Qualitative AI-Based Analysis:  Stakeholder Voice

 

While a newer player in the Monitoring & Evaluation field, Sopact’s Impact Cloud incorporates many of the features mentioned above and houses them all in one place on its cloud-based platform.

It’s a virtual data warehouse that thrives at simplifying the often complex data aggregation process from multiple data sources. It will help teams do away with the file after file of Excel spreadsheets by offering one place for data to be housed, managed, analyzed, and reported.

In short, it helps practitioners transform impact data into impact insights.

A theory of change model is only as good as the tools you have to make it a reality in your organization’s implementation and in the lives of the beneficiaries you hope to impact!

CONCLUSION

Choosing monitoring and evaluation tools shouldn't be difficult. But we have found the most organizations choose a difficult path.