MONITORING & EVALUATION TOOLS
TABLE OF CONTENTS
WHAT TO LOOK FOR IN A MONITORING AND EVALUATION TOOL?
The best monitoring and evaluation tool provides an integrated approach that is easy to adopt and takes you to continuous learning and improvement goals. Selecting monitoring and evaluation tools can be daunting. Some provide a point-to-point feature and have to use multiple tools to meet donor reporting requirements. Whereas some provide a suite of services which can cost a fortune combined with high risk of implementation failure and a longer customization time. This even when implemented may not meet requirements due to poor user experience.
Evidence and data are keys to long term social impact. The process of M&E tool selection must start with a well-defined goal. Many still focus on activity tracking and outcome harvesting, which might be sufficient for some donors who want impact justification (reporting), but it doesn't advance you towards evidence based approach. At the same time, a mission-oriented organization understands the purpose of monitoring and evaluation as listening to the stakeholder and improving the community's outcome.
- Which is a better approach? Which side are you on?
Post-COVID 19, there is a reawakening about how monitoring and evaluation are going to change?
Do you use mobile data collection based on open data kit or mobile apps like Kobotoolobox?
MONITORING AND EVALUATION - COMMON PITFALLS
- Cost for the tool
- Cost to train people to use the tools
- Hiring people with special skills on specific products you may be using
- The time it takes to set up the entire process - for continuous learning and improvement.
- Customization to suit specific needs
Managing end-to-end monitoring and evaluation or social impact measurement requires four major steps.
-1.png?width=693&name=Build%20Strategy-03%20(1)-1.png)
Download Guide & Requirement Gathering Template
CHECKLIST MONITORING AND EVALUATION TOOLS
-
DO NOT become ideological about frameworks such as Log Frame, Results-Based Accounting (RBA), Logic model, or "Theory of Change (TOC)." This is the fundamental point here.
-
Offline data collection can improve data accuracy.
-
Not all offline data collection is the same. Make sure the goals are clear:
- Faster data collection or even real-time data collection to improve longitudinal data
- Ease of use for field worker
- Beneficiary understanding: Language and education barriers
- Fieldworker trust (avoid fudging data)
- Training barriers
- Ease of analyzing that allows you closer to faster data.
-
Stay away from tools that require extensive customization. Well designed modern SAAS tools provide a configuration-based approach.
-
DO NOT overload M&E tool selection with a requirement such as a project management. Modern SAAS tools are the best of the breed (i.e., they provide the best user experience and functionality and cost significantly less)
- Consider end-to-end monitoring and evaluation tool(s) that is easy to configure to reduce all the core monitoring and evaluation functionalities.
- Data visualization tools should not be confused with impact reporting. Different data collection and reporting tools can keep you away from continuous learning and improvement goals. If you are using it to create many charts to demonstrate activities and outcomes then where is the learning?
- While many non-profits encourage the RFP process, this process may often increase your timeline, cost, bring favoritism and often not serve true M&E requirements. The best approach is to work with a vendor with a strong team to demonstrate lifetime customer success commitment and fully understand all the goals, design gap analysis. Ask them to build one program, FastTrack (30-Days with working exercise), and align the entire team together. This allows you to reduce implementation risk and reduce the total cost of ownership.
COVID-19 HAS SHIFTED DATA COLLECTION
More data collection efficiency is needed especially after COVID-19TWO MAJOR DATA COLLECTION TRENDS POST PANDEMIC
(HINT: MORE REAL-TIME)
FUTURE OF REMOTE DATA COLLECTION
-
What is the future of the Remote Data Collection in Monitoring, Evaluation, and Learning?
- How will Social Impact Measurement focus on real-time impact learning?
INTEGRATED OFFLINE DATA COLLECTION AND ANALYTICS
MONITORING AND EVALUATION TOOLS LANDSCAPE
Some of the challenges are listed below :
- Inconsistent Data Collection
- Activity & Output as Impact Reporting
- Impact Data Collection, Cleaning & Aggregation Costs
- Low-Quality Impact Data Collection
- Mobile Data Collection
- Excel or Google Spreadsheet
- Salesforce & Similar Program Data Management Tools
- Lack of consistency in reporting due to multiple funder requirements.
- Data collection is inconsistent and not aligned with funders.
- Data cleaning and aggregation for reporting are time-consuming.
- Relying on external data analytics tools like Tableau and Power BI can significantly increase time and require specialized skills.
- Often Theory of Change, data collection, and reporting are not aligned, making continuous learning and improvement a pipe dream.
Many organizations start with a long household survey; often, this survey fails to collect key outcomes aligned with change theory. We need a system that focuses on core aspects of impact management and outcomes aligned towards program goals. A Theory of Change driven and outcome-aligned stakeholder survey with well balanced qualitative and quantitative stakeholder feedback can achieve desired impact management.
As organization adds new programs, its reporting system becomes challenging and isolated. For example, for homelessness, a government funder may insist on using the HMIS system whereas Social services organizations may utilize case management systems, customize Salesforce-based program management, or Effort To Outcome system. While these systems might manage program data, often, their reporting tends to be relatively limited and does not even align well with funders reporting requirements. Hence, such organization often hires monitoring and evaluation analysts who usually export data from different systems and manually cleanse data with complex, tedious, and often error-prone spreadsheets. This process will keep an organization from good insight for continuous learning and improvement required for impact management.
-
Data gets unmanageable
-
Data integrity not preserved
-
Recreate formula every time with new data collection
-
Staff turnover can be vulnerable.
- Challenging to combine data from multiple tables
- Report building is too complex.
- Reporting for all agreed-upon metrics come from different external sources.
- Requires frequent reporting such as monthly, quarterly, and annually
TYPES OF EVALUATION TOOLS
- 01 Offline Data Collection
- 02 Online Survey Tools & Evaluation
- 03 Online Data Management Tools
- 04 Qualitative Evaluation
- 05 Nvivo
- ODK based Ecosystem
- Dimagi
- KoboToolbox
- SurveyCTO
- Magpi
- SocialCops
- Dharma.io
- TaroWorks
- and many more!
- Best for Basic Mobile Data Collection: KoBoToolbox, Magpi, Fulcrum
- Best for Mapping: ArcGIS Collector, Smap
- Best for Broadcast Messaging: engageSPARK, Magpi
- Best for Recurring Collection: TaroWorks, Akvo Flow
- Detailed product review
- Google Forms
- Survey Monkey
- Qualtrics
And many more! Complete comparison is beyond the scope as there are too many choices.
- MS-Excel
- Google Spreadsheet
- Google's AppSheet (has the offline capability)
- Airtable (Popular and Ease Of Use and Has Online Survey)
This software boasts that it is number one for qualitative data crunching. While hard to quantify where it ranks, it comes highly recommended by academics and researchers, especially in public health fields. One of the most significant benefits of this software is managing diverse data types, from text and audio to video and online surveys. An automated transcription feature adds to the efficiency-boosting nature it provides for qualitative data analysts.
Strength
- Designed for qualitative research
Weakness
- Limited use case
- More AI-based approach allows better sentiment, emotion, and keyword analysis. This approach is easier to analyze and understand.
MONITORING TOOLS
- 01 Monitoring Single Organization
- 02 TolaData
- 03 LogAlto
- 04 ActivityInfo
- Core technique: Aggregate result for a single enterprise for an internal purpose
- Appropriate for Organization: Medium size nonprofits
- Implementation: Relatively easy to set up and cost-effective
CONS
- Focuses on activity and output reporting
- Often rely on third-party visualization tools for analysis.
- Not appropriate for continuous learning and improvement
An impact-focused tool (they declare you can align with SDGs), TolaData offers real-time data management, combining built-in data collection features as well as import capabilities to manage data collected offline. A results-framework feature complements the analysis tools so that users can track how they are doing in reaching their impact goals.
TolaData can aggregate results from external sources; it focuses on integration with a few well-known offline data tools.
Level
Small & Midsize NGO (Department Level) and foundationStrengths
- Easy to setup
- Integrates data feed from a few offline tools
Limitations
- Reliance on results framework
- Cannot scale with data governance required at different department or country with different level of data culture
- Aggregate and roll-up of the department, country data not feasible
- Impact Framework: Custom & Limited
- Dashboard: Basic
- Scorecard & Professional Impact Reporting: None
A cloud-based software solution, LogAlto heavily emphasizes the collaboration-boosting benefits of its offering. It purports to be easy to use, which invites stakeholders of all tech-savviness levels into the M&E journey. Its mobile app can be used to manage data in real-time while in the field or to take advantage of offline data collection functions. LogAlto is mostly used by NGOs and foundations or by government-led initiatives.
LogAlto is a simple activity and indicator management tracking platform designed for relatively simple M & E programs. LogAlto does not have a sophisticated indicator framework; hence building effective outcome-oriented reporting is somewhat challenging. Data aggregation capabilities are relatively basic, which can cause challenges for the M & E team, especially if you will frequently be collecting results from different sources.
Level
Small & Midsize NGO (Department Level)Strengths
- Easy to setup
Limitations
- Reliance on results framework
- Cannot scale with data governance required at different department or country with different level of data culture
- External data integration limited
- Aggregate and roll-up of the department, country data not feasible
- Impact Framework: Custom & Limited
- Dashboard: Basic
- Scorecard & Professional Impact Reporting: None
ActivityInfo
If you need to work together across many different organizational partners, ActivityInfo offers a powerful solution for collaborative M&E. Used in a diverse number of sectors such as education, health, and water and sanitation; it is built for impact leaders working at the front-lines of “humanitarian operations.” Its database builder ensures easier adoption even in less tech-savvy teams. See its full list of features here.
MONITORING AND EVALUATION SUITE
- 01 Monitoring Sector-Specific Multi Programs
- 02 Monitoring Aid Related Multiple Programs
- 03 DevResults
- 04 Salesforce Based
- 05 Continuous Learning and Improvement
Core technique: Designed for large M & E programs funded by public funds to monitor activity, output through Results Framework
Appropriate for Organization: Designed for large M & E programs financed by federal funds to monitor activity, output through Results Framework
Implementation: Relatively complex
Led by its activity mapping tool, which enables practitioners to visualize their interventions' geographic landscape, DevResults is undoubtedly a leader in M&E software solutions. It complements this unique feature with a framework built to help teams align and track key impact indicators during a project's life. It rounds out these features with a built-in work plan platform to keep organizations accountable to project timelines and goals.
Level
- Large Hierarchical Organizations
Strengths
- Comprehensive
- Highly customizable
Limitations
- While DevResults is a relatively robust and complete platform, you should seriously consider the following risks:
- More features than an organization can adopt successfully
- Cost
- Time to Implement
Salesforce is one of the most robust and popular cloud based platform. Salesforce freemium pricing has invited many nonprofits to use this platform. If you are building a data management solution, Salesforce may be a good robust solution. However, impact measurement is quite dynamic and specific to every organization's internal framework, changing over time. While vendors may develop the solution, they often may require significant customization. Changing future requirements for the different processes can add additional challanges.
- Vera solutions
- Social Suite
Core technique: Designed for INGO, Foundations, Nonprofits with multiple programs, Hierarchical Structure to Aggregate Results, and Raise Grant or Impact Capital
Appropriate for Organization: Multiple or Single Organization
Implementation:
- Comprehensive
- Flexible
- Relatively Fast
- Compatible with popular data collection tools
- Built-in sophisticated analytics designed for continuous learning and improvement
Unique Value Proposition
- Impact Framework: impact knowledge graph simplifies consistent and standardized impact theme, impact metrics, survey and due diligence implementation across different organizations
- Data Repository: Manage organization, partners, grantee and investee output, outcome, performance, story, and document in a single location
- Designed for Fundraising: Align funders with program results.
- Data Warehouse, Aggregate from many sources: Aggregate results from many sources and auto calculate metrics in seconds
- Seamless Offline Data: Seamless bidirectional integration with KoboToolbox, SurveyCTO and others (in progress)
- Collaboration between funders & partner: Collaborate with impact results, field notes, stories, and program evaluation notes
- Document Management: Simple and Seamless document search for all stakeholders (based on access)
- Data Mapping & Multi-Table Calculation: Complex calculation and multi-table alignment
- Automated Scorecard with SDG Alignment: Streamline Integrate stakeholder, investors, SDG and IMP based dashboards
- Qualitative AI-Based Analysis: Stakeholder Voice
While a newer player in the Monitoring & Evaluation field, Sopact’s Impact Cloud incorporates many of the features mentioned above and houses them all in one place on its cloud-based platform.
It’s a virtual data warehouse that thrives at simplifying the often complex data aggregation process from multiple data sources. It will help teams do away with the file after file of Excel spreadsheets by offering one place for data to be housed, managed, analyzed, and reported.
In short, it helps practitioners transform impact data into impact insights.
A theory of change model is only as good as the tools you have to make it a reality in your organization’s implementation and in the lives of the beneficiaries you hope to impact!