Monitoring and Evaluation Tools
- Data ownership,
- Time to set up,
- Quality of data collection,
- Advanced analytics.
Monitoring and Evaluation Tools Training
Post-COVID 19, there is a reawakening about how monitoring and evaluation are going to change?
Do you use mobile data collection based on open data kit or mobile apps like Kobotoolobox?
Monitoring and Evaluation - common pitfalls
- Cost of the tool
- Cost to train people to use the tools
- Hiring people with special skills on specific products you may be using
- The time it takes to set up the entire process - for continuous learning and improvement.
- Customization to suit specific needs.
Managing end-to-end monitoring and evaluation or social impact measurement requires four significant steps.
Checklist monitoring and evaluation tools
DO NOT become ideological about frameworks such as Log Frame, Results-Based Accounting (RBA), Logic model, or "Theory of Change (TOC)." This is the fundamental point here.
Offline data collection can improve data accuracy.
Not all offline data collection is the same. Make sure the goals are clear:
- Faster data collection or even real-time data collection to improve longitudinal data
- Ease of use for field worker
- Beneficiary understanding: Language and education barriers
- Fieldworker trust (avoid fudging data)
- Training barriers
- Ease of analyzing that allows you closer to faster data.
Stay away from tools that require extensive customization. Well designed modern SAAS tools provide a configuration-based approach.
DO NOT overload M&E tool selection with a requirement such as a project management. Modern SAAS tools are the best of the breed (i.e., they provide the best user experience and functionality and cost significantly less)
- Consider end-to-end monitoring and evaluation tool(s) that is easy to configure to reduce all the core monitoring and evaluation functionalities.
- Data visualization tools should not be confused with impact reporting. Different data collection and reporting tools can keep you away from continuous learning and improvement goals. If you are using it to create many charts to demonstrate activities and outcomes then where is the learning?
- While many non-profits encourage the RFP process, this process may often increase your timeline, cost, bring favoritism and often not serve true M&E requirements. The best approach is to work with a vendor with a strong team to demonstrate lifetime customer success commitment and fully understand all the goals, design gap analysis. Ask them to build one program, FastTrack (30-Days with working exercise), and align the entire team together. This allows you to reduce implementation risk and reduce the total cost of ownership.
Innovative data collection approachModern data collection requires faster and higher data collection
Two major data collection trends post-pandemic
(HINT: MORE REAL-TIME)
FUTURE OF REMOTE DATA COLLECTION
What is the future of the Remote Data Collection in Monitoring, Evaluation, and Learning?
- How will Social Impact Measurement focus on real-time impact learning?
INTEGRATED OFFLINE DATA COLLECTION AND ANALYTICS
Monitoring and Evaluation Tools Landscape
Some of the challenges are listed below :
- Inconsistent Data Collection
- Activity & Output as Impact Reporting
- Impact Data Collection, Cleaning & Aggregation Costs
- Low-Quality Impact Data Collection
- Mobile Data Collection
- Excel or Google Spreadsheet
- Salesforce & Similar Program Data Management Tools
- Lack of consistency in reporting due to multiple funder requirements.
- Data collection is inconsistent and not aligned with funders.
- Data cleaning and aggregation for reporting are time-consuming.
- Relying on external data analytics tools like Tableau and Power BI can significantly increase time and require specialized skills.
- Often Theory of Change, data collection, and reporting are not aligned, making continuous learning and improvement a pipe dream.
Many organizations start with a long household survey; often, this survey fails to collect key outcomes aligned with change theory. We need a system that focuses on core aspects of impact management and outcomes aligned towards program goals. A Theory of Change driven and outcome-aligned stakeholder survey with well balanced qualitative and quantitative stakeholder feedback can achieve desired impact management.
As organization adds new programs, its reporting system becomes challenging and isolated. For example, for homelessness, a government funder may insist on using the HMIS system whereas Social services organizations may utilize case management systems, customize Salesforce-based program management, or Effort To Outcome system. While these systems might manage program data, often, their reporting tends to be relatively limited and does not even align well with funders reporting requirements. Hence, such organization often hires monitoring and evaluation analysts who usually export data from different systems and manually cleanse data with complex, tedious, and often error-prone spreadsheets. This process will keep an organization from good insight for continuous learning and improvement required for impact management.
Data gets unmanageable
Data integrity not preserved
Recreate formula every time with new data collection
Staff turnover can be vulnerable.
- Challenging to combine data from multiple tables
- Report building is too complex.
- Reporting for all agreed-upon metrics come from different external sources.
- Requires frequent reporting such as monthly, quarterly, and annually
Surveys: Surveys are evaluation tools that can be used to gather a large amount of data from a large number of people. Surveys can be conducted online, by phone, or in person and can include questions on various topics, including attitudes, behaviors, and opinions.
Interviews: Interviews are a qualitative evaluation tool that allows for in-depth topic exploration. Interviews can be conducted with individuals or groups and structured or unstructured.
Focus Groups: Focus groups are a qualitative evaluation method involving bringing a small group of people together to discuss a specific topic. Focus groups can be useful for gaining a more in-depth understanding of a particular issue or perspective.
Case Studies: Case studies are a form of qualitative evaluation that involves an in-depth examination of a specific individual, group, or program. Case studies are useful in exploring a particular issue, program, or intervention.
Observational techniques: Observational techniques are a form of qualitative evaluation that involve observing people in their natural setting. This can include direct observation, video recording, or participant observation.
- 01 Offline Data Collection
- 02 Online Survey Tools & Evaluation
- 03 Online Data Management Tools
- 04 Qualitative Evaluation
- 05 Nvivo
- Sopact Survey is the best for organizations looking for integrated strategy, data collection, aggregation, and business intelligence
- Best for Basic Mobile Data Collection: KoBoToolbox, SurveyCTO
- Best for Mapping: ArcGIS Collector
- Best for Broadcast Messaging: engageSPARK, Magpi
- Best for Recurring Collection: TaroWorks, Akvo Flow
- Detailed product review
- Google Forms
- Survey Monkey
And many more! Complete comparison is beyond the scope as there are too many choices.
- Google Spreadsheet
- Google's AppSheet (has the offline capability)
- Airtable (Popular and Ease Of Use and Has Online Survey)
This software boasts that it is number one for qualitative data crunching. While hard to quantify where it ranks, it comes highly recommended by academics and researchers, especially in public health fields. One of the most significant benefits of this software is managing diverse data types, from text and audio to video and online surveys. An automated transcription feature adds to the efficiency-boosting nature it provides for qualitative data analysts.
- Designed for qualitative research
- Limited use case
- More AI-based approach allows better sentiment, emotion, and keyword analysis. This approach is easier to analyze and understand.
Monitoring tools allow you to track progress, identify challenges, and assess the implementation of a program or project. These tools can include key performance indicators (KPIs), dashboards, checklists, and monitoring plans. Monitoring tools are used to gather qualitative and quantitative data. The choice of tool will depend on the type of information needed, the resources available, and the specific monitoring goals and objectives. Examples of monitoring tools include:
- Key Performance Indicators (KPIs): KPIs are quantitative measures used to track progress and assess the performance of a program or project. Examples of KPIs include budget spent, number of participants, number of training sessions, and number of products sold.
- Dashboards: Dashboards are interactive visualizations that provide a quick overview of critical data and performance metrics. Dashboards can be used to track progress and identify trends in real-time and can be accessed from anywhere with internet access. Example: https://www.datapine.com/dashboard-software/
- Checklists are simple lists of tasks or activities that must be completed to ensure that a program or project is being implemented correctly. Checklists can be used to track progress, identify challenges, and ensure that all necessary steps are being taken. Example: https://www.checkli.com/
- Monitoring plans: Monitoring plans are detailed documents that outline the methods, tools, and indicators that will be used to track progress and assess the performance of a program or project. Monitoring plans can be used to ensure that all necessary data is being collected and can be used as a reference throughout the life of the program or project.
- 01 Monitoring Single Organization
- 02 TolaData
- 03 LogAlto
- 04 ActivityInfo
- Core technique: Aggregate result for a single enterprise for an internal purpose
- Appropriate for Organization: Medium size nonprofits
- Implementation: Relatively easy to set up and cost-effective
- Focuses on activity and output reporting
- Often rely on third-party visualization tools for analysis.
- Not appropriate for continuous learning and improvement
A monitoring and evaluation tool (with the capacity to align with SDGs or IATI reporting standards), TolaData offers real-time data management, combining built-in data collection features as well as import capabilities to manage data collected offline. A results-framework feature complements the analysis tools so that users can track how they are doing in reaching their impact goals. An activity management section allows users to assign tasks and manage approval workflows. TolaData can aggregate results from external sources; it focuses on integration with a few well-known offline data tools. It also has a basic form builder available for online data collection. Data can be disaggregated and separated into reporting periods within indicators. Data can be aggregated within projects or across multiple projects in the tool.
Small, Midsize, and Large NGO (Department Level) and foundation
- TolaData is an Off-Shelf tool so there is no setup required by the organizations.
- Integrates data feed from a few offline tools
- Dashboard reporting updates in real-time as data in entered at the indicator level
- Large aggregation capability with options for custom formulas
- All data from the platform is exportable
- Designed to facilitate collaboration during the project implementation phase
- Cannot scale with data governance required at different departments or countries with different levels of data culture
- Not designed for strategy development and planning stage
- Impact Framework: Custom & Limited
- Scorecard & Professional Impact Reporting: None
A cloud-based software solution, LogAlto heavily emphasizes the collaboration-boosting benefits of its offering. It purports to be easy to use, which invites stakeholders of all tech-savviness levels into the M&E journey. Its mobile app can be used to manage data in real-time while in the field or to take advantage of offline data collection functions. LogAlto is mostly used by NGOs and foundations or by government-led initiatives.
LogAlto is a simple activity and indicator management tracking platform designed for relatively simple M & E programs. LogAlto does not have a sophisticated indicator framework; hence building effective outcome-oriented reporting is somewhat challenging. Data aggregation capabilities are relatively basic, which can cause challenges for the M & E team, especially if you will frequently be collecting results from different sources.
LevelSmall & Midsize NGO (Department Level)
- Easy to setup
- Reliance on results framework
- Cannot scale with data governance required at different department or country with different level of data culture
- External data integration limited
- Aggregate and roll-up of the department, country data not feasible
- Impact Framework: Custom & Limited
- Dashboard: Basic
- Scorecard & Professional Impact Reporting: None
If you need to work together across many different organizational partners, ActivityInfo offers a powerful solution for collaborative M&E. Used in a diverse number of sectors such as education, health, and water and sanitation; it is built for impact leaders working at the front-lines of “humanitarian operations.” Its database builder ensures easier adoption even in less tech-savvy teams. See its full list of features here.
Monitoring and Evaluation Suites
- 01 Monitoring Sector-Specific Multi Programs
- 02 Monitoring Aid Related Multiple Programs
- 03 DevResults
- 04 Salesforce Based
- 05 Continuous Learning and Improvement
Core technique: Designed for large M & E programs funded by public funds to monitor activity, output through Results Framework
Appropriate for Organization: Designed for large M & E programs financed by federal funds to monitor activity, output through Results Framework
Implementation: Relatively complex
Led by its activity mapping tool, which enables practitioners to visualize their interventions' geographic landscape, DevResults is undoubtedly a leader in M&E software solutions. It complements this unique feature with a framework built to help teams align and track key impact indicators during a project's life. It rounds out these features with a built-in work plan platform to keep organizations accountable to project timelines and goals.
- Large Hierarchical Organizations
- Highly customizable
- While DevResults is a relatively robust and complete platform, you should seriously consider the following risks:
- More features than an organization can adopt successfully
- Time to Implement
Salesforce is one of the most robust and popular cloud based platform. Salesforce freemium pricing has invited many nonprofits to use this platform. If you are building a data management solution, Salesforce may be a good robust solution. However, impact measurement is quite dynamic and specific to every organization's internal framework, changing over time. While vendors may develop the solution, they often may require significant customization. Changing future requirements for the different processes can add additional challanges.
- Vera solutions
- Social Suite
Core technique: Designed for INGO, Foundations, Nonprofits with multiple programs, Hierarchical Structure to Aggregate Results, and Raise Grant or Impact Capital
Appropriate for Organization: Multiple or Single Organization
- Relatively Fast
- Compatible with popular data collection tools
- Built-in sophisticated analytics designed for continuous learning and improvement
Unique Value Proposition
- Impact Framework: impact knowledge graph simplifies consistent and standardized impact theme, impact metrics, survey and due diligence implementation across different organizations
- Data Repository: Manage organization, partners, grantee and investee output, outcome, performance, story, and document in a single location
- Designed for Fundraising: Align funders with program results.
- Data Warehouse, Aggregate from many sources: Aggregate results from many sources and auto calculate metrics in seconds
- Seamless Offline Data: Seamless bidirectional integration with KoboToolbox, SurveyCTO and others (in progress)
- Collaboration between funders & partner: Collaborate with impact results, field notes, stories, and program evaluation notes
- Document Management: Simple and Seamless document search for all stakeholders (based on access)
- Data Mapping & Multi-Table Calculation: Complex calculation and multi-table alignment
- Automated Scorecard with SDG Alignment: Streamline Integrate stakeholder, investors, SDG and IMP based dashboards
- Qualitative AI-Based Analysis: Stakeholder Voice
Impact Cloud - Modern and Innovative MEAL Tool
While a newer player in the Monitoring & Evaluation field, Sopact’s Impact Cloud incorporates many of the features mentioned above and houses them all in one place on its cloud-based platform.
It’s a virtual data warehouse that thrives at simplifying the often complex data aggregation process from multiple data sources. It will help teams do away with the file after file of Excel spreadsheets by offering one place for data to be housed, managed, analyzed, and reported.
In short, it helps practitioners transform impact data into impact insights.
A theory of change model is only as good as the tools you have to make it a reality in your organization’s implementation and in the lives of the beneficiaries you hope to impact!
Monitoring and Evaluation Tools Resources
The USAID Monitoring and Evaluation Support Project: This project, managed by the RTI International, provides a wide range of resources on monitoring and evaluation, including guides, tools, and templates for designing and implementing monitoring and evaluation systems. https://www.rti.org/impact/usaid-monitoring-and-evaluation-support-project
The Development Impact Evaluation (DIME) Network: The DIME Network is an international community of researchers and practitioners who work to improve the design and use of impact evaluations in development. They provide resources, training, and research on monitoring and evaluation. https://dime-network.org/resources/
The World Bank: The World Bank provides a wide range of resources on monitoring and evaluation, including guides, tools, and templates for designing and implementing monitoring and evaluation systems. These resources are designed for practitioners and policymakers working in development. https://www.worldbank.org/en/topic/monitoringevaluation
The Evaluation Resource Center (ERC): ERC is an online resource center for evaluation professionals, researchers, and practitioners. They provide resources, training, and research on monitoring and evaluation. http://www.evaluationresourcecenter.org/
The African Evaluation Association (AfrEA): AfrEA is a professional organization that promotes evaluation in Africa. They provide resources, training, and research on monitoring and evaluation. https://www.afrea.org/resources