Measuring impact result is becoming an essential requirement in philanthropy and impact investment sectors. Survey tools like SurveyMonkey, Qualtrics, and Excel have long been friends of practitioners, field managers and non-profits/social enterprises. There are many tools out there with a unique value proposition for each; fundamentally each serves a purpose of getting records of data. Once the data arrives, it is up to the data collector to build useful reports and tell compelling stories. Organizations using the master Excel for summarizing results from multiple excel sheets have many challenges. Let us see why?
1. Survey is just one piece of a puzzle
Often survey brings a slice of data. True social impact needs combination approach depending on the impact thesis of an organization or program such as,
- Ecosystem Investment based on outcome alignment
- Program investment based on program effectiveness
- Program investment based on demonstrating social change
Based on impact thesis, an organization needs to define a strategy that involves, operational capacity, scorecard, stakeholder assessment, alignment of required criteria that demonstrate risk and effectiveness. Often organizations have to develop their custom analysis process to bring all the data together to understand program effectiveness, social change, and outcome alignment. Without an integrated analysis, survey poses a danger to data integrity.
2. Survey vs. Results Collection
The survey is useful in gathering one-dimensional data from stakeholders as a baseline, pre, and post feedback. This process should not be confused with collecting results. At the high level, there are two types of result gathering from organizations or partners,
1) Output and outcome-oriented results
2) Process-based results with or without scorecard
While the first allows understanding overall output & outcome of an initiative, second is relevant for understanding program effectiveness and risk. Often organizations use well-defined standards or custom metrics/indicators to collect results. A simple survey can not collect RESULTS.
3. Better data analysis requires relationship
Social impact results gathering and analysis tend to be unique to each organization. The high precision result can come when metrics are assigned to the programs based on logical relationships. However, it is up to you to design data structure, often in tools like Excel which do not represent a relationship with Program, Partner, Product, Project, Applicant, etc. The well-designed platform allows you to collect data from the online or offline survey, index them in the database for a future reference and reporting without creating an island of data stories.
4. Pitfall of using irrelevant systems for impact management
Many of the grant management software include impact measurement application. Unfortunately, most of them provide the survey-like data gathered from grantees/investee. They are likely to respond based on a size of the investment, local compliance requirements before providing true results to the funder. Often funder designed the survey that matches program objectives.
Usually, grantee impact is unique, and survey questions are not relevant to them, or they do not have a data capacity ( a system that tracks activity, output or outcome ). Often grantee felt that they already answered the similar survey (frequent survey syndrome) and believed that another similar survey does not serve a purpose.
5.Difficult to understand deep impact
To understand an actual risk and impact you must aggregate results based on integrated stakeholder approach. For example, if a foundation is funding selected non-profits and wants to understand if recipient organization is improving volunteer capacity or not? Survey tools can measure volunteer management and then have to be compared against an indicator like volunteer opportunity. Once results come, all the data manipulation happens in MS-Excel with customized pivot tables and data analysis. As years and number of organizations increases, soon excel based approach becomes messy and unmanageable.
Survey cannot tell an insightful impact storyWhat do you need to tell an effective story?
- Collect data from partners and beneficiaries/clients that are relevant to them.
- Aggregate data in such a way that offer unique impact insight to programs, partners, or projects with logically similar entities.
- Provide a consistent and easy to use impact insight for different types of data aggregation in a single location.
The survey can not overcome a challenge of uniquely aggregating data from different stories in such a way that follows key principles.
Correct social impact measurement requires all the pieces that can complete a puzzle. We need to be using system like Impact Cloud to Measure Impact that provides a complete lifecycle of qualitative and quantitative metrics with different types of results collection based on impact thesis. It also starts with comprehensive social impact metrics selection, providing data collection process that incentivizes results provider, collect results that can be used with analysis that can be shared with multiple stakeholders. Survey tools are just a single piece of the large puzzle.