The need for social impact measurement tools is becoming more and more apparent, especially in the philanthropy and impact investment sectors.
Each delivers on a unique value proposition while also fundamentally serving the purpose of recording data. Once the data arrives, however, it is up to the data collector to analyze, build useful reports and tell compelling stories.
This is where the limits of the above tools are most felt. Organizations using a "master" Excel sheet for summarizing results from multiple Excel sheets are clearly going to face many challenges. Let's see why and explore how they might be overcome.
Things to keep in mind when using survey tools
1. Surveys are just one piece of a puzzle
Often, surveys unveil just a slice of data, just a piece of the full insight you can be garnering through your impact measurement efforts. True social impact demands a hybrid approach and depends on the impact thesis of an organization or program. Ask yourself:
- Is our ecosystem investment based on outcome alignment with investors?
- Is program investment based on program effectiveness (efficiency in implementation)?
- Is program investment based on demonstrating social change?
An impact thesis helps an organization define a strategy which covers operational capacity, scorecards, stakeholder assessment, and general alignment of required criteria that demonstrate risk and effectiveness.
Often organizations have to develop custom analysis processes to bring all the data together to understand program effectiveness, social change, and outcome alignment.
Without an integrated analysis, superficial survey data can pose a danger to data integrity.
2. Survey vs. Results Collection
Surveys are useful in gathering one-dimensional data from stakeholders, such as baseline data, and pre and post feedback data. This process should not be confused with collecting results. At a high level, there are two types of results-gathering from organizations or partners:
1) Output and outcome-oriented results
2) Process-based results with or without a scorecard
While the first allows for understanding of overall outputs & outcomes of an initiative, the second is relevant for understanding program effectiveness and risk.
Often organizations use well-defined standards or custom metrics/indicators to collect results. A simple survey cannot collect RESULTS.
3. Better data analysis requires better relationships
Social impact results-gathering and analyses tend to be unique to each organization. High precision results can be achieved when metrics are assigned to programs based on logical relationships.
However, it is up to you to design your data structure, often done using tools like Excel which unfortunately do not represent well the relationship with Program, Partner, Product, Project, Applicant, etc.
A well-designed impact management platform (such as the Impact Cloud shown above), on the other hand, allows you to collect data from the online or offline surveys and index them in the database for a future reference. This will ultimately help streamline your reporting without creating islands of data stories.
4. Pitfalls of using irrelevant systems for impact management
Many of the grant management software solutions available include impact measurement applications. Unfortunately, most of them enable only the survey-like data gathered from grantees/investees.
Survey respondents are likely to respond based on a size of an investment or based on local compliance requirements before providing true results to the funder. A funder-designed the survey is likely to to be limited to assessing solely for program objectives.
Usually, grantee impact is unique and survey questions are not always relevant to them and their impact context. Other times they simply don't have the necessary data capacity either (a system which tracks activity, outputs or outcomes).
Grantees may also feel that they have already provided answers in a similar survey (frequent survey syndrome) and believe that another survey with similar questions does not serve a useful purpose.
5. Surveys fail to understand deep impact
To understand actual impact risk you must aggregate results based on an integrated stakeholder approach.
For example, if a foundation is funding selected non-profits and wants to understand if recipient organizations are improving volunteer capacity or not, survey tools can measure volunteer management but then have to be compared against an indicator like volunteer opportunity.
Once results have been acquired, all the data manipulation usually happens in a tool like Excel, with customized pivot tables and data analysis. As the years and number of organizations increases, an Excel-based approach becomes increasingly messy and unmanageable.
In other words, it is difficult to leverage just survey data to tell a compelling impact story. What do you need to tell an effective story? We recommend:
- Collecting data from partners and beneficiaries/clients that are relevant to them.
- Aggregate data in such a way that offer unique impact insights to programs, partners, or projects with logically similar entities.
- Provide consistent and easy to use impact insights for different types of data aggregation in a single location.
Leveraging social impact measurement tools requires all the relevant pieces to complete the puzzle. We need to be using a system like the Impact Cloud to measure impact because it provides a solution for the complete lifecycle of qualitative and quantitative metrics, and accommodates different types of results collection based on unique impact theses.
It also contains a comprehensive social impact metrics database combined with data collection and analysis tools. And because it is cloud-based, stakeholders at all levels are empowered to be a part of the impact accountability process.
Thus, while survey tools can perform a useful function, their limits must be recognized. They are simply one piece of a much larger impact measurement puzzle.