Data collection tools strategy: Tips for successful implementation
April 24, 2017
Did you miss Part One of this blog? Read it here. Social impact measurement cannot be done without collecting impact data. And we cannot collect good impact data without effective data collection tools.
Of course, it is much easier said than done. But we've got some data collection tips for you to help turn that talk into action. There are a number of tools and practices that help simplify the process -- all without compromising the results.
This blog will continue the series on lean impact measurement by examining the third step: Data collection. More specifically, we will map out specific processes for acquiring data and how to implement them. We will also focus on existing tools, such as software applications, which will help make data acquisition easier and more effective for all stakeholders involved.
As in previous posts, Aravind Eye Care System will be used as a case example to help us move through the discussion in an applied manner. Up to this point with Aravind, we have completed some essential pre-work identified how to establish appropriate metrics and have reviewed creating a data collection strategy.
With this foundation, we now enter the implementation phase. It’s time to collect data answering the most essential question for purpose-driven organizations -- how much impact are we really creating?
Nuances of social impact programming, target market, and other context-specific characteristics will determine specific implementation needs.
However, at a high level we can follow a framework which makes those nitty-gritty decisions easier to arrive at, with more confidence that we are being comprehensive in doing so.
Since our brains internalize information better using mnemonics we’ll use such an approach to break down best practices for impact data collection. First, take a look at the “5 right principles of data collection” framework outlined here.
More specific to our Aravind case we’ll explore the five W’s in this blog: Who, What, Where, When, & Why. We’ve already looked at the “why” in a previous blog, so we’ll cheat a bit and simplify to four W’s. Framing implementation around the following gives structure to the process and a resource for ongoing accountability to that structure.
Stakeholders, both internal and external, will need to be engaged and managed. Essentially, the Who is about the human channels you will tap into for data acquisition. Borrowing from IDEO’s tools for human-centered design, it can be helpful to create “personas” for each stakeholder involved in your impact measurement chain to map out their capacities and needs. Also consider if your data sample is indicative of your overall beneficiary group (e.g., do the 150 survey respondents sufficiently represent the 2,000 participants who received the intervention?).
In the Who for Aravind’s case, we have the beneficiary who received the treatment, the data collector, the hospital administrator, etc. What might they need to be successful in this impact measurement journey? Their success means quality data and plenty of it.
To some extent this has been covered in a previous blog, but in addition to the metrics you will use, it is important to establish and utilize appropriate research methods to capture those metrics. In Aravind’s case, and looking at the metric “patient perception of healthcare delivery,” Aravind may want to employ a mixed methods approach, using survey questionnaires that have been validated with existing research and/or a structured interview to capture qualitative data.
What they use depends largely on the Who they have established. Will patients be open/available to do an interview? Does Aravind have people trained to conduct interviews?
In the simplest sense, where is the data being acquired? We’ll need to explore the daily activities of those personas examined in the Who section. We may also want to consider existing research investigating how the place of data collection influences the quality of data acquired. In Aravind’s case, looking at the “percent regaining employment” metric, our implementation could occur in the beneficiary community -- perhaps even as a visit to the place of employment. However, this could be unfeasible for a number reasons, and if so, an in-person survey could be conducted at the hospital during a routine follow-up. The latter might be more cost-effective and yield a larger sample size.<p>There is no right or wrong in this example, it is important to balance these characteristics so that sufficient, reliable data can actually be acquired.
The time period in which a data point is collected could have a huge influence on the eventual results. When to probe for impact data will depend on the nature of the metric you are trying to measure and the type of intervention delivered.
In Aravind’s case, and using the same metric as above, they will want to establish a time period (or more than one) in which they will poll patients about regaining employment after the cataract eye surgery. Their own market and stakeholder research will guide the specifics, but transparency throughout will support reliability of responses and data.
These W’s begin to paint a solid portrait of on-the-ground implementation. The next primary question is how you can leverage tools to facilitate data collection and to manage those data you have collected. Luckily, there is an abundance of resources for that.
Our metrics data journey is advancing. We have identified who is carrying out this process, which research methods they are using, and when and where they implement them. Now, which hard tools should be used to acquire data? And how can those data be managed once acquired?
To do this, we can leverage a variety of technological tools. There are already many advanced tools for comprehensive and integrated measurement and assessment, and which solve for challenges faced in data collection and reporting, from internet connection limitations to data management.
As a starting point, check out this impressively comprehensive resource by the Foundation Center. It includes more than 150 tools and resources for assessing social impact. Identifying foundations or categories in that database which are similar to yours will likely yield something useful for your own measurement needs.
More specifically in this blog, we’ll examine data acquisition tools for mobile data collection (MDC). Because in addition to (or in combined with) survey instruments and other resources previously discussed, we can also turn to a bevy of mobile applications that make engaging beneficiaries easier for all parties.
MDC is ideal for gathering simple quantitative data, for example, through surveys delivered and answers via mobile phones. If appropriate for your data type and respondent, MDC has a number of benefits:
Cost reduction - less administrative overhead, field workers, etc.
Efficiency - respondents can be reached faster and data returned much quicker (digitally)
Reliability - point of collection errors reduced, digital data is theoretically easier to “clean” and back up
Positive environmental impact! - less need for paper, travel, etc.
For the right kind of data and population these benefits are considerable. In choosing a specific tool to implement a MDC process we have quite a few options. The following list provides a few examples.
While not an exhaustive list, researching these platforms and their various functionalities will give a sense of the available options and where they may fit in with your data collection needs.
If still feeling overwhelmed with MDC tools, check out this guide from Magpi for some further guidance. World Bank blog also provides a balanced overview of some issues to be aware of in the MDC field.
The next question is, where and how do you store that data that you are receiving? While some of the above applications offer management tools, the following graphic details some further options based on differing levels of data management needs.
One or perhaps a combination of tools will be necessary. Researching and selecting a tool, training staff, and creating effective workflows with these tools may seem daunting.
However, in the long-term doing so will almost certainly make your data management more efficient (cost-effective), your data analysis better streamlined (time-saving), and your eventual impact conclusions more reliable.
But the best part of it, you’ll have a clear understanding of the impact you are creating.
And no matter that conclusion, you will be better positioned to leverage those insights into improved outcomes for your organization and those you serve.
Learn More: Impact Measurement
Topics: impact measurement
Alan is a social sector consultant and one of the founding directors of Quantica Education, a school of social entrepreneurship in Colombia.