How do you clean open-ended survey responses?
Preserve the original response, record its response status, and make any cleaning changes in a separate working copy. Then define the coding rules and the denominator for each reported percentage. Cleaning should make the data usable without silently changing what respondents said or who counts in the result.
Use this reference when your analysis module includes open-ended answers. You will produce a cleaning log, a response-status table and a report statement a colleague can reproduce. Document analysis is a separate optional reading, not a prerequisite.
Consider a fictional program asking, “What barriers did you face?” Some people describe several problems; others write “none.” Deleting the short answers before analysis can make the remaining responses look more negative. But “none” does not prove the program removed a barrier. It reports an absence in the context of the question, period and respondent’s account.
Keep response status separate from the theme
Response status describes whether and how the question was answered. A theme describes the content of a substantive answer. Do not make one field do both jobs, and do not assign meaning from the word alone when the question or surrounding response changes it.
| Response | Possible interpretation | What to record |
|---|---|---|
| “None” to “What barriers did you face?” | No barriers reported for the question’s stated period | Substantive absence, if the context supports that reading |
| “N/A” | Not applicable, misunderstood, or another meaning | Use the instrument rules; retain ambiguity if applicability is not established |
| “Prefer not to say” | An explicit refusal to provide this answer | Refusal, distinct from a blank; do not invent a reason |
| Empty field | Skipped, not shown, interrupted or missing in export | Use routing and collection records where available; otherwise unknown |
| “None of the buses ran on time” | A transport problem | Substantive text; a global replacement of “none” would be wrong |
Decide your missing-value rules before calculating a rate. A blank after a skipped branch is not the same as a displayed question left unanswered. “No comment” may be a refusal under your codebook, but it does not establish that the question was sensitive or that the person understood it.
Pew Research Center’s published open-response codebook is an example of explicit coding instructions, including rules for non-substantive responses. Its categories fit its research question; use the practice of documenting rules rather than copying its labels into an unrelated study.
Preserve the source before normalizing
Keep the original text with its response ID, question version, date and permitted record link. Use separate fields for normalized text, translated text, redactions and codes. If you correct a processing error, retain the change history instead of making the source appear never to have contained the error.
Trimming accidental outer whitespace can be a useful working-copy change. Standardizing case may help matching, but test whether case carries meaning in abbreviations or identifiers. Treat “busfare” and “bus fare” as variants only when their meaning is clear. Do not convert an unfamiliar term into a familiar one just to make coding easier.
Translation and readable transcription can be appropriate analytical steps. They are transformations, not verbatim originals. Label them, retain their connection to the source where permitted, and review consequential ambiguity with someone competent in the language and context. Continue with the multilingual feedback lesson for that workflow.
For a published quotation, distinguish verbatim text from a translated or lightly edited quotation. Protect confidentiality and avoid displaying unnecessary identifying details. Retaining an original for authorized review does not mean publishing it without redaction.
Do not delete identical responses automatically
Two people may both answer “transport.” Those are two responses, not necessarily duplicate records. Conversely, an upload retry may produce two copies of one submission. Check submission IDs, timestamps and the collection process before removing anything.
Keep a deletion or exclusion reason in the working dataset. If a suspected duplicate cannot be resolved, flag it and explain how the analysis handles it. Text similarity alone is not proof of a duplicate person or fraudulent response.
Practice: calculate the denominator before writing the headline
In this fictional exercise, 100 people were invited and 80 returned the survey. Everyone who returned it was shown the barrier question. Of those 80, 40 reported at least one barrier, 20 explicitly reported none, eight gave a response confirmed as not applicable under the study’s rules, seven declined and five left it blank.
| Status | Count | Interpretation |
|---|---|---|
| At least one barrier reported | 40 | Substantive answer |
| No barriers reported | 20 | Substantive absence |
| Confirmed not applicable | 8 | Separate response status under the study rules |
| Explicit refusal | 7 | No substantive barrier answer |
| Blank after being shown the question | 5 | Item nonresponse; reason unknown |
| Survey not returned | 20 | No survey response from these invitees |
Among the 60 substantive answers, 40/60 = 66.7% reported a barrier. Among all 80 returned surveys, 40/80 = 50% reported a barrier. These are different reporting bases, not competing calculations of the same quantity. Neither says what all invited people experienced, because some did not answer the question or return the survey.
A transparent sentence is: “Of 80 returned surveys, 60 provided a substantive barrier answer; 40 of those 60 (66.7%) reported at least one barrier. Twenty reported none. The other returned surveys contained eight not-applicable responses, seven refusals and five blanks.”
Now suppose 24 of the 40 barrier-reporting respondents mention transport. That is 24/60 = 40% of substantive answers, or 24/40 = 60% of respondents reporting any barrier. Name the base. If a person can receive several theme codes, theme percentages need not sum to 100%.
Keep survey nonresponse separate from item nonresponse. Also distinguish people, responses and mentions: a participant may mention transport three times in one answer, but that does not make three people reporting it. Record the counting unit in the codebook.
Create a codebook without pretending fifty answers is enough
Start with a varied pilot set that includes different periods, groups, languages, long answers and short answers. There is no universal rule that fifty responses reveal every important theme. The appropriate review depends on diversity, complexity and the purpose of the analysis.
You can develop codes from the responses, start from a defined framework, or combine both. An existing theory or reporting requirement is not inherently a bias to discard. Make the approach explicit and leave a route for unexpected evidence rather than forcing every response into a planned category.
For each code, write a definition, inclusion and exclusion rules, and examples. Specify whether multiple codes are allowed and whether the counting unit is a respondent, response or passage. Pilot the rules with another reviewer where appropriate, discuss disagreements and revise the codebook before scaling.
Pew’s September 2026 methodology describes a model-assisted codebook process combining human interpretation and machine learning. It demonstrates a documented analytical workflow; it does not mean an unreviewed model output is sufficient for a different project.
Use a cleaning log that another person can follow
- Preserve the source and identifiers. Retain the original response and question context in the approved storage environment.
- Classify response status. Apply documented rules for substantive answers, absence, applicability, refusal and missingness.
- Normalize a working copy. Log transformations and retain uncertain terms for review.
- Check duplicates using record evidence. Do not remove responses solely because the text repeats.
- Apply the versioned codebook. Keep codes separate from source text and record the coding version.
- Review exceptions and a varied sample. Inspect disagreements, rare categories and consequential classifications.
- Reconcile counts and report the base. A reviewer should be able to reproduce the denominator and understand exclusions.
Use the shared-definition lesson to put the codebook and denominator rules beside the quantitative measures. Documenting the rules supports comparison across waves, but does not guarantee comparability if the question, population or collection method changes.
Plan the recurring coding and reporting work
Your cleaning log and codebook should remain usable when new responses arrive. Plan the effort of revising definitions and recalculating the same clearly stated reporting bases.
A workflow with repeated manual work
- Define from an initial sampleRead material and agree on the codebook.
- Apply it across the datasetCode responses and check the result.
- Revise a definitionReturn to affected material and recode it.
- Reconnect the numbersReconcile coded results with ratings and context, then rebuild the view.
The Sopact workflow
- Your team owns the definitionsDecide what each code means and improve it as you learn.
- Apply coding across the eligible dataAutomate application; people review quality and exceptions.
- Reprocess after a definition changesReapply the revised definition across the configured scope instead of recoding each response by hand.
- Ask across coded text and numbersKeep the response, rating and relevant record context connected; inspect the evidence behind the result.
This compares workflow patterns, not a claim that every research tool requires manual coding or separate files. Some already automate parts of this work; compare the complete cycle.
For this codebook-based workflow, the main saving is repeated application and reconnection—not the removal of human judgment. A changed definition can be reapplied across the configured data while reviewers concentrate on quality, exceptions and interpretation. Coded text stays connected to the relevant ratings and context.
Count the recurring work in ownership cost. Include setup, coding, recoding after revisions, source reconciliation, review and reporting, plus your actual platform and processing expenses. A worked scenario of four cycles of 4,000 responses illustrates 272 fewer annual staff hours; it is an assumption-based example, not a customer benchmark. Existing automation, review needs and implementation effort can substantially change the result.
Adjust the workload assumptions and compare total effort →
A reliable assistant should calculate from the selected records and let a reviewer open the supporting evidence. Check the data scope, definition, denominator and access permissions. Reproducible arithmetic does not make every AI interpretation correct.
Watch: Why Qualitative Analysis Stays Small — And How to Scale It
Watch this 2-minute 58-second explanation of repeated coding, revised definitions and connected analysis. Then estimate the work for your own collection cycle.
What changes when responses arrive throughout the year?
A new response may reveal a category your first codebook did not contain. Decide whether the change requires recoding earlier responses, mapping compatible versions or reporting a break in the series. Do not silently apply a new meaning only to the latest wave.
Keep the earlier coding output and the revised one when reprocessing, with the version and review date. A new theme can be added without erasing the original words. Check whether old responses meet the new definition rather than assuming that they do.
In a configured Sopact workflow, test how responses are analyzed on arrival and how the team reviews classification, supporting text and exceptions. Confirm what happens when the rubric changes, including whether earlier responses are reprocessed. Do not assume automatic backfilling or a correct denominator without testing the actual workflow.
A practical customer context: the King Center
The King Center’s published story describes using pre- and post-survey evidence across seven programs and making qualitative feedback useful for review. For a team doing similar work, this lesson’s contribution is a transparent route from original response to code and reported count. The story does not validate the fictional percentages above.
Test the workflow before using it in a report
Build a small test set containing a valid “none,” an ambiguous “N/A,” a refusal, a skipped branch, a blank, repeated text from different people and a genuine upload retry. Add an answer with two themes and one whose meaning depends on language or context.
Write the expected status and counting rule for each. Then add a later batch with a new theme. Pass when originals remain available to authorized reviewers, uncertain cases stay visible, versions are recorded and the reported counts reconcile. A model confidently assigning every answer to a category is not a pass if the underlying rules are wrong.
Watch: keeping qualitative evidence connected
Watch the video · 2 minutes 34 seconds. This companion explainer connects qualitative evidence with continuing records. Use the worked example above to check the cleaning and reporting decisions. Browse more videos in the video library.
Frequently asked questions
Should “none” be treated as missing?
For a barrier question, it may be a substantive report of no barriers. Interpret it using the question and context. Do not assume every occurrence means zero, or that it proves a program caused improvement.
Should “N/A” always be excluded from a denominator?
No universal rule applies. Determine whether the item was applicable and define the reporting base. Keep the response status visible; an unexplained “N/A” may need review rather than automatic exclusion.
Can I correct spelling or translate responses?
Use a labeled working copy, retain the source where permitted and document consequential changes. A translation can be useful, but should not be presented as the original wording.
How many responses should I read before creating codes?
Use a varied pilot appropriate to the dataset and purpose. Fifty is not a universal sufficiency threshold. Review across groups and periods, refine definitions and check whether important evidence remains uncaptured.
Are identical answers duplicates?
Not necessarily. Different respondents may use the same short answer. Check submission and collection evidence before removing records.
Can one response have several theme codes?
Yes, if the codebook allows it. State the counting unit and denominator; overlapping theme percentages may add to more than 100%.
What happens when a new theme appears later?
Version the codebook and decide how earlier responses will be reviewed or compared. Keep a record of recoding and disclose any break in comparability.
What is the difference between cleaning and analysis?
Cleaning prepares and documents the usable dataset. Coding and analysis interpret content under defined rules. Keep those decisions visible so a cleaning operation cannot silently become an unsupported finding.
When your feedback includes several languages
Take the source text, working copies and codebook into the multilingual feedback lesson. You will examine how language and translation affect interpretation without losing the original context.
Lesson reviewed September 12, 2026. All numerical examples are fictional; the King Center reference is drawn from its published customer story.