How do you analyze open-ended survey responses?
Analyze open-ended survey responses by reading the text, defining what you want to learn, grouping relevant ideas, checking those interpretations against the original answers and reporting patterns with clear coverage. Keep the question wording and useful response context attached throughout the process.
For an operational survey, a documented codebook can help a team organize recurring topics consistently. It is one practical approach, not the only way to conduct qualitative analysis. An exploratory research study may require a different analytical method and deeper interpretation.
This guide works through a small fictional response set, then shows how to apply the same discipline to a larger dataset with AI assistance. For question design, see open-ended survey questions.
1. Define the question your analysis must answer
“Summarize the comments” is too broad to guide a decision. “Which registration difficulties should we investigate before the next intake?” gives the analysis a purpose. It also defines its limits: comments may identify reported difficulties, but they cannot alone establish why someone abandoned an application.
Record the survey audience, dates, question wording, who was shown the question and who answered it. Note whether a comment was optional or shown only after a particular rating. A response set collected only from dissatisfied customers has a different meaning from one offered to everyone.
Keep an untouched copy of the source data. In the working copy, document excluded duplicates, test submissions or unusable entries. A short negative answer is not automatically low quality. Do not rewrite spelling, translation or grammar over the original evidence.
2. Read for meaning before fixing categories
Read a varied selection: short and long answers, different ratings, groups, languages and periods. Note ideas that recur, unexpected experiences and contradictions. Starting with a few known business categories is reasonable, but allow the evidence to change them.
For example, “registration problem” may initially seem sufficient. On reading the responses, you might find separate issues with opening the form, understanding eligibility and receiving confirmation. Those distinctions matter because different teams would address them.
Also distinguish an issue from the tone used to describe it. “The staff were kind, but I still could not upload my document” contains positive interpersonal feedback and an unresolved technical problem. A single positive or negative sentiment label would lose useful meaning.
3. Build a codebook reviewers can use
A codebook is a set of category definitions with guidance on when to use each one. For this example, codes describe reported registration issues. More than one code may apply to an answer.
Scroll horizontally to see all columns →
| Code | Include | Do not include |
|---|---|---|
| Form access | Difficulty opening or loading the form | Unclear wording in a form that opened successfully |
| Eligibility clarity | Uncertainty about who can apply or what evidence is required | A clearly understood decision that the respondent dislikes |
| Document upload | Difficulty attaching or submitting a file | A question about which document is needed |
| Confirmation | No confirmation, unclear completion status or uncertainty that submission arrived | A later decision delay after receipt was confirmed |
| Other issue for review | A relevant concern not covered by current codes | Blank answers |
Add an example and a version date to each definition. Decide whether the unit is the whole response, a passage or a person’s combined answers. If the report counts responses mentioning a theme, repeating the same theme three times in one answer still counts as one response.
“No issue reported” can be a meaningful answer. Blank, unclear, out of scope and no issue should not become one catch-all category. Keep enough distinction to understand the response coverage.
For additional methodological guidance, the CDC’s qualitative analysis chapter describes systematic reading and refinement of a codebook as part of the analysis. Choose an approach appropriate to the question and data.
4. Code a small set and inspect the result
The table below is fictional. It illustrates a review process rather than customer results.
Scroll horizontally to see all columns →
| Response | Original text | Codes and interpretation |
|---|---|---|
| A | The form would not open on my phone. I used a colleague’s laptop. | Form access. The respondent found a workaround. |
| B | I submitted everything but could not tell whether it had arrived. | Confirmation. Do not assume the submission actually failed. |
| C | I did not know which document was needed, then the upload kept failing. | Eligibility clarity; Document upload. Two separate issues. |
| D | The instructions were clear and I received confirmation immediately. | No issue reported. Positive experience in the described steps. |
| E | It took too long. | Clarification needed. The answer does not identify which step or why. |
For C, retain the passages supporting each code. For E, resist assigning a plausible cause without evidence. An analyst can report that the response describes delay, but should not turn it into “slow approval” when the respondent did not say that.
Have another reviewer examine a varied set where team consistency matters. Discuss disagreements and revise unclear rules. Agreement is useful feedback on this coding task; it is not proof that the interpretation is the only valid one. If the codebook changes, decide which earlier responses need recoding.
5. Use AI assistance with a review plan
AI can propose codes, group similar responses and help locate source passages. Before using the results widely, test the configured workflow on examples your reviewers have examined. Include multi-topic answers, negation, irony, vague responses, mixed sentiment and the languages actually collected.
Review both mistaken assignments and missed themes. A model that labels every comment “service issue” may look consistent while failing to provide a useful distinction. A confident explanation is not evidence that the code fits.
- Keep the original text and the question it answered.
- Record the codebook and analysis configuration used.
- Make reviewer corrections distinguishable from initial suggestions.
- Retain a category for uncertainty rather than forcing every answer into an existing theme.
- Review new topics and important concerns during the collection cycle.
- Recheck performance when wording, languages or the response population changes.
A fixed codebook helps define expectations. It does not by itself guarantee identical AI output on every run. Test repeatability and preserve the reviewed result used in a report. If the system or definitions change, document the change before comparing outputs.
For sensitive or urgent matters, follow the organization’s established response arrangements. Do not depend on sentiment analysis to detect every serious concern. Restricted source access and an appropriate review process remain necessary.
6. Count themes with an explicit denominator
Suppose 500 people were invited, 200 completed the questionnaire and 80 wrote an optional comment. Of those 80 comments, 24 mention confirmation and 16 mention document upload; eight mention both.
- Confirmation appears in 24 ÷ 80 = 30% of comments.
- Upload appears in 16 ÷ 80 = 20% of comments.
- Either issue appears in (24 + 16 − 8) ÷ 80 = 40% of comments.
The last calculation avoids counting the overlap twice. If the report instead counts mentions, label that measure and explain that one response can contribute several mentions. These percentages do not establish the prevalence among all 500 invited people or the wider customer population.
Show comment coverage: 80 of 200 completed questionnaires included a comment. If some people submitted several responses, do not relabel response counts as unique people without an appropriate deduplication rule.
7. Read themes alongside ratings and context
Ratings can help organize a review, and comments can help interpret what respondents experienced. Keep both at the submission level so a reviewer can see which comment accompanied which rating. An anonymous survey can support this within-submission connection without collecting identity.
For group comparisons, check how many people in each group were asked and how many commented. A branch with 20 comments out of 25 responses and another with 5 out of 100 should not be treated as equally represented in a theme comparison.
For individual change over time, a suitable persistent identifier may be needed, with appropriate permission and access. For repeated anonymous group surveys, compare the group-level results and explain changes in composition. Do not describe these as the same people improving.
When local questionnaires differ, retain local wording and map only sufficiently similar concepts into shared categories. A question asking “What went wrong?” and one asking “What helped?” do not provide equivalent opportunities to mention a problem.
8. Report a finding that a reader can check
A useful finding includes the question, the observed pattern, the denominator, representative evidence and limits. It then identifies the decision or investigation that follows.
Example: “Of 80 respondents who left a registration comment, 24 mentioned uncertainty about confirmation. Several described completing the form without knowing whether it had arrived. The team will test the confirmation message and delivery logs before the next intake. Optional comments may overrepresent people with strong experiences.”
Use quotations that illustrate the pattern without exposing identifying details. Include exceptions when they affect interpretation. Do not select only the most dramatic comment, or use one positive quote as evidence that the whole audience was satisfied.
For a wider report structure, see survey report examples. For an outcome-focused report, use the impact reporting ebook and report examples. A well-presented report still needs claims matched to the evidence available.
Make the next reporting cycle easier to maintain
Keep the original response, question version, period, relevant context, reviewed codes and publication decision connected. Assign owners for updating definitions and approving what leaves the team. Store stable participant or organization details only where the workflow needs them, and keep access proportionate.
Sopact’s relevant approach combines collection, analysis and governance so the operating team can maintain this process as responses and sources grow. Evaluate the configured workflow with your real data: inspect a theme’s source, correct a classification, compare a later period and check who can see the underlying comments.
A spreadsheet may be sufficient for a small one-off analysis. A recurring multi-source workflow can justify more structured support. Compare the total work involved in imports, review, definition changes and reporting, rather than assuming that either manual work or AI software is always the right answer.
How Sopact reduces coding and reporting work
Compare the complete recurring workflow: define the codebook, apply it, revise it and ask across coded text and numbers. The ownership cost includes every return to that work.
A workflow with repeated manual work
- Define from an initial sampleRead material and agree on the codebook.
- Apply it across the datasetCode responses and check the result.
- Revise a definitionReturn to affected material and recode it.
- Reconnect the numbersReconcile coded results with ratings and context, then rebuild the view.
The Sopact workflow
- Your team owns the definitionsDecide what each code means and improve it as you learn.
- Apply coding across the eligible dataAutomate application; people review quality and exceptions.
- Reprocess after a definition changesReapply the revised definition across the configured scope instead of recoding each response by hand.
- Ask across coded text and numbersKeep the response, rating and relevant record context connected; inspect the evidence behind the result.
This compares workflow patterns, not a claim that every research tool requires manual coding or separate files. Some already automate parts of this work; compare the complete cycle.
For this codebook-based workflow, the main saving is repeated application and reconnection—not the removal of human judgment. A changed definition can be reapplied across the configured data while reviewers concentrate on quality, exceptions and interpretation. Coded text stays connected to the relevant ratings and context.
Count the recurring work in ownership cost. Include setup, coding, recoding after revisions, source reconciliation, review and reporting, plus your actual platform and processing expenses. A worked scenario of four cycles of 4,000 responses illustrates 272 fewer annual staff hours; it is an assumption-based example, not a customer benchmark. Existing automation, review needs and implementation effort can substantially change the result.
Adjust the workload assumptions and compare total effort →
A reliable assistant should calculate from the selected records and let a reviewer open the supporting evidence. Check the data scope, definition, denominator and access permissions. Reproducible arithmetic does not make every AI interpretation correct.
Watch: Why Qualitative Analysis Stays Small — And How to Scale It
See why revising a codebook creates repeat work, and how connected coding and quantitative analysis change that workload.
Watch the reporting companion
This Sopact video discusses reporting from collected evidence. Use it alongside the practical guidance above.
Frequently asked questions
Do I need a codebook before reading the answers?
You can begin with known topics, but read a varied set before finalizing definitions. Allow unexpected ideas to change the codebook, and document revisions.
Can one response belong to several themes?
Yes. Define the coding unit and counting rule. Theme percentages can overlap when the same response contains more than one theme.
Can I analyze responses in Excel?
Yes. A spreadsheet can support a small analysis with source text, codes and review notes. Evaluate additional software when recurring collection, multiple sources, permissions or reporting create substantial maintenance work.
Is sentiment analysis the same as thematic analysis?
No. Sentiment concerns expressed tone or evaluation; themes concern the topics or meanings being discussed. A positive comment can still describe an unresolved problem.
Does a quote prove the cause of a low rating?
A quote supplies the respondent’s account. It can suggest an explanation to investigate, but it does not independently establish causality.
Can I compare themes across survey waves?
Yes, when question meaning, populations, coding definitions and coverage are sufficiently comparable. Document changes and avoid interpreting a different respondent mix as individual change.
Choose your next step
King Center
See how a team makes open-ended program feedback available for review and learning.
Read the customer story →