Analyze survey data in five steps: clean, describe, cross-tab, code the open-ended responses against a fixed codebook, then join themes back to the ratings.
Analyze survey data in five steps: clean the responses and resolve duplicates, describe each closed question with a distribution rather than a mean alone, cross-tabulate by the subgroups you care about, code the open-ended responses against a fixed codebook, then join the themes back to the ratings so every number carries its reason. The last step is the one most workflows skip, and it is the one that makes the analysis usable.
The practical problem is not statistical. It is that the closed questions take an afternoon and the open-ended column takes a week, so the open text gets summarized by whoever has time, or quietly dropped. The result is a report that says confidence fell four points and cannot say why. This page is about running both halves in one pass. If you want the concept rather than the procedure, that is on survey analysis.
Key takeaways
Count how many times a single survey response gets handled between submission and the finished report. In most teams it is four or five: exported, de-duplicated, recoded, read for themes, then pasted into a deck. Each pass is a place where a definition drifts and a number stops matching the one in last quarter's report.
Sopact calls the standard The One-Pass Test: a response passes if it arrives clean, coded against your codebook, bound to a participant record, and available to the report without being touched again. It is a diagnostic rather than a feature — you can run it against any workflow, including a spreadsheet one, and the count tells you where the analysis week actually goes.
The difference is where the work happens. A form-centric workflow collects first and cleans afterwards, so every fix is retroactive and has to be repeated next cycle. A record-centric one validates and codes at the moment of arrival, so the cleaning is done once, at the source. The property that produces — analysis that is finished when collection is — is covered on survey analysis. The stage below runs one closed survey both ways.
Start by cleaning: remove duplicate submissions, decide a rule for partial responses before you look at results, and check that every response can be tied to a respondent and a wave. Deciding the partial-response rule after seeing the data is how an analysis stops being defensible.
Then describe. Report the full distribution for every closed item, not the mean alone — a mean of 3.2 can be a tight cluster around 3 or a program that worked brilliantly for half the cohort and failed the other half, and those need different responses. For Likert items, report the median and the distribution together.
Cross-tabulate by the subgroups that could plausibly differ — site, cohort, demographic, starting level — and check whether a gap is larger than the variation you would expect by chance before describing it as a finding. Subgroup work in depth is in analyze results by demographic subgroup.
Comparing across waves has one hard requirement: match respondents individually. Two wave averages built from different people tell you about the mix of who answered, which is why attrition quietly inflates most reported improvement. The wave mechanics are on longitudinal data collection software.
| You have | Use | What it tells you |
|---|---|---|
| A single closed item | Frequency distribution and mean | The level — where the group sits right now |
| A closed item across two groups | Cross-tabulation with a group difference test | Whether the gap between groups is bigger than noise |
| A closed item across two waves | Paired comparison on matched respondents | Whether individuals moved, not whether averages differ |
| A Likert item | Median and full distribution, not just the mean | Where the mass sits — a bimodal split a mean would hide |
| Many correlated closed items | Correlation or factor structure | Which items are measuring the same underlying thing |
| An open-ended item | Codebook-based thematic coding, then counts | Why the numbers moved, in respondents' own words |
| Open text plus a rating | Theme distribution cut by rating band | Which reason drives which score |
| Responses in several languages | Code in the source language against one codebook | Comparable themes without translation drift |
Write the codebook before you read the responses. Define each theme in one sentence with an inclusion rule and an exclusion rule, code every response against it, mark anything that does not fit as uncoded, then report theme counts alongside the ratings they explain. Themes invented while reading cannot be reproduced, and a distribution you cannot reproduce is not evidence.
This is the step that separates a summary from an analysis. Asking a model to "summarize the themes" returns plausible themes, and asking twice returns different plausible ones — which is not a coding method a reviewer will accept. Anchoring the model to a codebook you defined makes the output fast and reproducible: the same responses coded twice return the same distribution. Cleaning messy free text first is covered in clean open-ended survey responses, and the full procedure in analyze open-ended survey responses.
Sentiment is worth extracting separately from theme, because they answer different questions — theme tells you what the response is about, sentiment tells you how the respondent feels about it, and a single negative theme with mixed sentiment usually means the codebook needs splitting. Keep the verbatim attached to every coded response; the quote is what makes the count credible to a board. Coding across languages without translating first is on multilingual survey analysis.
Join the two halves on the respondent, not on the report: cut each theme by the rating band of the people who expressed it, so you can say which reason drives which score. Reporting a chart section and a quotes section separately is presentation, not analysis.
| Question | Numbers alone | Numbers joined to themes |
|---|---|---|
| What happened? | Confidence fell 4 points | Confidence fell 4 points |
| Where? | Available by subgroup | Concentrated in one site |
| Why? | Not answerable | Transportation and shift conflicts, in 41 responses |
| Who said it? | Not answerable | Named participants, each traceable to a verbatim |
| What do we do? | Discuss it | Change the schedule at that site before the next wave |
Every row after the first depends on the rating, the reason, and the subgroup sitting on one respondent record. That is the whole argument for joining at collection rather than in the report: a survey analyzed this way answers the second question a board asks, not only the first. Instrument design for both halves is on mixed-method surveys, and the wider method comparison on qualitative vs quantitative.
If coding begins after the deadline, the finding lands after the cohort has moved on, and the analysis becomes a record rather than a decision. Reading responses as they arrive keeps the analysis level with collection. That is the premise of the Loop, Sopact's method for continuous impact intelligence: collect clean at the source, analyze the moment data arrives, improve while you can still act.
The Loop is also what makes a survey analysis defensible. Every theme count traces to the response that produced it, and the same codebook returns the same distribution on a second run. That standard has its own chapter in reliability and reproducibility. Where this sits in the wider practice is on impact measurement.
One method, three moves that never stop
Then the cycle runs again, a little sharper each wave. Read the method: the Loop methodology →
The fastest way to test the One-Pass Test is to run it against your last survey and count the passes. Each prompt below pastes into Sopact Sense's Assistant, or reasons through with your team; the arrow above each links the Academy walkthrough that shows the expected output and the tips.
Academy walkthrough → Code open-ended responses against a codebook
Code these open-ended survey responses against my codebook: [PASTE CODEBOOK + RESPONSES]. Return the theme distribution with counts and percentages, three verbatim quotes per theme, and a list of responses you could not classify marked UNCODED. Do not invent themes that are not in the codebook. If a theme in the codebook has zero responses, say so rather than omitting it.
Academy walkthrough → Clean the free-text column first
Clean this open-ended survey column before coding: [PASTE RESPONSES]. Flag blank and near-blank answers, test entries, responses in a language other than the expected one, and answers that respond to a different question than the one asked. Do not delete anything — return the original text with a status column. Return a table: Response ID / Original text / Status / Reason.
Academy walkthrough → Join the themes to the ratings
Join these coded themes to the closed-question ratings on the respondent: [PASTE CODED THEMES + RATINGS]. Show the theme distribution cut by rating band, name the themes that appear disproportionately among low scorers, and give two verbatims for each. State plainly where the sample in a cell is too small to read anything into. Return a table: Theme / Share among low / mid / high scorers / Verbatims.
Academy walkthrough → Cross-tab by subgroup honestly
Cross-tabulate these survey results by the subgroups available: [PASTE RESULTS + SUBGROUP FIELDS]. For each closed question report the distribution per subgroup, flag gaps larger than would be expected by chance, and mark any subgroup whose cell size is too small to interpret. Do not report a gap as a finding where the cell size does not support it. Return a table: Question / Subgroup / Distribution / Gap / Interpretable?
Each walkthrough is short and practical: what to do, the prompt to run, the output to expect, and the tips that keep it reliable.
Watch: running the quantitative and open-ended halves of a survey in one pass instead of two projects.
Clean the responses and resolve duplicates, describe each closed question with its full distribution, cross-tabulate by meaningful subgroups, code the open-ended responses against a codebook written in advance, then join the themes back to the ratings on the respondent. Sopact's diagnostic for the workflow is the One-Pass Test: a response should arrive clean, coded, bound to a participant, and report-ready without being handled again.
The five steps are the same in either spelling: clean, describe, cross-tab, code the open text, join it back to the numbers. The order matters — deciding your rule for partial responses after seeing the results, or inventing themes while reading them, are the two places an analysis stops being defensible. Sopact runs all five on arrival rather than after the survey closes.
For closed questions: frequency distributions, cross-tabulation with group difference tests, paired comparison on matched respondents across waves, and correlation between related items. For open-ended questions: codebook-based thematic coding with counts, plus sentiment extracted separately from theme. The method-selection table on this page maps each question type to the method that fits it. Sopact's position is that the open-ended methods are where most of the unrealized value sits.
Write the codebook first, defining each theme in one sentence with an inclusion and an exclusion rule. Code every response against it, mark anything that does not fit as uncoded rather than forcing it, then report theme counts beside the ratings they explain and keep a verbatim attached to each coded response. Sopact anchors the AI to your locked codebook so the same responses coded twice return the same distribution — asking a model to summarize themes freely returns different themes each run.
Survey coding is assigning each open-ended response to one or more defined categories so free text can be counted. Done properly it is codebook-first: the categories exist before anyone reads the responses, and coders mark unclassifiable answers rather than stretching a category to fit. Sopact treats a coding pass as reproducible only if a second run over the same responses returns the same distribution, which is the test to apply to any AI coding tool.
Join them on the respondent rather than in the report: cut each theme by the rating band of the people who expressed it, so you can say which reason drives which score. Presenting a charts section and a quotes section separately is layout, not analysis. Sopact keeps the rating, the reason, the subgroup and the wave on one record, which is what makes the join available without a merge step.
Report the distribution, and the median for Likert items. A mean of 3.2 can describe a group tightly clustered around 3 or a program that worked well for half the cohort and failed the other half — and those two situations call for opposite responses. Sopact's guidance is to show the shape first and the summary statistic second, because the shape is usually the finding.
Match respondents individually and compare only the matched set, then report the matched change alongside the raw change. Comparing two wave averages composed of different people measures the change in who responded, and because attrition is rarely random — people whose experience is deteriorating stop answering — it usually inflates apparent improvement. Sopact binds every wave to one persistent participant ID so the matched comparison is the default.
In a workflow that fails the One-Pass Test, one to three weeks, most of it spent cleaning exports and reading free text. In a workflow that passes it, the analysis is finished when collection is, because the cleaning and coding already happened on arrival. Sopact treats the gap between those two numbers as the actual cost of a form-centric tool, and it is paid again every cycle.
For closed questions on a single wave, yes — pivot tables handle frequencies and cross-tabs well. Two things break: coding several hundred open-ended responses by hand, and matching respondents across waves when the identifier is an email address that changed. Those are the two tasks that consume the analysis week. Sopact codes the open text on arrival against your codebook and holds the identity across waves, which is what removes them.
Next: read the concept behind the procedure on survey analysis, or design an instrument that supports both halves on mixed-method surveys.
