Academy / Foundations / Lesson 4
Additional course links
You leave with: A response-status table, a small codebook, a theme count with its denominator and a cleaning log a colleague can follow.
Where this fits: Lesson 4 sends you here from the review meeting, when the training team reaches the second half of its question: what stopped the others? You bring back a response-status table, a theme count with its base, and a cleaning log the reviewer can check.
How do you clean open-ended survey responses?
In short: Keep the original answer on its record, give every answer a response status before you look for themes, and make any cleaning changes in a separate working field. Then write the coding rules and name the denominator for every count you report.
The common damage is not a typo left in. It is a short answer deleted as “noise”, a blank treated as “no problem”, or a theme reported as a share of people who were never asked. The export-and-paste path hides it: hundreds of answers beside the numbers overwhelm a general chat model, the same file pasted twice can return two theme lists, and once answers leave their records nobody can check a theme against the words behind it.
Training example · fictional
At the 30-day follow-up, the yes/no question about using the skill is followed by an open one: “What helped, or what got in the way?” Of 40 spring completers, 25 replied: 15 said Yes and 10 said No. The answers from the 10 who said No are the ones that explain what stopped the others. Every answer sits on the learner’s ID beside intake, attendance, mentor notes and exit.
Why keep response status separate from the theme?
In short: Status says whether and how the question was answered. A theme describes what a substantive answer says. One field doing both jobs is how a refusal turns into “no barrier”.
| Answer from a learner who said No | Possible reading | What to record |
|---|---|---|
| “Nothing really” | No barrier named, though the person did not use the skill | Substantive absence; flag the tension for review rather than resolving it |
| “N/A” | Not applicable, misunderstood, or something else | Keep it ambiguous unless your rules or other records settle it |
| “Prefer not to say” | An explicit refusal | Refusal, distinct from a blank; do not invent a reason |
| Empty field | Skipped, interrupted or not shown | Use the form’s logic to tell “not shown” from “shown and left blank” |
| “Nothing would load on the work laptops” | A tool-access barrier | Substantive; a global rule that treats “nothing” as no barrier would be wrong |
Decide missing-value rules before you calculate anything. The 15 who said Yes answered too, but mostly about what helped; count their answers separately, not in the barrier base. The 15 who did not reply to the follow-up are unknown, not “no barrier”. Pew Research Center’s published open-response codebook shows explicit rules for non-substantive answers. Borrow the practice of writing rules down, not its labels.
How do you preserve the source before normalizing?
In short: The original answer stays on the record with its ID, question wording and date. Normalized text, translations, redactions and codes each get their own field.
Trimming stray spaces or matching “work laptop” with “work-laptop” is fine in a working copy. Do not turn an unfamiliar term into a familiar one to make coding easier, and note any fix to a processing error.
Translation is a transformation too. It can be the right analytical step, but it is not the verbatim original. Label it and keep it beside the source. Several of the training team’s answers arrived in Spanish; Multilingual feedback picks up there. For a published quotation, mark whether it is verbatim, translated or lightly edited, and remove identifying details.
What is the right denominator for a theme?
In short: Count statuses first, then themes, and name the base in the sentence. With ten answers, report counts; a percentage of eight looks more precise than it is.
| Status, spring cohort | Count | Treatment |
|---|---|---|
| No reply to 30-day follow-up | 15 | Unknown; outside every barrier base |
| Said Yes, used the skill | 15 | Answers coded separately for what helped; outside the barrier base |
| Said No, described at least one barrier | 8 | Substantive; coded for themes |
| Said No, prefer not to say | 1 | Refusal |
| Said No, left the open question blank | 1 | Item nonresponse; reason unknown |
All 40 completers are accounted for: 15 + 15 + 8 + 1 + 1 = 40. Fictional.
Now code the eight substantive answers. Each answer may carry more than one code, so the themes need not add up to eight.
| Theme | Answers | Base |
|---|---|---|
| No time to practise | 5 | of 8 who described a barrier |
| No chance to use it at work yet (manager, tools, role) | 3 | of 8 |
| Not ready to use it for real (confidence) | 2 | of 8 |
Five answers mention no time. That is 5 of 8 who described a barrier, or 5 of 10 who did not use the skill, or 5 of 25 who replied. It is not 5 of 40, and it says nothing about the 15 whose outcome is unknown. A sentence the reviewer can reproduce: “Of the 10 respondents who had not used the skill at 30 days, 8 described a barrier; 5 of those 8 mentioned no time to practise. One declined and one left the question blank. Fifteen completers did not reply.”
Are identical answers duplicates?
In short: Not by default. Two people can both write “no time”. An upload retry can produce two copies of one submission. Check the ID and timestamp, not the text.
With every answer tied to an ID at collection, the check is quick: the same ID twice within a minute is a likely retry; two IDs with the same words are two answers. Record a reason for anything you set aside.
How do you write a codebook that keeps themes checkable?
In short: Each code gets a definition, what it includes and excludes, and a real example linked to its record. A theme a reviewer cannot trace back to someone’s words is not ready to report.
ONE CODE · EXAMPLE (ILLUSTRATIVE)
You can build codes from the answers, start from a framework, or combine both; state which. Read a varied pilot before fixing the definitions, with long and short answers, both languages and each site. There is no universal number of answers that reveals every theme. Pew’s September 2026 methodology documents a codebook built with people and machine learning together; it shows a documented process, not permission to skip review.
What does AI change, and what stays with your team?
In short: AI can apply your codebook to every answer as it arrives and point each theme at its source. People still write the definitions, review exceptions and decide what the themes mean.
In Sopact Sense, an Intelligence Cell reads each open-ended answer on arrival with a prompt you configure, such as your codebook, and writes the result beside the original on the same record. Field selection keeps names and contact details out of what is sent to AI APIs. When you ask the AI Assistant for the barrier themes, each line of the answer links to a record you can open, so the reviewer checks the words instead of trusting the summary.
Models still miss negation (“the manager was fine about it”) and assign plausible codes on thin evidence. Read every rare code, every code resting on a few words and a sample of the rest. Eight answers can be read in full; four hundred cannot, which is where connected coding pays off. When you revise a definition, test how earlier answers are handled in your setup and keep the earlier output rather than assuming everything was recoded.
A three-minute explainer on why hand-coding stays small and what changes when definitions are applied across connected records. Watch the step where a definition is revised: that is where your cleaning log earns its keep. Watch on YouTube ↗
What goes in a cleaning log?
In short: Enough that a colleague can start from the originals and reach the same counts.
01 · SOURCE
Originals stay on the record with ID, question wording and date
02 · STATUS
Each answer classed: substantive, absence, refusal, blank, not shown
03 · WORKING COPY
Normalizing and translation logged, uncertain terms kept
04 · CODES
Codebook version noted; exceptions and rare codes reviewed
05 · RECONCILE
Counts add back to the population; the base is named
Keep it with the Lesson 4 change log: who approved each rule, and from which date.
What changes when answers arrive all year?
In short: New answers bring new themes. Add them without rewriting what earlier cohorts said, and record the date the codebook changed.
Suppose the next cohort, who get the practice session, write “the practice slot clashed with my shift”. That is a new code. Either re-read earlier answers against it, or mark a break in the trend from the date it was added. Both are defensible if written down; silently applying a new meaning to the latest wave is not.
ASK ANY TOOL, INCLUDING OURS
Bring 50 of your own open-ended answers, including a few blanks, refusals and one-word replies. Ask for a status count before any themes, then for themes with the base stated. A good answer keeps refusals and blanks visible, names the denominator without being asked, and lets you open the original words behind each theme. Paste the same set in twice: the counts should not change.
Try it on your own data
Open your working evidence plan ↗
- Pick one open-ended question from your most recent wave. Write who was shown it, and why.
- Build the status table so every person in the population is accounted for.
- Write two codes with a definition, what they include and exclude, and one linked example each.
- Write the report sentence with its base, and add the cleaning rules to your change log under the Lesson 4 roles.
Check your reasoning
For the training team, the population is 40 completers: 15 unknown, 15 who said Yes (counted separately), 8 substantive, 1 refusal, 1 blank. “No time to practise” is reported as 5 of 8 who described a barrier, with the refusal, the blank and the 15 unknowns stated beside it. The reviewer can open all five answers from their IDs. The finding supports testing a practice session; it does not show that time is the main barrier for everyone.
Questions teams ask
Should “none” be treated as missing?
No. To a barrier question, “none” or “nothing really” can be a substantive report of no barrier. Record it as an absence, not a blank, and read it in context: if the same person also said they did not use the skill, flag the tension for review. It is not proof that a program removed a barrier.
Should “N/A” always be excluded from a denominator?
There is no universal rule. Decide whether the question applied to that person, using routing and other records where you can, and define the base. An unexplained “N/A” may need review rather than automatic exclusion, because excluding it changes who the percentage describes.
Can I correct spelling or translate responses?
Yes, in a labeled working field. Keep the original on the record and note any change that could affect meaning. A translation can be the right basis for coding, but it should never be presented as the respondent’s own wording.
How many responses should I read before creating codes?
Enough varied answers to cover the groups, sites, languages and answer lengths you have. There is no fixed number that guarantees every theme is found. Pilot the codes with a second reader where you can.
Can one response have several theme codes?
Yes, if the codebook allows it. State the counting unit and the base. Overlapping theme counts can add up to more than the number of answers, and percentages can add up to more than 100%. Say so in the table note.