A funder-ready narrative pairs every figure with a verbatim quote. Here is the prompt that grades each claim green, amber, or red so nothing unsupported ships.
This chapter resolves check 08 Reliable of the eight checks.
The annual report is due Friday. Three hundred and forty participants finished the endline survey, and 71 per cent of them reported more confidence than at baseline. The programme officer needs a voice to sit beside that figure, so she opens the open-ended column, sorts it by length, and reads the twenty longest answers. One is four hundred words, beautifully structured, and genuinely moving. It goes into the report directly under the 71 per cent. Illustrative example.
Nothing in that is dishonest. But the number came from 340 people and the story came from one, and she found that one by sorting for length. Two claims now sit side by side on the page looking like they came from the same place, and they did not.
Because your number describes everyone and your story describes whoever writes well. The statistic is computed across the whole cohort. The quote is chosen — and it is chosen for how it reads. Length, fluency, emotional force and a satisfying arc are what make a response quotable, and none of them has anything to do with how many people share that experience. The reader cannot see the difference, so the mismatch never gets challenged. It just leaves a faint sense that the stories in impact reports are a bit too good.
Selection is happening, and it is happening at the moment of reading rather than in the analysis, which is why it escapes review. Sampling gets scrutinised. Quote-picking does not.
Put those four together and you have a filter that reliably selects the most articulate, most engaged, most comfortable participants in the programme — the people least like the ones you were funded to reach.
The fix is a change in the order of operations. Today the sequence is: write the finding, then go looking for a quote that supports it. Reverse it. Theme the responses first, count how many responses sit in each theme, and let the counts decide which themes get a voice in the report at all. Then, inside the chosen theme, pick the response that states it most plainly — not the one that states it best.
And then say the quiet part out loud on the page. A quote with a count beside it is a different kind of evidence from a quote alone:
Unpaired
71% of participants reported increased confidence.
"Before this programme I could not look a stranger in the eye. I now chair a committee of eleven people and last month I spoke in front of the district council…"
Paired
Of the 340 participants who completed the endline, 71% reported more confidence than at baseline. Asked what changed, 96 of them described speaking up in a group. One wrote:
"I can talk in the meeting now. Before I just sat."
The second version is less stirring and much harder to argue with. The quote is doing a specific job: it is showing what those 96 responses sound like. It has stopped being decoration and started being a sample.
This is a discipline before it is a feature, and it is worth running by hand once so you can see what it costs.
Done carefully on a few hundred responses, this works. There is nothing proprietary in it.
It breaks in three places, and they all break the same way — quietly, under deadline.
The count and the quote come from different places. The 71 per cent lives in a survey export. The quotes live in a document somebody made while reading. Nothing joins them, so nobody can check whether the person quoted is even inside the group the number describes. By the time the report is designed, the two are separate objects on a page and the link exists only in one person's memory.
Length bias returns whenever time is short. The method above assumes someone reads the whole theme. In the last week before a board deadline, they read the top of the sorted column, exactly as before.
The quote pool ossifies. Without a log, the same handful of participants supply the human voice of the organisation year after year. Their outcomes drift away from the cohort's — they are the engaged ones — and the reports get steadily more optimistic than the data, with no single decision anywhere responsible for it.
The narrow thing a system does here: because every response stays attached to the participant record it came from, a theme in Sopact Sense carries its own count and its own citations, so a quote can be pulled with the number of responses it represents already attached, and the original wording is preserved so what appears in the report is what the participant typed. Which theme matters, which quote reads honestly, and what the finding means are still your judgements to make.
Use: Your last published report. Take every participant quote in it and try to answer two questions for each: how many responses shared that theme, and is the quoted person inside the group the neighbouring statistic describes.
Pass: Every quote has a theme count you can recover from the data, every quoted participant sits inside the denominator of the claim beside it, and no participant is quoted more than once.
Fail: You cannot tell how many people a quote stands for. Or the two most powerful quotes turn out to be the same two people you used last year.
Do this on a report that has already gone out. Testing it on the one you are still writing lets you fix the quotes instead of seeing the pattern.
It is a richer one, and it tells you more about that person. It does not tell you that more people feel the same way. Length correlates with education, free time, comfort writing in the survey's language and confidence that the response will be read — all of which vary systematically across a cohort in ways your programme probably cares about.
Yes, and you should when theirs is the typical response. A plainly worded quote in a report signals that the selection was not made on style. Quote it exactly as written; do not tidy the grammar to make it presentable, because the moment you rewrite it you are quoting yourself.
The size of the group the quote is representing — how many responses fall in that theme — and the base it came from. "96 of 340 endline responses described this" is enough. Avoid attaching a quote to a statistic it has no relationship with, which is the most common version of this error.
Then say so and give the count without a quote. "The most common single response to this question was some version of 'the bus cost too much' — 96 responses" is a legitimate and rather strong sentence. A theme does not need a voice to be reported; it needs a number.
No. Use it, and label it as what it is: one person's account, not an illustration of the cohort. The damage is not caused by publishing a remarkable story. It is caused by placing it under a whole-population statistic so the reader reads it as the typical case.
No. A composite is a sentence no participant said, presented in quotation marks. If you need to convey a shared experience in one line, write it in your own voice as a summary with a count, and quote one real response beneath it. The distinction is small on the page and total in terms of what the report can be held to.
It concentrates the bias, because responses in the report's language need no work and everything else does. Keep the original text with the translation attached to it, quote the original alongside, and check whether your quotes are drawn from the language distribution of the responses or just the convenient part of it.
There is a useful side effect to keeping the quote log. After two or three reporting cycles it stops being an admin file and becomes a small dataset about your own organisation: whose words get carried to funders, boards and websites, and whose never do. If the same six people keep appearing, that is not a writing habit. It is a description of whose voice your systems can reach — and it is measuring something no survey in your programme was designed to ask about.
ChatGPT, Claude, and Gemini are fine for a quick test — but not for an answer you'll put in front of a funder or board. When it has to hold up, run it in Sopact Sense.
Try it in Sopact →
