Sopact is a technology based social enterprise committed to helping organizations measure impact by directly involving their stakeholders.
Copyright 2015-2026 © sopact. All rights reserved.
A general AI re-guesses every run, so the numbers move. Trust is the opposite: one scoring guide applied the same way to the same data, so the figure is the same on the second look as the first — and a number you can reproduce is a number you can defend.
Because a general AI re-guesses from scratch on every run — so ask it the same thing twice and the numbers move. Trust is the opposite: one scoring guide applied the same way to the same data, so the figure is the same on the second look as the first. A number you can reproduce is a number you can defend.
You’ve collected clean data against your definitions. This step is about making sure the analysis of it holds still — because a report whose numbers change when someone re-checks them loses a funder’s confidence fast.
Key takeaways
Everyone has felt the drift: paste a set of responses into a chat tool, get a clean summary, run it again an hour later, and the counts have shifted. For an internal brainstorm that’s harmless. In a funder report it’s the difference between evidence and a guess.
The fix isn’t a cleverer model — it’s a fixed method. When the same scoring guide is applied the same way to the same stored answers, the result doesn’t wander between runs, and every number carries the words that earned it.
Take a handful of open-ended answers and a simple scoring guide. Run it, then run it again unchanged.
Sort each answer below into exactly one category — ACCESS, COST, TIME, CONFIDENCE, OTHER. Quote the words that justify the choice. If no barrier is stated, mark NOT STATED and don’t guess. Return: answer · category · quote. Then repeat the exact same task again. The two results must be identical, line for line. Answers: [paste 5–10]
If they match, you’ve felt what a trustworthy number is: reproducible, and tied to the words behind it.
On a handful of answers, a general model can look consistent. Push it and the cracks show: on a few hundred responses it starts to drift and drop things, the counts change between runs, and it can’t tell you which answers produced a given number. It’s honest to say a chat tool can give a sharper-sounding summary — but it will occasionally invent detail, and it won’t give you the same figure twice. For a report, same-every-time beats clever.
In Sopact Sense the scoring guide you wrote is applied the same way to every response, held in a durable record you can re-query. The count is the same next week unless the data changed, it holds on thousands of responses, and each figure stays tied to its source. One nonprofit leader described the switch in exactly those terms:
“The reports I’m getting are much more consistent and accurate — the same prompt used to give me very different results. Now I can see where the numbers came from, and they’re consistent from report to report.” — Marco, Open Play Foundation
Reach for it whenever a figure will be quoted, compared across groups, or put in front of a funder. For a rough first pass nobody will check, drift doesn’t matter — use whatever’s quick.
Frequently asked questions
It samples a fresh line of reasoning each run and re-reads the input from scratch, so counts and wording shift. A fixed scoring guide over stored data returns the same result every time.
It trades a little improvised flair for accuracy you can defend. For a funder report, that’s the right trade.
Reproduce it, and point to the responses behind it. Consistency plus a source is what earns trust.
Yes — large sets are where general tools drift most. One fixed rule applied to a durable record avoids that.
Next: Show Funders Where Every Number Came From → · or see consistent analysis in Sopact Sense →
Open Sopact Sense, paste your program description, and put it to work.
Try in Sopact