Sopact is a technology based social enterprise committed to helping organizations measure impact by directly involving their stakeholders.
Copyright 2015-2026 © sopact. All rights reserved.
Ask a general AI the same question twice and you get two different numbers. Reliability is the opposite: a fixed rubric over a durable record, so the same input returns the same graded, cited answer every time — the trait that separates a number you can present from one you can only hope holds up.
Because a general chat model re-reasons from scratch on every run — so ask it the same question twice and you get two different numbers. Reliability is the opposite: a fixed rubric over a durable record, so the same input returns the same graded, cited answer every time. This is the single trait that separates an impact number you can present from one you can only hope holds up.
Everyone has felt the drift. You paste a set of responses into ChatGPT, get a clean summary, run it again an hour later, and the counts have moved. For a blog post that is harmless. For a funder report, it is the difference between evidence and a guess.
Key takeaways
The instinct is to want the smartest possible answer. The more useful goal is the reproducible one. Sopact treats consistency as the product, not a nice-to-have.
“If you write the report, you can ask the report twice, it gets the same result. If you put it in ChatGPT twice, it’s going to give you two different results. Sopact’s job is to give you reliable, traceable results.” — Unmesh Sheth
The mechanism is plain: analysis runs against a rubric you defined, applied the same way to every response, and written to a durable record you can re-query. Because the scale is fixed and the rule is mechanical, the grade does not wander between runs — and because the record persists, the same question returns the same figure tomorrow.
This is not a claim we make about ourselves. It is what an Open Play Foundation leader wrote after moving from a cleverer-but-drifting analysis to a deterministic one.
“The reports I’m getting are much more consistent and accurate — in the context of my recent query about the same prompt producing vastly different outputs. I can more easily gauge where the Assistant got its numbers from, and they’re consistent from report to report.” — Marco, Open Play Foundation
Read that closely: two differentiators in one sentence. The numbers are consistent report to report (reliability), and he can see where they came from (traceability). Those travel together, and neither is something a free-form prompt gives you.
Reliability costs something, and it is worth naming. A model given free rein can produce sharper, more surprising qualitative commentary — and will occasionally invent it. Sopact deliberately trades a little of that flair for answers you can stand behind: predictable and accurate beats clever and drifting, every time a funder is going to check your work.
The fastest way to feel reliability is to try to break it. Grade the same responses twice with a rubric that is deterministic by construction, and compare.
Grade every response to “What was the biggest barrier?” against a FIXED scale — ACCESS, COST, TIME, CONFIDENCE, OTHER. Rules: exactly one label per response; quote the words that justify it; if no barrier is stated, label NOT STATED and do not infer. Return response id · label · quote. Then run it again unchanged. The two tables must be identical, row for row.
When the two tables match, you have something a chat window cannot give you: a number you can defend because you can reproduce it. And because every row carries its quote, “where did this come from?” is answered on the page.
| Situation | Why it matters |
|---|---|
| A figure will be quoted or compared | It must be the same on the second read as the first |
| Large response sets | Where general tools drift most — context limits and re-reading |
| A number must reconcile to source | Every graded value carries the words it came from |
| Situation | Why |
|---|---|
| Brainstorming or first drafts | Drift is harmless when nothing is being quoted |
| One-off exploration | No need for reproducibility if the output is never reused |
Frequently asked questions
General models sample a fresh reasoning path each run and re-read the input from scratch, so counts and wording shift. A deterministic rubric applied to a stored record returns the same graded output every time.
Same input, same rule, same result — a fixed scale, a mechanical grade, a quote required for every score, and NOT STATED instead of an inferred guess.
It trades a little qualitative flair for accuracy and reproducibility. That is a deliberate choice, because an insight you can’t reproduce can’t be defended.
Every graded value carries the source words, and you can ask how a figure was computed and which fields it used — the audit trail is on demand, not reconstructed later.
Yes — that is where general tools drift most. Applying one fixed rule to a durable record avoids the context-window and re-reading problems that cause the drift.
For brainstorming or first drafts, drift is fine. Reach for reliability when a figure will be quoted, compared, or audited.
Next: Back to What Is The Craft · or Try Sopact Sense →
Open Sopact Sense, paste your program description, and put it to work.
Try in Sopact