Eight checks decide whether evidence stays usable across cycles: self-driven, one record, volume, longitudinal, qualitative, documents, assistant, reliable. Run them on one complete cycle — not a demo of one form.
Connected Data Intelligence is the practice of keeping every piece of evidence — survey answers, open comments, uploaded files, notes, and records from other systems — attached to the person, organization, or event it describes, so the record continues instead of resetting each cycle. It is not a survey tool with better charts. The test is whether one complete cycle survives eight checks: whether your own team controls the instrument, whether each response lands on the right record, whether everyone is covered, whether cycles stay comparable, whether comments explain the numbers, whether files keep their context, whether AI respects consent, and whether a reviewer can reproduce the report.
Key takeaways
In short: they are the eight things that have to hold across one complete collection cycle for evidence to stay usable. Each one is a place where a survey platform normally hands the work back to you.
Read that list as a sequence, not a menu. Checks 01 to 03 decide whether the evidence gets created at all. Checks 04 to 06 decide whether it stays usable once it exists. Checks 07 and 08 decide whether anyone outside your team will accept the result.
In short: the survey market is split between products that are cheap and self-serve but cannot keep a record, and platforms that can keep a record but route the instrument and the analysis through purchased services. Connected Data Intelligence is the category that has to pass both halves.
Start with the cheap half. SurveyMonkey's own help centre states that "it isn't possible to merge 2 survey designs or the results of 2 separate surveys" (SurveyMonkey Help Center, retrieved August 2026). The documented workaround, multi-survey analysis, is a paid feature from the Advantage Annual plan up and does not support Custom Data (Contacts) — which is the field a persistent participant identifier would live in. Its cross-survey export carries a Respondent ID and a separate Survey ID, so following one person across two waves is a join you perform somewhere else. Check 02 fails, and check 04 fails with it.
The same tool's qualitative and document handling is documented just as narrowly. Thematic Analysis works only on Comment box and Single textbox questions, requires the survey to be in English, is capped between 1 and 10,000 responses, is unavailable outside the US data centre, and is not available to HIPAA-enabled accounts (Thematic Analysis). In the cross-survey view, open text appears as a list rather than as themes. Uploaded files are attachments rather than data: PDFs and Word files cannot be previewed in-product, "files aren't included when you export your survey data," dashboards "don't display File Upload data or the file names," and File Upload is one of three question types unsupported in multi-survey analysis (File Upload Question). Checks 05 and 06 fail on the vendor's own documentation.
Now the capable half. Qualtrics publishes no list price; its Pricing & Plans page carries no numbers and meters the platform on "interactions," which it defines as "a data record collected or/and processed by Qualtrics" (Qualtrics Pricing, retrieved August 2026). Configuration is positioned as a purchased service: a Solution Delivery team that "will work according to your program blueprints to deliver a fully-configured program" (Qualtrics XM Services), with Success Checks available only to organizations that have bought a Success Package (Qualtrics Support). Analysis is sold the same way — the Market Research Services page lists open-ended coding, data cleaning, weighting and tabulation among its deliverables, staffed by "200+ expert consultants and practitioners" (Qualtrics Research Services). In FY2022 Qualtrics reported $1,458.6M total revenue against $1,223.7M subscription revenue, implying roughly 16% from professional services and other revenue (SEC 8-K, January 2023).
None of that makes Qualtrics a bad research platform. It makes it a platform whose shape works against checks 01 and 03. If routine changes to the instrument route through someone else, your team is not self-driven. And if the meter is the count of data records, then covering every participant costs more than covering some of them — which quietly converts check 03 from a design decision into a budget decision.
| The check | Response-first tool (e.g. SurveyMonkey) | Research platform sold with services (e.g. Qualtrics) | Connected Data Intelligence |
|---|---|---|---|
| 01 Self-driven | Yes — anyone can build a form | Configuration positioned as purchased delivery | Your team changes questions and rules directly |
| 02 One record | Vendor states two surveys' results cannot be merged | Achievable with configuration | A persistent identifier is the default, not a project |
| 03 Volume | Response caps per plan; per-response overage | Metered on data records collected or processed | Coverage is a design choice, not a line item |
| 04 Longitudinal | Cross-wave joins performed outside the tool | Possible; usually a services engagement | Cycles stay comparable while the instrument evolves |
| 05 Qualitative | English-only, 10,000-response cap, list view across waves | Open-ended coding offered as a service | Themed on arrival, cited to the exact response |
| 06 Documents | Attachment only; excluded from exports and dashboards | Varies by configuration | Files read as evidence, with the passage cited |
| 07 Assistant | Feature availability varies by region and account type | Governed, with configuration | Consent and anonymity rules enforced before AI sees data |
| 08 Reliable | Reproducibility depends on your export discipline | Defensible with documented methodology | Every number traces to the response it came from |
The squeeze in the middle row is the whole argument. Cheap tools let you change anything and keep nothing. Service-led platforms keep everything and let you change little. The organizations in the next section needed both, and none of them were trying to buy a survey.
Run these against one complete cycle of your own work — a real cohort, a real member survey, a real review round. A demo of a single form will pass all eight and tell you nothing.
Your program, research, or learning team should be able to change questions, add a wave, adjust access, and revise reporting rules without a vendor ticket or an internal build request.
Use: One real cycle, and one question you decide to change halfway through.
Pass: The change is made by your team, is logged, and does not orphan the responses already collected.
Every response, comment, file, and note lands on the correct person or organization without asking the respondent to remember who they are.
Use: One participant who responds three times across two cycles, plus one respondent who declines consent to be re-contacted.
Pass: The three responses join automatically; the declining respondent is not linked.
Coverage should be a design decision. If the cost of reaching every participant rises with each response, the instrument will shrink until the response rate is defensible and the finding is not.
Use: Your full population, not a pilot slice.
Pass: Coverage, response rate, non-responders, and missing groups are visible as numbers you can act on.
Cycles have to stay comparable even though the instrument changes. A new question should extend the series, not restart it, and any change to a repeated question should be visible in the output.
Use: Two cycles where one repeated question was reworded and two new questions were added.
Pass: Real change is not confused with a changed question, and the revision is disclosed in the report.
Open comments carry the reason behind the score. They should be themed as they arrive, in the languages people actually answered in, with the original wording preserved and citable.
Use: Supportive, critical, and contradictory comments — in more than one language, with at least one small subgroup.
Pass: Each theme cites the passage it came from, and small-group responses are de-identified before they appear.
An uploaded strategic plan, lab result, transcript, or grantee report is evidence. It should be readable in place and attached to the same record as the survey answers, not filed somewhere a dashboard cannot reach.
Use: Several authorized files of different types, attached to different records.
Pass: A finding drawn from a document cites the document and the passage, and the file stays on the record.
An assistant can theme comments, draft a section, and answer a new question. It should do so under the consent and anonymity rules already attached to the evidence, and it should not be the thing that decides.
Use: A subgroup small enough to re-identify, and one ambiguous comment.
Pass: The output suppresses identity, cites its evidence, states its uncertainty, and routes the decision to a person.
A reviewer who did not build the report should be able to reproduce a headline number and a theme from the underlying evidence.
Use: One finished report and one reviewer who was not involved in producing it.
Pass: Definitions, filters, population, calculation, theme configuration, and source records are all inspectable.
In short: the eight checks keep evidence attached to the right person across time. They say nothing about what a number means. Two organizations can pass all eight and still report different figures for the same programme, because "enrolled" was never defined — and no amount of record continuity fixes that.
There are two independent axes here, and most measurement failures are one or the other:
"Is this the same person, and did their answer survive?"
Identity, continuity, coverage, source, consent, reproducible retrieval. When this fails you cannot tell whether anything changed for anyone.
Governed by the eight checks — this course.
"What does this number actually count?"
Population, unit, period, boundary, calculation, disaggregation. When this fails everybody agrees on the participants and still produces different totals.
Governed by the data dictionary.
Checks 04 and 08 depend on both axes, which is worth being explicit about because it is where teams get caught. A cycle-over-cycle comparison is only meaningful if the measure definition held steady as well as the record — otherwise you are comparing two different questions asked of the same people. And a report is only reproducible if the calculation is written down somewhere a reviewer can read, not carried in the head of whoever built it.
That definition work has its own method, and it is not this course. Start with giving every number one definition to settle what a single measure means, then build the governed data dictionary that holds those definitions as fields with owners, allowed values, and versions. If you also report against external frameworks, mapping one dictionary to IRIS+, GRI, and ESRS covers how to do it by comparing definitions rather than labels.
One terminology note, because the two axes use similar-sounding words for different things. A persistent ID identifies a person across waves — that is check 02. A stable field ID identifies a column across dictionary versions. Both matter and neither substitutes for the other.
In short: the eight checks tell you what to test. Which of them will actually hurt depends on what your record follows — a person you contact again, several people describing one person, many programs you want one picture of, or a network where each member needs their own part. There are four common shapes and they make different checks difficult.
That is a decision worth making before anyone builds a form, and it has its own page: which shape is your data? — with the four shapes side by side, how to tell which one you are, what to do if you are more than one, and a chapter for each.
In short: the eight checks are also a diagnostic. Teams usually describe their problem as "our data is a mess," but the first failing check names it precisely — and each one has a different first move.
| What the team says | The check that is actually failing |
|---|---|
| "Every cycle we start over." | 02 and 04 — there is no continuing record for a new response to join. |
| "We can't get a good response rate." | 03 — the instrument was shortened to fit a cost or a cap, and coverage went with it. |
| "We collect open-ended feedback but never analyse it." | 05 — the comments arrive faster than anyone can read them. |
| "Our reports take a quarter to produce." | 01 — the work routes through someone who is not on your team. |
| "The board asked where a number came from and we couldn't say." | 08 — the calculation lives in a spreadsheet nobody can re-run. |
| "Legal won't let us use AI on this." | 07 — consent and anonymity rules are not attached to the evidence itself. |
All eight are achievable by hand. A careful team with a source map, a real identifier, and a disciplined analyst can pass every one of them for one cohort. The difficulty is repetition: the same eight checks have to hold for the next cohort, the next language, the next reporting period, and the next person who inherits the work.
The pattern is well documented in the sector, if dated. Innovation Network's State of Evaluation 2016, a survey of 1,125 US nonprofits, found that only 8% had staff whose primary responsibility was evaluation, 27% worked with an external evaluator — rising to 49% among organizations above $5M — and 55% used four or more separate methods to store their data, which the report identifies as a direct obstacle to analysis. Limited staff time, money, and expertise were the top three barriers for the third consecutive edition (State of Evaluation 2016). Read it as evidence of a durable structural pattern rather than a current statistic.
In short: the difficult bit is not storing what people tell you. It is sorting each answer out as it comes in — while it is one answer, from one person — so that months later you can look something up instead of reading everything again.
Here is what that means in practice. Someone writes you a paragraph about why they nearly dropped out. Right now that paragraph is just text sitting in a box. It only becomes something you can use once three things are known about it: which of your measures it relates to, what it tells you, and which sentence in it backs that up — all of it filed against the right person.
If that happens as answers arrive, everything afterwards is easy. Someone asks a question in March and you look up the answer. If it does not happen, then every single question means a person sitting down and reading hundreds of paragraphs again. That is the whole reason open-ended feedback gets collected and never used. Not laziness, and not the wrong software. Just nobody has three weeks.
Most of this course sits on top of that one thing. You can compare this year to last year because both years were sorted the same way. You can say what people's comments add up to because they were sorted as they landed, not in a pile at the end. And someone can check your figure because there is something underneath it to check.
Most teams in this space are stalled on a reasonable assumption: that a general AI assistant, the CRM, or the document store already covers it. Each of those solves a real problem well, and none of them is doing this job.
| What teams assume will cover it | What it actually does, and what it leaves |
|---|---|
| An AI chat assistant | Very good at thinking through whatever you paste in. But it works out a number by reading your text, not by counting your records — so ask the same thing next week and the number can come back different. Useful for exploring. Not something to put in a funder report. |
| Your database or CRM | Keeps track of people reliably. But it does not read a paragraph and work out what it means, so comments sit in a big text box that no report can do anything with. |
| Your file storage | Keeps documents safe and easy to find by name. What is inside them stays outside your reporting — so a plan or an interview transcript never sits alongside that person's survey answers. |
Same gap in all three: none of them sorts the answer out as it arrives. And that sorting cannot be switched on in a settings menu, because it depends on your measures, your people and your rules about who sees what. That is why it never comes ready-made.
This sector has run this experiment already, with CRM. Capable platform, real licence spend, and then the discovery that operating it required skills the organization did not have on staff. The usual outcomes were a system nobody fully used, or a permanent consultant line in the budget.
The same thing is happening with AI work now, and one difference makes it worse. A consultant builds you something and leaves. Then a new cohort starts, or the questions change, or the tools change — and the person who knew how it worked is gone. Organisations can spend a lot across several engagements and end up no more able to do it themselves than on day one, because nobody set out to leave a skill behind.
More consultants will not fix that. What fixes it is a set-up your own team can run — where the hard technical work is already done, and what is left is the judgement your programme staff already have.
And to be clear about the part people worry about most: this should not require you to write prompts, or to learn how AI works. If getting an answer out of your own data means someone on staff becoming good at phrasing instructions to a chatbot, that is not a solution — it is a new dependency wearing a friendlier face. The right test is whether a programme manager who has never thought about AI can ask a question in plain English and get an answer they can defend. That is what check 01 is really asking. A capability your own team cannot run is a capability you are renting.
You do not need to know how any of this is implemented to establish whether it holds. The tests are already above, and every one of them is observable from the outside:
Judge it on those. People still decide what to collect, review sensitive interpretations, correct identity errors and own the conclusion — the system's only job is to make sure the number in the report was computed rather than guessed.
Take the last report your team produced and run check 08 on it. Hand it to a colleague who did not build it and ask them to reproduce one headline number from the source records. Time how long it takes and note what they had to ask you for. Whatever they could not find on their own — the definition, the population, the filter, the calculation, the responses — is the first check you are failing, and it is almost never the one the team expected.
It is the practice of keeping evidence from surveys, comments, files, notes, and other systems attached to the person, organization, or event it describes, so the record continues across cycles. In practice it is verified by eight checks on one complete cycle: self-driven, one record, volume, longitudinal, qualitative, documents, assistant, and reliable.
A survey platform is organized around the response; this is organized around the record. The difference shows up at check 02. If a tool's own documentation says two surveys' results cannot be merged, it can collect data for years without ever producing a comparable series — which is a different product, not a cheaper one.
Qualtrics is a capable research platform, and much of this is achievable on it. The constraint is shape rather than capability: it publishes no list pricing and meters on data records collected or processed, while its own documentation positions configuration and open-ended coding as purchased services. That works against check 01 and check 03 specifically.
Because check 03 is about coverage. When each additional response has a price, the rational move is to survey fewer people or ask fewer questions, and both decisions weaken the finding. One global body received 285 responses from 1,600 competitors and described the result as enough to be useful but not enough to rely on.
Yes, for one cohort. A team with a persistent identifier, a documented instrument, and an analyst can pass all eight by hand. The problem is the second cohort, the second language, and the second reporting period — repetition is what turns a method into a system requirement.
By theming at the group level rather than attributing to individuals, and by de-identifying before analysis where a subgroup is small enough to be recognizable. A single respondent from a category with one member is identifiable no matter what the report says, so the suppression rule has to be applied to the evidence, not to the chart.
It should. A strategic plan, transcript, or grantee report often carries the explanation the survey question missed. Check 06 asks whether that file can be read in place and cited on the same record — in some tools uploaded files are excluded from exports and dashboards entirely, which means they are storage rather than evidence.
Stay in Connected Data Intelligence when the problem is collection, identity, sources, access, or history. Move to Impact Measurement & Reporting when the problem is frameworks, definitions, interpretation, funder context, or what you are allowed to claim.
In short: don't read this course front to back. Run the eight checks, find the one that broke first, and start at the chapter that resolves it.
| If this check failed | Start here |
|---|---|
| 02 One record | Collect evidence offline and in the field — how a persistent identifier survives a field visit with no connectivity |
| 03 Volume | Survey attrition — who dropped off — find who is missing, compare responders against non-responders, and grade the sample before reporting |
| 04 Longitudinal | Track one person's change across years, then analyze pre / mid / post data and measure how long outcomes last |
| 05 Qualitative | Clean open-ended responses at the source, then connect quantitative and qualitative data. Working across languages? Analyze multilingual feedback |
| 07 Assistant | 360 feedback software — the clearest case of reading comments while protecting anonymity, because both are the same problem |
| 08 Reliable | Write a cited impact narrative — every claim carries a number and the words behind it |
Two checks do not yet have a chapter of their own. 01 Self-driven and 06 Documents are being written, because they are the two that most often decide a purchase and the two most survey platforms handle worst. Until then, the tests in the section above are enough to run them yourself against a live cycle.
If your problem turns out not to be collection at all — if the evidence is connected but you cannot agree what it means or what you are allowed to claim — that is a different course: Impact Measurement & Reporting.
ChatGPT, Claude, and Gemini are fine for a quick test — but not for an answer you'll put in front of a funder or board. When it has to hold up, run it in Sopact Sense.
Try it in Sopact →
