How do you analyze documents as evidence?

In short: Attach each document to the person or organization it describes, read it against a rubric you agreed in advance, record the exact passage and where it sits, and keep what the document says apart from what you conclude.

The training team receives more documents than surveys admit. Resumes arrive with applications. Follow-up interviews produce transcripts. Some employers send a short letter confirming what a learner now does at work. On the survey, Excel, ChatGPT path, these sit in a shared folder, and the evidence inside them never reaches the report.

Picture the funder call. The delivery manager is asked: "Apart from what learners told you, is there any independent sign they used the skill?" An employer wrote about Maria weeks ago, somewhere in a folder of PDFs. She cannot say which learners have letters, what each says, or where. The files exist; the evidence does not, yet.

Why aren't documents treated like long survey answers?

In short: A document answers the author's question, on the author's schedule, in the author's format. It is evidence that a claim was made, not proof the claim is true.

The last row is the honest limit of the whole method. An employer letter saying Maria "now runs client intake interviews" is a record of what the employer reported. It corroborates her own 30-day answer from a second source; it does not measure how well she does it. Treat it as a reported claim, deliberately.

Why does a finding from page 2 have to cite page 2?

In short: Because someone will challenge the finding, and "it's in the letter somewhere" does not survive the challenge. Record the page, section or timestamp and the exact passage.

Citing the passage also stops a quieter failure. When people cite whole files, they paraphrase from memory, and the paraphrase drifts each time it is repeated. "Employer confirms skill use" is a summary. The letter may say Maria "ran two client intake interviews without support in her second week in the new role", which is sharper and more useful than the summary that replaced it.

For a transcript, keep the speaker label and timestamp. For a table in a PDF, keep the row label, unit and any footnote. For a scan, keep the page and look at the image, not only the recognized text.

How do you set this up by hand first?

In short: Two spreadsheet tabs and a filing rule: a register with one row per document, and an excerpt table with one row per passage that matters. Doing it by hand once makes the requirements obvious.

Never paraphrase into the passage column. Your interpretation goes beside it.

Note how far you looked: "No start date found on the reviewed page" is a finding about your review, not proof the date is absent. Read for the register as documents arrive, not for the report the week before; a stack read in one sitting lets the longest and most recent documents shape the story.

What changes when each document is read on arrival against a rubric?

In short: The register fills itself as documents land, using your rubric and your prompt, and a person reviews the proposal with the passage beside it. The judgment stays with the team; the reading stops piling up.

In Sopact Sense, a document uploaded to a learner's record is read by an Intelligence Cell with a prompt your team configures. The prompt carries the rubric, for example your definition of "uses the skill on the job", and asks for a rating, the supporting passage and its location. Because the document sits on ID 0417, the result appears beside Maria's survey answers and mentor note, and the original passage stays behind the rating.

The same approach works for a follow-up interview with a learner who did not use the skill. The rubric asks for the main barrier, the passage and the timestamp: "No time to practise", with the speaker's words at 14:20. One comment is one person's account. It does not show that lack of practice caused the result for others; it tells the team what to ask the next cohort.

For resumes, keep the rubric tied to the program's stated entry criteria, and have a person check before anyone is turned away. AI proposes; people decide.

A rubric prompt to adapt

Keep "not found in reviewed material", "unreadable", "not applicable" and "conflicting" as separate labels. A blank cell should never stand for all four.

Where does document review break?

In short: Backlogs, formats that defeat the reading, and two sources that disagree without anyone noticing.

The backlog. Documents arrive when senders are ready, not when you have time. Reading on arrival is the defence, with a weekly look at what is still unreviewed.

The format. A scanned letter may have no searchable text. A number inside a PDF table may come out without its unit. A long interview may hold the key sentence at minute thirty-eight. A file that could not be read must be marked unread, not returned as empty.

The disagreement. An employer letter dates a learner's new duties before the program ended; the learner's follow-up says she started after. Neither is necessarily wrong. Keep both, record the conflict and ask. Do not average two dates or quietly prefer the one that suits the report.

What should you check when AI reads the document?

In short: Check both the extraction and the inference. A correct quote can sit beside a conclusion it does not support, and a plausible citation can point to the wrong place.

NIST's Generative AI Profile lists fabricated content and citations among the risks to evaluate. Open a sample of citations and every exception before accepting a batch. For scans, compare the rating with the page image. Text recognition is widely available, including in tools such as SharePoint's OCR; the question is not whether a file can be searched but whether a finding can be tied to a person, a period and a passage that a reviewer has checked. Keep the original and the proposed extraction, so a correction does not erase the history.

Documents also carry personal detail. Decide which document fields may be sent to AI and who may quote a transcript before analysis starts; What your assistant may see covers those rules.