When outcome data is incomplete, begin by separating what you observed, what you can calculate and what remains unknown. A benchmark may help test a planning scenario; it cannot supply missing evidence of what happened to your participants or partners. The first useful output is an evidence-gap plan, not necessarily an impact estimate.
This lesson is for portfolio and program teams facing an incomplete report. Bring the requested claim, available records and reporting boundary. You will prepare an evidence table, examine a simple missing-data bound and choose the next collection action.
Define the question before filling gaps
“We need an impact number” does not tell you what evidence is missing. Specify the outcome, population, period and decision. Is the audience asking about current delivery, observed change, contribution or monetary value? Each requires different information.
Missing records, incompatible definitions and an unsuitable evaluation design are different problems. Collecting another survey may help one while leaving another untouched. Identify the problem precisely before requesting more data.
Separate observations, calculations and assumptions
| Evidence type | Example | How to report it |
|---|---|---|
| Observation | A participant reports employment at a stated date | Name the source, date and verification status. |
| Calculation | 36 employed among 60 known statuses | Show the denominator and formula. |
| External benchmark | A published rate from another population | Record its source and relevance limits. |
| Scenario assumption | An assumed outcome rate for planning | Keep it separate from evaluated results. |
| Unknown | No usable follow-up for 20 people | Retain the gap rather than replacing it with zero. |
An observed outcome is not automatically measured causal impact. Likewise, observing a change does not turn a monetary proxy into a directly measured financial value.
Use a missing-data bound where the question permits it
A fictional program has 80 eligible participants. Sixty have a known binary employment status at the specified follow-up, and 36 are employed. Twenty statuses are unknown.
Among known statuses, employment is 36/60, or 60%. Coverage is 60/80, or 75%. Across all eligible participants, 36/80, or 45%, are confirmed employed. If every unknown status were employed, the total would be 56/80, or 70%.
The 45–70% range is a bound under those stated definitions and assumptions about the missing binary statuses. It is not a confidence interval, a forecast or an estimate of the program’s effect. It also does not resolve errors in the records classified as known.
Do not automatically extend this technique to measures such as income with no justified finite range. The form of the question determines whether a simple bound is useful.
Examine who is missing
Compare available, appropriate characteristics of respondents and non-respondents without inferring sensitive attributes. Review contact validity, language, accessibility, timing and whether follow-up was actually due. A low response count may reflect several different problems.
At portfolio level, retain unassessed partners in the inventory. Show their known scope, reporting status and evidence gaps. An assessed subset should not silently become the entire portfolio.
Use compatible shared definitions across partners. If one reports attendance and another reports distinct people, that is a definition problem, not just a missing cell. Preserve both and resolve the comparison before aggregation.
Use benchmarks for a clearly labeled purpose
An external study may help frame an expectation or scenario. Review its population, context, period, measure and method before borrowing a figure. Record where your situation differs.
Do not multiply a sector outcome rate by your enrollment count and describe the result as measured outcomes. A benchmark SROI ratio applied to an investment amount is also a modeled scenario, not evidence that the funded activity created that value.
When a scenario is useful, show the assumed value and alternatives beside the calculation. Keep it separate from observed results, and explain the decision it informs. Transparency is necessary but does not make an unsuitable assumption valid.
Choose the next collection action
Prioritize the gap that most affects a real decision, considering burden and whether the information can still be obtained. A missing reference date may be resolved through clarification. An absent historical baseline may not be recoverable; redesign future collection rather than presenting recall as an original baseline.
| Gap | Possible next action |
|---|---|
| Wrong-period attachment | Request the relevant document or confirm why the earlier source was provided. |
| Unknown follow-up status | Use the agreed contact process and record remaining nonresponse. |
| Undefined shared measure | Agree unit, population, period and mapping before requesting another total. |
| Unsupported contribution claim | Narrow the wording and review what evaluation evidence would be needed. |
One new field does not automatically turn weak evidence into a strong finding. Record what the action can resolve and what remains uncertain.
Use Sopact to make gaps actionable
Connect each available response, document and clarification to the right person or organization and period. Configure the dictionary and analysis to distinguish missing, not yet due, conflicting and reviewed evidence. Assign an owner and next action for unresolved issues.
AI can help organize the evidence and surface candidate gaps. Ask it to use supplied sources, preserve unknowns and avoid inventing outcomes, proxies or confidence scores. Review the resulting table before it informs a report.
A useful prompt:
Practice: write the limited finding
- State the question and eligible population.
- Classify the available evidence using the table above.
- Reproduce the fictional result and missing-data bound.
- Write a finding that preserves coverage and avoids a causal claim.
- Assign one feasible next action and explain what it will improve.
Frequently asked questions
Should benchmarks fill missing outcome records?
No. Keep those records unknown. A suitable benchmark can inform a separately labeled planning scenario, with its assumptions and limits visible.
Can we report when data is incomplete?
Often yes, with a clearly bounded claim and visible gaps. Whether an estimate is suitable depends on the question and evidence, not merely on adding a caveat.
Does an organization’s own outcome data prove impact?
No. It supports statements about observed outcomes within its quality and coverage limits. Causal contribution requires additional reasoning and evidence.