What is social impact analysis?
Social impact analysis is the work of examining evidence about how a program, investment or policy affects people and communities: what changed, for whom, how much, compared with what, and why, including harms and effects nobody planned. It turns collected results into conclusions a decision can rest on.
The order of the work matters more than the software. Analysis goes wrong when numbers are compared before anyone checks that they count the same thing, or when a chart is explained without asking the people behind it. The OECD’s evaluation guidance treats impact as all significant effects, positive or negative, planned or not, so a success story is one part of the evidence, not the whole.
THE SHORT VERSION
- Check definitions first: only numbers that count the same people, in the same window, can be added or compared.
- Then compare, with earlier cohorts, with peers on the same definition, and with outside data such as local wages for the same job.
- Then ask why, from people’s own words, and state a claim no stronger than your design supports.
How is social impact analysis different from measurement and reporting?
Measurement decides what to count and collects it; analysis examines what the counts and comments mean; reporting tells a reader the findings and what happens next. The three feed each other, and analysis is where a weak definition or a missing voice shows up.
Scroll horizontally to see all columns →
| Activity | Main task | What it produces |
|---|---|---|
| Measurement | Agree what change matters and how each metric is defined | Metrics, definitions, collection plan |
| Analysis | Examine patterns, comparisons and explanations | Findings that name their people, period and limits |
| Reporting | Explain findings and decisions to a reader | Results, uncertainty and next actions |
When analysis finds that a definition is ambiguous, fix it for the next cycle and say so in the report. Do not quietly rewrite earlier evidence to fit the conclusion you hoped for.
How do you conduct a social impact analysis?
Work in seven steps, and do them in order: question, definitions, data preparation, description, comparison, explanation, conclusion. Skipping ahead to comparison is the most common way analyses mislead.
Scroll horizontally to see all columns →
| Step | What you do | Question to ask |
|---|---|---|
| 1. Ask a narrow question | Name the group, the change and the period | Could the evidence answer this? |
| 2. Check definitions | Hold each metric against its dictionary row | Does every number count the same thing? |
| 3. Prepare the data | Remove duplicates, check dates and units, mark missing values | Unique people, or visits? |
| 4. Describe | Counts, rates and spread, with denominators | How many answered, of how many? |
| 5. Compare | Earlier cohorts, peers, outside data | Same definition, place and period? |
| 6. Explain | Read open answers and interviews beside the numbers | What do people say got in the way? |
| 7. Conclude | State the finding, its limits and the next action | Is the claim as strong as the design? |
“What is our impact?” is too broad for step 1. “How many people we trained were placed in a job within 90 days of exit, and how did their starting wages compare with the local range?” can be answered. A theory of change suggests which questions matter; ask the people affected which changes they care about too.
The video below is Sopact’s introduction to impact measurement and management in the age of AI. Watch it with step 7 in mind: an analysis is finished only when a finding reaches a decision, and people, not the AI, make that call.
Why must definitions come before any comparison?
Because two numbers with different definitions answer different questions, and adding or ranking them produces a figure that describes nobody. Step 2 is where you catch this, before it reaches a chart.
Take a fictional regional workforce fund with four job-training partners, A to D, used as the worked example on this page. Its agreement defines “placed in a job” as within 90 days of exit. Partners A, B and D reported 42, 31 and 27, a portfolio total of 100. Partner C reported 55, counted within six months.
C’s 55 is not wrong; it answers a longer window. Adding it would have made 155 placements that match no definition, so the fund held it and asked C for the 90-day count. The check is a data dictionary: one row per metric saying what it counts, how it is broken down and whether it can be added across partners.
The Who column matters for analysis as much as the count. Splitting by gender, age and disability shows who a program reached and who it missed, which a single average hides. The chapter A shared data dictionary shows how to write each row.
What should you compare social impact results with?
Compare with three things, each on the same definition: your own earlier cohorts, peers counting the same metric, and outside data such as the local wage for the same job. A result alone tells you what happened; a fair comparison gives a board the context to judge it, though not the cause.
Earlier cohorts are the cheapest comparison, as long as the definition did not change between years. Peers work when a metric shares a dictionary row or a standard code. Outside data answers the question your records cannot: what do people in this job, in this area, usually earn?
For the workforce fund, the agreement asks for starting wage, hourly, by track. Each track trains for a job with an occupation code, and the BLS Occupational Employment and Wage Statistics publish hourly wages for that code by state and metropolitan area. The chapter Compare your results with outside data walks through the full match.
Scroll horizontally to see all columns →
| Check | Workforce fund (fictional) |
|---|---|
| Our number | Starting wage, hourly, by track, for people placed within 90 days |
| Who is in it | The 100 people placed by A, B and D; C held until it splits wage by track |
| Outside number | BLS hourly wage, 25th percentile and median, same occupation |
| Known difference | BLS covers all workers in the job; ours covers new hires only |
Because new hires usually start below the median, set each track beside the 25th percentile as well. Partner C’s single average wage cannot enter the comparison at all, since it mixes tracks.
How do you find out why results changed?
Read what people said, in their own words, beside the numbers they belong to. Numbers show the size and spread of a result; open answers and interviews show what helped, what got in the way and effects the metrics missed.
Keep each comment attached to the person’s record, so you can read the answers of people who were not placed separately from those who were. Look for disagreement and surprises, not only confirmation. Count people, not passages: ten quotes from two people are two voices. The mixed-methods analysis guide shows how to set numbers and words side by side without hiding disagreement.
In Sopact Sense, an Intelligence Cell reads each open answer as it arrives, with a prompt your team configures, and the AI Assistant answers questions with every line linked to the record behind it. Field selection keeps names and emails out of what AI receives. Whatever tool you use, check every theme against the answers it came from.
PROMPT · PASTE INTO CLAUDE, CHATGPT OR YOUR AI TOOL
You are helping us analyze one program result. No names or contact details are included. Inputs: 1. Our data dictionary rows for the metrics involved: [PASTE] 2. The result table, with counts per group and the number who answered: [PASTE] 3. Open answers to "What got in the way?", each tagged with outcome (placed / not placed) and group: [PASTE] Tasks: 1. Check each number against its dictionary definition. List any that use a different window, unit or population, and do not combine them with the rest. 2. Describe the result with its denominator and the count of people with no answer. 3. Group the open answers into themes. For each theme, give the number of distinct people, not passages, and two short quotes. 4. Say where the themes differ between placed and not placed. Rules: do not invent or estimate numbers. If something is not in the inputs, write "not in our data". Do not claim the program caused any change.
The video below makes the case for mixed methods: numbers show what changed, open answers and interviews suggest why, and the two only combine if they were collected with the same context. Watch for where that context comes from.
How strong a claim can a social impact analysis support?
Only as strong as its design: description shows what was observed, a before-and-after comparison shows change, and only a planned comparison group or similar design supports a claim about cause. Outputs, such as people trained, are never evidence of outcomes on their own.
Outside data gives context, not proof. If graduates earn above the local 25th percentile, report it, but the people who enroll may differ from the typical local worker, and wage estimates describe a year already past. Self-reported placements and wages, and people who stop answering follow-ups, add further uncertainty.
A number that traces perfectly to its records can still rest on a biased sample or a weak measure. AI can sort and summarize; people decide what a finding means, after checking it against the record.
Start with one result and one comparison
Pick one outcome you already report and take it through all seven steps once, with one comparison, before building anything larger.
- Write one narrow question, naming the group, the outcome and the period.
- Find the metric’s dictionary row, or write one, and check every number you plan to use against it.
- Count who answered and who did not, and put that denominator beside the result.
- Choose one comparison: an earlier cohort on the same definition, or one outside figure such as the local wage for the same job.
- Read the open answers of people with the weakest result, and note the themes by number of people.
- Write three sentences: the finding, its main limit, and the decision it informs.
After the first cycle you have one checked result, one fair comparison and the reasons behind it, which is the core of any impact report.
Frequently asked questions
What makes social impact analysis credible?
A narrow question, definitions checked before any comparison, denominators shown beside every rate, comparisons made on the same definition, and conclusions no stronger than the design. Traceability helps a reviewer confirm each number, but it does not fix a weak measure or a biased sample. Name who is missing from the evidence, and report unwelcome findings along with the rest.
What are the main methods of social impact analysis?
Descriptive analysis of counts and rates, before-and-after comparison on matched people, comparison with earlier cohorts or peers, comparison with outside data, and qualitative analysis of open answers and interviews. Causal designs with comparison groups sit on top of these when a decision needs them. Most program teams combine two or three methods to answer one question.
Can pre and post surveys prove social impact?
They can describe change among people who answered both times, if both answers are linked to the same person. On their own they cannot separate the program’s effect from other influences, such as the job market or who chose to enroll. Make causal claims only when a comparison group or another planned design supports them.
Where can we find outside data to compare with?
For workforce programs, the BLS Occupational Employment and Wage Statistics give wages by occupation and area, the Department of Labor publishes employer enforcement records, and ProPublica’s search covers IRS Form 990 filings. Match definitions before comparing, name the area and reference year, and treat the result as context rather than proof.
Should funders see individual participant records?
Not by default. Give funders aggregate findings, the definitions behind them and a way to verify them. Access to identifiable records should follow the purpose, the permissions people gave and your confidentiality rules. When you use AI on the records, choose which fields it may see and keep names and contact details out.
