What is program evaluation?
Program evaluation is a planned, systematic look at a program to answer specific questions about its design, delivery, results or value, so that someone can make a decision on evidence rather than impressions. It is broader than counting activities, and it does not always need to estimate causal impact.
Start with who will use the finding and what they will do with it. A team improving delivery next quarter needs different evidence from a funder deciding whether to renew a grant, and both differ from a board asking whether the program caused the change.
THE SHORT VERSION
- Choose the type of evaluation by the question and the decision: process for delivery, outcome for what changed, impact for what the program caused.
- Write the questions from your theory of change, level by level, and agree how each outcome is defined before any data is collected.
- Say what your design can claim: an outcome evaluation describes change for the people you reached, and only a comparison speaks to cause.
What are the main types of program evaluation?
Five types cover most needs: formative, process, outcome, impact and economic evaluation, and each answers a different question and supports a different kind of claim. The last column is the one teams most often skip.
| Type | Typical question | Possible evidence | What it can claim |
|---|---|---|---|
| Formative | How should the program be designed or improved? | Needs assessment, interviews, pilot findings | What to change before scaling |
| Process | Was it delivered as intended, and to whom? | Delivery records, observation, participant accounts | How the program ran |
| Outcome | What changed for participants? | Defined measures per person at set time points | Change among people reached |
| Impact | What change is due to the program? | A design with a comparison or counterfactual | Contribution or attribution, within limits |
| Economic | How do costs relate to results or alternatives? | Defined costs plus outcome evidence | Value, under stated assumptions |
The categories overlap, and one evaluation often combines two. Name the actual question and design rather than relying on a label to explain what the evaluation can conclude.
How do you write evaluation questions from a theory of change?
Read the theory of change level by level: the delivery levels give process questions, the outcome levels give outcome questions, and the arrows between them give the why and how-much questions an impact evaluation takes on. The model tells you which questions matter; the decision tells you which to answer first.
Take a fictional job-training partner of a regional workforce fund, the running example in the course Measurement and reporting. Its pathway: training and mentoring → certification → placement within 90 days → retained at 12 months → living-wage jobs. Its reporting agreement already names the outcomes, which spares the evaluation a definitional argument later.
Process questions sit on the first boxes. Did every trainee get a mentor? Did people who enrolled finish the track, and who dropped out, at which point and why?
Outcome questions sit on the outcome boxes. How many graduates were placed in a job within 90 days of exit, and how many were in the same job at 12 months? What was the hourly starting wage on each track, and who is missing from each count?
Impact and economic questions sit on the arrows: would these graduates have found similar jobs anyway? Do employers hire because of the certification? What does each placement cost, set against the wage it leads to?
If you do not have a model yet, the course chapter One change model, four formats builds one level by level, with the assumption under each arrow. The assumption you trust least is usually your best first evaluation question.
The video below walks a skills program from activities to impact in about seven minutes. As you watch, sort each result it names into process or outcome; that sorting is the first step of an evaluation plan.
What does an outcome evaluation need before it starts?
It needs agreed definitions and records that follow each person, because an outcome evaluation can only compare results counted the same way, for the same people, over the same window. Outputs can be counted at the end; outcomes have to be measured per person over time.
Definitions decide what an evaluation can say. When the workforce fund added up placements within 90 days, Partners A, B and D reported 42, 31 and 27, a total of 100. Partner C reported 55 placed within six months, a different window, so the fund held that number rather than add it.
An evaluator would face the same choice. Mixing windows makes the total larger and the finding wrong, so the definitions have to be settled in the agreement, before collection. The chapter Roll up and benchmark portfolio results shows the full roll-up.
Per-person outcomes also need the same person at each point: intake, exit, the placement survey and the 12-month follow-up. In Sopact Sense, each participant keeps one ID from the first form and later surveys add to the same record, so the evaluation starts from linked data instead of matching spreadsheets by name.
How do you know whether the program caused the change?
You need a comparison that shows what would likely have happened without the program; a before-and-after change on its own cannot separate the program from everything else that moved. A stronger local job market, for instance, can lift placements whatever the training did.
Comparison groups formed before the program starts give the strongest answer, and usually need an evaluator. A program team can still do honest, lighter comparisons: earlier cohorts measured the same way, or outside data on the same definition, such as a county median wage for the same job. The chapter Compare your results with outside data covers when that is fair.
Choose the claim before the design: “graduates were placed at this rate” needs good outcome data. “The program contributed to placements” needs outcome data plus evidence on the arrows. “The program caused this many placements” needs a counterfactual design, planned before collection.
The video below compares the two designs most teams choose between, following the same people over time or comparing groups at one point, and asks which one funders accept as evidence the program caused the change.
What steps does the CDC framework recommend, and what goes in the plan?
The 2024 CDC framework organizes evaluation around assessing context, describing the program, focusing the questions and design, gathering credible evidence, generating supported conclusions and acting on findings, with collaboration, equity and use running through all of it. Read the current framework.
Treat it as a planning structure, not a rigid sequence; something found during collection may send you back to the design. The plan itself can be short. Here is the template, filled in for the fictional workforce partner.
| Plan element | Decision to record | Workforce example |
|---|---|---|
| Purpose | Who uses the findings, for what decision, when | Fund and partner, grant renewal, next year |
| Program | What work, population and period | Training and mentoring, one year of cohorts |
| Questions | Which uncertainties matter most | Placement within 90 days; retention at 12 months |
| Design | What evidence and comparison can answer them | Outcome evaluation, earlier cohorts as context |
| Collection | Sources, timing, responsibilities | Placement survey quarterly; follow-up at 12 months |
| Analysis | Calculations, coding, quality checks | Unique people, by track, missing named |
| Use | How findings are discussed and acted on | Annual review; one change agreed |
Add the budget, access needs and the main limits. A question that needs evidence you cannot get should be narrowed, not quietly answered with what you have.
PROMPT · PASTE INTO CLAUDE, CHATGPT OR YOUR AI TOOL
Below is our theory of change and the decision our evaluation must inform. Draft evaluation questions and a short plan. Rules: 1. Sort questions into process, outcome and impact, and say which level or arrow of our model each one comes from. 2. For each outcome question, state the definition it depends on (unit, time window, who is counted), using only definitions in our materials. 3. For each question, say what design could answer it and what claim that design supports. Do not suggest a causal claim without a comparison. 4. Do not invent numbers, baselines, targets or findings. Where something we need is missing, write "not in our data". 5. End with the one question you would answer first for our decision, and why. Our decision: [PASTE] Our theory of change: [PASTE] Our reporting agreement or indicator list: [PASTE, OR WRITE "NONE"]
Where does evaluation fit in monitoring, evaluation and learning?
Evaluation is the third step of a yearly loop: agree what to monitor, monitor inside the work, evaluate what changed, for whom and why, then decide what changes next cycle. Monitoring feeds the evaluation its data, and learning feeds the findings back into the program.
Here is how a finding travels. Suppose the partner’s completion is high but placements within 90 days lag on one track. A process look finds that track has few employer contacts, so the fix is outreach to employers, not more training hours.
That finding guides delivery; it does not estimate how many placements the program caused. Write each conclusion with its evidence, interpretation and limit, then agree the action, an owner and a review date. For the monitoring half of the loop, see using a theory of change in M&E.
What can a program evaluation not tell you?
An evaluation can only be as strong as its design and its data, and most program evaluations describe change without proving cause. A satisfaction score, however high, cannot stand in for a skills assessment or a job.
Non-response is the quiet threat: the graduates who stop answering by 12 months may be the ones who lost their jobs. Document who is missing and compare respondents with non-respondents on what you already hold. A few interviews explain mechanisms well but cannot establish a population rate.
AI can help organize records and draft findings, but check every line against the records it cites, and let an accountable person decide what the results mean. Involve the people affected in interpreting findings, not only in supplying data.
Start with one evaluation question this cycle
Pick the one decision coming up soonest and evaluate only what it needs.
- Write the decision and who will make it, with a date.
- Find the level or arrow of your theory of change that decision depends on.
- Write one evaluation question and name its type: process, outcome or impact.
- Check the definition behind it against your reporting agreement, and fix any mismatch before collecting.
- List the records you already hold, and collect only what is missing.
- Write the finding with its evidence, its limit and one agreed action.
After the first cycle you have one answered question, a definition both sides trust, and a clear sense of which question, and which design, comes next.
Frequently asked questions
Is monitoring the same as evaluation?
No, though they feed each other. Monitoring tracks a few agreed indicators on a fixed schedule while the program runs, such as placements each quarter. Evaluation asks a specific question at a planned point, such as why placements lag on one track, and supports a judgment or decision. Good monitoring data makes an evaluation lighter to run, because many of the records it needs already exist.
What is the difference between outcome and impact evaluation?
An outcome evaluation measures what changed for the people a program reached, such as how many were placed in a job within 90 days. An impact evaluation asks how much of that change the program caused, which needs a comparison showing what would likely have happened without it. Many programs need the first every year and the second only occasionally.
Does every evaluation need a control group?
No. The design follows the question. Process and outcome evaluations do not need one. A control or comparison group matters when you want to estimate what the program caused, and even then earlier cohorts or outside data on the same definition can offer a lighter, clearly labelled comparison when a full design is not possible.
When should program evaluation start?
Plan it when the program is designed, not when the report is due. Definitions, baselines and follow-up points must be in place before collection, or the evaluation will depend on records that were never kept. Some questions can be answered afterward from existing records, but say plainly which limits that creates, especially for anything about cause.
Who should conduct a program evaluation?
Choose people with the skills and the independence the purpose needs. Program teams can run process and outcome evaluations well, especially with clean per-person records. Impact evaluations and evaluations meant to inform high-stakes funding decisions often benefit from an external evaluator. Internal and external roles can combine, with staff supplying data and context and the evaluator owning the design.
