New webinar on 3rd March 2026 | 9:00 am PT
In this webinar, discover how Sopact Sense revolutionizes data collection and analysis.
Survey analysis methods organized by the question they answer: descriptive, comparative, and explanatory questions, each paired with the methods that fit and the data they require.
The same methods catalogued elsewhere by family, viewed here from the question side. Pick the question your research is asking — descriptive, comparative, or explanatory — and the methods that fit follow. Most real questions cross types; the catalogue is broken out so the components are visible.
“What does the distribution look like?”
Quantitative
Qualitative
“Do two or more groups differ?”
Quantitative
Qualitative
“What drives the outcome?”
Quantitative
Mixed-methods
Most methods references organize by family — quantitative tests, qualitative techniques, mixed-methods designs — because that is how textbooks teach them. The companion catalogue on this site does the same. This page does the opposite.
Organizing by question type matches how readers usually arrive at the choice. Almost no one starts a survey thinking I need an ANOVA. They start with a question: did baseline-to-follow-up confidence change?, which curriculum elements predicted the largest gain?, did the cohort effect hold across income brackets? The methods follow from how that question is shaped.
Three question types cover most program-evaluation work. Descriptive questions describe what is. Comparative questions test whether groups differ. Explanatory questions ask what drives an outcome. Each maps to its own set of methods on both the quantitative and qualitative sides.
Most real questions cross types. The closing section of this page covers what to do when a single research aim splits across descriptive, comparative, and explanatory components — which is the rule, not the exception. For the same methods organized by family rather than question, see the methods canonical page. For the procedural sequence of running an analysis, see the step-by-step guide. For the broader topology, see the discipline page.
Descriptive questions ask what the responses look like in aggregate, without claiming generalization beyond the sample. What was the average satisfaction score? How are responses distributed across income brackets? What were the most common themes in open-ended feedback? These are the questions that summarize a program for a single audience at a single moment.
On the quantitative side, descriptive statistics handle the closed-ended responses. Frequencies count how often each option was chosen. Means and medians summarize numeric responses; the median is preferred when the distribution is skewed. Standard deviation describes spread. One-way cross-tabulation extends the description to a single demographic dimension — what is the distribution within each cohort?
On the qualitative side, thematic analysis identifies the recurring patterns in open-ended responses; content analysis applies a predetermined framework to count category frequencies. Either method produces a structured description of what people said.
What descriptive methods don't do: they don't test hypotheses, they don't claim that observed patterns generalize beyond the sample, and they don't answer why the patterns exist. The temptation to read inferential meaning into descriptive output (this group looks different from that one, so they must be different) is the most common analytical error. If the question is whether two groups truly differ, the question is no longer descriptive — it is comparative, and it needs comparative methods.
Sample size is unconstrained for descriptive analysis. Even small samples produce defensible descriptive output, as long as the report makes clear that the description applies to the sample only.
Comparative questions ask whether two or more groups differ in a way unlikely to be explainable by chance. Did confidence improve from baseline to follow-up? Did completion rate vary across cohorts? Did satisfaction differ between participants who attended in person and those who attended remotely? Each is a comparative question with a defined answer shape: yes or no, and how big.
On the quantitative side, the choice of test depends on the data shape. The t-test compares two group means and is the right tool for paired baseline-follow-up comparisons. ANOVA compares three or more groups in a single test, controlling for the inflated false-positive rate that running several t-tests pairwise would introduce. Chi-square tests whether two categorical variables are related — useful when the outcome is itself a category (completed vs not, agreed vs disagreed).
Each test produces a p-value indicating whether the observed difference is likely real. The p-value alone is never enough. An effect-size measure — Cohen’s d for mean differences, eta-squared for ANOVA, phi or Cramer’s V for chi-square — answers the second question that always follows: is the difference large enough to matter in practice? With a large enough sample, even tiny differences become statistically significant. Reporting those without effect size produces findings that are technically true and practically meaningless.
On the qualitative side, the comparative job is done by tracking theme frequency by subgroup — how often each coded theme appears within each demographic, cohort, or condition. Framework analysis is well-suited here because it produces a matrix of cases by themes, and the matrix can be aggregated by subgroup to show how the qualitative content differs across groups.
Sample size matters for comparative work. Below roughly thirty respondents per group, parametric tests lose power and effect-size estimates become unstable. Non-parametric alternatives — Mann-Whitney U, Kruskal-Wallis, Fisher’s exact — handle small samples better at the cost of some statistical power.
Explanatory questions ask what drives an outcome. Which curriculum elements predicted the largest confidence gain? Why did some participants score higher than others on the post-survey? What program features differentiate the highest-impact cohorts from the rest? Explanatory work is the most demanding kind of survey analysis — it claims a causal or near-causal relationship and has to defend that claim.
On the quantitative side, regression is the dominant tool. Linear regression estimates how a continuous outcome changes as a function of one or more predictors. Logistic regression handles binary outcomes (yes/no, completed/dropped). Ordinal regression handles ranked outcomes (low/medium/high satisfaction). Multiple regression with several predictors lets the analysis isolate the effect of one variable while holding others constant — does cohort still predict completion after controlling for prior education?
Regression has the strictest data requirements of any method on this page. The model assumes the predictors are independent of each other, the relationship is approximately linear (for linear regression), and residuals are roughly normally distributed. Sample size needs to be roughly ten to twenty cases per predictor to produce stable coefficient estimates. Skipping the assumption checks produces output that runs cleanly and means very little.
On the qualitative side, explanatory work requires linking themes to outcomes through the same participants. The frequency of a coded theme can be mapped against the distribution of a quantitative score — participants who mentioned hands-on labs scored higher on confidence gain. Without persistent participant identifiers connecting open-ended and closed-ended responses, this kind of explanatory integration is not possible.
Mixed-methods designs are the most common framework for explanatory work because the why behind a quantitative pattern usually doesn’t reduce to a regression coefficient. Sequential explanatory design runs the regression first, then qualitative analysis to explain the patterns. Sequential exploratory design runs qualitative analysis first to identify the variables that matter, then quantitative tests to measure them. Embedded design uses one method as primary and the other as supplementary — useful when the qualitative work is for context rather than confirmation.
The same catalogue, summarized in five columns. Read the row for the kind of question you have; the methods, data, sample requirements, and output type follow across.
Most program-evaluation research questions are not pure descriptive, pure comparative, or pure explanatory. A typical funder report has all three components: describe the cohort, compare baseline to follow-up, explain what drove the change. The practical move is to break the question into its components and run the matching method for each.
Descriptive plus comparative. The most common pairing. Describe the cohort with frequencies and means, then compare baseline to follow-up with paired t-tests. The descriptive part sets the context; the comparative part tests whether the program changed anything.
Comparative plus explanatory. Test whether outcomes differed by subgroup, then run regression to identify which factors predicted the differences. The comparative finding (cohort B improved more) becomes the question the explanatory analysis answers (which features of cohort B’s experience explain the gap?).
Descriptive plus explanatory. Less common but useful for program-design work. Describe the patterns in qualitative responses, then test which patterns correlate with quantitative outcomes. The qualitative description identifies the candidates; the quantitative test measures their relationship to the outcome.
When all three types are needed in a single report, the architectural requirement is the same as it is for any mixed-methods analysis: persistent participant identifiers linking responses across surveys, methods, and time. Without that identifier, integrating descriptive, comparative, and explanatory findings into a coherent narrative requires manual reconstruction at every stage. For the procedural walkthrough that runs all three in sequence, see the step-by-step guide.
The choice of method is downstream of the question. A question framed clearly will tell you what method to use.
A working principle
A descriptive question asks what the responses look like in aggregate (“what was the average satisfaction score?”). A comparative question asks whether two or more groups differ (“did completion rate vary by program track?”). An explanatory question asks what drives an outcome (“which curriculum elements predicted the largest gain?”). The grammatical clue: descriptive uses what is, comparative uses do X and Y differ, explanatory uses what predicts or why.
Descriptive questions are answered by descriptive statistics for closed-ended responses (frequencies, means, medians, standard deviations) and by thematic or content analysis for open-ended responses. Cross-tabulation by one demographic dimension extends descriptive analysis to subgroups. None of these methods test hypotheses or claim generalization beyond the sample — they describe what the sample looks like.
Comparative questions call for inferential tests paired with effect-size measures. T-test compares two group means; ANOVA extends to three or more groups; chi-square tests categorical relationships. Each produces a p-value indicating whether the difference is likely real, paired with an effect-size measure (Cohen’s d, eta-squared, phi) indicating whether the difference is large enough to matter in practice. On the qualitative side, theme frequency by subgroup performs the same comparative job for open-ended responses.
Explanatory questions call for regression — linear for continuous outcomes, logistic for binary, ordinal for ranked. Most explanatory questions also need qualitative analysis through linked participant records to surface the why behind the quantitative pattern. Mixed-methods designs (sequential explanatory, sequential exploratory, embedded) integrate the two through persistent participant identifiers.
Most real research questions do. A typical program-evaluation question is descriptive (what does the cohort look like?) plus comparative (did baseline-to-follow-up scores improve?) plus explanatory (which curriculum elements drove the improvement?). The practical move is to break the question into its three components, run the matching method for each, and integrate the findings in the report. Mixed-methods designs are the formal name for this integration when both quantitative and qualitative components are involved.
Quantitative regression generally needs ten to twenty cases per predictor variable to produce stable estimates, so small samples constrain how many predictors a model can include. Qualitative analysis through linked participant records is more sample-tolerant — even a small dataset can produce defensible explanatory findings if the cases are coded carefully. Mixed-methods designs are often the right choice for small-sample explanatory work because they lean on the qualitative side for what the quantitative side cannot estimate.
Longitudinal questions are typically comparative or explanatory with a time dimension. Paired t-tests compare baseline and follow-up means within the same participants. Repeated-measures ANOVA extends to three or more time points. Within-participant regression models how an outcome changes as a function of time and other predictors. All of these require persistent participant identifiers linking responses across surveys — the same architectural primitive that enables mixed-methods analysis.
A research question is one moving part of a larger discipline. The discipline sits one level up.
The hub page covers what survey data analysis is, the three approaches, the four outputs that frequency tables can’t produce, and how the methods named here fit into the broader workflow.
See how survey data analysis ladders up ↗
