Whether you run a fellowship, scholarship, or award, the application is where most of the outcome is decided. Get the key design decisions right up front and the review runs itself; get them wrong and you spend the whole cycle firefighting.
In short: Design the application backward from the decision you have to make — who is the right applicant, and what are you actually buying. Then lock a handful of choices before you build the form: which fields gate eligibility, which you score, and which you only track; how you weight each criterion; whether it is one step or several; and how you pilot it on real submissions before launch. The form is the last thing you build, not the first. The decisions below apply whether you run a fellowship, a scholarship, an award, or a cohort program.
The walkthrough shows the end-to-end shape: collect from any source, read each application on arrival, then ask the assistant for a ranked, evidence-backed pool your committee can decide from.
In short: A complete application process has ten stages. Most teams over-invest in the form and under-invest in the decisions around it.
The rest of this chapter walks the decisions that matter most — starting with the one everything else hangs on.
Before a single field, write down what makes someone the right applicant — and be honest that you are usually buying long-term contribution, not a polished résumé. Every field then has to earn its place by informing that decision. Most weak processes invert this: they list questions first and discover too late that they never asked the thing that actually predicts a good fit.
In short: Every field on the form does one of three jobs. Eligibility fields are pass/fail gates checked before anything is scored. Criteria fields — essays, resumes, letters, or closed-ended questions — are read against a rubric, quantified, weighted, and summed into the score. Metadata fields — name, email, zip code — identify and filter, but never enter the score. Sorting them before you build is the single decision that keeps scoring fair and reporting clean.
Eligibility — the gate. Hard requirements: citizenship, age, deadline, a required document attached. Evaluated pass/fail before scoring, so an ineligible application never competes and never wastes reviewer time. Keep these out of the score — an applicant does not earn a low score for being ineligible; they simply do not enter the ranked pool.
Criteria — what you score. Anything you score — a qualitative essay, a resume, a recommendation letter, even a closed-ended question — is read against a rubric that turns it into a number, weighted, and rolled into one score. A fixed rubric, written anchors, controlled instructions, and cited evidence make even a qualitative judgment far more consistent and auditable: the same submission tends to earn the same score, and every score points to the sentence behind it. A single criterion can draw on several fields — “leadership” might read the essay and the resume and a recommendation.
Metadata — what you track. Name, email, zip code, demographic tags, a contact ID. They identify the applicant, dedupe records, drive filters, and let you disaggregate the cohort by geography or discipline. Context, not criteria. The rule that saves you later: anything you will filter or report by must be a clean structured field captured at intake, or your geography cuts break the first time someone free-types their city.
How each field earns its place:
| Field | Role | How it’s used | In the score? |
|---|---|---|---|
| Citizenship / age / deadline | Eligibility | Hard requirement, checked first | No — pass/fail gate |
| Required documents attached | Eligibility | Completeness check | No — pass/fail gate |
| Motivation essay | Criteria (qualitative) | Rubric 1–5, evidence cited | Yes — weighted |
| Resume / CV | Criteria (qualitative) | Rubric 1–5 on relevant experience | Yes — weighted |
| Recommendation letter | Criteria (qualitative) | Rubric 1–5 on strength of endorsement | Yes — weighted |
| Years of experience | Criteria (closed-ended) | Banded to points (0–4 = 1 … 10+ = 5) | Yes — weighted |
| Name / email / contact ID | Metadata | Identity, dedupe, tracking | No — tracking only |
| Zip code / region | Metadata | Filters, geographic disaggregation | No — filter only |
Worked example. Say your published weights are motivation 25%, regional ties 20%, leadership 20%, recommendation strength 20%, experience 15%. Each criterion is quantified 1–5 by the rubric — whether it reads an essay, a resume, or a closed-ended answer — then normalized to its weight and summed to 100. For one applicant:
Eligibility and metadata both sit outside the 100: an applicant who fails citizenship or never uploaded a required document never reaches the weighted stage, and name or zip code never move the score — they just let you find the record, filter the pool, and check the cohort’s balance afterward.
For many demanding cohort programs, participant drop-out is one of the most consequential selection risks — and you cannot screen for commitment against a vague ask. Publish the real load — hours, travel, cost — and make applicants acknowledge it, so they self-select honestly and can never say “we didn’t know.” Where relevant, ask an employer to confirm in writing that they will release the person for the time. That one confirmation does more anti-drop-out work than any essay.
In short: Every question you add costs you applicants. Justify each one, publish the real completion time, and remove the friction that quietly filters out the people you most want.
Before a field ships, ask whether the decision actually needs it; if not, cut it. Then design for access: a form that works on a phone, in more than one language, with save-and-return so no one loses an hour of work; alternative submission formats and a clear way to request an accommodation; and an honest look at whether the documents you require — audited financials, a formal reference — quietly disadvantage early-stage or less-resourced applicants. Burden is not neutral; it changes who applies.
Decide your weights up front and publish the criteria and weights. Transparency raises the quality of who applies and defends the decision afterward. Treat it as a stated rule of engagement, not a black box. Some applicants still won’t follow it; that is fine — it tells you they didn’t take the time.
Here is what that looks like on a real application. Carnegie Mellon’s Forge Pitch challenge tracks a few identity fields, matches on a closed-ended vertical, and asks for one .pdf brief that covers everything the rubric scores — each section capped at 500 words:
Every submission is read on arrival and scored against those weights — evidence pulled straight from the .pdf, gaps flagged where a claim isn’t backed. Here is the scorecard Sopact Sense produced for one applicant:
Every criterion needs three things: a weight, the evidence that feeds it, and a scoring scale with written anchors describing what a 5 looks like versus a 3. A criterion without anchors is a vibe, and two reviewers will score it differently. But don’t build a strict examination paper — keep the rubric slightly liberal so you can tune the scoring as real applications reveal what you actually meant.
Lock the shape early, because it changes both the form and the timeline. Is it application-only, or application → recommendation forms → a shortlisted round of video interviews before final selection? A short interview stage before you commit is common and worth deciding now, not mid-cycle.
In short: Publish a simple end-to-end timeline, and decide up front exactly what applicants hear at each step.
Attach dates to each stage and put them on the page. Then decide the messages: a confirmation the moment they submit, a reminder before the deadline, a prompt if something is missing, and — the one teams skip — a timely, respectful decline for everyone you do not select. How you tell people no is how they remember you.
It is tempting to require nominations to hold volume down. Resist it: a nomination-only requirement can disadvantage qualified candidates who have less access to established professional networks — often the very people you most want — and when AI reads every application, volume is a good problem rather than a bottleneck. Let everyone apply. Incomplete applications screen out on eligibility, and the effort of a complete application is itself a signal of intent.
In short: A rubric only holds if reviewers use it the same way. Decide the governance up front: who reviews, how conflicts are handled, and how disagreements resolve.
Brief reviewers on the anchors so a 4 means the same thing to everyone; require anyone with a personal or professional tie to an applicant to recuse; and consider blind review — names and identifying details hidden — for the first scoring pass. Have every application read by more than one reviewer, and write down how you break ties, when a score can be overridden, and how that override is recorded. If you allow appeals, say so before you open. We go deeper in How Do You Review Applications Without Reviewer Bias?
Put it in writing for your committee and your applicants: AI scores against your rubric and surfaces the evidence; people select. The output is a curated, ranked pool with cited scorecards that the committee reviews and decides from — never an AI verdict. That line is both good practice and good governance.
In short: Applications hold resumes, demographics, references, and other sensitive data. Decide who can see it, how long you keep it, and whether it can be reused — before you collect a single response.
State plainly who on the committee can access which fields, set a retention period and delete on schedule, and be explicit if you intend to keep non-selected applicants on file for future rounds — with their consent. Handling applicant data with care is both a legal obligation and a trust signal that raises the quality of who applies.
Draft and argue your rubric in any AI chat first — it is faster than building. Then run five to eight real submissions through the live form before you open it to the world. What behaves cleanly on an invented example behaves nothing like real applications; every rubric is wrong on first contact, and the pilot is where you fix it. Calibrate by having your committee score the same few blind and comparing against the AI.
In short: The same loop, in one place — collect clean at the source, analyze each application the moment it lands, then chat with the whole round: rank, compare two applicants side by side, or ask any question and get an evidence-cited answer.
Because the rubric is fixed and every score cites its source, review stays consistent and auditable — you can trace any number back to the sentence behind it, the difference between a defensible pool and a black box.
It is not a one-time transaction. Give every applicant a persistent record, run a short baseline survey of the cohort after selection to match teams, and keep the people you don’t select in touch — they are the most valuable list you will build for future cohorts, events, and sponsors. Design for the ten-year alumni network, not just this round.
Once your criteria are set, these run in the Sopact Sense Assistant — or any AI working over your application data:
Here is my rubric: [paste criteria and weights]. Score each application only against these criteria, cite the sentence behind each score, grade it green/amber/red, and flag anything the applicant left incomplete. Do not reward writing quality that isn’t backed by evidence.
Rank all applications by weighted score and give me the top [N], each with the one line of evidence behind it and the region and discipline, so I can see the cohort balance at a glance.
Ten stages: (1) define the decision and the applicant; (2) set eligibility rules; (3) choose scored questions, tracked fields, and required documents; (4) design for low burden and access; (5) lock the timeline and stages; (6) build the rubric and publish weights; (7) set reviewer roles and conflict rules; (8) pilot on real submissions; (9) communicate with applicants throughout; and (10) select, notify, and retain records responsibly.
Eligibility criteria are pass/fail requirements — citizenship, age, a required document — checked before anything is scored; miss one and the application does not compete. Selection criteria are the weighted evidence — essays, resumes, closed-ended answers — used to rank the applicants who clear eligibility. Keeping them separate keeps the score fair: no one is scored down for being ineligible; they simply never enter the ranked pool.
Score the fields that carry evidence of fit — essays, resumes, recommendation letters, and closed-ended questions — by running each through a rubric that turns it into a weighted number. A fixed rubric with written anchors and cited evidence makes even a qualitative answer far more consistent and auditable: the same input tends to earn the same score, traced to the sentence behind it. Fields like name, email, and zip code are metadata — they track the application, dedupe records, and drive filters and disaggregation, but they never enter the score.
Only as long as the decision requires. Every question you add costs you applicants, so justify each field against a criterion or a filter you will actually use, cut the rest, and publish the real completion time so applicants can plan. Length is a cost, not a signal of rigor.
Drop questions the decision doesn’t need, make the form work on a phone and in the languages your applicants speak, add save-and-return, and offer an accommodation path. Watch for required documents — audited financials, formal references — that quietly disadvantage early-stage or less-resourced applicants.
Publish the criteria and their weights — it raises application quality and defends the decision. Do not publish the detailed scoring anchors; applicants who see exactly what earns a 5 will write to it, and you lose the signal that separates a strong candidate from a coached one.
Have at least two reviewers score each application so you can measure agreement and catch outliers, and brief them on the anchors so a 4 means the same thing to everyone. Blind review — hiding names and identifying details on the first pass — reduces bias for most competitive programs, and anyone with a tie to an applicant should recuse.
Draft and argue the rubric in any AI chat, then run five to eight real submissions through the live form before opening it. Every rubric is wrong on first contact; the pilot is where you fix it. Calibrate by having your committee score the same few blind and comparing against the AI.
State the real time, travel, and cost commitment and make applicants acknowledge it, and where relevant require an employer release confirming the person will be given the time. Honest, specific commitments up front remove much of the drop-out risk before anyone is selected.
Use AI to score every application against a fixed, published rubric and to surface the evidence sentence behind each score — then have people make the decision. AI should produce a ranked, cited pool a committee reviews, never a final verdict. A fixed rubric, written anchors, and cited evidence keep the scoring consistent and auditable; human judgment and a documented override keep it accountable.
Next: How Do You Review Applications Without Reviewer Bias? · How Do You Score a Grant Proposal?
Open Sopact Sense, paste your program description, and put it to work.
Try it in Sopact →
