Skip to content
finalstate.aiThe Generation-Verification Gap: When the Right Answer Loses
120Atlas

120THE RIGHT ANSWER IS THERE

The Generation-Verification Gap: When the Right Answer Loses

A correct answer can be present among generated candidates while the selector returns a wrong one. Measuring that loss requires the same tasks, candidate sets and correctness standard.

Correct candidate C3 stays in the set while the selector returns wrong candidate C1.Four candidate documents are an excerpt from an illustrative set of one hundred. A reference check identifies C3 as correct. A separate selection moves C1 into the returned position; the correctness mark never follows it.C1C2C3C4CorrectReturned
Illustrative failure: four shown candidates are an excerpt from a set of 100. The correct answer is C3; the selector returns C1. These are not observed frequencies.

THE GAP

Compare two rates at the same K.

A vertical bracket compares Pass at K and selection success at one fixed candidate budget.Illustrative success curves share a zero-to-one-hundred-percent scale and coincide at K=1 under forced selection. At K=100 a bracket measures their vertical difference. The curves are schematic, and the gap narrows over their final interval; no automatic widening law or area measure is implied.100%01100Candidates (K)Success ratePass@KSelectedGap
Schematic curves, not study data. The bracket measures a vertical difference at K=100; its size need not grow as K increases.
  • Pass@K: share of tasks with at least one correct candidate among K responses
  • Verification success: share of those same tasks whose selected response is correct
  • Gap: Pass@K minus verification success, in percentage points

Both rates use the same candidate sets and the same correctness labels. With one candidate and forced selection, they coincide. Review time is a separate cost. Section 3 defines the metric.

Find out whether generation or selection failed.

In Saad-Falcon et al.'s definition, a task contributes to the gap when its candidate set contains a correct answer but selection misses it. If every candidate is wrong, generation failed on that task. Correctness must be established to tell these cases apart.

WEAK VERIFIERS

Weaver combines unequal verification signals.

A constant positive signal stops at a filter while two informative signal patterns enter a weighted combination.Three schematic binary signal rows stand for part of the initial pool of thirty-three verifiers. An uninformative all-positive row stops; the remaining rows reach a weighted combination and a candidate selection. Line widths are illustrative, not fitted weights. The drawing promises no correct selection on an arbitrary task; reported dataset results remain in the caption.FilterWeightSelect
Saad-Falcon et al., arXiv:2506.18203v2, Table 1: MATH500 93.4%, GPQA Diamond 72.1%, MMLU College 94.9%, MMLU Pro 90.2%; 87.7% is their rounded, equal-weight average, not pooled task accuracy. The appendix labels GPQA differently and does not resolve the Table 1 subset size. The three drawn signal rows illustrate filtering and weighting; they are not measured verifier scores.

Weaver normalizes scores, discards uninformative verifiers and estimates weights with weak supervision. Its setup uses a small labeled development subset for task statistics. Table 1 reports higher success than majority voting and averaging the top ten reward models. This comparison does not establish reliability on untested work.

  • Llama 3.3 70B Instruct generated 100 candidates per question
  • 33 reward models and language-model judges supplied the initial signals
  • Table 1 reports 87.7% average selection success across four datasets

THE OFFLOAD RULE

Give the check a defined job and an accountable owner.

A candidate is compared with a reference before it moves to an owner whose decision remains open.The recurring candidate and a separate reference converge at a comparison. After that condition is checked, the candidate moves to the owner. Empty decision brackets remain empty. This schematic depicts one objective check; learned judges need representative evaluation and unresolved correctness still needs expert review.CandidateReferenceC3C3Owner
Illustrative objective check: a candidate matches a reference and reaches its owner for review. The match is not automatic authorization.
  • An execution test, reconciliation or sealed answer checks a specified condition
  • A learned judge needs evaluation on representative work with relevant errors
  • Where correctness remains unresolved, keep expert review and record its limits

This is an operator application of the metric. The owner must decide whether the check covers the decision and whether its remaining errors and review cost are acceptable. A passing test establishes the tested condition; it does not settle every consequence of acting.

More candidates need not produce more correct selections.

In Figure 3, several baseline selectors plateau as the candidate budget rises, while Weaver improves. Selection can become the constraint even when generation keeps raising the oracle ceiling. This is a result for the tested systems, not a rule that additional samples can never help.

FELT VERSUS MEASURED

Completion time contradicted participants' estimates.

Post-study estimated completion time is twenty percent lower; the adjusted measured time is nineteen percent higher.Fixed points share a completion-time-change axis with zero as AI-disallowed work. The post-study estimate is minus twenty percent. The adjusted measured effect is plus nineteen percent, with a ninety-five-percent confidence interval from plus two to plus thirty-nine percent. These summarize METR's historical trial of sixteen developers and two hundred forty-six tasks; the marks do not animate in size.Completion timePost-study−20%Measured+19%-20%0%+20%+40%
METR, Early-2025 AI Experienced Open-Source Developer Study, Figure 1 and §§2.2–2.4. All values are changes in completion time relative to AI-disallowed work, not reciprocal speed gains. The measured effect is adjusted; the interval belongs only to that estimate. This exhibit does not measure the generation-verification gap or isolate review cost.

The trial concerns the tools and work tested then. METR's February 2026 follow-up describes selection and measurement problems that prevent a reliable current estimate. Its May 2026 survey reports workers' perceptions, a different kind of evidence. None converts this historical trial into a current productivity verdict.

  • METR, February–June 2025: 16 experienced developers, 246 tasks in familiar repositories
  • Adjusted observed completion time: +19% with AI allowed; 95% confidence interval +2% to +39%
  • Post-study estimate: −20% completion time with AI

THE OPERATOR'S CHECKLIST

Measure both rates on the same eligible tasks.

A four-task ledger separates one correct selection, one selector miss, one generation miss and one unresolved task.Each task has four candidates and one returned candidate. Task A contains and returns a correct answer. B contains one but returns a wrong answer. C contains only wrong answers and returns a wrong answer. D has unknown correctness and is excluded from both denominators. Pass at four is two thirds; selection success is one third; their difference is one third. Review cost is recorded separately from these correctness rates.CandidatesReturnedABCD?????2/3 − 1/3 = 1/3
Illustrative audit, K=4 for every task. A returns its correct candidate; B misses a correct candidate; C contains none; D has unresolved truth. Three eligible tasks supply both denominators. A check mark denotes correctness, a cross denotes incorrectness, and a question mark denotes an unknown label.

This audit gives metacognitive demand an external record. In the illustrated ledger, two of three labeled tasks contain a correct candidate and one returns it: the gap is one third, or about 33 percentage points. The fourth task has unknown truth and contributes to neither rate. These rates describe the labeled subset. A selector allowed to abstain also needs a separately stated coverage measure.

  • Fix K, the task sample, candidate sets and selection rule; retain the correctness source
  • For each task, record correct candidate present and selected candidate correct
  • Exclude unresolved correctness from both rates and report those exclusions
  • Keep review cost, abstentions and the date for retesting in separate fields

BUILD THE CHECK

Spend where correctness was lost.

Separate routes identify generation failure, selection failure and unresolved ground truth.Three schematic routes retain their different evidence states: no correct candidate directs attention to generation; a correct candidate missed by the returned answer directs attention to selection; an unknown correctness label stops before either diagnosis and calls for ground truth. No route grants approval to deploy.GenerationSelection???Ground truth
  • No correct candidate: improve generation
  • Correct candidate present but missed: improve selection
  • Correctness unresolved: establish ground truth before claiming either diagnosis

That diagnosis informs cheap generation, expensive checking. It identifies the failed step; the owner still has to decide whether repairing it is worth the cost.

Read the transcript

01 · THE RIGHT ANSWER IS THERE

A model produces one hundred answers to a hard question. In this illustrative set, only candidate C three is correct. The figure shows four of the candidates, an excerpt rather than the whole hundred. The correct answer is already present. Then the selector ranks candidate C one first and returns it to the user. Generation supplied a correct response, but selection lost it. Across a set of tasks, that difference is the generation-verification gap.

02 · THE GAP

The paper defines the gap with two rates at the same candidate budget, K. Pass at K is the share of tasks where at least one of the K responses is correct. Verification success is the share where the selected response is correct. Both rates use the same tasks, the same candidate sets and the same correctness standard. At a fixed K, subtract selection success from Pass at K. Their vertical difference is the gap, measured in percentage points. The curves here are schematic, not experimental data. With just one candidate and a rule that must return it, the two rates coincide. Review time belongs in a separate account.

03 · GENERATION IS NOT SELECTION

Jon Saad-Falcon and colleagues formalize this distinction in Shrinking the Generation-Verification Gap with Weak Verifiers. A task where every candidate is wrong cannot be rescued by choosing a different member of that set. But when a correct candidate is present and the selected response is wrong, selection has failed. A final wrong answer alone does not tell you which happened. You need a defensible way to label the candidates.

04 · WEAK VERIFIERS

Weaver combines imperfect verifiers. It puts their scores on comparable scales and discards uninformative signals. It then estimates unequal weights from their patterns of agreement, using weak supervision and a small labeled development subset for task statistics. Those weights guide candidate selection. In Table one, Llama three point three, seventy billion parameter Instruct generated one hundred responses per question, and thirty-three reward models and language-model judges supplied the initial signals. Weaver reported average selection success of eighty-seven point seven percent across four mathematics and reasoning datasets. That is an equal-weight average of dataset rates, not a pooled accuracy across every question. Table one and the appendix use inconsistent G P Q A dataset labels, leaving its exact task count unresolved. The tested baselines did worse; a new kind of work still needs its own evaluation.

05 · THE OFFLOAD RULE

For an operator, the first step is to name what the check establishes. An execution test can compare a candidate with a specified expected result. A reconciliation can check that recorded totals agree. Each covers a condition, and the owner must judge whether that condition covers the decision. A learned judge needs representative evaluation, including the errors that would matter in practice. Unresolved correctness requires expert review with its limits recorded. A checked candidate can then reach the owner, whose decision remains open. The cost of running checks and reviewing the remainder belongs in the offload decision, alongside the consequences of getting it wrong.

06 · THE BINDING CONSTRAINT

Additional samples can make a correct candidate more likely to appear. Their value depends on the selector recovering it. In the paper's Figure three, several baseline selectors plateaued as the candidate budget grew, while Weaver continued to improve. Selection can become the constraint. This does not rule out gains from more samples; it gives a reason to measure whether your selection method realizes those gains.

07 · Advertisement · Bubble AI App Builder

Some ideas do not need another document before they become testable. With Bubble AI, you describe the app you want, and Bubble creates a working starting point: interface, data, and logic you can inspect. From there, you refine visually, connect AI models and services, and turn the first version into something real enough to use.

08 · FELT VERSUS MEASURED

External measurement can also correct human self-assessment. In METR's February to June twenty twenty-five trial, sixteen experienced developers completed two hundred forty-six tasks in familiar open-source repositories. The adjusted estimate was nineteen percent more completion time when AI was allowed, with a ninety-five-percent confidence interval from two to thirty-nine percent more time. After the study, participants estimated twenty percent less completion time with AI. Those quantities concern time, not reciprocal changes in speed. METR's February twenty twenty-six follow-up describes obstacles to a reliable updated estimate. Its May survey records workers' self-reported benefits, a different kind of evidence. The historical trial shows a mismatch between estimates and measured time in that sample. It neither measures the generation-verification gap nor isolates the cost of checking.

09 · THE OPERATOR'S CHECKLIST

Build an audit that can distinguish the failures. Fix the task sample, the candidate budget and the selection rule. Keep the candidates and record where their correctness labels came from. In the illustrative ledger, each task has four candidates. The first task contains and returns a correct answer. The second contains a correct answer but returns a wrong one. The third contains no correct answer. The fourth has unresolved truth, so leave it outside both rates and report the exclusion. Among the three eligible tasks, two contain a correct candidate and one selects it. Two thirds minus one third leaves a gap of one third, about thirty-three percentage points. These rates describe the labeled subset. Log review cost separately. If a selector may abstain, state its coverage as well as its success rate, and set a date to test it again.

10 · BUILD THE CHECK

Follow the evidence to the failed step. If no correct candidate appeared, investigate generation. If a correct candidate was present but lost selection, investigate the selector. If correctness is unresolved, stop before either diagnosis and establish ground truth. The audit identifies where the loss occurred. An accountable owner still decides whether the improvement justifies its cost.

01 / 10 · THE RIGHT ANSWER IS THERE0:00 / 6:57