Skip to content
finalstate.aiCheap to Make, Expensive to Check
005Atlas

005THE HINGE

Cheap to Make, Expensive to Check

A quarterly report arrives before the analyst has opened its source files. The draft is finished; the evidence still needs checking. That review belongs in the cost estimate.

A completed quarterly report sits beside source files that still need inspection.In this illustrative scene, the report appears first. The source files then open beside it; their unresolved connection shows the review still required. No cost ratio or measured time is shown.ReportSource
Illustrative quarterly report. Producing the draft leaves its claims unverified.

THE GATE

Price a check that can reject the output.

A report row is compared with its source; a mismatch interrupts acceptance.The report starts with a shorter value mark than its source. The comparison connects corresponding rows and highlights that existing discrepancy; the report remains behind a closed barrier. The example demonstrates one detected fault, not an established miss rate.SourceReportHold
  • Compare the check with remaking the work
  • Include generation, building and running checks, and likely repair
  • Include normal review in the manual baseline too

The analyst compares a report row with its source and finds a mismatch. This check can stop that error; one caught example does not establish its reliability across other failures. Saad-Falcon and colleagues, §3, define a different generation-verification gap: a correct candidate exists among the responses, but the verifier fails to select it. That selection metric does not establish a business-cost inequality. Here the operator must test whether a relevant check catches consequential errors, then compare the whole offload with a manual baseline that includes its own review at the same required quality.

THE LAST THIRTY

The quick draft leaves finishing work.

An unnumbered illustrative curve becomes steeper as finishing work accumulates.A fixed curve relates task progress to cumulative effort. It illustrates one possible work shape without asserting a universal curve, a measured seventy/thirty division, or a staffing allocation.EffortTask progressIllustrative
Illustrative work shape. No measured shares or calibrated cost axis.Addy Osmani, The 70% Problem, December 4, 2024

In The 70% Problem, dated December 4, 2024, Addy Osmani describes a fast first pass followed by difficult integration and production work. Finishing the job can still require expert diagnosis. The curve sketches rising cumulative effort; its unnumbered axes do not assign percentages to work, cost, or staffing.

  • Scaffolding can arrive before correctness, security, and edge cases are resolved
  • The 70/30 split is a heuristic, not a measured ratio

Include review in the cost estimate.

For a claim in the quarterly report, the analyst may need to reconstruct the calculation and read the underlying records. That work counts even when the draft took seconds. If checking already costs as much as remaking, generation and repair cannot be treated as a saving.

WHERE ORACLES LIVE

A useful check has a bounded question.

A program, a crystal, and a protein candidate connect to three different tests.Three rows pair a program with execution, a crystal lattice with DFT, and a binder with an assay. Each test produces a bounded measurement record. The schematic records carry no invented values, pass rates, or equivalence between test costs.RunDFTAssay
  • AlphaEvolve: run proposed programs and score specified objectives
  • GNoME: calculate predicted crystal stability
  • AlphaProteo: measure binding to specified targets

AlphaEvolve uses automated program evaluators. GNoME uses density-functional calculations in its crystal discovery pipeline; computational stability alone does not establish that a material can be synthesized or will be useful. AlphaProteo tests designed binders in the laboratory; binding alone does not establish a medicine. These are different kinds of verifiable space, with different costs. The operator must match the tested property to the decision.

TWO STUDIES, TWO SETTINGS

Measure the workflow you intend to change.

A fixed exhibit separates support output per hour from developer completion time.The support result is about plus fifteen percent issues resolved per hour. A separate completion-time axis places the developers' post-study estimate at minus twenty percent and the adjusted trial estimate at plus nineteen percent, with a ninety-five percent interval from plus two to plus thirty-nine percent. Values remain fixed throughout.Support agentsIssues resolved / hour+15%Developers, early 2025Completion time−40%0+40%Post-studyestimate−20%Adjustedtime+19%
Different populations, tasks, tools, dates, and outcome units. This juxtaposition is not a causal comparison. METR's interval describes its adjusted time estimate; it does not describe participants' beliefs.Brynjolfsson, Li & Raymond, QJE (2025); METR (2025, 2026)
  • Support: about 15% more issues resolved per hour
  • Developers: 19% more completion time in a historical trial

The published Generative AI at Work, QJE (2025), studied 5,172 support agents at one firm during a staggered assistant rollout. Its adjusted estimate was about 15% more issues resolved per hour. METR's early-2025 randomized trial covered 16 experienced developers and 246 tasks in familiar repositories, primarily with Cursor and Claude 3.5/3.7 Sonnet. AI access increased estimated completion time by 19% (95% CI: +2% to +39%); developers' post-study estimate was a 20% reduction. The time result adjusts self-reported active implementation time before and after pull-request review for task difficulty. Neither study isolates verification cost or explains the other's result. METR's February 2026 update identified selection and timing problems that prevent a reliable current estimate from its later experiment. Its May 2026 survey measures reported perceptions separately. Felt speed needs a local baseline before it enters the cost estimate.

THE SHELF

A cheap tie-out cannot certify the whole report.

A single balance tie-out contrasts with a report explanation that needs several source files.The upper path joins one reported balance to one bank record. The lower path gathers three source files before reconstructing an explanation. No bar lengths encode prices, time, or a safe-to-offload ranking.BalanceExplanation
Illustrative checking paths. Agreement on one balance supports that balance only.

In the illustrative report, the analyst ties one balance to a bank statement. The agreement does not validate the explanation beside it. That claim may require several source files and a reconstructed calculation. The paths have different checking demands; their cost must be measured locally. The reverse-centaur problem also concerns authority: who may challenge the output, reopen a decision, and stop acceptance. Expensive review alone does not establish that the operator has lost those rights.

  • A bank tie-out can test a stated balance
  • A narrative claim may require rebuilding the evidence
  • Estimate review cost for each consequential claim

BUILD THE CHECK

Set the criterion before inspecting the draft.

  1. 01Record the source, threshold, owner, and stop rule in advance
  2. 02Use model criticism to find questions worth testing
  3. 03Give reviewers the access and authority their role requires

Precommitment makes an acceptance rule harder to move after seeing a convenient result. Scheel, Schijen and Lakens (2021) found author-reported full or partial support for the first hypothesis in 146 of 152 standard reports (96%) and 31 of 71 Registered Reports (44%), using the first preregistered hypothesis for the latter. These were the study's own comparison groups, not a causal experiment; formats, hypotheses, authors, and unblinded coding could affect the difference. The operator's proposed adaptation is to seal the criterion, challenge the draft, and verify the resulting questions against evidence. A fresh prompt or model does not create independent assurance. Under the IIA's July 2026 position, an assurance role needs appropriate scope, access, reporting, resources, and safeguards against conflicts. Hiring an outsider alone is insufficient; the Three Lines Model keeps accountability with the organization.

A sealed criterion precedes a model challenge; a separate reviewer has a direct source-access path.The criterion is recorded before the draft is challenged within a management boundary. A review request then reaches a separate person, who receives a direct path to the source records. The figure shows proposed access, not automatic assurance or a claim that a solo operator reproduces formal Three Lines governance.ManagementCriterionDraftModel challengeSourceReviewer

DIRECTOR OR LIMB

Give the owner a reason to stop.

An analyst holds a report after its balance fails the precommitted source comparison.The comparison rule is bound to the source before a trial mismatch is introduced. The mismatch reaches the owner, who closes the acceptance barrier. The final state is a held report, not a premature approval or a claim of established reliability.SourceReportOwnerHold
  • Output: the quarterly report's stated balance
  • Source: the bank statement for the same account and date
  • Threshold: agreement to the declared reporting precision
  • Owner: the analyst responsible for resolving discrepancies
  • Stop rule: hold acceptance when the balance fails to reconcile

Test this check with plausible mismatches, including wrong dates and missing entries. In the example, the trial report fails the agreed comparison and the owner holds it. That demonstrates the stop rule; more failure testing is needed before relying on the check. Count a saving only with evidence of check reliability appropriate to the risk, and when generation, review, and likely repair together beat the manual baseline at the required quality. Otherwise improve the check, narrow the task, or retain the work.

Read the transcript

01 · THE HINGE

The quarterly report arrives before the analyst has opened its source files. The draft is complete. Its tables align and its explanation reads smoothly. Then the analyst opens the records behind it. Knowing whether the claims survive those records still takes work. In this illustrative case, fast generation has left a review bill. Include that bill before deciding whether the offload saved anything.

02 · THE GATE

The analyst compares a report row with its source. The values disagree, so this report cannot pass the check. One detected mismatch establishes that this fault was caught. It does not establish how often the check misses other faults. Saad-Falcon and colleagues define the generation-verification gap more narrowly: a correct response exists among the candidates, but the verifier fails to select it. That is a selection metric. The operator's cost question requires another calculation. Price building and running the check against remaking the work, then add generation and likely repair. Include the normal review needed for the manual alternative too. Compare both routes at the same required quality. Count a saving only when the whole offload costs less.

03 · THE LAST THIRTY

Addy Osmani's December 2024 essay, The Seventy Percent Problem, describes a fast first pass followed by difficult finishing work. Scaffolding and familiar patterns can arrive before integration, correctness, and security are resolved. Seventy and thirty are a heuristic, not measured shares of cost, work, or staffing. The curve illustrates cumulative effort getting steeper toward completion. Another task could have a different shape. The point for the analyst is to price the unresolved work, including the expertise needed to check it.

04 · REVIEW IS THE WORK

Checking a claim in the quarterly report may mean reconstructing a calculation and reading the underlying records. That effort belongs in the estimate even if the draft took seconds. If checking already costs as much as remaking, generation and repair cannot be entered as a saving. The relevant comparison is the total cost of reaching the same required quality.

05 · WHERE ORACLES LIVE

Research systems show how a check can answer a bounded question. AlphaEvolve runs proposed programs and scores specified objectives. In GNoME, density-functional calculations assess predicted crystal stability. A stable calculation alone does not establish a useful material that can be synthesized. AlphaProteo tests designed binders in laboratory assays. Binding to the specified target alone does not establish a medicine. The program run, the calculation, and the assay have different costs and answer different questions. Match the tested property to the decision you need to make.

06 · TWO STUDIES, TWO SETTINGS

Two assistance studies show why the workflow needs its own baseline. The published 2025 paper Generative AI at Work studied five thousand one hundred and seventy-two support agents at one firm. During a staggered assistant rollout, the adjusted estimate was about fifteen percent more issues resolved per hour. METR's early-2025 randomized trial involved sixteen experienced developers and two hundred and forty-six tasks in familiar repositories, primarily using Cursor and Claude three point five or three point seven Sonnet. AI access increased estimated completion time by nineteen percent, with a ninety-five percent interval from two to thirty-nine percent. That estimate adjusts self-reported active implementation time before and after pull-request review for task difficulty. Afterward, developers estimated that AI had reduced their time by twenty percent. These are different settings and outcome units. Neither study isolates verification cost or explains the other's result. METR's February 2026 update describes selection and timing problems that prevent a reliable current estimate from its later experiment. Its May survey measures reported perceptions separately. Treat the early trial as historical evidence and measure the workflow you are changing.

07 · Advertisement · 11x AI Growth Workers

Pipeline is not one task. It is a chain of small handoffs: find the buyer, research the account, respond fast, qualify cleanly, follow up on time. 11x gives sales, marketing, and RevOps teams digital workers for that background motion. The system keeps work moving, while people focus on the conversations that deserve them.

08 · THE SHELF

Return to the report. The analyst ties a stated balance to the bank record. Agreement supports that balance for the account and date checked. It does not validate the explanation beside it. To check that explanation, the analyst gathers several source files and reconstructs the claim. Those paths require different work; this drawing assigns neither a price. Measure the review locally. An operator who keeps responsibility for approval also needs a workable way to challenge the output. Cost alone does not tell us who has that authority.

09 · BUILD THE CHECK

Before inspecting the draft, record the source, threshold, owner, and stop rule. Seal that criterion first. Then ask a model to challenge the draft. Its criticism supplies questions to test against evidence; a fresh prompt or model does not make the review independent. For consequential assurance, arrange a reviewer with direct access to the source records. The role also needs appropriate scope, reporting, resources, and safeguards against conflicts. Hiring an outsider alone is insufficient. The IIA's July 2026 position preserves the organization's accountability even when work is outsourced. Precommitment has a related example in psychology. Scheel and colleagues found author-reported support for the first hypothesis in ninety-six percent of their standard reports and forty-four percent of their Registered Reports, counting full or partial support. Those were their own comparison groups. Nonrandom formats and other differences prevent a causal conclusion. The proposed practice here is to define acceptance before a convenient result can change the rule.

10 · DIRECTOR OR LIMB

Make the rule concrete for this report. Bind the stated balance to the bank statement for the same account and date. Declare the reporting precision, name the analyst who will resolve discrepancies, and require a hold when the balance fails to reconcile. Now try a report with a mismatched balance. The comparison detects the disagreement. The owner holds the report and investigates. That exercises one stop rule; test other plausible failures, including wrong dates and missing entries, before relying on the check. Count a saving only with evidence of reliability appropriate to the risk, and when generation, review, and likely repair together beat the manual baseline at the required quality. Otherwise improve the check, narrow the task, or retain the work.

01 / 10 · THE HINGE0:00 / 7:31