Skip to content
finalstate.aiThe Most Valuable Thing an AI Can Say Is "I Don't Know"
003Atlas

TWO ANSWERS, ONE VOICE

The Most Valuable Thing an AI Can Say Is "I Don't Know"

Two equally finished answers whose different correctness becomes visible only after evaluation.Two schematic answer sheets have the same typography and structure. Evaluation reveals one correct answer and one wrong answer. The marks illustrate the distinction between appearance and correctness, not the measured shares in the consulting study.Same finishDifferent outcomes
The two answers are illustrative, not participant counts. The experiment used GPT-4 in 2023; its results do not describe every task or a current model.

In a preregistered consulting experiment, one interface supported work with different measured outcomes. An answer's finish tells the operator little about the evidence behind it.

  • The 2023 GPT-4 study found gains on one task set and worse correctness on another
  • Evaluation established the difference; fluency did not

SEVEN HUNDRED CONSULTANTS

The edge was drawn from outcomes, not appearance

Eighteen inside-study task marks and one separate outside-study case, divided by a schematic jagged line.The left group contains exactly eighteen fixed marks. One separate case sits to the right. Participants were assigned to disjoint task arms; the positions of the marks carry no empirical distance or difficulty scale.18 tasks1 case
Dell'Acqua et al., Organization Science (2026), doi:10.1287/orsc.2025.21838, §§3–4, Tables 2, 5–7 and Figure 5. Inside: 385 participants; outside: 373, in a separate managerial case. The diagram shows task counts, not a geographic or universal capability map.

The jagged frontier was established for selected tasks. The outside reduction averaged 19 percentage points. Inside, the reported 25.1% time reduction measured reaching question 18 after the first 17, not finishing the whole session.

  • 758 consultants; separate inside and outside task arms, each with three randomized conditions
  • Inside: 18 tasks, 12.2% more completed and 32% higher rated quality with AI
  • Outside: correctness 84.5% without AI, 70.6% with GPT-4, 60.0% with GPT-4 plus an overview

Polish does not locate the boundary

The researchers could compare outputs with expected outcomes. An operator reading a new answer may lack that comparison. Confidence in the wording cannot replace evidence about performance on the work at hand.

EVALUATION SUPPORT

A familiar task can require an untested step

A tested three-step task compared with a new task that requires a fourth, untested step.The lower task shares three steps with the evaluated task above, then extends beyond its evidence boundary. The added step is unresolved. This schematic concerns evaluation coverage; it does not depict a language model searching its training data or classify the task as out of distribution.EvaluatedRequired?

Use out of distribution for a case outside a stated reference distribution. Lack of evaluation support is a different claim. Neither establishes that the task was absent from training data.

  • Relevant evaluation must cover the required capability
  • Unsupported by evaluation does not mean incapable
  • Out of distribution needs a specified reference distribution

RISK AND UNCERTAINTY

Uncertainty must be taken in a sense radically distinct from the familiar notion of Risk, from which it has never been properly separated.

Frank Knight, Risk, Uncertainty and Profit (1921), Part I, Chapter I

Knight's economic distinction concerns whether uncertainty can be measured. It is an analogy here. A probability written in an answer still needs an evidential basis; the absence of a per-answer check alone does not establish that usable odds are impossible.

THE WINS YOU CAN MEASURE

State the task, population and measure

Separate measurement exhibits for support output per hour and coding completion time.The support index is 100 at baseline and 115 with assistance, representing the published average estimate. An independent minutes scale shows coding means of 160.89 without Copilot and 71.17 with it, conditional on successful completion. These are different estimands from different studies, not a common productivity scale.Support indexNo AI100AI115No AI160.89Copilot71.170120Coding minutes0180
Brynjolfsson, Li & Raymond, QJE (2025), §V.A/Table II: adjusted resolutions per hour, shown as an index with baseline 100. Peng et al., arXiv:2302.06590v1, Results/Figure 6: mean 160.89 versus 71.17 minutes among completers; reported reduction 55.8%, 95% CI 21–89%. Completion probability is a separate outcome. The scales are independent.

The support study followed staggered adoption of a GPT-3-based assistant. The coding experiment randomized 95 developers; 70 completed the task, 35 in each group. Such gains leave open the strategy question of what value survives imitation.

  • Support: 5,172 agents at one firm; an estimated 15% more issues resolved per hour
  • Coding: one JavaScript HTTP-server task; 55.8% less completion time among successful completers
  • These studies measure bounded assistance gains, not the benefit of an abstention policy

FIND A VERIFIER

A completed check supports a bounded claim

Two candidates wait at an external check before one passes and the other is held for an owner.The upper candidate receives a checkmark only after its check completes successfully and then moves to the answer route. The lower candidate fails its check and moves toward an accountable owner. An unavailable check also needs a review route; a pass covers only the tested criterion.CheckAnswerOwner

In a verifiable space, a test can decide a specified claim. Elsewhere, a statistical rejection policy may control aggregate answered-case error under its validated conditions. It does not independently certify each answer.

  • Test the candidate against an external criterion
  • A pass supports only what that check covers
  • A failed or unavailable check needs a defined review route

TWO ERRORS

A refusal policy can fail in two directions

A matrix of returned or withheld candidates, each independently labelled correct or wrong.The wrong-and-returned cell is false acceptance. The correct-and-withheld cell is false refusal. The matrix includes only candidates whose correctness has been established, including retained withheld candidates. Unknown outcomes are excluded rather than counted as errors.CorrectWrongAnswerHold

Use held-out cases with known outcomes. An unlabelled answer is not a wrong answer. A policy that refuses everything has zero coverage; its answered-case error rate is undefined, not zero.

  • False acceptance: a wrong candidate is returned
  • False refusal: a correct candidate is withheld
  • Retain and independently label withheld candidates to measure the second error

RISK AND COVERAGE

Measure the rejection policy on relevant cases

A complete schematic risk-coverage curve with an operating point moving along the curve.The horizontal axis is the proportion of evaluated cases answered. The vertical axis is the error proportion among those answers. Only the operating point moves; the illustrative curve remains fixed and complete. Its shape represents a useful ranking, not an inevitable tradeoff or a measured language-model guarantee.RiskCoverage010%100%
Geifman & El-Yaniv, NeurIPS (2017), §§2–3 and §5. Curve is schematic, not measured LLM performance. Risk control assumes labelled cases sampled independently from the same underlying distribution; deployment shift requires fresh evidence. A poor ranking need not lower error as rejection increases. At zero coverage, selective risk is undefined.

Geifman and El-Yaniv's selective-classification method bounds risk under stated sampling conditions. Their image-classifier experiments do not demonstrate language-model calibration or guarantee a useful threshold in every setting.

  • Coverage = answered cases ÷ evaluated cases
  • Selective risk = wrong answers ÷ answered cases
  • A useful threshold must meet both an error limit and a coverage requirement

THE HANDOFF

Make refusal a tested route to an owner

Evaluation informs a rejection boundary; a withheld task is routed to an accountable owner.Labelled evaluation cases inform a policy threshold before a new case is routed. A refused case crosses to an owner for further judgment. The threshold does not establish individual correctness or make the human decision disappear.EvaluateThresholdOwnerHeld case

If the tested policy cannot meet both requirements, narrow the scope or increase review. A human owner remains responsible for the unresolved decision and for judgment beyond prediction.

  • Set the acceptable error rate and minimum useful coverage
  • Validate the rule on representative held-out cases
  • Assign an owner for refusals; record outcomes and retest after changes
Read the transcript

01 · TWO ANSWERS, ONE VOICE

One interface can produce a polished answer on either side of a capability boundary. In a consulting experiment using GPT-4 in 2023, AI assistance improved work on one task set and reduced correctness on a different task. The researchers scored the work to establish that difference. The appearance of an answer did not establish correctness.

02 · SEVEN HUNDRED CONSULTANTS

The experiment involved seven hundred and fifty-eight Boston Consulting Group consultants in separate inside and outside task groups. Within each group, people received no AI, GPT-4, or GPT-4 with a prompting overview. Inside, eighteen tasks showed gains in completion and rated quality. Outside, on one selected managerial case, correctness fell by an average of nineteen percentage points with AI. That boundary belongs to the tasks and model tested.

03 · NO WARNING AT THE EDGE

A polished response may be right or wrong. To separate them, the researchers needed outcomes and a way to score the work. An operator facing a new task may have neither. A refusal policy has to be evaluated against those outcomes too. Merely asking the model to sound less certain gives us another sentence to assess.

04 · EVALUATION SUPPORT

Name the boundary precisely. A task can resemble evaluated work and still require an untested step. The first three steps may be familiar; the fourth may need evidence the evaluation never collected. That leaves a gap in support, not proof of inability. Out of distribution is a separate description: it needs a case and a specified reference distribution. Neither label tells us that the prompt was absent from training data. Ask what evidence covers the capability required here.

05 · RISK AND UNCERTAINTY

Frank Knight distinguished measurable risk from uncertainty in his 1921 book. This economic distinction is an analogy here, not an explanation of machine learning. A probability written in an answer needs an evidential basis. But lacking a test for this particular answer does not, by itself, prove that useful probabilities are unavailable. We need to distinguish checking one claim from measuring error across a class of cases.

06 · THE WINS YOU CAN MEASURE

In the published support study by Brynjolfsson, Li and Raymond, access to an assistant raised issues resolved per hour by an estimated fifteen percent across five thousand one hundred and seventy-two agents at one firm. Adoption was staggered, not randomized. Peng and colleagues randomized ninety-five developers on a JavaScript HTTP-server task. Among the seventy who completed it, the Copilot group took fifty-five point eight percent less time. This is conditional completion time, not the probability of finishing. Neither study tests a refusal policy. Each gives a task, population and measure against which broader claims can be checked.

07 · FIND A VERIFIER

There are two routes to evaluate. First, test a candidate against an external criterion. Wait for the result. A successful check supports the claim it actually tests; then the candidate can follow the answer route. A failed or unavailable check takes the review route. Passing code tests, for example, does not establish every security property. Second, where individual checks are unavailable, a tested rejection rule can select which cases to answer. Its evidence concerns error across accepted cases, not certainty about this one.

08 · Advertisement · Bubble AI App Builder

Some ideas do not need another document before they become testable. With Bubble AI, you describe the app you want, and Bubble creates a working starting point: interface, data, and logic you can inspect. From there, you refine visually, connect AI models and services, and turn the first version into something real enough to use.

09 · TWO ERRORS

A rejection rule can make two mistakes. It can return a wrong candidate: false acceptance. It can withhold a correct candidate: false refusal. To measure the second error, retain the withheld candidates and establish whether they were correct. Otherwise, the claim that the system could have answered is an untested counterfactual. Keep unknown outcomes separate. Refusing everything gives zero coverage, with no answers on which to calculate an error rate. Answering everything avoids refusals but leaves every candidate error in the returned set.

10 · RISK AND COVERAGE

Coverage is the fraction of evaluated cases answered. Selective risk is the fraction of those answers that are wrong. Evaluate several thresholds to see what each retains. A useful ranking may let us reduce error by answering fewer cases. A poor ranking can discard correct answers while retaining errors. Geifman and El-Yaniv developed a risk-control method for classifiers using labelled samples from the same underlying distribution. They tested image classifiers, not language models. This curve is illustrative. The actual policy still has to meet both the error limit and the coverage requirement.

11 · THE HANDOFF

The phrase I don't know becomes useful when it changes what happens next. Set the maximum acceptable error and minimum useful coverage. Validate the rejection rule on representative held-out cases. If it cannot meet both requirements, narrow the scope or add review. Then assign an accountable owner to refused cases. That owner may consult a source or another tool, but the unresolved decision still belongs to someone. Record outcomes and retest when the model or domain changes.

01 / 11 · TWO ANSWERS, ONE VOICE0:00 / 6:05