Work That Never Enters the Comparison
When AI changes which software tasks developers choose to attempt, completion times on selected tasks cannot establish whether the changed work plan was more valuable.
Before asking a coding agent about a software migration, a developer read a ticket, its associated GitHub pull-request page and an internal wiki. They pasted wiki content into an inquiry about changes still needed for a migration involving an external dependency. A workplace-shadowing preprint, based on observing 11 developers for approximately 15 hours, documents that sequence. It does not establish that the inquiry would have been skipped without AI or what migration decision followed.
On November 5, AI evaluation organization METR reported a change in Anthropic’s research progress. An index combining the company’s model benchmark scores had grown at an estimated 1.9 times its pre-October 2025 rate. METR judged that AI likely explained most of the acceleration as of October 2026, with substantial uncertainty. The number compares historical progress rates, not researchers randomly assigned to work with or without AI.
For someone choosing what to give a coding agent, the gap between an inquiry and its consequence matters. Assistance might change which jobs are worth starting, as well as how quickly they finish. Timing selected jobs cannot establish whether a changed plan was better. That requires understanding what the added work delivered and what made room for it.
Before the task gets timed
METR encountered a missing part of this comparison in a developer productivity experiment. Participants submitted tasks before random assignment to AI-allowed or AI-disallowed conditions. In a February 2026 update, METR reported that 30–50% of surveyed developers withheld some tasks because they did not want to do them without AI. That percentage counts developers, not omitted tasks.
METR thought selection likely biased speedup estimates downward, but judged its overall productivity signal unreliable. Lower participation pay and difficulty timing concurrent work also complicated interpretation.
Random assignment happens too late to recover a task never submitted. Even perfect timing of the admitted work would leave the withheld work outside that comparison. The omissions could matter precisely because developers expect AI to help on those tasks.
An expectation is still different from a result. A developer might accurately recognize a job an agent handles well. They might overestimate the saving or simply prefer the assisted version. The same decision to withhold a task could follow from any of those reasons. Calling the omissions a hidden benefit would decide their value before observing it.
Nor does keeping a task out of an experiment mean leaving it undone in ordinary work. A developer might complete it with AI rather than risk an unassisted assignment. The reported selection therefore cannot count projects that only AI made possible. It establishes a limit on the comparison, while leaving the value beyond that limit unresolved.
For a team planning its next week, that distinction separates two choices: how to carry out work already accepted, and which work to accept in the first place. A convincing answer about execution speed need not answer the planning question.
What a cheaper inquiry could change
Return to the migration inquiry. Consider a hypothetical developer deciding whether to add that kind of investigation to a plan. Assembling the relevant records might previously have cost too much for the expected benefit. Cheaper assistance could make the inquiry worth attempting. Nothing in the observed example establishes that this is why the actual participant asked their question.
The hypothetical inquiry could reveal a dependency that makes postponement sensible. It could rule out a suspected obstacle and support proceeding. It could also restate familiar material without resolving the uncertainty. All three could end with a completed agent response, yet leave the developer in different positions.
Even an unchanged decision could conceal useful work. If the answer rules out a serious concern, proceeding as planned may now be better justified. The benefit would be stronger grounds for the decision, rather than a different decision or a faster migration. Conversely, a novel answer that changes the plan could still be wrong. Neither novelty nor a change of direction guarantees value.
The alternative use of the time matters too. A useful inquiry could displace an urgent repair. It could replace a weaker investigation, or fit into time otherwise spent waiting. Those are different comparisons. Making the inquiry affordable does not establish that it was the best available use of attention.
This also limits what a lower implementation cost can fix. It may make collecting evidence cheaper. It cannot settle whether the remaining uncertainty matters, whether the answer is trustworthy, or whether another job deserves priority. Those judgments determine which affordable tasks belong on the plan.
From more attempts to better work
The Anthropic assessment includes an interview account of trying more ideas as implementation became easier. Other interviewees favored non-AI explanations for acceleration. Much of METR’s most informative evidence was redacted, and its numerical estimate depends on how benchmark scores are combined.
Trying more ideas could help explain faster research progress. But the public account does not trace a particular newly attempted experiment through to the aggregate improvement. The interview and the progress estimate identify a plausible connection, not a demonstrated chain from additional work to better results.
A coding team faces a similar question without needing an organization-wide model of research progress. An agent’s faster completion of an existing job can be valuable on its own. If assistance also changes the plan, judging that change means following more than the completed artifacts. It means examining why a job became worth attempting, what its result made clearer, and what else could have been done instead.
The migration inquiry makes the difference concrete. An answer about remaining work might justify delaying the migration, proceeding with it, or investigating further. The consequential achievement would be a better-grounded next step. A productivity account that stops when the response arrives leaves that achievement unexamined.
Sources
- METR, Independent Assessment of AI R&D Acceleration at Anthropic in October 2026, November 5, 2026.
- METR, We are Changing our Developer Productivity Experiment Design, February 24, 2026.
- Yi-Hung Chou and colleagues, Developers as Managers of AI Agents: A Workplace Shadowing Study, October 24, 2026, preprint version 1.