Skip to content

Before an AI Agent Resumes the Task

Desktop experiments show why restoring an AI agent’s earlier state must be judged separately from completing the user’s task.

Wesley Hartman’s conference itinerary needed to work where he would attend. In his November 2025 account of using ChatGPT’s agent, the assistant assembled an itinerary but converted the conference’s Eastern times to Pacific because of his location. Hartman asked it to change them back.

Earlier, the assistant had solved a different problem. Twice it failed to drag the agenda page’s scrollbar upward. It then pressed Home and reached the top. That workaround helped it gather the schedule; it did not establish that the finished schedule would serve the person attending.

These are different kinds of recovery. One finds a workable action. Another repairs the result. For people handing computer work to an assistant, the important question is not simply whether it can get moving again, but what the renewed movement is supposed to accomplish.

What counts as being back?

The October 20 preprint Lost in Recovery tests induced desktop errors using screenshots and replay. Explicit instructions improve restoration but also change objectives and target information. In separate completion tests against no recovery, 2 of 4 models improve and 2 worsen: −3.33 to +2.80 percentage points over 2 runs.

A return score answers a useful, narrow question. It cannot by itself decide whether returning was the right thing to do. If another route reaches the user’s goal, insisting on the old route can mistake adaptation for failure. Hartman’s Home-key workaround is an everyday illustration of that distinction, not a test of the new research system.

Nor is one successful continuation a certificate for every part of a computer. Evidence that an intended sequence works does not logically establish that unrelated data is unchanged. A recovery requirement needs a scope: the interface needed for the next operation, the document’s contents, or some wider consequence. Otherwise, “back to normal” hides the very question a test needs to answer.

The earlier study Human-Guided Harm Recovery for Computer Use Agents chooses a different destination: a safe state that reflects human preferences. Its recovery examples include consequences such as malicious software and lost data. That work is not a replication of Lost in Recovery. It makes clear why an earlier screen and an acceptable outcome need not be interchangeable. A familiar-looking desktop does not tell the person affected whether the harm has been addressed.

The next action has prerequisites

In one example, a terminal covers the font controls; the agent scrolls as if they were open. The workflow also truncates history, announces recovery, adds repair actions and uses model-based error detection. These components were not isolated experimentally.

A correct font choice does not make scrolling sensible: the list must be available for that action.

That is the difference between a plan and readiness to resume it. A plan describes how work could proceed from a particular situation. Repair may leave a different situation. Until the assistant recognizes that difference, a perfectly sensible old step can become a new mistake.

There are several possible responses. Reopen the controls. Choose another way to set the font. Report that the attempted restoration did not establish the required conditions. These are design possibilities, not remedies the study has validated. Their value would have to appear in what happens afterward, including their cost and any new errors they introduce.

Recovery has to earn its interruption

A repair process has to justify the interruption it creates: detecting a problem, choosing what to preserve, attempting a change and deciding how work resumes. Better performance on one part cannot settle the value of the package. A fair practical comparison would also have to account for resources spent continuing, restarting or involving a person.

The same distinction changes what a handoff should communicate. “Recovery attempted” and “ready for the next operation” are different statements. Treating them as synonyms asks the continuing assistant to trust a conclusion that may still need checking. An alternative is to communicate what was actually established and leave the remaining uncertainty visible. Whether that produces better task outcomes is an empirical question, not an answer supplied by a more careful message alone.

Hartman’s itinerary eventually needed an ordinary clarification about the trip. No amount of successful scrolling could establish the intended timezone. The assistant needed both a way to operate the page and a correct understanding of the destination. Recovery earns its place when those come back into agreement, not merely when the cursor starts moving again.

Sources

Understand How People and Agents Work Together.

Practical essays, experiments, and lessons from building software for people and agents.

Occasional emails, sent when there is something worth reading. Free to subscribe. Unsubscribe at any time.