Skip to content

Can AI Tell What You’ll Keep?

AI models struggled to predict professionals’ use of generated contributions in recorded work, suggesting that assistants learning from revisions need to distinguish useful material being adapted from material being discarded.

An architect asked Claude, an AI assistant, to arrange rooms within an existing floor plan. The architect liked the proposed arrangement of bedrooms and bathrooms, but found problems with movement through the lobby and the placement of storage. In an annotated account from a new study, the architect then rebuilt walls and partitions in Revit, software for designing buildings. The useful arrangement survived; the connections needed work.

That matters for anyone using AI to develop something they will finish themselves. A changed answer can contain an idea worth keeping. If an assistant is to learn from your revisions or decide what help to offer next, it needs to understand which part you valued and which part you repaired. A single verdict on the whole response can lose that distinction.

The study, called Kolab and posted as a research paper on October 15, tests whether AI models can predict professionals’ decisions to use, change or reject generated contributions. The models evaluated recorded work; they were not doing the jobs themselves. They struggled to anticipate the decisions, particularly changes and rejections, even though most reactions the researchers labeled involved unchanged use. My reading is that the difficulty points to an important requirement for helpful assistance: understanding an answer in relation to the work it enters.

Where the answer goes

The architect’s room arrangement and the routes between rooms were separable. Keeping one while changing the other made the AI contribution part of a design process. Counting the whole proposal as a failure would erase the retained arrangement. Counting it as finished would erase the design work still required.

Kolab makes some of that continuing work visible. Its researchers recruited 50 professionals to record 100 sessions on problems from their own jobs. Participants had to produce concrete deliverables and use supplied Claude access; other AI tools were allowed. They were encouraged to interact repeatedly with AI. These were paid, planned sessions, rather than an unobtrusive sample of an ordinary working week.

The researchers connected generated contributions to the human actions that followed, including actions in other applications. About two-thirds of the annotated reactions involved using a contribution unchanged, and roughly another fifth involved using it with modifications. These labels record incorporation into the work. They do not independently establish that the resulting work was correct or that using AI saved time.

A separate study of professional editors using ChatGPT alongside Word places important decisions outside the conversation too. Its published abstract describes 6 editors revising 4 Dutch business letters. They asked ChatGPT for candidate changes ranging from complete rewrites to local reformulations. Most evaluation, adaptation and integration took place in Word. The researchers describe particular revision activities being replaced, rather than the editing task as a whole.

This is background evidence about editing, not a replication of Kolab’s prediction tests. It shows why following the work beyond chat matters. The conversation can contain a candidate sentence while the document contains the decisions that make it usable. The same apparent end to a chat could precede careful adaptation, immediate use or abandonment. Silence alone cannot identify which happened.

Anticipating the next decision

Kolab gave models two different views of a contribution’s fate. In one test, they could see the professional’s response and had to recognize what had happened. In the other, their input ended after the AI contribution, before the human reaction. They had to forecast the reaction from the information available so far. Seeing the response generally made the classification easier.

In the video-only forecasting test, Gemini 3.1 Pro predicted unchanged use for 52% of the cases that professionals modified and 71% of those they rejected. That pattern was not universal: a tested Qwen model strongly favored predicting modification instead. Different models could miss the person’s decision while favoring quite different answers.

A default of expecting unchanged use would often be right in this dataset. But for an assistant deciding what to offer next, the departures matter. A person adapting a useful proposal might need help carrying a change through the rest of a document. Someone discarding it might need a different approach. These are possible uses for a good forecast, not benefits demonstrated by the study. An assistant that cannot distinguish the decisions has less basis for choosing between them.

One plausible difficulty is the standard against which the person judges. The architect has to fit a room arrangement into an existing building plan. The editor has to fit a sentence into a particular letter. A response can satisfy part of a request while conflicting with another part of the developing work. The person may also discover what needs changing only while trying to use it.

That interpretation does not require treating every revision as evidence of a hidden preference. Some requirements may be explicit and still hard for a model to follow through a recording. Kolab’s professionals supplied plans and were encouraged to think aloud, so the evaluators had more to work with than bare chat. Difficulties interpreting the recordings and tendencies to favor particular labels are serious alternative explanations for the forecasting errors. The study does not isolate missing requirements as their cause.

There is also a limit to what can be known in advance. In another Kolab example, a person making a video abandoned a generated clip whose audio was unusable and had no credits left for another attempt. That decision involved both the material and the resources available to continue. Predicting use is a broader problem than assessing an answer’s intrinsic quality.

What should an assistant learn?

More history might seem the obvious remedy. Yet supplying preceding work did not consistently improve Kolab’s forecasts. Some settings improved; more history was not a dependable solution across the tested conditions. Better selection or presentation of information remains an open possibility.

The architect’s changes suggest what such information might need to convey. A record of a wall moving shows an action. An explanation of the circulation problem connects that action to a requirement. Preserving both the retained arrangement and the reason for changing its connections would give an assistant a more specific lesson than a single approval score.

This is a design direction, not a repair tested by Kolab. It would also impose a cost: explaining decisions takes time, and people cannot always state a reason before working through a change. An assistant asking for explanations after every edit could consume the attention it is meant to save. The value would depend on whether the explanation helps with consequential later work.

For people comparing AI tools, the same distinction changes what counts as useful evidence. Counting unchanged outputs can make sense when the purpose is to avoid editing. When the purpose is to develop a design or improve a document, that count can overlook useful starting material. Counting every modification as success would be equally misleading if rescuing the output took longer than starting over. Kolab does not supply that time comparison.

The architect’s record supports a more specific conclusion: the room arrangement was worth retaining, while its connections needed work. A future assistant learning from that revision would need to preserve the first judgment as well as act on the second. Otherwise, the next proposal could lose the very idea that made this one worth changing.

Sources

Understand How People and Agents Work Together.

Practical essays, experiments, and lessons from building software for people and agents.

Occasional emails, sent when there is something worth reading. Free to subscribe. Unsubscribe at any time.