Skip to content

Where an AI Review Goes Next

For people reviewing an AI assistant to improve its future behavior, useful follow-through means deciding which concerns should change its instructions and keeping those decisions connected to answers they can later examine.

For one university statistics tutor, saving an AI conversation was a way to get another person’s judgment. The tutor was reviewing completed exchanges between students and an AI learning assistant, deciding how the assistant ought to have responded. A new button kept an exchange available for later discussion with teaching assistants or fellow tutors. A teaching question could become something colleagues considered together.

An October 12 research paper shared ahead of formal publication follows 8 tutors doing this work over 3 months, alongside teaching tutorials and grading exercises. They valued new controls for choosing conversations and sharing cases. Yet they remained unsure what their feedback changed. The lecturer had deliberately restricted feedback-driven changes while the feedback’s quality and possible effects were assessed.

That combination matters for anyone reviewing an assistant’s answers or directing a coding agent. Making a review easier to give is useful; making it possible to learn what happened to the concern is another job. My reading of the study is that reviews intended to improve future behavior need an intelligible next step: a decision about what should change, followed by a chance to examine the result. An explanation of why a change was withheld can be part of that work, too.

What the saved conversation made possible

The tutors reviewed anonymized conversations after students had received answers. Each had to give feedback on at least 5 conversations a week. Their task involved judging how the assistant should teach, rather than simply marking answers correct or incorrect.

Repetition was a persistent complaint. Still, participants generally found the overall demands manageable. The added filters let them choose conversations by properties such as length. They particularly valued that control and the opportunity to discuss cases with peers.

Those controls change what a reviewer can do with their attention. Selecting a conversation creates room to pursue a question instead of taking the next item in a queue. Sharing it gives colleagues the same exchange to discuss. Tutors already reported sharing anonymized interactions through private channels before the new feature existed; the interface gave that practice a place in the reviewing process.

The saved conversation illustrates a further possibility. It need not carry only a verdict about the assistant. It can preserve an unresolved judgment for someone else to help make. That is why the tutor’s use of a save button matters: a feature intended for returning to work also supported asking for guidance.

Some tutors found value even without a change to the software. One reported using the assistant’s answers to reflect on their own teaching. Another felt that overseeing AI pulled effort away from helping students, the purpose they associated with tutoring. Visible system change was therefore neither the only source of meaning nor a sufficient account of what these people wanted from the work.

Deciding how much help to give

The assistant’s instructions did change. In week 9, the researchers distilled recurring themes in the tutors’ feedback into a revised system prompt, the instructions that guide its responses. The changes mainly called for shorter context explanations and hints that revealed less of the answer.

This was the only collective feedback-driven update during the study. The lecturer wanted to assess the feedback’s quality and how it might affect the assistant before exposing students to changes. The tutors’ uncertainty arose within that deliberately limited arrangement. It cannot establish that an AI repeatedly ignored corrections applied to it.

The caution has substance. An AI learning assistant can answer correctly while giving away too much of an exercise. Deciding how much help to provide requires a view of what the student should do independently. Accepting every tutor’s preference faster would not automatically produce a coherent teaching policy.

The tutors themselves asked for common guidance. Their judgments sometimes conflicted; one participant described seeing a problem in one conversation and the opposite problem in another. The authors interpret this as a missing shared standard. A screen could show that standard or help people debate it, but somebody would still have to choose what the assistant should do.

A useful response to a review could therefore explain which teaching goal governed the decision. Was the requested change accepted, narrowed or rejected? For the next conversation, that reason could be more informative than a notice that a comment was received. This is a proposed use for follow-through, rather than a benefit measured by the study.

It assumes someone has authority to decide and time to explain. Evaluating feedback takes work, and batching changes delays what reviewers can observe. The lecturer’s quality check is a serious reason to withhold an update. Keeping that decision connected to the original concern would let the reviewer distinguish a considered delay from a suggestion that has simply disappeared.

What the next version still asks of you

An earlier study of developers offers a contrast. In interviews with 17 experienced software-agent users, conducted in July and August 2025, participants described reviewing code, asking an agent for repairs and then reviewing again. Most worked at the authors’ technology company. Two said that a repair could involve changes elsewhere in the code. One began with the overall change, inspected key modified areas and then moved into smaller details.

The depth of scrutiny depended on what was being built. One participant insisted on understanding every line in serious, interconnected software; another granted much more freedom for a demonstration prototype.

Here a request could lead promptly to a revised artifact. That still left the work of establishing what the revision did. The tutors faced a guarded decision about changing instructions; the developers described repeated scrutiny of changes already made. A faster response to feedback can move the uncertainty to the next stage.

For a review tool, this suggests keeping the concern, the decision and the actual revision connected. In teaching, a reviewer might return to later answers to examine the intended balance of help. In software, they might inspect the changed code and test the behavior at issue. The relevant evidence would depend on the concern: seeing a changed instruction would establish that it was edited, while judging its effect would require examining what followed.

That does not require pursuing every comment indefinitely. Checking has a cost, and a prototype can warrant different scrutiny from an integrated production system. The narrower aim is to keep a worthwhile concern available for the next person who can act on it or judge the result.

The saved statistics conversation already did part of that job. It let a tutor carry an uncertain teaching judgment into a discussion with colleagues. If the judgment later informs an instruction change, preserving that connection would give the next review a specific question to return to: is the assistant now offering the kind of help they decided students should receive?

Sources

Understand How People and Agents Work Together.

Practical essays, experiments, and lessons from building software for people and agents.

Occasional emails, sent when there is something worth reading. Free to subscribe. Unsubscribe at any time.