Skip to content

Bringing an AI Conversation Back to the Draft

When writers use AI to examine an argument, the useful connection is the decision about the draft; a classroom study found an association between discussion-and-editing sequences and score gains, without showing that its structured tutor produced better essays.

Students in four Swiss secondary-school classrooms wrote about a question their tools would soon make practical: “Does using AI change what it means to learn?” After drafting independently, they were allotted about an hour to revise with an AI tutor. The students had to supply the writing. Both versions of the tutor were instructed not to produce essay text.

One tutor was designed to diagnose weaknesses in an argument and guide the writer through questions toward a revision. Yet the November 5 preprint reporting the study showed no demonstrated writing advantage over the comparison chatbot. Composite essay scores improved for 12 of 30 students using the structured tutor and 13 of 32 using the comparison system.

For someone discussing a proposal, article or explanation with AI, the interesting question is what happens between the exchange and the document. Does the writer have a reason to change a passage, investigate a missing fact or leave the draft alone? The classroom study found a promising association between discussion-and-editing sequences and score gains. It did not establish that the ideas being discussed actually entered the revisions.

What would make the conversation useful?

The paper illustrates its ambition with a passage that equates obtaining a correct answer with learning. In an explicitly constructed example, not an observed classroom exchange, the tutor questions that assumption. The imagined student distinguishes copying from understanding, then revises the claim to account for the difference.

The revision matters because it changes what the argument can support. The writer has not merely added words or prolonged a conversation. A broad assertion has become a more defensible claim. That is a recognizable purpose for discussing a draft with a teacher, another reader or an AI assistant.

The experimental tutor, MAIKE, tried to support this connection by analyzing the essay’s argument, assessing its quality and selecting questions aimed at weaknesses. It kept an evolving model of the student’s responses and reviewed successive revisions. These were AI-generated assessments, including inferences about the student’s understanding. A confident diagnosis was not necessarily a correct one.

The comparison system was more capable than a generic chatbot with no context. On each turn, it received the essay question, current essay and conversation history, under a single educational tutoring prompt. It shared MAIKE’s language model, interface, editor and Think/Write modes. Both systems were told to withhold answers and essay text.

The comparison therefore tested a package of diagnostic and conversational support. It could not establish that withholding generated prose, asking questions or separating thinking from writing was responsible for any difference. Those features were already present on both sides.

A visible draft, an incomplete connection

Students could read and edit their essay alongside the chat, then resubmit it for another round. The schedule allowed about 30 minutes for an initial draft and 60 minutes for interaction and revision. Throughout the task, they could not consult external sources or tools, including reference materials and search engines. This was an exercise in examining an argument, not researching and preparing an article for publication.

Of 65 students who consented to research use of their data, three were excluded because their activity was almost entirely off-task. That left 62 for the writing and process analyses. Assignment was random and balanced within each class, but the analysis did not retain every consenting student. English was mostly a second language, regularly used in their education. The findings concern these students working on one supported essay.

The outcome measure also needs care. Each student’s drafts went to the same teacher, without identifying the condition, conversation or draft order. Both teachers scored the drafts of 10 students to check agreement. The researchers then dropped evidence and dialectical reasonableness from the original five scoring traits because agreement did not exceed chance. Agreement on the retained traits and composite was fair to good, and each trait had only three score bands.

Those scores provide a limited view of change. The nearly matching improvement counts demonstrate neither MAIKE’s superiority nor equivalence between the systems. A small study with coarse scores could miss differences that a larger study or a more reliable assessment would detect.

To examine the process more closely, researchers coded messages that engaged with the argument’s ideas and edits that changed meaning. An idea-level message might take up something supplied by the tutor; it was not necessarily an independently generated insight. A substantive edit changed claims, examples or their connections, rather than just wording or formatting. That classification did not say whether the change was good.

The researchers paired each substantive edit with the most recent earlier idea-level message. Other messages and actions could intervene. Several idea-level messages followed by one edit counted as one pair. This recorded a sequence, not a verified connection between the content of the discussion and the content of the revision.

Within MAIKE, students with more pairs tended to gain more on their essay scores. The reported correlation was 0.49, compared with a weakly negative −0.14 in the control group. Across 13 exploratory process measures, counts of ideas or substantive edits alone did not clearly distinguish the groups’ relationships with improvement. The suggestive result concerned how the activities were arranged.

A well-targeted question could help a student reach a useful revision. But a more persistent or capable student could also sustain the exchange and improve the draft. A weaker starting essay could leave more room for gains. Random assignment of the tutor does not randomly assign these behaviors within each group. The result gives researchers a reason to inspect the connection more closely, not writers a reason to add discussion-and-editing cycles for their own sake.

Which part of the work is worth protecting?

There is another limit to treating idea-level conversation as the goal: writers do not all want help with the same work.

A separate 2025 study interviewed 15 writers with experience using AI writing tools, 11 of whom reported professional writing experience. A professional literary-fiction writer, identified as W7, located ownership in “sentences themselves and how sentences are sculpted”. The same person was more willing to delegate formula-constrained academic writing. Their boundary changed with the task.

A hobby fantasy writer in that study used AI for starting prompts and background settings to get to the writing they wanted to do. These are reported preferences from a small, selected group, not evidence of better performance. They nevertheless complicate a hierarchy in which discussing ideas is always the serious work and choosing words is merely execution.

For an argumentative classroom essay, examining a claim and its reasons is central. For W7’s literary work, rhythm and sentence structure could be the contribution worth retaining. An assistant that kept drawing that novelist into abstract discussion might be directing effort away from the work they valued.

That makes the destination of a writing conversation more specific. The useful decision could be to qualify a claim, reshape a sentence, find evidence or reject a suggestion. Counting edits would miss a considered decision to preserve the passage, just as counting messages can miss a conversation that has stopped helping.

A question can create the wrong work

Withholding prose does not remove the assistant’s influence. A question points toward an apparent problem. If the system misreads an argument, the writer may spend time repairing something that was not broken or accept a framing that obscures the intended point.

The MAIKE authors identify this diagnostic reliability problem: an incorrect assessment can produce a misdirected question, and the tested interface gave students no way to distinguish a sound diagnosis from an unreliable one. Before deciding how to revise, a writer still has to decide whether the question fits the draft.

The elaborate system also consumed more time. Median response latency was 18.3 seconds for MAIKE and 4.1 seconds for control. Duration measures included waiting, so more time in the interface was not necessarily more time thinking. Under a deadline, even useful questions compete with reading, checking facts and writing.

The study followed one supported essay in one session. It did not establish durable learning, better later writing without AI or reduced reliance on assistance. Those would require evidence beyond changes to the current document.

For ordinary writing, a practical interpretation is to bring the conversation back to the passage it is meant to help. What does the exchange give the author reason to change, preserve or investigate? This is a proposed way to judge assistance, not a procedure the classroom study validated.

In the paper’s constructed example, distinguishing copying from understanding gave the imagined writer a reason to qualify a claim. A real writer also needs the option to decide that the question is misplaced and keep the sentence. The document, rather than the length of the exchange, is where that judgment has to matter.

Sources

Both sources are licensed CC BY 4.0. This article condenses, paraphrases and critically interprets them; the short interview quotation retains its wording. Their authors do not endorse this interpretation.

Understand How People and Agents Work Together.

Practical essays, experiments, and lessons from building software for people and agents.

Occasional emails, sent when there is something worth reading. Free to subscribe. Unsubscribe at any time.