Skip to content

Would You Spot What Changed in Code You Built With AI?

In a small Java experiment, programmers spent less time writing code with a chatbot, but later spotted fewer deliberately planted errors in that code than in code they had written without the chatbot.

One participant in a November programming experiment described checking code they’d produced with an AI chatbot: “tests pass, zap, done.” The researchers had another job for them: look at the code again, this time with errors the researchers had planted, and mark what was wrong. No chatbot was allowed for that part.

The 35 people in the study’s analysis spent less time on their small Java assignments when they could use the chatbot. But in the later bug hunt, they took longer and caught fewer of the changes on average in code they’d finished with its help. This happened in the same session, not after months away from the code.

One assignment was to calculate an invoice from a customer’s type and order amount. For a team changing a billing rule, tests can check whether particular invoices come out right. A disputed charge asks something else of the person investigating it: can they connect the total on that invoice to the rule the software applied? A passing test tells the team about the code’s behavior, not whether someone can account for that behavior when it needs explaining.

What the bug hunt took away

The experiment included students, academic staff and professional developers. Each wrote two different Java methods, one with a chat interface to Qwen3-32B and one without it. This was a chatbot, not an agent operating the editor. The original submissions passed the researchers’ tests. Then an experimenter inserted 4 changes into each submission, such as switching an arithmetic operator or reversing a condition. The altered code still compiled. To find the changes, participants had the task description, an example and the altered code, but couldn’t consult other resources.

They found 88.5% of the planted changes in code they’d written without chat and 80% in code they’d made with it. They still did better than on a third assignment whose description they’d read but whose solution they hadn’t seen: there they found about 67%. That third comparison is rougher because the researchers supplied the solution and planted 5 changes rather than 4.

The chatbot’s initial time advantage was much larger than the difference in inspection time. In an exploratory analysis, participants spent an average of 14 minutes and 31 seconds working on the programming task with chat, versus 24 minutes and 41 seconds without it. That excludes the time the researchers spent testing or giving feedback. Participants later took 30 seconds longer on average to submit their inspection of the chat-assisted code. They also found fewer changes, so the comparison doesn’t say how much work it would have taken to catch the same number or to fix them.

What a reviewer needs to recognize

Five participants said that having made mistakes while writing their solutions helped them know where to look during the bug hunt. Writing the method gave them a chance to find out where an apparently reasonable approach went wrong. But participants also credited more of their discoveries in code written without chat to remembering its original form. Recognizing a changed line is useful, but it doesn’t always mean you can explain the rule it implements. And the researchers didn’t test identical programs with identical errors: a particular implementation might itself make a change easier or harder to spot.

A separate GitHub Copilot study asked 15 graduate students to add features to an existing web app. They finished the first subtask faster and passed more tests with Copilot, but there was no clear difference on questions about the existing app, such as what would happen if a block of code were removed. They always used Copilot second, when they’d had more time with the codebase. This wasn’t another test of whether people could spot changes to their own fresh code. It helps separate two things a team might need from a developer: knowing the part they just changed and knowing how it fits into the rest of the program.

Outside the lab, a team needn’t rely on one developer’s memory. A colleague, a version history or a chatbot with enough context can help trace a calculation. And for a small function that only removes the final line break from a string, another of the experiment’s assignments, a few precise examples may tell a team most of what it needs. How much of the implementation someone needs to understand depends on what that person will have to do next.

Suppose the invoice rule grants a discount only above a given order amount. A reviewer could predict the totals for two orders on opposite sides of that threshold, then trace where the code makes the distinction. The study didn’t test this as a way to improve later repairs. What it would show the team is more immediate: whether someone can tell a customer why one charge received the discount and the other did not.

Sources