Skip to content

Studying With AI for an Exam Without It

In a university trial, students did better on exams taken without AI when their classes introduced instructor-designed AI tools rather than waiting a term to introduce them.

At the end of each module, one university instructor gave students a 10-minute quiz. On paper were 5 situations, each with an AI-generated answer. The students had to decide whether the answer needed changing or what they would ask the AI next. The instructor described the quiz to researchers. The AI’s work was there to read, but there was no live assistant to tell students what to make of it.

That quiz suggests a useful starting point for anyone designing a study tool: decide which part of the work you want students to be able to do themselves. An assistant can help them practice questioning a plausible answer. The paper quiz can then ask them to make that judgment without offering another hint. Those are separate jobs, even in a course that teaches students to use AI.

A November preprint from Jun Li and colleagues reports a randomized trial across 19 universities. The courses covered subjects such as statistics, forecasting and data analysis. In the subset included in the exam analysis, introducing AI tools that instructors had designed improved scores on supervised exams taken without AI, compared with delaying the tools for a term. There was no detectable change in measures of course activity or satisfaction. The authors do not say whether the paper quizzes contributed to the exam result.

Where a course tool had a job

The trial didn’t prescribe one tutor. Instructors set up simulations, role-plays, guided practice and tools that organized course materials. A direct answer about a deadline can spare a student a search. A direct answer to a practice question can take over the work they came to practice. Even in one course, those requests need different responses.

One instructor described students using their own AI for weekly assignments, but believed they made extensive use of the course’s practice-exam tool. Another repeatedly revised instructions for a case-discussion chatbot. It kept giving away details and answering the discussion questions, so the instructor stopped using it. These are two teachers’ accounts, not a record of which students improved. They show why giving a tool a specific job also means checking what it actually does when a student asks for help.

What changed on the exams

Instructors proposed tools before they knew which term they would introduce them. Both groups received design and technical support. Researchers assigned the timing by class, not by student, and compared changes from each class’s past scores. Delaying the course tool wasn’t a ban on students’ usual AI use. The question was what happened when instructors added their own tools to that setting.

The exam analysis included 44 class offerings covering 4,768 enrollments, as well as earlier terms for comparison. The gain estimated from individual scores was 0.32 standard deviations, about a third of the usual spread in scores within a class, not 32% more points. The 95% confidence interval ran from 0.06 to 0.59. Using class summaries, which included students who hadn’t agreed to share individual scores, the researchers estimated a gain of 0.26 standard deviations.

Most exams were on paper. The rest used closely monitored or restricted computers. The researchers excluded 10 class offerings whose exams didn’t reliably rule out AI assistance, a restriction they hadn’t specified in advance. That makes the scores easier to interpret as unaided performance, but narrows the sample after the study was planned. A separate analysis that kept 2 offerings assigned to introduce AI that didn’t meaningfully do so gave similar results. This is evidence about exams in these courses, not about what students retain next year or can do in a new setting.

Better practice scores aren’t the same result

In a high school math experiment in Turkey, students scored higher on practice problems with either a basic chatbot or a version designed to offer hints. On subsequent unaided tests, the students offered the basic chatbot scored 17% lower than students who hadn’t been offered either tool. The hint-based version showed no reliable exam difference from the no-AI group. Scoring the practice problems alone would have missed that distinction.

Nor does adding course materials guarantee an advantage. In a separate university trial, instructors could choose the course materials and how the tutor responded, but most kept the default mode, which offered direct explanations. Students offered the tutor earned lower final grades in the comparison restricted to sections of the same course. Final grades aren’t the same as supervised exam scores, and neither trial tells us that the difference came from who designed the tool. More practice with the kinds of problems a course tests could be part of the gain. Li’s team didn’t compare its course tools with a generic tutor or with the same amount of practice without AI.

The paper quiz makes the choice concrete. If the skill is spotting a missing assumption, a study tool could wait for a student to identify it, then explain what the student missed. A later quiz could supply a different AI answer to inspect without live help. The generated answer can still be on the page. The reason to trust it, revise it or ask another question has to come from the student.

Sources