Deciding What an AI-Reconstructed Study Will Actually Do

The AI had found the rule for choosing the numbers, but not the numbers themselves. In an author-provided demonstration of new research software, an extraction pipeline read an older experiment’s description of numerical anchors while missing values in a scanned table. Those values mattered: the experiment asked people to make estimates after encountering a high or low starting number.

The tool, HumanStudyProgram, was turning the published experiment into a runnable program. It marked the missing values beside their source passage instead of supplying plausible replacements. The demonstration shows a researcher consulting the original table and entering the values, keeping their source attached. This was a worked example, not a measurement of how reliably the tool reconstructs unfamiliar papers.

For someone asking AI to build on a study, this is consequential work before any claim of replication. A readable account can leave instructions implicit. Executable code needs to settle what will happen. A November 9 preprint describes a system that brings some of those decisions into review, and reports how 12 experienced researchers used it with studies they knew well. Their work shows why approval needs to mean more than accepting a program that runs.

A number has to come from somewhere

The scanned-table case is a recoverable omission: a value was absent from the extraction but present in the available source. Other gaps are harder. If the record does not say what happened, checking it again cannot supply the missing fact. A researcher may need more materials, author contact, or an explicit decision to depart from the original procedure.

That distinction appeared in the hands-on expert review. A researcher identified as P7 encountered a conflict between a scale described in a paper’s methods and the scale used in its experiment. The tool flagged the mismatch and suggested contacting the authors. P7 said a careful reading had also made the discrepancy seem suspicious. The new paper reports this encounter; it does not establish that the authors were contacted, the conflict resolved, or the original publication corrected.

The interface puts extracted fields beside their source evidence. Researchers can revise a field, supply information, ask for another extraction, and retain notes about unresolved questions. Approval therefore covers choices with different meanings: confirming a value read from a table is not the same as accepting a substitute because the record is incomplete.

Across the 12 reconstructions, the tool surfaced 98 items for confirmation, with 4–15 per study. The items included verification, missing information, ambiguity, conflict and accepted departures. That count describes work brought to researchers’ attention. It is not 98 independently verified errors corrected, an error-detection rate, or a guarantee that every important gap was found.

What running the reconstruction commits to

HumanStudyProgram organizes a study into materials, participants, procedure, responses and analysis. Its participant-facing operations present information and collect answers, while Python handles assignment to conditions, branching, calculations and the sequence of events. The simulated respondent’s model and configuration remain separate from the procedure.

The paper’s repeated group-task example makes the sequence concrete. Each round collects guesses, calculates a group result and gives feedback before the next round. A reconstruction containing the right questions but omitting that feedback would give later answers a different informational basis. Matching words is only part of matching the experiment.

Once the researcher approves the structured description, the tool generates an executable package. Procedure code gathers records; separate measurement code turns them into measures and statistical comparisons. The tool inspects code structure and performs a test run before making the package available. These are operational checks. They do not establish that a chosen scale, assignment rule or analysis faithfully represents the intended experiment.

Nor did this paper invent reviewed extraction into executable studies. The same team’s earlier HumanStudy-Bench already separated agent implementations from protocols and used human review of extracted materials and generated configurations. The new work contributes a researcher-facing representation, connected authoring workflow and hands-on expert evaluation. Its value has to be judged at that level.

The reviewer needs a basis for approval

The 12 participants were pursuing or had completed doctoral training, had more than 3 years of human-subject research experience, and had used language models and simulated participants before. They chose familiar studies, sometimes their own or collaborators’ papers. A tutorial and help with interface operation accompanied the remote session.

All completed review, configured simulated participants, executed a package and inspected its results. That is encouraging evidence that this selected group could navigate the connected workflow. It does not establish how well a novice would reconstruct an unfamiliar paper, or how many incorrect decisions passed unnoticed.

The workload also remained visible. P5 and P9 found the volume of reconstructed information demanding and wanted a map of the relationships between its parts. Each spent 20–25 minutes reviewing the reconstruction. P2 and P3 wanted simpler guidance for configuring runs.

Across the group, review and approval took approximately 20 minutes, with reported bands spanning 10–15 to 25–30 minutes. Extraction was cached before the sessions; the tutorial, configuration, runs, results inspection and interview were separate. There was no controlled comparison showing time saved or reconstruction accuracy improved. Source links and flagged questions give a reviewer something specific to investigate, but still depend on that person’s attention and knowledge.

After the program runs

The record of a run offers another place to check the reconstruction. The system stores participant settings and, for each call, the visible context, raw model output, parsed answer and resulting study state. Identifiers connect the analyzed records with the calls that produced them.

During the expert review, researchers moved from overall findings to conditions, response distributions and individual exchanges. P9 described being able to keep going deeper into a result; the authors observed similar navigation by P4, P8 and P11. The system gives an unexpected chart a trail back to concrete inputs and answers, rather than asking the researcher to accept the aggregate alone.

That trail does not explain a model’s behavior by itself. It lets a reviewer investigate several different questions: what the original record said, what the reconstruction chose, what context the artificial respondent received, and how its answer became a measurement.

The framework is limited to text-based procedures, and its authors explicitly distinguish synthetic runs from evidence about human populations. A program can preserve an experiment’s sequence while its artificial respondents behave unlike the people the experiment concerns. Faithful instructions and representative participants are separate achievements.

The missing anchor can be recovered from a table. P7’s scale conflict may need a different resolution. Keeping those decisions attached to the runnable reconstruction preserves what its success does and does not mean: which parts were recovered from evidence, which were chosen, and which scientific questions remain open.

Sources

Study descriptions, findings and participant accounts are adapted and critically interpreted from Xuan Liu, Haoyang Shang, Yuanjun Feng, Shao-Yu Chu, Qiyu Li, Yash Raj Shrestha and Haojian Jin, Executable Human Study Programs with Silicon Participants, arXiv version 1, November 9, 2026, licensed under CC BY 4.0. This essay changes and interprets the material and does not imply the authors endorse its conclusions.

Historical comparison uses Liu and colleagues’ HumanStudy-Bench: Towards AI Agent Design for Participant Simulation, arXiv version 2, August 29, 2026, also licensed under CC BY 4.0.