Skip to content

Who Decides When an AI Errand Is Done?

LifeVal makes personal errands testable by pairing owners’ requirements with checks of completed actions, but its average score cannot show how often an errand meets every important requirement.

Sky Canaves asked ChatGPT’s agent to plan and buy ingredients for a Japanese breakfast for four. The order eventually arrived, but first she took over the cart to remove unwanted items, including ingredients already in her pantry. Her public account of the experiment discloses no spending ceiling or pantry inventory in the initial request. Getting an order delivered and discovering what she wanted to order were parts of the same job.

OpenAI’s LifeVal benchmark, announced November 2, tackles a related difficulty: testing an AI on work whose correct outcome depends on the person asking. Its 169 tasks pair personal information with simulated environments where agents can act. The people contributing the tasks also write the scoring criteria, called rubrics, that specify what a satisfactory completion requires.

That makes more of a personal errand inspectable. A useful shopping assistant has to connect a person’s requirements to the groceries it selects, not merely operate the website. LifeVal tests such connections and reports substantial progress within its setup. Its average score still leaves an important question open: how often does a completed errand meet every condition the person cannot afford to have missed?

Giving an errand an answer key

One of the paper’s example requests asks for a meal plan during a three-week visit and groceries for the first week. It includes a nut allergy and a father’s recent diabetes diagnosis. The example has been deidentified or synthetically altered to protect contributors’ privacy. It is a benchmark request, not a documented family’s completed grocery order.

Those details give the agent work that a generic menu cannot settle. The meals must fit the people eating them, and the shopping must support the plan for the requested period. A list of plausible recipes and a populated cart leave those connections unchecked unless the evaluator knows what to look for.

LifeVal draws that knowledge from people with experience of the task. Contributors had to have performed their task repeatedly and lived with its results for more than a year. They supplied their own rubrics and relevant material, including ChatGPT histories shared with consent and deidentified for research. They were paid contractors recruited through task-writing pools and targeted outreach, rather than a random sample of households. Tasks passed several rounds of human review and model-assisted checks.

The resulting criteria let an evaluator check work without sharing the owner’s life. That is a useful way to make personal requirements repeatable. It does not make the rubric an exhaustive account of what the owner might accept: another meal plan could be equally good, or a written condition could omit something that matters later.

Canaves’ experience shows why the starting brief also matters. Her cart review brought pantry duplicates and unwanted purchases into the decision. The disclosed account does not show an agent ignoring a budget or inventory she had supplied. It shows requirements becoming clearer during the work. A test built around specified conditions can assess execution more cleanly than it captures that negotiation.

Looking for the work itself

An answer key would be little help if an agent could collect its points by announcing success.

LifeVal’s grader receives evidence of the completed attempt: the request, personal context, rubric, final response, created files and changes in the task environment. The GPT-5.6 grader must verify that claimed artifacts exist and use environment changes to check actions. A separate grader assigns zero when an agent accesses internal environment interfaces, which the authors exclude because ordinary sites are unlikely to expose them.

The rubric and the environment record do different jobs. One establishes what the agent owed the person; the other helps establish what happened. A polished final message cannot stand in for a missing file or an unperformed action under these instructions.

These are controlled environments, not a study of groceries purchased and delivered to households. Browsing uses a web cache that the authors acknowledge can change between runs. The setup makes detailed checks possible while placing a limit on what they establish about a live service.

There is also judgment inside the check: the grader comes from the same provider as the models being tested. The authors compared its grades with human trainers’ grades on 30 GPT-5.5 attempts. The average grades were close, and the grades across attempts were strongly correlated. That is meaningful support for the grading method, although a small agreement study cannot validate every consequential condition. The humans were grading model work, not performing the errands themselves.

Reading the 80.6%

OpenAI’s headline chart reports average scores rising from 49.3% for GPT-5.2 to 80.6% for GPT-6 Astra. Both scores use xhigh, the models’ extra-high reasoning setting. The chart averages attempts within each task, then gives each of the 169 tasks equal weight. The result measures credit earned under the rubrics.

It does not say that Astra finished 80.6% of the errands perfectly. More attempts might now satisfy every requirement; many attempts might instead have improved while still missing something. Both changes could raise the average. The headline does not separate them.

The meal-planning example makes the practical stakes clear. An allergy constraint and the duration covered by the grocery plan are not interchangeable concerns. The published example does not reveal their rubric weights or how any model performed on them, so it supplies no evidence of a particular failure. It does explain why a share of points cannot certify that every indispensable condition held.

For builders, partial credit remains useful. It can reveal improvements that a strict pass-or-fail result would hide. For someone deciding whether to delegate an errand, another useful measure would be how often the important conditions hold together. The two measures answer different decisions: whether a system is improving, and whether a particular kind of job is ready to rely on.

How much checking remains?

Even a perfectly graded result leaves the question of how much attention the person spends before accepting the work.

In an April account of grocery automation, Brad Carter described initially checking an earlier agent’s cart so closely that he saved no time. After a month, he reported saving roughly half an hour a week. For a later run, he supplied repeat-purchase preferences and quantity guidance, and invited educated guesses. He reported a completed cart but did not double-check it. That records a change in his willingness to rely on the work, not independently verified accuracy.

His progression is a useful counterweight to the troublesome cart. Inspection has a cost, and permanent line-by-line review could consume the convenience an assistant offers. At the same time, choosing to stop checking cannot establish that the result is correct. Neither of these older, individual accounts tests LifeVal’s models. They show practical decisions that a controlled score leaves to the person.

LifeVal gives those decisions a firmer starting point by tying personal requirements to inspectable work. For a grocery assistant, the next useful assurance is that the requirements which matter most are met together in the finished order. Canaves removed items before approving hers; Carter described reaching a point where he stopped checking each one. A better score can inform that decision. It cannot make it for the household.

Sources

Understand How People and Agents Work Together.

Practical essays, experiments, and lessons from building software for people and agents.

Occasional emails, sent when there is something worth reading. Free to subscribe. Unsubscribe at any time.