Skip to content

Giving an Agent’s Choices to a Faster Model

In tests of five software agents, handing action choices to a specialized decision model cut selection time and estimated cost, with high success on simulated household tasks but substantially fewer completions in two phone agents.

During a test of software agents operating Android apps, an agent tried several recipe-form edits at once in its first run. The form shifted, and the agent reported putting a value into the wrong field. In the researchers’ account of that run, a version using Jev, a model that selects among supplied options, instead performed one operation, read the changed interface and chose the next operation. It completed the recipe task in all 3 runs. The change involved how actions were executed as well as which model chose them.

Jev’s choice interface returns one of those options, rather than a free-form answer. An October 9 preprint compares this way of choosing actions across 5 agent designs using text descriptions of their surroundings. Across those designs, estimated cost per action-selection call fell 66–78%, and time per call fell 74–82%. Two agents retained high success on simulated household tasks, with lower overall resource use. Two phone-agent integrations completed substantially fewer tasks. The recipe case was a useful exception within that less favorable phone result.

For anyone building an agent to operate apps, the handoff is more specific than replacing one model with another. You could give repeated choices to a cheaper model while retaining generated text, new screen readings and other work around them. The integration can also change the sequence of actions, as the recipe form illustrates. The tests measure machine resources and task completion, leaving the time people spend checking or correcting an assistant unmeasured.

Choosing from what is available

Selecting an operation and generating what it needs are different jobs. A model might choose to enter text, while another model supplies the words. It might choose an available command, or choose to request further reasoning before acting. The October study retained language-model calls for open-ended text, summaries and reflections around Jev’s choices.

In the household simulation, agents received textual descriptions and could work toward goals such as cleaning or placing objects. The Jev versions selected from commands the environment made available. They could also choose a THINK option that invoked language-model reasoning. Faster selection did not mean eliminating that reasoning.

One design, ReAct, shows the promise clearly. Across 134 tasks repeated 3 times, its original version completed 392 of 402 attempts; the Jev version completed 386. Average time per attempt fell from about 71 seconds to 37, and estimated model-call cost roughly halved. The second household design, which could reflect and try again, retained 98% success with Jev. The observed success declines were small, although the study does not establish equal reliability. In this setting, much of the resource saving survived the extra reasoning work.

The command menu changed too. Original household agents generated commands without that menu, so the researchers tested language models given the same candidates and selection instructions as Jev. Success remained close in those comparisons. The phone experiments lacked that same-interface control, and the recipe agent also changed from potentially batched calls to one tool operation per step. The study compares integrations; it does not isolate an intrinsic Jev advantage or defect.

Another look changes the next move

Return to the recipe form. Several operations referred to a layout that changed as they were carried out. The authors describe the Jev version receiving updated observations between its single-tool operations. The next choice could therefore use the new layout. This documents a changed execution sequence, without isolating single-step operation as the cause of the performance difference.

More frequent opportunities to look did not guarantee that the requested state would be reached. In the same phone-agent family, FastAgent, the Jev version opened quick settings and clicked a switch labeled “Off”. The label did not identify Wi-Fi. It declared success after 3 decisions while Wi-Fi remained off, in all 3 runs. The original version succeeded in all 3.

A short task can therefore contain a consequential judgment about an ambiguous target, followed by another about whether to stop. The switch was available to click. Its label did not establish what it controlled, and clicking it did not reach the requested Wi-Fi state.

Across the phone tasks, FastAgent’s success fell from 50% to 31.9%. Another design lost about 13 percentage points; the third had a smaller, uncertain decline and low success in both versions. These results concern fixed task instances and text-only observations, with 3 repetitions. They are not a ranking of every phone agent or a forecast for a different app.

When evaluating your own integration, consider following one screen-changing action through to the next decision: who receives the updated screen before choosing what to do? This is a suggested check, not a remedy tested by the study. The recipe and Wi-Fi accounts make both parts worth inspecting: what information arrived, and what the agent decided from it.

Keeping some work outside the handoff

A faster choice also leaves time-consuming work elsewhere. In the successful household example, ReAct’s action-selection time fell sharply, even after adding separate reasoning calls. In FastAgent, cheaper selection came with more decisions, more time in supporting language-model calls and more time outside model calls. Average runtime per attempt stayed near 120 seconds.

With fewer completions, total runtime divided by successful outcomes rose from about 239 to 378 seconds. That ratio charges failed attempts to the outputs the system delivered. It is not a measured wait until a particular user’s request succeeds: someone might repair the result, change the request or stop trying. Nor do the paper’s estimated model-call costs include all deployment expenses.

Replacing every choice is also only one design option. An older, independent study of a system called REFLEX let Jev handle selected decisions and called a stronger language model when its reported confidence did not clear a threshold. On 100 held-out tasks in a constructed tool environment, it completed 95%, versus 88% for the strong-only agent, while making 72.7% fewer strong-model calls. That is evidence for a selective design in that environment, not a repair demonstrated on these phone tasks.

The alternative need not use a specialized model at all. On 60 paired external tasks, the same REFLEX researchers found that a cheap generative model allowed to escalate to a stronger one cost less than REFLEX and was numerically more successful. The success difference was uncertain. The comparison broadens the practical choice: a specialized decision layer needs to earn its place alongside economical generative alternatives.

Who checks the result?

The time-and-cost totals stop short of the person receiving the work. In a separate 8-person study using existing agents, participants tried travel, shopping, document, image and game tasks. Researchers assigned step-by-step checking for the first task and end checking for the remaining tasks. The study observed how people checked under those arrangements, rather than how freely they chose a checking schedule.

One travel-task activity description said July 4 while the screenshot showed July 9. Participants inspected results, searched execution histories, supplied corrections and waited for replanning. In 2 document-processing trials, however, people downloaded the result and fixed minor errors themselves instead of asking the agent to repeat the work. This small study did not test Jev or measure its effect on human effort.

Those document repairs mark a further decision that an inexpensive action selector does not settle: whether another automated attempt is worth requesting. A builder can make choosing the next operation cheaper while someone receiving the file still decides to finish it by hand. Software can also retain some verification outside the action selector, giving a changed state its own check before accepting completion.

Droidrun’s Mobile Jev demo documents that separation for Android’s dark-theme setting. Its documentation separates setup, execution and verification timing. This is a developer-authored demo, not a demonstrated repair for the study’s Wi-Fi failure or evidence of a cost-free check. In this handoff, the model can report completion. The runner still reads the screen again and checks that the Dark theme switch is on.

Sources

Understand How People and Agents Work Together.

Practical essays, experiments, and lessons from building software for people and agents.

Occasional emails, sent when there is something worth reading. Free to subscribe. Unsubscribe at any time.