Which Icons Become the Agent's Test?
In sampled Android task demonstrations, the textless action icons people touched were more likely to carry descriptions than untouched icons on those screens, so an agent's test needs to be understood through its targets and available clues, not just its list of apps.
Before a phone agent taps an unfamiliar icon, it may have more to go on than the picture. AndroidControl, a dataset for training and evaluating AI that operates apps, pairs human demonstrations with screenshots and machine-readable interface records. Those records can supply a control’s name as text. An agent receiving that name faces a different identification problem from one interpreting only the image.
A November 11 audit found that the demonstrated targets were not equally distributed across those conditions. Among tappable graphics without visible text, about 70% of targets people touched carried content descriptions, compared with 61% of other eligible icons on the visited screens. The researchers examined 10 of AndroidControl’s 20 data files. They measured description presence and human target selection, not AI success.
For someone deciding whether an assistant is ready to handle app work, the practical question is which uncertainties its test has exercised. A successful test on better-described targets does not establish equal ability on undescribed ones. How much that matters depends on the operations the assistant will need and the information it receives. The audit does not show that any particular agent’s score is inflated.
What the interface tells the agent
A content description is a text field that can convey what a graphic means or does. Android’s labeling guidance says that an action icon’s description should identify the action, rather than merely the shape. A screen reader such as TalkBack can announce this information to someone navigating by sound.
An arrow illustrates the distinction. It might lead to another page, reveal more choices or close part of the current view. These are hypothetical uses of a familiar shape, not interchangeable operations. An accurate action name resolves some of that ambiguity before a person or agent presses anything. Without it, surrounding text, earlier actions or the picture itself may still provide useful evidence.
Displayed text can already be available to accessibility services. Editable fields can use hints or associated labels; decorative images need no name. A useful name does not always require filling an empty description field.
The audit’s broader count of elements judged to need descriptions had 57% coverage in AndroidControl. That population includes potentially decorative images. Its narrower action-icon comparison avoids that ambiguity, but description presence still does not establish correct wording.
Which part of the screen becomes the task
The difference between touched and untouched icons was not solely a matter of visiting better-described screens. About 3.5 percentage points of the 9-point gap came from selection within screens. Yet the direction reversed when the researchers compared all elements judged to need descriptions, rather than action icons. The result belongs to its defined population, not every object on a phone screen.
A demonstration selects operations as well as apps. It follows particular controls through particular screens, leaving others unused. A benchmark can include an app while never requiring some of its unnamed controls. Adding another app and adding another kind of target are different expansions of a test.
This selection would be consequential if proposed everyday tasks require more undescribed targets and the agent does worse without their descriptions. Neither condition was tested here. The untouched icons are a comparison population, not a forecast of what users will ask an assistant to press. If deployment uses much the same targets, the measured difference could have little practical consequence.
Nor does a missing description remove the picture. AndroidControl records screenshots as well as accessibility trees, the structured interface records. Different agents can use different combinations of these channels. An image-based agent may recover information absent from a text field. This audit does not measure how much it recovers, and no experiment added or removed descriptions to isolate their effect.
When recognizing the shape is enough
A separate study of blind TalkBack users shows why the available clues deserve attention. Researchers tested 4 selected Android apps with 5 participants per app, deliberately guiding tasks through known accessibility issues. These were human usability sessions, not tests of AI agents.
In the tested AnkiDroid flashcard app, all 5 participants inferred the purpose of an unnamed Add button. Automatic visual recognition identified a plus, and preceding text directed them to that symbol. In the card editor, an unnamed collapse button was harder: recognition supplied “up arrow,” but some participants could not infer that it would close the associated editing field. These observations concern the tested builds and tasks, not today’s releases.
The contrast does not make vision a solution or missing names an automatic failure. It shows what the extra evidence must accomplish: connect a recognizable shape to the operation needed here. An assistant with vision still needs to make that connection; an assistant with an accurate action description receives some of it explicitly. Human outcomes cannot tell us how well either model will do.
A benchmark report could separate results for described and undescribed targets and state whether the agent received images, structured records or both. The tasks and controls might still differ in other ways, so that breakdown would not isolate a label effect. It would help a reader judge which conditions a reported success actually covers.
For a proposed assistant handling the card editor, the next useful test might stay inside the same app. Ask it to collapse the editing field, using the information it will actually receive, and check the resulting view. A successful Add demonstration leaves that operation unresolved.
Sources
- Obada Kraishan and Kulsawasd Jitkajornwanich, Trained on What Screen Readers Cannot Read, November 11, 2026, version 1. Methods and Tables I and III; results were not independently reproduced here.
- Google Research, AndroidControl dataset documentation, and Wei Li and colleagues, On the Effects of Data Scale on UI Control Agents, NeurIPS 2024. Dataset format and observation channels.
- Google, Content labels, Android Accessibility Help, inspected November 16, 2026. Implementation guidance, not performance evidence.
- Syed Fatiul Huq, Ziyao He, Yirui He and Sam Malek, Bridging the Gap between Automated Intervention and Actual User Experience: A Mixed-Methods Study on Mobile Accessibility Issues for Screen Reader Users, CHI 2026, sections 4.1 and 4.2.1. Participant accounts are condensed and interpreted from this CC BY 4.0 work; no endorsement is implied.