Skip to content

News Before the Next Click

Washington Post readers who received AI search answers were supplied a wider mix of news topics, but the study did not establish whether they found the information they wanted.

During a 2024 Washington Post field test, some readers received an answer before choosing a news article. Above the usual ranked search results, the site automatically displayed AI-written text with citations to up to 5 source articles. The conventional results were still below it.

A search could now supply material from several reports before the reader opened any of them. The list offered places to read; the answer supplied something to read on the search page itself.

A working paper posted September 30 reports what changed along that path. In records for 37,561 anonymous browser-cookie IDs, the combined material in displayed answers and opened articles covered a less concentrated mix of topics. More material concerned subjects widely encountered across the audience, while the mix also shifted toward less-popular topics.

Those findings describe what the site supplied, not what readers understood. But they identify a change that article visits alone would miss: some of the news material now arrived before the next click.

What the answer adds

The researchers divided the archive’s content among 58 modeled topics. They identified a core of 14 topics that reached the most distinct article readers in the conventional-results group. Their measure of material supplied on those subjects rose 69% per reader in the answer group.

An article open contributed one unit, divided among its estimated topics. A substantive answer display contributed another. In the main calculation, the researchers mixed together the estimated topics of the articles an answer cited, giving each source equal weight. A short answer and a long report therefore initially counted as one unit each. Events without topic estimates were left out of these measures.

The researchers checked different answer weights and analyzed answer text, too. The important directions persisted. The findings were not simply dependent on giving every answer the weight of one article, though none of these checks could show which sentences someone attended to.

The difference becomes visible when answers are removed from the count. Considering only opened articles, the authors found no statistically detectable increase in individual topic breadth. Article selection did shift toward less-popular topics, and the audience’s article mix became less concentrated. The broader individual mix reported in the main analysis depended on including answers.

A reader could thus be supplied an overview spanning more subjects without opening more varied articles. That makes the answer page part of the information path. It does not make it the end of the reader’s task.

The Post test ran from October 1 through November 7, 2024. The publisher reports random assignment, but the researchers could not inspect the allocation code or logs. Adjusting for measured differences left the estimates for shared-topic material and search frequency similar. The paper’s causal interpretation assumes otherwise comparable readers entering on the same day. An earlier conference description describes a larger visitor population and different search invitations; its relationship to the new paper’s design remains unresolved. This comparison cannot isolate answer text from everything else that changed on the screen.

Finding the particular report

A search sequence in a 2013 paper by Microsoft researchers shows why the task matters. Someone entered “greenfield, mn accident,” opened a report and spent 36 seconds there. Then the searcher tried again, specifying a fatal accident involving a woman, and opened another report.

The first report dated from 2010; the second was from July 2012 and described a different incident. The searches happened a day after the newer accident. The researchers inferred that this was probably the incident the person wanted. They could not verify that intention from the log, but the next query made the first click look less successful.

Both reports concerned accidents in the same town. For someone seeking a particular incident, that overlap was insufficient. A subject could be present while the information sought was absent.

This historical case did not involve an AI answer, and it cannot tell us what Post readers wanted. It clarifies what the new paper’s topic measures leave open. Two people supplied material on the same subject need not encounter the same facts or viewpoints. Nor does a broader mix within one newspaper establish diversity across publishers.

Asking again

After a Post search, the next recorded action was another search in 21.6% of cases for the answer group, compared with 10.4% for the conventional-results group. About 21% of searches ended the session in either group.

Another query could pursue something interesting, clarify an incomplete answer or repair a failed search. These logs do not establish which. The Greenfield sequence is one documented reason to keep that question open, rather than treating every extra search as curiosity.

The answer group recorded 18.9% fewer article visits per search, yet 4.9% more per reader. Searches per reader were 29.4% higher. More searching and fewer visits after each search can coexist with slightly more visits overall. These figures count all recorded article-page visits, including repeated opens; “reader” still means a cookie ID. The raw visit count rose from about 1.75 to 1.84 per ID, with a 95% relative confidence interval of 1.3% to 8.7%.

Some opened articles had been cited by an answer earlier in the session. That classification did not identify the link clicked: a reader could have reached the article another way. It also could not show whether the reader was checking the answer. The source article’s presence in a log and the reason for opening it remain separate questions.

The destination of the next query matters, too. On the Post’s site, it stays inside the Post’s archive. On a broad search service, it need not bring a reader to a publisher.

A separate Google field experiment posted in August analyzed 1,100 active participants. After 3 baseline days, one group had its searches redirected to AI Mode for 7 days. Compared with ordinary Google search, assignment to that experience reduced external click-through by 18.8 percentage points and daily search sessions by 0.92.

This was forced adoption in a different interface, population and task mix. It shows why the Post’s extra searching cannot be assumed to replace lost visits elsewhere. Participants’ accounts also varied: some valued saved time and useful summaries; another complained that, when trying to reach a webpage, the system would “tell me about the website instead.” An explanation of the destination had arrived where a route to it was wanted.

The Post later changed how it handled a similar distinction. In February 2025 reporting by Technical.ly, product and design head Gitesh Gohel said user feedback had prompted the removal of AI summaries for simple-term searches. Queries framed as questions or prompts could still receive an answer. A simple term returned the article results without one.

That was a later product change, not the experimental treatment or a verified description of today’s interface. It left readers two ways into the reporting: an answer for a question, or a list of articles for a search term.

Sources

Understand How People and Agents Work Together.

Practical essays, experiments, and lessons from building software for people and agents.

Occasional emails, sent when there is something worth reading. Free to subscribe. Unsubscribe at any time.