Skip to content

What Did the Page Say When the AI Read It?

Opening a cited webpage today can show what it says now without settling whether a saved AI answer accurately reported what the page said earlier.

TechRadar writer Eric Hal Schwartz asked Google’s Gemini to track his packages. The shipping confirmations were in his email, but the status of each package was on a carrier’s website. Gemini checked both sites, Schwartz reported. One package showed as out for delivery. The other had been marked “Shipping Label Created” for several days.

Schwartz didn’t report going back later. But suppose the second page now says the package was delivered. That would tell him what the carrier reports now. It wouldn’t by itself tell him whether Gemini got the earlier status right. The page can still be useful for tracking the package while no longer showing what the assistant saw.

That matters when an answer gets saved or forwarded. The next reader might open a link to learn what’s true now, or to find out whether the answer accurately described what was posted at the time. A live link can serve the first purpose without serving the second. A November study put the difference to work by grading the same AI answers weeks apart.

Same answer, another grade

The researchers behind EverydayBench, a test of how assistants handle everyday online tasks, checked factual claims against the exact webpages the assistants cited. One task in the paper asks for studio rentals. The grader can recalculate the monthly cost from the amounts an assistant reports. Checking whether a landlord advertised the stated rent and availability requires opening the listing. A change to the listing won’t affect the arithmetic.

In a separate scoring exercise, the researchers kept a set of assistants’ outputs fixed and checked the live versions of their cited pages. The average score fell from about 69% when the work was first graded to 63% after 20 days and 61% after 30. Those scores combine several checks. An 8-point drop doesn’t mean that 8% of the claims became false. The paper does not identify the individual claims or pages that accounted for the change, or say whether the rental task was among them.

Imagine a saved rental recommendation lists a studio at $1,600 a month. The listing now shows $1,750. The assistant might have copied the price accurately before it changed, or it might have gotten the price wrong in the first place. If the listing won’t open at all, there’s even less to go on. The researchers’ rules count an inaccessible page as unverified, not as proof an answer is wrong. The scores don’t tell us how much came from edits to the pages, difficulty reading them or differences in the grader’s judgments.

The paper also tested 3 assistants on fresh runs 30 days apart and graded the work soon after each run. Those scores showed no systematic decline. A new search can find a different, currently available studio. It doesn’t tell you where the $1,600 in the saved answer came from.

When the page keeps its history

A tracking page might show “Delivered” at the top and still list the earlier events underneath. If it records when the label was created and when the carrier first scanned the package, a later reader can check parts of the old report without finding an old copy of the page. What matters is the dated entry behind the claim, not whether the page as a whole has changed. The entry still records what the carrier logged, not what someone found at the door.

Some sites make the earlier document itself available. This essay links to the study’s first version. If the authors revise the paper, arXiv still lets readers request version 1. On a listing that keeps no public history, retaining a dated copy of the relevant passage could help. That costs more effort than keeping a link, and a copy that omits the unit or the lease terms might not explain the price. The study didn’t test keeping records as a remedy.

If you’re deciding whether to tour the studio, today’s listing is worth opening. If someone asks why the studio made the shortlist, the useful thing to have is the earlier listing with the unit, price and terms still together.

Sources