Judging AI After the Standard Changes

One practitioner testing EvalGen, a tool for building checks on AI-generated answers, twice gave a bad grade to stay consistent with earlier grades, rather than because they considered the output bad. The supplied job was extracting named entities, such as people or organizations, from tweets.

A November 10 working paper reports rising rejection of fixed-version model pairs on Chatbot Arena, a website comparing AI responses. For people checking AI work, that raises a consequential question: when should a changed judgment be corrected for, and when does it contain information worth keeping?

A stable grading rule helps preserve a comparison with earlier work. It can also leave a newly clarified requirement unmeasured. The challenge is not simply to make the evaluator more consistent or more demanding. It is to decide whether the evaluation is meant to diagnose a change in software or judge whether an answer meets the requirements now.

Better than its rival, useful for the job

The historical Arena interface let visitors choose either answer, a tie or “both are bad.” Selecting a winner does not separately record that the answer is acceptable. Rejecting both does not reveal which would have won a forced choice. The interface records different kinds of verdict, without independently measuring both preference and acceptance for every comparison.

The new paper analyzes 1,260,967 votes from May 2023 through August 2024. It estimates a 1.71-percentage-point monthly rise in rejection within model pairs. In 2024, among votes choosing a winner between models at least 20 rating points apart, the higher-rated model’s share showed no detected trend.

That comparison is conditional on someone choosing a winner. It cannot reveal preferences in the rejected comparisons. Nor would a stable order, by itself, establish stable quality: both responses could become worse together while one remained preferable. A changed requirement could likewise make both fail without reversing their order. A comparative result leaves the question of adequacy open.

Several checks on requests, how models were run and interface changes preserved the rejection trend. But the log lacked request and answer text. Fixed versions were not fixed answers or settings. Network-based identifiers did not verify people, and exposure to better AI was unmeasured. Changed tasks, configurations or voting populations remain possible contributors.

The result makes changing judgment worth investigating. It does not establish that complaints about worsening AI are mistakes, or that an adjustment should make them disappear.

A requirement can emerge during inspection

The EvalGen study followed 9 industry practitioners on the supplied extraction task, not everyday deployment. Two wanted to relax a rule after seeing outputs: most extracted entities could be proper nouns, rather than all. Experience did not simply make their evaluations stricter.

There are at least 2 ways to understand that revision. The original wording might have excluded information the application could use. Or the evaluators might have accommodated a model’s weakness. Seeing which outputs pass cannot settle which explanation justifies the new rule. That depends on what the extracted list is for.

This is where a changing evaluation becomes more than a measurement nuisance. Before inspection, a requirement can appear settled because no one has confronted an awkward case. Once the case arrives, consistency with earlier grades and usefulness for the task can pull in different directions. Freezing the rule preserves the old decision; it does not prove that decision was well chosen.

A separate EvalAssist usability study makes the intended job concrete. Its 15 IBM practitioners used AI to assess generated responses. One supplied exercise asked them to define how to judge an HR answer’s faithfulness to a policy document, using an inquiry about a business-class upgrade with a corporate card. This was evaluation work, not an observed travel purchase.

For that exercise, a response could be more polished than its rival while still adding advice the document did not support. Choosing the smoother answer would answer a different question from checking whether the policy warranted its claims. If inspection exposed an unsupported statement, preserving earlier grades would not make that statement suitable for use.

One software developer in the study explained that even bad outputs could be ranked: “Rankings are just relative preferences.” Other participants preferred pairwise comparison for flexibility on subjective tasks or easier criteria-writing. This small, single-employer study supplies reasons to choose an assessment for its purpose, not grounds for making individual grading the universal winner.

Keep the comparison, reconsider the permission

A revised requirement changes what a score means. If earlier answers were graded under one rule and later answers under another, the difference can reflect the evaluator as well as the software. Comparing the scores alone cannot allocate responsibility between them.

Applying the new rule to earlier answers could instead ask how the earlier system performed on the task as it is now defined. Preserving the original rule could help investigate software change under the old definition. The choice depends on the question. Neither comparison decides whether the new requirement is worth adopting.

There must also remain room for a real product regression. In its April 23 account of Claude Code quality problems, Anthropic reported a bug that repeatedly discarded earlier reasoning after an idle session resumed, making the assistant forgetful and repetitive. It said its API was unaffected. This provider explanation is not an independent audit or an explanation of the earlier Arena data. It illustrates why the surrounding product cannot be assumed stable from a model label alone.

The Arena authors distinguish fixed-standard evaluation from current satisfaction and note that changed needs can warrant retaining changed judgments. For a builder, the useful consequence is to keep the reason for a revision attached to the decision it changes. A score adjustment can help preserve a comparison; it cannot supply that reason.

The practitioner who gave a bad grade for consistency was trying to keep later judgments in line with earlier ones. A builder approving an extraction list has another responsibility: explain why the kinds of entities it contains are suitable for the intended use. The earlier grades can remain in the record without becoming the answer to that decision.

Sources

• Zihan Chen, Lei Nico Zheng and Di Zhu, AI Isn’t Getting Dumber, We’re Getting Pickier: Rising Standards in Human Evaluation of Large Language Models, November 10, 2026, v1 working paper. CC BY 4.0; condensed interpretation, no endorsement. Estimates not independently reproduced.
• Wei-Lin Chiang and colleagues, Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, March 7, 2024, version 1. Original interface methods.
• Shreya Shankar and colleagues, Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences, April 18, 2024, version 1. EvalGen usability study. Licensed CC BY 4.0; accounts condensed and interpreted.
• Zahra Ashktorab and colleagues, EvalAssist: Insights on Task-Specific Evaluations and AI-Assisted Judgment Strategy Preferences, August 6, 2025, version 2. Supplied tasks and practitioner preferences. Licensed CC BY 4.0; accounts condensed and interpreted.
• Anthropic, An update on recent Claude Code quality reports, April 23, 2026. Provider incident account.