When people say an AI assistant "got worse", we usually assume someone changed the model. A new working paper tests a second explanation that is much harder to see: the model stayed the same and the people judging it raised their standards. Using more than a million blind votes on AI models that never changed, the authors find that verdicts grew harsher every month while the ranking of which model was better barely moved.
That changes how you should read complaints, ratings and your own sense that a tool has slipped.
The paper
"AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models", by Zihan Chen and Di Zhu of Stevens Institute of Technology and Lei Nico Zheng of the University of Massachusetts Boston. It was posted to arXiv on 10 November 2026 (version 1) and is labelled a working paper, which means it has not yet been through peer review.
The finding in plain English
The researchers used Chatbot Arena, a public website where anyone types a question, gets answers from two anonymous AI models side by side, and votes for the better one, a tie, or "both are bad". Names appear only after the vote. The Arena team describes it as an open platform for ranking models by crowdsourced human preference.
The authors looked only at pairs of models whose versions were fixed: dated snapshots that vendors publish as unchanging, such as gpt-4-0613, and open models whose weights never change. For the same pair of unchanged models, the share of "both are bad" votes rose by 1.71 percentage points a month, starting from an average of 17.1%. Roughly one vote in six rejected both answers, and that share kept climbing for the same two models.
The clearest evidence needs no statistics. The authors found fourteen model pairs that were voted on at least fifty times in both 2023 and 2024. In every one of the fourteen, "both are bad" went up, from 15.5% to 23.3%.
Meanwhile, the ranking held. When voters did pick a winner, they picked the stronger model about as often at the end as at the start; in 2024 the trend in its share of decisive votes was zero. People still knew which answer was better. What changed was how often they felt neither was good enough.
How they did it
The data is Arena's public log of 1,260,967 blind votes cast between May 2023 and August 2024 on pairs of fixed models. The core method compares each pair of models only with itself over time, so any difference in model quality is held constant.
They then tried to explain the rise away, with controls for prompt language and difficulty, easy English prompts on their own, dated OpenAI and Anthropic snapshots, models whose answer length stayed steady (a sign the provider had not quietly changed how they were served), and the periods between changes to Arena's interface. Across 28 versions of the analysis, the monthly rise ranged from about 1 to 2.6 points and was positive in every one.
Finally they followed people. The same voter, judging the same pair more than a month apart, became harsher. Newcomers who arrived in 2024 were harsher on their very first vote than newcomers from 2023, which suggests people bring a higher bar with them from their wider use of AI. One encounter with a strong model did not make the next vote harsher, so this looks like a slow drift in expectations.
Why it matters
The treadmill picture is striking. Over the period, the models on Arena improved so much that a typical model from August 2024 would be expected to score about 78% against one from May 2023, meaning it would win most head-to-head comparisons, yet the overall share of "both are bad" votes was 19.4% at the start and 20.6% at the end.
For anyone running a product, this means a flat or falling satisfaction score is not proof that quality dropped. A 2023 study found the March and June 2023 versions of GPT-4 behaved very differently on the same tasks. But if your users compare you with the best AI they used last week, a product that stands still will be judged as getting worse.
On Arena the drift ran at about 21 points a year.
The paper also points to a fix. Because the drift was shared across models, it could be measured on some fixed pairs and subtracted from others. Seventeen of eighteen heavily judged models looked as if they had got worse on their own; after that adjustment, the average monthly rise fell from 1.40 points to 0.19. A fixed reference, judged at the same time as the thing you care about, separates "the product changed" from "the bar moved".
This also bears on the human feedback used to train models, a topic we covered in 7 ways RLHF goes wrong: an "acceptable" label from 2023 does not mean what the same label means in 2024.
Caveats
Unusual voters. Arena users are a self-selected group keen on comparing AI. The authors say the size of the drift in the general population is still an open question.
Old data. The log ends in August 2024. Whether standards keep rising at this pace, or level off as progress slows, is unknown.
Thin records. The log holds no prompt or answer text, and a "voter" is a hashed network address, which follows a device rather than a person.
Mechanism unclear. The data cannot tell a higher threshold apart from closer scrutiny of the same answer.
How builders and ordinary people can use it today
The principle: compare against something fixed, judged at the same time, before concluding on what has changed.
If you use AI tools day to day
When a tool feels worse, test it before you switch. Keep ten to twenty real prompts from your work, with the answers you saved when you were happy. When doubt creeps in, rerun them and compare new and saved answers without labels, shuffled by a colleague or a simple script, picking the better one each time. If new answers win or tie about half the time, your bar has probably moved. If the old ones win clearly, you have evidence of a real regression.
If you build an AI product
Freeze a benchmark set. Save 30 to 50 representative prompts with today's outputs
Re-score blind on a schedule. Monthly, have reviewers compare current and saved outputs side by side without knowing which is which. If you use a model as the grader, first check it against human labels, as in our guide to the calibrated judge.
Ask two questions, not one. The paper suggests recording both a forced choice ("which is better?") and a separate yes or no on "is this acceptable?". The first tells you about quality; the second tells you about your users' current bar.
Report two numbers. Show satisfaction against the fixed baseline and the raw current score. A falling raw score with a flat baseline means expectations rose, which calls for an upgrade plan rather than a bug hunt.
Plan upgrades on a calendar. Since standing still reads as decline, budget for regular model swaps and judge each one on outcomes, for example with cost per solved task.
A worked example
Say your support bot's "not helpful" rate was 20% in January and 26% in July. Before rolling back or blaming the provider, you run your January set blind. If reviewers still prefer July's answers to January's, quality held and the 6-point rise reflects rising expectations.
Sources
AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models, Chen, Zheng and Zhu, arXiv, 10 November 2026
Full text (HTML, v1) of arXiv 2611.14333, arXiv, 10 November 2026
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, Chiang et al., arXiv, 7 March 2024
How is ChatGPT's behavior changing over time?, Chen, Zaharia and Zou, arXiv, 18 July 2023

