A coding model that scores 11 points higher on a hard benchmark should, you would think, get noticeably more of your work finished. A new study that caught a rare accident in the wild says it did not, at least not in a way anyone could measure. When a US directive took developers' best model away for 19 days and then gave it back, the share of their code that made it into their projects barely moved.

My take is simple. A higher benchmark score is not the same as more shipped work, so measure your own merged output before you pay for the upgrade.

The paper

Higher-Scoring Coding Models Did Not Get More Work Done is a preprint by Yining Hua of Harvard University. It is a single-author paper and has not been through peer review. The full text runs to 20 pages with long appendices, and it is unusually open about its own weak spots.

What it found, in plain English

On 12 June 2026, Anthropic received a US export control directive that suspended access to Claude Fable 5, three days after the model launched. Anthropic published a statement on the directive the same day. Per the paper, the suspension ran from 12 June to 1 July, 19 days in all, and Fable 5 users had to fall back on other models. Most of them went to Opus 4.8: during the suspension, 74.6% of their pull requests named it.

Opus 4.8 scored 69.2% on SWE-Bench Pro, a widely quoted coding benchmark, against 80.0% for Fable 5, the figures the paper takes from Anthropic's system cards. Model launches lead with gaps like that one, 10.8 points or about 11.

Then the paper looked at what actually happened to work. With Fable 5 available, the share of developers' pull requests merged within seven days was only 1.4 percentage points higher than during the suspension, with a 95% confidence interval from −1.4 to +4.3. That interval includes zero, so the result is not statistically significant. In plain terms, the data cannot tell this apart from no effect at all, and it rules out a gain much bigger than about 4 points.

The author then checked five other model upgrades from 2026, covering 1,208 developers and 15,442 pull requests. Pooled, the merge rate of developers who switched to the newer model changed by −0.9 points (−1.9 to +0.2), and all five individual estimates were negative. Those upgrades came with SWE-Bench Pro gains of 5 to 18 points across the six where Anthropic reported both scores. On output, developers who switched got 2.4% more pull requests merged per week than those who did not switch (−4.9% to +10.2%), again not significant. The share of pull requests failing automated checks such as tests and builds moved by +0.3 points, which is also noise.

So across a forced downgrade and five voluntary upgrades, more benchmark points did not turn into measurably more accepted code.

How the study worked

The hard part of measuring any model's effect on productivity is that everything else changes too. The people who rush to a new model may be more skilled, and they may hand it harder tasks. Compare early adopters with everyone else and you mostly measure the people.

The suspension removed that problem for a while. Nobody chose to lose Fable 5; according to the paper, the directive took it away from every customer at once, and then it came back. That makes it a natural experiment, which simply means the world ran the trial for you: the same people, on the same projects, had the better model, lost it, then regained it.

To spot which model wrote what, the paper used a habit of Claude Code. By default, the tool adds a line to commits naming the model, such as "Co-Authored-By: Claude Opus 4.6". When that line also appears in a pull request description on public GitHub, you can tell which model was in use. The treated group was 297 developers who opened at least one pull request naming Fable 5 during its three available days. The control group was 8,908 developers who used Opus 4.8 throughout and never touched Fable 5.

The statistical method is called difference-in-differences. In everyday terms, you watch how the treated group's merge rate moves when the model is taken away and returned, and you subtract however much the control group's rate moved over the same days. Whatever change remains can be pinned on the model. The paper also adds "fixed effects" for each developer and each calendar day, which strips out the fact that some people always merge more and that some days are busier for everyone.

Before Fable 5 arrived, the two groups looked alike: both merged about 92% of pull requests within a week. That similarity matters, because the whole design rests on the assumption that both groups would have moved together if nothing had happened.

What this means when you pay for upgrades

Every few months a new model arrives with a higher score and, often, a higher price or a tighter usage limit. The score does real work in that decision. People quote it, buyers compare it, and it feels like an objective reason to switch.

This paper suggests the score was answering a different question from the one you care about. The author's leading explanation is that the tasks developers hand to agents already sit within what the older model can do. If most of your requests are routine fixes and small features, an extra 11 points on hard benchmark puzzles has little room to show up. The numbers back this up. Before the upgrades, developers who switched were already getting about 95% of their pull requests merged within a week, which left almost nothing to improve.

It also fits what earlier work found about benchmarks themselves. A March 2026 note from METR found that roughly half of test-passing SWE-bench Verified PRs would not be merged by the projects' maintainers. A benchmark counts a task as solved when tests pass; a real project asks whether the change fits, reads well and does what was meant. We covered the same gap from another angle in Co-Design Bench, where the best agent met 69.9% of requirements per task but fully finished only 26.4% of edits, and in BOBA-Bench, where agents finished 95% of single requests but only 51% of 20-request chains.

What most people miss is that a benchmark measures a model, while your bill buys a result. The two overlap less than the launch charts suggest, and the only place to see the overlap is your own work.

The caveats, and they are real

The paper is honest about its limits, and you should weigh them before you quote it.

  • One author, no peer review yet. Nobody outside has checked the analysis.

  • A narrow slice. Only about 1% of merged Claude Code pull requests name a model in their description, so the study sees a small part of all work. The author shows these pull requests merge about as often as other Claude Code pull requests, but it is still a sliver.

  • Merging is a weak signal of quality. In a sample of 2,019 pull requests, 95% were merged by their own authors and 90% had no approval from anyone else. A merge here means the developer decided the work was done, which is acceptance, not a code review.

  • A short window. Fable 5 was available for only three days before the suspension, and the suspension lasted 19 days, so the main estimate has wide error bars. The return period also overlapped another model release, a US holiday and a temporary usage limit on Fable 5.

  • Upgrades were chosen, not imposed. In the five-upgrade analysis, developers picked when to switch, so that part shows association rather than cause.

  • No significant effect does not mean zero effect. The data allow for a gain of up to about 4.3 points from Fable 5 access. Small gains could be hiding inside that range, and gains that only show up under outside review would be missed entirely.

None of this rescues the benchmark as a buying signal, though. Even under the most generous reading, an 11-point score lead produced at most a few points of extra merges, and probably less.

How to use this today

You do not need a natural experiment to test a model on your own work. You need two weeks and a spreadsheet. This works for a solo developer, a small team or anyone who uses AI to get tasks done, from code to copy to client reports.

Imagine you run a small agency and a new model launches at a higher price. Before you switch everyone, run a personal A/B test:

  1. Set up one sheet with these columns: date, model, task (one line), task type (fix, feature, content, other), shipped (yes or no), minutes until done, rework (did you or anyone have to fix it within seven days, yes or no), and cost for the task if you can see it.

  2. Week one, use your current model for everything. Log every task you hand it, including the ones you abandon. Abandoned tasks matter as much as shipped ones.

  3. Week two, switch to the new model and keep the same habits. Try not to save your hardest work for the new model, because that skews the result (the paper flags exactly this risk).

  4. At the end, work out three numbers for each week: tasks shipped, share of tasks shipped, and share that needed rework. For code, "shipped" means merged; for anything else, it means you used the output without redoing it.

  5. Look at minutes per shipped task too. A model that ships the same number of tasks in less of your time is still worth something.

Then apply a decision rule. Ours (a rule of thumb, not something the paper tested) goes like this. Upgrade if the new model shipped clearly more tasks, or needed clearly less rework, and the extra monthly cost is smaller than the hours it saved multiplied by what an hour of your time is worth. If the difference is a task or two either way, stay where you are and test again at the next release. Two weeks of your own work is a small sample, so you are looking for a gap you can see without squinting, not a decimal point.

A few extra habits help. Keep the sheet running after the test, because a monthly look at shipped tasks per model will tell you more than any launch post. Write down the task types where the older model struggles; those are the only places a stronger model has room to help. And if you work in a team, ask who actually reviews AI-written work, because when people merge their own pull requests, as 95% did in the paper's sample, a merge says little about quality.

For people who do not write code, the lesson carries over. If you use a chatbot for emails, summaries or research, count how often you send its output with light edits versus rewrite it. That ratio, tracked across a model change, tells you more than any leaderboard.

The benchmark tells you what a model can do on someone else's tests, and your spreadsheet tells you what it does for you, so let your money follow the spreadsheet.

Sources