The paper
Authors: Xilin Wei, Shengyuan Ding, Xinyu Fang, Tiancheng Gu, Yuchi Wang, Zhixun Li, Ziyu Liu, Yuhang Zang, Weixiang Yu, Cheng Lu, Chonggao Sun, Tong Zhou, Dahua Lin, Hang Zhang, Shuai Bai, Dayiheng Liu (16 authors). The paper lists Fudan University, Alibaba Group, Zhejiang University, The Chinese University of Hong Kong, Shanghai Jiao Tong University and Nanyang Technological University; its Hugging Face paper page is listed under the Qwen organisation.
Date: submitted to arXiv on 26 October 2026 (v1).
Link: arxiv.org/abs/2610.35138, with code on GitHub and a 96-task public subset on Hugging Face Datasets.
The finding in plain English
If you build with Claude Code, Cursor or Codex, you know the moment: the agent says it is finished, the page looks right, and the one button you cared about does nothing. This paper puts a number on it. The researchers took 273 editing requests from real user coding sessions, ran ten model endpoints through the same agent harness, and checked every result against a hidden list of requirements written by experts. The best of the ten models met 69.9% of the requirements per task on average but fully completed only 26.4% of tasks. Put the other way round, even the strongest model left at least one requirement unmet in 73.6% of tasks.
Dev's take: when your agent says "done", treat it as about 70% done. The missing part is usually one or two specific things you asked for, sitting inside an app that otherwise works, which is why they slip past you.
There is a second finding. On the repository editing track, every one of the ten endpoints scored lower when a single request bundled three or four types of change (structure, style, interaction, content) than when it involved one or two.
How they did it
The team started with 39,325 production queries from 26,314 real coding sessions, leaving 37,463 after deduplication. They rebuilt each app as it stood before the request, checked it would build and render in a browser, and dropped anything that relied on services they couldn't recover.
They favoured hard, checkable requests where the original user had struggled (rephrasing several times, or rolling back), and experts turned them into 273 fixed tasks with 2,331 hidden acceptance criteria across two tracks:
Multimodal Editing (100 tasks): focused changes to a page; 80 came with an image, video, audio clip or document, such as a receipt whose value must flow through to the totals.
Repository Editing (173 tasks): coordinated changes across a multi-file front-end app. Almost all (171) are single-page React apps, and 166 of the 173 need more than one type of edit.
Grading is the part worth copying. A judge model marks each criterion Pass, Partial or Fail from the code diff, screenshots and recorded browser clicks. A separate model then tries to knock down every Pass and Partial, hunting for stubs, hardcoded outputs, features that exist in code but can't be reached in the interface, and quietly reinterpreted requests. Interaction claims need a recorded click and the resulting state change; a handler in the source doesn't count. The authors call this "judge-then-refute".
Every model ran inside the Claude Code harness with the same prompts and tools, capped at three hours and 120 turns per task.
Why it matters
Most benchmark headlines report the average score, and averages flatter agents. A 70% requirement score sounds respectable. A 26.4% full-completion rate means roughly three in four edit requests needed another round from you.
For a solo builder, that gap is where the time goes: a customer finds the missing piece and you lose an afternoon working out what the agent skipped. The setup around the model matters too. When the authors swapped agent harnesses with the models held fixed, the gap between two models moved by up to 16.2 percentage points on the repository track. Your harness, prompts and checks are part of what you are buying.
Caveats
The edit-breadth result is descriptive. Broader requests scored lower on requirement coverage, but the full-completion rate was about the same (9.1% for one or two change types, 9.7% for three or four). The authors say plainly that this doesn't prove bundling causes the drop. Splitting requests is a sensible inference that the paper didn't test directly.
It's front-end work. Nearly every repository task is a React single-page app. Back-end, data and mobile work weren't tested.
The judges are models. Two front-end experts checked 200 graded outputs, and swapping the judge model moved scores by at most 0.053. That helps, though it doesn't rule out judging errors.
We don't know whose sessions these are. The paper describes "production" coding sessions without naming the product, so the mix of users and request styles is unknown.
How builders and ordinary people can use it today
To use the method yourself, all you need is a checklist written before the agent starts and requests small enough to keep that checklist short.
1. Write your acceptance checklist before you prompt
The researchers wrote hidden criteria first and graded against them. Do the same, in a text file next to your project. Here's a template:
CHANGE: [one sentence: what should be different when this is done]
CHANGE TYPE: [structure | style | interaction | content] (pick one)
MUST BE TRUE (check in the browser, not in the code):
[ ] 1. I can see [element] on [page] at desktop and phone widths
[ ] 2. Clicking [control] does [result], and [state] updates
[ ] 3. The edge case works: [empty / zero / maximum / after delete]
[ ] 4. Values match everywhere they appear: [list each place]
MUST STILL BE TRUE (nothing else broke):
[ ] 5. [Existing feature] still works exactly as before
[ ] 6. No new errors in the browser console
NOT ALLOWED:
[ ] 7. No hardcoded sample data, placeholder text or "TODO"
[ ] 8. No feature that exists in code but has no way to reach itItems 7 and 8 come straight from the refutation step: they are the failure patterns the second model was told to hunt for. For more on checks like these, see our write-up of Trial by Trace, which found that agents can pass a test while still breaking the rules.
2. Ask for one type of change at a time
Instead of this:
Redesign the pricing page with the new brand colours, add a monthly/yearly toggle that updates the prices, move the FAQ above the footer and rewrite the plan descriptions.
That single request covers style, interaction, structure and content. Send it as four turns, checking each against its own short checklist before moving on:
Turn 1 (interaction): Add a monthly/yearly toggle above the plan cards. Yearly shows the yearly price and a "billed annually" note. Don't change anything else. When you finish, list each place a price appears and confirm it updates.
Turn 2 (style): Apply these colours to the pricing page only: [hex codes]. Don't change layout or text.
Turn 3 (structure): Move the FAQ section above the footer. Change nothing inside it.
Turn 4 (content): Replace the three plan descriptions with this text: [paste].
It takes longer to type, but each turn is quick to check, and when something breaks you know which turn broke it.
3. Run your own judge-then-refute pass
After the agent says it's done, open a fresh session (or a different model) and paste in your checklist with this prompt: "Here's what was requested and the acceptance checklist. Try to prove each ticked item is not actually done. Look for stubs, hardcoded values and features that can't be reached from the interface." Check anything it flags yourself. Our guide to the read-back verification loop covers the same idea for agents that act on your behalf.
Sources
Wei et al., Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models, arXiv 2610.35138, submitted 26 October 2026
Full paper (HTML version), arXiv
Hugging Face paper page (posted under the Qwen organisation)
Co-Design Bench code, GitHub
Co-Design Bench dataset, Hugging Face
