If you only check whether your AI agent got the right answer, you are missing most of what went wrong on the way there. A new benchmark from TraceLite AI and five Chinese research institutions ran 3,420 agent runs and found that nearly a third of them finished the task while breaking at least one of the rules the surrounding software was supposed to enforce. Here is what the paper did, what it found, and a trace-audit checklist you can use on your own agents this week.

Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution, arXiv 2610.32285, first version submitted 23 October 2026.

Authors: Zhiyong Cao, Hengjie Zhu, Yongxin Wang, Xuchen Li, Jintao Li, Shuxia Liu, Dunqiang Liu, Bo Wang, Kairan Zhang, Cong Lin, JiangDi Yu, Siyuan Liu, Sijia Yao, Yi-Liang Wei, Lingxu Ma, Aijia Wang, Anxin Chen, Zixian Wei, Xiang Ruan, Ruizhi Xu, Ruqian Shi, Baihua Xiao and Sicheng Zhou, from TraceLite AI, the Institute of Automation and the Institute of Information Engineering at the Chinese Academy of Sciences, Zhongguancun Academy, Tsinghua University and Beijing Institute of Technology.

An AI agent is two things: the model that makes decisions, and the harness around it. The harness is the software that decides what the model sees, which tools it can call, how command results and errors get passed back, what happens when output is too long, how sub-agents hand work back, and what needs approval. Claude Code, Codex and similar tools are harnesses.

Most benchmarks score the whole agent on whether the final answer or file is correct. This paper scores the harness separately, on whether it did its job faithfully during the run. The headline: across four harnesses and nine models, 31.1% of runs achieved task success despite failing at least one harness criterion, and macro-averaged harness scores ranged from 59.8% to 72.0%.

Put those numbers next to each other and the picture gets starker. In the paper's results, 1,351 runs (39.5%) passed every task criterion, but only 286 runs (8.4%) passed every criterion on both the task and the harness. So roughly four in five "successful" runs had a process problem hiding underneath the correct result.

The rankings also changed depending on what you measure. Claude Code had the highest task success, while OpenClaw had the highest harness score:

Source: Table 6 of the paper, also shown on the TraceLite project page. Each harness was run on the same 855 model and task combinations. The gap between best and worst harness was 12.2 points on harness score, against 8.2 points on task success.

The benchmark, called TraceTrial-Bench, splits harness reliability into seven components: context (information kept or leaked across boundaries), execution (faithful process results and exit states), governance (approvals, guardrails, secret redaction), lifecycle (state transitions, recovery, retries that do not repeat actions), observability (diagnostics and replayable logs), tools (valid tool interfaces and transport) and verification (whether completion claims match what actually happened).

They built 95 tasks that each poke at one of these mechanisms inside ordinary work. Examples from the paper include outputs too large to fit, a command that exits with an error while leaving a success-looking file behind, pending background jobs, facts that must cross a session boundary, and values only available by asking the user. Each task has a ground truth the agent cannot touch, such as a private simulator ledger or a private baseline copy of the workspace.

Every run was graded twice. The agent layer checks the deliverable against that ground truth. The harness layer checks the trace: did the model actually see the timeout, did the child agent's result reach the parent, did the action run exactly once. Deterministic checks are used wherever possible, and an LLM judge (deepseek-v4-flash in all 3,420 evaluations) handles criteria that need interpretation, using only a bounded packet of selected evidence.

Failure attribution works through pre-declared "mechanism chains". Before a task runs, the authors write down which harness failure could plausibly cause which task failure, and through what intermediate distortion. If both ends fail in a run, the link is flagged as a candidate. The paper is careful here: a flagged link is a lead for review, and only manual review of event order and competing explanations turns it into a validated cause. Runs where the harness slipped but the agent still delivered fall into what the paper calls an absorption state: a harness zero with no task failure. That state is what the 31.1% figure counts.

The models tested were DeepSeek-V4-Pro, Gemini 3.7 Flash, GLM-5.3, GPT-5.6-Sol, Grok-4.6, HY3, Kimi K3, Opus 5 and Qwen3.8-Max.

If you build on top of an agent harness, a pass/fail test tells you less than you think. A run that reaches the right answer after silently dropping an error, repeating a payment call, or losing a fact during context compaction is fragile. The model covered for the plumbing that time, and the same slip can produce a wrong answer when the task or the model changes.

It also means "which tool should I use?" has two answers. The tool with the best task completion was not the one with the most faithful process. For low-stakes drafting, task success may be all you care about. For anything that touches money, customer data or irreversible actions, process faithfulness matters more.

This lines up with earlier work we covered. SCOPE turned written policies into yes/no checks a cheap model can run, which is the same idea applied to rules: break the expectation into small, checkable criteria. And if you already tune prompts against a held-out test set, this paper is an argument for adding process criteria to that test set rather than only outcome checks. It also complements Harness 2, which improves an agent's harness task by task: you need a way to measure the harness before you can improve it.

You do not need the full benchmark to apply the idea. The TraceTrial-Bench code on GitHub includes adapters for the four harnesses and a Docker runner if you want to test one directly. For your own agents, start smaller.

Builders: a 30-minute trace-audit checklist. Turn on full logging (all four harnesses in the paper expose JSON event streams or session records that can be saved), pick ten recent runs that "succeeded", and for each one answer yes or no:

Score each run on the seven items as well as on its outcome. Any run that passed the outcome but failed an item goes into your test set as a regression case. Re-run the checklist when you change model or harness version.

Ordinary users of agent tools: when an agent says it finished something important, open the activity log and look for red error text, repeated steps and "output truncated" notices before you trust the result. Check the real destination (the sent folder, the bank record, the file) rather than the agent's summary.

Choosing a harness: run five of your own representative tasks through two candidates and compare both the result and the checklist above. The paper's table suggests the two can rank differently.

From today's Daily Briefing →