Cloudflare runs an AI code reviewer on its internal merge requests: a coordinator that hands each change to specialist sub-reviewers. On 19 October the team published how it brought the average review down from $2.43 to $0.77, a 68% cut (the $2.43 baseline is its average over the two most expensive weeks before the work began), while serious findings that led to a code change rose from 229 to 321 per 1,000 reviews. All figures are Cloudflare's own, self-reported.
The useful part is how they decided what to cut: by what each sub-agent cost per finding somebody actually acted on.
The pattern
Give every sub-agent one scorecard number: cost per acted-on finding. Total spend for that role divided by the outputs a human (or a downstream step) actually used. Then make four changes and re-measure after each one:
Skip sub-agents that have nothing new to look at.
Inline the reference material each sub-agent always fetches, as a short index in its prompt.
Strip tool definitions the sub-agent cannot or should not use.
Set reasoning effort per role, based on that role's scorecard.
Run the numbers on Cloudflare's headline figures and the shift is stark. By our arithmetic, $2.43 across 0.229 acted-on serious findings per review is about $10.60 per acted-on finding. $0.77 across 0.321 is about $2.40. That is a 4x improvement on the metric that matters, which is bigger than the 68% headline.
When to use it and when not
Use it when you have a fan-out agent: one orchestrator, several specialists, run often enough that cost shows up on an invoice. Code review, content QA, lead research and support triage all fit, as long as you can observe whether an output was used: a line edited, a draft accepted, a lead contacted.
Skip it if you run a single agent a few times a week; the measurement effort costs more than the savings. Be careful where "acted on" is a poor proxy for value. A security reviewer may rarely trigger a change and still be worth every cent on the day it does. For those roles, set a floor (always run on certain file types or risk tiers) and judge the rest on the scorecard.
How it works
Input: a job (a merge request, a draft, a ticket) plus a record of what happened last time this job was reviewed.
Step: the coordinator classifies the job into a risk tier (Cloudflare uses trivial, lite and full), decides which specialists have anything new to check, and calls only those. Each specialist starts with a prompt that already holds its reference index, sees only the tools it needs, and runs at the effort level its scorecard justifies.
Output: findings, each tagged with role, severity and cost. A later check records which ones were acted on, and that feeds the scorecard for next week's tuning.
Components
A skip rule for repeat runs. Cloudflare found that about two-thirds of its jobs are a second or later review of the same merge request, and often three reviewers had nothing new to see. On full-tier re-reviews the AGENTS.md reviewer is now skipped 56% of the time, performance 23% and documentation 13%. The post does not spell out the skip signal; the obvious one to build is "did anything this role cares about change since the last pass?"
An inlined reference index. The standards reviewer used to spend its first three turns reading reference files with tool calls. Moving a table of standards, a skill file and a list of reference files into the prompt cut its LLM calls per run from 15.9 to 9.6 and its cost per run from $1.12 to $0.72. If every run fetches the same files first, those files belong in the prompt.
A trimmed tool list. Sub-reviewers could see the tool that posts the final review comment, a job only the coordinator does. They tried to call it in 57% to 76% of sessions and were rejected by the server. Removing it saved about 2.9k tokens of tool definitions in every sub-reviewer request, plus the wasted turns.
Per-role effort. Anthropic's effort parameter changes all tokens in a response, including how many tool calls the model makes. Cloudflare tested the code-quality reviewer at both levels: high effort costs 61% more per run and posts 58% more critical and warning findings. The deciding number was further down: medium led to roughly a quarter as many acted-on findings as high, so code quality is moving to high. By our arithmetic, high effort makes each acted-on code-quality finding roughly 60% cheaper, even though each run costs more. Security and performance stay on medium for now.
Cache-stable prompts. Caching still mattered. Cloudflare found that the first request of a session found a cached prefix only 14% to 26% of the time, and fixed it by splitting the shared prompt block from the per-job block. Stripping tools helps here too, because changing tool definitions invalidates the entire cache. We covered the mechanics in cache-stable agent context, so we won't repeat them.
An acted-on detector. Cloudflare counted a finding as acted on if the lines it pointed to changed in the author's next push. As a control, lines 30 to 40 away changed about 15% of the time in every period, so subtract that baseline before you celebrate.
Minimal implementation
Pseudo-code for a coordinator. It maps to an n8n flow as Trigger → Code node (skip rule) → Switch → one AI Agent node per specialist → Merge → write to a findings table.
ROLES = {
"quality": {"model": "mid", "effort": "high", "tools": ["read_file"], "watches": ["*.py","*.ts"]},
"security": {"model": "mid", "effort": "medium", "tools": ["read_file"], "watches": ["*"], "always_on_tier": "full"},
"docs": {"model": "small","effort": "low", "tools": [], "watches": ["*.md"]},
}
INDEX = load("standards_index.md") # short table + file list, inlined
def review(job, last_run):
tier = classify(job) # trivial / lite / full
changed = diff_since(last_run, job)
findings = []
for role, cfg in ROLES.items():
if not touches(changed, cfg["watches"]) and cfg.get("always_on_tier") != tier:
log_skip(role); continue
out = call_llm(system=SHARED_PREFIX + INDEX,
user=role_prompt(role, changed),
tools=cfg["tools"], effort=cfg["effort"], model=cfg["model"])
findings += tag(out, role=role, cost=out.cost)
post_comment(findings) # only the coordinator has this tool
return findings
def scorecard(week):
for role in ROLES:
acted = count_acted_on(role, week) - baseline_change_rate(role, week)
print(role, spend(role, week) / max(acted, 1))Two details are easy to miss. Keep SHARED_PREFIX byte-identical across jobs so it caches. And make sure cancelled jobs actually stop: in a sample of 446 cancelled jobs, 358 kept running until the review finished. Cloudflare's jobs now check their own status every 15 seconds.
Failure modes and evals
Cheaper and quieter. Skipping roles drops cost and findings together. Track acted-on findings per 1,000 jobs alongside cost; if acted-on falls, your skip rule is too aggressive.
Severity inflation. A sub-agent that marks everything critical looks productive. Cloudflare had blinded model judges from two providers rate 657 findings and saw over-severe findings per review fall from 0.46 to 0.14, a 70% drop. Sample 50 findings a week and grade severity yourself.
Stale index. An inlined index that goes stale makes the agent confidently wrong. Regenerate it from source on every deploy.
Small samples. Cloudflare is candid that its step comparisons were not a controlled experiment and some groups held only 35 to 66 findings. Run settings side by side (it split code quality 50/50 between effort levels) before you lock one in.
For the eval harness itself, a held-out set of past jobs with known good findings works well; our post on automated evals and prompt hillclimbing walks through one. If your coordinator is itself an LLM call, consider making routing deterministic, as in the thin orchestrator.
Build-it-this-weekend exercise
Pick one multi-step agent you run at least daily. List its sub-agents or steps.
Add three fields to every output: role, cost in dollars, and an
acted_onflag you fill later (edited, accepted, sent).Run it for two days unchanged. Compute cost per acted-on output for each role.
Make one change per role: add a skip rule to the worst performer, inline the files your most tool-hungry role always reads, and delete every tool a role never legitimately calls.
For the role with the most acted-on output, run half its jobs at a higher effort level for two days and compare cost per acted-on output.
Keep what lowered the number; revert the rest.
Our take: per-run cost is the wrong dashboard for a multi-agent system. Cost per acted-on finding tells you which roles to skip and which to pay more for, and Cloudflare's own data shows the answer is sometimes "pay more".