Risk-Tiered Code Review for AI-Written PRs

When an agent writes most of your code, review becomes the bottleneck. On 15 October the Claude Code team at Anthropic published how they restructured code review around that problem. Machines read every pull request first, and a classifier decides whether zero, one or two people need to approve it. Their stated goal is that most PRs merge "with at most one human in the loop, who can be the author, with no increase in reverts or incidents."

The team reports that merged pull requests more than doubled from June to September while the share that had to be reverted fell by more than 40%. Treat those as self-reported before-and-after numbers. The post describes no control group, and the window overlaps new model releases: Anthropic's newsroom lists Claude Opus 5.5 on 22 September and Claude Sonnet 5.5 on 28 September. Some of the gain could come from better models. The pattern is still worth copying, because each layer makes sense without the headline figures.

The pattern

Risk-tiered review has three parts:

Automated reading on every change. CI, an AI reviewer and a security pass run before any human looks.
A risk tier set by the change, not the author. The team says "the change sets the tier, not the author." Low risk needs no human once checks are green. Medium needs one person. High needs two. Some paths always need a person: anything owners have listed as sensitive, and any CI workflow file.
A bounded AI loop. After three rounds the reviewer stops asking for more fixes and tells the author to reconsider the approach.

For a solo builder, the second part is the useful one. It turns "should I read this agent-written PR?" from a gut call into a rule you can test.

When to use it and when not

Use it when:

An agent opens more PRs than you can read properly. If you have started skimming them, you have already lost most of the value of review.
Your changes are small and easy to undo. The team leans on flags, reverts and staged releases that "undo most mistakes in minutes."
You have CI that actually fails when something breaks.

Skip it, or keep everything at the top tier, when:

You have no tests. If there are no automated checks, a "low" tier just means nobody checked the change at all.
You cannot roll back quickly, for example database migrations without a down path, or mobile builds already in review.
The code handles money, auth or customer data and you have no second person. In that case the rule is simply "I always read it".

How it works

Input: a pull request (diff, file paths, line counts, PR description).

Steps (as the team describes them):

Before the PR: the agent runs tests, build and lint, and a second agent reviews the diff in a fresh context.
Required CI: lint, types, tests and build. A red check blocks the merge.
AI review: several agents hunt for bugs in parallel, a separate group votes on each finding, and only findings enough of them agree on are posted. Each one is tagged Important, Nit or Pre-existing, so a bug the PR did not introduce is not blamed on the author.
Security review: a separate pass. Open findings from either review block the merge.
Risk classifier: assigns the tier and therefore the number of human approvals.
Merge queue, then a flag, internal use, staged release and fast rollback.

Output: a merged change with the right number of humans involved, plus a record of findings you can use to grade the reviewer later.

Components

Reviewer prompt plus a REVIEW.md. The team suggests moving comments you keep repeating into this file.
Noise filter. Drop findings that are only about style or cleanup. The team says noise costs more when the author is an agent: a person waves off a nit in seconds, while "an agent fixes it, which means another commit and another round of review."
Risk classifier. Start with a deterministic one. Rules you can read beat a model you have to trust, which is the same argument as in keeping your agent router deterministic.
Round counter. Stored as a PR label or comment so the loop knows when to stop.
Path guards. CODEOWNERS plus branch protection, so CI, auth, billing and release files always request a human regardless of what the classifier says.

Minimal implementation

This pseudo-code assumes GitHub Actions and the Claude Code GitHub Action (anthropics/claude-code-action@v1, which takes anthropic_api_key and prompt inputs). Any LLM reviewer you can call from CI works the same way.

# .github/workflows/review-router.yml  (pseudo-code; adapt before use)
on: pull_request
jobs:
  review:
    steps:
      - checkout
      - run: npm test && npm run lint            # red blocks merge
      - id: tier
        run: python scripts/risk_tier.py > tier.txt
      - id: rounds
        run: echo "n=$(count_label ai-review-round)" >> $GITHUB_OUTPUT
      - if: steps.rounds.outputs.n < 3
        uses: anthropics/claude-code-action@v1
        with:
          anthropic_api_key: <ANTHROPIC_API_KEY from your repo secrets>
          prompt: |
            Review only the diff. Report bugs that change behaviour.
            Tag each: IMPORTANT | NIT | PRE-EXISTING. Skip style-only notes.
            Do not repeat findings already posted on this PR.
      - if: steps.rounds.outputs.n >= 3
        run: comment "Three AI rounds done. Step back and rethink the approach."
      - run: apply_label "tier-$(cat tier.txt)"   # branch protection reads this

# scripts/risk_tier.py — rule-based, no model
ALWAYS_HUMAN = [".github/workflows/", "auth/", "billing/", "migrations/", "Dockerfile"]
LOW_OK = ["docs/", "README", "tests/", ".md"]

def tier(files, lines_changed):
    if any(p in f for f in files for p in ALWAYS_HUMAN):
        return "high"
    if lines_changed > 400 or len(files) > 15:
        return "medium"
    if all(any(p in f for p in LOW_OK) for f in files) and lines_changed < 100:
        return "low"
    return "medium"

Start strict. The team advises starting with tight tiers and loosening them only with evidence, and never letting a bot's approval stand in for a required human's.

Failure modes and evals

The loop that never ends. The team found one CLI PR that went through more than fifty rounds of review and fixes overnight, and about 59% of what got flagged had been there since the reviewer's first pass. They added the round cap, diff-only re-review and a rule against repeating findings. After those changes, they report the share of PRs passing after one review-and-fix round rose from 65% to 78%, and the share needing five or more rounds fell from 8.9% to 1.5%.
A noisy reviewer. Sort every finding by what the author did with it: fixed, wrong, intentional, or didn't matter. If most findings are ignored, the reviewer is costing you time. Cost per acted-on finding is the number to track.
A missing reviewer. The team reports that the share of internal PRs getting substantive review comments went from 16% to 54% after launch. Before that, most PRs got no substantive comment at all. Check what your own baseline is.
Tier drift. Something risky gets labelled low because it touched an unlisted path. Fix this by adding paths to ALWAYS_HUMAN after every incident.
Silent regressions after a model or prompt change. Keep a small eval set of past bad PRs (reverts, bugs you shipped). Rerun it whenever you change the prompt, tools or model, which matches the team's practice of rescoring on every change. Also run the reviewer on known-clean PRs and count false alarms.

Give the reviewer enough context. Whole files around the diff tend to help: we covered research showing coding agents fix more bugs when you paste whole files.

Build it this weekend: the "do I even need to look at this?" router

Goal: every PR on one repo gets a label look: no, look: skim or look: read before you open it.

Saturday morning. Export your last 30 merged PRs: files touched, lines changed, and whether each one was later reverted or patched. This is your eval set.
Write risk_tier.py using the rules above. Run it against those 30 PRs. Every PR that caused trouble must come out high or medium. If one comes out low, add its path to ALWAYS_HUMAN and rerun.
Saturday afternoon. Add the workflow. Map low to look: no, medium to look: skim, high to look: read. Add .github/workflows/ to CODEOWNERS with yourself as owner.
Add the round cap. Count ai-review-round labels and stop the AI reviewer at three.
Sunday. Open five PRs on purpose: a docs typo, a test, a 50-line feature, a dependency bump and a workflow edit. Check that each gets the label you expected.
For the next two weeks, read every PR anyway and note when you disagreed with the label. Loosen a rule only when the log shows it was too strict.

Pass criteria: no reverted PR in your history scores low, every workflow file scores high, and no PR goes past three AI rounds.

From today's Daily Briefing →