Most solo builders protect a risky agent by approving every action by hand, which gets tiring fast, or by trusting the prompt. Shopify has published a third option you can copy in a weekend. A second agent, on a different model, watches the working agent live and can stop it.
If that watcher goes quiet, the work stops as well.
The pattern
Shopify calls its system the Cyber Chaos Monkey, described by Dan Scofield on the Shopify Engineering blog on 12 November 2026. Its AI agents attack Shopify's own live systems to find security holes before outsiders do, which is about as risky as agent work gets.
The core idea is borrowed from government security and red-team work. Two-person integrity means a second party reviews the work of whoever is acting to confirm it poses no security risk. Shopify's version replaces the second person with a second agent. It says it feeds the session logs from the offensive agent into a separately aligned agent, using a different model, in real time. Shopify calls the worker the executor and the watcher the judge. When the judge sees behaviour that breaks the rules, it makes a tool call that kills the executor instantly.
The second half of the design is the part most people would skip. Shopify requires what it calls positive control: if the judge times out, crashes or fails to rule on the executor's messages, the run ends. The orchestrator, a small program in charge of the run, sends a regular "are you alive?" signal, called a heartbeat, to both agents. If either stops answering, everything shuts down. The system fails closed: a missing verdict counts as a no.
Shopify reports that its agents have run tens of thousands of offensive engagements and acted on dozens of impactful findings without causing an outage or incident. Those are Shopify's own figures; it gives none on how often the judge stopped a run or what it costs.
OpenAI described a close cousin on the same day. In its post on securing persistent agents like OpenClaw, it explains Auto-review, where a second agent monitors and approves the actions of a primary agent. OpenAI's reason will sound familiar: constant human review creates fatigue, and people learn to click "accept" without reading. The reviewer clears routine actions so human attention goes to the ones that matter. OpenAI says it enforces auto review on all actions of its OpenClaw agents.
When to use it and when not
Use a judge when an agent takes actions you cannot easily undo and you are not watching every run. Sending email to customers, issuing refunds, spending on ads, posting publicly, editing your store or deleting files all qualify. The test is simple. If a bad run would cost you money, customers or a weekend of clean-up, it needs a watcher.
Skip it when the agent only reads, or only drafts something you review before it goes anywhere. You are already the judge there, and a second model adds cost and delay for no gain. Expect the judge to add tokens for every step it reviews and a second or two of waiting per action.
How it works (input → step → output)
Input: a task for the working agent, plus a short written rules list. Shopify calls its version the Rules of Engagement. They leave the agent free to try things inside its target and set hard limits on what it may do next.
Step: the worker proposes or takes each action. Every event, including its reasoning, tool calls and tool results, is passed to the judge as it happens. The judge returns a verdict: allow, or kill. Meanwhile the orchestrator checks that the judge is alive on a fixed interval.
Output: a finished run with a log of every verdict, or a stopped run with the reason.
Components
Worker agent. The one doing the job, on whatever model suits the task.
Judge agent. A different model, ideally from a different provider, with its own prompt. A different model is less likely to share the worker's blind spots.
Rules list. One page, written as things the agent must never do: "never send to more than 20 recipients", "never refund over $100", "never touch files outside /drafts".
Orchestrator. Plain code, with no model inside it, that starts both agents, routes events, sends the heartbeat and has the power to end the run.
Verdict log. Every action, verdict and reason, saved where you can read it the next morning.
Least privilege. Shopify also runs the worker in a locked-down container with short-lived credentials it can read but not change, and an allowlist of websites it may reach. The judge sits on top of those limits.
Minimal implementation
Here is the loop in Python-style pseudo-code. The key line is the timeout: a slow or broken judge produces a block, never an allow.
JUDGE_TIMEOUT = 10 # seconds
HEARTBEAT_EVERY = 15 # seconds
def run(task):
worker = start_worker(task, rules)
judge = start_judge(model="different-provider-small", rules=rules)
for event in worker.stream(): # thoughts, tool calls, results
if time_since(judge.last_pong) > HEARTBEAT_EVERY * 2:
return stop(worker, "judge heartbeat lost")
if event.type != "tool_call":
judge.observe(event) # context only
continue
try:
verdict = judge.review(event, timeout=JUDGE_TIMEOUT)
except (Timeout, Crash, BadOutput):
log(event, "BLOCK", "no verdict")
return stop(worker, "judge failed, fail closed")
log(event, verdict.decision, verdict.reason)
if verdict.decision != "ALLOW": # anything unclear counts as a no
return stop(worker, verdict.reason)
worker.execute(event) # only now does the action happenIf you build in n8n or a similar tool, the same shape works. Put a "Wait for judge" HTTP call before every node that sends, pays or deletes. Set the node timeout to ten seconds and route the error branch to "Stop and alert" instead of "Continue". Ask the judge for JSON with a decision and a reason, and treat anything that fails to parse as a block.
A judge prompt can be short: "You review one proposed action from an assistant. Here are the rules. Here is the action and the conversation so far. Reply ALLOW only if the action clearly follows every rule. Otherwise reply BLOCK with the rule it breaks."
Failure modes and evals
Silent fail-open. The most common bug: a try/except that logs the error and carries on. Test it by pointing the judge at a dead URL. The run must stop.
Rubber-stamp judge. A judge that allows everything looks fine until the bad day. Before you trust it, feed it 20 actions you have labelled yourself, 10 safe and 10 rule-breaking, and check it catches the bad ones. Our post on calibrating an LLM judge against human labels walks through this.
Nervous judge. Too many false blocks and you will switch it off. Loosen rules that fire on harmless actions.
Judge sees too little. Shopify's judge reads everything, including the worker's reasoning and tool results. A judge that only sees the final action can miss a poisoned instruction that arrived three steps earlier.
Judge as the only lock. A judge can be fooled. Keep hard limits in code too, such as spending caps and credentials scoped to the approved plan.
Build it this weekend
Pick your one riskiest agent, the one that sends email or touches your store.
Write the rules list. Five to ten "never" lines, each specific enough that a stranger could check it.
Wrap every action that sends, pays, posts or deletes so it calls the judge first. Leave read-only steps alone.
Choose a small, cheap model from a different provider than your worker as the judge.
Add the ten-second timeout and make every error path a block. Then break it on purpose: kill the judge mid-run and confirm nothing gets sent.
Run the 20-action test from the evals section. Fix the rules until it catches all 10 bad actions.
Log every verdict to a spreadsheet. On Monday, read the blocks and decide which were right.
If a blocked action should go to you instead of disappearing, add one more route: send blocks to your phone for a yes or no. That gives you OpenAI's split, with a model clearing the routine and you deciding the few that matter. For actions where you want to see the result before it is real, pair this with sandbox-first approval.
The decision rule to take away: any agent that can spend, send or delete gets a second model watching, and a silent watcher means stop.

