The paper
SCOPE: Compiling Natural-Language Policies into Neurosymbolic Predicate Programs, by Arnon Mazza and Elad Levi of Plurai. arXiv 2610.31062, version 1, submitted 22 October 2026.
The four benchmark tasks are published on Hugging Face.
The finding in plain English
If you hand a long rulebook to an AI model and ask "did this conversation break the rules?", you get an answer that is expensive and wobbly. Run it again and the verdict can change. This paper turns the rulebook into a small program instead: a set of narrow yes/no questions that a cheap model can answer, plus ordinary code that counts things and combines the answers into a verdict.
On the authors' four test tasks, programs built this way and run on cheap, non-reasoning open-weight models reached accuracy the paper calls comparable to frontier reasoning models. They were also far steadier. Over five repeated runs on the multi-label Tile-match task, only 5–7% of samples changed their prediction at least once for the two best cheap models, against 37% for gpt-5.5 and 26% for gemini-3.1-pro judging the whole policy in one go.
Cost moved the other way. The paper's estimate puts the compiled programs at 9 to 47 times cheaper to run than gemini-3.1-pro. On the customer-support escalation task, that is $0.23 per 1,000 samples against $10.71.
How they did it
A SCOPE program has three parts. Predicates are single classification questions with a fixed answer set, usually yes/no, answered by a small model with reasoning switched off. Extractors are plain functions that pull exact values out of the input, such as a word count or a list of turns. Composition code is Python that decides which question to ask about which slice of the input, and turns the answers into a verdict. Counting is done by code rather than estimated by a model.
The program is built by a loop of strong-model agents (the paper used Claude Opus 4.8 for every compiler role). It starts with one predicate: the whole policy. Each round, it runs the program on labelled examples, traces each wrong verdict back to the question or line of code that caused it, and rewrites, splits or adds questions. Any change has to be justified by a clause in the policy, not by a phrase spotted in one example.
The part the authors lean on is the adversary. After each revision, two generators write new test cases designed to make the program disagree with the policy: one probes single questions, the other probes how the questions combine. A strong "teacher" model labels each case against the full policy, and a case only joins the pool if its label matches the generator's intended one. Switching the adversaries off lowered accuracy on three of the four tasks, for example from 0.865 to 0.813 on the plan-checking task.
The search runs on the cheapest executor, gpt-5.4-nano, so it only keeps questions that a weak model can actually answer. With that executor, accuracy rose from 0.56 to 0.82 on the escalation task and 0.38 to 0.78 on the multi-label task compared with simply prompting the model with the policy. SCOPE scored best or tied in 10 of 12 task-and-model combinations.
Why it matters
Most creators running an AI support inbox, a community bot or a content checker are using the "big model reads the rules" approach. This paper is evidence for three things you can act on.
The model matters less once the rules are split up. Moving from the weakest to the strongest executor added 0.26 accuracy on average under prompt optimisation but only 0.09 under SCOPE. Good decomposition buys back most of what a pricier model would.
Programs travel. Programs compiled on gpt-5.4-nano ran on five open-weight models with no re-tuning, so you are not locked to one vendor's model.
You can see why. Every verdict comes with the list of questions asked, the exact text each one saw and its answer. When a customer disputes a decision, you can point to the failed check.
It also fits a pattern we have covered before: AI judges disagree with themselves when the question is fuzzy, and keeping the control logic in deterministic code makes agents easier to trust.
Caveats
Commercial interest. Both authors are from Plurai, which sells evaluation and guardrail tools for AI agents. A method that makes guardrails cheaper and steadier is close to its product. The paper also cites the authors' own earlier guardrail work.
Small, self-built benchmark. Four tasks, with test sets of 71, 286, 81 and 124 samples. Two tasks were built by the authors from real-world cases; the others extend public benchmarks with authored samples.
Frontier models still win some tasks. On the coaching-audit task, gpt-5.5 scored 0.94 and gemini-3.1-pro 0.96, ahead of every cheap model running a SCOPE program (best 0.90).
The teacher is the ceiling. The authors say plainly that correctness rests on the teacher's reading of the policy. If the strong model misreads a clause, the program inherits the error.
Costs are partial and assumed. The 9–47x figure covers running the program, priced at an assumed $0.15 per million input tokens with output treated as negligible. Compiling with Opus-class agents is a one-off cost the paper does not price. The data is released; I found no compiler code.
How builders and ordinary people can use it today
You do not need the SCOPE code to borrow the idea. Here is a creator's course refund policy, turned into checks.
Policy: "Full refund within 14 days of purchase if you've completed less than 30% of lessons. Coaching calls already held are not refundable. One refund per customer. Anyone who reports a technical problem we couldn't fix gets a refund regardless."
Step 1: split it into extractors (exact data, no AI). Days since purchase, lesson completion percentage and prior refund count come from your payment and course platforms. Never let a model guess these.
Step 2: write self-contained yes/no questions. Each one must carry its own definitions, because the cheap model never sees the full policy.
"Is the customer asking for their money back for the course? Answer yes or no. Asking about a discount or a pause is no."
"Does the customer describe a technical problem that stopped them accessing the course (login, video playback, missing files)? Yes or no."
"Does our support reply in this thread say the problem was fixed? Yes or no."
Step 3: combine in code.
def refund_verdict(t, data):
if not ask(Q1, t.last_customer_message):
return "not_a_refund_request"
if ask(Q2, t.customer_messages) and not ask(Q3, t.support_replies):
return "refund_full" # unfixed tech issue overrides
if data.prior_refunds >= 1:
return "decline_repeat"
if data.days_since_purchase <= 14 and data.completion < 0.30:
return "refund_course_only" # coaching calls held stay non-refundable
return "decline_outside_window"Step 4: test like the paper. Label 30 to 50 past tickets by hand and hold some back (we covered why a held-out set matters). Run each ticket five times and count how many verdicts flip. Then ask a strong model to write ten tricky tickets per question, such as a "refund" request that is really a pause request, and fix whichever check breaks.
The same recipe works for community moderation. "No self-promotion outside #wins; asking for feedback on your own work is fine; three strikes and you're muted" becomes: an extractor for the channel name, a question "does this post link to or pitch something the author sells?", a question "is the author asking for feedback rather than buyers?", and code that counts strikes. The paper's escalation task follows the same pattern, flagging a user who re-raises the same unresolved intent at least three times.
Run the questions on the cheapest model you trust, keep the trace, and send anything ambiguous to a human.