First-message model router: cut median agent cost 64%

Most agent setups send every request to the strongest model they can afford. A one-line config fix and a cross-file refactor get the same frontier model and the same bill. LangChain's Open SWE team tested an alternative: read the first message, pick a model tier for the whole thread, and measure whether anything breaks. Their write-up by Sydney Runkle and Eugene Yurtsev (1 October 2026) is a rare public A/B test of this pattern, and it is small enough to copy into your own agent this weekend.

The pattern

Before the agent starts work, a small classifier reads the first user message and chooses one of three tiers: fast, balanced or performance. The thread then runs on that model from start to finish. In LangChain's words, the base prompt "tells the classifier its job: pick the least expensive model likely to complete the task."

The results come from an A/B test in which half of threads went through the router and half always used the strongest model, GPT-6 Astra, across 973 threads. "The median routed thread cost $0.94 vs. $2.61 on control, 64% less." Quality, measured as merged pull requests, did not move in a way they could detect: "29.2% of routed threads ended in a merged PR vs. 27.3% of control (p = 0.49)." The mean cost fell 42% and the p90 fell 37%, so the saving was spread across the distribution and did not come from a few cheap outliers.

The authors are open about the caveat: "This result isn't surprising, given the control was our most expensive model." They call it a proof of concept. Read it as evidence that many tasks don't need the top tier. It does not show that routing beats a well-chosen mid-tier default.

When to use it and when not

Use it when:

Task difficulty varies a lot and a good share of requests are easy (lookups, small edits, formatting, short drafts).
Everything currently goes to a frontier model by default.
You have a per-task success signal you trust: a merged PR, a sent email, a published post, an accepted answer.

Skip it when:

You cannot measure quality per task. Without a success metric you are only measuring the saving.
Tasks often change difficulty partway through a thread. The router decides once, and switching models mid-thread has a cost: "switching models throws it away, so the new model re-reads the thread at full price." (If prompt caching is new to you, our post on cache-stable agent context explains why that matters.)
Your cheapest model is not good enough for real work. LangChain ran a second test against an always-fast arm, and "We ended this test within a day." Engineers reported low output quality that was disrupting their work.

How it works: input, classify, route, run, log

Input. The first human message in the thread, plus whatever the harness already knows: repo, user, task type.
Classify. One cheap call with structured output that returns a tier name and nothing else.
Route. Look the tier up in a routing table and store it on the thread state, so later turns reuse it instead of classifying again.
Run. The agent runs as normal on the chosen model. In Open SWE this is middleware using dynamic model selection, which swaps the model before each call, so the rest of the agent doesn't change. The public model_selection.py reuses a stored route if there is one and falls back to a default model when the classifier returns nothing.
Log. Record the tier, cost, outcome and A/B arm for every thread.

The authors argue that "An effective model router belongs in the harness, not a generic gateway." A gateway sees tokens. The harness knows the task, the tools and the domain, which is what the tier criteria are written against.

Components

Base prompt: the single instruction to pick the cheapest model likely to succeed.
Tier criteria: plain-language descriptions of what belongs in each tier, written for your own tasks.
Classifier model: a fast, cheap model with structured output. LangChain's first version used an LLM with structured output.
Routing table: tier to model, kept in config so you can swap models without touching the prompt.
Thread state: somewhere to store the chosen tier and the experiment arm.
Logger: cost, outcome and feedback per thread.

For reference, Open SWE's tiers as of 9 September 2026 were GLM-5.3-Flash for fast, GPT-5.6 Sol for balanced and GPT-6 Astra for performance. Of routed threads, 56% went to balanced, 34% to fast and 10% to performance. Median cost per thread was $0.097, $1.50 and $2.88 by tier, roughly a 30x spread from bottom to top.

Minimal implementation

A sample classifier prompt to adapt (this is ours, not LangChain's, whose prompt text is not published in the post):

You route tasks for an AI assistant. Pick the LEAST expensive tier
that will very likely complete this task well on the first try.

fast: single-step, low-risk work. Rewording, formatting, short
  replies, simple lookups, one-file edits with clear instructions.
balanced: multi-step work with some judgement. Drafting a full post,
  summarising several sources, small features, routine debugging.
performance: ambiguous, high-stakes or long-horizon work. Strategy,
  multi-file changes, unclear requirements, anything customer-facing
  where a mistake is costly.

If unsure between two tiers, choose the higher one.
Return JSON only: {"tier": "fast" | "balanced" | "performance"}

Task:
{first_message}

A sample routing table (fill in your own models and check current prices before you commit):

Tier	Model slot	Typical tasks	On failure
fast	your cheapest capable model	formatting, replies, lookups	retry once on balanced
balanced	your mid-tier model	drafts, summaries, small builds	retry once on performance
performance	your frontier model	strategy, multi-file, ambiguous	flag for a human

Pseudo-code for the harness:

ROUTES = {"fast": FAST_MODEL, "balanced": MID_MODEL, "performance": TOP_MODEL}

def start_thread(thread, first_message):
    thread.arm = "router" if hash(thread.id) % 2 == 0 else "control"
    if thread.arm == "control":
        thread.tier = "performance"
    else:
        try:
            out = classifier.structured(PROMPT.format(first_message=first_message[-8000:]),
                                        schema={"tier": ["fast", "balanced", "performance"]})
            thread.tier = out["tier"]
        except Exception:
            thread.tier = "balanced"          # safe default
    thread.save()

def call_model(thread, messages):
    return llm(ROUTES[thread.tier]).invoke(messages)   # tier fixed for the thread

def finish_thread(thread, outcome, cost):
    log(thread.id, thread.arm, thread.tier, cost, outcome, thread.feedback)

The n8n (or any no-code LLM-steps) version:

Trigger: webhook, form or chat message.
Code node: assign the arm by hashing the thread or ticket ID.
LLM node (cheap model): the classifier prompt above, with a structured output parser.
Switch node: route on tier to three AI Agent nodes, one per model.
IF node: if the agent errors or a check fails, send it to the next tier up.
Data table or sheet: append thread ID, arm, tier, tokens, cost and outcome.

If you already run workflows this way, it slots into the stack we described in the agentic workflow stack for creators.

Failure modes and evals

How to A/B it yourself. Split by thread, not by message, so a conversation never changes models. Keep your current setup as control. Choose one primary outcome before you start and stick to it.

Sample size. LangChain needed 973 threads, and even then the quality comparison only told them there was no detectable difference. A solo builder with 30 tasks a day will need several weeks for a firm answer. Cost differences show up much sooner than quality differences, so don't stop the test once the cost chart looks good.

What to log:

Thread ID, arm, tier and the classifier's raw output
Input and output tokens, cost, tool-call count and duration
The primary outcome (published, merged, sent, resolved)
Thumbs up or down, though the authors found feedback sparse and merged PRs the stronger signal
Every escalation and the reason for it

Guardrails:

Escalate on failure: one automatic retry on the next tier up, then a human.
Default up: on a classifier error or an unclear answer, pick balanced, never fast.
Hard overrides: let the user or a keyword force a tier, as Open SWE does when a model is requested explicitly.
Kill switch: one config flag that sends everything back to control.
Weekly drift check: read ten fast-tier threads every week to catch quality problems that your metric misses.

Judging output quality automatically is hard, which is one reason LangChain leaned on real-world outcomes. Our post on improving an agent's harness without a ground-truth checker covers the same problem from the research side.

Build-it-this-weekend exercise

Saturday morning: export a week of agent or chat logs. Label 50 tasks by hand as fast, balanced or performance.
Saturday afternoon: write your tier criteria from those labels, run the classifier prompt over the same 50 tasks, and check agreement. If more than one in ten hard tasks lands on fast, tighten the criteria.
Sunday: add the router behind a flag, with the arm split, escalate-on-failure and the logger.
Next week: run it alongside your current setup. Compare median cost and success rate per arm, and read every escalation.

The outcome to look for is LangChain's: a lower cost per task with a success rate you cannot tell apart from control. If quality drops on the routed arm, check the fast-tier threads first, then rewrite the criteria for that tier.

From today's Daily Briefing →