Most prompt tuning works like this: you change a line, rerun the three examples you remember, and keep the change if those look better. The prompt gets longer, the bill grows, and you have no idea whether real traffic improved. Anthropic's Lance Martin wrote up a tighter loop on the claude.dev blog : build an eval from real cases, split it, let an agent propose one change at a time, and keep a change only when both splits improve. On an internal support benchmark, he reports, held-out accuracy went from 78.6% to 90.5% at about one fifth of the cost.
This post explains the pattern so you can run it on any prompt that earns you money.
The pattern
First, build an eval: a set of real inputs with a way to score each output. Second, hillclimb: an agent reads failures on one part of the set, proposes a single patch to the prompt, model or settings, and the patch survives only if scores rise on the part it trained on and on a part it never saw. Martin's post packages this as two commands in Anthropic's claude-api skill for Claude Code, /claude-api build-eval and /claude-api hillclimb, but the method works with any model and a spreadsheet.
The objective does not have to be accuracy. Martin points out that cost is a strong goal even when an eval is saturated: ask the agent to cut spend while holding the score at parity.
When to use it and when not
Use it when:
A prompt runs often enough that a few points of accuracy or a few cents per call matter (support replies, lead scoring, routing, extraction).
The right answer is checkable: a label, a decision, a set of facts the reply must contain.
You want to move to a cheaper model and need evidence it holds up.
Skip it when:
You have fewer than about 30 real examples. With tiny sets, noise swamps any gain.
Quality is pure taste (brand voice, jokes) and you cannot write checkable criteria.
Your baseline already scores near the ceiling. Martin's skill warns when the baseline is at about 95% or higher and suggests optimising cost or latency instead.
How it works
Input: a prompt or agent config, plus labelled cases. Step: score the baseline, measure noise, then run rounds of single patches with a keep or revert rule. Output: the configuration that did best on the held-out split, with its score and cost per call next to the baseline.
The worked example in the source: a customer support benchmark of 44 tickets, 30 used for the search and 14 held out. The baseline was Opus 4.8 at default (high) effort, scoring 74.4% decision accuracy on the search tickets at 4.6 cents per ticket. The hillclimb first audited the prompt, removing mandatory tool-call rituals, a scratchpad step and contradictory rules. Opus 5.5 on low effort then scored 87.8% at 1.9 cents. Sonnet 5 on low effort scored 88.9% at about 1 cent. Prompt changes (routing rules and a refund-cap cross-reference) took Sonnet 5 to 98.9% on the search tickets at about the same cost. On the 14 held-out tickets, the final setup scored 90.5% against the original 78.6%.
Note the gap: 98.9% on the cases the agent studied, 90.5% on cases it never saw. That gap is why the held-out split exists. Quote the held-out number, never the training one.
Components
Case set. Martin's order of sources: production transcripts (after checking retention and sensitive data), bug reports and support tickets, five to ten hand-written cases, then synthesised cases.
Grader. Programmatic checks where possible. Otherwise an LLM judge that reads input, output and a rubric written as checkable claims, not a 1-to-5 scale. The judge should not be the model you are testing.
Grader validation. Run the grader twice on the same output and see whether the verdict changes; check for timeouts, API errors and cut-off answers; read a sample of scored transcripts yourself.
Train/test split. Random. The agent may read train transcripts; test stays unseen.
Proposer. An agent that reads the last round's train failures and writes one patch.
Ledger. One row per round: patch, train score, test score, cost per call, decision.
Minimal implementation
Python-flavoured pseudo-code. run calls your prompt on one case; grade returns 1 or 0; propose asks a strong model for one edit given the current prompt and failing train cases (summarised, never pasted in); failures returns the train cases that scored 0.
import random
def score(prompt, cases, reps=2):
results = [grade(c, run(prompt, c)) for c in cases for _ in range(reps)]
return sum(results) / len(results)
random.seed(7)
random.shuffle(cases)
train, test = cases[:40], cases[40:] # 60 cases: 40 train, 20 test
best = prompt
best_train, best_test = score(best, train), score(best, test)
noise = abs(score(best, train) - best_train) # rerun gap = rough noise floor
for rnd in range(6):
candidate = propose(best, failures(best, train))
t, h = score(candidate, train), score(candidate, test)
if t > best_train + noise and h > best_test:
best, best_train, best_test = candidate, t, h # keep
# else revert: train-only gains are overfitting
log(rnd, candidate, t, h, cost_per_call(candidate))In n8n the same loop is a Google Sheet of cases, a Loop node calling your LLM node, a second LLM node as judge, a Code node for the keep/revert rule and an Append row step for the ledger. Run each round by hand at first so you can read the transcripts.
One honest caveat: if the test split gates every decision, you are slowly tuning to it too. Hold back a small third set, ten cases you never look at, and score it once at the end.
Failure modes and evals
Overfitting to the tuning set. Signal: train rises, test stays flat. Martin shows an inbox-routing variant that added two worked examples and was reverted for exactly this. Rule: one patch per round, so you know what caused the move.
Leaking held-out data. The proposer must never see test transcripts, and should never paste failure content into the prompt. Martin also warns that models can "reward hack" by finding answers directly, so keep labels structurally out of the model's reach (separate file, no tool access).
Judge drift. The judge's verdict on the same output can wobble between runs or between judge model versions. Pin the judge model, rerun a fixed sample each session, and read scored transcripts before trusting a jump. We covered how unstable judges can be in LLM judges and novelty instability.
Cost blow-up. Each round costs cases times repetitions times (prompt plus judge). Sixty cases at two reps is 120 runs and 120 judge calls per candidate. Cap rounds, use a cheaper judge for the inner loop, and log cost per call so an accuracy win that doubles spend is visible. Prompt caching helps; see cache-stable agent context.
Gains inside noise. Martin's loop checks first that noise is smaller than the smallest improvement you would act on, and recommends against merging when the final gain is within noise. When the score stalls for two or three rounds, sort the remaining failures by cause before trying more patches.
Build it this weekend
Pick your support-reply or lead-qualification prompt.
Saturday morning: export 60 real cases. For leads, label each hot, warm or cold from what actually happened. For support, write three to five checkable claims per ticket ("states the refund window", "does not promise a callback").
Saturday afternoon: build the grader. Exact-match for lead labels; an LLM judge from a different model family for support claims. Run it twice on 10 outputs and read every verdict. Fix the rubric until the two runs agree.
Saturday evening: shuffle, split 40/20, set aside 10 of the 20 as a final check. Score the baseline twice to get your noise floor.
Sunday: run five or six rounds. One patch each: tighten a rule, delete a contradiction, add a definition. Keep only when train beats baseline by more than noise and test also improves.
Sunday afternoon: try a cheaper model at equal score. A per-request router like the one in first-message model router can then send only the hard cases to the expensive model.
Finish: score the 10 untouched cases once and record held-out score and cost per call against the baseline.
Done means a one-page ledger showing what changed, what it scored on unseen cases and what it costs per call. The support benchmark result belongs to Anthropic's internal test; yours will differ, and the ledger is how you will know by how much.