Hint, Don't Answer: Train a Cheaper Task Model

Shopify's Sidekick assistant runs on a frontier model, but some of its most expensive work no longer does. On 21 October 2026 Shopify Engineering described how Sidekick hands focused jobs to subagents running open models "we've post-trained for that one task". The clearest case is GraphQL query writing, where the subagent reads about 40 tokens for every one it writes, and a frontier API bills for every one of them.

Shopify reports that its post-trained model now scores 74.5 on its GraphQL judge against 73 for the frontier model, returns a full response in 0.87 seconds at the median against 6.50 seconds, and costs about 90% less to serve on that traffic. Two caveats before you copy anything. First, these figures come from Shopify's internal judge and replays of its own traffic, with no independent check. Second, the article is adapted from a talk at NVIDIA GTC Berlin 2026, the model is NVIDIA Nemotron 3.5 Lightning, and the training and serving sections lean on NVIDIA hardware and software, with some results credited to "NVIDIA's measurements". Treat the GPU material as partly co-marketing. The loop around it is the part a small team can use.

The pattern

Shopify runs four steps, in this order:

Define quality first. Product experts turn requirements into a rubric covering completeness, execution and response quality. How well two experts agree on the same conversations sets the bar. Shopify then tunes an LLM judge with the GEPA and ACE optimisers until it agrees with held-out human labels just as well.
Improve the frontier agent to set the target. An automated loop proposes one small change at a time and keeps it only if the judge score rises. The improved frontier agent becomes the score the cheap model must beat.
Choose an open model and measure serving immediately. In Shopify's words, "a model that's too slow for production isn't worth post-training."
Turn failures into training data, every night. This is the self-healing loop, and its key rule is that a frontier critic writes a hint, never the answer.

On GraphQL, the base open model scored 37.7 out of the box. One loop took it to 61.5, and after six loops and 61,000 healed examples it reached 74.5.

Why a hint? Shopify does not spell out its reasoning, so this is ours. If the critic wrote the answer, the training example would contain work the agent never did. With a hint, the agent does the rerun itself, the hint is then stripped out, and what remains is a clean conversation in which the original request leads to a correct result. That is a trajectory a small model can learn to reproduce.

When to use it and when not

Use it when:

One narrow subtask dominates your token bill, especially if it reads far more than it writes (schema lookups, long documents, big tool outputs).
The output can be checked: a query that runs, JSON that validates, a field that matches a source.
You already have production logs, because the loop feeds on real failures.
Latency matters to the user. Shopify reports its model's slowest 1% of responses finished 8x sooner than the frontier model's.

Skip it when:

You have no judge you trust. Without one, every later number is noise. Build that first; our post on automated evals and hillclimbing with a held-out test set covers it.
Volume is low. Training and hosting a model has a fixed cost, and a few hundred calls a day rarely covers it.
The task changes weekly, for example an API schema still in flux. Every change forces a fresh round of training.
Your real problem is picking the right model for the whole conversation. That is routing, which we covered in the first-message model router. This pattern sits one level lower, inside a single tool call.

How it works

Input: a sample of yesterday's logged conversations for one subtask.

Steps:

The judge scores each conversation against the rubric. Shopify sends anything short of perfect to self-healing.
A critic model reads the failed conversation and the judge's reasons, then writes a hint.
A simulated user replays the original request with that hint against the same environment. For Shopify, that means the same store.
The judge rescores the rerun. On a pass, the hint is removed, the conversation is anonymised and it joins the training set. On a fail, the critic tries again, up to three times.
Whatever still fails goes to people. Shopify's annotators correct the rest, and those corrections recalibrate the judge.

Output: a growing set of verified conversations, plus a scoreboard comparing each training run with the frontier score. Over 86 nightly runs, Shopify says self-healing repaired 59,535 GraphQL conversations, 69% of the failures it tried.

Components

Rubric with three to five criteria, each scored pass or fail with a short reason.
Human label set: 50 to 100 conversations labelled by two people, or by you twice a few days apart.
Judge prompt, accepted only once it agrees with the labels about as often as the labellers agree with each other.
Critic prompt that may only point at the problem, plus a code check that rejects hints that leak the answer.
Replay harness: a staging database or sandbox where a rerun cannot touch real customer data.
Training store: a JSONL file of anonymised conversations in chat format.
Fine-tuning service for an open model. Together AI, for example, offers LoRA and full fine-tuning and says it handles "data upload, training, hosting, and inference on a dedicated endpoint".
Serving benchmark that records median and p99 latency and cost per 1,000 calls.

Minimal implementation

# nightly_heal.py  (pseudo-code; adapt before use)
for convo in sample(logs.yesterday(task="sql_writer"), n=200):
    verdict = judge(convo, rubric)                 # {"pass": bool, "reasons": [...]}
    if verdict.pass:
        continue
    for attempt in range(3):
        hint = critic(convo, verdict.reasons,
                      rule="Point at the mistake. Never write the query.")
        if leaks_answer(hint):                     # no code, no SQL keywords, < 300 chars
            continue
        rerun = replay(convo.first_user_message, hint, env="staging")
        verdict = judge(rerun, rubric)
        if verdict.pass:
            train_set.append(anonymise(strip_hint(rerun)))
            break
    else:
        human_queue.append(convo)

# weekly
if len(train_set.new_since_last_run) > 500:
    job = finetune(base="small-open-model", data=train_set)
    report(score(job.model, holdout), score(frontier, holdout),
           latency(job.model), cost_per_1k(job.model))

In n8n, the same loop is a Schedule trigger at 2am, a Postgres node that pulls yesterday's calls, an LLM node for the judge, an IF node, a second LLM node for the critic, an HTTP Request node that replays against staging, and a Data Table node that stores passing reruns. Export the table to JSONL when you are ready to train.

Failure modes and evals

Hint leakage. Critics like to be helpful and will paste the fix. Check hints in code, and spot-read ten a week.
Judge gaming. The model learns the judge's quirks, and those drift away from what users need. Keep re-labelling a fresh sample by hand and recheck the judge's agreement against those labels each month, the way Shopify feeds human corrections back into its judge.
A held-out set that leaks. Freeze held-out conversations before the first training run and never heal them.
Reading too much into a narrow win. 74.5 against 73 is close to parity. Set parity on held-out data as your switching rule, then bank the speed and cost gains.
Tail latency. Shopify notes that another post-trained model of similar size was close at the median but more than twice as slow at p99. Measure p99, not only the median.
Privacy. Shopify anonymises conversations before training. Do the same before any log leaves your database.

Build it this weekend: one cheap specialist

Pick one subtask your agent repeats constantly: writing SQL against your own database, pulling invoice fields into JSON, or drafting Airtable formulas.

Saturday morning. Export 200 logged calls. Label 60 pass or fail with a one-line reason. Set 50 aside as a frozen held-out set.
Write the judge and run it on your labelled 60. Rewrite it until it matches you on at least 9 in 10.
Saturday afternoon. Run the heal loop on the other 140 with a frontier critic. Log your repair rate; Shopify's 69% is a useful reference point.
Sunday morning. Score your current frontier setup on the held-out 50. That is your bar.
Sunday afternoon. Fine-tune a small open model on the repaired and already-passing conversations, then score it on the same 50. Record median and p99 latency and cost per 1,000 calls.

Pass criteria: the small model matches or beats the frontier score on held-out data, its p99 latency is acceptable for your users, and the cost saving pays back the training run within a month. If it misses, keep the loop running nightly and try again once you have more repaired examples. Shopify needed six loops to get there.

Sources

Shopify Engineering, "How Shopify turns open models into task experts" (Andrew McNamara, Logan Vegna), 21 October 2026: https://shopify.engineering/open-model-task-experts
Together AI, Fine-tuning overview (docs), accessed 25 October 2026: https://docs.together.ai/docs/fine-tuning-overview
HackrLife, Tune prompts with a held-out test set: https://www.hackrlife.com/p/automated-evals-prompt-tuning
HackrLife, Route on the first message: https://www.hackrlife.com/p/first-message-model-router

From today's Daily Briefing →