The pattern
More of your customers now send an AI agent to do the job for them. They ask Claude Code, Codex or a chat assistant to book the appointment, pull the report or place the order, and the agent uses your API, your command-line tool or your MCP server (the connector standard agents use to reach outside tools) on their behalf. If the agent gets the job wrong, your customer blames your product, not the agent.
On 16 November Vercel published Agents are users, so your software needs evals, a step-by-step method for checking whether agents can actually complete tasks with your product. Its opening line sets the frame: "Agents are now users of software, doing work on people's behalf." The method is an end-to-end eval: give an agent a real request, let it work through the whole task in a controlled copy of your product, then check the final result against what the user asked for. Repeat it several times, because, as the post says, "one good manual test can't tell you whether a fix worked."
The idea worth taking away is in how you score it. Measure whether the agent gets the job right every time, not whether it can get it right once.
What it means for you
Vercel's worked example is a coffee-ordering tool. In five illustrative trials, an agent succeeds three times, a 60% success rate. Two numbers come out of that. Pass@5, the chance at least one of five tries succeeds, is about 99%. Pass^5, the chance all five succeed, is about 8%. In Vercel's words, a user who orders every weekday "gets at least one wrong coffee in 92% of weeks." Both numbers describe the same agent. Most demos quote the first one; your customers live with the second.
This matters for small products because a wrong action costs real money and trust: a booking at the wrong time, an invoice to the wrong client, an order over budget. An eval tells you, before a customer does, which of your product's tasks agents fail at and why. The fix is often small and in your product, not in the model: a clearer error, a missing flag, a total shown before the final step. Setting up the first eval is a weekend of work (an estimate for a solo builder with a working product), and after that it reruns on every release.
When to use it and when not
Build one when:
Customers or partners already reach your product through agents, or you sell an API, CLI or MCP server to builders.
A wrong action costs money or trust: payments, bookings, sending messages, changing records.
You are about to change something agents depend on (a command, an endpoint, your docs) and want to know if it helps or hurts.
Support tickets say "the agent did the wrong thing".
Skip it, or use a lighter check, when no agent touches your product yet, when every agent action is reviewed by a person before it takes effect, or when a failure costs only a quick edit. For a draft email the user reads before sending, a plain success rate is enough; Vercel's post makes the same distinction.
How it works: input, step, output
Input. A task: one real request, such as "a large iced latte with whole milk, ready by 9:00, under $8 including tip", plus a starting copy of your product with fixed test data, so every run begins in the same state.
Step. The agent works through the task using your real tool, as many steps as it needs. Everything it does is recorded: messages, commands, files and the final result. Vercel calls this record a trace.
Output. A verifier, a small script you write, checks the end state against each requirement separately (right item, on time, on budget, exactly one order) and scores the trial 1 only if every check passes. You repeat each task several times per agent setup and read the scores and traces together.
Components
Part | What it is | Small-product version |
|---|---|---|
Goal | One outcome users care about, found in your logs, support tickets or roadmap | Your most common support complaint about agents |
Tasks | 3–5 concrete requests: the common case, the edge cases, and one where the right move is not to act | One folder per task with the request, seed data and a check |
Environment | Your real tool against fixed fake data, with a frozen clock if time matters | A Docker container or a test account reset before each trial |
Verifier | A script that scores each requirement separately | 30 lines of Python reading the final state |
Variants | The agent setups you compare: model, prompt, tools, skills | The two agents your customers actually use |
Metrics | Success rate, pass@k, pass^k, and steps, time and cost on successful trials | A spreadsheet with one row per trial |
Vercel's own run uses Harbor, an open-source framework from the creators of Terminal-Bench that runs agents in containers, checks results and records traces. The pass^k measure comes from the τ-bench paper, which introduced it to measure how consistently agents behave across repeated trials. We covered the general case for testing prompts against fixed examples in tune prompts with a held-out test set.
Minimal implementation for a small product
Step 1: pick the goal and pick the metric. Decide what a failure costs your user. Vercel's rule of thumb: lead with pass^k when a failure cannot be undone (an order, a payment), with pass@k when the agent can check and retry cheaply (code with tests), and with plain success rate when a person reviews the result anyway.
Step 2: write four tasks. One common request, two edge cases that trip agents up, and one impossible request where nothing fits and the right answer is to tell the user. Name your tool in each instruction so the test measures whether the agent completes the job, not whether it can find you.
Step 3: write the verifier. Score each requirement separately so a failure shows what broke. A sketch, adapted from Vercel's example:
# check.py: score one trial from the final state
import json, sys
orders = json.load(open(sys.argv[1])) # what the agent actually did
o = orders[0] if len(orders) == 1 else {}
score = {
"one_order": int(len(orders) == 1),
"right_item": int(o.get("item") == "latte" and o.get("temp") == "iced"),
"on_time": int(o.get("ready_at", "99:99") <= "09:00"),
"on_budget": int(o.get("total", 99) <= 8.00),
}
score["pass"] = int(all(score.values())) # 1 only if every check holds
print(json.dumps(score))Keep the seed data and this script where the agent cannot edit them, and make sure the agent can change the checked records only through your product. Vercel warns that agents may otherwise "reward hack", for example by writing the orders file directly.
Step 4: run each task five times per agent. Harbor automates this. Without it, a short shell loop works: reset the test data, run the agent non-interactively with the task instruction (Claude Code and Codex both have a headless mode for scripts), save the trace, run the verifier, and append one row to a CSV. Four tasks, two agents and five tries each is 40 trials.
Step 5: compute the numbers per task. For each task and agent: success rate p, pass^5 estimated as p to the power of 5, and cost per success, which is total spend including failed trials divided by successful trials. Look at each task separately; an average can hide one task that fails most of the time.
Step 6: read the traces. Scores tell you how often; traces tell you why. In Vercel's example, a failed trial shows the agent saw a $7.50 menu price and ordered without asking for the total, which came to $9.27 with tax and tip. That points to a product fix, such as a --max-total flag that refuses anything over budget, which then gets rerun against the same tasks.
Failure modes and evals
Vercel's first real runs exposed three problems with the eval itself, and each one is likely in yours:
Everything scores zero. Its first run scored 0% on every variant because the agents did not know the tool existed. Name the tool in the instruction for this test, and treat discovery (your docs, skill or MCP listing) as a separate job. Our review of Dedica covers checking whether agents can find you at all.
The agent uses the wrong door. Codex kept looking for the tool as a connected integration and never ran it in the shell. That is an access problem; fix it in your instructions or docs, not in the model.
Passes that are not passes. On the impossible request, Codex passed almost every time, but 9 of 10 passes were the agent giving up without searching. The verifier only checked that no order was placed. Vercel tightened it to confirm the agent searched first, using a command log the agent cannot write to.
Two more to watch. With five trials, 100% means "no failures seen yet", not "never fails", so add trials before you promise reliability to customers. And evals saturate: once every agent passes every task, the eval still catches regressions but stops telling you what to improve, so add a harder task each month. The same discipline applies to a second model judging your agent, covered in judge the outcome.
Build it this weekend
Read your last 30 days of support tickets and logs. Write down the one task agents most often get wrong, or the one you most fear they will.
Decide what a failure costs your user, and choose pass^5, pass@5 or success rate to lead with.
Write four tasks, including one impossible request, each with fixed seed data and a known-good solution you have run by hand.
Write the per-requirement verifier and protect it and the seed data from the agent.
Run 40 trials across the two agents your customers use. Record result, steps, time and cost for each.
Read every failed trace and three passing ones. Group failures by cause.
Ship one product fix for the biggest cause, rerun the same 40 trials, and compare against the first run. Keep both results in a dated file next to the tasks.

