HALO: Human-Grounded Granular Evaluation of Proactive Agents, by Wenqi Pei, Xinyu Wang, Jinrui Liu, Cynthia Zhuge, Jie Sheng, Zihan Wang, Tenghao Huang and Muhao Chen. It was submitted to arXiv on 29 October 2026 and has not yet been peer reviewed. The arXiv listing does not show the authors' institutions; the team's public dataset card says the work was done in collaboration with 2077AI. The test environment is on GitHub and every graded response is free to download.

A proactive agent is an AI assistant that speaks or acts without being asked. It is the reminder that pops up before a deadline, the inbox bot that drafts a reply for you, or the background agent that reschedules a meeting on your behalf. The hard part of building one is judgement. It has to decide whether to step in at all, when to do it, and how far to go.

The researchers asked 20 experts to grade 8,000 unprompted responses from 10 AI models. When the experts rejected a response, the most common reason was that the assistant moved too early or went too far. In the paper's words, over-proactivity is "the dominant observed failure mode", with premature timing or excessive intensity accounting for 61.20% of rejected responses. Put simply, about three in every five bad interventions were over-eager ones.

Even the strongest models got it wrong often. The best proprietary model (a closed, paid model from a large lab) gave an acceptable response 72.88% of the time, roughly three times in four. The best open-source model, the kind anyone can download and run, managed 56.63%, a little over half. The team's release file counts 4,425 acceptable and 3,575 unacceptable responses across all ten models, so the average unprompted response was accepted only about 55% of the time. The same file shows 1,763 responses labelled premature, against 821 that came too late or never came at all. Early beat late by about two to one.

The team built a test world called HALO-Arena. It covers 10 everyday areas of life, with five simulated people in each, and every person has their own environment of records: structured databases and unstructured documents. That makes 50 personal environments in total.

Inside those environments sit 400 scenarios, each built around one of four kinds of need. A dependency is something that waits on something else, such as a booking that needs a confirmation first. A blockage is something stuck. A risk is something that could go wrong. An opportunity is something worth doing while there is time. Each scenario unfolds as a series of events, and at set moments, 2,160 of them in all, the agent sees what has happened so far, can search the records for evidence, and then decides what to do.

Every response was sorted onto a four-step ladder of action levels: L0 Observation (stay quiet and keep watching), L1 Attention (flag something to the person), L2 Preparation (get something ready, such as a draft) and L3 Execution (actually do it). Each of the ten models, which included systems from Anthropic, OpenAI, Google, xAI, DeepSeek, Zhipu and Alibaba's Qwen family, contributed 800 responses.

The experts then judged each response as acceptable or not at that moment. For rejected ones they recorded what went wrong on two dials. Timing could be timely, premature, belated or missing entirely. Intensity could be appropriate, insufficient or excessive. They also flagged other problems, such as "unsupported evidence", meaning the assistant claimed something the records did not show, and wrote short explanations. The team also logged how many search rounds and how much hidden reasoning each response used.

The written explanations show how fine the line is. In one civic-life scenario, an assistant's note about a volunteer roster was accepted because, with "the Friday-noon deadline a day away", it named the only trained backup and set out the next steps. A later note in the same scenario was rejected because the roster cut-off had already passed and the note "cannot advance the lost station". Same assistant, same task, different moment.

If you run a small business or build software, you probably already ship things that speak first: renewal reminders, abandoned-cart nudges, "I noticed…" messages, or an agent that tidies a calendar overnight. Each unwanted interruption spends some of the customer's attention and trust. An unwanted action can also cost money, from a refund sent too early to an email that should never have gone out.

The study gives you a sensible default. When your product is unsure, it should do less and wait a little. That matches what we argued in the proactive first message: speak first only when it matters. It also bears on self-scheduling agents, because an agent that wakes itself up needs a rule for when to stay quiet once it is awake.

It matters for buyers too. If you let an assistant act on your inbox or calendar, a fair working estimate from this test is that even the best model will misjudge the timing or the size of its help about one time in four. Real life is messier than a test, so treat that as a floor for caution.

For builders: an escalation ladder. Give every unprompted feature four rungs, mapped to the paper's levels: mention, suggest, draft, act. Start each case on the lowest rung that would help and climb only when there is evidence that more is needed.

Gate every rung on evidence. Before any unprompted message, make the agent name the record that shows the need: a date, an invoice, a message. If it cannot point to one, it stays silent. This is a direct guard against the "unsupported evidence" failure the experts flagged.

Time it to when the need is due. Work out the last sensible moment to help and aim a little before it, leaving the person time to act. The roster example shows the other edge: help that arrives after the deadline is also rejected.

A worked example. A freelancer's client is late paying. On day one, the assistant mentions it in the morning digest. On day seven, it suggests a reminder. On day fourteen, it drafts one for approval. It never sends without a click unless the freelancer has turned on automatic reminders for that client.

A checklist for any product that sends unprompted messages or actions:

A one-hour test. Pull 20 recent unprompted messages or actions from your product. Label each with its rung, its timing (early, on time, late) and whether real evidence existed. Then check which ones users ignored, dismissed or complained about. If most of the failures were early or too heavy, you are seeing the same pattern as the paper, and the fix is to move those cases down one rung and later in time. If your agent is vague about the need, borrow the habit from ask first and have it ask a question instead of acting.

For everyone using AI assistants. Set any assistant that touches your email, calendar or money to ask before it acts. Prefer "suggest" or "draft" modes over automatic ones for the first few weeks. When an assistant oversteps, tell it plainly what you wanted instead, and once a week glance through what it did on its own. A good assistant, like a good colleague, knows when to say nothing.

From today's Daily Briefing →