The paper: "From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings" by Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui, Ao Qu, Ou Zheng, Jiangbo Yu, Haris N. Koutsopoulos, Jinhua Zhao and Yuan Zhou. The authors are from MIT, Tsinghua University, UC San Diego, Tongji University, Peking University, McGill University and Northeastern University (the paper does not say which author sits where). Version 1 was posted to arXiv on 9 November 2026. The benchmark is called Ask2Decide.

Ask an AI agent to plan something, such as a delivery route, a staff roster or a week of appointments, and it will usually produce a confident answer. The trouble starts when the request leaves out a fact that changes the answer: a driver who can't work Tuesdays, a van that can't take pallets, a client who must be seen before noon.

The researchers tested exactly that situation. When every fact was handed over at the start, the agents' first plans were unworkable on 5–28% of tasks. When the agents had to ask for the missing facts themselves, their first plans were infeasible on 58–98% of tasks. "Infeasible" here means the plan breaks a real constraint and cannot be carried out.

The reason is in the questions the agents chose to ask. The benchmark hides two kinds of facts: ones that change the best plan, and look-alikes the authors call "twins", which the paper describes as items "with information about the state but none about the decision". Agents uncovered the facts that mattered only slightly more often than the irrelevant twins: 39–52% against 32–41%. Their questions landed almost as often on details that changed nothing as on details that decided the plan.

The second finding is about people. When a plan was wrong, real participants left the correcting fact out of 43% of their replies. A standard simulated user, the kind builders use to test agents, left it out of only 7%. People say "no, that won't work" and assume the reason is obvious. The authors sum it up in one line: models can solve a fully stated problem "but rarely find out what the problem is".

The team built 43 planning tasks covering routing, scheduling, staffing and other areas, starting from established optimisation problems where a solver (software that computes the best possible plan) can say exactly what the right answer is. For each task, some facts were hidden in an operator's private notes. A solver labelled each hidden fact by whether knowing it changes the best plan.

The agent's job was to question the operator, then propose a plan. The researchers scored the first plan, before any feedback, because that plan shows whether the agent asked well. They also scored the final plan after feedback.

They tested four models: GPT-6.1 Sol, Gemini 3.8 Flash, DeepSeek V4.1 Flash and GLM-5.3 Flash. Even the strongest, GPT-6.1 Sol, produced a first plan that could not be carried out on 58% of tasks when it had to ask. The authors add that "Asking more does not fix this": on one hard task set, GPT-6.1 Sol asked 11.2 questions per episode without closing the gap.

For the human side, 19 participants played the operator in 79 sessions across 22 of the tasks, answering the agent's questions and accepting or rejecting its plans. The authors then used these sessions to build a calibrated simulated user that behaves more like real people (it withheld the fix in 48% of replies).

Most of us now hand planning jobs to AI: quotes, rosters, travel, content calendars, stock orders. This paper shows the weak point is the step before planning. The agent does the maths well once the problem is fully stated. It does a poor job of finding out what the problem is.

That has three practical consequences.

First, a confident first draft tells you little. The plan looks finished, yet it may rest on facts the agent never asked about.

Second, feedback hides the problem. Plans improved after people pushed back, but the paper notes that feedback hides much of this gap. If you only judge the final result after five rounds of corrections, you will miss how much of your time the agent cost you.

Third, if you build agents, your tests are probably too kind. A simulated user that leaves out the fix in only 7% of replies behaves nothing like your customers. Real people reply "that doesn't work for us" and stop there.

The authors say plainly that "This is ongoing work and will be updated", so numbers may change. The tasks are formal optimisation problems; a messy small-business request may behave differently. Participants could not be experts in every task's domain, which the paper flags. The human study covered 22 tasks, not all 43, with 19 people. The authors also note that differing instructions about revisions limit a causal reading of one of their comparisons. Treat the ranges as a strong signal about a pattern, not a precise forecast for your agent.

The fix is the same whether you are building an agent or just using one: get the plan-changing facts on the table before any plan is made.

If you build agents: add a required-facts intake. For each kind of job your agent plans, write down the five to eight facts that most often change the answer. For a cleaning business roster, that might be:

Make the agent collect these before it drafts anything. Code the check so it cannot be skipped. Our post on making your agent confirm every detail before it acts shows how to pair intake with a read-back.

Add a "what would change my plan?" step. Before drafting, have the agent list which unknowns would change the plan and ask about those first. A prompt you can paste:

This targets the exact weakness the paper found: agents spending questions on look-alike details. It is the same instinct behind asking clarifying questions before the first build.

When a user rejects a plan, ask why. Since people withhold the fix almost half the time, treat "no" as a prompt for one follow-up: "What would need to be different for this to work?" Don't guess and redraft.

Test with a stingy simulated user. If you test with an AI playing your customer, instruct it to reject wrong plans without explaining why in roughly four replies out of ten. Score the first plan, not only the final one.

If you use an AI assistant: brief it with what it won't ask. Before asking for a plan, write two or three lines covering constraints, exceptions and deal-breakers. A worked example for booking a team offsite: "Budget A$4,000 all-in. One person uses a wheelchair. Two people can't fly before 9am. Must finish by 3pm Friday." Those four facts are the kind that change the answer, and an agent may never ask for them.

When you reject a plan, say why. "This doesn't work because Sam is off Thursdays" saves you a round of corrections. "Try again" invites another guess.

Agents that act on their own carry their own risks; our look at how proactive assistants fail by moving too soon covers the other side. Here the rule is simpler: the agent can do the maths, so your job is to make sure it knows what the problem is.

From today's Daily Briefing →