The paper

Authors: Xiaodong Wu, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li and Jianbing Ni. The paper states that all six authors are with Queen's University in Kingston, Ontario, Canada.

Submitted: 5 November 2026 to arXiv (2611.07650, software engineering).

The finding in plain English

A relay is a company that resells access to someone else's AI model. You get an API key from the relay, send your request to it, and it forwards the request to a model and passes the answer back with a bill. Many advertise the big models below the official price.

The Queen's team paid for access to 20 relays selling GPT-6 Astra and Claude Opus 5. Ten advertised prices below the official APIs and ten advertised the same or higher prices. They checked answers, permissions and bills.

On answers, the first round "flags 7/10 low-price and 2/10 same- or higher-price relays for behavioral differences". In practice, those relays got noticeably more answers wrong on a fixed set of knowledge questions than the official API did in the same test batch, for at least one of the two models.

On permissions, four relays returned tool calls despite tool_choice: none. A tool call is the model asking your app to run a function, such as a search or a script. Setting tool_choice to "none" tells the model not to ask. One relay returned 24 such calls.

On billing, "five relays report 4,200 to 60,000 more input tokens than the official API" for identical requests. Tokens are the word fragments you are billed for. In one example, the official API counted 36 input tokens for a question, while three relays counted 4,416 or 4,418 for the same question. The authors note that a fixed surplus like this fits a hidden system prompt added by the relay, though altered accounting would look the same from outside.

Six services let the researchers read their account balance before and after testing. At all six, deductions matched reported usage at listed prices to the cent. One of those six was also among the five with extra tokens, so at that service the extra tokens were billed. For the other four, the paper says it cannot tell.

How they did it

The team found candidates through chatbot recommendations, community lists and web searches. A relay qualified if anyone could sign up, it had public prices and an English site, and it took a card or PayPal with a minimum deposit of at most $15. A relay counted as low-price if its average price was under 95% of the official one. All requests came from the USA.

For answers, they picked knowledge questions near the edge of what each model knows, the kind of question a cheaper substitute model tends to miss. Each relay answered each question five times, and the score was compared with official answers collected in the same batch, on the same day except for part of the first batch, which was compared with the previous day's official answers.

For safety, they sent ordinary coding and setup requests and screened 3,332 responses for lookalike package names, odd web addresses, hidden shell commands and encoded content. They confirmed no injected content. They then compared token counts and balances, and repeated the whole test 11 to 18 days later on 35 of the 36 relay and model pairs.

Why it matters for a small builder

Cost. A discount is only real if the token count is honest. A rough illustration (our estimate, not the paper's): at one relay's Astra cache-read rate, listed at $0.15 per million tokens during the October 2026 tests, 4,224 extra cached tokens add about $0.0006 per request. That is about $6 per 10,000 calls, on every call, before any normal usage. On short prompts, extra tokens can be most of what you pay for.

Quality. If your product depends on the model getting facts right, a relay that scores lower on the same questions hands your customers worse answers under a familiar brand name. The model name field, which mostly matched the request, would not warn you.

Privacy and safety. A relay reads your requests in plain text. The paper did not test what relays keep or do with your data. If your agent automatically runs any tool call it receives, a call you switched off could still act. None of the tool calls in this study were run, and none appeared in the second round. The paper's advice is to enforce tool permissions inside your own app.

Caveats

The sample of 20 was not random, and the gap between price groups had p = 0.07, which misses the usual 0.05 bar for statistical significance. The authors say price alone does not predict quality. Flags were unstable: on retest, seven of 35 pairs gained or lost a flag. A lower score does not prove a cheaper model was swapped in, since an added instruction or a refusal rule can change answers too. Relays are anonymised, and the study covers English-language relays paid by card, tested from the USA.

How builders and ordinary people can use it today

The paper's recommendations for relay users come down to three checks: compare answers, audit tokens and enforce tool permissions yourself. If you buy through a reseller, run a 20-prompt check against the official API first.

The 20-prompt comparison.

  1. Pick 20 prompts from your real work: 8 typical tasks, 6 fact questions with one checkable answer (include some about recent events), 3 that must return strict JSON, and 3 that declare a tool but set tool_choice to "none".

  2. Send every prompt to the relay and the official API on the same day, with the same model name, temperature, maximum tokens and reasoning setting.

  3. Save the full raw responses, including the usage and model fields.

  4. Score facts right or wrong, JSON as valid or not, and count refusals separately.

  5. If the relay gets clearly more wrong, retest on a different day before deciding. The paper treated a 20-point accuracy gap as a large effect. On 20 prompts, our rule of thumb is that four or more extra misses deserve a retest.

The token-count audit. For each identical request, subtract the official input-token count from the relay's. For Claude-style responses, total input is input_tokens plus the two cache fields. For OpenAI-style responses, use prompt_tokens. Take the median difference. Then note your balance before and after the test, multiply each token type by its listed rate, and check the two figures agree. Price cached and uncached tokens separately, because their rates differ. A large, fixed surplus is a question to put to the relay in writing. Our earlier piece on cutting a coding agent's token bill covers where else tokens hide.

Enforce tools on your side. Reject any tool call when you have disabled tools, allow only tool names you declared, and inspect arguments, commands and destinations before running anything. Pair this with plan-scoped credentials so a stray call cannot reach much.

Decision rule. A reseller is acceptable when all four hold: answers stay within noise of the official API after a retest; any token surplus is explained, and the price after counting it still beats the official one; no disallowed tool calls appear; and you would be comfortable with a third party reading every request. If one fails, use the official API. Because flags changed within weeks, keep your 20 prompts as a fixed test set, the same idea as a held-out set for prompt tuning, and rerun it monthly.

If you are not a developer. If an app sells you a famous model for far less than the official subscription, ask it and the official app the same five questions with known answers, and compare. Do not paste private documents, client data or passwords into any service you cannot verify.

Sources

Comments

Avatar

or to participate