The paper
Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation by Noy Sternlicht, Simra Shahid, Peter Jansen, Daniel S. Weld, Pao Siangliulue and Tom Hope, from the Hebrew University of Jerusalem, the Allen Institute for AI, Microsoft, the University of Arizona and the University of Washington. Posted to arXiv as v1 on 1 October 2026. It is a preprint and has not yet been peer reviewed.
The finding in plain English
Teams building "AI scientist" systems often ask another AI model to grade how original the generated ideas are. This paper finds that grading is unreliable: the verdict depends heavily on how you word the question.
In one case, the same model, GPT-5.4, was shown the same pairs of ideas under two slightly different prompts. The two prompts gave different answers on at least 52.6% of the pairs, and the accuracy gap between them was 52.6 points. Under one of those wordings the model fell below chance. A change that helped one model could hurt another.
Two common fixes did little. Giving the judge related papers to read improved scores by at most 16 points, and only for some models. Raising reasoning effort from "low" to "high" was usually neutral, and the paper says some models did better with less reasoning. Finally, the cheapest plain-prompt setup beat two purpose-built novelty tools by 8 to 29 macro-F1 points, while costing up to 32 times less.
How they did it
The authors needed ground truth for "novel" and "not novel", so they mined ICLR 2026 reviews on OpenReview for explicit statements about novelty. High-novelty papers were accepted, had an average rating in the top 10% of their ICLR area, and drew positive novelty comments from a majority of reviewers with none negative. Low-novelty papers were rejected, rated in the bottom 10%, and drew negative novelty comments from a majority with none positive.
That produced two test sets:
Human-Only: 154 validated novel ideas paired against 145 low-novelty human ideas.
Human+Generated: the same 154 novel ideas paired against 154 ideas written by a simple one-pass AI idea generator, which the authors treat as less novel.
They tested six judge models: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. Each judge scored ideas one at a time (pointwise) and in pairs (pairwise). The team varied prompt wording, retrieval, reasoning effort, how verdicts were combined and how ideas were formatted. Retrieval used Allen AI's Paper-Finder pipeline to collect related work. The two dedicated tools they compared against were the Idea Novelty Checker and AI-Scientist's novelty check.
The biggest swing came from a small wording change. One prompt told the judge that reviewers had found exactly one of the two ideas novel and asked which. The other simply asked which idea was novel. On the human-only set, the gap between these two prompts for GPT-5.4 was about 7 points. On the set with AI-generated ideas, it was 52.6.
The team also built a stronger agentic judge that could search the web, restricted to arxiv.org. According to the paper, it did not beat the simpler setups. Other prompt variants removed the guardrails warning against trivial or incremental ideas, or dropped the definition of novelty entirely.
Why it matters
Many claims that one idea generator is "more creative" than another rest on an AI judge. If changing a sentence in the judging prompt moves the result by 50 points, a reported gain may reflect the judge setup more than the system being judged. The authors' recommendation is to validate judges on the kind of ideas they will actually score.
The instability is worst on AI-generated ideas, the kind these judges are built to score. Judges also tied more often on generated ideas. Claude Opus 4.5 and Sonnet 4.5 tied roughly twice as often as on human ideas, and Sonnet 4.5 tied on more than half of pairs in some setups.
The cost result is also useful. A specialised tool with more steps cost more and did not score better than a single well-chosen prompt. Our write-up of Harness 2 made a related point: an agent's results depend on the harness around a model as well as on the model itself.
Caveats
The "not novel" labels for generated ideas are assumed. The authors label the simple generator's output as less novel without checking each idea. They describe this as a weak label.
The expert check is small. One expert reviewed 30 pairs, 28 of them pairs the judge got wrong, and agreed with the labels on all 30. The authors call this a sanity check, not full validation.
The headline numbers are specific cases. The 52.6-point swing is GPT-5.4 on one prompt pair and one test set. The 32-times saving is the best case, for Claude Opus 4.6. For GPT-5.4 the paper reports roughly 6 times lower cost.
The domain is machine learning research. All ideas come from ICLR papers, so other fields are untested.
How builders and ordinary people can use it today
If you use AI to score anything subjective, such as business ideas, hooks, headlines, thumbnails or newsletter angles, you are running a small version of this experiment. Models work from statistical patterns in their input, so the wording of your question shapes the answer.
Ask two or three ways. Write two or three wordings of the same scoring prompt, for example "Which is more original?", "Which would a sceptical editor call derivative?" and "Rate each from 1 to 10 for originality." Run all of them.
Treat disagreement as "unsure". If the wordings agree, you have a stronger signal. If they split, mark the item as unsure and decide yourself or test with real readers.
Do not hint at the answer. Leave out what you expect or hope the answer is.
Test with a few known examples first. Before trusting a scorer, give it five headlines you know performed well and five that flopped. If it cannot separate them, its scores on new material mean little.
Start with a simple, cheap prompt before paying for a specialised evaluator. In this study the simplest setup beat both dedicated tools. Pay for a dedicated tool only if it beats your basic prompt on your own test set.
A 20-minute judge stability test
This is our own adaptation of the paper's approach. The thresholds are rules of thumb, not figures from the study.
Pick 10 items. Use ten past headlines or content ideas where you already know the outcome: five that did well and five that did poorly.
Write three wordings. Change only the phrasing. Use a pairwise format ("Which of these two is stronger?").
Run every pair under all three wordings. Five winners against five losers gives 25 pairs, so 75 calls. Running each pair a second time in swapped order catches position bias and brings it to 150 calls.
Score agreement. For each pair, note whether all three wordings picked the same item. Then count how often the agreed pick was the known winner.
Set your threshold. If fewer than about 80% of pairs get the same pick from all three wordings, treat the judge's single-prompt scores as "unsure". If the agreed picks match the known winners less than about 70% of the time, it is not tracking your audience.
Say you have two subject lines for Friday's issue. Ask "Which subject line is more likely to be opened?", then "Which one would a busy reader skip?", then "Score each from 1 to 10 for curiosity." If all three favour the same line, send it. If they split, the model is unsure, so run a small A/B test on part of your list. Leave out notes like "I prefer the first one". In the paper, the biggest swing came from a similar hint: telling the judge that one option had already been rated novel.
Sources
From today's Daily Briefing → DeepSeek's $0.75 Search API, 71.5% of AI Tokens Go Open-Weight, and How Reddit Lost Its Grip on ChatGPT's Answers