NEEDLE Benchmark Rebuilds Search Queries Every Hour to Block Label Leakage

Static benchmarks have a structural vulnerability when the system under test is a web search agent: the agent's own fetch capability can retrieve a publicly committed answer key before retrieval even begins. Even without that exploit, parametric memorisation means a model scoring well on a frozen query set may never have performed live retrieval at all. Keenable AI's response is NEEDLE — News, Everyday, Expert, Deep-tail, and Legal Evaluation — an MIT-licensed benchmark that eliminates the fixed target by regenerating its query set continuously from live public sources. News queries rebuild hourly from roughly 124 curated RSS feeds and Google Trends signals across 52 US geographies; finance, scholar, deep-tail, and legal queries rebuild daily from SEC XBRL, Wikidata, GLEIF, arXiv, Europe PMC, CourtListener, and public agent trajectory logs. There is no committed gold-label file to fetch or memorise.

This sits directly inside the problem the industry has not solved cleanly: as the frontier AI gated capability verification crisis illustrates, measuring what a capable agent actually does in deployment is fundamentally different from measuring what it scores on a curated test set.

Query Construction and Leak Prevention

Each vertical runs source material through a three-stage pipeline before a query reaches an engine. A source document — a trending topic, a 10-Q filing entry, a CourtListener opinion — is fed to an LLM that projects it into a plausible keyword query while refusing to produce evergreen content. The generated query then passes a machine leak-check verifying the answer is not restated in the query text. Legal queries are additionally rejected if the query leaks the citation being sought; the gold label is the reporter citation or docket with party tokens.

Scholar queries demonstrate how far the decomposition goes: one paper generates four distinct query styles — a degraded title, a full-text-only detail, a natural-language clue, and a hedged tip-of-the-tongue description. Scoring uses recall@10 by arXiv ID, DOI, or PMID. Deep-tail queries are drawn from public agent trajectory releases including DeepResearchGym, OpenResearcher, and LRAT, retained only when a BERT WordPiece tokenizer either splits a word into five or more subwords or returns the unknown token — a proxy for genuine lexical rarity approximating real agentic traffic.

The harness is a Python CLI installed with uv sync, driven by generate and run subcommands per benchmark. It requires an OpenRouter key for judging and one API key per search engine under test. Runs execute in public GitHub Actions and per-run artifacts are published to a Hugging Face dataset.

Scoring and the Ultimate Ceiling

News and deep-tail use an LLM judge rating each result 0–4, with a duplicate-URL penalty; the harness reports nDCG@5. Finance reports answer-recall@5. Scholar and legal are known-item tasks scored by identifier match.

The analytically distinct construct is the ultimate ceiling. For every query, NEEDLE pools results from all 15 engines into a synthetic oracle ranking scored identically to any individual engine — an empirical upper bound on what the entire field retrieved. A large engine-to-ultimate gap indicates a ranking or surfacing failure; a low ultimate score indicates that even pooling the entire market turned up little relevant evidence, distinguishing a ranking problem from a retrieval gap shared by the whole industry.

Published 7-day means for the window ending 2026-08-28 20:30 UTC:

Finance is the closest to solved, with the field clustered between 0.771 and 0.910 against a 0.965 ceiling. Deep-tail is the hardest: the leading engine reaches only 0.557 of what the whole field collectively found, and Bing reaches 0.199 — a result reflecting genuine retrieval failure on rare-entity queries drawn from actual agent logs.

Latency as a First-Class Metric

Because search agents call retrieval tools dozens of times per task, per-call latency compounds directly into total task latency. NEEDLE issues one request at a time with no concurrent load. Over the same 7-day window, Keenable-realtime recorded 193 ms p50 and 284 ms p95. Exa-instant came in at 451 ms p50 and 684 ms p95. Bing measured 2,767 ms p50 and 9,381 ms p95; Tavily measured 1,723 ms p50 and 3,326 ms p95. Failed calls carry no latency sample, meaning published figures represent only successful retrievals — a methodological choice that could understate effective latency for engines with meaningful failure rates.

NEEDLE also publishes an overlap analysis across engine pairs, flagging queries where two engines share a below-relevance URL or three or more URLs. Brave and You.com showed an 89.1% shared-query flag rate; Brave and Parallel 76.2%; Parallel and You.com 72.4% — consistent with shared upstream index dependencies rather than independent retrieval. Google and Tavily overlapped on 48.1% of comparable queries.

NEEDLE's structure reflects a recognition — consistent with the case that pipeline architecture, not better models, is driving AI gains in 2026 — that evaluation infrastructure is itself a systems engineering problem. The benchmark's ability to distinguish a ranking failure from a market-wide retrieval gap gives RAG and web agent developers a qualitatively different instrument than any static leaderboard can provide.