Meta FAIR's RPMs Cut H200 Time 1.6× by Ranking Experiments First

Meta FAIR, in collaboration with researchers from the University of Oxford and University College London, has published arXiv:2608.13940 introducing AI Research Preference Models (RPMs) — a framework that ranks unexecuted ML experiment candidates inside an autonomous research agent loop, so GPU budget flows to whichever candidate the model judges most likely to improve on the current best result. The problem RPMs target is structural: AI research agents can generate experiment proposals faster than any hardware budget can evaluate them, making selection policy the actual bottleneck on research throughput.

As infrastructure governance constraints shape safe agent deployment, the cost of undisciplined candidate execution compounds quickly. RPMs insert a ranking layer that costs tokens instead of GPU hours.

Where the RPM Sits in the AIRA-dojo Loop

AIRA-dojo runs an evolutionary tree search: it selects the node with the highest validation score, applies one of three mutation operators (Draft, Improve, Debug), and returns the highest-scoring node at termination. Without an RPM, one child is generated per operator call and immediately executed. With an RPM, the operator is applied 15 times in parallel to produce 15 unexecuted candidates, which then compete in a pairwise knockout tournament. Only the tournament winner is trained and scored; the other 14 candidates consume token inference budget rather than GPU hours.

Each pairwise comparison is grounded in context nodes retrieved by a BFS walk of the explored tree. Every context node is shown to the judge alongside the validation score it earned — the RPM never predicts an absolute metric value, a capability the team explicitly found language models to be unreliable at.

Two Variants: Inference-Only and Agentic

Both RPM configurations use frozen pretrained LLMs with no task-specific fine-tuning. Qwen3.6-27B serves as the backbone for both the AIRA-dojo operators and the RPMs themselves, so performance differences between variants cannot be attributed to a stronger underlying model.

Inference-only RPM: An LLM-as-a-judge evaluates candidate plans, code, and search history. Its prompt was optimized using MIPROv2 from DSPy, converging on a principal-investigator rubric that tolerates fixable bugs, rewards extensibility, and penalizes redundant research directions. Offline accuracy ranges from 57.7% to 59.0% on a held-out evaluation set.

Agentic RPM: The same judge gains access to a sandboxed clone of the agent's environment, including a single H200, with tools limited to python, bash, and submit_solution. It runs small-scale pilot experiments; a feedback model then either proposes the most informative next experiment or terminates the loop. Two deliberate engineering choices: the remaining budget is reported to the agent as 2,700 seconds when the real cap is 300 seconds, preventing premature termination; and pilots are capped at 30 with a 60-second threshold. Because pilot time competes directly with the main agent's clock, the agentic RPM is applied only on Draft and Improve operator steps — Debug steps revert to random selection.

AIRS-Bench Results

Evaluation ran across 20 public text and tabular tasks on AIRS-Bench, each with a 24-hour budget on a single H200, averaged over 10 seeds.

Oracles execute every candidate and retain the best-performing result — they are theoretical ceilings, not deployable baselines. The probability that either RPM variant improves over random selection on any given run is 0.5923 (inference-only) and 0.5913 (agentic), with 95% confidence interval lower bounds of 0.5066 and 0.5018 respectively. Self-hosted inference adds 0.660 hours per run; accounting for that overhead still yields a score of 0.708 at 23.34 hours for the inference-only variant.

Two new reported state-of-the-art results: WinoGrande at 94.1% with the agentic RPM, against a prior agentic SOTA of 90.4% from AIRA₂; and SVAMP at 95.7% with the inference-only RPM, against a prior human SOTA of 94.2%.

Scaling Dials

Offline evaluation on 1,000 sibling node pairs drawn from 40 unreleased image, video, and audio tasks reveals which knobs matter. Random selection scores 50%.

Context node count is the strongest lever for the inference-only variant: judging on code alone (0 context nodes) yields 60.7%; accuracy increases monotonically through 1, 10, and 100 nodes. Reasoning budget adds a steady 2.1-point gain when the judge is allowed higher-effort chain-of-thought. Candidate pool size shows that the selection advantage over the batch mean rises from 0.017 at n=2 to 0.073 at n=15 (the oracle ceiling at that pool size), explaining the design choice to generate 15 candidates per operator call.

For the agentic variant, pilot experiment time budget is the primary dial. A 30-minute pilot achieves 78.52% offline accuracy — the configuration used in end-to-end runs. An LLM arbiter ensemble across GPT-5, Gemini 3.1 Pro, and Claude Opus 4.8 reaches 69.35% offline accuracy versus 64.66% for GPT-5 alone, though these ensembles were not tested in the full end-to-end pipeline.

The AIRA-dojo scaffold and AIRS-Bench are open source, and the Qwen3.6-27B backbone carries open weights. This is a direct instance of the broader pattern where pipeline architecture, not better models, drives AI gains: the backbone is identical across all conditions, and the selection mechanism alone accounts for the measured improvements. For teams running autonomous research loops at scale, the 1.55–1.61× speedup to equivalent quality is a concrete operational lever — fewer H200 hours before a result is reached.