ByteDance's HarnessDev: Only 34 of 64 LLM Harness Changes Generalize

Benchmarks typically freeze the execution harness and measure what a model produces inside it. The Terminal-Bench 2.1 leaderboard already shows why that choice matters: GPT-5 solves 35.2% of tasks inside Terminus 2 but 49.6% inside Codex CLI using identical weights — a 14-point swing attributable entirely to harness architecture. HarnessDev, from researchers at ByteDance Seed, Singapore University of Technology and Design, Georgia Institute of Technology, M-A-P, and TokenWave.AI, inverts the evaluation target: the artifact under judgment is the runnable harness the model writes, not the answer the harness eventually returns.

For engineers building autonomous coding agents, this distinction is consequential. As covered in production AI failures rooted in architecture rather than model intelligence, the scaffolding around a model frequently determines deployed performance more than the weights do.

Two-stage evaluation: Creation and Evolution

Every creator LLM begins from a deliberately crippled seed — passive file, search, and process primitives plus result and trajectory writers, with no execution loop, planner, verifier, retry logic, or stopping rule. Unmodified, it scores 0 across all five benchmarks. In the Creation stage, each model receives a task-family specification, a short design tutorial, and one to three development cases, then builds a complete harness from that seed. The harness is frozen before hidden tasks run. Coverage spans 2,207 instances across SWE-bench Pro public split (731), Terminal-Bench 2.1 (89), MLE-bench (75), EQ-Bench3 (46), and BrowseComp (1,266). Each creator produces three harnesses per benchmark, reported as avg@3.

In the Evolution stage, each creator begins from its own frozen Creation harness and revises it using execution feedback from 100 SWE-bench Pro tasks and all 89 Terminal-Bench 2.1 tasks. Each official candidate completes both evaluations as a pair, with a budget of 10 pairs and at most 2 five-task probes between pairs. All 73 official versions are later scored on 630 held-out SWE-Pro instances the creator never observed during development.

Six creator LLMs were tested — Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro — working inside Claude Code 2.1.177, except GPT-5.5 which used Codex 0.144.3.

Creation results

Under Self-Eval, Opus 4.8 posts the highest average score at 67.8 against a human-engineered reference of 86.2. The gap is domain-dependent: Opus 4.8 reaches 69.3 on SWE-Pro versus an 80.0 reference; Gemini 3.1 Pro leads Terminal-Bench at 68.8 versus 88.8; and search shows the widest gap — 52.6 (GPT-5.5 on BrowseComp) against 92.2. Writing is the one area where models exceed the reference: Opus 4.8 scores 84.6 on EQ-Bench3 against an 83.7 reference, and both Opus 4.8 (32.9) and Gemini (32.4) beat the 24.0 MLE-bench reference.

Code volume predicted nothing. The 18 code harnesses added 17,111 net lines total, yet Gemini added the fewest (1,006 lines) and led Terminal-Bench. Self-test count barely correlated with score (Spearman 0.13 to 0.26), while revision call count reached 0.57. Of 108 code component instances, 72 trigger in real runs and 18 never fire — all from the state and memory category. Eleven of 18 harnesses define a State class, yet no checkpoint event appears across 26,679 trajectories. Dead code is also pervasive in writing harnesses: 124 of 587 writing features never execute.

Executor portability and Evolution reliability

Swapping the executor to a fixed Gemini 3.1 Pro reshuffled rankings significantly. Qwen 3.7 Max gained 17.6 points on BrowseComp and 12.9 on MLE-bench, signalling its own executor had been the ceiling. Opus 4.8's SWE-Pro score collapsed from 69.3 to 33.0 because one harness hard-coded a 120-step limit tuned for its original executor; that harness's duplicate-query rate jumped from 10.1% to 88.2% after the switch — a brittle coupling no internal test detected.

Token efficiency varied dramatically: GPT-5.5 achieved a 19.1 medal rate on MLE-bench using 29.3M tokens, while DeepSeek V4 Pro reached 19.6 with 208.4M tokens — roughly a 7-fold difference in token consumption for equivalent output.

Evolution results were similarly fragile. All five self-runtime lineages improved on held-out tasks, gaining between +1.43 and +4.44 points (mean +3.11). Under a fixed Gemini executor, only Opus improved; GPT-5.5 regressed 10.32 points. Of 64 adjacent version switches, 8 regressed on both benchmarks, 16 regressed on one, 27 gained only within the ±4.75 pair-score noise band, and 2 showed clear positive signal. Feedback and held-out scores moved in the same direction only 34 of 64 times (53.1%), and only 2 of 9 declared final versions were held-out optimal. Of 169 new functions or classes introduced during Evolution, 25 have no caller.

AI Mastery analysis

HarnessDev's 53.1% feedback-to-held-out alignment is the most operationally important number in the paper. When a model interprets its own execution feedback and decides a change is an improvement, it is wrong nearly half the time — a failure rate unacceptable in any human-managed CI pipeline. The benchmark also surfaces a subtler problem: models capable of writing syntactically complete harness machinery (State classes, checkpoint hooks, memory buffers) are systematically failing to wire that machinery into live execution paths. This is not a reasoning failure about what to build; it is a failure to verify that what was built actually runs. The dedicated trajectory analysis interface was called only twice across all Evolution runs, indicating models are not using available observability tools to close that gap.

This connects directly to architectural governance concerns around safe agent deployment: when a model's self-assessment of its own harness changes is unreliable, any auto-iterating agent architecture that treats execution feedback as a trustworthy training signal will accumulate silent regressions. The one clear win — Opus 4.8 detecting that 99 of 100 runs reported success while only 48 actually passed, then adding a completion gate — illustrates the diagnostic ceiling: models can fix problems they notice in aggregate statistics, but systematic trajectory analysis is not yet an emergent behaviour. For practitioners designing agent control layers, the implication is that harness Evolution loops require external verification checkpoints, not model-interpreted feedback alone.

HarnessDev establishes that LLMs can close a meaningful fraction of the gap to human-engineered harnesses in Creation, but the Evolution stage reveals that self-directed improvement under execution feedback is barely above coin-flip reliable at the held-out task level. The benchmark's 630-instance held-out evaluation set and its decomposition of dead-code rates and executor portability give teams a reproducible framework for measuring harness quality independently of task-answer accuracy.

Primary source

HarnessDev: Can LLMs Engineer Their Own Agent Harness? — MarkTechPost