Easy Bug Beats Every AI Model; Hard Ones Fall 16-for-16

Empirical evaluations of AI coding agents tend to focus on algorithmic difficulty: can the model solve a hard problem? A month-long study by Nhu Hoang, published in Towards Data Science on August 23, 2026, inverts that framing. Across 28 blind-scored debugging experiments on three production open-source libraries, the hardest bugs were solved correctly every time, while a superficially trivial one defeated every model and workflow tested. The finding has immediate operational implications: CI going green is not sufficient evidence that an AI-generated fix is safe to merge.

The Three Bugs and Why the Difficulty Ranking Inverted

Hoang selected bugs fixed upstream in July 2026 from ky, immer, and decimal.js, choosing that window to place the fixes after the probable training cutoff for the Claude 5 model family. Each fix shipped with regression tests that were withheld from agents and used as a hidden grader—agent claims of success based on visible test results were explicitly discarded.

The immer bug required reconstructing a structural-sharing invariant inside proxy internals: reverse() and sort() reorder array elements in place, so an index-based draft lookup can return the original object directly, allowing subsequent writes to escape the proxy and mutate base state. The decimal.js bug was subtler—the obvious fix of increasing working precision passes all 22,624 visible assertions while still failing all four hidden edge-case tests near the numerical boundary. The algebraically correct solution, reformulating 1 − x² as (1 − x)(1 + x), avoids catastrophic cancellation entirely. A sibling acos fix (PR #217) already used this formulation in the same codebase; three of the eight decimal.js runs cited that prior fix by name.

What Defeated Every Model and Workflow

All 16 runs on immer and decimal.js passed the hidden regression tests. All 12 runs on ky failed the decisive one. The ky failure is precise: agents correctly identified that Ky's generic deep-merge discards a numeric retry value when merged with an object. The obvious repair normalizes {retry: 3} to {retry: {limit: 3}} before merging—passing all 84 visible retry-suite tests—but silently rewrites user JSON payloads. If a request body contains any field named retry set to a number, the naive fix injects a limit key into that user data. The maintainer's actual fix scopes the normalization to the top-level retry option only, because user payloads can contain arbitrary keys. That scoping requirement appears nowhere in the bug report and nowhere near the function being patched.

Hoang tested three model tiers—Claude Haiku 4.5, Sonnet 5, and Opus 4.8—and three workflows: a naive single agent, a structured five-step investigation requiring explicit impact enumeration before implementation, and a parallel pipeline with two independent diagnosers, an implementer, and a reviewer empowered to rewrite the patch. Every combination failed. One structured-investigation run came closest: it noticed a nested user key named retry could incorrectly gain a limit field, checked Ky's option type definitions, found no such field there, and dismissed the risk. The correct answer lived in user data space, which no codebase artifact addresses.

This failure pattern connects to a broader finding: a separate benchmark on precise code edits found frontier models passing unit tests more than 76% of the time while matching the maintainer's exact edit less than 45% of the time. The gap points to the same issue—green tests show a fix works for the cases you tested, not that the fix is correct across real usage. This is also relevant to the security implications of AI coding agents operating with insufficient contract information.

The Reviewer That Caught the Bug and Approved It Anyway

The parallel pipeline produced the experiment's most instructive single run. The reviewer agent traced the corruption path precisely, writing verbatim: "the fix keys on the string 'retry' at every nesting depth… deepMerge({json:{retry:3}}, {json:{retry:{foo:1}}}) → {json:{retry:{limit:3,foo:1}}} (user request-body corruption)." It then approved the merge. Its reasoning: the deep-merge function already couples to option names at all depths, making this a pre-existing class of problem; a colliding key is unlikely in practice; a clean fix requires a broader refactor. The fix shipped in the simulated environment as a data-corruption bug that a human reviewer rejected in the actual PR.

The failure was not detection—it was the ship decision. The reviewer correctly weighed scope, likelihood, and refactor cost, and reached the wrong outcome. Hoang's proposed structural response removes discretion from that decision entirely: any review finding that identifies a potential unintended side effect or data-corruption risk blocks the merge automatically, with no likelihood weighting. Under that rule, the one detection in twelve runs becomes a catch. This connects to a broader pattern noted in analyses of agent control architectures—judgment calls distributed across agent layers tend to diffuse accountability rather than enforce it.

Sorting Bugs by Information Availability, Not Apparent Complexity

The operational takeaway is a routing heuristic that replaces difficulty estimation: ask whether the information necessary to produce the correct fix is fully visible in the repository and the ticket, or whether correctness depends on undocumented usage contracts. When everything needed is discoverable in the codebase—invariants, algorithmic structure, sibling implementations—the study's results suggest current models are substantially more capable debuggers than benchmark reputation implies. When correctness depends on how callers actually use the code, no model tier or workflow configuration solved the problem, and every failure came with green CI.

The practical leverage points Hoang identifies are the ticket itself—every contract made explicit is information the agent cannot otherwise infer—and the review gate, which should be mechanical rather than discretionary when any side-effect risk is flagged. Green tests establish that a fix addresses the reported symptom; they do not establish that the fix is correct across the full space of real usage.