LLM Judge Approved Its Own Errors: Three Biases Explained

A production SQL pipeline ran for weeks without incident — until it didn't. One agent generated SQL from natural-language questions; a second acted as judge, deciding whether each query was safe to execute automatically or required human sign-off. When a query silently dropped an implied filter clause and returned a wrong result with full confidence, the team discovered the judge had approved it without hesitation. Re-running the same query in isolation reproduced the same approval, confirming this was a structural failure, not a stochastic one. For any team treating "LLM judge approved it" as equivalent to "it is correct," this incident is the clearest documented argument to stop.

As agentic pipelines become the default execution layer for data work, LLM judges are being inserted into increasingly consequential decision points — query execution, code review, content gating — often without the adversarial testing applied to the models they oversee. This post-mortem names specific failure modes, shows their mechanisms, and describes the fixes that worked.

The Root Cause: Self-Preference Bias

The generator agent and the judge agent were built on the same underlying model — not by design, but as an artifact of cost-driven standardisation across the pipeline. When the team substituted queries generated by a different model (comparable quality on manual inspection), the judge became measurably stricter. It flagged issues in the external model's output that it had been passing in its own model's output.

The mechanism is perplexity familiarity. LLMs tend to assign higher approval probability to text that is stylistically predictable — text that looks like their own output. Stronger models may amplify this effect because they are better at recognising their own stylistic fingerprints. The bias does not require the judge prompt to instruct leniency; it operates independently of the rubric.

Three Systematic Ways a Judge Misleads You

Self-preference bias was the incident's proximate cause, but auditing past decisions revealed two additional failure modes operating in parallel:

None of these biases individually invalidate the LLM-as-judge pattern. Collectively, they rule out treating a judge's score as an objective measurement in the way a unit test pass or fail is objective. The correct frame is that of a consistent but biased reviewer whose blind spots must be mapped before its verdicts are weighted. This parallels the verification problem discussed in frontier AI gated capability verification: confidence and correctness are not the same signal.

The Fixes, in Order of Impact

The first and highest-leverage change was routing judgment to a model from a different family than the generator. If the generator uses a GPT-family model, the judge uses Gemini-2.5-Pro, and vice versa — costing nothing beyond an additional API call to a different provider. This removes the self-preference mechanism entirely rather than attempting to average it away statistically.

That change did nothing for verbosity bias, because verbosity bias is a function of what the rubric implicitly rewards, not model lineage. A neutral judge operating against a rubric that values thoroughness will still favour verbose output. The fix required rewriting the rubric to explicitly instruct the judge to penalise unnecessary length and to include a worked example in the prompt showing a concise correct query outscoring a longer over-hedged one. Concrete prompt examples outperformed generic "be objective" instructions. For teams investing effort in automated prompt optimisation for production pipelines, this is a directly applicable pattern: the judge prompt is itself a prompt, and it should be treated with the same rigour as any other.

Calibrating Against Human Judgment

Fixing the prompt and the model routing are necessary but not sufficient without ongoing measurement against ground truth. The team pulled a sample of previously scored queries, had a schema-familiar reviewer score the same sample without seeing the judge's prior verdicts, and measured simple agreement. The result came out in the low-to-mid 80s as a percentage — flagged explicitly as a figure specific to their pipeline and task, not a transferable benchmark.

The diagnostic value was not the percentage itself but its distribution across query categories. Queries that were plausible but subtly wrong — precisely the category the original incident fell into — showed the weakest judge-human agreement. Those queries are now routed to human review by default regardless of the judge's confidence score, because the judge's confidence in that specific category had already proven unreliable.

The calibration loop also revealed a practical upside: an LLM judge running at that agreement rate is genuinely useful for bootstrapping a labelled evaluation dataset. Score a rough batch automatically, have a human review the disagreements, and build a real evaluation set from that loop rather than labelling everything by hand from scratch.

The broader lesson is that LLM judges in production pipelines need the same adversarial instrumentation applied to the models they oversee. A review step that is never itself reviewed introduces a single point of failure that is invisible precisely because it always produces a confident, well-reasoned output — whether or not that output is correct. After this incident, three things became permanent policy: judge and generator are never from the same model family, the rubric penalises verbosity explicitly, and any query category where judge-human agreement has historically run weak is routed to a human by default.