LLM Evaluation
9 pieces on LLM Evaluation, including 1 step-by-step guide.
Guides
News & Analysis
Testing an LLM Editor: Planted Errors and Preservation Checks
A runnable preservation checker and a protocol for testing whether an LLM editor corrects planted errors while preserving code and citations.
Sakana AI's LLM Review System Catches 73% of Core-Claim Errors
Sakana AI's MLR system caught 73.43% of planted core-claim errors in research papers—about 5x the best baseline—but real retractions and prompt injection remain hard.
AI Portability: What Changes When the Evaluation Setup Changes
SWE-2, PC-ALM, FLM and HarnessDev show why benchmark transfer needs direct testing. Their results establish different kinds of evidence.
ByteDance's HarnessDev: Only 34 of 64 LLM Harness Changes Generalize
HarnessDev benchmarks the harness LLMs build, not the answers they return. Execution feedback matched held-out results only 53.1% of the time.
Intermediate Agent Checks: A Watchdog Pattern and Its Limits
A handoff evaluator can check intermediate payloads, but its threshold needs validation. Datadog does not establish a 40% silent-failure rate.
Risk-Scored Routing Cuts Human Review to High-Signal Queries Only
A text-to-SQL team replaced blanket approval gates with a four-signal risk router, sending only genuinely ambiguous actions to human reviewers.
LLM Judge Approved Its Own Errors: Three Biases Explained
A production SQL pipeline approved wrong queries for weeks. The judge and generator shared the same model — and that structural flaw caused the incident.
Stop Hand-Tuning Prompts: A Production Workflow for Automated LLM Optimization
Move beyond trial-and-error prompting with a measurable workflow for evaluating and optimizing LLM prompts in production systems.