LLM Evaluation
5 pieces on LLM Evaluation.
News & Analysis
ByteDance's HarnessDev: Only 34 of 64 LLM Harness Changes Generalize
HarnessDev benchmarks the harness LLMs build, not the answers they return. Execution feedback matched held-out results only 53.1% of the time.
40% of Production AI Failures Are Silent: Fix With a Watchdog
Datadog data shows 40% of production AI failures return HTTP 200. A lightweight 1B-parameter watchdog at each handoff seam catches what output-level eval cannot.
Risk-Scored Routing Cuts Human Review to High-Signal Queries Only
A text-to-SQL team replaced blanket approval gates with a four-signal risk router, sending only genuinely ambiguous actions to human reviewers.
LLM Judge Approved Its Own Errors: Three Biases Explained
A production SQL pipeline approved wrong queries for weeks. The judge and generator shared the same model — and that structural flaw caused the incident.
Stop Hand-Tuning Prompts: A Production Workflow for Automated LLM Optimization
Move beyond trial-and-error prompting with a measurable workflow for evaluating and optimizing LLM prompts in production systems.