Five MLOps Assumptions That Silently Pass Failed Agent Runs
In this article
Most teams ran the MLOps-to-AgentOps migration as an addition: new OpenTelemetry GenAI spans — create_agent, invoke_agent, execute_tool, plan — layered on top of an existing monitoring stack, with nothing removed. The inherited signals kept firing. Several now report healthy on runs that failed. Gartner puts the cost of that gap at more than 40 percent of agentic AI projects canceled by the end of 2027, citing escalating costs, unclear value, and inadequate risk controls.
The failure mode is not a missing dashboard widget. It is five structural assumptions baked into every traditional MLOps monitoring layer — assumptions that hold for a stateless scoring service and silently break for a system that runs a loop.
The Five Inherited Assumptions and What They Miss
Comparable outputs. Per-call accuracy sampled once per input cannot detect run-to-run inconsistency. Tau-bench's pass^k metric makes this concrete: a single gpt-4o attempt cleared roughly 61 percent of retail tasks, but the probability that all 8 attempts at the same task succeed drops below 25 percent. A dashboard scoring one run per input reports 2.4× the reliability users actually experience.
Stateless inference. Request-level success and latency cannot see a defect that propagates through the path while returning a clean output at each step. Anthropic's own multi-agent research system documented this directly: one failing step causes agents to explore entirely different trajectories. The MAST taxonomy, built from over 1,600 traces and sorting failures into 14 modes, found that the single largest failure category was system design — errors wired into how steps connect, not into any individual step's output. Retraining cannot fix a path defect.
One decision boundary. A per-step success rate of 85 percent looks healthy on any threshold-based alert. Compounded across ten steps, 0.85^10 is roughly 20 percent end-to-end success. Per-step monitoring reports the 85; users experience the 20.
Ground truth arrives. Drift monitors watching input distributions stay silent when a one-line system-prompt edit shifts agent policy with no data change at all. The substitute teams reach for — cheap automated verifiers — compounds the problem: MAST found many existing verifiers perform only superficial checks, such as confirming code compiles rather than confirming it is correct. A ChatDev-built chess program passed every such check and scored just 25 percent on the ProgramDev benchmark.
A human sits between model and consequence. When an agent acts rather than predicts, the trace becomes the only evidence the action was correct. A documented CrewAI issue shows agents generating text that looked like a real tool call and result without the tool ever running. Native tool calling (where the system executes the action, not the model) closes that specific hole, but the deeper problem survives: a fully accurate trace of work that was done wrong still reads green.
What to Instrument Instead
| Broken assumption | Signal that encoded it | What that signal cannot see | Replacement instrument |
|---|---|---|---|
| Outputs are comparable | Per-call accuracy, one sampled run | Run-to-run inconsistency on identical input | pass^k across repeated trials |
| Inference is stateless | Request-level success and latency | Path defect returning a clean per-step output | Trajectory replay with step-level state |
| One decision boundary | Per-step success rate | Compounding failure across the whole path | Trajectory completion rate |
| Ground truth arrives | Drift against a reference window | A prompt edit that shifts policy with no data change | Versioned agent config — prompt, tools, model, parameters — diffed per run |
| A human sits in between | Single-threshold alert | An unsafe action inside a run that scored fine | Pre-action gates on every side effect |
Two additional instrumentation decisions matter beyond the table. First, cost should be denominated per successful trajectory, not per call. Multi-agent systems already consume roughly 15 times the tokens of a chat interaction, and per-call dashboards rank a 40-tool failing run as cheaper than a 12-tool successful one. Second, recursion limits are not the same as hard caps: a LangGraph thread documents an agent looping to its recursion limit while burning tokens on a deterministic tool error it had no way to reason past. One practitioner recommends flagging after 3 to 5 identical retries, not after 20.
When the Old Stack Is Still Correct
The case for trajectory instrumentation is conditional. A single model call with no tools and no memory is a stateless scoring service — every assumption above holds, and monitoring it like a model is correct. A retrieval-augmented wrapper that retrieves once and generates once, with no loop, needs prompt versioning and output evaluation; LLMOps covers both. Trajectory tooling on a two-step pipeline produces storage costs and an unread dashboard.
The boundary comes down to two questions: Does the system act without a human between the decision and its consequence? Does it produce external side effects — writes to a datastore, payments, messages sent, deploys triggered? If the answer to both is no, the existing stack plus span-level tracing holds. If the answer to either is yes, the architectural choices governing how those steps wire together matter more than the model quality inside any single step. As the MAST taxonomy shows, inter-agent misalignment — where every component works and the coordination between them fails — is the second-largest failure category in real production traces.
The tracing layer was the cheap half of this migration. The expensive half is identifying which inherited signals to stop trusting before an agent with write access takes an action that a green dashboard certified as fine. That decision belongs on the instrumentation roadmap before the agent ships, not after the first incident.