95% of AI Agents Stall in Demo: Simulation-Driven Testing Fixes That

Moving AI agents from demo to production remains one of the most persistent problems in applied ML. Zhou Yu, Professor of Computer Science at Columbia University and founder of Arklex AI, presented at QCon AI a systematic methodology for simulation-driven testing that addresses why, by her account, 95% of agents stall permanently in demo phase. The core argument is architectural: single-turn static benchmarks cannot evaluate multi-turn agents whose tool calls mutate live data, and replacing them requires a purpose-built simulation layer integrated directly into CI/CD.

The compliance bottleneck Yu describes is concrete. A voice agent capable of opening a credit card — handling personalised data, recommendations, and database writes — cannot ship in finance until it can be evaluated against regulatory requirements at scale. Asking colleagues to manually chat-test an agent produces neither coverage nor repeatability: testers are not real users, their intentions skew toward technical edge cases, and every agent update invalidates previously collected evaluation data, forcing another manual round.

Synthetic User Persona Architecture

The simulation framework Yu's lab built, open-sourced as ArkSim under the Arklex AI repository, structures each simulated user from three independently parameterised components. First, a user profile encodes demographic and behavioural attributes — for a shopping agent, whether the simulated user is budget-conscious or not. Second, user goals are seeded directly from the agent's declared tool capabilities and sub-goals, ensuring simulated intent is grounded in what the agent can actually do. Third, context information supplies relevant documents and product knowledge a real user would bring into the conversation.

Permuting across these three components generates a scenario set. Each scenario maps one-to-one to one simulation run, and the full set can be pinned to a CI/CD pipeline so that any code push triggers a fresh batch of multi-turn conversations between the simulated user agent and the product agent. The simulated user also supports GUI actions — clicking on product cards, for instance — so testing covers rendered interface state, not only API responses. This architecture directly addresses the infrastructure governance challenges that gate safe agent deployment by making compliance-relevant test coverage continuous rather than episodic.

Trajectory Entropy as a Coverage Metric

Optimising the product agent for task completion is well understood. Optimising the user simulator for coverage requires different metrics. Yu's framework defines three. Tool call distribution entropy measures how broadly the simulated user invokes distinct tools across the agent's capability surface. Tool call transition entropy measures diversity in the sequences by which the simulator moves between tool calls. Trajectory distance requires that distinct simulation runs produce trajectories that are not clustered near each other — a simulator that always triggers the same return-product path would score poorly regardless of how many conversations it generates.

A fourth metric — agent failure identification rate — functions as downstream validation of the other three. Given a fixed product agent treated as a black box, the simulator variant that discovers the most distinct failure modes wins. Coverage metrics only earn their value if they correlate with catching real bugs before users do.

Realism provides a countervailing constraint. A simulator optimised purely for entropy could become an adversarial fuzzer rather than a user model. Yu addresses this with reward functions that penalise deviation from the declared user profile and goal, and by measuring distributional similarity against real production conversation samples where available.

Evaluation Dimensions and Error Taxonomy

The five error categories the framework surfaces are: failure to ask for clarification, repetition, disobeying user requests, providing false information, and lack of specific information. Task completion verification goes beyond inspecting agent responses — it queries backing databases and environment state to confirm whether an action, such as a product return, actually executed. This is the critical departure from LLM-as-judge alone: the judge cannot verify side effects without grounding in the environment.

Evolving the Golden Scenario Set from Production Logs

Simulation coverage degrades over time as user behaviour drifts — new promotions, seasonal demand shifts, and policy changes alter the real distribution. Yu's framework addresses this by continuously comparing the distribution of production conversation logs against the existing scenario set, then inducing new attribute values, user goals, and context parameters to fill observed gaps. The result is a self-updating golden scenario set that keeps the CI/CD pipeline calibrated to current user behaviour without requiring full manual reconstruction.

Cost is a practical constraint here. Running 10,000 simulated conversations per CI build would impose token costs that most teams cannot sustain. The framework's efficiency objective — minimum scenarios for maximum coverage — directly controls whether simulation-driven testing is economically viable as a gating step in continuous deployment. This connects to a broader pattern explored in pipeline architecture and systems-level optimisation, where disciplined workflow design produces gains that raw model capability cannot.

The signal for practitioners is direct: compliance-heavy verticals like finance and healthcare are not blocked on model capability but on the absence of verifiable, repeatable evaluation infrastructure. Simulation-driven testing with entropy-optimised persona generation and environment-state verification is a concrete path through that bottleneck. Teams shipping consumer-facing agents in lower-risk verticals today will face the same infrastructure demands as they expand into regulated domains — building the evaluation pipeline early is the leverage point.