Five Context Failure Modes That a Model Upgrade Cannot Fix

Production AI systems routinely fail not because their underlying models are weak, but because the surrounding context infrastructure is absent. Ricardo Ferreira, Principal Developer Advocate at Redis with more than 25 years of distributed systems experience, delivered a detailed post-mortem at QCon AI on building "My Jarvis" — an Alexa skill backed by OpenAI and the Redis Agent Memory Server (AMS) open-source project. The presentation is a practitioner-level accounting of every context failure mode he hit, how he resolved each one architecturally, and the cost consequences that followed. This maps directly to the broader argument that pipeline architecture, not better models, drives AI gains.

The Context Failure Taxonomy

Ferreira's first instinct when the skill began producing incoherent answers was to upgrade his OpenAI model — at higher cost. It changed nothing. He ultimately categorised the failure modes into five distinct types: context poisoning (nonsensical early responses), context distraction (excess information degrading answer quality), context confusion (irrelevant data mixing), context rot (quality degrading as conversation length grows), and context clash (conflicting versioned data producing contradictory outputs). Each required a separate architectural intervention rather than a model swap — a finding that aligns with the case for architectural specificity outperforming GPU scaling.

The Memory Architecture

The core data layer is Redis AMS, a thin wrapper over Redis that stores short-term memories as session-bounded, TTL-keyed JSON structures and promotes interactions to long-term memory (LTM) using a background LLM process. Ferreira set session TTL to five minutes. When TTL expiry caused the system to lose all conversational context, he added LTM retrieval via a ContentRetriever implementation in LangChain4j that issues HNSW-indexed vector searches against Redis, returning a configurable top-k result set per query.

Multi-tenancy surfaced immediately once his wife and teenage son began using the skill: without per-user post-filtering on vector search results by owner ID, memories from one family member contaminated answers for another. He resolved this with vector search post-filtering on metadata — a step that must be designed in from the start.

A third memory tier — knowledge bases — emerged when static household information (door codes, device instructions) needed to be retrievable without being tied to any user's personal memory. This required a queryRouter that calls the LLM inline to classify each incoming question and dispatch it to either the user-memory retriever or the knowledge-base retriever before the primary LLM call fires.

The full pre-flight pipeline Ferreira assembled, in execution order:

The reranking stage required manual calibration: Ferreira settled on an 80% minimum Cohere score after iterative tuning and explicitly warned that changing the underlying model resets that calibration entirely. His original intention was to use an ONNX-format ms-marco-MiniLM-L-6 model for local reranking, but the resulting JAR file exceeded both the AWS Lambda direct-upload limit of 50 MB and the S3-backed limit of 250 MB, forcing a switch to the Cohere API, which kept the JAR under 100 MB.

The Cost Problem: Linear Expectation, Exponential Reality

At three-user household scale — averaging 10 queries per user per day — Ferreira's LLM cost was a flat $4.20 and Cohere reranking cost was under $1.00. Those numbers look trivial. The danger is in the growth curve.

His early WorkingMemoryChat implementation, built to fix a tool-calling infinite loop in LangChain4j's built-in buffer, was a simple pass-through that allowed the session context JSON to grow without bound. Because the LLM makes bidirectional tool calls that each append to the context payload, token consumption grew exponentially with conversation length rather than linearly. Extrapolated to any meaningful user base, the cost profile becomes unmanageable.

The architectural fix — TokenWindowChatMemory capped at 4,096 tokens for GPT-3.5 — enforces a hard ceiling, but it demands a model-specific token count estimator because OpenAI and Anthropic parse tool-call payloads differently, producing different token counts from identical conversations. This cost dynamic is the production reality that automating LLM prompt optimisation tools alone cannot address; session memory architecture must be bounded at the design level.

The Broader Signal

Ferreira's account confirms that context engineering is not a refinement of prompt engineering — it is a separate systems discipline requiring explicit state management, retrieval pipelines, reranking calibration, and token budget enforcement. Every fix he applied added at least one additional LLM call to the pre-flight pipeline, which means context engineering trades inference cost for response quality in a way teams must budget for deliberately. The pattern echoes the infrastructure governance argument for safe agent deployment: the reliability of an AI system is determined by the scaffolding around the model, not the model itself.