ALTK-Evolve: Agent Memory Gains Depend on Model Tier, Not Just Size
In this article
IBM Research's ALTK-Evolve team, publishing through Hugging Face on August 18, 2026, has released a calibration study for agent memory — tested across eight models ranging from a 30B dense model to frontier proprietary systems on AppWorld, a 585-task multi-step benchmark spanning 9 simulated apps. The core finding cuts against the intuitive assumption that more memory is always better: injecting a full self-mined guideline set helped some models substantially, helped others less than selective retrieval, and produced zero measurable gain for at least one model already near its performance ceiling.
For engineers sizing agent control architectures and their associated inference budgets, the practical upshot is that memory configuration is a per-model decision, not a system-wide toggle.
How ALTK-Evolve Builds Agentic Memory Without Weight Updates
The memory mechanism operates entirely outside the model. The agent runs tasks, produces trajectories, and ALTK-Evolve extracts behavioral guidelines — successful strategies, failure patterns, and edge-case lessons — from both successful and unsuccessful runs. Those guidelines are consolidated into a reusable set mined once from AppWorld's training split, with no test-split data touching guideline construction and no model weights updated at any point. At inference time, the agent receives either the complete guideline set on every ReAct step, or a fixed high-confidence core plus a small variable portion retrieved by cosine similarity for each specific task.
Because weights never change, the same pipeline ran portably across all eight evaluated models.
Three Capacity Patterns Across Eight Models
AppWorld scores tasks on two metrics: Task Goal Completion (TGC), the share of individual tasks fully completed, and Scenario Goal Completion (SGC), an all-or-nothing bar requiring the agent to pass every variant within a scenario bundle. SGC is the stricter signal — an agent that handles most variants but fails one scores on TGC but not SGC.
Across the sweep, three patterns emerged: models with remaining headroom that absorbed the full guideline set, weaker models that performed better with selective retrieval, and models that showed no gain regardless of configuration.
| Model | Pattern | Baseline TGC / SGC | Best-memory TGC / SGC | Best Config | Δ TGC | Δ SGC |
|---|---|---|---|---|---|---|
| gpt-oss-120b (117B MoE) | Weak / selective | 39.9 / 21.4 | 56.0 / 37.5 | Curated retrieval | +16.1pp | +16.1pp |
| DeepSeek-V3.2 (671B MoE) | Strong w/ headroom | 79.8 / 64.3 | 89.3 / 80.4 | Full guideline set | +9.5pp | +16.1pp |
| Claude Opus 4.6 | Strong w/ headroom | 90.5 / 87.5 | 94.6 / 94.6 | Full guideline set | +4.1pp | +7.1pp |
| GPT-5.5 | Strong (near-ceiling) | 92.3 / 82.1 | 95.2 / 89.3 | Full guideline set | +2.9pp | +7.2pp |
| GLM-5 (745B MoE) | Saturated | 87.5 / 80.4 | 87.5 / 80.4 | Full guideline set | 0.0pp | 0.0pp |
The SGC column is the more informative read: DeepSeek's SGC gain of +16.1pp outpaces its +9.5pp TGC gain, confirming that guidelines particularly help an agent clear every variant of a scenario rather than merely improving average-case performance. Even GPT-5.5 and Opus 4.6, both above 90% TGC at baseline, still gained +7.2pp and +7.1pp SGC respectively, indicating memory keeps returning value as long as any addressable failure mode remains.
Token Overhead and the Prompt-Caching Lever
Full guideline injection re-sends the entire set on every ReAct step, which compounds across multi-step tasks. The measured overhead for DeepSeek-V3.2 on AppWorld was 148K tokens per task at baseline versus 263K with the full guideline set — a +78% increase. For gpt-oss-120b, full injection cost 110K versus 166K tokens, a +51% increase. Curated retrieval for the same model added only 5% overhead (110K to 116K), while delivering the top accuracy result of +16.1pp TGC. That combination — best accuracy at lowest cost — holds specifically for weaker models where selection outperforms injection.
For strong models running the full guideline set, prompt caching is the primary cost control: because the static guideline-set prefix is identical across ReAct steps, it is a natural cache target. DeepSeek also ran approximately 18–19 ReAct steps with and without memory, confirming that token overhead comes from input inflation, not longer reasoning trajectories.
The team also hypothesizes that context-window size independently influences which delivery mode works — models with larger windows may absorb full injection more effectively, while smaller-context models benefit from retrieval keeping injected content compact — but notes this has not yet been isolated in controlled experiments.
What Remains Open
The current retrieval mechanism ranks guidelines by cosine similarity, which the authors explicitly flag as an imperfect predictor of which guidelines will actually help a given task. A selector trained on outcome signal is the logical next step. Two additional open problems: self-distillation lacks signal below a minimum capability threshold, making teacher-distilled memory for very weak models a separate research track; and all results come from AppWorld alone, with broader benchmarks and real-world deployments listed as in progress.
For teams deciding how much memory to budget into production agent deployments, ALTK-Evolve's results make the sizing question empirically tractable: run the baseline, measure residual failure modes, and let observed headroom — not parameter count alone — determine whether full injection, selective retrieval, or no memory addition is the right configuration. The gains are annotation-free and weight-update-free, which substantially lowers the cost of running that calibration experiment.