Software Extraction Beats Hardware Acquisition at the AI Frontier

The dominant narrative in AI infrastructure has been that capability tracks compute — more parameters, more GPUs, more spend. Three independent developments published this month dismantle that assumption in concrete, measurable terms. Z.ai shipped frontier-level coding and security results without modifying a single weight in its 743B base model. Kog demonstrated that memory bandwidth enterprises already own is being left stranded by software inefficiency, not silicon limitation. Cactus compressed full tool-calling capability into a 45-million-parameter binary that fits in 14MB and runs on a Raspberry Pi. None of these involved a new chip, a larger model, or a bigger training budget. Together they represent something more significant than individual product launches: converging practitioner evidence that the highest-ROI frontier in AI has shifted from hardware acquisition to software extraction.

The Post-Training Proof: GLM-5.3

The cleanest demonstration of the thesis is also the most surprising in scope. Z.ai's GLM-5.3 achieved its gains without touching the 743B-parameter base that underpinned GLM-5.2. Every benchmark movement — and some of those movements are large — came from scaled post-training alone: expanded task environments, more environment types, and longer training runs against a frozen foundation.

The numbers on long-horizon coding tasks are not marginal. Terminal-Bench 3.0 jumped from 4.6 to 28.3, roughly a 6× increase. DeepSWE v1.1, which tests sustained multi-file edits across a codebase, moved from 46.2 to 66.9. Agents' Last Exam (CLI), covering cross-tool collaboration over long horizons, moved from 23.8 to 28.5. The pattern across all three benchmarks is consistent: gains compress at shorter horizons and compound at longer ones, which is precisely where agentic workloads live.

The security results were, by Z.ai's own account, unplanned. The team added vulnerability-discovery training data expecting improved single-bug reasoning. Instead, capability compounded as training scaled, and the model began constructing coherent plans across full exploitation chains. CyberGym reached 84.5%, edging past both Mythos 5 at 83.8% and GPT-5.6 Sol at 83.6%. ExploitBench more than doubled from 24.4% to 54.4%. GLM-5.3 trails Mythos 5 on ExploitGym — 105 tasks versus 181 at the 2-hour budget — but the direction of improvement, and the fact that it was achieved through post-training on a frozen base, establishes a concrete upper bound on what this approach can extract. Labs that retrain foundations to chase the same benchmark movement are paying a much higher engineering and compute cost for comparable or lesser results. Practitioners evaluating this for offensive security workflows should also consult the wider landscape of AI-assisted bug hunting on the Linux security mailing list to calibrate policy exposure.

The Bandwidth Argument: Kog

Kog's inference engine makes a structurally different but complementary argument. French CEO Gaël Delalleau's position is that successive GPU generations have shipped with increasing memory bandwidth that existing software stacks leave untapped during the decode phase of inference. Decode is memory-bandwidth-bound, not compute-bound — which is why it dominates agentic pipelines, where the model generates one token at a time, and why latency penalties compound most viciously across multi-step workflows.

Kog's methodology is deliberately low-level: reverse-engineering hardware at the assembly and binary level to exploit behaviour the chip was not explicitly designed for. Against a purpose-built 2B-parameter model called Laneformer 2B, the Kog Inference Engine produced 3,000 per-request tokens per second on AMD MI300X and Nvidia H200 hardware. The September milestone — a first major model implementation at 10× speed — is the proof point that will determine whether this methodology scales to production LLMs. Delalleau has tied that demonstration explicitly to Series A readiness. The critical caveat is real: Kog has not yet demonstrated equivalent throughput against a full-scale LLM, and a team of eleven faces a hard ceiling on how many GPU architectures it can support near-term. Each new GPU generation requires several weeks to months of dedicated engineering research before the KIE can target that hardware.

But the economic argument doesn't require Kog specifically to win. It requires the underlying claim to be correct: that stranded bandwidth exists, and software can recover it on hardware enterprises already own. Claude Code users waiting hours for multi-step completions, and Anthropic charging a price premium for Fast Mode, are market signals that inference speed has discrete dollar value independent of model capability. Kog's May Hacker News preview generated 200 concrete business leads — enough enterprise attention to anchor a seed round co-led by Varsity VC alongside Bpifrance and the French Tech 2030 program — suggesting the premise resonates with buyers before a full-LLM demonstration has occurred.

The Compression Proof: Needle 2

Needle 2 operates at the opposite end of the scale axis, and its existence makes the same point from a different direction. Cactus Compute's 45-million-parameter model ships as a single 14MB binary — weights included — runs a complete session in approximately 28MB of RAM, and achieves 500 tokens per second on a Raspberry Pi 5, 400–1,500 tokens per second on Meta Quest 3S and Apple Vision Pro, and 300–700 tokens per second on sub-$200 phones. It already runs locally inside Pebble's Index 01 app for offline voice actions. Developers evaluating best small language models on Hugging Face for on-device deployment will find those two numbers — 14MB binary, 28MB session RAM — define a meaningful new floor.

The architecture choices are instructive. Cactus replaced the feed-forward network with a Hadamard MLP, added engram key-value memory accessed via gather operations rather than matrix multiply, and trained quantization to CQ2-bit from the start rather than applying it as a post-hoc compression step. Of the 45M total parameters, 8M live in engram tables, leaving 35M matmul-active. The result is 70 MFLOPs per token — compared with 460 for LFM2.5 230M and approximately 6,000 for Apple FM. Three runtime mechanisms enforce correctness without adding memory overhead: byte-level grammar constraints that skip up to 98% of vocabulary projection on structural tokens; a contrastive retrieval head that scores tool schemas per turn and admits only the top five when more than five tools are declared; and confidence gating that returns an explicit empty call rather than hallucinating on off-topic requests.

Needle 2 trails on BFCL v4 — 42.6% versus 60.8% for LFM2.5 230M f16 — and Cactus discloses this directly, attributing it to distribution mismatch: the training corpus targets consumer device actions rather than the general and enterprise API surface BFCL v4 covers. On Seal-Tools, which better matches deployment context, Needle 2 leads both LFM2.5 230M and FunctionGemma 270M on both in-domain and out-of-distribution splits. The comparison framework most practitioners apply — leaderboard rank against cloud-scale models — is the wrong frame for the deployment context. For teams shipping regulated or offline products where inference cannot reach an external API, compute efficiency roughly two orders of magnitude below the next competitor changes the evaluation entirely.

Comparing the Three Approaches

The Strongest Case Against This Argument

The counterargument deserves a direct answer. Post-training, inference optimization, and compression are not new ideas. Flash Attention, speculative decoding, and quantization have been active research areas for years. The fact that three teams published results in the same month could reflect publication clustering rather than a genuine inflection point. More importantly, the gains described here are task-specific. GLM-5.3 trails closed frontier models on the hardest exploitation-chain benchmarks. Needle 2 trails on general API surface. Kog has not yet demonstrated its throughput claims against a production-scale LLM. A skeptic could argue that software optimization fills niches without threatening the general capability trajectory that scale produces.

That argument is partially correct and worth taking seriously. What is different now is the magnitude and the economic context. A 6× jump on Terminal-Bench 3.0 from post-training alone, on a frozen base model, is not a niche efficiency gain — it is a result that previously would have required a new model generation. The GPU supply constraint documented in Nvidia's supply chain dominance means that teams choosing between spending on compute and investing in post-training recipes are facing a real tradeoff with asymmetric returns. And the emergence of unplanned capability in GLM-5.3's security results — exploitation-chain reasoning that Z.ai did not target — suggests post-training is producing generalization, not just benchmark optimization.

What Would Have to Be True to Be Wrong

This argument breaks under three conditions. First, if frontier capability gains resume at a rate that software optimization cannot track — if GPT-6 or Gemini Ultra 3 produce general reasoning improvements large enough that task-specific post-training recipes are simply left behind, not just trailing on hard benchmarks but irrelevant to the deployment decisions practitioners actually face, then the scale narrative reasserts itself. Second, if Kog's September demonstration fails: without a live LLM running at meaningful speed multiples, the bandwidth-stranding thesis remains plausible but commercially unvalidated, and the inference optimization case rests on a thinner empirical base. Third, if the compression approach hits a hard floor below which task capability degrades non-linearly — if Needle 2's architecture proves to be near the limit of what 45M parameters can do for tool calling, rather than a proof of concept for a broader class of domain-specialized small models.

None of those conditions currently obtain. The coordinated disclosure ledger that Z.ai's models produced across 269 open-source projects — 2,436 vulnerabilities logged, 1,097 rated critical or high, with an average dormancy of 26.6 years — is evidence that post-training generalization is reaching production-grade output, not just benchmark movement. Until the falsifying conditions appear, the evidence from this month points in one direction: practitioners who treat software-layer optimization as second-order to scale are leaving substantial performance on the table, on hardware they already own.