Kog Targets 10x LLM Speed With Assembly-Level GPU Tuning
In this article
French startup Kog hit the front page of Hacker News in May, demonstrating 3,000 per-request tokens per second (TPS) on standard datacenter hardware — specifically the AMD MI300X and Nvidia H200 — and walked away with 200 tangible business leads. The demo's central argument: software-layer GPU optimization still has substantial headroom that purpose-built silicon vendors and conventional serving frameworks are both leaving on the table.
What KIE Actually Does
The Kog Inference Engine (KIE) does not bypass GPU drivers the way ZML's hardware-agnostic layer sidesteps CUDA. CEO Gaël Delalleau describes KIE as operating at a deeper level, closer in spirit to Stanford's Hazy Research lab than to portability-focused middleware. The methodology involves dedicating several weeks to several months of GPU engineering research per chip — reverse-engineering hardware behavior down to assembly language and binary code to surface memory-bandwidth capacity that standard serving stacks leave idle. Delalleau argues that newer GPU generations carry increasing memory bandwidth that conventional inference runtimes fail to fully saturate, and that this gap is structural rather than incidental.
His background — solid-state physics at École Polytechnique, followed by offensive cybersecurity and four DEFCON CTF finals — shaped this constraint-inversion mindset, and it is the culture he is building into an 11-person team.
The 2B Demo and the LLM Scaling Problem
The 3,000 TPS figure came from Laneformer 2B, a purpose-built approximately 2-billion-parameter model that has since been open-sourced. Critics were right to note that jumping from a 2B model to the large frontier models enterprises actually deploy is a qualitative change in memory footprint, attention complexity, and KV-cache pressure. Delalleau does not concede the architecture is incompatible with LLMs; he calls the idea that GPUs are poorly suited for autoregressive decoding a "misconception" that newer hardware has already begun to refute.
His stated milestone: a first major model running at 10x speed, targeted for September 2026, after which he plans to demonstrate customer traction and open a Series A. The 30x headline figure Kog uses refers to per-request single-request decoding acceleration; whether that transfers proportionally to multi-turn, multi-agent workloads at LLM scale remains unproven in public benchmarks.
Agentic Workflows as the Primary Market
Early design-partner feedback pointed to software engineering as the lead use case. Claude Code users regularly absorb multi-hour latencies on complex agentic pipeline runs — workloads that stack many sequential inference calls and make per-call latency multiplicative. Anthropic already prices speed as a premium feature through Claude's Fast Mode, which signals that the market has accepted latency as a monetizable variable. A second design-partner category involves game and app generation from prompts, where reduced inference latency translates directly to conversion.
Kog also found that prospective customers are not ready to fine-tune small models — a discovery that pushed the company to prioritise accelerating large-model inference over optimising the 2B-scale architectures its demo showcased.
Competitive and Operational Tradeoffs
| Dimension | Kog (KIE) | ZML | Purpose-built silicon (e.g., Cerebras) |
|---|---|---|---|
| Optimisation layer | Deep GPU micro-architecture, assembly-level | Driver abstraction, CUDA bypass | Hardware architecture |
| Target hardware | AMD MI300X, Nvidia H200 (per-chip research) | Hardware-agnostic across competing chips | Proprietary wafers |
| Demonstrated TPS | 3,000 (Laneformer 2B, single request) | Not disclosed in source | Not disclosed in source |
| Scaling cost | Weeks–months of engineering per new GPU | Lower per-chip marginal cost | Capex-intensive |
| LLM-scale validation | Pending (September 2026 milestone) | Not disclosed in source | Available at product launch |
The per-chip research cost is the most significant operational constraint. With 11 people, Kog cannot simultaneously support many GPU SKUs, which limits its addressable market until the agent-based pipeline it envisions for automating chip support is itself production-ready. The company counts Scaleway as a supporter and holds backing from Bpifrance and the French Tech 2030 program — European sovereignty positioning at a moment when the EU is actively investing in domestic AI infrastructure. Varsity VC, co-led by Delalleau's former Stribe co-founder Kamel Zeroual, co-led the seed round; the size was not disclosed.
Whether KIE's assembly-level gains replicate at 70B-plus parameter scales will determine whether software extraction outperforms hardware acquisition in this segment of the inference market. September's planned LLM demonstration is the first hard checkpoint. If the per-request speedup holds even partially at frontier-model scale, the cost calculus for agentic infrastructure shifts meaningfully toward existing GPU fleets.