Kog Bets Software Can Unlock 10x Faster LLM Inference on Existing GPUs
In this article
French startup Kog landed on the front page of Hacker News in May with a pointed claim: extremely fast single-request decoding is possible on the standard datacenter GPUs enterprises already own. The demo ran on AMD MI300X and Nvidia H200 hardware and hit a peak throughput of 3,000 per-request tokens per second (TPS) — achieved with the now open-sourced Laneformer 2B, a purpose-built model at roughly 2 billion parameters. That proof-of-concept generated 200 tangible business leads, according to CEO Gaël Delalleau.
The stakes are real for anyone building agentic workflows. Claude Code users routinely wait hours for results; Anthropic charges a price premium for Claude's Fast Mode precisely because latency has monetary value. If Kog's Kog Inference Engine (KIE) can deliver comparable acceleration on frontier-scale models, it reframes the infrastructure calculus for teams currently weighing purpose-built silicon against GPUs they already operate.
The Technical Claim and Its Current Limits
The 3,000 TPS figure and the "30x faster LLM inference" headline carry an important asterisk: the demo ran on a 2B-parameter model, not a production LLM. Kog's prospective customers, Delalleau confirmed, are not prepared to fine-tune small models — so since May the team has redirected engineering effort toward accelerating larger models. The self-imposed milestone is a first major model running at 10x speed, targeted for September 2026, at which point Delalleau plans to demonstrate customer traction ahead of a Series A raise.
The methodology is deliberately low-level. For every new GPU architecture, Kog spends several weeks to months conducting what Delalleau calls GPU engineering research on that specific hardware — reverse-engineering behaviour down to assembly language and binary code. This is not a CUDA-wrapper play; Delalleau positions KIE as closer in spirit to Stanford's Hazy Research lab than to ZML, the other French software-first inference startup that bypasses CUDA entirely for hardware-agnostic portability. The depth of per-chip work is both the technical differentiator and the binding constraint: with a team of 11, the number of supported chips is limited for the foreseeable future.
Delalleau's argument against GPU-skepticism for decoding workloads is that newer GPUs carry increasingly large memory bandwidth that existing software stacks leave underutilised. The approach, he argues, draws on solid-state physics (his training at École Polytechnique) and offensive security (he is a four-time DEFCON CTF finalist): understand the actual laws governing the hardware, then exploit them for purposes they weren't explicitly designed for.
Target Use Cases and Design Partners
Early demand concentrates in two areas. Software engineering workflows — particularly the long-running, multi-step agentic pipelines characteristic of Claude Code-style usage — represent the primary commercial target. Kog also has design partners building prompt-to-game and prompt-to-app products, where inference latency directly gates how many generation cycles a user can complete per session and thus affects revenue per user.
| Dimension | Kog (KIE) | ZML | Purpose-built silicon (e.g., Cerebras) |
|---|---|---|---|
| Hardware target | AMD MI300X, Nvidia H200 (per-chip deep research) | Hardware-agnostic, CUDA bypass | Proprietary wafer-scale chip |
| Optimisation depth | Assembly / binary level, weeks–months per GPU | Compiler abstraction layer | Silicon + firmware co-design |
| Proven throughput | 3,000 TPS (2B-parameter model) | Not stated in source | Not stated in source |
| Model size demonstrated | 2B parameters (Laneformer 2B, open-sourced) | Not stated in source | Not stated in source |
| Team scale | 11 people | Not stated in source | Not stated in source |
| Primary analogy | Hazy Research (Stanford) | Hardware-agnostic portability | Custom silicon IPO (May 2026) |
Funding and Sovereign Tailwinds
Kog's seed round was co-led by Varsity VC — whose founding partner Kamel Zeroual was Delalleau's co-founder at Stribe, a TechCrunch50 2009 participant. Additional backers include Bpifrance and the French Tech 2030 programme; Scaleway is a named infrastructure supporter. The European angle is deliberate: as the continent pushes to build independent AI capability at both the model and chip layer, a French startup promising to extract more from existing datacenter hardware without ceding ground to US or Asian silicon vendors fits a policy narrative that translates into non-dilutive support.
The September 2026 LLM milestone is what actually matters. Kog's longer-term plan to feed its per-chip methodology into agent-based orchestration pipelines — eventually automating support for more chips and models — is coherent, but entirely contingent on first proving that the physics-and-hacking approach that yielded 3,000 TPS on a 2B model survives the jump to the parameter counts enterprise inference buyers actually care about. That proof is still outstanding.