Kog Targets 10x LLM Speed by Exploiting GPU Memory Bandwidth
In this article
French startup Kog landed on Hacker News' front page in May with a technical claim that cuts against a growing narrative in AI infrastructure: that GPUs are poorly suited for the decoding-heavy demands of agentic workloads. Rather than chasing purpose-built silicon, Kog argues that software optimisation applied at a deep hardware level can unlock substantially more inference throughput from datacenter GPUs enterprises already own — specifically the AMD MI300X and Nvidia H200 it used in its public demo. The timing is pointed; Cerebras received a warm IPO reception that same month on exactly the premise that dedicated inference chips are necessary to break the latency wall.
The case for attention comes from a concrete result. Kog's demo hit 3,000 per-request tokens per second using its Kog Inference Engine (KIE) — a figure the company frames as evidence that "extremely fast single-request decoding" is achievable on standard datacenter hardware. The company claims a 30x faster LLM inference target. The significant caveat: that demo ran on Laneformer 2B, a purpose-built model with approximately 2 billion parameters that Kog has since open-sourced. Scaling the same methodology to frontier-sized LLMs remains unproven, and Kog's stated milestone is a 10x speed result on its "first major model," which CEO Gaël Delalleau expects to demonstrate in September.
What "Deeper" Actually Means Technically
Kog's differentiation claim centres not on the inference stack above the GPU but on the GPU itself. Delalleau draws a distinction between Kog's approach and that of ZML — another French software inference company — which releases hardware-agnostic software that bypasses Nvidia's CUDA layer to support fast inference across competing chips. Kog instead positions itself closer to Stanford's Hazy Research lab, operating at a lower level of GPU abstraction. For each new GPU architecture, the team commits several weeks to months of dedicated GPU engineering research on that hardware to understand and exploit its specific characteristics.
Delalleau's own background informs this methodology. He studied solid-state physics at École Polytechnique and worked in offensive cybersecurity, reaching the finals of DEFCON's CTF tournament four times. He describes the synthesis as understanding the laws of physics as applied to GPU architecture, combined with the reverse-engineering discipline of hacking down to assembly language and binary code. The practical consequence is that this approach does not generalise cheaply — with a team of 11, the number of GPU architectures Kog can support at any given time is tightly bounded.
His core technical counter-argument to GPU sceptics: newer GPUs carry increasing memory bandwidth that current software stacks fail to exploit. For agentic pipelines, where repeated per-request latency pressure is constant, memory bandwidth utilisation becomes a primary bottleneck, not raw compute.
Market Signal and Positioning
The May tech preview generated 200 tangible business leads, according to Delalleau. Early demand analysis shifted the startup's roadmap: prospective customers were not prepared to fine-tune small models, redirecting Kog's focus toward accelerating larger models. Software engineering workflows are the primary anticipated first use case — veteran Claude Code users already encounter multi-hour waits on complex agentic tasks. Anthropic's own pricing structure reinforces the point: the company charges a price premium for Claude's Fast Mode, signalling that latency has measurable commercial value.
Kog also counts design partners in the app and game generation space, where lower inference latency translates directly to user conversion and revenue per session.
| Company | Approach | Hardware scope | Abstraction level | Demonstrated result |
|---|---|---|---|---|
| Kog | Deep per-GPU hardware research, KIE software engine | AMD MI300X, Nvidia H200 (others require dedicated research cycles) | Below CUDA; assembly / binary level | 3,000 TPS single-request on Laneformer 2B |
| ZML | Hardware-agnostic CUDA bypass | Multiple competing chips via unified layer | Above GPU microarchitecture | Not specified in source |
| Cerebras | Purpose-built inference silicon | Proprietary wafer-scale chip | Custom hardware | Not specified in source |
Funding and the September Proof Point
Kog's seed round was co-led by Varsity VC, the firm of Kamel Zeroual — Delalleau's former co-founder from Stribe, his TechCrunch50 2009 company. Additional backing comes from France's Bpifrance and the French Tech 2030 program, with Scaleway as a named supporter. The company's path to a Series A is explicitly conditioned on the September milestone: Delalleau stated directly that demonstrating 10x speed on a major LLM is the prerequisite for showing customer traction and entering fundraising discussions.
The longer-term roadmap involves feeding Kog's per-GPU methodology into agent-based pipelines that could automate some of the chip-specific research work, eventually expanding the number of supported architectures. European AI sovereignty ambitions — particularly around models and hardware independent of US supply chains — represent a potential structural tailwind for a French inference startup with this profile. Whether September's result validates the approach at LLM scale will determine whether Kog's argument against the accelerator-diversification thesis carries weight beyond a compelling demo.