OpenAI's Jalapeño ASIC Beats Nvidia GB200/GB300 on Latency and Throughput

OpenAI published benchmark results on August 25, 2026, showing its Jalapeño ASIC outperforming Nvidia's GB200 and GB300 superchips on inference throughput and latency simultaneously — a combination that hardware vice president Richard Ho described as unusual, given that production AI systems typically must trade one for the other.

What Jalapeño Is

Jalapeño is an Application-Specific Integrated Circuit built in partnership with Broadcom. OpenAI first disclosed the chip in June 2026, and it is designed exclusively for AI inference — running trained models to complete tasks or drive agents — rather than training. The ASIC approach trades GPU programmability for fixed-function efficiency optimised for the arithmetic patterns inference demands, which is what enables the efficiency-per-watt and latency figures OpenAI is now claiming.

Benchmark Methodology and Claimed Metrics

OpenAI evaluated Jalapeño against the best results recorded on InferenceX, a third-party inference benchmarking platform, at the time of testing. The reference systems used Nvidia GB200 or GB300 superchips. Three models served as workloads: GPT-OSS 120B, DeepSeek R1, and Kimi K2.5 1T.

The latency metric InferenceX uses is time between tokens (TBT) — the interval between successive tokens delivered to the user. A 1.7× to 3.6× reduction in TBT across three architecturally distinct models, ranging from a dense 120B to a 1T mixture-of-experts, suggests the gains are not narrowly tuned to a single sparsity pattern. Ho framed the dual improvement as enabling faster user-facing responses, more responsive agents, and more reliable throughput at scale. The relevance to pipeline architecture and inference-layer optimisation is direct: shaving TBT at the silicon level compounds across every layer above it.

Deployment Timeline

OpenAI plans to deploy Jalapeño in small volumes before the end of 2026, with a volume ramp continuing into 2027. The company did not disclose target unit counts for either phase. Ho was explicit that Jalapeño will not replace OpenAI's existing chip lineup; Nvidia remains a stated partner, and the broader compute strategy retains GPU infrastructure alongside the new ASIC. OpenAI also confirmed that second- and third-generation iterations of Jalapeño are already in development.

What the Latency-Throughput Combination Means

The conventional inference hardware trade-off — batching more requests to improve throughput while accepting higher per-token latency, or prioritising low latency at the cost of utilisation — has shaped how inference providers price and architect their serving stacks. If Jalapeño genuinely breaks that trade-off at the scale claimed, it changes the economics for agentic workloads in particular, where low TBT directly affects multi-step task completion time rather than just perceived responsiveness. That has downstream implications for systems engineering approaches to AI stack gains, since latency budgets at the hardware layer propagate upward through orchestration, context management, and API pricing.

Custom silicon from a frontier lab has moved from roadmap speculation to benchmarked, deployed hardware — putting measurable pressure on Nvidia's inference dominance even as OpenAI continues to rely on Nvidia GPUs for the bulk of its compute. Two further Jalapeño generations already in development indicate this is a long-term infrastructure hedge, not a one-cycle experiment.