OpenAI's 2-Engineer Rust Rewrite: 6× CPU, 15× Memory Gains at 70M RPS

OpenAI's engineering blog, published September 11, 2026, details how Habitat—the company's internal online storage platform—grew from a single Python client library launched at DevDay 2023 into a distributed system handling more than 70 million requests per second, spanning more than 500 petabytes across almost 40 geographic regions for over 1 billion weekly users. The writeup is candid about deliberate sequencing: absorbing technical debt, deferring rewrites, and squeezing headroom from a Python serving stack that was always known to be temporary. For practitioners dealing with production AI failures rooted in architecture rather than model quality, this is a rare primary-source account of what that tradeoff actually costs and pays.

From library to service: why centralisation was unavoidable

Habitat's initial form—a Python client-side library wrapping Azure Cosmos DB calls—worked because its scope was narrow. Product engineers got schema routing, authorisation, encryption, and connection pooling without owning any of it. By mid-2025, that simplicity had become a liability. Rolling out a regional sharding migration required coordinating feature-flag rollouts across dozens of services. Each round-trip—routing logic, shadowing for correctness validation, a bug fix—consumed multiple days of cross-team coordination. When one team rolled back to a stale client for an unrelated reason, it triggered the exact outage the entire exercise was designed to prevent.

The fix was pulling Habitat into a standalone service behind an Envoy sidecar, with multiple Python worker processes per pod. Any platform-level change—ACL policies, rate limits, encryption primitives, circuit breakers—now takes effect immediately across every OpenAI product without client coordination.

Python at high throughput: tail latency as the primary enemy

Running Python as a high-throughput service amplifies one problem: asyncio scheduling delay. Habitat performs CPU-heavy work inline—routing, compression, encryption, checksumming, downstream health checking, request shadowing, and hedging—alongside I/O-bound request proxying. The GIL means none of that CPU work is parallel. At p99 and above, traces showed downstream Azure Cosmos DB responding quickly while requests sat unscheduled. Scheduling jitter reached hundreds of milliseconds and, in edge cases, several seconds.

Three mitigations were combined: (1) capping concurrent requests per process tightly and scaling out via many worker processes; (2) measuring delta between scheduled and actual background-task execution to track event-loop delay empirically; and (3) live CPU profiling to find discrete offenders. One culprit: Statsig feature-flag configs refreshed every 60 seconds with no jitter, causing all workers on a pod to simultaneously parse a configuration file covering every production rule across every service. Deploying a scoped config, extending the refresh interval, and adding jitter resolved the stall.

Connection pooling introduced a separate metastable failure. Python's aiohttp TCPConnector defaults to LIFO connection reuse: slower overloaded servers returned connections to the pool last, making them the most recently available—and therefore most frequently selected. This concentrated load progressively on already-degraded pods. Patching to FIFO reuse broke the loop; Istio and Envoy now handle connection pooling and load-aware balancing across OpenAI's broader infrastructure.

Why Habitat does less

Habitat's API deliberately exposes less than its underlying storage could support. Clients interact through a NoSQL API modelled on TAO's object-and-edge graph primitives rather than arbitrary SQL. Each object and its edges are colocated within a storage partition, but cross-object edge hops may span separate Azure Cosmos DB accounts in different regions—making graph traversals expensive client-side by design. Architectural specificity of this kind routinely outperforms raw compute scaling because it shapes demand before it reaches the storage tier.

For teams requiring complex queries, Habitat exposes a secondary read path via Rockset, populated via change data capture from the online store in near-real time. Each team provisions and scales its own Rockset instance, isolating analytical and search workloads from the OLTP path.

The Rust rewrite: efficiency numbers from a two-engineer migration

At peak throughput the Python service was handling more than 20 million requests per second. In Q2 2026, two engineers rewrote the entire service in Rust using Codex and GPT‑5.5. The Rust service now handles 95% of production traffic, with Python deprecation expected within weeks.

AI Mastery analysis

Habitat's evolution shows that the Python-to-Rust rewrite was not the hard part—identifying the correct API surface was. Locking clients into a NoSQL object-edge model early prevented the unbounded-query outages that plagued the prior Postgres era and made the eventual rewrite tractable because request semantics were stable. The 2-engineer Q2 2026 migration was only possible because that constrained interface had been load-bearing for years. Engineers rewriting a service with a wide or unstable API face an exponentially harder migration.

The post does not directly address one risk: the Rockset escape hatch creates a secondary scaling surface each product team owns independently, with divergent operational postures that could become a governance problem as agent workloads increase. CDC lag between the online store and Rockset is unquantified, which matters for any team treating the secondary view as near-real-time truth.

The broader pattern is that systems-engineering gains at the stack level are rivalling raw scaling investment: 6× CPU and 15× memory improvements from a language rewrite, combined with FIFO connection pooling and asyncio tuning, moved the throughput needle more than adding capacity would have in the near term. Part two of this series, covering multi-tenancy reliability and Azure Cosmos DB optimisation, will determine whether the storage-layer story is as rigorous as the service-layer account presented here.

Primary source

Rapidly scaling online storage to serve over 1 billion ChatGPT users — OpenAI Engineering