Infrastructure Optimization: Find the Bottleneck Before Changing Models
In this article
Performance work starts by locating the time and resources a task consumes. A model change cannot be assumed to fix storage latency, and a storage rewrite cannot be counted as an inference throughput improvement.
Correction, 22 October 2026: The earlier article treated unrelated reports as proof that infrastructure optimization outperforms model work. It also described Habitat too broadly as serving infrastructure and overstated cuDNN serialization. The reports support narrower conclusions.
Routing can preserve work already computed
AWS's benchmark evaluates prefix-aware routing against random routing with vLLM prefix caching enabled. Its long-context test reports 71–77% lower P50 TTFT; shorter ShareGPT-style workloads report 13–16%. The routing overhead was 1.3–1.9 milliseconds per request.
The distinction between the workloads belongs beside the saving. Repeated long prefixes create a different opportunity from mostly unique prompts. Test the actual distribution of prefixes, including how serialization and message construction affect reuse.
Keep time to first token separate from complete-response latency. A large prefill saving can represent a small share of a task dominated by output generation or slow tools.
Storage efficiency is a different layer
OpenAI's Habitat account describes online storage. It identifies asyncio scheduling delay, synchronized feature-flag parsing and connection-pool behavior as service problems. The Python tuning work and later Rust migration are distinct stages of that account.
OpenAI reports the Rust service as 6× more CPU efficient and 15× more memory efficient. The platform's more-than-70-million requests per second describe storage traffic; they are not model requests or generated tokens. Python had served more than 20 million requests per second at its peak.
A trace should identify which service is responsible for a delay before the result is assigned to an AI performance category. Storage, retrieval, authorization and inference may all contribute to the same user-visible response.
A cuDNN graph still needs a supported execution plan
NVIDIA's graph documentation describes operation graphs and engine selection. Fusion can avoid intermediate materialization when the chosen engine supports the graph. Describing operations together does not establish that every operation becomes one kernel.
The frontend overview distinguishes serializing a graph specification from serializing a fully built execution plan. The latter can save plan-creation work, but not all engine configurations support it. Tensor identifiers must remain consistent.
Measure cold initialization and warmed execution separately. Include graph construction, plan selection, workspace allocation and any compilation that remains on the actual path. Verify numerical behavior after changing precision or engine selection.
PAIR exposes parallel request capacity
NVIDIA's PAIR demonstration reports a five-subagent workload completing in 8 minutes 48 seconds across three devices versus 18 minutes on one laptop. It demonstrates that workload and configuration, not a general twofold acceleration for every request.
The request model routes an entire request to one eligible node. It does not split the request, pool GPU memory or move an in-flight request to another machine.
A workflow with independent tasks can benefit from more capacity. A serial dependency chain may have little work to distribute. Include network transfer and model availability when assessing either case.
Compare changes against the same accepted work
| Observed delay | Candidate change | Measurement to retain |
|---|---|---|
| Repeated prefill | Prefix reuse and routing | Hit rate, TTFT, load distribution |
| Storage tail latency | Scheduling, pooling or service changes | Per-service traces and tail latency |
| Kernel setup or intermediates | Supported fusion and plan reuse | Cold start, steady execution and numerical checks |
| Independent tasks queueing | Multi-node request routing | Completed-workflow time and failure handling |
Quantization and model substitution remain candidates when they address the measured bottleneck. Include implementation time, maintenance, quality and operating cost in the comparison. The reports above contain no common experiment that ranks those investments universally.
Our local deployment review covers the related memory and data-boundary checks before choosing the hardware configuration.
Frequently asked questions
What conditions produced SageMaker's largest reported TTFT improvement?
AWS used Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances with vLLM prefix caching. Long-context tests used 8,000-token shared prefixes and reported 71–77 percent lower P50 time to first token.
Was Habitat a model inference service?
No. OpenAI describes Habitat as an online storage platform. Its reported CPU and memory efficiency gains concern the Python-to-Rust storage-service rewrite, not faster model token generation.
Does serializing a cuDNN graph eliminate every startup compilation cost?
No. NVIDIA distinguishes graph-specification serialization from built-plan serialization, and not every engine configuration supports the latter. Validate the actual execution path.
Does PAIR combine GPU memory for one model?
No. Each request runs whole on one eligible node. Additional machines can support concurrent independent requests, rather than pooling their memory for one request.
Do these reports prove infrastructure work always beats quantization?
No. They examine different systems and metrics. Compare both options on the same workload, quality requirements and engineering budget.
Related Reading
cuDNN Graph API: Fusion, Autotuning, and Plan Reuse Explained
NVIDIA's cuDNN Frontend Graph API makes kernel fusion, engine selection, and compilation timing explicit — here's how each technique works.
OpenAI Decisions API Beta: 10x Faster Typed Answers, $0.10 Input
OpenAI's gpt-6-luna Decisions endpoint returns probabilities, choices and scores at $0.10 per million input tokens, with no output fees and a vendor-stated 10x speedup.
NVIDIA's Switchyard Routes LLM Traffic Between OpenAI and Anthropic APIs
NVIDIA open-sources Switchyard, an Apache 2.0 Rust proxy that translates and routes LLM traffic across OpenAI and Anthropic wire formats, including streams.