SageMaker HyperPod Inference Gateway Cuts TTFT P95 by Up to 98%

AWS shipped SageMaker HyperPod Inference Gateway on September 18, 2026, targeting a failure mode that costs GPU-heavy teams real money: Kubernetes round-robin routing has no visibility into KV cache saturation, queue depth, or LoRA adapter residency, so it distributes requests without regard for backend state. The practical result is first-token latency that spikes past 4 seconds during traffic bursts while adjacent pods sit underloaded. This is the class of problem that production AI systems routinely fail on at the architecture layer, not the model layer, and HyperPod Inference Gateway is a direct infrastructure answer to it.

Architecture: two tiers, one Kubernetes-native addon

The gateway deploys as a single EKS managed addon (amazon-sagemaker-hyperpod-inference, version v2.0.0-eksbuild.1) with no sidecars, no service mesh, and no application code changes required. Its Tier 1 per-cluster design rests on three components built atop the open-source Gateway API Inference Extension.

Envoy Gateway terminates incoming HTTPS traffic and exposes a single private endpoint per cluster. Body-Based Router (BBR) inspects each OpenAI-compatible request body, extracts the model field, and dispatches to the correct model pool — enabling one gateway to serve multiple models simultaneously. Endpoint Picker (EPP) is where the routing intelligence lives. EPP consumes real-time Prometheus metrics from every model-serving pod and scores backends across five dimensions: KV cache utilization, queue depth, LoRA adapter residency, prefix cache hit rate, and running request count. Each scorer carries a configurable weight, so operators can tilt routing toward latency minimization or throughput maximization depending on workload profile.

Tier 2 — a Global Inference Router (GIR) covering cross-cluster failover, global rate limiting, and cost-aware traffic shaping — is listed as coming soon. In the current failure table, cluster-level failover detects a stale heartbeat and redirects traffic within 35 seconds, with a gradual ramp-up on reintroduction.

The EPP's LoRA Affinity Scorer routes adapter requests to pods that already have the adapter loaded in GPU memory, eliminating swap latency. When no resident pod exists, the request goes to the pod with the most available capacity to load the adapter quickly.

Benchmark results

AWS benchmarked four models ranging from 8B to 235B parameters on p5.48xlarge (H100) and g5 (A10G) instances, with traffic flowing through internal Application Load Balancers and model servers isolated on a dedicated server node group to eliminate resource contention. All figures below are against a Kubernetes round-robin baseline using the gateway's default routing configuration — no hand-tuning applied.

The Qwen3-235B bursty result is instructive: P95 TTFT shows no measurable improvement, but P99 drops 89%, meaning the gateway eliminates tail latency spikes without helping median-load pods. On a fully uniform fleet under steady traffic, the gateway performs on par with round-robin — which AWS explicitly acknowledges. The gains are maximised under exactly the conditions production traffic reliably creates: heterogeneous hardware, bursty demand, shared prefixes. AWS's own headline example translates the 8B mixed-GPU result into human terms: a user waiting 4.4 seconds for the first token now sees it in under 800 ms.

AI Mastery analysis

The EPP's five-scorer weighted algorithm is the architectural lever that deserves the most scrutiny. Because each scorer weight is configurable, teams running latency-sensitive chat applications can heavily weight KV cache utilization and queue depth, while batch inference pipelines can weight throughput-related signals more aggressively. The source material does not specify what the default weights are, which means operators have no published baseline to reason from when diagnosing unexpected routing behaviour — a documentation gap worth watching.

The Tier 1/Tier 2 split reflects a pragmatic shipping decision. Cross-cluster and cross-region intelligence is deferred to GIR. Teams running multi-region active-active deployments today cannot yet rely on the gateway for global failover with intelligent routing; the 35-second cluster-failure recovery is local only. This aligns with the broader pattern where systems-level engineering gains compound across stack layers rather than arriving complete in a single release.

The Gateway API Inference Extension foundation matters for portability. Because the EPP is built on that open standard, the routing logic is inspectable, extensible, and replaceable. Any OpenAI-compatible model server — vLLM, SGLang, TGI — works without modification. The addon's EKS-managed lifecycle does tie it to HyperPod/EKS infrastructure specifically, which is a real constraint for organisations running inference on self-managed or other managed Kubernetes offerings.

The prefix cache hit rate scorer is the subtlest gain. Multi-turn conversation and document Q&A workloads share prompt prefixes across requests; routing those requests to pods with the prefix already cached avoids redundant KV computation that round-robin cannot prevent regardless of fleet uniformity.

Tier 2's promised cost-tier-aware traffic shaping, canary splitting via InferenceModelRewrite CRDs, and priority-band admission control (Critical, Standard, Sheddable) will collectively determine whether HyperPod Inference Gateway becomes a full inference platform layer or remains a sophisticated per-cluster load balancer. The margin in LLM inference infrastructure increasingly comes from routing and scheduling intelligence — and this release is a concrete, measurable step in that direction.

Primary source

Introducing Amazon SageMaker HyperPod Inference Gateway — AWS Machine Learning Blog