SageMaker Prefix-Aware Routing Cuts LLM P50 TTFT by Up to 77%

Serving a large language model at scale exposes a structural inefficiency most teams discover too late: prefix caching — one of the most effective latency optimisations available in frameworks like vLLM and TensorRT-LLM — degrades to near-zero effectiveness the moment you distribute load across multiple instances. A shared 3,000-token system prompt that would be computed once on a single instance gets recomputed from scratch on every machine in the fleet, because random routing ensures no single instance accumulates a warm cache for any given prefix. Amazon SageMaker Inference's newly launched PREFIX_AWARE routing strategy directly attacks this failure mode.

The mechanism is straightforward but consequential: the endpoint inspects the beginning of each incoming request and consistently routes requests sharing the same opening to the same instance, allowing that instance's KV cache to compound across requests rather than sit perpetually cold. This is consistent with the broader pattern that systems-engineering gains at the infrastructure layer rival model-level scaling — the gains here come from smarter request dispatch, not additional hardware.

How prefix-aware routing works

Two parameters govern the behaviour. PrefixLength (range: 1,024–65,536) determines how many bytes or characters from the request head are used as the routing key — bytes on the native Invoke API, characters from extracted message text on the OpenAI-compatible path. ConcurrencyThreshold (range: 1–1,024) sets the in-flight request ceiling on the target instance; once reached, the request is redirected to a less loaded instance rather than queuing.

Two additional design decisions matter operationally. First, scaling events do not invalidate caches globally: when instances are added or removed, only a small fraction of traffic redistributes, so warm caches on surviving instances remain intact. Second, an optional tenant-isolation ID — the X-Amzn-SageMaker-Prefix-Aware-Id header (up to 64 ASCII characters) on the native API, or prompt_cache_key on the OpenAI-compatible path — combines with the prefix hash to separate routing for tenants whose prompts are textually identical but must remain cache-isolated.

Benchmark results

AWS benchmarked the feature on Llama 3.1 70B Instruct across 7 ml.p5.48xlarge instances using vLLM with prefix caching enabled, running 16 test configurations covering single-model endpoints, inference component endpoints, the native Invoke API, and the OpenAI-compatible API. All tests completed with a 100% success rate.

The routing layer itself added 1.3–1.9 milliseconds per request against model TTFT measurements of 63–280 milliseconds — overhead that is negligible relative to the savings. Traffic distribution remained within 1 percentage point of an ideal even split across all 7 instances (each receiving 13.3–15.4% of requests), confirming that hash-based affinity does not create hot spots under the tested load patterns.

Three routing strategies on SageMaker real-time endpoints

SageMaker Inference now exposes three strategies via RoutingStrategy in the production variant configuration. RANDOM remains the default for general-purpose and non-LLM workloads. LEAST_OUTSTANDING_REQUESTS targets workloads with variable processing times where keeping all instances equally loaded is the priority. PREFIX_AWARE is recommended whenever requests share a common prefix and the serving container has prefix caching active. Switching between strategies requires only an endpoint configuration update — no model redeployment.

AI Mastery analysis

Several operational constraints deserve attention before treating these benchmark numbers as universally applicable. The 77% P50 TTFT reduction is specific to 8,000-token shared prefixes; the ShareGPT results (13–16% P50 improvement) confirm that shorter prefixes yield proportionally less KV cache computation to skip. Mixed workloads — some requests sharing long system prompts, others with short or unique context — will see blended results that require load testing against real traffic distributions.

The serialisation consistency requirement is non-trivial. Because PrefixLength on the native Invoke API operates on raw request bytes, any variation in JSON key ordering, whitespace, or floating-point formatting across client implementations corrupts the routing key and scatters requests that should be collocated. Teams using multiple SDK versions or heterogeneous request generators need explicit serialisation discipline.

The minimum two-instance requirement means the feature adds no value for single-instance deployments. And while the feature integrates with LoRA adapters by confining prefix-based selection to the sticky instance set that already has a given adapter loaded, the interaction between adapter-loading latency and cache hit rate in high-adapter-count deployments is not characterised in the published benchmarks.

The KV cache hit rate jump from ~25% to ~82% on long-context workloads represents a substantial reduction in redundant computation without touching model weights, quantisation, or hardware — a clean demonstration that architectural specificity can outperform GPU scaling at the serving layer.

Prefix-aware routing is available now on SageMaker real-time inference endpoints after updating the AWS SDK or CLI to pick up the new RoutingStrategy and PrefixAwareRoutingConfig parameters.

Primary source

Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference — AWS Machine Learning Blog