Chunked Prefill Beats Disaggregation Below 1,000 GPUs

Every major inference framework shipped prefill-decode disaggregation in 2026: NVIDIA built it into Dynamo, SGLang made it the default for large-scale deployments, and vLLM added a KV connector API to support it natively. The implication — that teams should split prefill and decode onto separate GPU pools to improve throughput — is wrong for most production environments, and acting on it prematurely introduces failure modes that produce no errors while silently corrupting output.

The core problem is hardware profile mismatch. Prefill is compute-bound, driving GPU utilization to 80–95% through parallel matrix multiplications across the full input sequence. Decode is memory-bandwidth-bound, reading the KV cache sequentially to emit one token at a time, with compute utilization below 5% on an H100. When both phases share a GPU, a single large prefill request arriving mid-decode inflates time-per-output-token by 2–30x under bursty workloads. Disaggregation is one fix — not the only fix, and for clusters below roughly a thousand GPUs, usually the wrong one.

What Chunked Prefill Actually Solves

Chunked prefill breaks long prefill requests into smaller chunks and interleaves them with decode batches on the same GPU, bounding rather than eliminating the interference. TNG Technology Consulting measured a 50% increase in total token throughput using standard vLLM with chunked prefill enabled — no separate node pools, no KV cache transfer over the network, no P:D ratio to tune. For workloads below roughly 50 requests per second with moderate prompt lengths, the interference bound that chunked prefill imposes is tight enough to stay within SLO. This matches the pattern in systems-level work showing that architectural specificity outperforms GPU scaling.

Doubleword's analysis reinforces this: a balanced disaggregated deployment matches colocated throughput, but at small GPU counts, rounding losses dominate because fractional GPU allocation is impossible. The practical benefit of disaggregation at that scale is independent SLO tuning, not throughput improvement.

The Three Costs Disaggregation Explainers Omit

The KV transfer tax. For a 70B-parameter model, each completed prefill generates roughly 2.6 GB of KV cache that must be shipped to a decode node. When prefill and decode share a node, that transfer stays in GPU memory. Once disaggregated across nodes, the transfer hits the network interconnect, and available bandwidth drops by orders of magnitude depending on topology. Without InfiniBand or NVLink in the same rack, network transfer cost dominates any compute specialization gain.

The operational surface. Disaggregation requires separate prefill and decode node pools, each with independent scaling policy. The optimal P:D ratio is workload-specific: LMSYS evaluated 4 prefill nodes and 9 decode nodes for DeepSeek-R1, and those numbers shift whenever the prompt-to-output length ratio changes. A chat workload and a RAG pipeline require different ratios. If node pools are static while traffic mix shifts, one side is over-provisioned while the other starves. There is no graceful fallback: if a prefill node goes down, decode nodes cannot substitute.

The silent failure cliff. SGLang issue #9266 documents systematic KV cache transfer failures at 64 or more concurrent requests, returning HTTP 400 to clients. Issue #30233 describes a worse failure: when input exceeds the maximum request length, the prefill side aborts but still transfers a single-token KV cache, causing the decode side to generate from uninitialized memory with no error surfacing to the caller. These failures are invisible at low QPS and only manifest at production concurrency. Modular's inference handbook reports a 20–30% performance drop from disaggregation on small or untuned workloads — teams pay the KV transfer cost on every request before any throughput gain materialises.

The Decision Matrix

Three Conditions That Must Hold Simultaneously

Disaggregation pays for itself only when three conditions are all true at once. First, the cluster must be large enough for clean P:D allocation — DeepSeek required thousands of GPUs before the ratio produced integer node counts matching their traffic mix; at 8–16 GPUs, the available ratios are 1:7 or 2:6, neither of which may fit a given workload. Second, network bandwidth must sustain the KV production rate of the prefill pool; if prefill generates caches faster than the interconnect delivers them, decode nodes idle and the bottleneck migrates from compute interference to network transfer. Third, dynamic autoscaling must be in place to handle shifting traffic mixes; static node pools against variable workloads guarantee over-provisioning on one side. If any single condition is absent, chunked prefill is the correct default.

The teams extracting the most from their inference clusters in 2026 are the ones instrumenting p95 TPOT, profiling prefill fraction of TTFT, and validating KV transfer reliability at production concurrency before committing to architectural changes. Disaggregation is the right answer above roughly a thousand GPUs with fast interconnect and dedicated infrastructure engineering capacity — a description that fits hyperscalers and large inference providers, not most teams shipping LLM features this year. As software extraction increasingly outperforms hardware acquisition at the AI frontier, the scheduling intervention alone often closes the gap.