DFlash Delivers 3.92x CPU Token Throughput on Qwen3.5-9B

DFlash speculative decoding support landed in vLLM v0.25.0, and benchmarks on an Amazon EC2 r8i.16xlarge — a single Intel Xeon 6975P-C with 32 cores and 512 GB DDR5 at 7200 MT/s — show Qwen3.5-9B reaching 3.92x average output token throughput over its autoregressive baseline at concurrency 1, translating to a 74.40% reduction in per-token generation cost at fixed concurrency.

The result matters because CPU inference is not a fringe case. Organisations running latency-bound LLM workloads at low concurrency — internal tooling, single-user agent loops, or regulated environments where GPU allocation is constrained — face a bottleneck that is fundamentally memory-bandwidth-limited, not compute-limited. This is exactly where speculative decoding earns its keep: per-token matrix-vector operations that lean on Intel AVX-512 become multi-position matrix-matrix operations that Intel AMX can accelerate, meaning target model weights loaded from DRAM are reused across several candidate positions rather than once per token. That reuse is the economic core of the speedup.

How DFlash Differs From Conventional Speculative Decoding

Standard speculative decoding runs a small draft model autoregressively to propose tokens, then checks the full sequence against the target in one verification pass. DFlash, developed by Z Lab and published at PMLR 306, replaces serial draft token generation with a block-diffusion drafter that predicts all masked positions in a single forward pass using bidirectional attention within the block. The number of positions proposed per block is controlled by the num_speculative_tokens parameter — set to 15 in the benchmarks below. DFlash also injects hidden features from the target model directly into each draft layer's KV cache, giving the drafter access to the target's contextual representation without requiring it to reconstruct full context independently. That combination raises acceptance rates without inflating drafter size.

The drafter for Qwen3.5-9B is available as z-lab/Qwen3.5-9B-DFlash on Hugging Face. Enabling it requires one additional flag in the vLLM Docker invocation:

docker run --rm \
  --name vllm-cpu-server \
  --network host --ipc host --security-opt seccomp=unconfined --cap-add SYS_NICE \
  -e VLLM_TARGET_DEVICE=cpu \
  -e VLLM_CPU_KVCACHE_SPACE=40 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest \
  Qwen/Qwen3.5-9B \
  --dtype bfloat16 \
  --trust-remote-code \
  --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-9B-DFlash", "num_speculative_tokens": 15}'

For models without a paired drafter, Z Lab's DFlash Hugging Face collection and Red Hat AI's Speculator-trained model collection are the recommended starting points.

Benchmark Results

The benchmark used vLLM 0.26.1rc1 on Ubuntu 24.04.4 LTS, tested by Intel as of July 2026, at concurrency 1 with output length 128. Three datasets covering distinct generation regimes were included: HumanEval (programming), GSM8K (mathematics), and MT-Bench (multi-turn conversational). Structured output domains consistently outperform conversation because acceptance rates are higher when token sequences are more predictable.

The HumanEval run with num_speculative_tokens 15 illustrates how vLLM's speculative decoding counters read in practice. Across 381 verification steps, 5,715 draft tokens were proposed (15 × 381), of which the target accepted 2,261, yielding a 39.56% acceptance rate. Each verification round committed an average of 5.93 accepted draft tokens plus one target-generated correction or bonus token, producing an acceptance length of 6.93. The per-position survival profile declines from 85.83% at position 0 to 11.81% at position 14. That decay does not mean the block should be shortened: 12% of blocks are fully accepted and would contribute nothing if truncated, while 14% have every proposal rejected regardless. Throughput on the specific hardware and dataset is the correct tuning signal for num_speculative_tokens, not acceptance rate alone.

Concurrency as the Key Deployment Constraint

The gains above are measured at concurrency 1, and the source is explicit that the advantage shrinks as concurrency rises. Speculative decoding's value proposition is trading spare compute for saved memory bandwidth. At low concurrency, compute units are largely idle while weights stream from DRAM, so wider verification is nearly free. As concurrent requests fill those compute units, spare capacity evaporates and the draft-plus-verify overhead becomes net-negative. Teams evaluating DFlash for production should benchmark at their realistic concurrency distribution before committing. The technique is a strong fit for single-user or near-single-user serving patterns, where the cost structure of CPU inference scales very differently from accelerator-based deployments.

Lossless 3.92x throughput on commodity Xeon silicon — without quantization, without hardware upgrades, and with a one-flag code change — demonstrates that well-tuned CPU inference can close a meaningful gap against GPU baselines in the single-request regime. Work on parallel drafting continues beyond DFlash: DSpark introduces semi-autoregressive correction and variable-length verification, while DFlash 2 adds a path selector and local convolutions to reduce draft-accuracy decay at the tail of each block. For teams running latency-bound workloads on CPU infrastructure today, DFlash on vLLM v0.25.0 is the highest-leverage, lowest-friction option currently available in a production inference framework.