2.4T-Parameter Qwen3.8 Runs on One Node With vLLM and NVFP4

On August 12, 2026, Alibaba's Qwen team released Qwen3.8-2.4T-A95B — the first time a Qwen-Max-class model has shipped as open weights. The model carries 2.4 trillion total parameters with 95 billion activated per forward pass, a hybrid linear-plus-full-attention architecture, a native context window of 262,144 tokens extensible to 1,010,000, and a maximum output length of 128K tokens. An AWS blog post published September 9, 2026 by Dmitry Soldatkin, Andrew Smith, and Vinay Arora documents a complete production deployment path on Amazon SageMaker HyperPod using vLLM and NVFP4 quantization on a single ml.p6-b300.48xlarge instance. As production AI systems increasingly fail on infrastructure choices rather than raw model capability, the granularity of this deployment recipe matters as much as the benchmark numbers.

Architecture: Why MoE and Hybrid Attention Change the Serving Math

Qwen3.8 is a fine-grained Mixture of Experts model with 512 routed experts plus 1 shared expert, activating 10 routed experts per token, across 92 layers. The layer layout repeats a 3:1 pattern: three Gated DeltaNet blocks (linear attention with a bounded recurrent state) followed by one Gated Attention block (full quadratic attention). That ratio produces 69 DeltaNet layers and 23 full-attention layers.

Traditional dense transformers grow their KV-cache linearly with every layer as context extends. Here, only the 23 full-attention layers contribute context-dependent memory growth. The 69 DeltaNet layers maintain a fixed recurrent state estimated at 50–100 GB total regardless of sequence length — a critical property for agentic workloads that accumulate tool outputs, code, and reasoning traces toward the 1M-token context ceiling.

The model also ships with native Multi-Token Prediction draft heads bundled into the weights, enabling speculative decoding without a separate draft model.

Hardware and Memory Budget

The ml.p6-b300.48xlarge instance provides 8 NVIDIA B300 Blackwell Ultra GPUs, each with 288 GB HBM3e, for 2.1 TB aggregate GPU memory. GPU-to-GPU interconnect runs at 14.4 TB/s bisection bandwidth over NVLink and NVSwitch. FP4 compute totals approximately 120 PFLOPS across the node. The instance also carries 192 vCPUs (Intel Xeon Emerald Rapids), 4,096 GiB of system memory, 6,400 Gbps EFA networking, and 3.8 TB of local NVMe.

At BF16 precision, the 2.4T parameters require approximately 4.8 TB for weights alone — exceeding any single node. NVFP4 (W4A4) quantization compresses that to approximately 1.2 TB, fitting within the 2.1 TB aggregate and leaving room for cache and activations.

The ml.p6-b300.48xlarge instance type is not available on-demand. Procurement requires a Flexible Training Plan — a committed GPU reservation allocated directly to the HyperPod cluster — with the target Availability Zone set to match the plan's allocation.

vLLM Configuration: Key Flags

The serving command targets the Inferact/Qwen3.8-2.4T-A95B-NVFP4 checkpoint on Hugging Face with --tensor-parallel-size 8 and --quantization nvfp4. Four additional flags drive the bulk of the deployment's capability:

• --enable-prefix-caching: Reuses computed KV-cache blocks across requests sharing the same prompt prefix. For agentic multi-turn conversations where system prompts and history repeat, this can reduce TTFT by 50–80% on repeated turns.
• --reasoning-parser qwen3: Extracts the model's <think>...</think> blocks into a separate reasoning_content field, keeping the thinking trace distinct from the final answer. Thinking can be disabled per-request via extra_body={"chat_template_kwargs": {"enable_thinking": False}}.
• --tool-call-parser qwen3_coder with --enable-auto-tool-choice: Enables OpenAI-compatible function calling with tool_choice values of auto, required, none, and named functions. When strict: True is set on a tool definition, vLLM enforces schema-constrained decoding on tool arguments.
• --speculative-config '{"method":"mtp","num_speculative_tokens":1}': Activates native MTP speculative decoding using the bundled draft heads, which reuse the model's existing hidden states with minimal overhead.

The InferenceEndpointConfig manifest sets VLLM_ENGINE_READY_TIMEOUT_S to 1800 seconds, accounting for the approximately 1.2 TB model download plus fastsafetensors deserialization. On a cold deployment with no cached weights, the full sequence takes 15–30 minutes; subsequent restarts with local NVMe caching are significantly faster.

Benchmark Results

AWS benchmarked four configurations on a single p6-b300 instance using 512 requests at 1,024 input tokens and 1,024 output tokens at concurrency 32, measuring improvement relative to a tensor-parallelism-only (TP=8) baseline:

MTP is the dominant optimization: enabling speculative decoding with just one draft token cuts TTFT from 1,244 ms to 513 ms. The draft head predicts the first output tokens in parallel with the final prefill steps, overlapping compute that would otherwise be sequential. EP alone provides modest gains (approximately 3.5% TTFT, approximately 1% throughput); its benefit is more pronounced at higher concurrency where expert routing becomes a bottleneck. The combined TP+EP+MTP configuration also improves inter-token latency from 17.97 ms to 16.36 ms — a 9% reduction.

One caveat: at high QPS under saturation, MTP's draft-and-verify overhead can reduce aggregate throughput. Monitor spec_decode_acceptance_rate via the vLLM /metrics endpoint and consider disabling MTP under heavy batch load if the acceptance rate drops below roughly 50%.

For sampling, recommended defaults are temperature=0.6, top_p=0.95, and top_k=20, with max_tokens set to 32,768–65,536 for complex reasoning tasks where reasoning_content consumes a meaningful share of the token budget. Setting --max-model-len to match the actual workload maximum — for example, 32,768 for typical coding agents — frees KV-cache slots for additional concurrent requests.

Serving a 2.4-trillion-parameter open-weight model on a single node is no longer a research exercise. The convergence of NVFP4 quantization, a hybrid linear-attention architecture that keeps per-layer memory bounded, and native speculative decoding without a separate draft model compresses what would have been a multi-node orchestration problem into a single declarative Kubernetes manifest. As software extraction increasingly outpaces raw hardware acquisition at the inference frontier, the stack described here — HyperPod Inference Operator, vLLM, NVFP4, and native MTP — represents exactly the systems-level leverage that narrows the gap between proprietary API access and self-hosted frontier capability.