G7 Beats G5 and G6 on MoE Inference With Half the GPUs

AWS's September 8, 2026 benchmarking post delivers controlled, apples-to-apples GPU comparisons using production-grade MoE models on real SageMaker endpoints. The study pits ml.g5.12xlarge (A10G), ml.g6.12xlarge (L4), and ml.g7.12xlarge (RTX PRO 4500 Blackwell) against each other using Qwen3-Coder-30B-A3B-Instruct-FP8, then deploys a second MoE model — NVIDIA Nemotron-3-Nano-30B-A3B-NVFP4 — across G6, G6e, and G7 variants to produce automated cost-efficiency rankings. For teams choosing between GPU generations today, this is one of the first published data sets that quantifies what the Blackwell jump returns at the 30B MoE scale.

Why MoE Inference Is Memory-Bandwidth Bound

Mixture-of-Experts architectures activate only a fraction of parameters per token during decoding, making token generation far more sensitive to memory bandwidth than to raw compute. The bottleneck is moving expert weights into cache rather than executing dense matrix multiplications. Higher memory bandwidth directly compresses inter-token latency and raises sustainable throughput.

G7's structural edge goes beyond bandwidth. The RTX PRO 4500 Blackwell carries native FP4 Tensor Core support, enabling hardware-accelerated inference over NVFP4-quantized weights. G5 (A10G) and G6 (L4) lack that hardware path: they can load NVFP4 weights but execute without the dedicated silicon. This is why the AWS team selected NVIDIA's own NVFP4-quantized Nemotron model for the second use case — the benchmark is designed to exercise exactly that hardware differentiation.

Use Case 1: Qwen3-Coder-30B Throughput and Latency

The benchmark uses DJL Large Model Inference (LMI) 28.0, a fixed workload of 100 requests at concurrency 4 with 128 mean input and 128 mean output tokens, and the SageMaker Generative AI benchmarking tool backed by NVIDIA AIPerf. G5 and G6 each deploy four GPUs totalling 96 GB of aggregate GPU memory; G7 uses only two GPUs with 64 GB.

G7 delivers 13.0% higher throughput than G5 and 60.8% higher than G6. Average latency falls 10.8% versus G5 and 37.6% versus G6. At P99, the latency reduction reaches 20.2% relative to G5 and 54.7% relative to G6 — significant for coding copilots and IDE integrations where tail latency degrades perceived responsiveness.

A streaming-mode run on G7 recorded a P50 TTFT of 117.9 ms, a P99 TTFT of 286.0 ms, and an average ITL of 8.9 ms at 408.1 output tokens per second — within the responsiveness envelope for interactive developer tooling. For more on how architectural choices shape these trade-offs, see architectural specificity outperforms GPU scaling.

Use Case 2: Nemotron-3-Nano-30B Cost-Per-Token Optimisation

The second use case uses SageMaker's generate_deployment_recommendations() API with vLLM, supplying PerformanceTarget.THROUGHPUT as the optimisation objective across G6 (4× L4, 96 GB aggregate), G6e (4× L40S, 192 GB aggregate), and G7 (2× Blackwell, 64 GB) configurations. Cost per million output tokens is derived post-hoc from measured throughput: ($/hour × 1,000,000) ÷ (output tokens/s × 3,600).

Two workload profiles reveal how input length reshapes the economics. Profile A (512 ± 50 input / 256 ± 30 output tokens) represents chat traffic; Profile B (3,500 ± 50 input / 200 ± 20 output tokens) represents RAG pipelines. Results:

Across both profiles, G7 delivers approximately 1.4× lower cost per token than G6e and approximately 4.9× lower than G6 for the chat workload and 5.6× lower for RAG. The RAG profile's throughput drop — from 2,397 to 816 tok/s on g7.48xlarge — illustrates how prompt-processing load erodes absolute output throughput even on identical hardware.

Regional Availability and Scope

G7 instances are currently generally available only in US East (Ohio) and US West (Oregon). The benchmarks are explicitly scoped to the models, quantization formats (FP8 for Qwen3-Coder, NVFP4 for Nemotron), LMI 28.0 or vLLM serving stacks, and the token distributions and concurrency levels used. AWS recommends re-running with application-representative traffic before committing to a production configuration.

The core finding is that MoE architectures at the 30B scale have entered territory where hardware-quantization co-design — Blackwell's native FP4 Tensor Cores paired with NVFP4 weights — produces cost and latency advantages that prior-generation configurations cannot recover through GPU count alone. As examined in software extraction beats hardware acquisition, software-side optimisation has limits; when new silicon introduces a structurally different execution path for a target quantization format, the hardware generation itself becomes the primary lever. SageMaker's generate_deployment_recommendations() tooling automates this evaluation across candidate instance families, materially lowering the engineering cost of finding that optimum before committing to deployment.