cuDNN Graph API: Fusion, Autotuning, and Plan Reuse Explained

NVIDIA's cuDNN Frontend Graph API surfaces the same kernel-fusion machinery that frameworks like PyTorch use internally, but hands it directly to the engineer — making fusions, engine selection, and compilation timing explicit decisions rather than opaque defaults. That shift matters because systems-level specificity increasingly drives inference efficiency gains more than raw compute budget, and the Graph API is one of the sharpest tools available for extracting that efficiency at the CUDA layer.

The tutorial works through seven progressive sections on a single Colab GPU, validating every fused kernel against a PyTorch reference. The baseline correctness check is not incidental: sections where the cuDNN Frontend result merely matches PyTorch confirm that the framework was already dispatching to cuDNN underneath — which marks out precisely where the lower-level API earns its keep: fusions with no framework-level equivalent, shapes hot enough to justify exhaustive engine search, and small kernels where launch overhead dominates wall time.

The Five-Step Build Pipeline and Fusion Mechanics

Every graph follows the same construction sequence: declare tensors by dimensions and strides, chain operations, then call validate, build_operation_graph, create_execution_plans, check_support, and build_plans in order. All I/O tensors are kept in channels_last (NHWC) memory format because cuDNN's tensor-core engines expect those strides; mismatched layouts silently degrade to slower paths.

The first fusion target is a convolution-bias-ReLU chain over a problem of shape N=32, C=128, H=56, W=56, with K=256 filters of size 3×3. Folding the bias add and activation into the same kernel eliminates two intermediate tensor writes. The matmul section extends the pattern further: a batch-size-16 matrix multiply (M=512, K=1024, N=512) carries a full epilogue — an alpha scalar passed by value from the host, a bias add, an activation (GELU where available, ReLU as fallback), and an AMAX reduction. The AMAX collected inside the same kernel is the FP8 training primitive: it produces the quantization scale factor for the next layer without a second pass over the output. The comparison against torch.baddbmm plus separate activation and amax calls makes explicit that the speedup comes from eliminating epilogue memory traffic, not from a faster GEMM core.

Autotuning: Exhaustive Engine Search vs. Heuristics

The autotuning section rebuilds the convolution graph but requests plans from heuristic modes A, B, and FALLBACK simultaneously, then compiles all surviving candidates with build_plan_policy.ALL. It walks the plan list, calls build_plan_at_index for each, allocates that plan's specific workspace, and times execution with execute_plan_at_index at 3 warmup and 15 timed iterations per candidate. Throughput and workspace footprint are printed for every surviving config.

The spread between the fastest and slowest engine across candidates is the operational result. As pipeline-level engineering increasingly rivals scaling for production AI gains, that spread quantifies the budget available for offline autotuning on a hot shape. The recommended production pattern is to persist the winning plan index — or the full serialized plan blob — rather than re-running heuristics at startup.

Plan Serialization, Dynamic Shapes, and CUDA Graph Capture

The serialization section builds a batched matmul (Bsz=8, M=256, K=512, N=256), serializes the compiled plan to a byte blob, then deserializes into a fresh graph object and executes via integer UIDs rather than tensor handles — skipping compilation entirely. The dynamic-shapes section shares one kernel cache across four convolution graphs differing only in batch size (8, 16, 24, 32), so build time drops for later shapes after the first JIT compilation. CUDA graph capture wraps the convolution plan from section 2, pinning the cuDNN handle's stream to the capture stream so kernel launches land inside the graph. The tutorial notes that CUDA graph capture logging must use level 10 (CUDNN_FRONTEND_LOG_INFO=1), because level 1 dumps tensor data and is not capture-safe.

AI Mastery analysis

The SDPA section exposes a tradeoff that practitioners frequently misread. The tutorial explicitly acknowledges that PyTorch's scaled_dot_product_attention may already dispatch to cuDNN or FlashAttention, so parity between the two timings is the expected, healthy result — not a failure of the lower-level API. The Graph API's attention path is valuable not for raw throughput over PyTorch's default, but for composability: custom epilogues, non-standard masking patterns, or quantization primitives that torch.nn.functional.scaled_dot_product_attention does not expose can be attached, and the result still runs as a single fused kernel. This is consistent with the broader pattern that production AI failures concentrate in architectural decisions rather than model capability — the framework's default dispatch is correct for the common case, but inflexible at the boundary conditions that matter most in deployment.

The kernel cache approach for dynamic shapes deserves scrutiny. Sharing a single cache across graphs with is_dynamic_shape_enabled=True works when shape variants are compatible with an already-compiled kernel's tile configuration, but the tutorial does not enumerate what "compatible" means in terms of tile size constraints. Engineers serving variable sequence lengths should validate that later build calls are genuinely hitting the cache rather than silently falling back to a full recompile — the per-shape build timing printed in the tutorial is the right signal to watch.

The serialization mechanism binds the blob to a specific GPU architecture and cuDNN backend version. In any deployment pipeline that touches multiple GPU generations or updates the cuDNN package, plan serialization requires a versioned artifact store and a fallback rebuild path. As software-level extraction of hardware performance replaces hardware acquisition as the primary efficiency lever, the pattern the tutorial demonstrates — describe once, autotune offline, serialize, capture — becomes a template for turning one-time engineering cost into permanent per-invocation savings.

Primary source

Inside NVIDIA's cuDNN Graph API: Fusion, Autotuning, and Plan Reuse with cuDNN Frontend — MarkTechPost