NVIDIA TRTMC: Hugging Face to C++ TensorRT in Two Commands, No ONNX

NVIDIA has pushed TensorRT Model Connect (TRTMC) into public preview as an Apache-2.0 open-source project that takes a supported Hugging Face or local checkpoint to native C++ TensorRT inference in exactly two CLI commands, with no intermediate ONNX or TorchScript export step. The output is a versioned .bundle artifact that loads via a native C++ task API, keeping PyTorch entirely out of the runtime path — a hard requirement for teams shipping inference inside embedded systems, robotics stacks, or automotive in-vehicle compute.

The conventional pain points are well understood: the PyTorch → ONNX → TensorRT → hand-written C++ glue route accumulates export failures on unsupported graph operators, validation effort that compounds across multiple conversion artifacts, and per-model rewrites of tokenization, sampling, and pre/post-processing logic. TRTMC replaces those boundaries with a family-owned build system and a task-oriented runtime.

The Two-Command Pipeline

The quick-start builds and runs Qwen3-0.6B:

trtmc build Qwen/Qwen3-0.6B --precision bf16 --max-cache-length 16384 --output qwen3-0.6b.bundle
trtmc run ./qwen3-0.6b.bundle --prompt "What is the capital of France? Answer in one word." --chat-template --no-thinking

The same .bundle loads from C++ with trtmc::load("./qwen3-0.6b.bundle"). Python owns checkpoint resolution and TensorRT engine construction; the native C++ runtime then calls task-oriented APIs — generate(), transcribe(), generate_image(), embed(), solve() — without maintaining per-model conversion stages. A small number of hybrid profiles invoke a helper Python executable, and those manifests declare that dependency explicitly. trtmc inspect exposes bundle kind, model family, precision, runtime identity, and engines, making the artifact auditable at any point in a CI pipeline.

Release Snapshot and Throughput Benchmarks

NVIDIA's July 29, 2026 comparison run on GB300 hardware covered 105 single-process release profiles across 76 model families. Of those, 102 beat their declared per-row reference by more than 5%, one landed within 5%, and two were slower by more than 5%. Bundle preparation, model loading, compilation, and warmup are excluded from the infer-p50 values; baselines are declared per row rather than uniformly against torch.compile. Families include decoder and hybrid language models, embeddings, OCR, ASR, TTS, diffusion image and video generation, segmentation, and time-series forecasting — comparable in breadth to the range of deployable checkpoints surveyed in the best small language models on Hugging Face.

Architecture and Provenance

TRTMC is explicitly not a single generic converter. It ships as a collection of family-owned reference implementations, each building directly against TensorRT APIs for its model class. That design eliminates the ONNX export layer but bounds coverage to the 76 families NVIDIA has implemented and validated as of the July 29 snapshot. NVIDIA also notes that the entire project — model implementations, performance tuning, tests, integrations, and documentation — was built using OpenAI Codex agents under human direction and review.

Availability and Fit

The aarch64-only wheel is the most immediate friction point for x86_64 development environments. Teams already inside Docker may absorb the source-build path without difficulty; those who do not will need to account for it before evaluating TRTMC against existing tooling. The broader trajectory — software-layer optimisation displacing hardware acquisition as the primary inference efficiency lever — is examined in software extraction beating hardware acquisition at the AI frontier.

With 102 of 105 measured profiles exceeding their declared reference by more than 5% on GB300, the throughput case is present for teams already inside the NVIDIA stack. The binding constraints today are the 76-family ceiling and the aarch64-only binary distribution — factors that will determine whether TRTMC earns adoption in general-purpose MLOps pipelines or remains, for now, an edge and robotics accelerator.