NVIDIA VoiceChat 11B: Open Full-Duplex Speech Model with 448 ms Latency

NVIDIA's NemotronLabs team has shipped VoiceChat 11B under a permissive OpenMDW-1.1 license — the first open-weight model to combine full-duplex speech-to-speech operation with live tool calling. The headline figure is a measured end-to-end turn-taking latency of 448 ms on Full-Duplex-Bench 1.0, achieved inside a single 11-billion-parameter network rather than across three separate inference calls. The model listens while speaking, so barge-in is a native behaviour rather than an interrupt handler bolted onto a half-duplex loop.

Architecture: One Network, Four Functional Paths

VoiceChat 11B is assembled from three existing NVIDIA components plus one new output path. A Fast Conformer speech encoder, drawn from Nemotron-Speech-Streaming-En-0.6b, continuously encodes an incoming 16 kHz stream. That feeds a Nemotron Nano v2 LLM backbone — a hybrid Mamba/Transformer — which consumes audio tokens and predicts agent text tokens alongside a running user transcription. A TTS decoder and codec renders those tokens as 22.05 kHz agent audio. The fourth path is a dedicated side channel for tool-call scripts, keeping function invocations out of the spoken reply stream entirely. Training spanned roughly 550,000 hours of real and synthetic audio, building on prior SALM-Duplex and Audio Flamingo 3 work. The context window is capped at two minutes of audio — a hard constraint NVIDIA documents as a current failure mode rather than a design target.

Benchmarks and Known Failure Modes

NVIDIA ranks VoiceChat 11B at #2 among open full-duplex models on both VoiceBench and Full-Duplex-Bench 1.0. The interrupt TOR of 1.00 at 480 ms — meaning the agent yields the floor in every tested case — is the standout conversational result. Tool-calling figures are less uniform: 82.5% tool selection accuracy collapses to 33% pass@1 and 27.5% on parallel-multiple calls, a gap that matters for production voice agents handling complex multi-step lookups. NVIDIA's own repository documents four additional failure modes: non-recoverable gibberish after several dialogue turns, runaway self-talk after a turn completes, dropped words in user transcription, and the two-minute audio context ceiling. The official characterisation is "research purposes only."

Live Tool Calling: Side Channel and On-Hold Messages

When the model generates text that triggers a tool call, the call is emitted on the dedicated side channel as a structured block; the operator's code returns a result block in the same format. System prompts and tool responses must be ASCII-only and formatted to be TTS-readable. The dead-air problem — silence while an external API runs — is addressed through per-tool on-hold messages: an operator defines a phrase the agent speaks the moment the tool call fires, before any response arrives. NVIDIA caps recommended session complexity at five tools and explicitly states the model cannot reliably invoke multiple tools simultaneously; the user also cannot interrupt during tool execution. For teams building voice front-ends over internal APIs, as explored in agentic AI patterns reshaping data science workflows, this side-channel architecture is a cleaner separation of concerns than injecting tool results mid-utterance in a cascaded stack.

Availability and Hardware Requirements

Weights, an NGC container, and the NeMo Speech repository are all publicly accessible. The minimum hardware specification is a single GPU with 80 GB of VRAM — A100, H100, RTX 6000 Pro, or B200 — running x86_64 Linux. No hosted API exists and no inference provider currently serves the model, meaning teams without direct GPU access cannot evaluate it in the current release window. The permissive OpenMDW-1.1 license does not restrict commercial experimentation, but the "research-only" label on the checkpoint is NVIDIA's practical guidance against customer-facing deployment. NVIDIA's broader supply chain position in high-end GPU hardware, well documented, means the 80 GB VRAM floor is more accessible to its existing enterprise customer base than it would be for a third-party release.

The release represents a concrete step toward collapsing the cascaded voice stack into a single deployable artifact — something OpenAI's realtime audio work has approached from the hosted-service direction. Whether the documented instability beyond a few turns can be resolved without architectural changes will determine whether the 448 ms latency figure matters outside benchmarking rigs. The open weights mean the research community can engage with that question directly.