SeedRealtime: ByteDance's Native Audio-Visual Full-Duplex LLM
In this article
ByteDance's Seed team has shipped SeedRealtime, a native audio-visual full-duplex large language model that processes audio, video, and text inside a single end-to-end architecture rather than routing them through chained modules. The architectural target is the cascade — the ASR → VLM → LLM → TTS pipeline that still underpins most production real-time voice systems — which SeedRealtime replaces with parallel perception, understanding, decision-making, and expression running continuously over live multimodal streams. For teams building real-time multimodal agents, this is a significant reference architecture, even though the weights are not yet public.
What the Architecture Actually Does Differently
The conventional cascaded stack has a structural flaw: each stage waits for the previous one to complete, and each handoff discards information. Speech-to-text collapses prosody; the VLM processes frames independently of audio; TTS receives only the text output of the reasoning stage. SeedRealtime removes every handoff. Perception of the incoming audio and video streams, joint understanding, the decision of whether and when to speak, and speech synthesis all operate in the same model simultaneously.
Turn-taking is the second structural change. Most real-time stacks still route the "is the user done speaking?" decision through an external voice-activity detector. SeedRealtime internalises this judgment — the model decides when to stay silent and when to interrupt based on the full multimodal context. ByteDance's own human evaluation reports that pacing issues were halved relative to cascaded stacks, though no latency figures or external benchmark scores have been published.
Four Scenarios That Demonstrate the Capability Envelope
ByteDance published seven demonstration scenarios. The four that carry the most architectural signal are:
Identity binding across modalities: At a group dinner, the model attaches names to faces as people are introduced, then maintains voice-to-identity attribution across the conversation — correctly assigning conflicting travel preferences to different speakers before synthesising a plan. This requires simultaneous processing of face appearance, voice characteristics, and conversational content inside one representational space.
Proactive speech from a held instruction: At Hebei Museum, a user asks to be alerted when a specific bronze screen stand appears. The camera continues panning; the model watches without prompting and speaks when the exhibit enters frame. The same behaviour was demonstrated on a ResNet paper — the model tracks rapid page flips, identifies the "3.4 Implementation" section autonomously, pauses on its own, and reads out the learning rate, momentum, and weight decay values.
Correction from visual state: Watching an espresso workflow, the model interrupts when whole beans are placed in the portafilter without grinding, then reads crema colour and volume and recommends shortening extraction by 2 to 3 seconds. The trigger is a visual state mismatch, not a user question.
Off-screen memory with interference suppression: At Beijing Daxing Airport, unrelated bystander chatter about a flight does not trigger a response. When the user subsequently asks, the model answers using departure-board information that had already scrolled off screen, and retrieves the baggage-carousel location from an online source.
Architecture vs. Cascade
| Capability | Cascaded Stack (ASR → VLM → LLM → TTS) | SeedRealtime (Native Full-Duplex) |
|---|---|---|
| Modality integration | Sequential handoffs; each stage receives text output of the prior | Audio, video, and text processed in parallel inside one model |
| Turn-taking | External VAD determines end-of-turn signal | Internal decision based on full multimodal context |
| Pacing errors (human eval) | Baseline | Halved vs. cascade (ByteDance internal eval; no external benchmark) |
| Cross-modal grounding | Visual context converted to text before reasoning | Visual and audio context share the same representational layer |
| Proactive interaction | Requires explicit user prompt | Model initiates unprompted when held conditions are satisfied |
| Current availability | Widely deployed; multiple open and API-accessible implementations | Live in Doubao app only; no weights, no API, no technical report |
Availability
SeedRealtime is live inside Doubao, ByteDance's consumer assistant. Beyond that deployment, ByteDance has released no technical report, no parameter count, no open weights, and no Volcano Engine or BytePlus API endpoint. Third-party integration is not currently possible.
What is actionable now is the architecture as a validated reference. The four demonstrated behaviours — cross-modal identity binding, proactive unprompted speech, correction from visual state, and interference-robust off-screen memory — define a concrete capability target for teams building camera-plus-voice products. The growing complexity of agentic system design makes unified multimodal perception increasingly relevant, and SeedRealtime establishes that the cascade is no longer the only viable path to production. The open question is whether ByteDance releases weights or an API before competitors close the architectural gap.