DoorDash's 2-Layer Filter Cuts Verbal-Abuse Incidents 50% at 4M Messages/Day

DoorDash processes over 4 million chat messages, 400,000 voice calls, and 200,000 image exchanges every day across its three-sided marketplace of consumers, Dashers, and merchants. Every chat message must be classified as safe or unsafe before delivery — a window measured in fractions of a second. At that volume, routing every message through an LLM produces per-call latency averaging 2 to 10 seconds and a cost curve that doesn't close. The engineering response, presented at QCon AI by DoorDash trust and safety engineer Bruna Pereira, is a named hybrid pattern now powering a generalised internal moderation platform.

Data Before Architecture

Before writing a single classifier, the team spent roughly one and a half months instrumenting chat and asynchronously calling a free, off-the-shelf moderation API to characterise their message corpus. The goal was not to make production decisions but to put a number on what they already suspected. The finding that shaped everything: only a small single-digit percentage of messages were actually unsafe. That statistic justified building an aggressive, cheap first layer tuned specifically to confirm safety rather than detect harm. Skipping this step, Pereira noted, is the most common mistake teams make — the economics of the hybrid pattern only hold if you know what fraction of traffic the cheap layer can absorb.

Two-Layer Pipeline: Classifier Then Multi-Axial LLM

The internal classifier — now in its ninth trained version — has three hard constraints: respond in under 100 milliseconds at the 90th percentile, carry no per-call cost (infrastructure cost only), and reliably identify messages that are obviously safe. It outputs a 0-to-1 score. Messages scoring above a 0.5 unsafe threshold proceed to the LLM layer; everything below is delivered. Because less than 10% of messages reach the LLM, the economics become tractable.

The critical design choice is what the LLM is asked to do. Rather than a Boolean safe/unsafe judgment, the LLM scores each message across multiple independent axes — threat level, profanity level, sexual content level, and others added over time. This multi-axial scoring enables graduated enforcement: low-severity content (swearing) triggers censorship and delivery; mid-severity content (insults) triggers message blocking; high-severity content (threats) triggers blocking plus an offer to cancel the order without charge; very high severity triggers order cancellation, offender warning, and removal of the affected party from the interaction loop. None of those four response tiers are possible with a Boolean output.

This aligns with the broader architectural principle — discussed in our coverage of pipeline architecture driving the biggest AI gains — that how you compose and sequence model calls matters more than which model you use.

Voice and image channels use structurally identical scoring engines with different cheap-layer implementations. For images, a commercial vision API handles the obvious-case filter in place of the internal classifier. For voice, real-time blocking is impossible — the audio has already been heard by the time transcription completes — so the action layer shifts to call termination and order cancellation.

Architecture at a Glance

From Product to Platform

After SafeChat produced roughly a 50% reduction in verbal-abuse safety incidents — a real-harm metric, not a model accuracy figure — other DoorDash teams requested their own versions: Dasher profile picture moderation, name validation at signup, food review moderation, and fraud detection in chat and calls. Rebuilding the same pattern repeatedly was untenable, so the team extracted it into a content-agnostic moderation platform.

The platform exposes three model primitives: internal models (trained, fine-tuned, and served on DoorDash infrastructure under a standardised input/output schema); external models (pre-integrated vendor APIs, billed per team via separate API keys); and external prompts (arbitrary LLM calls routed through an internal LLM gateway with configurable fallback and retry strategies, including structured-output retry when a model returns malformed JSON). Teams compose these primitives into moderation agents — directed pipelines with conditional branching — entirely through a UI without writing code.

Agents run either synchronously, holding an HTTP connection open to gate delivery decisions, or asynchronously, acknowledging the request immediately and publishing results to a Kafka topic when execution completes. The async path is preferred wherever gating is not required, because it removes the latency cap that synchronous execution imposes on each pipeline step.

A built-in backtesting workflow lets teams run candidate agents against historical datasets; human reviewers label results as true positive, false positive, true negative, or false negative, and the platform computes evaluation metrics before the agent reaches production. Pereira cited approximately 1,000 labeled examples as a practical threshold — enough to produce trustworthy metrics, small enough that reviewers remain engaged with the task.

Portable Lessons

The pattern DoorDash codified — cheap filter absorbing the high-confidence majority, expensive LLM handling the ambiguous tail, graduated action driven by continuous scores rather than binary labels — is directly portable. Any team operating a high-throughput content or fraud pipeline that reflexively reaches for an LLM on every request should read this architecture as a cost and latency corrective. As with the multi-agent control structures emerging across the industry, safety infrastructure benefits most from explicit layering where each component does only the work it is economically and technically suited for.