NVIDIA MPS Cuts ASR GPU Count from 16 to 4 on EC2

Heidi Health processes over 2.4 million clinical consultations per week across 190 countries, and a structural GPU waste problem sits at the heart of its infrastructure bill. A single inference request against the NVIDIA Parakeet TDT 0.6B V2 model consumes only 15–20% of an L40S GPU's 142 streaming multiprocessors. CUDA's default time-slicing serialises access, leaving the remaining 80% idle while processes context-switch. At acceptable latency thresholds—mean below 650 ms and p99 below 1,000 ms—one GPU tops out at roughly 62 requests per second, forcing 16 GPU instances to meet peak-traffic SLAs. A joint AWS, NVIDIA, and Heidi engineering effort now demonstrates that NVIDIA CUDA Multi-Process Service (MPS) on Amazon EC2, combined with NVIDIA Triton Inference Server, reduces that instance count to 4—a 75% infrastructure cut that holds those same latency SLAs.

Why MPS Beats the Alternatives

Three GPU-sharing mechanisms exist on NVIDIA hardware. This is the same systems-level reasoning that now rivals pure scaling gains in squeezing performance from fixed hardware budgets.

MPS funnels all CUDA work through a single daemon-managed GPU context, eliminating context-switch overhead while allowing kernels from different processes to execute simultaneously on separate streaming multiprocessors. Partition size is set via the CUDA_MPS_ACTIVE_THREAD_PERCENTAGE environment variable with no application code changes. For transcription, four concurrent MPS instances run at 25% SM each, consuming approximately 2.5 GB of the L40S's 48 GB VRAM per instance. Diarization uses eight instances at 12% SM, approximately 1.8 GB each, on a separate partition without contention.

Architecture: Three Layers on a Single EC2 Instance

The stack runs on g6e.4xlarge and g7e.4xlarge instances (NVIDIA L40S, 48 GB VRAM) with three containerised components orchestrated via Docker Compose. A FastAPI gateway—four uvicorn workers on port 8002—decodes uploaded audio (WAV, WebM/Opus, MP3, M4A, FLAC) via torchcodec into raw 16 kHz mono float32 tensors, then forwards to Triton over gRPC, keeping all audio decoding on CPU. The gateway exposes an OpenAI Whisper-compatible REST API (POST /v1/audio/transcriptions).

Triton manages two batching strategies. Transcription uses dynamic batching with preferred batch sizes of 4, 8, and 16 and a maximum queue delay of 50,000 microseconds (50 ms). Diarization uses sequence batching with per-recording state keyed by correlation ID; sessions expire after 600 seconds of inactivity.

The Triton backend calls model.forward() directly rather than routing through NeMo's model.transcribe(), removing approximately 50 ms of framework overhead per request. Combined with bfloat16 autocast and a dedicated CUDA stream per MPS instance, a single instance processes 45-second audio in approximately 160 ms. Model initialisation is serialised with fcntl.flock to prevent four simultaneous 600M-parameter loads from exceeding GPU memory. CUDA graph warmup pre-caches shapes at 5, 15, 30, 45, and 60 seconds (batch size 1) plus batch size 2 at 61 seconds; shapes within that envelope replay at approximately 165 ms, while novel shapes fall back to eager execution at approximately 500 ms. A wedge sentinel mechanism monitors for MPS instances rendered unrecoverable by CUDA errors, writing to tmpfs (/tmp/parakeet_wedged) for health-check detection.

Benchmark Results

The benchmark sweeps concurrency from 1 to 100, averaging five measurement rounds on clinical consultation audio. The SLA gate is mean latency below 650 ms AND p99 below 1,000 ms.

On g6e.4xlarge with Triton + MPS, the optimal operating point is concurrency 28: 60.8 RPS, 470.5 ms mean, 947.1 ms p99—inside the SLA envelope. Concurrency 32 pushes p99 to 1,169.1 ms and fails. This supports a recommendation of 4 GPUs versus the current 16, a 75% reduction.

On g7e.4xlarge with Triton + MPS—the selected production path—the optimal point is concurrency 32: 92.1 RPS, 352.5 ms mean, 768.8 ms p99. The g7e delivers over 51% more throughput and lower mean latency than the g6e at the same 4-GPU recommendation. Single-request latency reaches 121.0 ms p50 at concurrency 1.

Adding TensorRT + ONNX acceleration—routing the compute-heavy 24-layer, 1024-hidden-dimension Conformer encoder through TensorRT FP16 kernel fusion while keeping the TDT decoder in PyTorch CUDA—shifts the optimal point to concurrency 64: 111.6 RPS, 590.3 ms mean, 895.7 ms p99, reducing the instance recommendation to 2 GPUs from 16, an 88% reduction. The tradeoff is a longer deployment pipeline requiring ONNX re-export after each fine-tuning cycle.

For diarization, TensorRT warmup optimisation on the NVIDIA Streaming Sortformer 4-speaker v2 model reduces mean latency from 309.04 ms to 238.73 ms (−23%) and p99 from 499.45 ms to 389.21 ms (−22%). Standard deviation drops from 12.62 ms to 7.13 ms (−44%), tightening latency predictability. A 60-second consultation processed as four 15-second chunks completes in under 1 second total at 238 ms mean per chunk, a real-time factor of 0.016×.

Inference Economics

The result fits a broader pattern: pipeline architecture and systems engineering are generating efficiency gains that rival switching to a newer model. The MPS partitioning approach, direct forward-pass pattern, CUDA graph safety mechanism, and wedge sentinel are model-agnostic—the authors confirm the same architecture has been validated against NVIDIA Canary and OpenAI Whisper large-v3 checkpoints. Any encoder-decoder workload where individual requests consume a small fraction of available GPU compute is a candidate for the same treatment. The complete implementation—Dockerfiles, Triton model configurations, the FastAPI gateway, and benchmark scripts—is available in the accompanying GitHub repository and deploys to any EC2 GPU instance with three commands.