Deepgram Brings Billing-Accurate Metrics to SageMaker AI Endpoints
In this article
Teams running Deepgram speech-to-text and text-to-speech models as SageMaker AI real-time endpoints have historically faced a hard observability gap: standard endpoint metrics report invocation counts and latency, but the questions that drive cost reconciliation and capacity planning — exactly what you're billed for, which features your traffic exercises, and what the inference engine is doing per GPU — stayed invisible inside the vendor container. Deepgram's Enhanced Metrics integration, available now on SageMaker AI, closes that gap through two distinct capabilities that require no agent, no sidecar, and no additional IAM permissions.
The timing matters for practitioners thinking about pipeline architecture as a competitive differentiator: visibility into how individual components consume resources is a prerequisite for any serious optimisation effort, and speech pipelines running at production scale on managed infrastructure are no exception.
Billing Metrics Over the Logging Path
AWS Marketplace model packages run under network isolation — the container cannot make outbound connections. Deepgram routes around this by writing CloudWatch Embedded Metric Format (EMF) records to container stdout. SageMaker AI already forwards container output to the endpoint's CloudWatch log group, and CloudWatch Logs extracts EMF records into metrics automatically. No new network path is opened.
The result is two CloudWatch namespaces. Deepgram/SageMakerInference emits one record per completed request — streaming STT sessions, pre-recorded requests, and TTS requests — and its ConsumedUnits metric carries the same billable-unit values that drive AWS Marketplace metered billing. Dimensions are published at three granularities: [Category], [Category, Model], and [Category, Model, Transport], so a query can answer both "what did streaming STT cost this month" and "how much of that was nova-3" from the same namespace. Alongside ConsumedUnits, the namespace publishes AudioDurationSeconds for STT requests and CharCount for TTS. The billing stream cannot be disabled; it is part of the metering pipeline.
The second namespace, Deepgram/SelfHosted, is emitted by the Deepgram API server independently of billing and answers product-level utilisation questions. It is on by default and can be disabled with a single environment variable override (DEEPGRAM_API_01: emf.enabled=false).
Deepgram/SelfHosted breaks usage down by method, model tier, and enabled feature: AudioMs and Requests dimensioned by Method (streaming versus pre-recorded); TierAudioMs dimensioned by Tier (for example, nova-3 or flux); FeatureAudioMs and FeatureTokens dimensioned by Feature (diarization, smart formatting, redaction); and TtsCharacters, Tokens, and VoiceAgentMs for TTS and voice-agent traffic.
Both namespaces aggregate across all Deepgram endpoints in an AWS account and Region — dimensions include category, model, and transport but not endpoint name or instance ID. Per-endpoint, per-instance, and per-GPU breakdowns require the Prometheus and OpenTelemetry path below.
Prometheus and OpenTelemetry: Per-GPU and Engine-Level Visibility
SageMaker AI detailed observability runs an AWS-managed OpenTelemetry Collector on each instance backing an endpoint. The collector scrapes the Deepgram container's Prometheus endpoint directly and also exports per-GPU and host-level metrics. Three sources publish to the CloudWatch OTel-compatible metric store:
| Source | Example metrics | What it enables |
|---|---|---|
| GPU (DCGM exporter) | DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED |
Per-GPU series on multi-GPU instances; a saturated device cannot hide behind a summed utilisation number |
| Host (node exporter) | node_cpu_seconds_total, node_memory_MemTotal_bytes |
Standard Prometheus node-exporter metrics for each instance |
| Deepgram engine | engine_active_requests{kind="stream"}, engine_estimated_stream_capacity |
Engine-reported concurrent-stream load and headroom, previously available only to self-hosted customers running their own collectors |
engine_estimated_stream_capacity is the Deepgram engine's own estimate of sustainable concurrent streams for the instance. Comparing it against engine_active_requests yields an engine-reported headroom signal for autoscaling decisions. Every series carries SageMaker resource labels including aws.sagemaker.endpoint.name, variant name, and instance ID, enabling single-endpoint or single-GPU filtering across a scaled-out fleet.
Detailed observability is on by default for newly created endpoints, publishing at a 60-second frequency. Existing endpoints require a new endpoint configuration with EnableDetailedObservability: true passed in MetricsConfig, followed by an update-endpoint call — a blue/green operation that keeps the endpoint in service. Because the OTel Collector runs on the host outside the model container, this capability also operates cleanly under AWS Marketplace network isolation. The metrics are queryable with PromQL from the CloudWatch console, via the standard Prometheus-compatible HTTP API authenticated with SigV4, or through Grafana and any other Prometheus-native tooling.
Availability
Deepgram Enhanced Metrics require no setup and begin flowing into CloudWatch the moment a Deepgram SageMaker AI endpoint serves traffic. Deepgram models currently listed on SageMaker AI — Nova, Flux, and Aura-2 — include a 14-day trial at no additional Deepgram cost, though AWS infrastructure charges for endpoint hosting, CloudWatch logs and metrics, and associated networking apply from deployment start.
The integration reflects a broader pattern noted in systems-engineering gains that rival raw scaling: instrumenting individual inference containers with billing-accurate and engine-level signals at the infrastructure layer — rather than bolting on external observability tooling — is a concrete architectural advantage. Deepgram's use of existing SageMaker logging channels and the managed OTel Collector to deliver this without relaxing network isolation suggests a replicable model for any vendor offering containerised model packages on managed inference infrastructure.