NVIDIA's Switchyard Routes LLM Traffic Between OpenAI and Anthropic APIs

NVIDIA has open-sourced Switchyard, a Rust proxy and library that decodes, routes, and re-encodes LLM traffic across OpenAI and Anthropic wire formats under an Apache 2.0 license. The project targets a friction point that any team running multiple coding agents will recognise: Claude Code speaks the Anthropic Messages API, Codex CLI speaks OpenAI, and the model a team actually wants to serve sits behind vLLM, NVIDIA NIM, or Ollama. Rather than rewriting each agent, Switchyard intercepts traffic, converts it into provider-neutral Rust types, runs a routing algorithm to select a backend, re-encodes the request in that backend's own wire format, and translates the response — including streaming events — back into the format the client expects. Documentation lives at docs.nvidia.com/nemo/switchyard. NVIDIA labels the release pre-alpha and experimental, explicitly not for production, with the API and algorithms expected to change significantly before v1.0.

API Translation and Configuration

Switchyard accepts three inbound wire formats: OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Each configured LLM client declares one upstream format independently of what the client sends. That decoupling means any of the three inbound formats can address any configured route without the agent and the backend needing to agree on a protocol. Translation covers streaming events, not just unary responses.

A TOML deployment has three layers: llm_clients define base URL, wire format, credential environment variable, and retry policy; targets bind one upstream model ID to a client; routes expose one client-visible model ID and its algorithm. Secrets never appear in the file — api_key_env names an environment variable, and the proxy reads the credential at runtime. max_retries defaults to 2 and covers transport failures, timeouts, and HTTP 408, 429, and 5xx responses.

Three installation paths exist. uv tool install --python 3.12 "nemo-switchyard[cli]" delivers the launcher, which exposes switchyard launch claude, switchyard launch codex, and switchyard launch openclaw as named entry points. cargo install --locked switchyard-server installs the standalone proxy with a --dry-run config validator. The third path, switchyard-libsy, embeds the routing algorithms directly into a Rust application without owning an HTTP stack; the library never calls a model itself — it selects a target and returns that decision to the caller.

Routing Algorithms

A route exposes one client-visible model ID and one algorithm. Four algorithms ship with meaningfully different overhead profiles:

Strong, weak, capable, and efficient are roles assigned inside a route definition, not fixed properties of a model. The same upstream model can fill different roles in different routes. The llm_classifier path is directly relevant to teams building infrastructure governance layers for agent deployment, since it lets a lightweight judge gate access to expensive tiers rather than sending every request to the strongest available model.

Observability

GET /metrics returns Prometheus text from the server's process-wide OpenTelemetry provider. Metric families cover requests, errors, model-call latency, full-turn latency, prompt tokens, completion tokens, cached tokens, cache-creation tokens, reasoning tokens, and upstream HTTP attempts broken out by outcome and status code. A tier label carries strong or weak for distinguishable classifier decisions. Classifier calls are excluded from the request and token families, keeping cost attribution clean.

The operationally interesting metric is switchyard_routing_overhead_ms, which reports the algorithm's run time minus the model call that served the request. For passthrough and random, that number reflects the sub-millisecond cost of target selection. For llm_classifier routes, it captures the full classification latency, since classifier calls are not subtracted there. Histogram buckets start at 0.1 ms. Separately, --routing-log-file appends a JSON record per completed response, and GET /v1/routing/session-stats returns per-session call and token totals from that log — a more granular view than the aggregate Prometheus counters. This maps directly to the pipeline architecture patterns that have driven measurable cost reduction in multi-model deployments.

Context

Switchyard's release sits alongside Anthropic's acquisition of Stainless, which owns SDK generation tooling — a move that suggests the API compatibility layer between providers is becoming contested infrastructure. NVIDIA releasing an Apache 2.0 routing proxy that handles both sides of that boundary positions it at the translation layer without locking operators into a proprietary gateway. The pre-alpha label is honest: the API will change, and stage_router and llm_classifier are routing primitives that need production traffic to validate their tuning parameters. Engineers who can afford the evaluation overhead will find the routing overhead metric and per-session token accounting more operationally useful than most commercial gateway dashboards provide today.