Switchyard — NVIDIA's LLM Traffic Router and Translator
Imagine this scenario: you want to use a terminal autocomplete tool or coding agent like Claude Code or Codex CLI. It's a handy tool, but it's tailored to a specific API. If you want to redirect its requests to a local vLLM, Ollama, or NVIDIA NIM, you'll run into a problem: the endpoint formats and data structures won't match. Anthropic expects the Messages format, OpenAI uses Chat Completions, and your local model is tuned for a third format entirely.
The Switchyard project from the NVIDIA NeMo team solves exactly this problem. It's a proxy server and Rust library designed for API protocol translation and intelligent request routing across different language models.
What Switchyard Can Do
The project is built on two core mechanisms: on-the-fly request conversion and routing algorithms.
The client application continues to "think" it's communicating with the native Anthropic or OpenAI API. In reality, Switchyard receives the request, translates it into the target provider's format, sends it to the appropriate backend, and transforms the response back.
Currently, three main formats are supported:
- OpenAI Chat
- Anthropic Messages
- OpenAI Responses
This means you can point Claude Code at a local model via vLLM, and the client won't even notice the switch.
Smart Routing Instead of Plain Round-Robin
Any proxy server can balance load randomly. What makes Switchyard interesting is its traffic distribution scenarios. The authors built in four built-in algorithms:
- Stage Router. Evaluates signals within the dialogue itself. If a previous tool call ended in error, the router will redirect the next turn to a stronger model. If everything is going smoothly, a cheaper model will handle the request.
- Escalation Router. First sends the request to a weaker model. In parallel, a special judge model evaluates the response quality. If the result is unsatisfactory, the same request is automatically escalated to a flagship model.
- LLM Classifier. A separate lightweight model classifies incoming requests by complexity and selects the appropriate backend before main generation even begins.
- Random. Standard random distribution with a fixed split. Useful for A/B testing or evaluating costs across different providers.
Architecture and Integration Options
The developers structured Switchyard as several modules. The choice of operating mode depends on your task.
If you need to quickly launch a coding agent with the right settings, use the CLI launcher. The entire installation boils down to a single package manager command:
After that, the agent starts with a single line specifying the config:
To run as a network proxy, build the binary. It tracks metrics via Prometheus (token counts, latencies, errors) and is configured through a file.
If you're writing your own Rust application, you won't need to pull in a separate HTTP server. The crate embeds routing logic directly into your code. The library doesn't make network calls itself—it just makes a routing decision and returns the selected target backend to your application.
Current Project Status
The repository page carries a warning: Switchyard is in pre-alpha stage. At the time of this review, the project has close to 800 stars on GitHub and experimental software status.
This means the public API, TOML config structure, and internal crate architecture are subject to change. Using it right now in critical production is a risky move.
Who Should Keep an Eye on This Project
Switchyard looks promising for two categories of developers:
- Those who actively use terminal AI agents (Claude Code, Codex CLI) and want to save money by redirecting some simple tasks to local models or cheaper APIs.
- Teams building their own infrastructure around LLMs based on Rust. The ability to pull ready-made escalation algorithms and protocol translators from the crate will save a couple of weeks of work.
If you've been looking for a clean tool for transparent backend substitution without rewriting the agent's own code, this project is definitely worth following.
Related projects