Relay – a self-hosted LLM gateway with smart routing and request pacing

Relay is a self-hosted AI gateway that sits between applications and model providers. It connects OpenAI, Anthropic, Bedrock, Vertex AI, Groq, Together AI, Fireworks, Ollama, LM Studio, and more, then routes requests through one API. The project describes itself as an AI gateway built for agents and the people behind them.

Relay provides routing, queueing, pacing and model cooldowns, fallbacks, guardrails, limits, usage tracking, cost tracking, logs, and a realtime view of every request. It can pace requests to avoid provider RPM and token limits, absorb bursts with queue-first routing, and keep preferred models in play instead of immediately dropping to weaker backups.

Run it yourself with no separate database or queue to manage. Relay ships as a single Go application with the dashboard built in. Applications talk to Relay through one OpenAI-compatible API, and Relay handles routing, capacity, fallbacks, guardrails, and visibility before the request reaches a model.

Community Edition can classify requests and use that context with its available routing capabilities. Hosted Relay adds stronger request classification and Smart Groups that route automatically based on intent, domain, complexity, and configured model assignments.