Routing LLM Requests Across 14 Providers with Auto Failover

Every AI app I shipped in 2024 had the same problem. One provider goes down, or rate-limits, or quietly degrades a model, and the whole product breaks. Switching providers in code is easy. Switching providers in production without dropping requests is hard. That gap is why I built LLM Router.

OpenAI-compatible as the contract

The router exposes the OpenAI chat completions API. Any client written against OpenAI works unchanged. Internally, it translates the request to whichever provider actually serves it. This means a team can point their existing SDK at the router and get failover, routing, and observability for free, without touching application code.

Compatibility is not a feature. It is the entire value proposition. If clients have to change code to use your router, they will not.

Routing strategies

Different workloads want different routing. A cheap summarization job should go to the cheapest provider. A customer-facing chat reply should go to the lowest-latency one. A coding task should go to whichever model scores best on the relevant benchmark. The router supports weighted, latency-priority, cost-priority, and model-pinned strategies, configurable per request via a header.

• weighted: distribute load across providers by configured weight.
• latency: pick the provider with the lowest p50 over the last 5 minutes.
• cost: pick the cheapest provider that serves the requested model.
• pinned: send to one explicit provider, failover only on error.

Failover without retry storms

Naive failover retries the same request on the next provider. If the request is large or streaming, you have already spent tokens and time. The router tracks per-provider health with a circuit breaker: a provider in open state is skipped for a cooldown window. This prevents a single bad provider from cascading retries across the whole pool.

function selectProvider(
  candidates: Provider[],
  strategy: Strategy,
  health: HealthStore,
): Provider | null {
  const live = candidates.filter((p) => health.state(p.id) !== "open");
  if (!live.length) return null;
  return rankByStrategy(live, strategy)[0] ?? null;
}

Observability is the actual product

The routing logic is not that complicated. The value is in knowing what happened. Every request logs provider, model, latency, tokens, cost, and outcome. A dashboard shows p50/p95 per provider, error rate, and cost per million tokens. Once you can see provider behavior, the routing decisions almost make themselves.