Choosing an LLM Routing Strategy

The AI Gateway sits in front of your model providers and decides which model handles each request. Your routing strategy is the rule it uses to pick — optimizing for reliability, cost, speed, or quality depending on what matters most for a given workload.

How routing works

Every request that hits the gateway is matched against the models in your routing pool. The strategy you select determines the order in which candidates are tried, and failed or rate-limited providers are automatically skipped so traffic keeps flowing.

The strategies

• ▸Priority — Always tries your models in a fixed, ranked order, falling through to the next only when one is unavailable. Best when you have a clearly preferred model and want predictable, deterministic behavior.
• ▸Least-cost — Routes to the cheapest capable model first, using each provider's per-token pricing. Ideal for high-volume, cost-sensitive traffic where any model in the pool is good enough.
• ▸Least-latency — Sends each request to the model with the fastest recent response time, based on recent observed response times. Best for interactive, user-facing experiences where responsiveness wins.
• ▸Semantic — Attempts to match each request to a model suited for that kind of work (for example, code, summarization, or reasoning) based on the prompt's content. Useful for mixed workloads where request types vary widely.
• ▸Round-robin — Distributes requests evenly across the pool to spread load and utilization. Useful for balancing quota across multiple providers.
• ▸Weighted — Splits traffic by percentages you assign, so you can send, say, 90% to one model and 10% to another. Handy for gradual rollouts and A/B testing.

Choosing one

1. 1.Start with priority if you have one trusted model and want fallbacks for resilience.
2. 2.Switch to least-cost or least-latency once you know whether budget or speed is the constraint.
3. 3.Reach for semantic when a single pool serves many different request types.
4. 4.Use weighted to test a new model safely before promoting it.

Next, see Configuring providers and keys and Setting spend limits.