Model routing
In short
Model routing is when an AI system automatically picks which model should answer each request, sending easy questions to a cheap, fast model and hard ones to a more capable, expensive one. Its main aim is to cut cost without hurting quality.
- Also called
- LLM routing, AI router, Model router, LLM cascade
- Topic
- The AI business
- Level
- Some background helps
- First described
- An early proposal was FrugalGPT by Lingjiao Chen, Matei Zaharia and James Zou, posted in May 2023, which framed choosing between paid language models as a cost problem and introduced the LLM cascade1.
AI models vary a lot in price. On OpenAI's API, GPT-6 Astra costs $10 per million input tokens while GPT-6 Luna costs $0.104. Most requests do not need the most powerful model, so sending everything to it wastes money.
A router sits in front of several models. It looks at each request, often with a small classifier model, and predicts which model can handle it well enough. A related approach, the cascade, tries a cheap model first and only escalates if the answer looks weak1.
Research backs the savings. The FrugalGPT paper reported matching GPT-4's performance with up to 98% lower cost on its test tasks1. RouteLLM trained routers on human preference data and cut costs by more than half in some cases without lowering response quality2.
You meet routing in everyday products. OpenAI described GPT-5 as one system with a fast model, a deeper reasoning model and "a real-time router" that picks between them based on conversation type, complexity and explicit intent3. Cursor's Auto setting runs a classifier on each coding request and picks "the most cost-effective model that still produces comparable quality"5.
Routing is not the same as a mixture of experts, where a small router inside one model sends each token to a few of its internal "expert" blocks6. Model routing chooses between whole, separate models.
Example
A customer-service bot might send "What are your opening hours?" to a small, cheap model, but pass a complaint about a disputed refund to a stronger reasoning model.
Why it matters
It is one of the main ways AI companies keep costs down as usage grows. It also means the model answering you may change from one question to the next without you being told.
Questions people ask
Does ChatGPT use model routing?
Yes. OpenAI's GPT-5 system card describes a real-time router that decides between a fast model and a deeper reasoning model based on the conversation, task complexity, tool needs and requests such as "think hard about this"3.
How much money does model routing save?
It depends on the task mix. Research results range from more than halving costs in some RouteLLM tests2 to up to 98% savings in FrugalGPT's experiments1. Real products report less dramatic gains, and savings fall if most requests genuinely need a top model.
Is model routing the same as mixture of experts?
No. Mixture of experts happens inside one model: in Mistral's Mixtral, for example, each layer picks two of eight expert blocks for every token6. Model routing chooses between entirely separate models, often from different companies.
Related terms
Tools that use it
- Cursor RouterCursor system that sends each coding request to the cheapest model able to handle it well
- Ramp RouterRamp's single API for many AI models that sends each request to the cheapest model that fits
Sources
- Chen, Zaharia and Zou (2023), FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, arXiv:2305.05176
- Ong et al. (2024), RouteLLM: Learning to Route LLMs with Preference Data, arXiv:2406.18665
- OpenAI (2025), GPT-5 System Card, section 1
- OpenAI API documentation, Pricing
- Cursor documentation, Cursor Router
- Jiang et al. (2024), Mixtral of Experts, arXiv:2401.04088