Routing Beat Scaling for Most Production Workloads

The default architecture for a production feature built on language models is to pick the strongest model available and send everything to it. It is a reasonable starting point: it maximises quality, it is trivial to implement, and it removes a decision. It is also, for most real workloads, a considerable overpayment, because the distribution of requests hitting any deployed system is heavily skewed. A large share is routine — classification, extraction, short reformulations, questions answered directly by retrieved text — and a small share is genuinely hard. Routing exploits that skew by choosing capability per request. In the simplest arrangement, a small fast model handles the request and a larger one is invoked only when the first is unlikely to be sufficient, judged either by the nature of the request before it runs or by the confidence of the result after it does. The economics are appealing precisely because of the skew: if most traffic is easy, most traffic can be served cheaply, and the expensive path is reserved for the cases that justify it. Latency improves alongside cost, which is rare enough to be worth noting, since the common case gets shorter rather than longer. The engineering difficulty is concentrated entirely in the decision. A router that is too eager to escalate captures none of the saving while adding a stage; one too reluctant degrades quality in a way that is easy to miss, because the failures are distributed across a minority of requests and each looks like an ordinary bad answer rather than a systemic fault. Confidence signals from models are famously unreliable for this purpose — a wrong answer is often delivered with the same assurance as a right one — so escalation triggers built on stated confidence alone tend to disappoint. What works in practice is unglamorous: route on observable properties of the request rather than on the model's self-assessment, verify the cheap path's output against something external where the task allows it, escalate on verification failure rather than on a feeling, and hold out a sample of traffic that always goes to the strong model so the quality gap is measured continuously rather than assumed. Without that control sample, a routing layer degrades silently, and the saving it reports is real while the cost it moved is invisible.