Inference Is Moving Back Onto the Device, and the Reason Is Arithmetic

The default architecture for AI features has been to send the request to a hosted model and return the answer. It is the simplest thing to build, it gives access to the most capable models available, and it made obvious sense while the alternative — running a model on the user's own hardware — meant accepting dramatically worse output. Both halves of that comparison have shifted enough to make the default worth re-examining for a specific class of work. The arithmetic is straightforward. A feature that fires on every keystroke, every message, every photo taken or every document opened generates request volumes where per-call pricing compounds into a material line item, and where network round-trip latency is perceptible in a way that makes the feature feel worse than not having it. For those workloads — classification, extraction, ranking, redaction, transcription, short completions — the required capability is well within what compact models running on current phone and laptop silicon deliver, and the marginal cost of an additional inference on hardware the user already owns is zero. Privacy compounds the case rather than driving it alone. Processing that never leaves the device removes an entire category of question about data handling, cross-border transfer and retention, which matters most in exactly the health, legal and financial contexts where per-request costs are also least tolerable. The compliance argument and the cost argument point the same direction, which is unusual and is part of why the shift has momentum. The architecture that appears to be settling is not local-versus-hosted but a routing decision made per request: handle the high-frequency, low-complexity majority locally, escalate the genuinely hard minority to a hosted model, and treat the split point as something to be measured rather than assumed. The engineering difficulty moves to the router — deciding what is hard is itself a judgement — but the resulting system is cheaper, faster and less exposed than one that sends everything to a server, and better than one that sends nothing.