Small Models on the Device Are Winning the Jobs Where the Data Cannot Leave

The public conversation about AI has been dominated by the largest models, which are extraordinary and which run on infrastructure only a handful of operators own. Underneath them, quietly, a different class has become genuinely useful: models small enough to run on a laptop, a phone or a modest on-premises server, doing narrow work well rather than everything adequately. For summarising a document, classifying a message, extracting fields from a form or drafting a routine reply, the gap between a small local model and a giant remote one has closed to the point where, for that task, it no longer decides the outcome. The reason to run locally is usually not cost, though it can be. It is that the data must not leave. A law firm, a clinic, a bank or any business handling regulated or confidential information faces a simple constraint: sending client records to an external service to be processed is often either forbidden or a liability nobody wants to sign off. A model that runs on the same machine as the data sidesteps the question entirely — nothing is transmitted, nothing is retained by a third party, and the residency and privacy obligations are met by architecture rather than by a contract clause and a hope. Latency and offline operation are bonuses: the model answers without a round trip, and it works when the connection does not. The tradeoff is capability, and it is real. A small model has a shorter memory, weaker general reasoning and less breadth than the frontier systems, so the discipline is scoping. The deployments that work assign the local model a job it can actually do — a narrow, well-defined task with clear inputs and a checkable output — and route the genuinely hard, open-ended work elsewhere, with the confidential parts stripped out first. Pretending a small model is a small version of a large one, rather than a different tool, is how these projects disappoint. There is an operational tail that the privacy win can obscure. A model running on many devices is software that has to be distributed, updated, versioned and monitored like any other, and the same data-residency logic that kept the inputs local also keeps the failures local and harder to observe centrally. The organisations doing this well treat the on-device model as part of their software estate, with a plan for how it is updated and how its mistakes are caught, rather than as a one-time install that quietly drifts out of date on ten thousand laptops.