On-device AI inference reached a practical threshold in mid-2026 that changes the economics of mobile AI applications. Qualcomm's Snapdragon 8 Elite, shipping in flagship Android devices since Q1 2026, now runs Llama 3.1 7B at 30 tokens per second with quantized weights — fast enough for real-time chat, code completion, and summarization without a cloud round-trip.
MediaTek's Dimensity 9400 achieves comparable throughput through a different architectural approach, using a dedicated NPU cluster that draws only 3.2 watts at peak AI load. For a phone running on battery, that power envelope means continuous AI inference without meaningful impact on daily use time.
The implications are significant for three categories of applications. First, privacy-sensitive use cases — medical symptom checking, legal document review, financial analysis — can now stay entirely on the device. Second, latency-critical applications like real-time translation and live caption generation benefit from eliminating the 80-to-200ms cloud round-trip. Third, developers in markets with unreliable connectivity (large portions of Southeast Asia, Sub-Saharan Africa, Latin America) can build AI features that actually work offline.
Apple's A18 Pro in the iPhone 16 Pro established the benchmark that both Android chipmakers are chasing: 38 tokens per second on the on-device model powering Apple Intelligence features. The gap has closed from roughly 3x eighteen months ago to under 30% today.
Model compression has enabled this shift as much as raw silicon performance. Techniques like INT4 quantization, weight sharing, and structured pruning allow 7B-parameter models that would have required 14GB of RAM at full precision to run in under 4GB — within reach of mainstream flagship devices.
The next frontier is 13B models on mobile. Current silicon handles this class only at reduced token rates (8-12 tokens per second), but roadmaps from both Qualcomm and MediaTek suggest crossing the 20 tokens-per-second threshold for 13B models by early 2027, which would bring near-GPT-4-class capability to devices without any cloud dependency.