Cloudflare's AI Gateway has crossed 2 trillion tokens served since its launch in late 2024, making it one of the largest AI infrastructure proxies in production. The milestone was announced alongside the launch of a model routing feature that Cloudflare positions as the most commercially significant addition to the platform since launch.
AI Gateway sits between applications and AI provider APIs (OpenAI, Anthropic, Google Gemini, Cohere, and others), providing caching, rate limiting, cost tracking, and now intelligent routing. The routing feature evaluates each incoming request against a configurable capability matrix and routes it to the cheapest model that can handle it within specified quality thresholds — similar in concept to how content delivery networks select the optimal edge node for each request.
In testing disclosed by Cloudflare, routing reduced API costs by 31% on average for a sample enterprise customer running a mix of simple classification tasks (routed to cheaper models) and complex multi-step reasoning tasks (routed to GPT-4o or Claude 3.5 Sonnet). The key insight is that many production AI workloads contain a significant proportion of simpler requests that are currently over-provisioned to expensive frontier models by default.
The caching layer remains one of AI Gateway's most-used features. For applications with repetitive queries — customer support bots that handle the same 200 questions repeatedly, or classification pipelines with overlapping inputs — caching eliminates redundant API calls entirely. Cloudflare reports that customers save an average of 22% on API costs through semantic caching alone, before model routing is applied.
Privacy-conscious enterprises can run AI Gateway in "zero-logging" mode, which processes requests without retaining prompt or response data. This addresses a key procurement objection from financial services and healthcare companies.
At Cloudflare's edge scale (350+ data centers globally), AI Gateway can also reduce latency by routing requests to AI provider endpoints geographically close to the edge node handling the user request.