Caching Changed the Economics of Prompting More Than Any Model Release

Serving a language model has a cost structure that is not obvious from the outside. Processing the input is a different kind of work from producing the output, and a large share of the input is frequently identical from one request to the next: the system instructions, the tool definitions, the retrieved documents, the reference material that frames a conversation. Caching the intermediate state for that repeated prefix, so that it is computed once and reused, turns a substantial recurring cost into an occasional one. The consequence for system design runs against a habit teams spent the previous two years acquiring. The instinct has been to make prompts short, because input was billed by volume and long prompts were expensive. Under caching, brevity is no longer the objective — stability is. A long instruction block that never changes can be cheaper in practice than a short one that is rebuilt per request, because the long stable one is paid for once and the short volatile one is paid for every time. The design rule that follows is to order a request from most stable to least: fixed instructions first, then slowly changing context, then the user's actual input last. This quietly penalises patterns that look tidy. Injecting the current timestamp, a request identifier, or a personalised greeting near the top of a prompt invalidates everything after it, and the cost of that mistake is invisible in testing and material in production. The same applies to reordering retrieved documents on every call, or to templating that interleaves stable and volatile fields for readability. These are not bugs in any conventional sense — the output is correct — which is why they persist until someone reads an invoice carefully. The limits deserve stating plainly, because caching is often described as though it were free. Cached state occupies memory and is evicted, so the benefit depends on traffic patterns; a workload with many distinct prefixes used rarely gains little. Caches are typically scoped in ways that matter for multi-tenant systems, and anyone sharing a prefix across customers should understand exactly what the scope is before assuming isolation. And the saving is on input processing, so a workload dominated by long generated outputs sees less of it than the headline figures suggest.