Context windows of 100,000 tokens or more have become the baseline expectation in enterprise AI model evaluations, a threshold that would have been considered exceptional just 18 months ago. The shift reflects a maturation of long-context use cases from experimental to production, with enterprise buyers now treating large context capacity as a table-stakes requirement rather than a differentiating feature.
The proliferation of long-context models — Claude's 200K context, Gemini's 1M context, GPT-4's 128K context — has enabled workflows that were previously impractical: analyzing an entire codebase for security vulnerabilities, summarizing a full legal case file with all supporting documents, or reviewing a complete set of financial statements and footnotes for a due diligence analysis. These are tasks where the quality improvement from seeing the full document set outweighs the cost premium of long-context inference.
The economics of long-context inference have also improved. The "KV cache" efficiency problem — which historically made long-context inference disproportionately expensive because each token attended to all prior tokens — has been partially addressed through sparse attention mechanisms, prompt caching features (where repeated system prompts and document context are cached at reduced cost), and better KV cache management. Claude's prompt caching feature, in particular, has made long-document workflows substantially more cost-effective for applications that reuse the same large context across multiple queries.
Practical use of 1M token contexts remains constrained by performance rather than availability. Retrieval quality within very long contexts degrades for some models and some query types — the "lost in the middle" problem where information in the center of a very long context is less reliably retrieved than information at the beginning or end. This performance characteristic means that for many applications, a well-designed retrieval-augmented generation system outperforms a naive 1M context approach, despite the latter's greater simplicity.
The competitive pressure from long-context capabilities is driving models toward better retrieval performance throughout the context, with several labs publishing research on attention modifications that improve middle-of-context retrieval accuracy.