Long Context Did Not Kill Retrieval, and the Reason Is Mostly Economics

Each substantial expansion of model context windows has been accompanied by the argument that retrieval-augmented generation is a transitional hack: if the entire corpus fits in the prompt, why maintain an embedding pipeline, a vector index, and a chunking strategy? The argument is coherent, and for small corpora it is simply correct — a team whose knowledge base is a few hundred pages should put it in the prompt and skip the infrastructure. What has kept retrieval in place for larger deployments is not model capability but the cost and latency structure of attention over long inputs. The economics are straightforward and they compound. Sending a large corpus with every request means paying to process it with every request, and the growth is not free even where prompt caching offsets repeated prefixes, because caching helps most when the prefix is stable and helps least in exactly the multi-tenant, frequently-updated situations enterprise deployments tend to be. Retrieval inverts the arrangement: the indexing cost is paid once at write time, and each query processes only the small subset of material relevant to it. For a corpus of any real size queried at any real volume, the difference is not marginal. Quality behaves less simply than either camp claims. Long context genuinely outperforms retrieval when a question requires synthesis across many parts of a corpus at once, because retrieval can only reason over what it retrieved, and a chunking strategy that fragments a document can destroy the relationships the question depends on. Retrieval outperforms long context when the corpus contains material that is contradictory, outdated, or irrelevant, because filtering happens before the model sees it rather than relying on the model to ignore distractors — an ability that degrades as the volume of distractors grows. Neither is a general winner. What production systems have largely converged on is retrieval that returns considerably more material than earlier pipelines did, exploiting large context windows to be less aggressive about filtering rather than to eliminate filtering. Retrieve generously, keep whole documents rather than fragments where they fit, let the model do the final selection. That design keeps the cost profile of retrieval while removing its most damaging failure mode, which was never that the model could not handle the material — it was that the retriever silently declined to supply the passage that contained the answer.