When enterprise teams first started integrating large language models into production systems in 2023, the dominant question was: should we fine-tune a model on our data, or retrieve relevant context at inference time and feed it to a general model? Two years later, the answer from most production deployments is clear: retrieval-augmented generation (RAG) won the first round, but fine-tuning is staging a comeback in specific, well-understood use cases.
RAG's advantages are practical. You don't need to retrain a model every time your documents change — you just update the vector database. You get citations, which are essential for regulated industries. The cost of fine-tuning and hosting a custom model is significant; RAG on top of an API like GPT-4o or Claude costs a fraction. And for the most common enterprise use case — answering questions from internal documents, product catalogs, or customer support knowledge bases — RAG performs well enough that fine-tuning offers diminishing returns.
The stack that most enterprises have settled on looks roughly like this: a general-purpose LLM from Anthropic, OpenAI, or Google accessed via API, a vector database (Pinecone, Weaviate, or pgvector inside their existing Postgres instance), a chunk-and-embed pipeline for documents, and a retrieval layer that pulls the three to five most relevant chunks before each generation call. For teams without ML expertise, platforms like Langchain, LlamaIndex, and Cohere's enterprise offering abstract most of this complexity.
Where fine-tuning is winning: tone and format consistency for high-volume content generation, domain-specific classification tasks where precision matters more than fluency, and code generation tuned to proprietary frameworks. Salesforce has fine-tuned its own CodeGen models for Apex code (its proprietary language) because general-purpose models trained on Python and JavaScript consistently make Apex-specific errors. Similarly, Bloomberg's BloombergGPT represents a finance-domain fine-tune that outperforms general models on financial NER and sentiment tasks.
The emerging middle ground is "PEFT" (parameter-efficient fine-tuning), particularly LoRA (Low-Rank Adaptation), which lets you fine-tune a model on a laptop using a fraction of the compute of full fine-tuning. LoRA adapters are increasingly being shared as model "plugins" — you run the base model and load a small adapter for a specific task. This is making fine-tuning accessible to teams that couldn't previously justify the infrastructure cost.
For most enterprise teams starting their LLM journey today, the advice from practitioners is consistent: start with RAG, measure whether your use case is retrieval-bound or knowledge-bound, and only invest in fine-tuning once you have a clear gap that RAG cannot close. The tooling for RAG has matured significantly, and the cost per query continues to fall as model providers compete on efficiency.