RAG, short for retrieval-augmented generation, is a way of making an AI chatbot look things up before it answers. Instead of replying only from what the model absorbed during training, the system first searches a chosen set of documents, such as the web, a company’s policy files or a legal database, and places the most relevant passages in front of the model. The model then writes its answer from those passages, often with citations. It is how most “chat with your documents” and AI search tools work today.
The problem RAG solves
A large language model learns from a huge body of text, then its knowledge is fixed. Our guide to how large language models work explains that training process. Three gaps follow from it:
- Frozen knowledge. The model knows nothing that happened after its training data was collected, and retraining is expensive.
- No access to private material. It has never seen your company’s HR policy, your insurer’s terms or last week’s board minutes.
- Plausible guessing. When it lacks a fact, it may still produce a fluent answer. We cover why in why AI chatbots “hallucinate”.
The original RAG paper, by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University, set out the same gaps. It noted that a model’s ability to “access and precisely manipulate knowledge is still limited”, and that “providing provenance for their decisions and updating their world knowledge remain open research problems”. Their answer, presented at NeurIPS 2020, paired a language model with a searchable index of Wikipedia. They found it generated “more specific, diverse and factual language” than the same kind of model working from memory alone.
How RAG works, step by step
Most RAG systems follow the same pipeline. The first three steps happen in advance; the last three happen each time you ask a question.
- Chunking. The documents are split into small pieces. Anthropic describes chunks as “usually no more than a few hundred tokens”, roughly a few paragraphs each.
- Embedding. Each chunk is passed through an embedding model, which turns text into a list of numbers that captures its meaning. OpenAI’s documentation defines an embedding as “a vector (list) of floating point numbers” and explains that “the distance between two vectors measures their relatedness”. Passages about the same idea end up close together, even if they use different words.
- Storing. The vectors go into a vector database, built to find the nearest vectors quickly. These can be specialist products or extensions to ordinary databases: pgvector, for example, adds “vector similarity search” to the widely used PostgreSQL database.
- Retrieval. When you ask a question, it is turned into a vector in the same way, and the system fetches the chunks closest to it. Many systems also run a traditional keyword search alongside and re-rank the combined results.
- Prompt assembly. The top chunks are pasted into the model’s prompt with your question and an instruction along the lines of “answer using only these passages and cite them”.
- Generation with citations. The model writes its answer. Good tools link each claim to the passage it came from, so you can check it.
The key point is that the model itself is unchanged. RAG changes what goes into the model’s working memory, its context window, for each question.
RAG vs fine-tuning vs long context
RAG is one of three common ways to make a model useful on specific material. They solve different problems.
| RAG | Fine-tuning | Long context | |
|---|---|---|---|
| What it does | Searches documents and adds relevant passages to each prompt | Further trains the model on examples | Puts whole documents into the prompt |
| Best for | Large, changing or private document sets; answers that need sources | Consistent style, format or a narrow repeated task | A few documents that fit within the model’s limit |
| Updating knowledge | Add or replace documents in the index | Retrain | Paste in the new version |
| Shows sources | Yes, if the tool is built to | No | Can quote, if asked |
| Main weakness | Fails if the search misses the right passage | Poor way to add new facts; costly to redo | Cost per question; models can miss details in long inputs |
OpenAI’s developer guidance frames fine-tuning as a way to improve accuracy on “a specific task” by showing the model many examples of it done correctly, while RAG gives the model “access to domain-specific context” it lacks. It even gives a case where adding RAG to a fine-tuned task made accuracy worse: the right tool depends on the problem.
Long context is a real alternative for small collections. Anthropic’s engineering team wrote in September 2024 that if a knowledge base is smaller than 200,000 tokens, “about 500 pages of material”, you can include all of it in the prompt “with no need for RAG”. The trade-off is that you pay to process all of it on every question, and a 2023 study by Liu and colleagues, “Lost in the Middle”, found that model performance “significantly degrades” when the relevant information sits in the middle of a long input.
Where RAG fails
Retrieval reduces errors but does not remove them. A 2024 study of AI legal research tools, whose makers promoted RAG as the way to curb fabrication, found they still hallucinated between 17% and 33% of the time on the researchers’ test questions. The common failure points:
- The search misses. If the right passage is not retrieved, the model answers from the wrong passages or from memory. Anthropic gives the example of a chunk that says “The company’s revenue grew by 3% over the previous quarter” without saying which company or quarter, so it is hard to find and easy to misuse.
- Bad or outdated sources. RAG faithfully repeats what is in the index. An old policy document left in the collection produces an old answer.
- Misreading the passage. The model can still summarise a correct passage wrongly, or blend it with what it “remembers”.
- Citations that do not support the claim. A link can be real while the sentence it is attached to says something the source does not.
- Security and privacy. The OWASP security project lists vector and embedding weaknesses among its top risks for AI applications in 2025, including weak access controls that expose sensitive documents, leaks between users of a shared database, and “data poisoning” through planted content. Text hidden in a retrieved web page or file can also act as an instruction to the model, a risk that grows when the system can take actions, as our explainer on AI agents describes.
For Indian organisations, the privacy point is practical. A bank or insurer that indexes customer emails or claim files into a RAG chatbot is processing personal data, so the duties set out in India’s data protection law apply; see our DPDP Act explainer.
How to tell if an AI tool is using retrieval
You usually cannot see inside a product, but there are reliable signs:
- It shows sources. Numbered citations or links next to specific sentences usually mean passages were retrieved. Google’s Gemini API, for instance, returns answers grounded in search with annotations that “link a text segment” to a source URL, along with the search queries it ran.
- It shows a “searching” step. Many tools display the queries they are running or the files they are reading.
- It knows recent or private facts. If it correctly answers about last week’s news or a file you uploaded, it is retrieving rather than recalling.
- It says when it found nothing. A well-built RAG tool tells you the documents do not cover your question instead of guessing.
None of these proves the answer is right. Open the sources and check that they say what the answer claims, especially for figures, rules and dates.
The point: RAG makes an AI tool search a chosen set of documents and answer from what it finds, which keeps answers current, lets them use private material and makes them checkable. It does not make the model infallible: if the search misses or the source is wrong, so is the answer. Treat citations as an invitation to verify, not as proof.
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Lewis et al., NeurIPS 2020
- Optimizing LLM accuracy, OpenAI developer documentation (accessed October 2026)
- Vector embeddings, OpenAI developer documentation (accessed October 2026)
- Introducing Contextual Retrieval, Anthropic, 19 September 2024
- pgvector: open-source vector similarity search for Postgres, pgvector project documentation
- Grounding with Google Search, Google Gemini API documentation (accessed October 2026)
- Lost in the Middle: How Language Models Use Long Contexts, Liu et al., Transactions of the Association for Computational Linguistics, 2023
- Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Magesh et al., arXiv, May 2024
- LLM08:2025 Vector and Embedding Weaknesses, OWASP Gen AI Security Project