A language model only knows what it saw during training, plus whatever you put in the prompt. If you want it to answer questions about your own handbook, product docs or support history, you have to get that text in front of it at the right moment. Retrieval-Augmented Generation (RAG) is the pattern that does exactly that.
The idea in one paragraph
Instead of asking the model a question directly, you first search your documents for the most relevant passages, then give the model both the question and those passages, and instruct it to answer using only that material. The model supplies language skills; your documents supply the facts.
The pipeline, step by step
Ingest your documents (PDF, HTML, Markdown, database rows) and extract clean text.
Chunk the text into passages small enough to be specific, but large enough to keep their meaning.
Embed each chunk: convert it into a vector that captures its meaning.
Store the vectors with their source metadata in a vector index or database.
Retrieve: embed the user question and find the closest chunks.
Generate: send the question and retrieved chunks to the model with clear instructions.
Chunking decides more than you expect
Most disappointing RAG systems fail at chunking, not at the model. Chunks that are too large dilute the match; chunks that are too small lose context. A good starting point is a few hundred words per chunk with a small overlap, split on natural boundaries such as headings and paragraphs rather than a fixed character count.
Keep the source title, section heading and page number with every chunk.
Split on structure (headings, paragraphs) before splitting on length.
Re-test chunk size on real questions, not on guesses.
Retrieval: embeddings plus a sanity check
Semantic search finds passages that mean the same thing even when the words differ. It is not magic, though. Exact identifiers such as error codes or product names are sometimes better served by keyword search, which is why many production systems combine both and re-rank the combined results.
# Simplified retrieval step
query_vector = embed(question)
matches = index.search(query_vector, top_k=5)
context = "\n\n".join(
f"[{m.source} p.{m.page}] {m.text}" for m in matches
)The prompt that keeps the model honest
The generation step needs firm instructions. Tell the model to answer only from the provided context, to say when the context does not contain the answer, and to cite the source labels you attached.
If the answer is not in the context, say so. A confident wrong answer is worse than "I could not find that."
Where RAG usually breaks
The right passage never gets retrieved
If retrieval misses, the model cannot recover. Log the retrieved chunks for every question during testing so you can tell whether a bad answer was a retrieval problem or a generation problem.
The answer spans several documents
Questions that need information from multiple places are harder. Retrieving more chunks helps, but so does restructuring documents so related facts live together.
Stale content
An index is a snapshot. Decide how often documents are re-ingested and how deleted or updated files are removed from the index.
A sensible way to start
Begin with a small, well-understood set of documents and a list of twenty real questions with known answers. Measure how often the right passage is retrieved before you tune anything else. Once retrieval is reliable, improving the prompt and adding citations becomes straightforward.
