Retrieval-augmented generation is the standard way to make a language model answer questions about documents it was never trained on. The mechanism is straightforward: find the relevant passages, put them in the prompt, ask the model to answer using them. Nearly every implementation that disappoints does so at the first step.
Chunking decides more than it should
Documents have to be split before they can be indexed, and how they are split determines what can be found. Cut too small and a passage loses the context that made it meaningful. Cut too large and each result is mostly irrelevant text that costs tokens and dilutes the model's attention. Splitting on structure — sections, headings, natural boundaries — beats splitting on a fixed character count almost every time, and almost every tutorial teaches the fixed count.
Semantic search is not always better search
Embedding-based retrieval finds text with similar meaning, which is excellent for vague questions and unreliable for precise ones. Ask about a specific product code, a person's surname or a section number and traditional keyword search frequently wins outright. The robust answer is to run both and combine the rankings; the common answer is to use embeddings alone because that is what the framework demonstrates.
What to do when nothing relevant is found
The failure mode that damages trust is the confident answer assembled from passages that had nothing to do with the question. A retrieval step needs a relevance threshold and a path for "I don't have anything on that", and the model needs to be instructed to use it. Answering from general knowledge when retrieval fails is precisely what the system was built to avoid.
Citations are the feature
Returning the passage each claim came from is what makes the output checkable, and checkability is the entire value proposition over asking the model directly. An implementation without citations has taken on all of the engineering complexity and kept none of the benefit.