Retrieval-Augmented Generation: How RAG Works and Where It Fails

Retrieval-augmented generation (RAG) combines a language model with a search step. Instead of relying only on the model’s general training, the application finds relevant material and includes it in the model’s context before asking for an answer. Google Cloud’s RAG overview describes this combination of retrieval and generation.

RAG can make an answer easier to ground in a supplied source, but it does not ensure the source is correct, current, authorized or relevant. The application still needs to check access rights, search quality and the model’s response.

The stages

1. Prepare sources. Identify which documents are authoritative, who can access them and how updates or deletions should be handled.
2. Index them. Split documents into useful passages, retain their source and version, and create a searchable index. Search may use keyword matching, vector similarity or a combination.
3. Retrieve. Use the question and the user’s permissions to select relevant passages. A document the user cannot access should not be included simply because it matches well.
4. Generate. Ask the model to answer using the returned material and make clear what to do when the material is insufficient.
5. Check. Show or record source references where appropriate, and test whether the answer is supported by the actual passage.

Worked example: an internal policy assistant

Imagine an employee asks, “How do I request a new laptop?” The index contains a current equipment policy, an old policy and a general IT handbook. The assistant should first filter documents to those the employee is allowed to view. It should favor the current policy, retrieve the section describing the request process and return a concise answer with a link to that section.

Now change the scenario: the current policy does not specify the approval route. The correct result is to say that the available source does not answer that part and direct the employee to the responsible team. A plausible but invented approval step would be a failure. If an old policy is retrieved instead, the fix may be in the index or source lifecycle rather than in the prompt.

A practical test set

Build a small, reviewed set of questions with expected source passages. Include an answer found in one paragraph, a question spanning two documents, a question with no answer, a recently updated policy and a document the test user cannot access. For each case record both the retrieved passages and the final answer. This helps separate retrieval failure from an unsupported generation.

Treat retrieved documents as data, not trusted instructions. A malicious document can contain text that attempts to change the model’s behavior; OWASP explains this form of prompt injection. Validate actions outside the model and avoid granting write permissions the task does not require.

RAG is useful when a task needs answers tied to a changing document collection. It may be unnecessary for a small, stable FAQ with a reliable keyword search. It also cannot compensate for missing source material or poor access control.