How to Implement Generation in RAG
In the previous two posts I looked at retrieval - how we find the right documents, and augmentation - how we package those documents into a prompt. In this post I'll look at the last step: generation. This is where the LLM actually reads the augmented prompt and produces a response.
Generation is the step the user directly experiences. And, if retrieval and augmentation are done correctly, the LLM has everything it needs to produce a useful, grounded answer.
What is Generation?
Generation is the process of passing the augmented prompt to a large language model and receiving a response. The LLM doesn't have access to your data source, the retrieval index, or anything outside of what's in the prompt - it just reads the context it's been given and generates the most likely useful continuation.
The LLM can't "look things up" at generation time. Everything it needs to answer the question should already be in the prompt.
The quality of the generation step is therefore hugely dependent on the quality of retrieval and augmentation. A well-constructed prompt with the right context gives the highest chance of producing a coherent, accurate response. A poorly constructed one, filled with irrelevant documents, conflicting information, or vague instructions, will produce something much less useful.
How LLMs Generate Responses
LLMs generate text by predicting the most likely next token (a word or part of a word) given everything that came before it. This process continues token by token until the model decides the response is complete (or reaches a length limit).
The model was trained on a huge amount of text, which gives it a rich understanding of language, context, and common patterns of reasoning. In a RAG scenario, the system prompt and retrieved context bias the model towards generating a response that's consistent with the provided information, rather than drawing on general training knowledge.
It's important to understand that this is probabilistic, not deterministic. Two calls to the same model with the same prompt will produce different responses. This is controlled by a parameter called temperature...
Techniques for Effective Generation
Temperature and sampling
Temperature controls how "creative" or "random" the model's output is.
A temperature of 0 makes the model always pick the most likely next token, for example, if the preceding tokens made up "Hello, how are", then the model would always pick "you" as the next token (assuming that the text it was trained on contained the words "Hello, how are you", more than, say, "Hello, how are cats"). However, a model with non-zero temperature will often pick "you" but might sometimes pick "your".
A temperature of 0 means that the model produces mostly deterministic responses - given the same prompt, a model with temperature 0 will likely produce the same response. Higher temperatures introduce more randomness, leading to more varied (but potentially less reliable) outputs.
For RAG use cases - particularly those where accuracy is fundamentally crucial, like regulated industries - a low temperature (0 to 0.3) is usually the right choice. You want the model to stick closely to the context rather than improvise.
Max tokens
Setting a maximum output length prevents runaway responses and helps control latency and cost. Think about what a reasonable response looks like for your use case. For example, a quick factual answer needs far fewer tokens than a detailed report. The token limit should be set according to your use case.
System prompt design
As discussed in the augmentation post, the system prompt is where you set the rules for how the model should behave. For generation quality specifically, a few things are worth being explicit about:
- Staying grounded: "If the answer is not found in the provided context, say that you don't know." This is one of the most important instructions you can give, as LLMs have a natural tendency to produce plausible-sounding answers even when they don't have the information.
- Output format: if you need a structured response (JSON, bullet points, a specific template), specify it clearly. Models follow explicit formatting instructions well.
- Citing sources: asking the model to reference specific documents in its response makes the output more auditable and gives users a way to verify the answer.
A note that even given these instructions, the nature of LLMs is such that they may not be followed 100% of the time, and all output should still be validated.
Structured outputs and citations
Asking an LLM to return free-form text is fine for simple use cases, but in production RAG systems you usually want more control over the shape of the response. Structured outputs - where you define a schema and ask the model to return data that conforms to it solves several problems at once:
- The response is machine-readable and easy to render in a UI
- Citations become a first-class field, not an afterthought buried in text
- Validation is straightforward - if the response doesn't conform to the schema, you know immediately
Most LLM frameworks and APIs support this via JSON mode or tool/function calling. In Python, Pydantic is a standard way to define the schema. It allows you to ensure that the response has the correct format, with automatic retry on failure.
A schema for a RAG response with mandatory citations might look like this:
from pydantic import BaseModel
class Citation(BaseModel):
source_id: str # matches the label used in the prompt, e.g. "Response 1"
quote: str # the quote from the source that supports the claim
class RAGResponse(BaseModel):
answer: str
citations: list[Citation]
To use this schema with the Azure OpenAI client, you pass it via the response_format parameter:
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from openai import AzureOpenAI
credential = DefaultAzureCredential()
token_provider = get_bearer_token_provider(
credential,
"https://cognitiveservices.azure.com/.default"
)
client = AzureOpenAI(
azure_endpoint="https://your-foundry-account.services.ai.azure.com/",
api_version="2024-08-01-preview",
azure_ad_token_provider=token_provider,
)
def generate_response(augmented_prompt: str) -> RAGResponse:
completion = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[
{
"role": "system",
"content": (
"You are a helpful assistant. "
"Only use information from the provided context to answer the question. "
"If the answer is not found in the context, say that you don't know. "
"Cite the source of each piece of information using the labels provided in the context."
),
},
{"role": "user", "content": augmented_prompt},
],
response_format=RAGResponse,
)
return completion.choices[0].message.parsed
The .parse() method handles deserialising the response directly into your Pydantic model. If the model returns malformed JSON that doesn't match your schema, the SDK raises a validation error rather than returning bad data.
By making citations a required field, you enforce at the schema level that the model must cite its sources. It can't return a valid response without doing so. The source_id field maps back to the labels you included in the augmented context (as discussed in the augmentation post), and the quote field gives an auditable snippet of the exact passage that was used.
This also makes it easy to validate that every cited source_id is one that was actually in the retrieved context - a simple check that catches any cases where the model has hallucinated a source reference.
Handling ambiguity and uncertainty
Not every question has a clear answer in the retrieved documents. A well-designed generation step should handle this gracefully - producing an "I don't have enough information to answer that" response rather than hallucinating an answer. This comes back to the grounding instruction in the system prompt, but it also depends on the model itself. Some models are more prone to hallucination than others, and testing your specific model against ambiguous queries is an important part of evaluation.
With a structured output schema, you can handle this even more explicitly. You can add a confidence field or an answer_found: bool field to the schema, so that low-confidence or unanswerable queries are flagged in a structured way rather than silently producing a vague response.
As noted in the overview post, even with a well-designed system, there is still an inherent propensity for LLMs to fall back on training data or include unexpected information, even when instructed not to. Adding explicit grounding instructions ("Do not use any information outside of what is contained in the given context") significantly reduces this, but it doesn't eliminate it entirely. This means that even in a RAG system, all responses must be validated against expected outputs, especially in any domain where an incorrect answer has real consequences.
Evaluation and Quality Metrics
Evaluating generation quality is harder than it sounds, because there's rarely a single "correct" answer to compare against. A few metrics that are commonly used:
- Faithfulness - does the response only use information that is present in the provided context?
- Relevance - does the response actually answer the user's question? A response can be faithful to the context but still miss the point of what was asked.
Faithfulness and relevance are usually measured using an "LLM as a judge" - i.e. you feed the prompt and the response into a separate LLM and ask it to give you a score based on these factors. Using this, you can then evaluate any changes you make as to whether they've had a positive or a negative impact.
- Completeness - does the response cover all the relevant information in the context, or has it missed something important?
Completeness is a bit harder to automate, as "are all the relevant points covered" is subjective. This is sometimes done using an LLM as a judge, but often is done via...
- Human evaluation - automated metrics are useful, but there is no substitute for a Human-in-the-Loop (HITL) process that validates the responses. This is especially true in early development, when you're still tuning your prompts and retrieval strategy. This usually involves domain experts manually checking the outputs, especially ones which are flagged with a low confidence.
Real-World Applications
RAG and its generation step are well-suited to a wide range of real-world scenarios:
- Customer support - answering questions grounded in a company's own documentation, FAQs, or support history, rather than generic LLM responses that might give inaccurate information about specific products or policies.
- Knowledge management - enabling employees to query internal documents, meeting notes, or wikis in natural language, with responses grounded in the actual content rather than generalised summaries.
- Research and analysis - summarising information from a large collection of documents (reports, papers, data) in response to specific questions, with traceable references back to the source material.
- Survey analysis - gaining insights from large volumes of qualitative responses, such as customer surveys, employee feedback, or product reviews. RAG allows you to ask questions across hundreds or thousands of responses ("What are the most common themes in low-scoring responses?") and get grounded, cited answers without needing to read every entry manually.
- Search augmentation - enriching traditional search results with a generated summary that synthesises across multiple documents, rather than just returning a ranked list of links.
Conclusion
Generation is where the RAG pipeline delivers its value to the user, but it's the output of all three steps working together. Good generation depends on good retrieval (the right documents), good augmentation (a well-structured prompt), and careful configuration (the right model settings and instructions).
When all the stages are working well together, you have the greatest chance of responses grounded in up-to-date, domain-specific, data, with references to specific documents that can be cross-checked. You can use smaller, more efficient models because you're not relying on them to have memorised everything. And you retain the ability to apply fine-grained security and access controls at the retrieval layer, rather than exposing everything to the model.
But generation is also where things can go wrong if you're not careful. LLMs are probabilistic, and even the best RAG system doesn't eliminate the risk of unexpected outputs. Evaluation, validation, and human oversight remain crucial.
