How to Implement Retrieval in RAG
In my previous blog I gave an overview of Retrieval Augmented Generation (RAG). In this post I'll dive deeper into the "retrieval" part of the solution.
Retrieval is the first step in the RAG pipeline, and in many ways the most important. Before any augmentation or generation can happen, you need to find the right information.
Retrieval is the process of querying a data source and returning the documents or records most relevant to a given input. That input might be a user's question, a search query, or some other piece of text. The output is a set of documents that will be passed on to the augmentation step.
The data source could be almost anything - a database, a document store, a knowledge base, an email archive, or a collection of web pages. What makes retrieval in RAG interesting is that it often goes beyond a simple keyword lookup. Rather than asking "does this document contain these words?", we often want to ask "is this document relevant to what the user is asking?".
Why Retrieval is Important
Retrieval of relevant information, rather than just augmenting the prompt with all available data, is an incredibly important step in RAG processing. This is because as prompt input sizes increase, the responses from the LLM degrade. Limiting the context to a smaller, targeted subset of relevant information is the best way to get useful responses.
Alongside this, good retrieval is also what determines the quality of everything that comes after it. If the wrong documents are fetched, the generated answer will be wrong or incomplete, no matter how well the rest of the pipeline is designed.
Because retrieval surfaces an explicit set of source documents, those sources can be specifically cited by any answers - meaning that answers can be validated and audited more easily.
Finally, the retrieval index can be updated continuously as new data arrives, this means the knowledge available to your RAG system can grow and evolve over time without any retraining or fine-tuning.
Types of Retrieval
Database Query
The simplest form of retrieval is a structured query against a database. If your data lives in a relational database, a query language like SQL lets you filter records precisely - by date range, category, status, or any other field in your schema.
This approach works really well when you know exactly what you're looking for and your data is well-structured. It's fast, deterministic, and easy to understand why certain documents have been retrieved.
The downside is that it's brittle - it requires you to know the right field values up front, and it can't handle anything fuzzy or conceptual. If a user asks "what went wrong last month?", a database query can't help you unless you already know exactly which fields represent "what went wrong".
Database queries are often used as a pre-filtering step in more sophisticated RAG pipelines - narrowing a large dataset down to a relevant subset before applying a more intelligent retrieval method.
Keyword Search
Keyword search matches documents based on the presence of specific words or phrases.
This is a fast and easily-understandable approach - it's still very easy to understand why certain results have been returned. Tools like Azure AI Search and Elasticsearch are built for exactly this, and they support things like:
- Fuzzy matching (handling spelling mistakes and variations)
- Stemming (matching "running" when you search "run")
- "Stop word" filtering which removes words like "a", "is" and "the" from the search criteria. This is important because if a user searches for a whole sentence - like "what is the biggest complaint", then many keyword searches would return any documents which contain any word in that sentence. And, you can imagine that most documents likely contain the word "the"!
The limitation is that keyword search can only match documents that contain the words you searched for. A document that says "the package arrived two weeks late" won't match a search for "slow delivery" even though it means the same thing. For that, you need something more semantic.
Vector Search
Vector search addresses the limitation of keyword search by working with meaning rather than words. It uses embeddings (numerical representations of text) to find documents that are conceptually similar to a query, even when they share no words in common.
An embedding is produced by passing text through an embedding model (such as OpenAI's text-embedding-3-small or open-source alternatives like sentence-transformers). The model outputs a vector (an array of numbers) that encodes the semantic meaning of the text. Crucially, texts that mean similar things end up with vectors that are close together in this high-dimensional space.
For example, when you embed the text "Shipping took forever" and "Delivery was very slow", they'll have similar vector representations despite using different words, because they mean similar things. Meanwhile, "Material feels cheap and flimsy" will be far away in vector space.
In practice, generating an embedding is a single API call. Here's a minimal example using Azure OpenAI with Azure AD authentication:
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from openai import AzureOpenAI
credential = DefaultAzureCredential()
token_provider = get_bearer_token_provider(
credential,
"https://cognitiveservices.azure.com/.default"
)
client = AzureOpenAI(
azure_endpoint="https://your-foundry-account.services.ai.azure.com/",
api_version="2024-02-01",
azure_ad_token_provider=token_provider,
)
def get_embedding(text: str) -> list[float]:
response = client.embeddings.create(
input=text,
model="text-embedding-3-small"
)
return response.data[0].embedding
You would call get_embedding on each chunk of your documents to build up your retrieval index.
When a user submits a query, that query is also embedded, and the retrieval system finds the documents whose vectors are closest to the query vector - typically using cosine similarity to calculate the "distance" between vectors. The most semantically similar documents are returned, regardless of exact wording.
Using the same model for both is essential, as different models produce incompatible vector spaces.
Using Azure AI Search, a vector search looks like this:
from azure.search.documents.models import VectorizedQuery
from azure.search.documents import SearchClient
from azure.identity import DefaultAzureCredential
search_client = SearchClient(
endpoint="https://your-search-service.search.windows.net",
index_name="reviews",
credential=DefaultAzureCredential(),
)
def vector_search(query: str, top: int = 5) -> list[dict]:
query_vector = get_embedding(query) # using the function from above
vector_query = VectorizedQuery(
vector=query_vector,
k=top,
fields="content_vector", # the field in your index that stores embeddings
)
results = search_client.search(
search_text=None,
vector_queries=[vector_query],
select=["id", "content", "source"],
top=top,
)
return [dict(doc) for doc in results]
Calling vector_search("What are customers saying about delivery?") will return the reviews whose embeddings are closest to the query embedding, even if they use completely different words.
One important consideration is chunking - how you split your documents before embedding them for the index. Embedding a 50-page document as a single vector will lose a lot of nuance, so in practice documents are split into smaller chunks (paragraphs, sections, or fixed-size windows) before being embedded and indexed. Choosing the right chunk size and overlap is one of the more impactful tuning decisions in a RAG system.
The index itself is typically stored and queried using a vector database, such as Azure AI Search.
Hybrid
In practice, neither keyword search nor vector search is universally better, both are powerful in different situations. Hybrid retrieval combines both, running keyword and vector searches in parallel and then merging the results.
The most common merging strategy is Reciprocal Rank Fusion (RRF), which combines the ranked lists from each search method by giving each result a score based on its position in each list. This allows you to narrow down to documents that rank well in both searches. If a document is relevant both in its keywords and semantically, you can be fairly confident it's what the user is looking for.
Azure AI Search supports hybrid retrieval natively, making it a good fit for RAG pipelines on Azure.
Common Tools and Techniques
The retrieval types above each require different tooling, but in a real RAG system there are a few layers that nearly always appear.
Embedding models
Embedding models form the basis of vector search. Popular choices include OpenAI's text-embedding-3-small and text-embedding-3-large, which are available via the Azure OpenAI Service. If you'd rather keep everything on-premises or avoid API costs, open-source alternatives like sentence-transformers are widely used. The right choice of embedding model depends on domain, language and document types.
Most importantly - the model used to embed your documents at index time must be the same one used to embed queries at retrieval time. Using different models can produce incompatible embeddings for the same phrase, resulting in a completely different vector.
Vector stores and search services
These handle storing and querying your embeddings. As mentioned above, Azure AI Search is a managed service that supports keyword, vector, and hybrid search out of the box - making it a convenient choice for Azure-based workloads. But, depending on your technology stack there are a lot of options out there (Pinecone, Weaviate, Qdrant, etc.).
Orchestration frameworks
Orchestration frameworks like LangChain, LlamaIndex, and Microsoft's Semantic Kernel provide higher-level abstractions for building RAG pipelines. Rather than writing the plumbing yourself (embed the query, query the index, format the results, build the prompt), these frameworks offer pre-built components for each step that can be wired together and swapped out. They also tend to integrate with a wide range of data sources and model providers, which makes it easier to experiment with different retrieval strategies without rewriting your whole pipeline.
Challenges and Best Practices
Here are some of the main challenges to think about when designing a retrieval system:
Relevance ranking
A retrieval system might return ten documents, but if the most relevant one is ranked eighth, you may be passing less relevant information into the generation step. Hybrid search and re-ranking models (which apply a second, more expensive scoring pass over the top results - providing a more accurate rank) can both help here.
Chunking strategy
Chunking has a large impact on quality. Chunks that are too small may lack enough context to be useful, whilst chunks that are too large may dilute the relevant signal with noise. There's no universally correct answer - it depends on your documents, your embedding model, and the kinds of questions users will ask. Experimenting with different chunk sizes and overlap amounts is usually necessary to get the best results.
Data freshness
If your underlying data changes frequently, you need to think about data freshness. If documents in your index are updated or deleted, the index needs to reflect that - particularly in regulated domains where accuracy is critical.
Latency
If you are working at scale, latency can become a concern. Vector similarity search over millions of embeddings - API calls to generate the embeddings, then the search itself, combined with a keyword search and re-ranking, adds up quickly.
Most production systems use "approximate nearest neighbour" algorithms - which instead of performing exact matching on all embeddings in the index, use an index which is organised in vector space to discount results which are likely to be irrelevant. This trades some accuracy, but produces much faster results. Solutions also often cache frequently-asked queries where possible.
Security and access control
As in any data system security and access control are important to get right. If your data store contains documents that different users should have different levels of access to, retrieval must respect those boundaries. Returning a document in the retrieved context that the user isn't authorised to see (even if they can't see the document directly) could leak sensitive information in the generation step. The safest approach is to apply access filters at query time, so only documents the user is permitted to see can ever be retrieved.
The advantage of RAG here is that, if you instead just trained a model on all the data, there would be no way to enforce different security boundaries for different users. The ability to use fine-grained access control is one of the huge strengths of a RAG architecture.
Conclusion
Retrieval is the foundation that the rest of RAG is built on.
The key message is that there's no single "correct" retrieval strategy - the right approach depends on your data, your users, and the kinds of questions being asked.
Structured database queries work well for precise, known criteria. Keyword search is fast and auditable. Vector search handles semantic similarity where keywords fall short. And hybrid approaches combine the strengths of both.
In the next post, I'll look at the augmentation step - how the retrieved documents are prepared, formatted, and injected into the prompt to give the LLM the context it needs to generate a useful response.
