Build a Multi-Query RAG Retriever with Elasticsearch and LangChain

A RAG pipeline is only as good as the query that enters the retriever. A user types “what is the nasa sales team?”, the embedding model turns that exact string into a vector, and Elasticsearch returns whatever happens to sit near that vector. If the user’s wording does not match the language of the indexed chunks, relevant paragraphs are missed even though the knowledge base contains them. Multi-query retrieval attacks that problem before retrieval happens: an LLM generates several paraphrased or decomposed questions from the original, each variant is embedded and searched independently, and the combined results are passed to the final answer-generation stage. You trade higher query-time cost and latency for meaningfully better recall on documents where terminology is inconsistent.

This guide is adapted from Elasticsearch Labs’s chatbot-with-multi-query-retriever.ipynb, part of the elastic/elasticsearch-labs repository and used under the Apache-2.0 licence. The tested code is reproduced here as-is. Prose explanations, warnings, and cost guidance are added independently.

The workflow is aimed at engineers building LangChain RAG applications on Elasticsearch who have already seen single-shot retrieval fail on real support tickets or employee handbooks. It costs more than a naive retriever: at query time you pay for one LLM expansion call, one embedding call per generated query, the retrieval work for each query, and a final LLM answer call. Because the notebook uses remote OpenAI embedding and generation APIs, no local GPU is required. Latency still grows from one retrieval round to several.

Prerequisites

The source notebook assumes:

• Python with pip; no specific package versions are pinned.
• An Elastic Cloud deployment or equivalent Elasticsearch cluster.
• An OpenAI API key with embedding and LLM access.
• Outbound network access to Elasticsearch and OpenAI.

The code runs in a hosted notebook such as Colab or a local Jupyter environment. No local GPU is required.

Step 1: Install dependencies and load the LangChain pieces

The first cell installs the Python packages and imports the components that the rest of the workflow depends on. The package list includes jq, lark, and tiktoken; tiktoken is used later by the token-aware splitter, and the other packages supply LangChain parsing and loading dependencies. The imports pull in the OpenAI embeddings model, the Elasticsearch vector store, the OpenAI LLM, and the multi-query retriever.

!python3 -m pip install -qU jq lark langchain langchain-elasticsearch langchain_openai tiktoken

from langchain_openai.embeddings import OpenAIEmbeddings
from langchain_elasticsearch import ElasticsearchStore
from langchain_openai.llms import OpenAI
from langchain.retrievers.multi_query import MultiQueryRetriever
from getpass import getpass

The installation is not pinned, which means a future release of LangChain or langchain-elasticsearch can change import paths or default behaviour. If the retriever import fails, that is the first thing to check.

Step 2: Connect to Elasticsearch and create the vector store handle

The notebook uses getpass so credentials are not stored in the source file. ElasticsearchStore is created before the index exists; it holds the connection details and the embedding function that will be used when documents are indexed and when queries are embedded later.

# https://www.elastic.co/search-labs/tutorials/install-elasticsearch/elastic-cloud#finding-your-cloud-id
ELASTIC_CLOUD_ID = getpass("Elastic Cloud ID: ")

# https://www.elastic.co/search-labs/tutorials/install-elasticsearch/elastic-cloud#creating-an-api-key
ELASTIC_API_KEY = getpass("Elastic Api Key: ")

# https://platform.openai.com/api-keys
OPENAI_API_KEY = getpass("OpenAI API key: ")

embeddings = OpenAIEmbeddings(openai_api_key=OPENAI_API_KEY)

vectorstore = ElasticsearchStore(
    es_cloud_id=ELASTIC_CLOUD_ID,
    es_api_key=ELASTIC_API_KEY,
    index_name="chatbot-multi-query-demo",
    embedding=embeddings,
)

The index_name value matters because it must match the name used in the indexing step. A mismatch produces an empty retriever or an index-mapping error, often after expensive embeddings have already been created.

Step 3: Download the sample dataset

The tutorial ingests a public JSON dataset from the Elasticsearch Labs repository. The loader expects a JSON file on disk, so the remote content is written to temp.json first.

from urllib.request import urlopen
import json

url = "https://raw.githubusercontent.com/elastic/elasticsearch-labs/main/example-apps/chatbot-rag-app/data/data.json"

response = urlopen(url)
data = json.load(response)

with open("temp.json", "w") as json_file:
    json.dump(data, json_file)

This step is deliberately separate from parsing because LangChain’s JSONLoader reads a local file path, not a URL.

Step 4: Parse JSON, preserve metadata, and split documents

The dataset contains records with a content field plus several metadata fields. The metadata_func receives each JSON record and copies only the fields that will be useful in the final prompt. That metadata is valuable later because the answer-generation prompt prints a SOURCE: line before each retrieved chunk.

from langchain.document_loaders import JSONLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter


def metadata_func(record: dict, metadata: dict) -> dict:
    metadata["name"] = record.get("name")
    metadata["summary"] = record.get("summary")
    metadata["url"] = record.get("url")
    metadata["category"] = record.get("category")
    metadata["updated_at"] = record.get("updated_at")

    return metadata


# For more loaders https://python.langchain.com/docs/modules/data_connection/document_loaders/
# And 3rd party loaders https://python.langchain.com/docs/modules/data_connection/document_loaders/#third-party-loaders
loader = JSONLoader(
    file_path="temp.json",
    jq_schema=".[]",
    content_key="content",
    metadata_func=metadata_func,
)

text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=512, chunk_overlap=256
)
docs = loader.load_and_split(text_splitter=text_splitter)

The splitter uses a tokenizer-aware encoder. chunk_size=512 means roughly 512 tokens per passage, and chunk_overlap=256 means adjacent passages share half their content. That overlap preserves context across chunk boundaries, but it also creates near-duplicate text that can reappear in retrieval results. If you later increase chunk sizes toward very long contexts, the prefill tradeoffs are not free; see AI Mastery’s coverage of long-context prefill.

Step 5: Index the passages and create the multi-query retriever

The from_documents call embeds every split document with the configured OpenAI embeddings and bulk-indexes the vectors into Elasticsearch. The same index_name is passed again explicitly. Then an OpenAI LLM with temperature=0 is created. Deterministic output is important for query expansion because a creative paraphrase can drift away from the user’s intent. MultiQueryRetriever.from_llm wraps the vectorstore’s retriever and adds the expansion LLM.

documents = vectorstore.from_documents(
    docs,
    embeddings,
    index_name="chatbot-multi-query-demo",
    es_cloud_id=ELASTIC_CLOUD_ID,
    es_api_key=ELASTIC_API_KEY,
)

llm = OpenAI(temperature=0, openai_api_key=OPENAI_API_KEY)

retriever = MultiQueryRetriever.from_llm(vectorstore.as_retriever(), llm)

MultiQueryRetriever generates several alternative questions from the input and retrieves documents for all of them. The notebook does not set the number of generated queries, so it relies on the LangChain default. The generated questions are not shown to the end user; they are internal retrieval aids.

Step 6: Build the question-answering chain

The final chain has three stages. RunnableParallel runs the retriever against the original question and combines the retrieved documents into one context string, while the question passes through unchanged. The context and question are then formatted into LLM_CONTEXT_PROMPT, and finally the LLM generates the answer. The logging configuration prints the multi-query retriever’s generated queries, which is useful for debugging but should be disabled or lowered in production if queries contain sensitive text.

from langchain.schema.runnable import RunnableParallel, RunnablePassthrough
from langchain.prompts import ChatPromptTemplate, PromptTemplate
from langchain.schema import format_document

import logging

logging.basicConfig()
logging.getLogger("langchain.retrievers.multi_query").setLevel(logging.INFO)

LLM_CONTEXT_PROMPT = ChatPromptTemplate.from_template(
    """You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Be as verbose and educational in your response as possible. 
    
    context: {context}
    Question: "{question}"
    Answer:
    """
)

LLM_DOCUMENT_PROMPT = PromptTemplate.from_template(
    """
---
SOURCE: {name}
{page_content}
---
"""
)


def _combine_documents(
    docs, document_prompt=LLM_DOCUMENT_PROMPT, document_separator="\n\n"
):
    doc_strings = [format_document(doc, document_prompt) for doc in docs]
    return document_separator.join(doc_strings)


_context = RunnableParallel(
    context=retriever | _combine_documents,
    question=RunnablePassthrough(),
)

chain = _context | LLM_CONTEXT_PROMPT | llm

ans = chain.invoke("what is the nasa sales team?")

print("---- Answer ----")
print(ans)

LLM_DOCUMENT_PROMPT uses the name metadata field captured in Step 4. If that metadata did not survive indexing, the final context would still contain page content but the source attribution would be incomplete.

Choosing between a single retriever and multi-query expansion

The table below summarises the tradeoffs you are accepting when you replace a single retrieval pass with multi-query expansion.

What to watch out for

Multi-query retrieval fixes a wording gap. It does not fix bad chunking, an embedding model that does not understand your domain, or an index that was built with mismatched fields.

The expansion LLM can drift. At temperature=0 the drift is limited, but a question with ambiguous words can still be paraphrased into variants that leave the original intent behind. If the generated queries are visible in logs during development, inspect them before trusting the architecture.

Duplicate context is a real problem with chunk_size=512 and chunk_overlap=256. Half of each passage is shared with its neighbour. Retrieval across several paraphrased queries can return the same or near-duplicate chunks, filling the final prompt with redundant text and raising token cost without adding information. Deduplicate results or consider a smaller overlap before sending the combined context to the LLM.

Every question now costs more. A single user query triggers the expansion LLM call, several embedding calls, several retrieval operations, and the final generation call. For a low-traffic internal assistant this is acceptable. For a public-facing product, monitor OpenAI spend and p95 latency carefully. Teams that need to control per-token cost while scaling RAG might also want to look at AI Mastery’s coverage of cost-reduction work in coding agents, which raises similar tradeoffs around token budgets.

Connection and index hygiene matter. The index_name appears in both the vectorstore constructor and the from_documents call. If you change one, the retriever queries a different index, or none at all. A reused Elasticsearch index with an incompatible mapping can also fail on the bulk import.

Finally, multi-query retrieval does not guarantee precision. It increases recall by retrieving more documents, but some of those documents will be irrelevant. The final prompt sends all combined results to the answer LLM. In this notebook the LLM is instructed to say when it does not know, but it is still possible for a confident-sounding answer to use weak context. For higher precision, add a reranking stage after retrieval instead of assuming the union of retrieved passages is clean.

Where to go next

The next practical step is to adjust the retriever’s num_queries, add a reranker, or replace the single vectorstore retriever with an Elasticsearch hybrid search that fuses BM25 and vector scores. The Elasticsearch Labs repository includes additional notebooks that build on this exact dataset, and LangChain’s documentation covers the full MultiQueryRetriever API, including custom prompt templates for domain-specific paraphrasing. Treat this guide as the baseline: the code works, but the retrieval strategy is only one dial in a RAG system.