Build a sentence-window RAG retriever with Haystack

Most retrieval-augmented generation (RAG) systems face a granularity trade-off. Split documents into large chunks and you keep context but dilute the exact sentence that answers the user's question; split into tiny sentence-level chunks and you get precise BM25 or vector matches that often read like fragments. Sentence-window retrieval resolves this by indexing sentence-level documents and expanding the match into a window of neighboring sentences at query time. Haystack's SentenceWindowRetriever implements this pattern: it receives one or more matching documents from an upstream retriever, uses split metadata to locate surrounding sentences in the same document, and returns both raw window strings and document objects.

This guide is adapted from Haystack's 42_Sentence_Window_Retriever.ipynb in the haystack-tutorials repository, distributed under the Apache-2.0 license. Code blocks are reproduced from that tutorial. The guide adds the connection notes, window-size trade-offs, and failure modes that the notebook leaves implicit.

Who needs this: engineers building Q&A systems over news archives, policy text, or documentation where a single sentence is ambiguous but a 2,000-token fixed chunk is wasteful. The tutorial's code requires no API keys or GPU-specific packages, and the InMemoryDocumentStore keeps every sentence-level document in RAM. That makes this a learning and prototyping setup; for production-scale corpora you would swap in a persistent document store and potentially a vector retriever, but the pipeline structure remains the same.

Prerequisites

• Python environment with pip; no GPU required.
• No accounts or paid APIs. The only network access is for installing packages and downloading the BBC news CSV.
• Enough RAM to hold the BBC dataset as sentence-level Haystack Document objects. The tutorial uses an in-memory store, so the dataset must fit in available RAM.
• The tutorial uses the latest haystack-ai and nltk packages; it does not pin versions.

Step 1: Install Haystack and NLTK

First install the packages. The source notebook runs this as a bash cell.

%%bash

pip install --upgrade pip
pip install haystack-ai nltk

Step 2: Split a document into sentence-level units

The core idea starts with a splitter configured to produce one sentence per document. In this first example the splitter uses split_by="period" to split on sentence-ending periods. The split_length=1 and split_overlap=0 settings keep one sentence per document with no overlap.

from haystack import Document
from haystack.components.preprocessors import DocumentSplitter

splitter = DocumentSplitter(split_length=1, split_overlap=0, split_by="period")

text = (
    "Paul fell asleep to dream of an Arrakeen cavern, silent people all around  him moving in the dim light "
    "of glowglobes. It was solemn there and like a cathedral as he listened to a faint sound—the "
    "drip-drip-drip of water. Even while he remained in the dream, Paul knew he would remember it upon "
    "awakening. He always remembered the dreams that were predictions. The dream faded. Paul awoke to feel "
    "himself in the warmth of his bed—thinking thinking. This world of Castle Caladan, without play or "
    "companions his own age,  perhaps did not deserve sadness in farewell. Dr Yueh, his teacher, had "
    "hinted  that the faufreluches class system was not rigidly guarded on Arrakis. The planet sheltered "
    "people who lived at the desert edge without caid or bashar to command them: will-o’-the-sand people "
    "called Fremen, marked down on no  census of the Imperial Regate."
)

doc = Document(content=text)
docs = splitter.run([doc])

DocumentSplitter.run() returns a dictionary with a documents key. Each sentence becomes its own Haystack Document with metadata fields that later allow the window retriever to reconstruct context.

Step 3: Write sentence documents to an in-memory store

The SentenceWindowRetriever needs access to the split documents, so write them to an InMemoryDocumentStore. DuplicatePolicy.OVERWRITE prevents duplicate accumulation if you rerun the cell.

from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.document_stores.types import DuplicatePolicy

doc_store = InMemoryDocumentStore()
doc_store.write_documents(docs["documents"], policy=DuplicatePolicy.OVERWRITE)

Step 4: Retrieve a context window around one sentence

This is the component that does the work. It takes a document_store and a window_size. The retriever does not accept a user query directly; it accepts a list of retrieved documents and returns the neighboring sentences for each.

from haystack.components.retrievers import SentenceWindowRetriever

retriever = SentenceWindowRetriever(document_store=doc_store, window_size=2)
result = retriever.run(retrieved_documents=[docs["documents"][4]])

The input document at index 4 is "The dream faded.". With window_size=2, the returned window contains two sentences before and two after, for a total of five sentences when boundaries permit.

Step 5: Inspect both output formats

The result contains two keys: context_windows returns the expanded text as strings, while context_documents returns the same content as Haystack Document objects sorted by their split index.

result["context_windows"]

result["context_documents"]

The string form is convenient for prompt construction; the document form preserves metadata such as title, category, and split position for tracing.

Step 6: Read and download the BBC news dataset

Now move to a more realistic multi-document pipeline. The tutorial uses the BBC news dataset, already preprocessed into a single CSV file. The reader skips the header and creates one Document per row with category and title metadata.

from typing import List
import csv
from haystack import Document


def read_documents(file: str) -> List[Document]:
    with open(file, "r") as file:
        reader = csv.reader(file, delimiter="\t")
        next(reader, None)  # skip the headers
        documents = []
        for row in reader:
            category = row[0].strip()
            title = row[2].strip()
            text = row[3].strip()
            documents.append(Document(content=text, meta={"category": category, "title": title}))

    return documents

Download and load it:

from pathlib import Path
import requests

doc = requests.get("https://raw.githubusercontent.com/amankharwal/Website-data/master/bbc-news-data.csv")

datafolder = Path("data")
datafolder.mkdir(exist_ok=True)
with open(datafolder / "bbc-news-data.csv", "wb") as f:
    for chunk in doc.iter_content(512):
        f.write(chunk)

docs = read_documents("data/bbc-news-data.csv")
len(docs)

Step 7: Index BBC articles as sentence-level documents

This pipeline splits each article into sentences and writes them to a fresh InMemoryDocumentStore. Note the change from the earlier split_by="period" to split_by="sentence", which uses a more robust sentence tokenizer for real news copy.

from haystack import Document, Pipeline
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.document_stores.types import DuplicatePolicy

doc_store = InMemoryDocumentStore()

indexing_pipeline = Pipeline()
indexing_pipeline.add_component("splitter", DocumentSplitter(split_length=1, split_overlap=0, split_by="sentence"))
indexing_pipeline.add_component("writer", DocumentWriter(document_store=doc_store, policy=DuplicatePolicy.OVERWRITE))

indexing_pipeline.connect("splitter", "writer")

indexing_pipeline.run({"documents": docs})

Step 8: Compose BM25 retrieval with sentence-window expansion

The retrieval pipeline chains two components. The InMemoryBM25Retriever takes the user query and returns the best single sentence. Its output feeds directly into SentenceWindowRetriever.

from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.retrievers import SentenceWindowRetriever

sentence_window_pipeline = Pipeline()

sentence_window_pipeline.add_component("bm25_retriever", InMemoryBM25Retriever(document_store=doc_store))
sentence_window_pipeline.add_component("sentence_window__retriever", SentenceWindowRetriever(doc_store, window_size=2))

sentence_window_pipeline.connect("bm25_retriever.documents", "sentence_window__retriever.retrieved_documents")

The connection is the critical part: bm25_retriever.documents must be connected to sentence_window__retriever.retrieved_documents. If you wire it to a query input, the pipeline will fail because SentenceWindowRetriever has no query input.

Step 9: Run a query and compare outputs

Run the pipeline for the query "phishing attacks", requesting only the top BM25 match and telling Haystack to include the BM25 outputs in the result.

result = sentence_window_pipeline.run(
    data={"bm25_retriever": {"query": "phishing attacks", "top_k": 1}}, include_outputs_from={"bm25_retriever"}
)

Inspect the BM25 sentence first:

result["bm25_retriever"]["documents"]

Then inspect the expanded context:

result["sentence_window__retriever"]["context_windows"]

result["sentence_window__retriever"]["context_documents"]

BM25 returns a single sentence fragment from the matching article; the sentence-window retriever returns that sentence plus two neighbors on each side, turning a cryptic fragment into coherent context.

What to watch out for

The component is simple, but its assumptions create real failure modes.

First, SentenceWindowRetriever depends on metadata created by DocumentSplitter. If you pass documents that were not produced by the same splitter run, or if you construct documents by hand without the split position metadata, the retriever can raise an error or produce an empty window. Always source retrieved_documents from the same document store that the retriever is configured with.

Second, the retriever does not accept queries. It is a post-processor, not a standalone retriever. The pipeline input must be shaped as retrieved_documents, and your upstream retriever must already return matching sentence-level documents. This is why the tutorial uses InMemoryBM25Retriever, but you can replace it with any Haystack retriever.

Third, document boundaries are not padded. If the matching sentence is the first or last in an article, the returned window is shorter than 2 * window_size + 1. A prompt builder should handle truncated windows gracefully; do not assume exactly five sentences.

Fourth, window_size is a direct context multiplier in sentences, which translates into token growth. With top_k=1 and window_size=2, the BM25 hit expands to five sentences. If you set top_k=10 and window_size=3, you can be composing prompts from dozens of sentences. That extra context has a generation cost too; see Claude Haiku 5.5's 1M context pricing for a broader view of how context length affects cost.

Fifth, keyword retrieval has lexical blind spots. The query "phishing attacks" will miss an article that says "fraudulent emails" instead. The sentence-window retriever cannot fix that; it only expands whatever the upstream retriever returns. For higher recall, pair the same window mechanism with a vector or hybrid retriever.

Finally, InMemoryDocumentStore is ephemeral and RAM-bound. The BBC dataset is small enough for a learning environment, but a production corpus with millions of articles will exceed available memory. The pipeline pattern transfers to persistent stores, but you will need to plan for memory, indexing time, and metadata consistency.

Where to go next

The next step is to swap the BM25 retriever for a dense embedding retriever and compare retrieval quality on the same BBC dataset. Small embedding models have been improving quickly; Perplexity's pplx-embed-v2 is one recent example worth evaluating. You can also experiment with different window_size values and split_by options, or add a PromptBuilder and generator after the sentence-window retriever to measure downstream answer quality. The Haystack documentation covers the SentenceWindowRetriever, DocumentSplitter, and additional retrievers in more detail.