Evaluate a Qdrant RAG Pipeline with Ragas and OpenRouter

Most RAG checklists stop at one number: retrieval hit rate. That tells you whether the right chunks reached the prompt, but nothing about whether the final answer is supported by those chunks or actually answers the question. This guide builds an end-to-end evaluation loop that separates the two halves of a Qdrant-backed RAG pipeline: retrieval quality and generation quality. You will index the BioASQ mini benchmark in a local, in-memory Qdrant collection, retrieve passages with FastEmbed, generate answers through OpenRouter, and score the results with Ragas. The output is a per-question diagnosis, not a single average, so you can tell whether to fix the retriever, the writer, or both.

This workflow is aimed at ML engineers who are about to ship a RAG feature or tuning an existing one, and at search teams that need evidence before changing a prompt or a retrieval model. It is deliberately cheap to run: you need one OpenRouter API key, no GPU, and no hosted Qdrant instance. The notebook does not record a cost figure or embedding runtime; both vary with the host, concurrency, and OpenRouter pricing, so measure them on a small sample before scaling. The tradeoff is that Qdrant runs in local mode, which is fine for evaluation but not for production workloads.

Adapted from Qdrant’s rag_pipeline_evaluation.ipynb in the Qdrant examples repository, used under the Apache-2.0 licence.

Prerequisites

The notebook pins exact package versions; keep them as written.

%pip install -q "ragas==0.4.3" "langchain-community==0.4.1" "qdrant-client[fastembed]==1.19.1" "openai==3.3.0" datasets pandas

You also need an OpenRouter API key. In Colab, store it as a secret named OPENROUTER_API_KEY; anywhere else, the script will prompt for it.

Step 1: Configure Environment and Evaluation Constants

This cell sets up the models and the evaluation parameters. The generator and judge are deliberately different model families: a model judging its own output tends to score that output more favourably. The judge also has reasoning disabled so the scores reflect the judge’s direct evaluation rather than long internal deliberation.

import os
from getpass import getpass

try:
    from google.colab import userdata

    os.environ.setdefault("OPENROUTER_API_KEY", userdata.get("OPENROUTER_API_KEY"))
except Exception:
    pass  # not in Colab, or the secret is not set
if not os.environ.get("OPENROUTER_API_KEY"):
    os.environ["OPENROUTER_API_KEY"] = getpass("OpenRouter API key: ")

GENERATOR_MODEL = "anthropic/claude-haiku-5.5"
JUDGE_MODEL = "openai/gpt-5.4-nano"  # a different model family from the generator
JUDGE_EMBEDDING_MODEL = "openai/text-embedding-3-small"
RETRIEVAL_MODEL = "BAAI/bge-small-en-v1.5"

N = 100  # questions to evaluate; more questions give more stable means
SEED = 42
K = 10  # passages retrieved per question
MAX_CONCURRENCY = 16

RECALL_TARGET = 0.5
FAITHFULNESS_TARGET = 0.8

The targets at the end are used later to place each query in a 2×2 diagnostic grid. Set those before you run, not after you have seen the numbers.

Step 2: Load and Clean the BioASQ Mini Benchmark

The rag-mini-bioasq dataset provides the three things an end-to-end evaluation needs: a passage corpus, questions labelled with their relevant passage IDs, and a reference answer for each question. The labels drive recall@10, while the reference answers drive context precision.

Some corpus rows contain the literal string nan rather than text. This step drops those rows, drops any labels that point to them, and drops questions left with no usable label.

import json
import random

from datasets import load_dataset

corpus = load_dataset("rag-datasets/rag-mini-bioasq", "text-corpus", split="passages")
qa = load_dataset("rag-datasets/rag-mini-bioasq", "question-answer-passages", split="test")

passages = {row["id"]: row["passage"] for row in corpus if row["passage"].strip() != "nan"}

questions = []
dropped_labels = 0
for row in qa:
    labels = json.loads(row["relevant_passage_ids"])
    usable = [pid for pid in labels if pid in passages]
    dropped_labels += len(labels) - len(usable)
    if usable:
        questions.append(
            {"id": row["id"], "question": row["question"], "reference": row["answer"], "relevant": set(usable)}
        )

print(f"{len(passages):,} passages kept, {len(corpus) - len(passages):,} empty rows dropped")
print(f"{len(questions):,} questions kept, {len(qa) - len(questions):,} dropped, {dropped_labels:,} labels pointed at empty rows")

random.seed(SEED)
sample = random.sample(questions, N)

The deterministic seed applies only to question sampling. It does not make downstream LLM calls deterministic, which matters later when you compare runs.

Step 3: Index the Corpus in Qdrant

Qdrant’s FastEmbed integration embeds each passage locally during upload, so there is no separate embedding pipeline to run. The collection lives in memory. Embedding and uploading the full passage set is the slowest part of the setup; Qdrant will also warn that local mode is intended for small collections, which is acceptable here.

from qdrant_client import QdrantClient, models

client = QdrantClient(":memory:")
client.create_collection(
    "bioasq",
    vectors_config=models.VectorParams(
        size=client.get_embedding_size(RETRIEVAL_MODEL), distance=models.Distance.COSINE
    ),
)
client.upload_collection(
    "bioasq",
    ids=list(passages),
    vectors=[models.Document(text=text, model=RETRIEVAL_MODEL) for text in passages.values()],
    payload=[{"text": text} for text in passages.values()],
)
print(client.count("bioasq").count, "points")

No GPU is required. If you move this to a production collection, use a Qdrant server or Qdrant Cloud rather than :memory:.

Step 4: Measure Retrieval with Recall@10

Recall at 10 is the share of a question’s relevant passages that appear in its top 10 retrieved results. It isolates the retrieval half of the pipeline. The same snippet also stores each question’s retrieved contexts for generation and judging later.

def recall_at_k(retrieved_ids, relevant_ids):
    return len(set(retrieved_ids) & relevant_ids) / len(relevant_ids)


assert recall_at_k([1, 2, 3], {1, 9}) == 0.5

for q in sample:
    points = client.query_points(
        "bioasq",
        query=models.Document(text=q["question"], model=RETRIEVAL_MODEL),
        limit=K,
        with_payload=True,
    ).points
    q["retrieved_contexts"] = [p.payload["text"] for p in points]
    q["recall"] = recall_at_k([p.id for p in points], q["relevant"])

print(f"mean recall@{K}: {sum(q['recall'] for q in sample) / N:.3f}")

If a question has more than 10 labelled relevant passages, it cannot reach a recall@10 of 1.0 even if every retrieved passage is relevant. Read recall alongside the number of labels, not in isolation.

Step 5: Generate Grounded Answers

The generator receives the retrieved passages through a grounding prompt that forbids outside knowledge. Using an asynchronous client and a semaphore keeps generator calls bounded at MAX_CONCURRENCY. Failed generations return an empty string, which is treated as a pipeline failure rather than silently ignored.

import asyncio

from openai import AsyncOpenAI

openrouter = AsyncOpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
    max_retries=5,
    timeout=60,
)
limit = asyncio.Semaphore(MAX_CONCURRENCY)

GROUNDED_PROMPT = """You are answering questions using retrieved source material.

Answer the question below using only the provided context.
If the context does not contain the answer, say so explicitly.
Do not rely on outside knowledge.

Context:
{retrieved_context}

Question:
{query_text}
"""


async def generate(prompt_template, q):
    prompt = prompt_template.format(
        retrieved_context="\n\n".join(q["retrieved_contexts"]), query_text=q["question"]
    )
    async with limit:
        try:
            response = await openrouter.chat.completions.create(
                model=GENERATOR_MODEL,
                max_tokens=1024,
                extra_body={"reasoning": {"enabled": False}},
                messages=[{"role": "user", "content": prompt}],
            )
        except Exception as error:
            print(f"generation failed: {type(error).__name__}: {error}")
            return ""  # an empty answer is scored as a pipeline failure
    return response.choices[0].message.content or ""


baseline_answers = await asyncio.gather(*(generate(GROUNDED_PROMPT, q) for q in sample))
print(baseline_answers[0])

The prompt is kept in a named string so that a prompt change becomes a visible, reviewable diff.

Step 6: Score Answers with Ragas

Ragas measures answer quality using an LLM judge. The three metrics cover the most common failure modes: hallucination, off-topic answers, and misranked support passages.

import pandas as pd
from ragas.embeddings.base import embedding_factory
from ragas.llms import llm_factory
from ragas.metrics.collections import AnswerRelevancy, ContextPrecision, Faithfulness

judge = llm_factory(
    JUDGE_MODEL,
    client=openrouter,
    max_tokens=4096,
    extra_body={"reasoning": {"enabled": False}},  # no thinking mode
)
judge_embeddings = embedding_factory("openai", model=JUDGE_EMBEDDING_MODEL, client=openrouter)

faithfulness = Faithfulness(llm=judge)
answer_relevancy = AnswerRelevancy(llm=judge, embeddings=judge_embeddings)
context_precision = ContextPrecision(llm=judge)


async def score(metric, **fields):
    async with limit:
        try:
            return (await metric.ascore(**fields)).value
        except Exception as error:
            print(f"{metric.name} failed: {type(error).__name__}: {error}")
            return float("nan")


async def score_answers(answers):
    faithfulness_scores = asyncio.gather(*(
        score(faithfulness, user_input=q["question"], response=a, retrieved_contexts=q["retrieved_contexts"])
        for q, a in zip(sample, answers)
    ))
    relevancy_scores = asyncio.gather(*(
        score(answer_relevancy, user_input=q["question"], response=a)
        for q, a in zip(sample, answers)
    ))
    return await faithfulness_scores, await relevancy_scores


async def score_context_precision(q):
    if not (q.get("reference") or "").strip():
        return float("nan")  # reference-based metric: nothing to score against
    return await score(
        context_precision, user_input=q["question"], reference=q["reference"], retrieved_contexts=q["retrieved_contexts"]
    )


baseline_faithfulness, baseline_relevancy = await score_answers(baseline_answers)
baseline = pd.DataFrame({
    "question": [q["question"] for q in sample],
    "recall@10": [q["recall"] for q in sample],
    "context_precision": await asyncio.gather(*(score_context_precision(q) for q in sample)),
    "faithfulness": baseline_faithfulness,
    "answer_relevancy": baseline_relevancy,
})
baseline.mean(numeric_only=True).round(3)

Here is what each metric catches:

A failed judge call becomes a NaN rather than stopping the run. Averages hide those failures, so inspect the missing values explicitly.

Step 7: Inspect Baseline Failures

Before comparing runs, look at how many judge calls failed and which questions scored worst on faithfulness.

baseline.isna().sum()  # judge calls that failed, per metric

baseline.nsmallest(10, "faithfulness")

The row index in the lowest-faithfulness table matches baseline_answers[i] and sample[i]["retrieved_contexts"]. Read those answers next to their retrieved passages to see whether the generator is hallucinating or simply working with bad retrieval.

Step 8: Change One Thing — the Generation Prompt

The second run replaces the grounded prompt with an open-book prompt that lets the generator use its own knowledge as well as the retrieved context. Retrieval is untouched, so recall@10 and context precision carry over unchanged. Any movement in faithfulness or answer relevancy therefore belongs to generation.

OPEN_BOOK_PROMPT = """Answer the question below. The context may help, and you may also use your own knowledge.

Context:
{retrieved_context}

Question:
{query_text}
"""

changed_answers = await asyncio.gather(*(generate(OPEN_BOOK_PROMPT, q) for q in sample))
changed_faithfulness, changed_relevancy = await score_answers(changed_answers)
changed = baseline.assign(faithfulness=changed_faithfulness, answer_relevancy=changed_relevancy)

pd.DataFrame({
    "grounded prompt": baseline.mean(numeric_only=True),
    "open-book prompt": changed.mean(numeric_only=True),
}).round(3)

Step 9: Place Each Query in the 2×2 Diagnostic Grid

Comparing means across runs is useful, but the per-query movement tells you which half of the pipeline to investigate. Each query is classified as high or low recall and high or low faithfulness using the targets from the configuration cell.

def quadrant(recall, faithfulness):
    retrieval = "high" if recall >= RECALL_TARGET else "low"
    generation = "high" if faithfulness >= FAITHFULNESS_TARGET else "low"
    return f"{retrieval} recall, {generation} faithfulness"


pd.DataFrame({
    name: run.dropna(subset=["faithfulness"])
    .apply(lambda row: quadrant(row["recall@10"], row["faithfulness"]), axis=1)
    .value_counts()
    for name, run in [("grounded prompt", baseline), ("open-book prompt", changed)]
}).fillna(0).astype(int)

The signal to read is the movement between the two columns. Since recall@10 is identical in both runs, any query that leaves high faithfulness under the open-book prompt moved because of generation, not retrieval.

Step 10: Record the Run

Both the generator and the judge sample their output, so a rerun changes the numbers. Keep model IDs and package versions next to every result you compare.

import datetime
import importlib.metadata

print("run date:", datetime.date.today())
for package in ["ragas", "langchain-community", "qdrant-client", "fastembed", "openai", "datasets", "pandas"]:
    print(f"{package}=={importlib.metadata.version(package)}")
print(f"generator={GENERATOR_MODEL} judge={JUDGE_MODEL} judge_embeddings={JUDGE_EMBEDDING_MODEL}")
print(f"retrieval={RETRIEVAL_MODEL} N={N} SEED={SEED} K={K}")

What to watch out for

• Local mode is not production. Qdrant’s in-memory collection is fine for evaluating a small benchmark, but it is not persisted, not replicated, and not designed for large production corpora. Use a Qdrant server or Qdrant Cloud when you move beyond evaluation.
• Embedding time is the setup bottleneck. Embedding the full passage set on CPU is the slowest part of setup, and a free Colab runtime may take longer than a local laptop. Do not mistake this for retrieval latency; it is a one-time indexing cost.
• Recall@10 has a ceiling when labels exceed K. Some questions may have more than 10 labelled relevant passages. A recall of 0.5 on such a question can still mean every retrieved passage was relevant. Read recall alongside the number of labels, not in isolation.
• Averages hide failing queries. Judge calls that fail are returned as NaN, not zero. If you only look at mean(), a metric with many failed calls can look healthy. Always inspect isna().sum() and the worst-scoring rows.
• The 1024-token generation cap affects scoring. The judge scores the text as written. If a long answer gets truncated by max_tokens=1024, the judge might mark it incomplete or less faithful even when the retriever was fine. Raise the cap if your answers run long.
• A model judging its own output is biased. The source deliberately chooses a judge from a different model family. Changing the judge to the same family as the generator will tend to inflate scores and mask regressions.
• Reasoning mode should stay off. The generator and judge both set reasoning.enabled=False. Enabling it may change cost, latency, and failure rates, and it can make scores harder to compare across runs.
• Nondeterminism is not fully controlled. SEED fixes the question sample, but LLM completions are sampled. Two runs with the same seed can still produce different answers and scores. Keep run metadata and the prompt strings alongside results.
• OpenRouter rate limits and costs are real. Concurrency 16 with retries can mask throttling as latency. The notebook does not record cost; OpenRouter billing depends on model, tokens, concurrency, and retries, so monitor spend on a small run before scaling.

Where to go next

The natural next step is to replace the BioASQ mini benchmark with your own corpus and question set, then run the same grounded-versus-open-book comparison. If you want to compare FastEmbed’s BAAI/bge-small-en-v1.5 against a newer embedding model, our coverage of Perplexity’s PPLX Embed v2 is a useful starting point. For a deeper discussion of the evaluation framework itself, see Qdrant’s Evaluating Pipeline Output Quality documentation.