Build a Multi-Modal RAG Pipeline with LlamaIndex and Qdrant

Retrieval-augmented generation works well for text, but real-world documents are rarely text-only. Specification sheets carry tables and diagrams. Research reports embed photographs. Wikipedia articles mix prose with infographics. A pipeline that ignores anything that isn't plain text leaves most of that signal on the floor. Multi-modal RAG fixes this by indexing images and text into separate but jointly-queryable vector stores, retrieving across both modalities at query time, and feeding the combined evidence to a vision-capable LLM that can reason over screenshots, photographs, and written content simultaneously.

Who needs this? Anyone building knowledge assistants over product documentation, financial filings with embedded charts, scientific literature with figures, or any corpus where information is split across media types. The technique covered here — adapted from LlamaIndex's Multi_Modal_RAG_System.ipynb (MIT licence) — uses OpenAI's GPT-4 Vision, CLIP embeddings, and a local Qdrant instance to build an end-to-end pipeline you can run on a single machine.

Cost and time expectations. GPT-4 Vision charges per image token in addition to text tokens, so a corpus with hundreds of images will cost meaningfully more than a pure-text RAG system of the same size. CLIP embedding runs locally on CPU but a GPU cuts indexing time significantly. An OpenAI API key with GPT-4 Vision access is required; standard tier access is sufficient.

---

Prerequisites

• Python 3.9 or later
• OpenAI API key with access to gpt-4-vision-preview
• 8 GB RAM minimum; 16 GB recommended for the full Wikipedia corpus
• Disk space: ~2 GB for the mixed-wiki image set and Qdrant on-disk store
• No GPU strictly required; CLIP will run on CPU

---

Step 1: Install dependencies

The pipeline pulls in several packages that don't travel together by default. Install them all before importing anything, because llama-index-embeddings-clip has its own CLIP dependency that must be satisfied by the GitHub build rather than the PyPI stub.

!pip install llama-index-multi-modal-llms-openai
!pip install llama-index-vector-stores-qdrant
!pip install llama_index ftfy regex tqdm
!pip install llama-index-embeddings-clip
!pip install git+https://github.com/openai/CLIP.git
!pip install matplotlib scikit-image

---

Step 2: Set your OpenAI API key

import os

os.environ["OPENAI_API_KEY"] = "sk-..."

Replace the placeholder with your real key. In a shared environment, prefer a secrets manager or a .env file rather than hardcoding it here.

---

Step 3: Verify multi-modal inference with a single URL image

Before building the full pipeline, confirm that GPT-4 Vision is reachable and returning sensible descriptions. This catches auth problems and quota issues early.

from llama_index.multi_modal_llms.openai import OpenAIMultiModal

from llama_index.core.multi_modal_llms.generic_utils import load_image_urls


image_urls = [
    "https://res.cloudinary.com/hello-tickets/image/upload/c_limit,f_auto,q_auto,w_1920/v1640835927/o3pfl41q7m5bj8jardk0.jpg",
]

image_documents = load_image_urls(image_urls)

openai_mm_llm = OpenAIMultiModal(
    model="gpt-4-vision-preview", max_new_tokens=300
)

response = openai_mm_llm.complete(
    prompt="Describe the images as an alternative text",
    image_documents=image_documents,
)

print(response)

max_new_tokens=300 is deliberately modest — enough for a descriptive caption but not runaway generation. The openai_mm_llm object is reused throughout the rest of the guide, so keep it in scope.

---

Step 4: Download the local image corpus

The pipeline's real data is a set of Tesla Model Y specification screenshots. Download them into a dedicated directory.

from pathlib import Path

input_image_path = Path("input_images")
if not input_image_path.exists():
    Path.mkdir(input_image_path)

!wget "https://docs.google.com/uc?export=download&id=1nUhsBRiSWxcVQv8t8Cvvro8HJZ88LCzj" -O ./input_images/long_range_spec.png
!wget "https://docs.google.com/uc?export=download&id=19pLwx0nVqsop7lo0ubUSYTzQfMtKJJtJ" -O ./input_images/model_y.png
!wget "https://docs.google.com/uc?export=download&id=1utu3iD9XEgR5Sb7PrbtMf1qw8T1WdNmF" -O ./input_images/performance_spec.png
!wget "https://docs.google.com/uc?export=download&id=1dpUakWMqaXR4Jjn1kHuZfB0pAXvjn2-i" -O ./input_images/price.png
!wget "https://docs.google.com/uc?export=download&id=1qNeT201QAesnAP5va1ty0Ky5Q_jKkguV" -O ./input_images/real_wheel_spec.png

Visualise what you've got:

from PIL import Image
import matplotlib.pyplot as plt
import os


def plot_images(image_paths):
    images_shown = 0
    plt.figure(figsize=(16, 9))
    for img_path in image_paths:
        if os.path.isfile(img_path):
            image = Image.open(img_path)

            plt.subplot(2, 3, images_shown + 1)
            plt.imshow(image)
            plt.xticks([])
            plt.yticks([])

            images_shown += 1
            if images_shown >= 9:
                break

image_paths = []
for img_path in os.listdir("./input_images"):
    image_paths.append(str(os.path.join("./input_images", img_path)))
plot_images(image_paths)

plot_images is also used later to render image sources returned by the query engine — define it once and reuse it.

---

Step 5: Load local images as documents and verify vision understanding

from llama_index.multi_modal_llms.openai import OpenAIMultiModal
from llama_index.core import SimpleDirectoryReader

# put your local directore here
image_documents = SimpleDirectoryReader("./input_images").load_data()

response = openai_mm_llm.complete(
    prompt="Describe the images as an alternative text",
    image_documents=image_documents,
)

print(response)

At this point you're sending all five specification images to GPT-4 Vision in a single call. If the model describes range figures, pricing, and wheel specs, your local image loading is working correctly.

---

Step 6: Build the mixed-modality Wikipedia corpus

This is where the pipeline earns its name. The code below fetches plain-text article content and up to 15 images per article for 11 electric and performance vehicle pages.

import requests


def get_wikipedia_images(title):
    response = requests.get(
        "https://en.wikipedia.org/w/api.php",
        params={
            "action": "query",
            "format": "json",
            "titles": title,
            "prop": "imageinfo",
            "iiprop": "url|dimensions|mime",
            "generator": "images",
            "gimlimit": "50",
        },
    ).json()
    image_urls = []
    for page in response["query"]["pages"].values():
        if page["imageinfo"][0]["url"].endswith(".jpg") or page["imageinfo"][
            0
        ]["url"].endswith(".png"):
            image_urls.append(page["imageinfo"][0]["url"])
    return image_urls

from pathlib import Path
import requests
import urllib.request
import time

image_uuid = 0
# image_metadata_dict stores images metadata including image uuid, filename and path
image_metadata_dict = {}
MAX_IMAGES_PER_WIKI = 15

wiki_titles = {
    "Tesla Model Y",
    "Tesla Model X",
    "Tesla Model 3",
    "Tesla Model S",
    "Kia EV6",
    "BMW i3",
    "Audi e-tron",
    "Ford Mustang",
    "Porsche Taycan",
    "Rivian",
    "Polestar",
}

data_path = Path("mixed_wiki")
if not data_path.exists():
    Path.mkdir(data_path)

for title in wiki_titles:
    response = requests.get(
        "https://en.wikipedia.org/w/api.php",
        params={
            "action": "query",
            "format": "json",
            "titles": title,
            "prop": "extracts",
            "explaintext": True,
        },
    ).json()
    page = next(iter(response["query"]["pages"].values()))
    wiki_text = page["extract"]

    with open(data_path / f"{title}.txt", "w") as fp:
        fp.write(wiki_text)

    images_per_wiki = 0
    try:
        # page_py = wikipedia.page(title)
        list_img_urls = get_wikipedia_images(title)

        # print(list_img_urls)

        for url in list_img_urls:
            if (
                url.endswith(".jpg")
                or url.endswith(".png")
                or url.endswith(".svg")
            ):
                image_uuid += 1
                # image_file_name = title + "_" + url.split("/")[-1]

                urllib.request.urlretrieve(
                    url, data_path / f"{image_uuid}.jpg"
                )
                time.sleep(1)
                images_per_wiki += 1
                # Limit the number of images downloaded per wiki page to 15
                if images_per_wiki > MAX_IMAGES_PER_WIKI:
                    break
    except Exception as e:
        print(e)
        print(
            "Number of images found for Wikipedia page: {} are {}".format(
                title, images_per_wiki
            )
        )
        continue

The time.sleep(1) call between image downloads is intentional — Wikipedia's API will rate-limit aggressive scrapers. Removing it causes intermittent failures on larger corpora.

---

Step 7: Set up dual Qdrant vector stores

Text and images are embedded by different models (OpenAI text embeddings for prose, CLIP for images) and must live in separate Qdrant collections. StorageContext ties them together so LlamaIndex knows which store to query for which modality.

from llama_index.core.indices import MultiModalVectorStoreIndex
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.core import SimpleDirectoryReader, StorageContext
import qdrant_client


# Create a local Qdrant vector store
client = qdrant_client.QdrantClient(path="qdrant_mm_db")

text_store = QdrantVectorStore(
    client=client, collection_name="text_collection"
)
image_store = QdrantVectorStore(
    client=client, collection_name="image_collection"
)
storage_context = StorageContext.from_defaults(
    vector_store=text_store, image_store=image_store
)

The path="qdrant_mm_db" argument creates an on-disk Qdrant instance — no Docker, no server process required. This is fine for development but not production; a networked Qdrant instance is preferable once your corpus grows beyond a few thousand documents or requires concurrent access.

---

Step 8: Index the mixed corpus

# Create the MultiModal index
documents = SimpleDirectoryReader("./mixed_wiki/").load_data()

index = MultiModalVectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
)

SimpleDirectoryReader automatically distinguishes text files from image files and routes them to the appropriate loader. CLIP runs locally during this step to produce image embeddings — expect several minutes on CPU for the ~160-image corpus.

---

Step 9: Build the query engine and run a query

from llama_index.core import PromptTemplate
from llama_index.core.query_engine import SimpleMultiModalQueryEngine

qa_tmpl_str = (
    "Context information is below.\n"
    "---------------------\n"
    "{context_str}\n"
    "---------------------\n"
    "Given the context information and not prior knowledge, "
    "answer the query.\n"
    "Query: {query_str}\n"
    "Answer: "
)
qa_tmpl = PromptTemplate(qa_tmpl_str)

query_engine = index.as_query_engine(
    llm=openai_mm_llm, text_qa_template=qa_tmpl
)

query_str = "Tell me more about the Tesla Model X"
response = query_engine.query(query_str)

print(response)

The prompt template is deliberately strict: "given the context information and not prior knowledge." Without it, GPT-4 Vision will answer from its training data rather than your retrieved sources, defeating the purpose of RAG entirely. This same prior-knowledge bleed is a known failure mode in LLM-as-judge evaluation pipelines, where a model's pretrained knowledge contaminates what should be retrieval-conditioned scoring.

---

Step 10: Inspect cited sources

from llama_index.core.response.notebook_utils import display_source_node

for text_node in response.metadata["text_nodes"]:
    display_source_node(text_node, source_length=200)
plot_images(
    [n.metadata["file_path"] for n in response.metadata["image_nodes"]]
)

response.metadata carries two separate node lists: text_nodes for retrieved text chunks with their source file paths and relevance scores, and image_nodes for retrieved images. Rendering both lets you audit exactly which evidence the model used — critical for any production system where hallucination carries real cost.

---

Modality and store option comparison

---

What to watch out for

CLIP retrieval noise. CLIP embeds images into the same space as text queries using visual-semantic similarity, not document-level relevance. A query about "Tesla battery range" may surface a photograph of a scenic road because CLIP associates open roads with the vehicle context. Always inspect image_nodes before trusting them as evidence.

GPT-4 Vision token costs at scale. Each image passed to the model costs additional tokens proportional to its resolution and tile count. A corpus with 160 images and a retrieval top-k of 5 per query means every query potentially sends 5 high-resolution images to the API. Profile your per-query cost before deploying at volume.

Wikipedia image quality is unpredictable. The get_wikipedia_images helper retrieves whatever images appear in the article, including icons, flags, infobox thumbnails, and licence badges. Many carry no useful signal for automotive queries. A production system should filter by minimum dimensions or run a relevance classifier before indexing.

image_metadata_dict is declared but never populated. The source notebook initialises this dictionary but does not write to it during the download loop. If your downstream code depends on it beyond the scope shown here, you will need to populate it yourself.

On-disk Qdrant is not thread-safe for concurrent writes. If you're running indexing and querying simultaneously — in a web server, for example — switch to the networked Qdrant client. The path= mode is single-process only.

SVG images are downloaded but saved with a .jpg extension. The download loop accepts .svg URLs, which PIL cannot open. These will raise exceptions when plot_images or SimpleDirectoryReader tries to process them. Filter SVG URLs out before saving, or wrap image loading in a try/except.

The gpt-4-vision-preview model identifier may be deprecated. OpenAI periodically retires preview endpoint names. If you receive a model-not-found error, check the current canonical name in the OpenAI documentation and substitute it into the OpenAIMultiModal constructor.

---

Where to go next

With a working multi-modal RAG pipeline, the natural extensions are: adding a re-ranker stage to post-filter image results before they reach the LLM; experimenting with open-weight vision models to reduce per-query API cost (see Liquid AI's LFM2-5-VL 3B as a lightweight locally-hostable alternative); and replacing the on-disk Qdrant store with a persistent networked instance backed by proper metadata filters. The LlamaIndex documentation covers MultiModalVectorStoreIndex persistence and incremental ingestion for keeping your index current as source documents change.