Build Retrieval-Augmented Conversational Memory with Milvus

When an assistant can see only the latest user message, it forgets earlier preferences, corrections, and constraints. The fix is conversational memory: store each exchange in a vector database, retrieve the relevant fragments at inference time, and inject only those fragments into the prompt. This guide builds that pattern with Milvus and LangChain using the exact code from Milvus Bootcamp's conv_mem_langchain.ipynb notebook. The notebook is licensed under Apache-2.0 and is available at https://github.com/milvus-io/bootcamp.

Who needs this? Engineers building support bots, onboarding agents, or any assistant where user preferences and prior turns should survive across requests. The trade-off is explicit: you keep token costs low by not resending the full history, but you add a vector database, an embedding step, and retrieval latency. In this tutorial, the external service cost is OpenAI API usage. The code seeds five facts, so memory.save_context runs five embedding operations, followed by the example completion calls. The Milvus default server runs locally through the milvus package, so no separate Milvus service or container is required for the walkthrough.

Prerequisites

The source notebook assumes the following:

• A Python environment with langchain, milvus, pymilvus, and python-dotenv installed. The notebook also imports openai, so install that too if LangChain does not already pull it in.
• An OpenAI API key stored in a .env file as OPENAI_API_KEY.
• A local machine that can run the embedded Milvus instance through default_server. The source does not use a Dockerized Milvus deployment.
• Enough disk space for a small local collection. The seeded examples are tiny and run comfortably on a development laptop.

Step 1: Install the dependencies

The original notebook includes a commented install command for reproducibility. Run it in a terminal or uncomment it in a cell.

# !pip install langchain milvus pymilvus python-dotenv

If your environment does not already include openai, install it separately or add it to the command.

Step 2: Import LangChain components and set the OpenAI key

The memory pattern uses LangChain's VectorStoreRetrieverMemory, a conversation chain, and a prompt template with a {history} placeholder.

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.llms import OpenAI
from langchain.memory import VectorStoreRetrieverMemory
from langchain.chains import ConversationChain
from langchain.prompts import PromptTemplate

Load the environment and configure the OpenAI client.

import os
from dotenv import load_dotenv
import openai
load_dotenv()
openai.api_key = os.getenv("OPENAI_API_KEY")

Step 3: Start the local Milvus server and reset the collection

The milvus package provides default_server, a lightweight embedded Milvus instance. You start it without a separate service.

from milvus import default_server

default_server.start()

After it starts, the server is reachable through default_server.listen_port, which the connection code uses below.

Before creating the vector store, import LangChain's Milvus wrapper and instantiate the embedding model.

from langchain.vectorstores import Milvus
embeddings = OpenAIEmbeddings()

The source then connects to the embedded Milvus and drops any existing LangChainCollection. This makes the guide re-runnable. Without this step, a second run can duplicate the seeded vectors or fail on a collection conflict.

# !pip install pymilvus
from pymilvus import utility, connections

connections.connect(host="127.0.0.1", port=default_server.listen_port)
utility.drop_collection('LangChainCollection')

The extra commented pymilvus install is from the original notebook cell and is harmless if the package is already installed.

Step 4: Create the vector store and wrap it as retriever memory

The vector store is created empty. from_documents normally accepts a list of documents, but here {} deliberately creates an empty collection that will be filled later through the memory object.

vectordb = Milvus.from_documents(
    {},
    embeddings,
    connection_args={"host": "127.0.0.1", "port": default_server.listen_port})

Wrap the collection as a retriever, then pass that retriever into VectorStoreRetrieverMemory.

retriever = Milvus.as_retriever(vectordb, search_kwargs=dict(k=1))
memory = VectorStoreRetrieverMemory(retriever=retriever)

The search_kwargs setting k=1 means only the single most similar prior exchange is injected into the prompt each turn. That is a deliberate cost control: you trade recall breadth for a smaller prompt.

Before building the conversation chain, it is worth comparing the memory strategy with the two common alternatives.

This guide implements the third row: vector retrieval injection. It keeps the prompt compact, but it can fail when an answer requires combining multiple past facts that are not individually similar to the current question. The k=1 setting makes this especially aggressive.

Step 5: Seed five user preferences and check retrieval

The memory object saves each exchange as an input-output pair. These pairs are embedded and stored in Milvus.

about_me = [
    {"input": "My favorite snack is chocolate",
     "output": "Nice"},
    {"input": "My favorite sport is swimming",
     "output": "Cool"},
    {"input": "My favorite beer is Guinness",
     "output": "Great"},
    {"input": "My favorite dessert is cheesecake",
     "output": "Good to know"},
    {"input": "My favorite musician is Taylor Swift",
     "output": "Same"}
]
for example in about_me:
    memory.save_context({"input": example["input"]}, {"output": example["output"]})

Before involving the LLM, check what the memory retrieves for a direct question.

print(memory.load_memory_variables({"prompt": "who is my favorite musician?"})["history"])

This check isolates the retrieval path. With k=1, it should return the most similar seeded exchange rather than all five facts, confirming that the vector lookup works independently of the language model.

Step 6: Build the ConversationChain with a retrieval-aware prompt

The prompt template has two input variables: history receives the retrieved exchange, and input receives the current user message. The instruction (You do not need to use these pieces of information if not relevant) gives the model permission to ignore a poor retrieval result instead of treating it as a command.

llm = OpenAI(temperature=0) # Can be any valid LLM
_DEFAULT_TEMPLATE = """The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Relevant pieces of previous conversation:
{history}

(You do not need to use these pieces of information if not relevant)

Current conversation:
Human: {input}
AI:"""
PROMPT = PromptTemplate(
    input_variables=["history", "input"], template=_DEFAULT_TEMPLATE
)
conversation_with_summary = ConversationChain(
    llm=llm, 
    prompt=PROMPT,
    memory=memory,
    verbose=True
)
conversation_with_summary.predict(input="Hi, my name is Gary, what's up?")

Because memory is attached to the conversation chain, this first turn is saved into the same vector memory after the model responds. That means the seeded facts are not the only retrievable history. New exchanges become part of the memory too.

Step 7: Ask the assistant to recall seeded facts

Now query the assistant for stored facts.

conversation_with_summary.predict(input="who is my favorite musician?")

conversation_with_summary.predict(input="Whats my favorite dessert?")

conversation_with_summary.predict(input="What's my name?")

Each call performs a fresh similarity search and injects only the most relevant prior exchange. The musician and dessert questions should retrieve the seeded facts, while the name question should retrieve the Hi, my name is Gary exchange saved during the first conversation turn. The typo Whats is in the source notebook and is preserved here so the code still matches the tutorial.

Step 8: Stop the server

When finished, release the embedded Milvus instance.

default_server.stop()

What to watch out for

Old LangChain import paths. The source uses from langchain.embeddings.openai import OpenAIEmbeddings and from langchain.llms import OpenAI. Those are older LangChain API paths. On recent LangChain releases they may raise deprecation warnings or fail outright. Keep the bootcamp environment pinned, or treat this code as an architectural reference and adapt the imports to your installed LangChain version.

k=1 is very restrictive. If a question requires two stored facts, such as asking for both the user's snack and dessert, top-1 retrieval will inject only one prior exchange. The model may answer one part and ignore or hallucinate the other. Increase k when multi-fact recall matters, but remember that larger retrieved history increases prompt length and cost.

Re-running without the drop step causes duplicates. The guide calls utility.drop_collection('LangChainCollection') before creating the vector store. If you omit that line and rerun the notebook, from_documents can fail with a collection conflict or leave stale vectors behind. Keep the drop step unless you have a deliberate persistence strategy.

The embedded server is not production persistence. default_server is convenient for prototyping, but do not assume its data survives a restart or that it can handle production traffic. For anything beyond a tutorial, run a standalone Milvus deployment with a persistent volume and replace the connection arguments accordingly.

Embedding and completion calls both cost money. Each memory.save_context call embeds text through OpenAI. That is negligible for the five seeded examples, but a long-running agent that saves every turn into Milvus will accumulate embedding API charges. Separately, even with retrieval, the prompt still includes the retrieved history plus the current input. For a deeper look at how prompt length affects cost, see Tokengate's long-context prefill analysis.

Retrieval quality depends on the embedding model. OpenAI embeddings are a sensible default, but domain-specific slang, product names, or multilingual text may need a model tuned for your data. A poor embedding space causes the retriever to inject irrelevant prior turns, and the model may struggle to ignore them despite the prompt's instruction.

Where to go next

The Milvus Bootcamp repository is a useful next step for more notebook patterns involving vector memory and Milvus. If you need production memory, explore Milvus standalone or Zilliz Cloud, and consider LangChain's ConversationSummaryBufferMemory for hybrid summarization-plus-retrieval pipelines. The Tokengate article linked above is also a useful follow-up when prompt length becomes the limiting factor.