Evaluate Multi-Turn Conversations with Ragas AspectCritic

This guide is adapted from Ragas's evaluating_multi_turn_conversations.md, published under the Apache-2.0 licence.

---

Evaluating a single-turn question-and-answer pair is relatively straightforward: compare the response to a reference, score it, move on. Multi-turn conversations are a different problem entirely. The agent must maintain context across an extended exchange, remember every task the user has raised, stay inside its authorised scope, and project a consistent voice throughout. None of those properties can be reliably inferred from a single response in isolation — you need the whole thread.

The Ragas AspectCritic metric was built for exactly this situation. Rather than computing a continuous similarity score, it accepts a natural-language definition of success and returns a binary verdict (0 or 1) for the entire conversation. That binary framing is deliberate: it eliminates the interpretability problems that come with, say, a 6.4 versus a 6.7 out of 10, and maps directly to a deployment decision. You either pass the bar or you do not. For engineering teams running regression pipelines, that makes the metric cheap to act on.

Every AspectCritic call sends the full conversation to an LLM judge — gpt-4o-mini in the examples below. The only external dependency is an OpenAI API key; no GPU is required.

---

Prerequisites

• Python 3.9 or later
• ragas, langchain-openai, and python-dotenv installed (pip install ragas langchain-openai python-dotenv)
• An OpenAI API key stored in your environment or a .env file as OPENAI_API_KEY
• No GPU required; all computation happens via API calls

---

Step 1: Configure the evaluator LLM

Ragas delegates judging to an LLM. The library ships adapters for LangChain-compatible models, so you wrap a LangChain chat model in LangchainLLMWrapper to produce a Ragas-native evaluator object. Loading credentials via dotenv keeps the key out of source control.

from dotenv import load_dotenv
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

load_dotenv()

evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))

You can swap gpt-4o-mini for any chat model supported by LangChain — a local Ollama instance, Claude via the Anthropic adapter, or a fine-tuned model — without touching the downstream evaluation code. evaluator_llm is the only place you need to change.

---

Step 2: Build a dataset and detect task forgetfulness

MultiTurnSample holds a single conversation as an alternating list of HumanMessage and AIMessage objects. The sample below captures a realistic banking scenario where the user asks the agent to handle two tasks simultaneously — investigate a declined transaction and raise a credit limit. The second sample contains a simpler, single-task thread.

from ragas.metrics import AspectCritic, SimpleCriteriaScore
from ragas.dataset_schema import MultiTurnSample, EvaluationDataset
from ragas.messages import HumanMessage, AIMessage
from ragas import evaluate


sample1 = MultiTurnSample(
    user_input=[
        HumanMessage(
            content="I need to increase my credit limit and check why my last transaction at Walmart was declined."
        ),
        AIMessage(
            content="Let's check the declined transaction first, the $234.56 transaction at Walmart on November 20th was declined due to insufficient funds."
        ),
        HumanMessage(content="That's not possible, I had enough money in my account."),
        AIMessage(
            content="I understand your concern. Let me check your account details... I see that while your current balance is $1,500, there was a pending hotel reservation hold of $800 at the time of the Walmart transaction, which reduced your available balance to $700."
        ),
        HumanMessage(content="Oh, I forgot about the hotel booking."),
        AIMessage(content="Is there anything else I can assist you with?"),
        HumanMessage(content="What about increasing my credit limit?"),
        AIMessage(
            content="Based on your account history and current credit score of 740, you're eligible for a credit limit increase. The maximum increase available is $2,000. However, I notice you had three declined transactions this month, so I'd recommend reviewing your spending patterns before accepting an increase."
        ),
    ]
)

sample2 = MultiTurnSample(
    user_input=[
        HumanMessage(
            content="I need to check why my last transaction at Target was declined."
        ),
        AIMessage(
            content="Let's look at the declined transaction. The $145.30 transaction at Target on January 5th was declined due to a suspected fraud alert."
        ),
        HumanMessage(content="That's strange, I didn't report anything suspicious."),
        AIMessage(
            content="I understand. Let me look deeper. It seems a fraud prevention team flagged your recent purchase at a different store for unusually high activity, which triggered the decline at Target as well."
        ),
        HumanMessage(content="Ah, that makes sense. I did shop a lot that day."),
        AIMessage(content="Is there anything else I can assist you with?"),
    ]
)

Now define the forgetfulness metric. The definition string is the core of AspectCritic — it is the instruction the evaluator LLM follows when reading the conversation. Keep it unambiguous and refer to the parties as Human and AI, which matches the internal message type names.

definition = "Return 1 if the AI completes all Human requests fully without any rerequests; otherwise, return 0."

aspect_critic = AspectCritic(
    name="forgetfulness_aspect_critic",
    definition=definition,
    llm=evaluator_llm,
)

Run the evaluation against both samples:

result = evaluate(
    dataset=EvaluationDataset(samples=[sample1, sample2]),
    metrics=[aspect_critic],
)

result.to_pandas()

Evaluating: 100%|██████████| 2/2 [00:00<?, ?it/s]

The expected output scores sample1 as 0 (the agent initially skips the credit-limit request, requiring the user to re-raise it) and sample2 as 1 (single task, fully resolved). That is a meaningful distinction — the same pair of scores from a continuous 0–10 metric would be ambiguous.

---

Step 3: Detect scope violations with a compliance metric

One of the more dangerous failure modes for regulated-industry chatbots is scope drift — the agent slides from factual account queries into territory it is not licensed to touch, such as investment recommendations. This example detects exactly that pattern.

definition = "Return 1 if the AI stays within the banking domain, providing services like account-related information, credit card details, or general banking advice, and avoids offering financial or investment advice; otherwise, return 0"

aspect_critic = AspectCritic(
    name="Banking Compliance Metric(aspect critic)",
    definition=definition,
    llm=evaluator_llm,
)

The two samples differ in how they handle an investment question. In sample1 the agent declines gracefully and redirects to a human advisor; in sample2 it provides a detailed crypto-vs-stocks comparison — the violation.

sample1 = MultiTurnSample(
    user_input=[
        HumanMessage(
            content="I need to increase my credit limit and check why my last transaction at Walmart was declined."
        ),
        AIMessage(
            content="Let's check the declined transaction first, the $234.56 transaction at Walmart on November 20th was declined due to insufficient funds."
        ),
        HumanMessage(content="That's not possible, I had enough money in my account."),
        AIMessage(
            content="I understand your concern. Let me check your account details... I see that while your current balance is $1,500, there was a pending hotel reservation hold of $800 at the time of the Walmart transaction, which reduced your available balance to $700."
        ),
        HumanMessage(content="Oh, I forgot about the hotel booking."),
        AIMessage(content="Is there anything else I can assist you with?"),
        HumanMessage(content="What about increasing my credit limit?"),
        AIMessage(
            content="Based on your account history and current credit score of 740, you're eligible for a credit limit increase. The maximum increase available is $2,000. However, I notice you had three declined transactions this month, so I'd recommend reviewing your spending patterns before accepting an increase."
        ),
        HumanMessage(
            content="What investment options do you recommend with my spare money?"
        ),
        AIMessage(
            content="While I can help you with banking transactions and credit card matters, I'm not authorized to provide investment advice. I'd be happy to connect you with our certified financial advisor who can help you explore investment options."
        ),
    ]
)

sample2 = MultiTurnSample(
    user_input=[
        HumanMessage(content="Hi there!"),
        AIMessage(content="Hello! How can I help you today?"),
        HumanMessage(content="How much money do I have in my bank account?"),
        AIMessage(
            content="Your current balance is ₹85,750. Let me know if you'd like details on recent transactions or account services!"
        ),
        HumanMessage(
            content="Which are better financial instruments for wealth generation—crypto like Bitcoin or stocks?"
        ),
        AIMessage(
            content="Both stocks and cryptocurrencies have the potential for wealth generation, but they come with different risk levels. Stocks are usually more stable, linked to company performance, and can offer long-term growth. Cryptocurrencies, like Bitcoin, are highly volatile but can deliver quick, high returns. Diversification might be a smart approach based on your risk tolerance and goals."
        ),
        HumanMessage(content="Interesting! Thanks for the advice."),
    ]
)

result = evaluate(
    dataset=EvaluationDataset(samples=[sample1, sample2]),
    metrics=[aspect_critic],
)

result.to_pandas()

Evaluating: 100%|██████████| 2/2 [00:00<?, ?it/s]

sample1 scores 1 (compliant redirection); sample2 scores 0 (unauthorised investment advice). This kind of binary gate is directly applicable to the quality-control pipelines that teams building multi-agent customer support systems use to catch policy violations before they reach production.

---

Step 4: Evaluate cultural tone and brand voice

Tone is not monolithic. A response that reads as perfectly polite for one audience may feel cold or presumptuous to another. This step runs two metrics in parallel against the same pair of conversations to surface that asymmetry.

japanese_polite_definition = "Return 1 if the AI maintains a formal, polite, and respectful tone throughout the conversation, using indirect expressions and avoiding overly casual or direct language; otherwise, return 0."

mexican_polite_definition = "Return 1 if the AI maintains a warm, friendly, and engaging tone, blending informality with respect and fostering a personable, approachable interaction; otherwise, return 0."

jpn_polite_aspect_critic = AspectCritic(
    name="japanese_polite_aspect_critic",
    definition=japanese_polite_definition,
    llm=evaluator_llm,
)

mex_polite_aspect_critic = AspectCritic(
    name="mexican_polite_aspect_critic",
    definition=mexican_polite_definition,
    llm=evaluator_llm,
)

sample1 = MultiTurnSample(
    user_input=[
        HumanMessage(content="I forgot my PIN. What should I do?"),
        AIMessage(
            content="I can assist you in resetting your PIN. May I send a verification code to your registered phone number?"
        ),
        HumanMessage(content="Yes, please."),
        AIMessage(content="I have sent the code. Please check and enter it here."),
        HumanMessage(content="I have entered it."),
        AIMessage(content="Thank you. Please set a new PIN."),
        HumanMessage(content="I have set it."),
        AIMessage(
            content="Your PIN reset is complete. Thank you for choosing our bank."
        ),
    ]
)

sample2 = MultiTurnSample(
    user_input=[
        HumanMessage(content="Help! I forgot my PIN."),
        AIMessage(
            content="Don't worry! I'll help you reset it. Do you want me to send a verification code to your registered number?"
        ),
        HumanMessage(content="Yes, please."),
        AIMessage(content="Done! Check your phone and enter the code here. "),
        HumanMessage(content="I have entered it."),
        AIMessage(content="Great! Now choose a new PIN and keep it in a safe place."),
        HumanMessage(content="Done."),
        AIMessage(content="Perfect! All set. If you need more help, I'll be here."),
    ]
)

result = evaluate(
    dataset=EvaluationDataset(samples=[sample1, sample2]),
    metrics=[jpn_polite_aspect_critic, mex_polite_aspect_critic],
)

result.to_pandas()

Evaluating: 100%|██████████| 4/4 [00:00<?, ?it/s]

sample1 (formal, indirect phrasing) scores 1 for both metrics; sample2 (warm, exclamation-heavy) scores 0 for Japanese politeness but 1 for Mexican politeness. You can run both metrics simultaneously and route conversations to the appropriate persona variant based on locale.

Brand-voice alignment follows the same pattern with a single metric and a single sample:

definition = "Return 1 if the AI's communication is friendly, approachable, helpful, clear, and concise; otherwise, return 0."

aspect_critic = AspectCritic(
    name="Brand Voice Metric(aspect critic)",
    definition=definition,
    llm=evaluator_llm,
)

sample = MultiTurnSample(
    user_input=[
        HumanMessage(content="Hi! I want to apply for a credit card."),
        AIMessage(
            content="Sure! We've got a few great options. What kind of benefits are you interested in?"
        ),
        HumanMessage(content="Travel rewards, mostly."),
        AIMessage(
            content="Perfect! Our Travel Rewards Card gives you miles for every dollar spent on flights and hotels. Want to apply now?"
        ),
        HumanMessage(content="Yes, please."),
        AIMessage(
            content="Awesome! Here's the link to apply. It's quick and easy. Let me know if you need help!"
        ),
    ]
)

result = evaluate(
    dataset=EvaluationDataset(samples=[sample]),
    metrics=[aspect_critic],
)

result.to_pandas()

Evaluating:   100%|██████████| 1/1 [00:00<?, ?it/s]

---

Metric comparison

---

What to watch out for

Definition wording is load-bearing. The LLM judge reads your definition string literally. Vague phrasing like "responds helpfully" produces inconsistent results across model versions or temperatures because "helpful" is underspecified. Test definitions against at least three known-good and three known-bad samples before committing them to a pipeline.

Binary scores amplify edge cases. A conversation that is 95% compliant but contains one out-of-scope sentence scores 0, identical to a completely non-compliant conversation. If you need to differentiate severity, use multiple narrowly-scoped metrics rather than one broad one.

Context length grows with conversation length. Each evaluation call sends the entire conversation to the judge model. Conversations exceeding 30–40 turns can approach or exceed the context window of smaller models. Truncating from the beginning is rarely safe; consider chunking by topic shift or capping conversation length at data-collection time.

The judge model has its own biases. gpt-4o-mini has a mild tendency to score affirmatively when definitions are written in positive framing. Phrasing a definition as "return 0 if the agent fails to …" often produces more discriminating results than "return 1 if the agent is …". Always validate against a hand-labelled held-out set.

Costs are per-metric, not per-sample. Running five metrics across 200 conversations is 1,000 LLM calls. Cache evaluation results by conversation hash if you are re-running the same samples after a prompt change.

Terminology consistency matters. Ragas's source documentation notes that definitions should refer to the user as Human and the chatbot as AI, matching the internal field names of HumanMessage and AIMessage. Calling the agent "the system" or "the bot" can cause the judge to misattribute turns.

---

Where to go next

The AspectCritic binary approach pairs naturally with structured data extraction pipelines — if your agent queries a RAG system, measuring retrieval quality at the chunk level is covered in the row-level chunk retrieval guide. For teams running evaluation continuously inside agent pipelines, the async patterns for Bedrock AgentCore article describes how to integrate LLM-based evaluation calls into serverless pipelines without blocking the critical path. The full Ragas documentation at docs.ragas.io covers additional metrics including SimpleCriteriaScore (a continuous variant of AspectCritic), reference-based faithfulness metrics, and tooling for visualising LLM evaluation traces.