Self-Consistency Voting with Outlines and gpt-4o-mini

Self-consistency is a sampling strategy that treats a language model's stochastic output as a distribution to reason over rather than a single answer to trust. Instead of asking the model once and hoping for the best, you generate a large batch of independent reasoning chains and then vote on the most common final answer. The intuition is solid: wrong reasoning paths tend to diverge in unpredictable ways, while correct reasoning paths converge on the same numerical or categorical conclusion. For multi-step arithmetic and symbolic reasoning — the class of tasks where chain-of-thought prompting already helps — self-consistency improves accuracy over greedy decoding, often closing the gap between smaller and larger models without any fine-tuning.

This guide is adapted from Outlines' self_consistency.py, available under the Apache-2.0 licence. The technique pairs naturally with Outlines because constrained sampling ensures each sampled response follows a predictable structure, making the final answer extraction step reliable rather than fragile. Engineers building evaluation pipelines, tutoring systems, or any agent that must solve word problems or logic puzzles will find this immediately applicable.

The cost profile is worth understanding before you commit. You are multiplying your token consumption by whatever sample count you choose — ten samples means ten times the prompt tokens plus ten times the completion tokens. On larger or self-hosted models the inference cost becomes material quickly. If you are running open-weight models and thinking about KV-cache efficiency across those parallel samples, the architectural tradeoffs discussed in our coverage of PagedAttention and RadixAttention are directly relevant. Latency also multiplies unless you parallelise: the source code requests all samples in a single API call using n=10, so wall-clock latency is closer to one request than ten, but your serving infrastructure must be able to handle the load.

Prerequisites

• Python 3.10 or later
• outlines installed (pip install outlines)
• openai Python SDK installed (pip install openai)
• numpy installed (pip install numpy)
• An OpenAI API key set in your environment (OPENAI_API_KEY)
• A prompt template file at prompts/self_consistency.txt — this is a Jinja-style template that Outlines' Template system will render; it expects question and examples variables

The source uses gpt-4o-mini as the backend. Any OpenAI-compatible endpoint works, but the n parameter (requesting multiple completions in one call) must be supported by your provider.

Step 1: Import dependencies and define your few-shot examples

The few-shot examples are the backbone of chain-of-thought self-consistency. Each example shows the model not just the answer but the full reasoning path that leads to it. This is what gives the voting mechanism its leverage: models that have been shown deliberate, step-by-step reasoning are more likely to produce structured completions where the final number is unambiguous.

import re

import numpy as np
import openai

import outlines
from outlines import Template

examples = [
    {
        "question": "There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?",
        "answer": "We start with 15 trees. Later we have 21 trees. The difference must be the number of trees they planted. So, they must have planted 21 - 15 = 6 trees. The answer is 6.",
    },
    {
        "question": "If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?",
        "answer": "There are 3 cars in the parking lot already. 2 more arrive. Now there are 3 + 2 = 5 cars. The answer is 5.",
    },
    {
        "question": "Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?",
        "answer": "Leah had 32 chocolates and Leah's sister had 42. That means there were originally 32 + 42 = 74 chocolates. 35 have been eaten. So in total they still have 74 - 35 = 39 chocolates. The answer is 39.",
    },
    {
        "question": "Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?",
        "answer": "Jason had 20 lollipops. Since he only has 12 now, he must have given the rest to Denny. The number of lollipops he has given to Denny must have been 20 - 12 = 8 lollipops. The answer is 8.",
    },
    {
        "question": "Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?",
        "answer": "He has 5 toys. He got 2 from mom, so after that he has 5 + 2 = 7 toys. Then he got 2 more from dad, so in total he has 7 + 2 = 9 toys. The answer is 9.",
    },
    {
        "question": "There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. How many computers are now in the server room?",
        "answer": "There are 4 days from monday to thursday. 5 computers were added each day. That means in total 4 * 5 = 20 computers were added. There were 9 computers in the beginning, so now there are 9 + 20 = 29 computers. The answer is 29.",
    },
    {
        "question": "Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more. How many golf balls did he have at the end of wednesday?",
        "answer": "Michael initially had 58 balls. He lost 23 on Tuesday, so after that he has 58 - 23 = 35 balls. On Wednesday he lost 2 more so now he has 35 - 2 = 33 balls. The answer is 33.",
    },
    {
        "question": "Olivia has $23. She bought five bagels for $3 each. How much money does she have left?",
        "answer": "She bought 5 bagels for $3 each. This means she spent 5",
    },
]

question = "When I was 6 my sister was half my age. Now I'm 70 how old is my sister?"

Notice that the final example is intentionally incomplete — it cuts off mid-sentence to let the model continue the reasoning pattern naturally rather than copy a complete answer. The target question is a classic misdirection problem: the naive answer is 35, but the correct answer is 67, because the age gap is fixed at three years regardless of the narrator's current age.

Step 2: Load the prompt template and initialise the model

Outlines' Template system handles prompt construction from a file, keeping your Python logic clean and your prompt text version-controllable separately. The generator is intentionally unconstrained here — you want free-form reasoning chains, not a single token or a structured schema. The structure comes from your examples, not from regex or grammar constraints.

few_shots = Template.from_file("prompts/self_consistency.txt")

model = outlines.from_openai(openai.OpenAI(), "gpt-4o-mini")
generator = outlines.Generator(model)
prompt = few_shots(question=question, examples=examples)
answers = generator(prompt, n=10)

The n=10 argument is the key lever. It requests ten independent completions from the API in a single round-trip. All ten see the same prompt and the same few-shot examples, but because sampling temperature is non-zero they will diverge in their reasoning paths. Treat n as a hyperparameter: too low (two or three) and the vote can easily be a coin flip; too high (thirty or more) and you are paying a large token cost for diminishing accuracy returns. Ten is a sensible default for most arithmetic benchmarks.

Step 3: Extract the final numerical answer from each completion

Each of the ten completions is a free-text reasoning chain that ends, following the few-shot pattern, with a sentence like "The answer is 67." The extraction logic uses a simple regex to grab the last digit sequence in each string, which reliably captures the final stated answer without needing the model to output a constrained token.

digits = []
for answer in answers:
    try:
        match = re.findall(r"\d+", answer)[-1]
        if match is not None:
            digit = int(match)
            digits.append(digit)
    except AttributeError:
        print(f"Could not parse the completion: '{answer}'")

The [-1] index is deliberate: you want the last number in the string, not the first. A reasoning chain will naturally contain intermediate numbers (ages, differences, partial sums), and you only want the conclusion. Any completion that cannot be parsed is logged and silently dropped from the vote rather than crashing the pipeline.

Step 4: Vote and report the consensus answer

With a list of extracted integers, standard majority voting is a one-liner using NumPy. The source then formats a human-readable confidence report alongside the winning answer.

unique_digits, counts = np.unique(digits, return_counts=True)
results = {int(d): int(c) for d, c in zip(unique_digits, counts)}
print(results)

max_count = max(results.values())
answer_value = [key for key, value in results.items() if value == max_count][0]
total_count = sum(results.values())
print(
    f"The most likely answer is {answer_value} ({max_count / total_count * 100}% consensus)"
)

A healthy run on the sister-age problem will produce output like {67: 8, 35: 2} followed by The most likely answer is 67 (80.0% consensus). The consensus percentage is directly useful as a confidence proxy: if you see a 50/50 split across ten samples, that is a strong signal to either increase n, switch to a more capable model, or flag the question for human review rather than accepting the "winner."

Choosing your sample count

What to watch out for

The regex is brittle for non-integer answers. The re.findall(r"\d+", answer)[-1] pattern only captures whole numbers. If your domain involves decimals, currency with cents, or percentages, the extractor will silently truncate or misparse the answer. You need a more sophisticated pattern and corresponding test cases before moving beyond integer arithmetic.

Voting does not fix systematic prompt errors. If the few-shot examples anchor the model toward a particular wrong reasoning strategy, all ten samples will converge on the same wrong answer with high confidence. A 90% consensus score feels reassuring but tells you nothing about whether the shared reasoning path was correct. Calibrate the system on a held-out labelled benchmark before trusting the confidence metric operationally.

Temperature matters more than you might expect. The source does not explicitly set temperature, relying on the API default. At very low temperatures (approaching greedy), all ten samples will be nearly identical and the vote adds no information. At very high temperatures, samples become incoherent and parsing failure rates climb. A temperature between 0.5 and 0.8 tends to produce the useful diversity that makes voting meaningful.

Ties need a defined resolution strategy. The source code's [0] index on the list comprehension silently picks the numerically smallest tied answer. For a production system this is an invisible assumption that can produce wrong outputs without any warning. Either break ties explicitly (for example, by requesting additional samples) or surface them as low-confidence cases for human review.

Provider n support is not universal. The n parameter is an OpenAI API convention. If you switch to a self-hosted model behind a vLLM or similar server, check that the endpoint implements n correctly — some implementations run n calls sequentially rather than batching them, which eliminates the latency advantage entirely.

Dropped parses silently reduce your vote pool. If three of ten completions fail regex extraction, you are effectively voting on seven samples, but total_count in the percentage calculation reflects only successful parses. A 70% consensus printed to the user may actually represent seven out of seven parsed completions — a much weaker signal than seven out of ten.

Where to go next

Once you have self-consistency working for numerical answers, the natural extension is to apply structured output constraints within each sample rather than relying on regex post-processing — Outlines' grammar-constrained generation can enforce that every completion terminates with a parseable answer token, eliminating the extraction brittleness entirely. For teams thinking about how this fits into larger reasoning pipelines, the architecture patterns in our async inference pipeline coverage are worth reviewing: parallelising across multiple questions (rather than just multiple samples of one question) is where throughput gains become significant at scale. If your workload involves retrieval-augmented contexts being fed into these reasoning chains, the cost dynamics discussed in our RAG compression guide compound with the per-sample multiplication described here — worth modelling before you commit to a production architecture.