Build a Tool-Calling Agent with a Local Small Model and Ollama (2026)

Published Β· Updated β€’ guides
OllamaAI AgentsTool CallingSmall Language ModelsLocal LLMsPython

This guide is adapted from the examples in Ollama's official Python library, ollama/ollama-python (MIT licence): tools.py, multi-tool.py and structured-outputs.py. Code is reproduced as published; the explanations are ours.

An "agent" is less mysterious than it sounds: a model that can ask your code to run a function, see the result, and decide what to do next. Every major hosted model does this, and so do small open models running on your own machine, which is attractive when data cannot leave your network, when you want zero per-token cost, or when you need to work offline.

Small models are less forgiving, though. They pick the wrong tool more often, pass malformed arguments, and drift from the output format you asked for. Ollama's Python library handles the plumbing, and its own examples quietly include the defensive habits that make small models usable. This guide builds three things from them: a single-turn tool agent, a multi-tool agent that streams its reasoning, and structured JSON output you can validate.

Prerequisites

  • Ollama installed and running. Our complete guide to running LLMs locally covers installation and hardware.
  • The models the examples use: ollama pull llama3.1 and ollama pull qwen3.
  • The Python library: pip install ollama, plus pydantic for the structured-output example.

Step 1: Turn Python functions into tools

In Ollama's library a tool can simply be a Python function. Its signature and Google-style docstring become the schema the model sees:

from ollama import ChatResponse, chat


def add_two_numbers(a: int, b: int) -> int:
  """
  Add two numbers

  Args:
    a (int): The first number
    b (int): The second number

  Returns:
    int: The sum of the two numbers
  """

  # The cast is necessary as returned tool call arguments don't always conform exactly to schema
  # E.g. this would prevent "what is 30 + 12" to produce '3012' instead of 42
  return int(a) + int(b)

Read the comment twice. The type hints say int, but the model's arguments can still arrive as strings, and "30" + "12" is "3012". Treat every argument as untrusted input: cast it, validate it, or both. With a small model this is not optional.

You can also define a tool by hand as a JSON schema, which is useful when the function lives elsewhere or needs a description the docstring cannot express:

def subtract_two_numbers(a: int, b: int) -> int:
  """
  Subtract two numbers
  """

  # The cast is necessary as returned tool call arguments don't always conform exactly to schema
  return int(a) - int(b)


# Tools can still be manually defined and passed into chat
subtract_two_numbers_tool = {
  'type': 'function',
  'function': {
    'name': 'subtract_two_numbers',
    'description': 'Subtract two numbers',
    'parameters': {
      'type': 'object',
      'required': ['a', 'b'],
      'properties': {
        'a': {'type': 'integer', 'description': 'The first number'},
        'b': {'type': 'integer', 'description': 'The second number'},
      },
    },
  },
}

Step 2: Run the tool-call loop

The model never runs anything itself. It returns a request to call a tool; your code runs it and sends back the result. The whole cycle in Ollama's example:

messages = [{'role': 'user', 'content': 'What is three plus one?'}]
print('Prompt:', messages[0]['content'])

available_functions = {
  'add_two_numbers': add_two_numbers,
  'subtract_two_numbers': subtract_two_numbers,
}

response: ChatResponse = chat(
  'llama3.1',
  messages=messages,
  tools=[add_two_numbers, subtract_two_numbers_tool],
)

if response.message.tool_calls:
  # There may be multiple tool calls in the response
  # Add the assistant message with tool calls to the conversation
  messages.append(response.message)

  for tool in response.message.tool_calls:
    # Ensure the function is available, and then call it
    if function_to_call := available_functions.get(tool.function.name):
      print('Calling function:', tool.function.name)
      print('Arguments:', tool.function.arguments)
      output = function_to_call(**tool.function.arguments)
      print('Function output:', output)
    else:
      print('Function', tool.function.name, 'not found')
      output = 'Function not found'

    # Add each tool result as a separate message
    messages.append({'role': 'tool', 'content': str(output), 'tool_name': tool.function.name})

  # Get final response from model with all tool call results
  final_response = chat('llama3.1', messages=messages)
  print('Final response:', final_response.message.content)

else:
  print('No tool calls returned from model')

Four details make this robust:

  • available_functions is an allow-list. The model can only trigger functions you mapped by name. A hallucinated tool name gets "Function not found" instead of an exception or, worse, a call to something you did not intend.
  • The assistant message is appended before the results. The model needs to see that it asked for the tool, then the answer, in order.
  • Each result is its own role: 'tool' message, labelled with tool_name, so multiple calls in one turn stay distinguishable.
  • The second chat call has no tools. It turns results into an answer and cannot start another round.

That last point makes this a single-step agent. A multi-step agent repeats the call-run-append cycle, passing tools each time, until the model returns no tool calls. Cap the number of rounds when you build that: a small model can loop.

Step 3: Several tools, streaming, and visible thinking

multi-tool.py asks one question that needs two different tools, streams the response, enables the model's reasoning output with think=True on Qwen3, and then sends the results back. The complete file:

import random
from typing import Iterator

from ollama import ChatResponse, Client


def get_temperature(city: str) -> int:
  """
  Get the temperature for a city in Celsius

  Args:
    city (str): The name of the city

  Returns:
    int: The current temperature in Celsius
  """
  # This is a mock implementation - would need to use a real weather API
  import random

  if city not in ['London', 'Paris', 'New York', 'Tokyo', 'Sydney']:
    return 'Unknown city'

  return str(random.randint(0, 35)) + ' degrees Celsius'


def get_conditions(city: str) -> str:
  """
  Get the weather conditions for a city
  """
  if city not in ['London', 'Paris', 'New York', 'Tokyo', 'Sydney']:
    return 'Unknown city'
  # This is a mock implementation - would need to use a real weather API
  conditions = ['sunny', 'cloudy', 'rainy', 'snowy']
  return random.choice(conditions)


available_functions = {
  'get_temperature': get_temperature,
  'get_conditions': get_conditions,
}


cities = ['London', 'Paris', 'New York', 'Tokyo', 'Sydney']
city = random.choice(cities)
city2 = random.choice(cities)
messages = [{'role': 'user', 'content': f'What is the temperature in {city}? and what are the weather conditions in {city2}?'}]
print('----- Prompt:', messages[0]['content'], '\n')

model = 'qwen3'
client = Client()
response: Iterator[ChatResponse] = client.chat(model, stream=True, messages=messages, tools=[get_temperature, get_conditions], think=True)

for chunk in response:
  if chunk.message.thinking:
    print(chunk.message.thinking, end='', flush=True)
  if chunk.message.content:
    print(chunk.message.content, end='', flush=True)
  if chunk.message.tool_calls:
    for tool in chunk.message.tool_calls:
      if function_to_call := available_functions.get(tool.function.name):
        print('\nCalling function:', tool.function.name, 'with arguments:', tool.function.arguments)
        output = function_to_call(**tool.function.arguments)
        print('> Function output:', output, '\n')

        # Add the assistant message and tool call result to the messages
        messages.append(chunk.message)
        messages.append({'role': 'tool', 'content': str(output), 'tool_name': tool.function.name})
      else:
        print('Function', tool.function.name, 'not found')

print('----- Sending result back to model \n')
if any(msg.get('role') == 'tool' for msg in messages):
  res = client.chat(model, stream=True, tools=[get_temperature, get_conditions], messages=messages, think=True)
  done_thinking = False
  for chunk in res:
    if chunk.message.thinking:
      print(chunk.message.thinking, end='', flush=True)
    if chunk.message.content:
      if not done_thinking:
        print('\n----- Final result:')
        done_thinking = True
      print(chunk.message.content, end='', flush=True)
    if chunk.message.tool_calls:
      # Model should be explaining the tool calls and the results in this output
      print('Model returned tool calls:')
      print(chunk.message.tool_calls)
else:
  print('No tool calls returned')

When streaming, tool calls arrive inside chunks, so the loop checks every chunk for thinking, content and tool_calls. The thinking stream is the most useful debugging aid you have with a small model: when it picks the wrong tool, the reasoning usually shows why, often a vague docstring.

The example's tools are mocks that return random weather for five cities. One small inconsistency: get_temperature is annotated -> int but returns strings such as 'Unknown city'. It works because the result is converted with str() before being sent back, but match your annotations to reality in your own tools, because the model reads them.

The second request (after "Sending result back to model") passes tools again, and the example prints any further tool calls instead of running them. This is where you would continue the loop.

Step 4: Get JSON you can trust

Many agent steps do not need tools, just output your code can parse. Ollama constrains the reply to a JSON schema through the format argument:

from pydantic import BaseModel

from ollama import chat


# Define the schema for the response
class FriendInfo(BaseModel):
  name: str
  age: int
  is_available: bool


class FriendList(BaseModel):
  friends: list[FriendInfo]


# schema = {'type': 'object', 'properties': {'friends': {'type': 'array', 'items': {'type': 'object', 'properties': {'name': {'type': 'string'}, 'age': {'type': 'integer'}, 'is_available': {'type': 'boolean'}}, 'required': ['name', 'age', 'is_available']}}}, 'required': ['friends']}
response = chat(
  model='llama3.1:8b',
  messages=[{'role': 'user', 'content': 'I have two friends. The first is Ollama 22 years old busy saving the world, and the second is Alonso 23 years old and wants to hang out. Return a list of friends in JSON format'}],
  format=FriendList.model_json_schema(),  # Use Pydantic to generate the schema or format=schema
  options={'temperature': 0},  # Make responses more deterministic
)

# Use Pydantic to validate the response
friends_response = FriendList.model_validate_json(response.message.content)
print(friends_response)

The same Pydantic model does two jobs: it generates the schema that constrains the reply, and it validates the reply afterwards. Keep both. The schema makes valid output likely, and validation turns the remaining failures into an exception you can retry, instead of bad data flowing into the next step. temperature: 0 makes the output repeatable, which is what you want for extraction and routing.

Choosing a model and what to watch for

Risk with small modelsWhat the examples doWhat to add
Arguments of the wrong typeCast with int() inside the toolValidate ranges and formats; return a clear error string the model can correct from
Calling a tool that does not existavailable_functions allow-listLog unknown names. A recurring one means your tool descriptions are unclear
Endless tool loopsSingle round, or print instead of re-runningCap rounds per task and stop with a message when the cap is hit
Malformed structured outputformat= schema plus Pydantic validationRetry once on validation failure, then fail loudly
Wrong tool chosenthink=True to show reasoningSharpen docstrings; offer fewer tools per step

Tool-calling quality differs far more between small models than general chat quality does. Our comparison of the top small language models for agentic tool calling is the place to start, but the deciding test is your own: run your real tools and prompts against two or three candidates and count wrong calls.

Mind the cost of loops on local hardware. An agent calls the model at least twice per tool round. On a CPU an 8B model can take seconds per call, so a five-round task feels slow. A GPU or Apple Silicon makes the difference between a demo and a usable tool.

Where to go next

When your agent needs memory, retries or branching, a framework helps more than a longer loop: our LangChain getting-started guide builds a retrieval agent with create_agent, and the self-correcting RAG agent with LangGraph and Milvus runs entirely on local models. To force a local model's output to a grammar at the inference-engine level, see guaranteed JSON from llama.cpp with GBNF grammars.

Frequently asked questions

How do I give a local model tools with Ollama in Python?

Pass plain Python functions in the tools list of ollama.chat. The library builds each tool's schema from the function's signature and its Google-style docstring. When the response contains tool_calls, run the matching function, append the assistant message and a role 'tool' message with the result, and call chat again for the final answer.

Why does Ollama's tool example cast arguments with int()?

Models do not always return arguments that match the schema. Ollama's own example notes that without the cast, 'what is 30 + 12' could produce '3012' instead of 42, because the numbers arrive as strings. Validate or cast every argument before using it, especially with small models.

Which small local models support tool calling in Ollama?

Ollama's examples use llama3.1 (the 8B model) and qwen3; any model tagged with tool support in Ollama's library can be used. Tool-calling reliability varies a lot between small models, so test your own tools before choosing one.

How do I get reliable JSON from a local model with Ollama?

Pass a JSON schema as the format argument, for example a Pydantic model's model_json_schema(), and set temperature to 0. Then validate the reply with the same Pydantic model's model_validate_json, so malformed output raises an error instead of flowing into your agent.

Do I need a GPU to run a tool-calling agent locally?

Not strictly. An 8B model runs on a recent CPU with enough RAM, but slowly; a GPU or Apple Silicon makes agent loops, which call the model several times per task, far more usable. Smaller models trade speed for weaker tool use.

Free interactive tools for the decisions this piece raises.

Related Guides