Build a Tool-Calling Agent with a Local Small Model and Ollama (2026)
In this article
This guide is adapted from the examples in Ollama's official Python library, ollama/ollama-python (MIT licence): tools.py, multi-tool.py and structured-outputs.py. Code is reproduced as published; the explanations are ours.
An "agent" is less mysterious than it sounds: a model that can ask your code to run a function, see the result, and decide what to do next. Every major hosted model does this, and so do small open models running on your own machine, which is attractive when data cannot leave your network, when you want zero per-token cost, or when you need to work offline.
Small models are less forgiving, though. They pick the wrong tool more often, pass malformed arguments, and drift from the output format you asked for. Ollama's Python library handles the plumbing, and its own examples quietly include the defensive habits that make small models usable. This guide builds three things from them: a single-turn tool agent, a multi-tool agent that streams its reasoning, and structured JSON output you can validate.
Prerequisites
- Ollama installed and running. Our complete guide to running LLMs locally covers installation and hardware.
- The models the examples use:
ollama pull llama3.1andollama pull qwen3. - The Python library:
pip install ollama, pluspydanticfor the structured-output example.
Step 1: Turn Python functions into tools
In Ollama's library a tool can simply be a Python function. Its signature and Google-style docstring become the schema the model sees:
from ollama import ChatResponse, chat
def add_two_numbers(a: int, b: int) -> int:
"""
Add two numbers
Args:
a (int): The first number
b (int): The second number
Returns:
int: The sum of the two numbers
"""
# The cast is necessary as returned tool call arguments don't always conform exactly to schema
# E.g. this would prevent "what is 30 + 12" to produce '3012' instead of 42
return int(a) + int(b)
Read the comment twice. The type hints say int, but the model's arguments can still arrive as strings, and "30" + "12" is "3012". Treat every argument as untrusted input: cast it, validate it, or both. With a small model this is not optional.
You can also define a tool by hand as a JSON schema, which is useful when the function lives elsewhere or needs a description the docstring cannot express:
def subtract_two_numbers(a: int, b: int) -> int:
"""
Subtract two numbers
"""
# The cast is necessary as returned tool call arguments don't always conform exactly to schema
return int(a) - int(b)
# Tools can still be manually defined and passed into chat
subtract_two_numbers_tool = {
'type': 'function',
'function': {
'name': 'subtract_two_numbers',
'description': 'Subtract two numbers',
'parameters': {
'type': 'object',
'required': ['a', 'b'],
'properties': {
'a': {'type': 'integer', 'description': 'The first number'},
'b': {'type': 'integer', 'description': 'The second number'},
},
},
},
}
Step 2: Run the tool-call loop
The model never runs anything itself. It returns a request to call a tool; your code runs it and sends back the result. The whole cycle in Ollama's example:
messages = [{'role': 'user', 'content': 'What is three plus one?'}]
print('Prompt:', messages[0]['content'])
available_functions = {
'add_two_numbers': add_two_numbers,
'subtract_two_numbers': subtract_two_numbers,
}
response: ChatResponse = chat(
'llama3.1',
messages=messages,
tools=[add_two_numbers, subtract_two_numbers_tool],
)
if response.message.tool_calls:
# There may be multiple tool calls in the response
# Add the assistant message with tool calls to the conversation
messages.append(response.message)
for tool in response.message.tool_calls:
# Ensure the function is available, and then call it
if function_to_call := available_functions.get(tool.function.name):
print('Calling function:', tool.function.name)
print('Arguments:', tool.function.arguments)
output = function_to_call(**tool.function.arguments)
print('Function output:', output)
else:
print('Function', tool.function.name, 'not found')
output = 'Function not found'
# Add each tool result as a separate message
messages.append({'role': 'tool', 'content': str(output), 'tool_name': tool.function.name})
# Get final response from model with all tool call results
final_response = chat('llama3.1', messages=messages)
print('Final response:', final_response.message.content)
else:
print('No tool calls returned from model')
Four details make this robust:
available_functionsis an allow-list. The model can only trigger functions you mapped by name. A hallucinated tool name gets "Function not found" instead of an exception or, worse, a call to something you did not intend.- The assistant message is appended before the results. The model needs to see that it asked for the tool, then the answer, in order.
- Each result is its own
role: 'tool'message, labelled withtool_name, so multiple calls in one turn stay distinguishable. - The second
chatcall has notools. It turns results into an answer and cannot start another round.
That last point makes this a single-step agent. A multi-step agent repeats the call-run-append cycle, passing tools each time, until the model returns no tool calls. Cap the number of rounds when you build that: a small model can loop.
Step 3: Several tools, streaming, and visible thinking
multi-tool.py asks one question that needs two different tools, streams the response, enables the model's reasoning output with think=True on Qwen3, and then sends the results back. The complete file:
import random
from typing import Iterator
from ollama import ChatResponse, Client
def get_temperature(city: str) -> int:
"""
Get the temperature for a city in Celsius
Args:
city (str): The name of the city
Returns:
int: The current temperature in Celsius
"""
# This is a mock implementation - would need to use a real weather API
import random
if city not in ['London', 'Paris', 'New York', 'Tokyo', 'Sydney']:
return 'Unknown city'
return str(random.randint(0, 35)) + ' degrees Celsius'
def get_conditions(city: str) -> str:
"""
Get the weather conditions for a city
"""
if city not in ['London', 'Paris', 'New York', 'Tokyo', 'Sydney']:
return 'Unknown city'
# This is a mock implementation - would need to use a real weather API
conditions = ['sunny', 'cloudy', 'rainy', 'snowy']
return random.choice(conditions)
available_functions = {
'get_temperature': get_temperature,
'get_conditions': get_conditions,
}
cities = ['London', 'Paris', 'New York', 'Tokyo', 'Sydney']
city = random.choice(cities)
city2 = random.choice(cities)
messages = [{'role': 'user', 'content': f'What is the temperature in {city}? and what are the weather conditions in {city2}?'}]
print('----- Prompt:', messages[0]['content'], '\n')
model = 'qwen3'
client = Client()
response: Iterator[ChatResponse] = client.chat(model, stream=True, messages=messages, tools=[get_temperature, get_conditions], think=True)
for chunk in response:
if chunk.message.thinking:
print(chunk.message.thinking, end='', flush=True)
if chunk.message.content:
print(chunk.message.content, end='', flush=True)
if chunk.message.tool_calls:
for tool in chunk.message.tool_calls:
if function_to_call := available_functions.get(tool.function.name):
print('\nCalling function:', tool.function.name, 'with arguments:', tool.function.arguments)
output = function_to_call(**tool.function.arguments)
print('> Function output:', output, '\n')
# Add the assistant message and tool call result to the messages
messages.append(chunk.message)
messages.append({'role': 'tool', 'content': str(output), 'tool_name': tool.function.name})
else:
print('Function', tool.function.name, 'not found')
print('----- Sending result back to model \n')
if any(msg.get('role') == 'tool' for msg in messages):
res = client.chat(model, stream=True, tools=[get_temperature, get_conditions], messages=messages, think=True)
done_thinking = False
for chunk in res:
if chunk.message.thinking:
print(chunk.message.thinking, end='', flush=True)
if chunk.message.content:
if not done_thinking:
print('\n----- Final result:')
done_thinking = True
print(chunk.message.content, end='', flush=True)
if chunk.message.tool_calls:
# Model should be explaining the tool calls and the results in this output
print('Model returned tool calls:')
print(chunk.message.tool_calls)
else:
print('No tool calls returned')
When streaming, tool calls arrive inside chunks, so the loop checks every chunk for thinking, content and tool_calls. The thinking stream is the most useful debugging aid you have with a small model: when it picks the wrong tool, the reasoning usually shows why, often a vague docstring.
The example's tools are mocks that return random weather for five cities. One small inconsistency: get_temperature is annotated -> int but returns strings such as 'Unknown city'. It works because the result is converted with str() before being sent back, but match your annotations to reality in your own tools, because the model reads them.
The second request (after "Sending result back to model") passes tools again, and the example prints any further tool calls instead of running them. This is where you would continue the loop.
Step 4: Get JSON you can trust
Many agent steps do not need tools, just output your code can parse. Ollama constrains the reply to a JSON schema through the format argument:
from pydantic import BaseModel
from ollama import chat
# Define the schema for the response
class FriendInfo(BaseModel):
name: str
age: int
is_available: bool
class FriendList(BaseModel):
friends: list[FriendInfo]
# schema = {'type': 'object', 'properties': {'friends': {'type': 'array', 'items': {'type': 'object', 'properties': {'name': {'type': 'string'}, 'age': {'type': 'integer'}, 'is_available': {'type': 'boolean'}}, 'required': ['name', 'age', 'is_available']}}}, 'required': ['friends']}
response = chat(
model='llama3.1:8b',
messages=[{'role': 'user', 'content': 'I have two friends. The first is Ollama 22 years old busy saving the world, and the second is Alonso 23 years old and wants to hang out. Return a list of friends in JSON format'}],
format=FriendList.model_json_schema(), # Use Pydantic to generate the schema or format=schema
options={'temperature': 0}, # Make responses more deterministic
)
# Use Pydantic to validate the response
friends_response = FriendList.model_validate_json(response.message.content)
print(friends_response)
The same Pydantic model does two jobs: it generates the schema that constrains the reply, and it validates the reply afterwards. Keep both. The schema makes valid output likely, and validation turns the remaining failures into an exception you can retry, instead of bad data flowing into the next step. temperature: 0 makes the output repeatable, which is what you want for extraction and routing.
Choosing a model and what to watch for
| Risk with small models | What the examples do | What to add |
|---|---|---|
| Arguments of the wrong type | Cast with int() inside the tool | Validate ranges and formats; return a clear error string the model can correct from |
| Calling a tool that does not exist | available_functions allow-list | Log unknown names. A recurring one means your tool descriptions are unclear |
| Endless tool loops | Single round, or print instead of re-running | Cap rounds per task and stop with a message when the cap is hit |
| Malformed structured output | format= schema plus Pydantic validation | Retry once on validation failure, then fail loudly |
| Wrong tool chosen | think=True to show reasoning | Sharpen docstrings; offer fewer tools per step |
Tool-calling quality differs far more between small models than general chat quality does. Our comparison of the top small language models for agentic tool calling is the place to start, but the deciding test is your own: run your real tools and prompts against two or three candidates and count wrong calls.
Mind the cost of loops on local hardware. An agent calls the model at least twice per tool round. On a CPU an 8B model can take seconds per call, so a five-round task feels slow. A GPU or Apple Silicon makes the difference between a demo and a usable tool.
Where to go next
When your agent needs memory, retries or branching, a framework helps more than a longer loop: our LangChain getting-started guide builds a retrieval agent with create_agent, and the self-correcting RAG agent with LangGraph and Milvus runs entirely on local models. To force a local model's output to a grammar at the inference-engine level, see guaranteed JSON from llama.cpp with GBNF grammars.
Frequently asked questions
How do I give a local model tools with Ollama in Python?
Pass plain Python functions in the tools list of ollama.chat. The library builds each tool's schema from the function's signature and its Google-style docstring. When the response contains tool_calls, run the matching function, append the assistant message and a role 'tool' message with the result, and call chat again for the final answer.
Why does Ollama's tool example cast arguments with int()?
Models do not always return arguments that match the schema. Ollama's own example notes that without the cast, 'what is 30 + 12' could produce '3012' instead of 42, because the numbers arrive as strings. Validate or cast every argument before using it, especially with small models.
Which small local models support tool calling in Ollama?
Ollama's examples use llama3.1 (the 8B model) and qwen3; any model tagged with tool support in Ollama's library can be used. Tool-calling reliability varies a lot between small models, so test your own tools before choosing one.
How do I get reliable JSON from a local model with Ollama?
Pass a JSON schema as the format argument, for example a Pydantic model's model_json_schema(), and set temperature to 0. Then validate the reply with the same Pydantic model's model_validate_json, so malformed output raises an error instead of flowing into your agent.
Do I need a GPU to run a tool-calling agent locally?
Not strictly. An 8B model runs on a recent CPU with enough RAM, but slowly; a GPU or Apple Silicon makes agent loops, which call the model several times per task, far more usable. Smaller models trade speed for weaker tool use.
Related Guides

Structured Output with Local LLMs: When Valid JSON Is Not Enough
Gemma 4's 4B model returns schema-valid JSON that still includes the wrong device. Here's the decomposition pattern that fixes it.
How to Run Qwen3 Locally with Ollama: Setup, API, and a Gradio App
Set up Qwen3 locally in minutes using Ollama. Covers every model variant, thinking mode control with /think and /no_think tags, CLI, REST API, Python SDK, and a practical Gradio reasoning app.
Build a DSPy ReAct Agent That Searches and Reads Wikipedia
Reproduce DSPy's ReAct-with-tools example, define Wikipedia search and page tools, cap tool calls, and inspect the agent's trajectory.