Fine-Tune Tiny Aya for Tool Calling with QLoRA and TRL

Published • guides
Tool CallingHugging Face

Engineers who run agents quickly learn that a model that can call a tool in a demo is not the same as a model that calls it reliably. Fine-tuning a compact model on a small corpus of tool calls can change the failure from malformed JSON, invented parameter names, or ignored schemas into a parsing problem with a predictable boundary. This guide adapts Hugging Face TRL's examples/sft_tool_calling/sft_tiny_aya_tool_calling.py from the TRL repository, released under Apache-2.0. The code blocks below reproduce the relevant sections of that script unchanged and teach tool calling to CohereLabs/tiny-aya-global using supervised fine-tuning with QLoRA on bebechien/SimpleToolCalling.

The source model does not have native tool-calling support. The script extends its Jinja2 chat template so that tool schemas are serialized into the system preamble, and assistant calls are rendered as structured <tool_call> XML inside <|START_RESPONSE|> / <|END_RESPONSE|> delimiters. The modified template is saved with the tokenizer, so inference can load the tokenizer from the output directory and call apply_chat_template(tools=TOOLS) instead of reconstructing prompts by hand. This is the same failure class that larger routing systems face, for example the tool dispatch problem discussed in our coverage of DeepSeek's DSH superagent router.

What the source script does and does not specify

The TRL example does not publish exact GPU requirements, measured training times, or cloud costs. Its memory choices are explicit: 4-bit quantization, LoRA rank 32, batch size 1 with gradient accumulation over 4 steps, and activation offloading. Use those settings to estimate whether your environment has enough VRAM; do not treat a specific GPU model or hourly rental figure as part of the source.

Prerequisites

You need a CUDA-capable GPU. The script's inline dependency metadata lists trl[peft], bitsandbytes, kernels, and trackio. Install them with:

pip install "trl[peft]" bitsandbytes kernels trackio

kernels is used only when use_kernels=True is passed to model loading. trackio is the logging backend; if you do not want that integration, remove the reporting arguments or set report_to="none". You also need tiny_aya_chat_template.jinja in the same directory as the script, or adjust chat_template_path. If you intend to push the adapter to the Hub, run huggingface-cli login first.

Step 1: Define the tool schemas and conversation mapping

The script begins by importing the training stack and declaring the two functions in the dataset: search_knowledge_base and search_google. The schemas are not comments; they are embedded in every prompt so the model sees the exact parameter names, types, required fields, and return type it is allowed to emit. create_conversation maps each raw sample to a user prompt, an assistant tool_calls completion, and the shared TOOLS list.

import json
from pathlib import Path

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

from trl import SFTConfig, SFTTrainer


# These are the tool schemas that are used in the dataset
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "search_knowledge_base",
            "description": "Search internal company documents, policies and project data.",
            "parameters": {
                "type": "object",
                "properties": {"query": {"type": "string", "description": "query string"}},
                "required": ["query"],
            },
            "return": {"type": "string"},
        },
    },
    {
        "type": "function",
        "function": {
            "name": "search_google",
            "description": "Search public information.",
            "parameters": {
                "type": "object",
                "properties": {"query": {"type": "string", "description": "query string"}},
                "required": ["query"],
            },
            "return": {"type": "string"},
        },
    },
]


def create_conversation(sample):
    return {
        "prompt": [{"role": "user", "content": sample["user_content"]}],
        "completion": [
            {
                "role": "assistant",
                "tool_calls": [
                    {
                        "type": "function",
                        "function": {
                            "name": sample["tool_name"],
                            "arguments": json.loads(sample["tool_arguments"]),
                        },
                    }
                ],
            },
        ],
        "tools": TOOLS,
    }

The arguments field is parsed with json.loads(sample["tool_arguments"]). That tells you the dataset stores arguments as JSON text, while the trainer expects structured objects. If you replace the dataset, this function is the first place to adjust.

Step 2: Load and split the tool-calling dataset

main sets the model, dataset, and output identifiers and loads only the train split. The map call applies create_conversation and removes the original columns. The script then creates a 50/50 train/test split. The test half is useful for manual evaluation, but the script does not run an evaluation loop on it.

def main():
    model_id = "CohereLabs/tiny-aya-global"
    dataset_name = "bebechien/SimpleToolCalling"
    output_dir = "tiny-aya-global-tool-calling-SFT"

    # Load and format dataset
    dataset = load_dataset(dataset_name, split="train")
    dataset = dataset.map(create_conversation, remove_columns=dataset.features)
    dataset = dataset.train_test_split(test_size=0.5, shuffle=True)

Step 3: Load the base model with 4-bit QLoRA

The base model is loaded in 4-bit NF4. Frozen base weights stay quantized, while LoRA adapters are trained in higher precision. The loader also enables SDPA attention, FP16 compute in quantized matrix multiplications, double quantization, and use_kernels=True.

    # Load model
    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        attn_implementation="sdpa",
        dtype=torch.float16,
        use_kernels=True,
        quantization_config=BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_compute_dtype=torch.float16,
            bnb_4bit_use_double_quant=True,
            bnb_4bit_quant_type="nf4",
        ),
    )

The following matrix summarizes the quantization choices.

ArgumentValue in scriptWhat it doesTradeoff if changed
load_in_4bitTrueStores base weights in 4-bit to reduce VRAMSetting False can multiply memory use and cause OOM
bnb_4bit_compute_dtypetorch.float16Dequantizes to FP16 for matrix multiplicationsUsing FP32 is slower and uses more memory; lower precision may hurt convergence
bnb_4bit_use_double_quantTrueApplies a second quantization pass to the scaling factorsSaves memory with negligible quality loss; disabling it needs more VRAM
bnb_4bit_quant_type"nf4"Uses NormalFloat4, a data-aware 4-bit representationThe older FP4 type usually performs worse for language models

Step 4: Configure LoRA and SFT training

LoRA targets query, key, value, output, gate, up, and down projections with rank 32 and alpha 32. The training arguments use batch size 1 and four gradient accumulation steps, a 2e-4 learning rate, paged 8-bit AdamW, a 1024 token maximum length, and activation offloading. The most important item is chat_template_path.

    # Configure LoRA
    peft_config = LoraConfig(
        r=32,
        lora_alpha=32,
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    )

    # Train
    training_args = SFTConfig(
        output_dir=output_dir,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=4,
        # Use the tool-aware chat template
        chat_template_path=str(Path(__file__).parent / "tiny_aya_chat_template.jinja"),
        warmup_steps=5,
        learning_rate=2e-4,
        optim="paged_adamw_8bit",
        logging_steps=1,
        report_to="trackio",
        trackio_space_id=output_dir,
        max_length=1024,
        activation_offloading=True,
        push_to_hub=True,
    )

The Jinja template must exist because it is the entire mechanism that makes tool schemas visible during training. If the file is missing, training will fail early; if the path is wrong, the model sees conversational prompts but no structured tool definitions, so it will not learn reliable function calls.

Step 5: Train, save, and validate structured tool use

The trainer is created with the train split, training arguments, LoRA config, and model. Training runs, then the adapter and tokenizer are saved locally and pushed to the Hub. The tokenizer is saved with the updated template, so validation does not require manual prompt construction.

    trainer = SFTTrainer(
        model=model,
        args=training_args,
        train_dataset=dataset["train"],
        peft_config=peft_config,
    )
    trainer.train()

    # Save model and tokenizer (tokenizer carries the updated chat template)
    trainer.save_model(output_dir)
    trainer.push_to_hub(dataset_name=dataset_name)

    trainer.end()


if __name__ == "__main__":
    main()

After the run, load the tokenizer from output_dir and call apply_chat_template with tools=TOOLS. Inspect the assistant output for a well-formed <tool_call> block with valid JSON. Because the script does not wire an evaluation loop, that manual check is the actual validation step. You can parse the block, execute the function, and append the result as the next assistant message.

What to watch out for

The 50% test split reserves half the available data and does nothing during training. If your goal is maximum tool-call accuracy on a very small dataset, reconsider that split or add an explicit evaluation loop.

use_kernels=True requires the kernels package and a supported CUDA environment. On unsupported GPUs or older drivers, model loading can raise an import or runtime error even though the rest of the stack is installed. If that happens, test without the flag, but the unmodified script enables it.

max_length=1024 truncates examples that exceed that token count. For a small model and two simple tools, that is usually fine, but check your dataset's tokenized lengths if you swap to a different schema set or longer tool descriptions. Truncation will silently cut off part of the completion and teach the model the wrong boundary.

Activation offloading reduces peak VRAM at the cost of slower iteration. If you have headroom, it may be worth testing without it; if you are near the VRAM limit, leave it on and expect longer wall-clock time. The same tradeoff applies to paged_adamw_8bit and gradient_accumulation_steps=4, both of which keep memory low while changing the effective batch size.

The script sets both push_to_hub=True in SFTConfig and an explicit trainer.push_to_hub(dataset_name=dataset_name) call. Before running, ensure your Hub namespace permits writes and that the same output_dir does not collide with an existing repository.

Once the adapter is validated, the natural next step is to wire the same tokenizer and LoRA weights into a minimal agent loop that parses the <tool_call> block, executes the function, and returns the result as the next assistant message. If you are also running larger tool-using systems, the routing and validation concerns from AWS Continuum's code vulnerability findings are a useful reminder that structured output is only half the problem; the tool execution path still needs its own safeguards.

Frequently asked questions

How do I fine-tune tiny-aya-global for tool calling with TRL?

Use Hugging Face TRL's sft_tiny_aya_tool_calling.py, which loads CohereLabs/tiny-aya-global and bebechien/SimpleToolCalling, applies QLoRA with LoraConfig(r=32, lora_alpha=32), and trains with SFTTrainer. The companion tiny_aya_chat_template.jinja is the part that serializes tool schemas into the system preamble. Save and push the adapter with the tokenizer so inference can call apply_chat_template(tools=TOOLS).

What dataset does the TRL tiny Aya tool-calling example use?

The example uses bebechien/SimpleToolCalling and loads only the train split. create_conversation maps each sample to a user prompt, an assistant tool_calls completion, and the two tool schemas. It then applies train_test_split(test_size=0.5, shuffle=True), but the script does not evaluate on the test half.

How do I run inference with the fine-tuned tiny Aya tool-calling adapter?

Load the tokenizer from the output directory rather than the base model tokenizer because the saved tokenizer contains the modified chat template. Call apply_chat_template with tools=TOOLS on each new user input. The assistant output should contain a structured <tool_call> block, which you then parse as JSON before executing the function.

Why does the TRL script require tiny_aya_chat_template.jinja?

CohereLabs/tiny-aya-global does not have native tool-calling support, so the Jinja2 template wraps tool schemas in the system channel and emits assistant calls as <tool_call> XML inside the model's native response markers. If the file is missing, training fails early because chat_template_path cannot be resolved. Without it, the model would not see the structured tool definitions it is supposed to learn.

What does use_kernels=True do in the tiny Aya tool-calling script?

use_kernels=True enables a kernel path during model loading and requires the kernels package on a supported CUDA environment. On unsupported GPUs or older drivers it can raise an import or runtime error even after the rest of the stack installs cleanly. If that happens, test without the flag; the unmodified script has it enabled.

How much VRAM does the tiny Aya tool-calling fine-tune need?

The source script does not publish a minimum VRAM requirement. It is configured to be memory-constrained with 4-bit NF4 quantization, LoRA rank 32, batch size 1 with gradient accumulation over 4 steps, and activation offloading. A CUDA-capable GPU is still required, but the exact limit depends on driver support, dataset length, and whether use_kernels=True works in your environment.

Related Guides