SigLIP LoRA Fine-Tune Cuts Under-Labeling from 9.4% to 3.4%

Alma Media's real estate platform processes hundreds of thousands of listings annually, each carrying dozens of unlabeled photos. The engineering team built a 23-class multi-label image classifier to tag room types (LIVING ROOM, KITCHEN, BEDROOM, and others), schematic content (floor plans, site plans), and exterior content (aerial shots, garden photos). Their published report documents the full decision path from prompted VLM to LoRA-adapted google/siglip-base-patch16-224 — and critically, it quantifies when frozen backbones fail in ways that raw F1 scores obscure.

The core finding: a frozen SigLIP classifier left 9.4% of photos without any label at its best operating threshold, 2.4× the natural unlabeled rate of 3.9% in the test set. LoRA fine-tuning drove that figure to 3.4%, essentially matching the floor. That gap — not the 4-point F1 lift — is what justified the engineering investment. For practitioners weighing pipeline architecture decisions over model upgrades, the under-labeling rate is the right metric to watch first.

Three Questions That Gate the Decision

Question 1: API or custom model? Using Google's Agent Platform with gemini-3.5-flash at July 2026 pricing costs roughly $1.50 per 1,000 images at 1K resolution — approximately $1,500 per million photos. Running their own classifier on an AWS EC2 g4dn.xlarge (single NVIDIA T4, 16 GB VRAM) at $0.53/hour on-demand, with throughput of at least 400 images per second, costs roughly $0.37 per million — about 1/4,000th of the API cost for inference compute alone. Below a few hundred photos per day, the cost difference is immaterial. At scale, it is decisive.

A secondary issue with API routes is confidence calibration. Verbalized confidence from VLMs is known to be poorly calibrated, and token log-likelihoods from an API rarely correspond to the class probabilities actually needed. Custom classifiers expose per-class scores that are explicit, thresholdable, and calibratable.

Question 2: Which foundation model? SigLIP (Google) is trained on captioned images and attends to caption-worthy objects — making it strong on KITCHEN but weak on GARDEN, which is predominantly background. DINOv2 (Meta) uses patch-level self-supervised objectives, weighting every image region equally. That distinction showed up directly: frozen DINOv2 scored 58% F1 on GARDEN, a 15-point improvement over frozen SigLIP — recovering more than half the gap to fine-tuned SigLIP without any training. Yet DINOv2 never predicted the MARKETING class and dropped 7 points on KITCHEN relative to SigLIP, showing that foundation model selection is task-topology dependent, not a general ranking.

Question 3: Freeze or fine-tune? Linear classifiers trained on frozen representations can be fit on a CPU in minutes from a 100k-instance set. LoRA, applied to the self-attention query and value layers of the SigLIP ViT backbone, trains only ~0.6% of model parameters — roughly a 99% reduction compared to full fine-tuning — using the PEFT library with BCE loss across 23 independent logistic heads. A 40-trial random hyperparameter sweep over learning rate (1e-5 to 1e-3, log-uniform), batch size ({16, 32, 64}), and LoRA rank ({8, 16, 32} with lora_alpha locked to lora_r) ran entirely on a single T4, with the full sweep costing approximately $30. Training used fp16 mixed precision with loss scaling over 20 epochs.

Benchmark Results

All models trained on 40,000 manually annotated proprietary photos, validated on 1,800 held-out test examples from distinct listings.

The F1 improvement is largely a recall story: +6.5 points over frozen SigLIP, +6.9 over frozen DINOv2, while precision moved only ~1 point. Per-class gains are more dramatic: GARDEN improved 26 points and DINING AREA 15 points versus frozen SigLIP. The one regression was KITCHEN, which dropped 5 points under LoRA. Precision–recall curves confirm fine-tuning dominates at every operating threshold — meaning the advantage is structural, not an artifact of threshold selection. The frozen models' unlabeled rates climb steeply as the threshold rises, making them inflexible for applications that need to trade recall for precision.

When the Math Favors Staying Frozen

The team is explicit about when LoRA is not the right call. If periodic retraining is required, the cost asymmetry matters: 23 logistic regression heads on frozen embeddings train in minutes on CPU; a full LoRA hyperparameter sweep takes days on a GPU instance, and that cost recurs with every retraining cycle. If under-labeling is not a measurable problem for a given deployment, the frozen approach avoids that recurring expense entirely. And if the classification task requires structured output rather than class labels — extracting JSON from building schematics, for example — neither a frozen head nor LoRA solves the problem; a prompted VLM does.

The broader signal for practitioners is that the under-labeling rate, not micro-averaged F1, should be the primary diagnostic for deciding whether fine-tuning is warranted. When downstream automation consumers need reliable per-photo coverage — as search and recommendation systems do — a frozen model's confidence distribution may be structurally incapable of meeting that requirement regardless of threshold tuning. LoRA's value here was not incremental accuracy; it was making the classifier's confidence regime usable in production without class-specific threshold engineering.