Four-Layer Local AI Stack Runs SLMs With No Cloud Dependency

The gap between spinning up a local model in a terminal and running one that meaningfully accelerates a development workflow is almost entirely a tooling problem. With open-weight models in the 1B–14B parameter range now running on consumer hardware carrying 8–24 GB of VRAM or Apple Silicon unified memory, the hardware ceiling has dropped enough that stack architecture — not raw compute — is the primary variable under a developer's control. A four-layer framework covering serving, IDE integration, terminal automation, and context retrieval gives practitioners a structured basis for making those choices without locking in any single vendor.

This matters beyond hobbyist tinkering. As pipeline architecture increasingly drives AI productivity gains over model selection alone, getting the serving and retrieval layers right has a measurable effect on output quality for small models operating under tight context budgets.

Layer 1 — Serving: The Control vs. Convenience Spectrum

Every upstream layer inherits the constraints and capabilities of the inference engine underneath. Ollama runs as a lightweight background service, handles VRAM allocation and hardware detection without configuration, and exposes a REST API that most higher-level tools already target. Its main limitation is that it abstracts away deeper performance tuning.

LM Studio presents a fully visual desktop interface for pulling models from the Hugging Face Hub and serves as a drop-in OpenAI API replacement — well-suited for side-by-side model evaluation, poorly suited for headless deployments.

Llama.cpp is the inference engine Ollama wraps. Accessing it directly unlocks granular control over quantization formats and compilation targets, including CPU-only and edge hardware, at the cost of steep manual setup. vLLM solves a different problem: built around PagedAttention and continuous batching, it is a GPU-native engine designed for high-throughput concurrent requests, relevant when serving a local model to an entire engineering team rather than a single developer.

Layer 2 — IDE Integration: Agents vs. Autocomplete

Cline is the leading option for VS Code-embedded agentic coding. It operates as an autonomous agent: plans multi-step tasks, creates and edits files, and executes terminal commands, with a Plan/Act separation that surfaces the model's proposed approach before any action is taken. It integrates with the Model Context Protocol for access to external tools and APIs, is model-agnostic, and works directly against a local Ollama endpoint. With over 5 million VS Code installs and 60,000+ GitHub stars, it has consolidated the open-source coding-agent category.

One architectural cost of agentic tools is context consumption. A 7-billion-parameter model on consumer hardware exhausts its context window far faster under agentic task execution than under simple autocomplete, making context window size a first-class model selection criterion.

Cursor, which acquired Continue.dev in June 2026, now incorporates lighter-weight Copilot-style completions, but it is a commercial IDE rather than a local-first tool. Continue.dev itself has been discontinued — its GitHub repository is read-only with no further releases planned — making Cline the most direct migration path for developers who built setups around it.

Layers 3 & 4 — Terminal Automation and Retrieval

CLI tools extend model access into repo-wide refactoring, headless pipelines, and CI/CD integration. Aider brings Git-native version control to AI-assisted edits, committing changes with generated commit messages and tracking modifications across files, but requires leaving the visual editor environment. OpenCode, written in Go and carrying more than 165,000 GitHub stars, is provider-agnostic, manages file reading, shell execution, and LSP integration, and is explicitly designed for headless embedding in automated pipelines — though its rapid development pace introduces occasional breaking changes between versions. Claude Code offers strong multi-file reasoning but requires an internet connection for authentication even when model inference is routed through a local Ollama endpoint, disqualifying it for fully air-gapped deployments.

The retrieval layer determines what context the model sees at inference time. The direct corpus interaction pattern for AI agents that has gained traction in agentic systems depends on this layer functioning correctly. Embedded options such as Chroma and LanceDB run in-memory or on local disk with no infrastructure setup. Qdrant handles large embedding collections and multi-user access; pgvector adds vector search to an existing PostgreSQL stack without a new infrastructure component.

Composing the Stack

The layered structure's practical value is that each decision is independent: swapping the retrieval database does not require rebuilding the serving layer. A baseline individual-developer configuration — Ollama serving, Cline for IDE-embedded agentic work, Aider or OpenCode for terminal automation, and Chroma or LanceDB for local retrieval — covers the full range of daily development tasks with no cloud dependency and no per-token cost. Scaling to higher concurrency means moving to vLLM at the serving layer; scaling retrieval across a larger codebase or multiple users means moving to Qdrant. The upgrade is surgical, not structural.

For practitioners exploring which small language models are worth running on this infrastructure, model selection matters less than context window configuration and quantization format choices — decisions the serving layer controls directly. The local SLM ecosystem has crossed a maturity threshold where the binding constraint is no longer tool availability but informed assembly.