Run DeepSeek V4.1 Flash Locally with llama.cpp (v1.0.0)
In this article
- What You're Setting Up
- Size the Machine from the Actual File Layout
- Step 1: Build llama.cpp v1.0.0 with CUDA
- Step 2: Download the ggml-org GGUF
- Step 3: Start llama-server
- Step 4: Test with a Fixed Prompt
- Step 5 (Optional): Speculative Decoding with the DSpark Draft
- Step 6: Troubleshoot with the Known Bugs
- What Performance to Expect
- When to Use vLLM or SGLang Instead
- Practical Verdict
- What to Read Next
What You're Setting Up
DeepSeek V4.1 Flash is an MIT-licensed multimodal mixture-of-experts model. Per DeepSeek's model card, it has 552B backbone parameters arranged as a 20-layer causal encoder followed by a 20-layer decoder, which lets it activate only 8B parameters per token during prefill and 16B during decode. The full checkpoint is larger than that headline: Hugging Face counts 763.2B parameters in the 1017 safetensors, because it also carries 196B parameters of "Engram conditional memory", a DSpark speculative-decoding draft and a vision encoder. The official FP8/FP4 weights are 510.3 GB.
This guide runs the 1017 checkpoint (released 17 October 2026, a minor update to the 10 September original) with llama.cpp v1.0.0 and the official ggml-org GGUF, on a single machine that combines GPU memory with system RAM.
November 2026 status: llama.cpp support for the
deepseek41architecture was merged in PR #31086 on 25 October 2026 and first shipped in build b11870. The stable v1.0.0 release (13 November) includes it, plus external per-layer embedding shards (--shard/--ple). Five correctness bugs for this model are still open; each one has a row in the troubleshooting table below.
How this guide was checked: We have not run this procedure on our own hardware. Every flag below was checked against
common/arg.cppat the v1.0.0 tag, every file size against the Hugging Face API, and every bug against its GitHub issue on 22 November 2026. The only throughput figures quoted come from a llama.cpp maintainer and are attributed to them.
Requirements:
- Linux with an NVIDIA GPU and a CUDA toolkit (the build below uses CUDA)
- About 190 GB of combined VRAM and system RAM for Q2_K_S, plus context (see the sizing section)
- About 310 GB of free NVMe storage for the GGUF and optional draft model
- Python 3 for the Hugging Face download CLI
Size the Machine from the Actual File Layout
The ggml-org repository ships two text quantizations, each split into six shards. Reading the GGUF headers shows that the fifth shard of both holds a single tensor, per_layer_token_embd.weight, the per-layer embedding (PLE) table that the GGUF model card says is stored as IQ4_NL. At IQ4_NL's 4.5 bits per weight, 110.6 GB is about 196.6B parameters, which matches the 196B that DeepSeek gives for the model's Engram memory. DeepSeek also describes that memory as "sparsely accessed via token-based lookup", which is why it does not need to sit in RAM.
| Shard | Contents | Q2_K_S | Q2_K |
|---|---|---|---|
| 00001 | Metadata | <0.1 GB | <0.1 GB |
| 00002–00004 | Routed expert weights (ffn_up/gate/down_exps) | 178.3 GB | 215.1 GB |
| 00005 | per_layer_token_embd.weight (PLE table, the Engram memory) | 110.6 GB | 110.6 GB |
| 00006 | Attention, indexer, shared-expert, router, Engram KV, hyper-connection and output tensors | 9.1 GB | 9.1 GB |
| Total | 298.0 GB | 334.8 GB |
That fifth shard is what makes this model fit on one machine. llama.cpp v1.0.0's --lazy-mode defaults to auto, which its help text defines as reading the rows of large tensors "from disk on demand instead of keeping them resident", applied to tensors larger than 4 GiB, and it requires mmap. The PLE table is one 110.6 GB tensor, so under the default it stays on disk.
What has to be resident is therefore the rest:
- Q2_K_S: 298.0 − 110.6 = about 187 GB of weights
- Q2_K: 334.8 − 110.6 = about 224 GB of weights
Add the KV cache for your chosen context and llama.cpp's compute buffers on top. These numbers are arithmetic from file sizes, not a measurement: confirm them against the memory llama-server reports at load time. If you have room for neither, see when to use vLLM or SGLang instead. To budget other models the same way, use the VRAM calculator.
Two practical consequences:
- Put the GGUF on fast NVMe. The PLE table is read on demand during inference, so disk latency becomes part of your token latency.
- Do not pass
--no-mmap. Lazy mode depends on mmap. Disabling it forces the whole 110.6 GB table into memory.
Step 1: Build llama.cpp v1.0.0 with CUDA
Clone the release tag rather than master, so the binary you test is the binary you can rebuild:
git clone --branch v1.0.0 --depth 1 https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
Configure with CUDA and build the server and CLI. The commands follow the project's build documentation:
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j --target llama-server llama-cli
Confirm the binary runs:
./build/bin/llama-server --version
Record the printed version with your deployment notes. If you later move to a newer bNNNNN build for a bug fix, rerun your validation prompts before trusting it.
Step 2: Download the ggml-org GGUF
Install the Hugging Face CLI:
pip install -U "huggingface_hub[hf_xet]"
Download the six Q2_K_S shards, pinned to the repository revision this guide was checked against:
hf download ggml-org/DeepSeek-V4.1-Flash-1017-GGUF \
--revision 6bc1f89d4b7ae7125ce9a795ea20ef6b5d0e8099 \
--include "DeepSeek-V4.1-Flash-1017-Q2_K_S-*.gguf" \
--local-dir ~/models/dsv41-flash-1017
For Q2_K instead, change the pattern to DeepSeek-V4.1-Flash-1017-Q2_K-*.gguf. Per the model card, Q2_K keeps the routed down-projection experts in MXFP4, while Q2_K_S converts them to Q2_K. Neither quantization used an imatrix.
Check that all six shards arrived:
ls -lh ~/models/dsv41-flash-1017
You should see -00001-of-00006 through -00006-of-00006. llama.cpp loads a split GGUF when you point it at the first shard, so a missing shard fails at load time, not at download time.
Do not substitute the Unsloth 1017 GGUF here. In issue #31385, maintainer CISC explained that it was built with an unofficial quantization type ID that clashes with the FP8 type merged into ggml. A user confirmed it then loaded but "outputs gibberish".
Step 3: Start llama-server
./build/bin/llama-server \
--model ~/models/dsv41-flash-1017/DeepSeek-V4.1-Flash-1017-Q2_K_S-00001-of-00006.gguf \
--alias deepseek-v4.1-flash \
--host 127.0.0.1 \
--port 8080 \
--jinja \
--ctx-size 32768 \
--fit on \
--flash-attn on \
--cache-ram 0 \
--temp 1.0 \
--top-p 1.0
What each flag is doing, with the v1.0.0 help text where it matters:
| Flag | Why it is here |
|---|---|
--jinja | Uses the chat template embedded in the GGUF. DeepSeek's own release ships no Jinja template, but the ggml-org GGUF embeds one ported from DeepSeek's encoding.py, and v1.0.0's chat parser has V4.1-specific detection for its tool-call format. |
--ctx-size 32768 | The model supports 1,048,576 tokens, but start small. Raise it only after the server loads with headroom. |
--fit on | "Adjust unset arguments to fit in device memory." It is the default in v1.0.0; it is written out here so the intent is explicit. It decides how many layers and experts stay on the GPU. |
--flash-attn on | Fine on CUDA. Switch to off on Vulkan (see issue #31480 below). |
--cache-ram 0 | Disables the host-memory prompt cache. This is the workaround for issue #30944, where reusing a cached prompt prefix corrupts output. You lose cross-request prompt reuse in exchange for correct tokens. |
--temp 1.0 --top-p 1.0 | DeepSeek's recommended sampling for the 1017 checkpoint, from its model card. |
--host 127.0.0.1 | Keeps the server local. If you bind to 0.0.0.0, add --api-key. |
Loading maps about 300 GB from disk, so the first start takes a while. When the terminal reports that the server is listening, note the memory figures it logged for each device and compare them with your budget from the sizing section.
Tuning flags, once it works. --fit handles placement automatically. If you want manual control, v1.0.0 offers --n-cpu-moe N ("keep the Mixture of Experts (MoE) weights of the first N layers in the CPU"), and a new expert cache, --moe-cache-mib N, a "GPU cache size in MiB for the MoE experts kept in the CPU" that is disabled by default. Change one flag at a time and rerun the same prompt.
Step 4: Test with a Fixed Prompt
Use the OpenAI-compatible endpoint with a short prompt whose answer you can check:
curl http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-v4.1-flash",
"messages": [{"role": "user", "content": "Solve 5x + 12 = 47. Show your work."}],
"chat_template_kwargs": {"reasoning_effort": "low"}
}'
chat_template_kwargs passes options to the embedded template. Per the template's own header, it accepts:
thinking_mode:"thinking"(the default) or"chat"reasoning_effort: an integer from 1 to 100, or"low","high"(the default) or"max"
Thinking at high effort can produce very long reasoning before the answer, so "low" keeps the first smoke test quick.
Then repeat the same request two or three times. Correct output once and corrupted output on a repeat is the signature of the prompt-cache bug. If --cache-ram 0 is set and you still see it, record your build and GGUF revision and compare against issue #30944.
Step 5 (Optional): Speculative Decoding with the DSpark Draft
The ggml-org repository also ships DeepSeek's DSpark draft model as dspark-DeepSeek-V4.1-Flash-1017-MXFP4.gguf (7.9 GB) and a BF16 version (8.5 GB). Download it alongside the main model:
hf download ggml-org/DeepSeek-V4.1-Flash-1017-GGUF \
--revision 6bc1f89d4b7ae7125ce9a795ea20ef6b5d0e8099 \
--include "dspark-DeepSeek-V4.1-Flash-1017-MXFP4.gguf" \
--local-dir ~/models/dsv41-flash-1017
Then add three flags to the Step 3 command:
--model-draft ~/models/dsv41-flash-1017/dspark-DeepSeek-V4.1-Flash-1017-MXFP4.gguf \
--spec-type draft-dspark \
--spec-draft-p-min 0
--spec-draft-p-min 0 is required. v1.0.0's speculative decoder throws "DSpark draft has no confidence head: please set --spec-draft-p-min 0" otherwise. Speculative decoding only helps if the draft's tokens are accepted often enough to repay its cost, and the draft also needs memory. Compare decode speed with and without it on your own prompts, and keep it only if it wins.
Step 6: Troubleshoot with the Known Bugs
All five issues below were open on 22 November 2026. Match your symptom before changing sampling settings: these are engine bugs, and adjusting temperature will only hide them intermittently.
| Symptom | Where it was reported | Workaround |
|---|---|---|
| A wrong token appears when a prompt prefix is reused (repeated requests, multi-turn chat) | #30944 | --cache-ram 0 (already in Step 3). One tester reported no corruption on b12375 with defaults; the issue stays open. |
| Gibberish on the Vulkan backend with flash attention | #31480 | --flash-attn off |
| Garbage output on CUDA sm_120 GPUs (RTX 50-series, RTX PRO 6000 Blackwell), reported with an Unsloth IQ1_M quant | #32367 | --no-op-offload |
| CPU-only run on Intel AMX aborts in the scheduler | #32197 | No documented workaround. Use a GPU build. |
| An Unsloth 1017 GGUF loads but outputs gibberish | #31385 | Use the ggml-org GGUF from Step 2. |
What Performance to Expect
There is no official performance figure for llama.cpp. The one public data point comes from llama.cpp maintainer pwilkin, in his review of PR #31086 on 24 October 2026. He ran a pre-merge build, with a small patch, on an AMD R9700, an RTX 5060 and 128 GB of RAM, using a smaller community GGUF (ISTA-DASLab's GSQ quantization), not the ggml-org file used here:
| Phase | Tokens | Speed |
|---|---|---|
| Prompt processing | 8,318 | 31.32 tokens/s |
| Generation | 850 | 6.55 tokens/s |
He called prefill "somewhat disappointing right now" and decode "quite decent". At 31 tokens per second, an 8K-token prompt takes over four minutes to process. Plan interactive use around short prompts, or measure your own hardware before committing to long-context work.
When to Use vLLM or SGLang Instead
llama.cpp is the path for one machine with mixed GPU and system memory. If you have datacenter GPUs, the server engines are better documented for this model:
- vLLM: the official recipe requires vLLM 0.31.0 or later and lists a 614 GB minimum VRAM for the full checkpoint. It also documents a single GB300 configuration with a 244 GB GPU budget, which keeps the Engram tables and 60 GiB of experts in Grace host memory.
- SGLang: maintains a DeepSeek V4.1 cookbook page with launch recipes and feature compatibility.
- Ollama: the library entry offers only
cloudand1017-cloudtags, so it is not a local option yet.
For the exact vLLM commands on 4x H200 and 8x H100, see Serving DeepSeek V4.1 Flash on vLLM in our self-hosting guide.
Practical Verdict
Unlike V4 Flash, which needed a work-in-progress community fork, V4.1 Flash runs on a stable upstream llama.cpp release with an official ggml-org GGUF.
It is still an evaluation setup, not a turnkey one:
- About 187 GB of resident weights at the smallest official quantization puts it beyond most single workstations.
- Five open correctness bugs mean you should validate repeated and multi-turn prompts, not just one-shot answers.
- The only public throughput figure shows slow prefill, so long prompts are expensive.
If you have the memory, start with Q2_K_S, the exact flags above, and a fixed validation prompt. Record the build version and GGUF revision, and change one variable at a time.
What to Read Next
- Run DeepSeek V4 Flash Locally with llama.cpp: the previous generation on a single 96 GB GPU
- Self-Host DeepSeek V4 with vLLM: multi-GPU serving with tensor parallelism
- DeepSeek V4.1 Flash: 552B MoE at 890 bytes per token: what changed in the architecture
- VRAM calculator: budget memory for other models before you download them
Frequently asked questions
Can llama.cpp run DeepSeek V4.1 Flash?
Yes. Support for the deepseek41 architecture was merged in llama.cpp PR #31086 on 25 October 2026 and first shipped in build b11870. The stable v1.0.0 release (13 November 2026) includes it, along with loading the per-layer embedding table from disk on demand. Several correctness bugs are still open, so validate output on your own hardware before relying on it.
How much memory does DeepSeek V4.1 Flash need in llama.cpp?
The ggml-org Q2_K_S GGUF is 298.0 GB, but 110.6 GB of that is a single per-layer embedding tensor that llama.cpp's default lazy mode reads from disk on demand. Plan for roughly 190 GB of combined VRAM and system RAM for the remaining weights, plus KV cache and compute buffers. That figure is arithmetic from the file sizes, not a measurement.
Which GGUF should I use for DeepSeek V4.1 Flash?
Use the official ggml-org/DeepSeek-V4.1-Flash-1017-GGUF conversion (Q2_K_S at 298.0 GB or Q2_K at 334.8 GB). A llama.cpp maintainer reported that the Unsloth 1017 GGUF loads on upstream builds but outputs gibberish, because it used an unofficial quantization type ID that clashes with the newly merged FP8 type.
What is the difference between DeepSeek V4.1 Flash and V4.1 Flash 1017?
DeepSeek describes 1017 as a minor update to V4.1 Flash with better reasoning, coding and agentic results and shorter outputs. Both checkpoints are 510.3 GB in their official form and MIT licensed. DeepSeek recommends temperature 1.0 and top_p 1.0 for 1017.
Related Guides
Run DeepSeek V4 Flash Locally with llama.cpp (2026)
Run DeepSeek V4 Flash locally with llama.cpp, a GGUF build, one 96GB GPU, and explicit checks for the current experimental support path.
The Complete Developer Guide to Running LLMs Locally: From Ollama to Production
Everything you need to run LLMs on your own hardware in 2026: VRAM sizing, model formats, an 8-tool comparison table, a full local RAG pipeline, and Docker production deployment with GPU passthrough and Nginx auth.

Benchmark Local Models with llama.cpp's llama-eval.py
Use llama.cpp's single-file eval harness to run reproducible AIME, GSM8K, and GPQA benchmarks against a local server with Wilson confidence intervals.