Quantize Falcon-7B with LitGPT to Cut GPU Memory to 5.4 GB

Published • guides
QuantizationLLM Inference

In the LitGPT quantize reference run, 4-bit NF4 quantization cuts Falcon-7B generation memory from 28.95 GB to 5.72 GB while staying nearly as fast as the 32-bit baseline. Double quantization lowers it further to 5.38 GB. The commands and outputs below are reproduced from the quantize.md documentation in the Apache-2.0 licensed LitGPT project. Treat the values as reference numbers from that documentation, not a guarantee for your GPU.

Running a 7B model locally used to mean either renting a datacenter GPU or accepting slow CPU inference. In 32-bit precision, Falcon-7B reports 28.95 GB of GPU memory during generation, which is too much for a 24 GB RTX 3090 or 4090. LitGPT's generate command accepts a --quantize flag that shrinks the model weights to 8 or 4 bits at load time. The main benefit is not always speed: quantization is primarily a memory-optimization technique. It lets you fit larger models or serve more concurrent requests on the same GPU, but you may pay with throughput depending on the quantizer.

Before you start

  • A Linux machine with a CUDA GPU. The bnb.* quantizers rely on bitsandbytes, which is CUDA-oriented in the reference environment; Windows users should plan to use WSL2.
  • A GPU with bfloat16 support if you want to run the bf16-true examples; otherwise substitute 16-true.
  • A working LitGPT installation with litgpt on your PATH.
  • Internet access to download tiiuae/falcon-7b from Hugging Face on first run.

No separate export step is required. Each command loads and quantizes the checkpoint during generation.

Step 1: Establish the 32-bit baseline

Before quantizing, record what full precision costs. Run generation in 32-bit precision and note the memory and throughput LitGPT reports:

litgpt generate tiiuae/falcon-7b \
  --precision 32-true \
  --max_new_tokens 256

The reference output reports 6.93 seconds and 36.96 tokens/s, with 28.95 GB of memory used. Write these numbers down: every quantizer is compared with them.

Step 2: Halve memory with BF16

The cheapest optimization is to stop using 32-bit floats. LitGPT converts the model weights to bfloat16 when you pass --precision bf16-true:

litgpt generate tiiuae/falcon-7b \
  --precision bf16-true \
  --max_new_tokens 256

Memory drops to 14.50 GB, and the run finishes in 5.37 seconds at 47.66 tokens/s. On hardware that supports bfloat16, that makes BF16 both faster and smaller. If your GPU does not support bfloat16, use --precision 16-true instead; the timing will differ.

Step 3: Apply NF4 4-bit quantization

If BF16 still does not fit your target GPU, install bitsandbytes and set --quantize bnb.nf4:

pip install bitsandbytes

litgpt generate tiiuae/falcon-7b \
  --quantize bnb.nf4 \
  --precision bf16-true \
  --max_new_tokens 256

The bnb.nf4 quantizer applies 4-bit NormalFloat4 quantization while loading the model. Memory falls from 28.95 GB to 5.72 GB. The run completes in 6.80 seconds at 37.62 tokens/s, effectively the same speed as the 32-bit baseline at a fraction of the memory.

Validation is direct: if the model loads and tokens are produced, the quantized weights are in use. A bad quantizer value or unsupported environment fails before generation begins.

Step 4: Add double quantization with nf4-dq

Double quantization reduces memory further by quantizing the quantization constants themselves. LitGPT exposes this as bnb.nf4-dq:

pip install bitsandbytes

litgpt generate tiiuae/falcon-7b \
  --quantize bnb.nf4-dq \
  --precision bf16-true \
  --max_new_tokens 256

Memory drops from 5.72 GB to 5.38 GB. That is a modest absolute saving for a 7B model, but it can matter more for a larger model or a smaller card. The run slows to 8.09 seconds at 30.87 tokens/s. Double quantization makes the most sense when you are memory-bound and accept the extra dequant overhead.

Step 5: Try FP4 and FP4-DQ variants

LitGPT also supports pure FP4 and its double-quantized form:

pip install bitsandbytes

litgpt generate tiiuae/falcon-7b \
  --quantize bnb.fp4 \
  --precision bf16-true \
  --max_new_tokens 256

Pure FP4 uses 5.72 GB and runs at 36.98 tokens/s, very close to NF4.

pip install bitsandbytes

litgpt generate tiiuae/falcon-7b \
  --quantize bnb.fp4-dq \
  --precision bf16-true \
  --max_new_tokens 256

FP4 with double quantization uses 5.38 GB but is the slowest 4-bit option at 25.54 tokens/s. If latency matters, NF4 or plain FP4 is usually the better starting point.

Step 6: Evaluate INT8 quantization

The final built-in option is bnb.int8. The reference example pairs it with --precision 16-true, not bf16-true:

pip install bitsandbytes

litgpt generate tiiuae/falcon-7b \
  --quantize bnb.int8 \
  --precision 16-true \
  --max_new_tokens 256

Memory is 8.70 GB, higher than the 4-bit methods, and the run takes 20.22 seconds at 12.66 tokens/s. Use INT8 only when you need higher numerical fidelity than 4-bit and cannot afford 16-bit memory; the throughput penalty here is severe.

Compare the results

The matrix below summarizes the reference run. “Memory used” is what LitGPT printed after generation; “tokens/s” is total tokens generated divided by total time.

Quantization Precision flag Memory used Time Tokens/s Notes
None (32-bit) 32-true 28.95 GB 6.93 s 36.96 Baseline; too large for typical 24 GB cards
None (BF16) bf16-true 14.50 GB 5.37 s 47.66 Fastest reference option
bnb.nf4 bf16-true 5.72 GB 6.80 s 37.62 Close to 32-bit speed at much lower memory
bnb.nf4-dq bf16-true 5.38 GB 8.09 s 30.87 Best 4-bit memory in the reference table
bnb.fp4 bf16-true 5.72 GB 6.92 s 36.98 Comparable to NF4
bnb.fp4-dq bf16-true 5.38 GB 10.02 s 25.54 Slowest 4-bit option
bnb.int8 16-true 8.70 GB 20.22 s 12.66 More memory than 4-bit; very slow

Use this matrix as a starting point, not a promise. Your GPU, driver, prompt length, and batch size will change the absolute numbers. The validation that matters is the trend: the command completes, tokens are generated, and memory drops in line with the selected quantizer.

What to watch out for

bitsandbytes is CUDA-focused. The bnb.* quantizers depend on bitsandbytes, which is designed for CUDA builds rather than CPU, macOS, or native Windows execution. Windows users generally need WSL2 with an NVIDIA runtime.

Bfloat16 support is not universal. The BF16 examples assume your GPU supports bfloat16. On older NVIDIA cards, replace --precision bf16-true with --precision 16-true. The INT8 example already uses 16-true and should not be switched to bf16-true blindly.

Speed is not guaranteed to improve. BF16 is faster than 32-bit in the reference run, but every 4-bit quantizer is slower than BF16 for this 7B model. If latency is critical, BF16 may be the better choice until memory forces you lower.

Double quantization's payoff depends on model size. For Falcon-7B, nf4-dq and fp4-dq save only 0.34 GB over their single-quantized counterparts while adding measurable overhead. That can be enough when a model is right at a memory limit, but the throughput drop is real.

These examples quantize in memory. The litgpt generate --quantize workflow applies quantization while loading the model; it does not export a new quantized checkpoint to disk. If you need a reusable quantized artifact, use a separate conversion workflow.

Quality is not benchmarked in the source. The reference material reports throughput and memory, not downstream task quality. Treat those measurements as the primary comparison, and run your own evaluation suite before shipping a quantized model.

Long context can shift the bottleneck. Once weights fit in memory, attention and KV-cache costs may become dominant. If you plan to run long inputs on a quantized model, our analysis of Tokengate long-context prefill explains where the next memory wall appears.

Where to go next

Start with bnb.nf4 and --precision bf16-true as the default quantized inference path. If you need more memory headroom, move to bnb.nf4-dq and verify that the throughput penalty is acceptable for your workload. Once inference fits, consider quantized fine-tuning workflows such as QLoRA-style adapters, which keep base weights in lower precision while training a small adapter; check the LitGPT documentation for current commands.

Frequently asked questions

How much GPU memory does Falcon-7B use with LitGPT quantization?

In the LitGPT reference run, 32-bit inference uses 28.95 GB, bf16 uses 14.50 GB, and bnb.nf4 or bnb.fp4 uses 5.72 GB. Double quantization with bnb.nf4-dq or bnb.fp4-dq lowers that to 5.38 GB, while bnb.int8 uses 8.70 GB. These are the reported memory-used values for generating 256 tokens with tiiuae/falcon-7b.

Which LitGPT quantize flag should I start with for Falcon-7B?

Start with --quantize bnb.nf4 --precision bf16-true. In the reference run it cuts memory from 28.95 GB to 5.72 GB and remains nearly as fast as the 32-bit baseline at 37.62 tokens/s. Use bnb.nf4-dq only if you need the drop to 5.38 GB and can accept the lower 30.87 tokens/s.

Why is bf16-true faster than 32-true in the LitGPT reference run?

The BF16 run completes 256 tokens in 5.37 seconds at 47.66 tokens/s, compared with 6.93 seconds and 36.96 tokens/s for the 32-bit baseline. It is the fastest option in the reference table, so it is worth trying before any quantizer on hardware with bfloat16 support.

Does litgpt generate --quantize export a quantized checkpoint?

No. The --quantize flag applies quantization while loading the model for generation and reports memory and throughput afterward. It does not write a new checkpoint to disk; the original downloaded weights remain unchanged. You need a separate conversion workflow if you want a reusable quantized artifact.

Can I use the bnb.* quantizers on macOS or native Windows?

The reference commands depend on bitsandbytes, which is designed for CUDA builds rather than CPU, macOS, or native Windows execution. Windows users generally need WSL2 with an NVIDIA runtime, and unsupported platforms should expect the bnb.* flags to fail. Check the current bitsandbytes installation documentation before starting.

Free interactive tools for the decisions this piece raises.

Related Guides