LOCAL LLM FIELD GUIDE

Quantization: what is reduced, and how to choose

Weight quantization reduces the fixed model footprint. KV-cache quantization reduces memory that grows with context and concurrent requests. Start with a format the runtime can actually load, then choose bit width.

TWO SEPARATE BUDGETS

Weights are fixed. KV cache grows with work

Weights: Reduce model footprint and bytes read per token
KV cache: Reduce memory per context × sequence

Thai explanatory diagramThai-language diagram. The English article text provides the matching terminology and caveats.

USE IT ON REAL WORK

Use quantization on the actual bottleneck

Start with the runtime you will operate, then choose the artifact and bit width. Weight quantization and KV-cache quantization reduce different memory pools, so inspect the breakdown separately.

If the model does not fit, inspect weight quantization first. If it fits but long context or concurrent requests do not, inspect KV cache, working context, and batch.

Symptom	Adjust	Verify
Weights do not fit	Choose a weight quant supported by the runtime	Checkpoint, kernel, and task quality
Long context exhausts KV	Choose a supported KV dtype	KV per sequence and long-context prompts
Running on a Mac	Use an MLX or GGUF/Metal path	Artifact converted for that runtime

Before downloading a model

Record the runtime and its release
Keep weight quant separate from KV cache dtype
Set working context from ordinary requests, not the maximum limit
Test important prompts before lowering bits further

Configuration record to compare

runtime = vllm
weight_format = AWQ
kv_cache_dtype = fp8_e4m3
working_context = 32768

Weights and KV cache are different budgets

Weights remain resident for the whole run, so they decide whether a model starts at all. KV cache grows with active tokens and sequences. long-context workloads can become KV-bound after weights already fit.

Match the artifact to the runtime

vLLM / SGLangUse a checkpoint and kernel supported by that runtime
llama.cpp / OllamaUse a GGUF artifact
MLXUse MLX-converted weights

Do not choose from the quant name alone: Artifacts called Q8, AWQ, or GPTQ are not automatically interchangeable across runtimes.

Quality risk is workload-specific

Four-bit weights often make a model fit, but validate reasoning, code, tool calling, and Thai prompts. Lower-precision KV can affect long-context behavior differently from lower-precision weights.

Use the calculator in order

Choose hardware and framework first. Inspect weights, KV cache, and overhead separately, reduce the actual bottleneck, then benchmark real requests before production.

Sources

Try your configuration

Put the intended model, quantization, context, framework, and hardware into the calculator, then validate with a production-like benchmark.

Open calculator