LOCAL LLM FIELD GUIDE

EXL3 is ExLlamaV3's trellis code with a target average bitrate

EXL3 is the quantization format of ExLlamaV3, the inference library by turboderp. The project README calls it a streamlined variant of QTIP from Cornell RelaxML. QTIP, Quantization with Trellises and Incoherence Processing, is a NeurIPS 2024 Spotlight paper. The converter takes an average bits-per-weight target, including fractional targets, and assigns integer widths per tensor. A runtime can decode the file only when that runtime has EXL3 kernels.

TRELLIS WEIGHTS

The average bitrate and the trellis code are different layers

QTIP: Cornell trellis after a Hadamard rotation
mul1: Default codebook in the EXL3 converter
Average: Such as 2.90, then integer widths per tensor

Thai explanatory diagramThai-language diagram. The English article text provides the matching terminology and caveats.

USE IT ON REAL WORK

Read an EXL3 pack before you pick hardware

Start from the average bitrate on the card, then read the per-tensor widths and the tensors left at higher precision.

Then check whether the engine you run has kernels for the format, or whether the recipe pins a specific image.

Question	Where to look	Rule used on this page
What the bpw number is	An average after allocation, with integer widths per tensor	2.90 on the Mia card does not mean every weight is stored at 2.9 bits
Which codebook	The converter default is mul1	mcg and 3inst are for testing, per the conversion guide
Why the file is larger	Embeddings start at 16 bits and lm_head starts at 6	DeepSeek Engram tables sit outside the pack
Dropping the file on vLLM	Check that the image has EXL3 kernels	Stock vLLM and SGLang wheels do not load this format
Will two Sparks fit	Compare the official 475.251 GiB tree with the 196 GiB pack	256 GB across two machines does not hold the official tree

Before trusting a low-bit quality claim

Record the codebook name and the --bits target used to convert
Separate the pack average from attention, expert, head, and embedding widths
Name the engine: ExLlamaV3, a community image, or the fork a recipe pins
For DeepSeek, locate the Engram tables outside the EXL3 pack
Measure KL or your own prompts, and do not reuse the base model's published scores

Flag shape from the ExLlamaV3 conversion guide

bits = 2.90
codebook = mul1
head_bits = 6
embed_bits = 16
hq = on
mtp_bits = 4

What QTIP contributes

QTIP compresses trained weights with trellis-coded quantization instead of a vector codebook whose size grows exponentially with dimension. The structure the paper uses for parallel decode is a bitshift trellis. Reconstruction is computed from the trellis state, so inference does not keep a giant lookup table. The paper calls these computed codes and gives 1MAD and 3INST as lookup-free examples.

Before coding, QTIP applies incoherence processing with the random Hadamard transform from QuIP#, so the weights look closer to a Gaussian and a few huge outliers do less damage. The paper's lookup-free experiments use a 16 by 16 block, 256 values, as one trellis sequence so it matches a 16 by 16 matrix-multiply tile, and a tail-biting trellis so the stored stream does not keep a separate initial state. EXL3's own quantize.py comment expects that 16 by 16 tile permutation in the tensor-core kernels.

The paper reports Llama perplexity against QuIP# and AQLM at 2, 3, and 4 bits, and those tables land closer to the FP16 baseline than the two comparison methods. The paper does not publish a GGUF Q4 or Q5 table, so this page does not treat EXL3 as a winner over GGUF on every model.

What the EXL3 converter actually sets

convert.py takes an unquantized Hugging Face model and a -b or --bits target, a float average bits per weight. The conversion guide accepts three codebook names through --codebook: mul1, the default, plus mcg and 3inst. The guide says to stay on mul1, because some optimized paths, including int8 GEMV and CPU expert offload, require it, and another codebook is for testing.

The -hq or --hq flag raises the floor on layers the converter selects, such as attention and shared experts. The file can end up above the requested --bits. On MoE models the guide puts that increase at about 0.05 to 0.10 bpw for the architecture's default promotion, and -hq 2 doubles the promotion.

Other defaults make the file larger than the decoder average. --embed_bits starts at 16, so the token embedding stays unquantized. --head_bits for lm_head starts at 6. --mtp_bits starts at 4. --vision_bits starts at 6 or 16 depending on the architecture. Vision and MTP layers are quantized without calibration.

2.9 bpw is an average, not one code width

A card that says 2.9 bpw is reporting the average after allocation, not a fractional code stored on every weight. The card for Mia-AiLab/DeepSeek-V4.1-Flash-1017-EXL3-EAQ-v1-2.9-bpw states an average of 2.90 over the decoder body. Routed experts are mostly 3 bit, layers 18 to 22 are 2 bit, attention is 5 bit, shared experts are 4 to 5 bit, the indexer wk is 8 bit, the head is 6 bit, and the MTP and DSpark tensors are 4 bit.

That card calls the pack EAQ v1 and says the file format is ordinary EXL3: quant_method exl3, codebook mul1, out_scales always. It says the main-model tensor keys and per-tensor widths match a standard EXL3 2.9 bpw build the card compared. EAQ v1 changes the conversion, on exllamav3 v1.6.1 plus the EAQ patch set. A loader that already reads EXL3 DeepSeek-V4.1 checkpoints can read this pack without EAQ-specific code.

The card reports teacher-forced KL against the original weights at revision 1bae0e10, in the authors' evaluation port. On 100 short conversations the KL is 0.0137 nats and top-1 agreement is 96.6 percent, over 89,880 scored assistant tokens. Confidence intervals and the per-task table are on the card. That measurement is not free generation, not vision quality, not DSpark quality, and not context past the longest session on the card, 31.2k tokens.

Which stacks can run it

The ExLlamaV3 docs point at the library itself, and they recommend TabbyAPI when you want an OpenAI-compatible server. The published install path is PyTorch with CUDA 12.4 or newer. This page does not describe an AMD EXL3 kernel from that install guide.

Stock vLLM and stock SGLang do not load EXL3 as a built-in format. A community image that ports the kernels is a different path from the stock wheel. TensorFold's release model table lists GLM-5.3-Flash EXL3/TR3 as an experimental checkpoint on two CUDA ranks. DeepSeek is not in the table of checkpoints a release binary serves. The two-Spark recipe below therefore pins a different engine line.

Quality still depends on calibration. The converter's default is its bundled example mix, and --cal_data can replace that mix with packed token rows from a real trace. A workload-matched trace is an input the tool accepts. It is not a promise about downstream task scores. Measure the prompts you will actually serve.

Why the 2.9 bpw pack is the two-Spark path

The official DeepSeek-V4.1-Flash-1017 tree recorded by this calculator on 2026-10-19 is 48 safetensors summing to 510,296,708,312 bytes, or 475.251 GiB. Two 128 GB DGX Sparks provide 256 GB of unified memory, so that whole tree does not fit as resident weights. Subtracting the two Engram shards measured at 189.127 GiB still leaves about 286 GiB, which is also above 256 GB. Three machines provide 384 GB, enough for the remainder when the Engram tables stay on disk. Four machines provide 512 GB, enough for the whole tree resident. This is arithmetic on measured file sizes, not a benchmark of a three-Spark or four-Spark recipe.

The MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks recipe serves the 196 GiB EXL3 pack above, about 98 GiB of weights on each Spark, through TensorFold's Zig 0.17.0 line and BertholomusAI's dsv41 family at TP=2, one rank per machine, over the ConnectX-7 ports with RoCE v2. The advertised window is 1,048,576 tokens, with image input, tool calling, thinking, and DSpark from the checkpoint's own draft blocks. On a head that also runs other software, the README calls 524288 the practical CONTEXT ceiling.

The Engram tables are not inside the quantized pack. You still need shards 47 and 48 from the same base revision. The card puts each near 95 GiB. The recipe README says those tables must sit on local disk on both Sparks. When they are read over NFS, prefill in that report falls to about 690 to 890 tok/s.

Read the recipe tok/s with its conditions

The recipe README reports sparkDash numbers from one pair of Sparks. Both GPUs were capped at 2,200 MHz, the server CONTEXT was 524288, drafts were on, the head was also running a desktop and about 18 small containers, and the worker was otherwise idle. At one stream, prose at temperature 0 with thinking off was 80.7 tok/s with TTFT 133 ms. Prose at temperature 0.7 with thinking on was 63.9 tok/s with TTFT 214 ms. Code at temperature 0 with thinking off was 118.1 tok/s with TTFT 213 ms.

Prefill in the same table stays near 2,100 tok/s from about 4K to about 256K tokens when the Engram tables are local on both machines. The 4,114-token point measured 2,105.9 tok/s, and the 262,163-token point measured 2,065.0 tok/s.

Important note: that table is one pair under the conditions above. It is not the tok/s the llm-cal calculator estimates from its roofline, and it does not transfer to a different clock, context, or draft setting.

Frequently asked questions

Can I swap EXL3 and GGUF?

No. EXL3 is ExLlamaV3's weight format, a trellis code after a Hadamard rotation, with a fractional average bitrate target. GGUF is the llama.cpp container, and a label such as Q4_K_M is one level inside that container. The kernels differ, so compare them on one model with a named measurement.

Does 2.9 bpw mean every weight uses 2.9 bits?

No. The 2.90 figure on the Mia pack is the decoder average. On that card most experts are 3 bit, some layers are 2 bit, attention is 5 bit, and the head is 6 bit. The embedding table and the Engram tables stay at higher precision and sit outside that average.

Sources

Try your configuration

Put the intended model, quantization, context, framework, and hardware into the calculator, then validate with a production-like benchmark.

Open calculator