KNOWLEDGE BASE

Local LLM Guides: hardware, models, quantization, and serving

A practical guide library for choosing local LLM hardware, models, quantization, KV cache, serving, and performance paths.

Quantization: Where VRAM is Reduced and What Quality Trades AwayIn-depth guide to 4-bit and 8-bit quantization for weights and KV cache, summarizing QAT, MXFP4, AWQ, GPTQ, and GGUF with VRAM calculations, quality trade-offs, and hallucination risks.Why LLM tok/s Hits Memory Bandwidth BottlenecksDeep-dive into the Roofline Model for Local LLMs: why Decode is memory bandwidth bound, why Prefill is compute (FLOPS) bound, distinguishing TTFT, TPOT, ITL, and reading calculator results like a pro.Choose Local LLM Hardware by Workload, Not GPU NamesComprehensive guide to choosing hardware for Local LLMs. Summarizing pros and cons across Apple Silicon Mac, NVIDIA CUDA, and AMD ROCm with VRAM, memory bandwidth, specs, and Thailand market planning prices.Serving frameworks: choose from workload, hardware, and model filesCompare popular LLM serving frameworks: vLLM, SGLang, llama.cpp, MLX-LM, and Ollama by strengths, limits, hardware fit, and supported file artifacts like Safetensors and GGUF.Choose Local LLMs by work: Thai, chatbot, agent, coding, workflowSelect local LLMs for Thai, chatbots, agents, coding, and structured workflows by workload profile instead of ungrounded rankings.Speculative decoding and attention backends: faster only when the conditions fitUnderstand speculative decoding, acceptance rate, and attention backends for local LLMs, including when they improve or hurt performance.VRAM Management for Large MoE Models on Multi-GPU ClustersDeep-dive case study on VRAM capacity planning and multi-GPU serving for large MoE models such as DeepSeek-V3 and DeepSeek-R1 using EP, TP, PP, and MLA.EXL3 is ExLlamaV3's trellis code with a target average bitrateEXL3 is ExLlamaV3's weight format, a practical form of QTIP. It uses a trellis after a Hadamard rotation and a target average bitrate. This guide covers the mul1 codebook, the hq flag, and the 2.9 bpw DeepSeek pack on two DGX Sparks.TensorFold is the Zig server that serves models on Metal and CUDATensorFold serves models from a Zig binary on Apple Silicon and NVIDIA GPUs. The binary reads the checkpoint and runs the kernels itself. This guide covers exact decoding, the release model table, and the forked DeepSeek recipe on two DGX Sparks.