KNOWLEDGE BASE

Local LLM Guides: hardware, models, quantization, and serving

A practical guide library for choosing local LLM hardware, models, quantization, KV cache, serving, and performance paths.

Quantization: Where VRAM is Reduced and What Quality Trades AwayIn-depth guide to 4-bit and 8-bit quantization for weights and KV cache, summarizing QAT, MXFP4, AWQ, GPTQ, and GGUF with VRAM calculations, quality trade-offs, and hallucination risks.Why LLM tok/s Hits Memory Bandwidth BottlenecksDeep-dive into the Roofline Model for Local LLMs: why Decode is memory bandwidth bound, why Prefill is compute (FLOPS) bound, distinguishing TTFT, TPOT, ITL, and reading calculator results like a pro.Choose Local LLM Hardware by Workload, Not GPU NamesComprehensive guide to choosing hardware for Local LLMs. Summarizing pros and cons across Apple Silicon Mac, NVIDIA CUDA, and AMD ROCm with VRAM, memory bandwidth, specs, and Thailand market planning prices.Serving frameworks: choose from workload, hardware, and model filesCompare popular LLM serving frameworks: vLLM, SGLang, llama.cpp, MLX-LM, and Ollama by strengths, limits, hardware fit, and supported file artifacts like Safetensors and GGUF.Choose Local LLMs by work: Thai, chatbot, agent, coding, workflowSelect local LLMs for Thai, chatbots, agents, coding, and structured workflows by workload profile instead of ungrounded rankings.Speculative decoding and attention backends: faster only when the conditions fitUnderstand speculative decoding, acceptance rate, and attention backends for local LLMs, including when they improve or hurt performance.VRAM Management for Large MoE Models on Multi-GPU ClustersDeep-dive case study on VRAM capacity planning and multi-GPU serving for large MoE models such as DeepSeek-V3 and DeepSeek-R1 using EP, TP, PP, and MLA.