LOCAL LLM FIELD GUIDES
Choose local LLMs with evidence
Choose hardware, models, quantization, KV cache, serving frameworks, and performance paths for local LLMs.
START HERE
Start from the question you need answered
Choose Local LLM Hardware by Workload, Not GPU NamesComprehensive guide to choosing hardware for Local LLMs. Summarizing pros and cons across Apple Silicon Mac, NVIDIA CUDA, and AMD ROCm with VRAM, memory bandwidth, specs, and Thailand market planning prices.Choose Local LLMs by work: Thai, chatbot, agent, coding, workflowSelect local LLMs for Thai, chatbots, agents, coding, and structured workflows by workload profile instead of ungrounded rankings.Quantization: Where VRAM is Reduced and What Quality Trades AwayIn-depth guide to 4-bit and 8-bit quantization for weights and KV cache, summarizing QAT, MXFP4, AWQ, GPTQ, and GGUF with VRAM calculations, quality trade-offs, and hallucination risks.Why LLM tok/s Hits Memory Bandwidth BottlenecksDeep-dive into the Roofline Model for Local LLMs: why Decode is memory bandwidth bound, why Prefill is compute (FLOPS) bound, distinguishing TTFT, TPOT, ITL, and reading calculator results like a pro.Serving frameworks: choose from workload, hardware, and model filesCompare popular LLM serving frameworks: vLLM, SGLang, llama.cpp, MLX-LM, and Ollama by strengths, limits, hardware fit, and supported file artifacts like Safetensors and GGUF.Speculative decoding and attention backends: faster only when the conditions fitUnderstand speculative decoding, acceptance rate, and attention backends for local LLMs, including when they improve or hurt performance.EXL3 is ExLlamaV3's trellis code with a target average bitrateEXL3 is ExLlamaV3's weight format, a practical form of QTIP. It uses a trellis after a Hadamard rotation and a target average bitrate. This guide covers the mul1 codebook, the hq flag, and the 2.9 bpw DeepSeek pack on two DGX Sparks.TensorFold is the Zig server that serves models on Metal and CUDATensorFold serves models from a Zig binary on Apple Silicon and NVIDIA GPUs. The binary reads the checkpoint and runs the kernels itself. This guide covers exact decoding, the release model table, and the forked DeepSeek recipe on two DGX Sparks.