# Choose Local LLM Hardware by Workload, Not GPU Names https://llm-cal.p7z.pw/en/guides/hardware/ The popular question 'Which GPU should I buy for LLMs?' cannot be answered by GPU model names alone. Selecting the right hardware requires defining your target model size, serving framework, required concurrency, and budget. Mac Apple Silicon excels as a personal single-machine workstation for large models, NVIDIA is the gold standard for high-throughput production servers, and AMD offers attractive VRAM-per-dollar value while requiring careful ROCm support matrix validation. ## Mac Apple Silicon: everyday work and local LLMs The Unified Memory Architecture (UMA) of Mac Apple Silicon (M2/M3/M4 Series) is a game-changer for local LLM workflows by sharing a single RAM pool across CPU, GPU, and Neural Engine! - Strengths: Configurable with 64 GB, 96 GB, 128 GB, or 192 GB RAM to run 70B or 120B models (via GGUF Q4 or MLX) smoothly on a single desktop without complex multi-GPU setups, while doubling as a general workstation for coding, media, and daily tasks. - Limitations: UMA RAM is shared with macOS and other apps, non-upgradeable post-purchase, and memory bandwidth (e.g. M4 Max at 410-546 GB/s, M3 Ultra at 819 GB/s) remains below high-end HBM3 GPU server clusters (2,000 - 3,000+ GB/s). Thus, Macs are ideal personal workstations or dev machines, but not primary high-concurrency production serving nodes. ## Mac and vLLM: available, but not CUDA parity The developer community created vLLM-Metal, a hardware plugin leveraging MLX backend to expose OpenAI-compatible HTTP serving on Mac hardware. It is inaccurate to say Macs cannot run vLLM. However, feature support depends on model architecture, and server-grade features (cross-card Tensor Parallelism, certain speculative decoding methods, LoRA serving, VLMs, complex MoEs) remain unsupported or unvalidated. For production fleet scaling, do not design architecture around CUDA vLLM features expecting identical performance or feature parity on Mac Metal! ## NVIDIA: the broadest server-inference ecosystem NVIDIA CUDA remains the undisputed gold standard for server inference, featuring the industry's most mature software ecosystem. - Software Stack Maturity: Frameworks like vLLM, SGLang, TensorRT-LLM, FlashAttention-3, ModelOpt, and profiling tools are developed and optimized for CUDA first. - Hardware Generations & Precision Paths: - Blackwell (RTX 5090, B200, DGX GB300): Features FP4 Tensor Cores supporting NVFP4 / modelopt_fp4 and native FP4 execution for breakthrough decode speeds. - Ada Lovelace & Hopper (RTX 4090, L40S, H100): Supports FP8 (E4M3 / E5M2) for weights and KV Cache, halving memory footprint with zero speed penalty. - Interconnect: High-end datacenter GPUs feature NVLink / NVSwitch providing 900 - 1,800 GB/s inter-card bandwidth, making multi-GPU Tensor Parallelism (TP=2, TP=4, TP=8) seamless without PCIe bottlenecks. ## AMD: strong memory value, but validate the support matrix AMD ROCm / HIP hardware offers attractive VRAM-per-dollar ratios, such as Instinct MI300X (192 GB HBM3) or consumer Radeon RDNA GPUs. Key considerations for AMD deployment: - Software & Kernel Support Matrix: While vLLM-ROCm, SGLang, and llama.cpp support on AMD has improved dramatically, kernel availability (FlashAttention, AWQ loaders, FP8 KV, Speculative Decoding) depends heavily on GPU generation, ROCm drivers, and runtime releases. - Instinct vs Radeon: Enterprise Instinct MI300X/MI350 GPUs feature tight enterprise tuning for FP8, FP8 KV, AWQ, and AMD Quark (including MXFP4). Consumer Radeon RDNA GPUs may lack certain CUDA-equivalent kernel optimizations. Recommendation: Do not assume AMD cannot run, nor that identical quantization names achieve equal speed across all AMD generations. Always inspect calculator-supported options for your target GPU. ## Consumer, workstation, datacenter Hardware is categorized into 3 main tiers based on budget and workload: - Consumer GPUs (RTX 5090 / 4090 / 3090): Ideal for local development, personal workstations, and budget single-GPU inference. Offers high memory bandwidth (1,000+ GB/s) at accessible pricing, but limited by VRAM caps (24-32 GB) and lack of NVLink in newer models. - Workstation GPUs (RTX 6000 Ada, Radeon PRO): Expands VRAM to 48 GB+, includes ECC memory for error protection, and features server-friendly power/form-factor designs. - Datacenter GPUs (H100, H200, B200, DGX Station GB300, MI300X): Designed for 24/7 enterprise fleets requiring HBM3/HBM3e memory bandwidth (3,000+ GB/s), liquid cooling, and NVSwitch fabric for seamless multi-card Tensor Parallelism. (The table below lists hardware and prices from the calculator, representing Thailand planning estimates rather than commercial quotes). ## Questions before buying Five essential questions before buying or renting hardware: 1. What model size (B) and quantization format will you run? 2. What working context length is required (e.g. 4K, 32K, 128K)? 3. How many concurrent users must be served (Batch Size)? 4. Is the primary goal low personal latency (TTFT) or high aggregate throughput? 5. What operating environment expertise does your team have (macOS, Linux CUDA, ROCm)? Plug these parameters into the LLM VRAM Calculator to verify Peak VRAM and speed before purchase! ## Choose hardware from the deployment path Start with the runtime and artifact you must use. They determine the accelerator path before price. Then compare reserved memory, bandwidth, and operating effort. More VRAM does not automatically make a workflow easier. Compare the model, context, and concurrent requests with a real path to run them. | Option | Start with | Verify | | --- | --- | --- | | Mac Apple Silicon | MLX or GGUF on Metal | Unified memory, conversion path, local workflow | | NVIDIA | A supported CUDA runtime and kernel | VRAM, bandwidth, driver, serving topology | | AMD | ROCm and HIP support in the runtime | GPU model, kernel path, artifact support | ### Questions before buying or renting - Which model and quantization must run - How many working-context tokens and concurrent sequences - Whether TTFT, per-user tok/s, or aggregate throughput matters - How much Linux, CUDA, or ROCm operation the team can support - Whether the plan is one device, replicas, or tensor parallel ### Inputs to place in the calculator ``` hardware = framework = model = working_context = batch = ``` ## Frequently asked questions ### Can an RTX 5090 run a 70B model? An RTX 5090 has 32 GB of VRAM, so a 70B model at full precision is out of reach because the weights alone exceed 100 GB. A 4-bit checkpoint squeezes weights to about 35 GB, which is still tight once KV cache and runtime overhead join in. Put the full weights plus KV plus overhead budget into the calculator before deciding. ### How is Mac unified memory different from VRAM? Mac unified memory is one pool shared by the CPU, GPU, and Neural Engine, configurable up to about 192 GB, so a single workstation can load large models through MLX or GGUF. The tradeoff is far lower bandwidth than datacenter HBM cards, and memory cannot be upgraded later. Treat it as a personal workstation path, not a high-concurrency production server. ## Sources - [Apple Mac Studio technical specifications](https://www.apple.com/mac-studio/specs/) - [vLLM-Metal](https://docs.vllm.ai/projects/vllm-metal/en/latest/) - [NVIDIA TensorRT precision support](https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/capabilities.html) - [AMD ROCm vLLM optimization](https://rocm.docs.amd.com/en/develop/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html) - [MLX-LM](https://github.com/ml-explore/mlx-lm) - [llama.cpp](https://github.com/ggml-org/llama.cpp)