Choose Local LLM Hardware by Workload, Not GPU Names

Mac Apple Silicon: everyday work and local LLMs

The Unified Memory Architecture (UMA) of Mac Apple Silicon (M2/M3/M4 Series) is a game-changer for local LLM workflows by sharing a single RAM pool across CPU, GPU, and Neural Engine!

• Strengths: Configurable with 64 GB, 96 GB, 128 GB, or 192 GB RAM to run 70B or 120B models (via GGUF Q4 or MLX) smoothly on a single desktop without complex multi-GPU setups, while doubling as a general workstation for coding, media, and daily tasks.
• Limitations: UMA RAM is shared with macOS and other apps, non-upgradeable post-purchase, and memory bandwidth (e.g. M4 Max at 410-546 GB/s, M3 Ultra at 819 GB/s) remains below high-end HBM3 GPU server clusters (2,000 - 3,000+ GB/s). Thus, Macs are ideal personal workstations or dev machines, but not primary high-concurrency production serving nodes.

Mac and vLLM: available, but not CUDA parity

The developer community created vLLM-Metal, a hardware plugin leveraging MLX backend to expose OpenAI-compatible HTTP serving on Mac hardware.

It is inaccurate to say Macs cannot run vLLM. However, feature support depends on model architecture, and server-grade features (cross-card Tensor Parallelism, certain speculative decoding methods, LoRA serving, VLMs, complex MoEs) remain unsupported or unvalidated.

For production fleet scaling, do not design architecture around CUDA vLLM features expecting identical performance or feature parity on Mac Metal!

NVIDIA: the broadest server-inference ecosystem

NVIDIA CUDA remains the undisputed gold standard for server inference, featuring the industry's most mature software ecosystem.

• Software Stack Maturity: Frameworks like vLLM, SGLang, TensorRT-LLM, FlashAttention-3, ModelOpt, and profiling tools are developed and optimized for CUDA first.
• Hardware Generations & Precision Paths:
• Blackwell (RTX 5090, B200, DGX GB300): Features FP4 Tensor Cores supporting NVFP4 / modelopt_fp4 and native FP4 execution for breakthrough decode speeds.
• Ada Lovelace & Hopper (RTX 4090, L40S, H100): Supports FP8 (E4M3 / E5M2) for weights and KV Cache, halving memory footprint with zero speed penalty.
• Interconnect: High-end datacenter GPUs feature NVLink / NVSwitch providing 900 - 1,800 GB/s inter-card bandwidth, making multi-GPU Tensor Parallelism (TP=2, TP=4, TP=8) seamless without PCIe bottlenecks.

AMD: strong memory value, but validate the support matrix

AMD ROCm / HIP hardware offers attractive VRAM-per-dollar ratios, such as Instinct MI300X (192 GB HBM3) or consumer Radeon RDNA GPUs.

Key considerations for AMD deployment:

• Software & Kernel Support Matrix: While vLLM-ROCm, SGLang, and llama.cpp support on AMD has improved dramatically, kernel availability (FlashAttention, AWQ loaders, FP8 KV, Speculative Decoding) depends heavily on GPU generation, ROCm drivers, and runtime releases.
• Instinct vs Radeon: Enterprise Instinct MI300X/MI350 GPUs feature tight enterprise tuning for FP8, FP8 KV, AWQ, and AMD Quark (including MXFP4). Consumer Radeon RDNA GPUs may lack certain CUDA-equivalent kernel optimizations.

Recommendation: Do not assume AMD cannot run, nor that identical quantization names achieve equal speed across all AMD generations. Always inspect calculator-supported options for your target GPU.

Consumer, workstation, datacenter

Hardware is categorized into 3 main tiers based on budget and workload:

• Consumer GPUs (RTX 5090 / 4090 / 3090): Ideal for local development, personal workstations, and budget single-GPU inference. Offers high memory bandwidth (1,000+ GB/s) at accessible pricing, but limited by VRAM caps (24-32 GB) and lack of NVLink in newer models.
• Workstation GPUs (RTX 6000 Ada, Radeon PRO): Expands VRAM to 48 GB+, includes ECC memory for error protection, and features server-friendly power/form-factor designs.
• Datacenter GPUs (H100, H200, B200, DGX Station GB300, MI300X): Designed for 24/7 enterprise fleets requiring HBM3/HBM3e memory bandwidth (3,000+ GB/s), liquid cooling, and NVSwitch fabric for seamless multi-card Tensor Parallelism.

(The table below lists hardware and prices from the calculator, representing Thailand planning estimates rather than commercial quotes).

Questions before buying

Five essential questions before buying or renting hardware:

1. What model size (B) and quantization format will you run?
2. What working context length is required (e.g. 4K, 32K, 128K)?
3. How many concurrent users must be served (Batch Size)?
4. Is the primary goal low personal latency (TTFT) or high aggregate throughput?
5. What operating environment expertise does your team have (macOS, Linux CUDA, ROCm)?

Plug these parameters into the LLM VRAM Calculator to verify Peak VRAM and speed before purchase!

Sources

• Apple Mac Studio technical specifications
• vLLM-Metal
• NVIDIA TensorRT precision support
• AMD ROCm vLLM optimization
• MLX-LM
• llama.cpp