Serving frameworks: choose from workload, hardware, and model files

Start with the artifact, not the framework name

In local LLM engineering, model files are not interchangeable. Each server runtime is tailored to execute specific artifact types:

• Safetensors: The standard Hugging Face format for GPU servers and the direct path for vLLM and SGLang, providing fast memory-mapped loading into VRAM and high security.
• Pre-quantized Formats (AWQ / GPTQ / FP8 / compressed-tensors / native low-bit): Pre-compressed weights requiring dedicated loaders and accelerated compute kernels matching the GPU architecture.
• GGUF: A binary container format from GGML / llama.cpp designed for local PCs and Macs, bundling weights, specs, and tokenizer metadata into a single file. The primary native path for llama.cpp and Ollama.
• MLX Model: Formatted specifically for Apple Silicon to maximize Unified Memory throughput.
• Ollama: A model-management layer with intuitive CLI and APIs capable of importing Safetensors model folders, Safetensors adapters, GGUF models, and GGUF adapters through Modelfiles for supported architectures.

vLLM: broad features for GPU servers

vLLM is the industry heavyweight for high-concurrency multi-user GPU server inference:

• Flagship Features: Includes PagedAttention (managing KV Cache memory like OS virtual memory pages to eliminate memory waste), Continuous Batching, Chunked Prefill, Prefix Caching, OpenAI-compatible API, Speculative Decoding, Multi-GPU Parallelism (TP / PP), and Disaggregated Prefill-Decode (PD).
• Broad Quantization Ecosystem: Natively supports Hugging Face formats including FP8, MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ, GGUF, compressed-tensors, and NVIDIA ModelOpt.
• Important Caveats: Server deployment requires matching CUDA/ROCm versions, drivers, kernels, and GPU generations. GGUF in vLLM remains experimental and under-optimized, so it is not the default for a performance-sensitive or feature-complete server. vLLM-Metal is separate and is not CUDA parity.

SGLang: explicit prefix reuse and serving topology

SGLang is a premier production serving framework engineered for ultra-low latency, high throughput, and complex agentic workflows.

• Core Advantages: Features RadixAttention (intelligently caching and reusing shared system prompts and multi-turn conversation history across requests without redundant computation), Multi-GPU Parallelism, Structured Outputs (guaranteeing strict JSON and regex schema conformance with minimal latency), Tool Calling, Speculative Decoding, and Prefill-Decode (PD) Disaggregation.
• Supported File Paths: Directly loads from Hugging Face repos or local checkpoint directories (starting with Safetensors and falling back to PyTorch bin), reading pre-quantized AWQ, GPTQ, and FP8 configurations with online quantization options.
• Important Caveats: Configuration, kernel backend, and topology must be validated against the exact release and hardware. Do not choose SGLang merely because a GGUF file exists start with a supported Hugging Face checkpoint.

llama.cpp: the direct GGUF path with broad backends

llama.cpp is the ultimate portable engine and the universal foundation for executing GGUF files across virtually any hardware:

• Cross-Platform Flexibility: Runs seamlessly on standard CPUs, Apple Silicon (Metal), NVIDIA GPUs (CUDA), AMD GPUs (ROCm/HIP), Intel GPUs (SYCL), and Vulkan.
• Fully Featured Server: The included llama-server provides an OpenAI-compatible API, Continuous Batching, Multi-user Parallel Decoding, Prometheus metrics monitoring, Schema-constrained JSON, Tool Calling, and Speculative Decoding out of the box.
• Strengths: Compact C/C++ binary, lightweight memory footprint, and broadest hardware backend compatibility.
• Limitations: Requires models converted into GGUF format with valid architectures and chat templates. Safetensors is not the default direct-load artifact. Large multi-node cluster fleets require manual scaling and lifecycle orchestration compared to cluster-native engines.

MLX-LM: native Apple Silicon, not a production server

MLX-LM is developed by Apple's machine learning research team specifically for Mac Apple Silicon (M-Series) chips.

• Native Mac Power: Loads MLX-compatible repositories or converted local paths directly into Mac Unified Memory, supporting fast inference, 4-bit quantization, LoRA fine-tuning, and speculative decoding. Includes an HTTP server with OpenAI-compatible chat endpoints.
• Ideal Use Case: Frictionless personal local development and AI research on an everyday Mac workstation.
• Engineering Limitation: MLX-LM documentation clearly states that the MLX-LM server provides only basic security checks and is not recommended for production serving. MLX models and GGUF are different artifacts for GGUF files on Mac, choose llama.cpp on Metal instead.

Ollama: fast local API and model management

Ollama is the easiest bridge for developers and beginners to deploy local LLMs within minutes:

• Simplicity and Speed: One-click installation, single-command model downloads (ollama run ...), and an immediate local REST API for applications without manual compiler configuration.
• Flexible Model Ingestion: Supports Modelfiles to create, customize, and manage models. Imports Safetensors model folders, Safetensors adapters, GGUF models, and GGUF adapters for supported architectures, with built-in quantization from FP16/FP32 sources.
• Important Caveats: Ollama's simplicity does not replace the capacity planning, kernel selection, detailed telemetry, or distributed cluster orchestration provided by vLLM and SGLang. Always validate peak VRAM and latency under production traffic before deployment.

Before production

Essential checklist before deploying to production:

1. Sizing and Limits: Set max-model-len and concurrency (max-num-seqs) strictly within your calculated VRAM budget.
2. Network Security: Always bind the service to 127.0.0.1 (localhost) first, routing external traffic through a reverse proxy or API gateway with authentication.
3. Record Blueprint: Log the exact artifact hash, framework release, GPU driver version, and runtime flags.
4. Empirical Testing: Measure TTFT, Inter-Token Latency (ITL), Output Throughput (tok/s), p95 tail latencies, peak memory, and answer quality on real workload prompts. The fastest framework on a short benchmark may not be the most stable under real-world traffic.

Sources

• vLLM capabilities
• vLLM quantization
• SGLang quantization
• SGLang PD disaggregation
• llama.cpp server
• MLX-LM server
• Ollama model import