LOCAL LLM FIELD GUIDE

Serving frameworks: choose by serving shape

No framework wins every job. Start with hardware and artifact, then choose among a local single-user flow, a simple API, a high-throughput server, or an NVIDIA-optimized production path.

DECISION TREE

Start with the request shape, not a framework name

For Mac local work, start with MLX or llama.cpp on Metal
For GGUF or a simple local API, start with llama.cpp or Ollama
For a GPU server that handles concurrent requests, consider vLLM or SGLang
For an NVIDIA-specific production path, evaluate TensorRT-LLM

Thai explanatory diagramThai-language diagram. The English article text provides the matching terminology and caveats.

USE IT ON REAL WORK

Choose a framework from request shape

A local runner focuses on loading a model for a few people. A server runtime manages batching, KV cache, concurrency, and observability.

Choose from the artifact you have, the hardware you run, and the people who operate the service. Do not start from the most popular framework name.

Framework	Start here when	You own
llama.cpp	You need `GGUF` on CPU, GPU, or Metal	Model file, backend flags, benchmark
Ollama	You want a simple local API and model management	Model lifecycle, memory limit, feature support
MLX	You work on Apple Silicon	Conversion, local API, memory headroom
vLLM / SGLang	A GPU server handles concurrent requests	Batching, KV, backend, rollout
TensorRT-LLM	The team supports an NVIDIA production path	Engine build, version compatibility, operations

Before opening a service

Bind to 127.0.0.1 until there is a reason to expose it
Set max-model-len and max-num-seqs from the capacity plan
Record the checkpoint, runtime release, and flags
Have a smoke test, load test, and rollback path

Shape of a vLLM server command

vllm serve <checkpoint> \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len <tokens> \
  --max-num-seqs <sequences>

Start from the runtime path

Mac/Apple Silicon: start with MLX or llama.cpp/Metal. GGUF local use: llama.cpp or Ollama. GPU servers needing batching, cache, and metrics: vLLM or SGLang. For NVIDIA production paths, evaluate TensorRT-LLM only where the team can support it.

vLLM and SGLang

They suit GPU serving needing continuous batching, KV management, and concurrency. vLLM offers a widely used engine/API. SGLang adds serving composition and prefill/decode separation paths. Verify model, quantization, and backend support in the exact release.

llama.cpp, MLX, and Ollama

llama.cpp provides a broad GGUF path across Metal, CPU, and GPU backends. MLX is native to Apple Silicon for conversion, quantization, and local work. Ollama lowers local API and model-management friction, but does not replace production capacity planning.

Avoid a battle card

Use a decision tree: Do you need concurrency? GGUF or safetensors? Mac or GPU server? Which cache, speculation, and attention backend? Who operates observability and rollout? Benchmark the chosen path on real traffic.

Sources

Try your configuration

Put the intended model, quantization, context, framework, and hardware into the calculator, then validate with a production-like benchmark.

Open calculator