LOCAL LLM FIELD GUIDE

Why LLM tok/s often hits memory bandwidth

During decode, the system repeatedly reads the weights and state needed to produce one next token. That often makes data movement.not FLOPS.the limit. Framework, kernel, attention, and speculation still modify the observed result.

ROOFLINE PERFORMANCE MODEL

This equation is a planning model, not a benchmark

Think of tok/s as available data movement, adjusted by the software path and bytes that must be read.

tok/s≈Bandwidth × Framework × Quant-Kernel × Attention × Spec-Decode÷Bytes Read per Token

Thai explanatory diagramThai-language diagram. The English article text provides the matching terminology and caveats.

USE IT ON REAL WORK

Read tok/s for the thing you measured

Decode emits one output token at a time, so it often rereads weights and KV cache. Prefill accepts many prompt tokens together and has a different bottleneck.

The calculator helps rank options. Confirm latency, capacity, and cost decisions with an equivalent benchmark.

Metric	Answers	Does not replace
TTFT	How long a user waits for the first token	Generation speed
ITL	Time between output tokens	Prefill time
output tok/s	Decode speed of one request	Aggregate throughput
peak memory	Headroom before OOM	Answer quality

Keep these fixed when comparing

The same model, quantization, and runtime
The same working context and batch
Comparable prompts, sampling, and power mode
Measure TTFT, ITL, tok/s, and peak memory separately

Roofline form used by the calculator

readPerStepGB = (activeWeightsBytes + N * kvReadPerStreamBytes) / 1e9
tok/s_total = bandwidthGBps * frameworkEff * quantKernelEff
              * attentionEff * specMult(N) * N / readPerStepGB

Decode is not prefill

Prefill processes many prompt tokens together and can be more compute-heavy. Decode emits one token at a time, rereading weights and often becoming bandwidth-sensitive. one tok/s number is not all latency.

Bytes per token is the core

Lower weight precision can reduce bytes read per token, but only with an effective kernel and access pattern. Long context also increases attention and KV work, reducing throughput.

Read an estimate as a model

The calculator starts from hardware bandwidth and adjusts for framework, quant kernel, attention, and speculation. It is a planning estimate, not a benchmark guarantee. request mix, clock, and power matter.

Validate the workload

Measure TTFT, inter-token latency, output tok/s, concurrency, and peak memory independently. Match prompt, context, batching, and sampling to production to see whether the bottleneck changes.

Sources

Try your configuration

Put the intended model, quantization, context, framework, and hardware into the calculator, then validate with a production-like benchmark.

Open calculator