LOCAL LLM FIELD GUIDE

Speculative decoding and attention backends: faster only when the conditions fit

Speculative decoding drafts tokens ahead with a draft model or method, then lets the target model verify them in parallel. It helps when acceptance is strong enough to repay overhead. Attention backends reduce attention cost but depend on GPU, runtime, context, and kernel support.

VERIFY BEFORE YOU TRUST

Drafting helps only when the target accepts the drafted tokens

Measure baseline: TTFT, ITL, output tok/s
Enable speculation and inspect acceptance rate/length
Compare p95 and cost on identical traffic

Thai explanatory diagramThai-language diagram. The English article text provides the matching terminology and caveats.

USE IT ON REAL WORK

Enable a feature, then measure in order

Speculative decoding lets a draft propose tokens and the target verify them. It helps when enough drafted tokens are accepted to repay the extra work.

An attention backend is a kernel and memory-access path. Compatibility depends on model, GPU, runtime release, context, and KV dtype.

Signal	Good sign	Stop sign
Acceptance rate or length	Drafted tokens are accepted in runs	Low acceptance adds overhead
TTFT	No user-visible regression	First token is slower than baseline
ITL and output tok/s	Decode improves on the same workload	Only short prompts become faster
p95 and peak memory	Within the SLO and headroom	Worse tail latency or near OOM

Benchmark sequence

Measure a baseline without speculative decoding
Enable a method supported by the runtime and model
Inspect acceptance, TTFT, ITL, tok/s, p95, and peak memory
Use the same prompts, context, and concurrency
Keep the feature only when the improvement persists

Record results for comparison

method = MTP
working_context = 32768
concurrency = 4
measure = TTFT, ITL, output_tok_s, p95, peak_memory

How speculative decoding works

A draft model, MTP, or n-gram proposes tokens ahead. the target model verifies and accepts a run. With low acceptance, verification and draft management may not pay off, so measure acceptance length/rate together with latency.

Do not read average speed alone

Speculation can improve output throughput while TTFT, tail latency, sampling, tool calls, and prompt distribution change the outcome. Enable it only for clearly supported model/runtime combinations and compare with the baseline on identical traffic.

Attention backends

FlashAttention, FlashInfer, and newer backends are memory-access and kernel strategies for attention. Compatibility depends on CUDA, ROCm, or Metal. GPU architecture. context. KV dtype. and runtime release.not one universal toggle.

How to select

Start with the backend recommended by the runtime that supports your KV quantization. Benchmark prefill, decode, long context, and concurrency. disable a feature when it adds complexity without improving p95 or cost for the actual workload.

Sources

Try your configuration

Put the intended model, quantization, context, framework, and hardware into the calculator, then validate with a production-like benchmark.

Open calculator