LOCAL LLM FIELD GUIDE

Speculative decoding and attention backends: faster only when the conditions fit

Speculative decoding pairs a smart target model with a fast draft model (or method) to propose candidate tokens ahead, allowing the target model to verify them in parallel. It accelerates generation significantly when drafted tokens are accepted at a high rate. Attention Backends are hardware-optimized math algorithms that compute token relationships inside fast GPU cache without getting bottlenecked by slow memory transfers.

VERIFY BEFORE YOU TRUST

Drafting helps only when the target accepts the drafted tokens

Measure baseline: TTFT, ITL, output tok/s
Enable speculation and inspect acceptance rate/length
Compare p95 and cost on identical traffic

Thai explanatory diagramThai-language diagram. The English article text provides the matching terminology and caveats.

USE IT ON REAL WORK

Enable a feature, then measure in order

Speculative decoding lets a draft propose tokens and the target verify them. It helps when enough drafted tokens are accepted to repay the extra work.

An attention backend is a kernel and memory-access path. Compatibility depends on model, GPU, runtime release, context, and KV dtype.

Signal	Good sign	Stop sign
Acceptance rate or length	Drafted tokens are accepted in runs	Low acceptance adds overhead
TTFT	No user-visible regression	First token is slower than baseline
ITL and output tok/s	Decode improves on the same workload	Only short prompts become faster
p95 and peak memory	Within the SLO and headroom	Worse tail latency or near OOM

Benchmark sequence

Measure a baseline without speculative decoding
Enable a method supported by the runtime and model
Inspect acceptance, TTFT, ITL, tok/s, p95, and peak memory
Use the same prompts, context, and concurrency
Keep the feature only when the improvement persists

Record results for comparison

method = MTP
working_context = 32768
concurrency = 4
measure = TTFT, ITL, output_tok_s, p95, peak_memory

How speculative decoding works

Standard LLM inference generates tokens autoregressively one by one, sweeping through all model weights on every step.

Speculative Decoding transforms this workflow into a two-stage pipeline:

Drafting Ahead: A smaller Draft Model, an integrated Multi-Token Prediction (MTP) head, or an n-gram lookup proposes 3-5 candidate tokens in rapid succession.
Parallel Verification: The large Target Model verifies all candidate tokens simultaneously in a single parallel execution pass (GEMM).
Speed Multiplication: When the target model accepts 3 drafted tokens, the system outputs 3 tokens in the time of a single step, boosting decode speed by 1.5x - 2.5x!

Key Rule: Always measure the Acceptance Rate and Acceptance Length alongside latency. If the acceptance rate is low, drafting overhead makes total generation slower than standard inference.

Do not read average speed alone

While speculative decoding can boost aggregate output throughput on predictable text, performance depends heavily on prompt complexity and sampling parameters:

Predictable Text (High Acceptance): Highly structured code, formal prose, and boilerplate templates yield high acceptance rates and significant speedups.
Complex Reasoning (Low Acceptance): Difficult mathematical derivations, creative open-ended writing, or high-temperature sampling cause frequent draft rejections, neutralizing performance gains.
Metrics to Monitor: Inspect Time to First Token (TTFT), Inter-Token Latency (ITL), and tail latencies (p95/p99) on identical traffic to ensure speculation does not introduce user-visible stuttering.

Attention backends

Attention is the core Transformer mechanism connecting relationships across tokens, but it demands quadratic compute and memory bandwidth as context grows.

Attention Backends optimize this computation at the hardware kernel level:

FlashAttention and FlashInfer: Restructure attention math into tiled blocks computed directly within ultra-fast on-chip GPU SRAM, drastically minimizing slow roundtrips to main VRAM. This accelerates long-context prefill and saves memory.
PagedAttention: Partitions KV Cache memory into non-contiguous physical blocks like OS virtual memory paging, preventing memory fragmentation.
Compatibility Matrix: Attention backends are not universal toggles. Compatibility depends strictly on GPU architecture (CUDA / ROCm / Metal), KV data types (FP16, BF16, FP8), and serving framework releases.

How to select

Structured implementation workflow:

Start with the Recommended Backend: Select the default attention backend recommended by your runtime that natively supports your chosen KV cache quantization.
Measure the Baseline: Benchmark TTFT, decode tok/s, and peak memory without speculation under standard traffic.
Validate Speculative Decoding: Enable speculation only on officially supported model/runtime pairings, testing against real production prompt distributions.
Disable Under Low Returns: If speculation yields low acceptance, worsens p95 tail latency, or pushes memory close to OOM limits, disable the feature to preserve system stability.

Sources

Try your configuration

Put the intended model, quantization, context, framework, and hardware into the calculator, then validate with a production-like benchmark.

Open calculator