Skip to Content
KV CachingPrompt and KV caching

Prompt and KV caching

The live KV cache stores attention state for generations currently in progress. The prefix cache retains reusable model state after a prefill so a later request with the same token prefix can skip repeated work.

Use prefix caching for stable system prompts, tool schemas, long documents, and multi-turn conversations. It improves time to first token when requests share enough leading tokens to make restoration cheaper than recomputation.

Applications continue to send the complete prompt or conversation. Skippy does not perform fuzzy or semantic prompt matching. Reuse ends at the first token that differs.

Default policy

prefix_cache = "auto" follows the admitted model-family policy. Automatic payload selection uses the loaded graph and native memory exporters so hybrid or recurrent models retain all required state. If no complete representation is available, prefix reuse is disabled while inference remains available.

[cache] prefix_cache = "auto" prefix_cache_budget = "auto" prefix_cache_ram = "off" kv_cache_disk = "off"

Control prompt reuse

[cache] prefix_cache = "on" # auto, on, off, or record prefix_cache_budget = "8GiB" prefix_cache_max_entries = 256 prefix_cache_min_tokens = 512 prefix_cache_stride_tokens = 128 prefix_cache_record_limit = 4

Use record to populate state without serving hits. off disables lookup and recording. Exact-state snapshots are indivisible, so one snapshot may exceed a configured soft retention target.

How prefix matching works

Skippy renders and tokenizes the full request, finds the longest compatible retained token prefix, restores its admitted model state, then prefills only the unmatched suffix. Cache compatibility includes the model and runtime state needed to resume inference, not only the prompt text.

A stable application cache key creates a deterministic namespace for related requests, but it does not replace prefix comparison. Requests must still carry the complete content and matching tokens. See the request guide for examples.

Choose live KV precision

[kv] kv_cache_type_k = "q8_0" kv_cache_type_v = "q8_0" kv_cache_offload = true flash_attention = "auto"

Supported explicit live K/V values are f16, q8_0, and q4_0. Quantized V requires compatible Flash Attention and model geometry. An incompatible explicit choice fails rather than silently changing precision.

Observe cache behavior

OpenAI responses report reused tokens as usage.prompt_tokens_details.cached_tokens. Skippy telemetry also reports hit status, matched prefix tokens, suffix prefill tokens, and restore source.

Cache reference

SettingPurpose
prefix_cacheSelect auto, on, off, or record-only mode
prefix_cache_budgetSet the resident retention target
prefix_cache_max_entriesLimit retained prefix entries
prefix_cache_min_tokensAvoid recording prefixes below a token threshold
prefix_cache_stride_tokensSet shared-prefix checkpoint spacing
kv_cache_type_k, kv_cache_type_vChoose live key/value precision
kv_cache_offloadPlace supported live KV work on the backend
flash_attentionSelect the compatible attention implementation

For cache namespaces in OpenAI-compatible and Anthropic-compatible requests, see Deterministic caching.