Prompt and KV caching
The live KV cache stores attention state for generations currently in progress. The prefix cache retains reusable model state after a prefill so a later request with the same token prefix can skip repeated work.
Use prefix caching for stable system prompts, tool schemas, long documents, and multi-turn conversations. It improves time to first token when requests share enough leading tokens to make restoration cheaper than recomputation.
Applications continue to send the complete prompt or conversation. Skippy does not perform fuzzy or semantic prompt matching. Reuse ends at the first token that differs.
Default policy
prefix_cache = "auto" follows the admitted model-family policy. Automatic
payload selection uses the loaded graph and native memory exporters so hybrid
or recurrent models retain all required state. If no complete representation
is available, prefix reuse is disabled while inference remains available.
[cache]
prefix_cache = "auto"
prefix_cache_budget = "auto"
prefix_cache_ram = "off"
kv_cache_disk = "off"Control prompt reuse
[cache]
prefix_cache = "on" # auto, on, off, or record
prefix_cache_budget = "8GiB"
prefix_cache_max_entries = 256
prefix_cache_min_tokens = 512
prefix_cache_stride_tokens = 128
prefix_cache_record_limit = 4Use record to populate state without serving hits. off disables lookup and
recording. Exact-state snapshots are indivisible, so one snapshot may exceed a
configured soft retention target.
How prefix matching works
Skippy renders and tokenizes the full request, finds the longest compatible retained token prefix, restores its admitted model state, then prefills only the unmatched suffix. Cache compatibility includes the model and runtime state needed to resume inference, not only the prompt text.
A stable application cache key creates a deterministic namespace for related requests, but it does not replace prefix comparison. Requests must still carry the complete content and matching tokens. See the request guide for examples.
Choose live KV precision
[kv]
kv_cache_type_k = "q8_0"
kv_cache_type_v = "q8_0"
kv_cache_offload = true
flash_attention = "auto"Supported explicit live K/V values are f16, q8_0, and q4_0. Quantized V
requires compatible Flash Attention and model geometry. An incompatible
explicit choice fails rather than silently changing precision.
Observe cache behavior
OpenAI responses report reused tokens as
usage.prompt_tokens_details.cached_tokens. Skippy telemetry also reports hit
status, matched prefix tokens, suffix prefill tokens, and restore source.
Cache reference
| Setting | Purpose |
|---|---|
prefix_cache | Select auto, on, off, or record-only mode |
prefix_cache_budget | Set the resident retention target |
prefix_cache_max_entries | Limit retained prefix entries |
prefix_cache_min_tokens | Avoid recording prefixes below a token threshold |
prefix_cache_stride_tokens | Set shared-prefix checkpoint spacing |
kv_cache_type_k, kv_cache_type_v | Choose live key/value precision |
kv_cache_offload | Place supported live KV work on the backend |
flash_attention | Select the compatible attention implementation |
For cache namespaces in OpenAI-compatible and Anthropic-compatible requests, see Deterministic caching.