Skip to Content
KV CachingDurable disk tier

Durable disk tier

The disk tier stores verified prompt state below the in-process prefix cache. It lets reusable prefixes survive process restarts and extends retention beyond the resident cache. It is disabled by default and must be enabled deliberately.

Use it when repeated prompts are valuable across restarts or do not fit in the resident retention budget. Keep it off when local storage latency, capacity, or data-retention policy makes durable prompt state undesirable.

Enable the disk tier

[cache] prefix_cache = "on" kv_cache_disk = "32GiB" kv_cache_disk_dir = "/var/lib/skippy/kv-cache" kv_cache_min_free = "16GiB"

Or override one run from the CLI:

skippy serve \ --model Qwen3-0.6B-Q4_K_M \ --kv-cache-disk 32GiB \ --kv-cache-disk-dir /var/lib/skippy/kv-cache \ --kv-cache-min-free 16GiB

Use auto instead of a fixed size for an automatically bounded disk budget, or off to disable the tier.

How the tiers interact

Skippy checks compatible resident state first, then optional host-RAM and disk state. Restored disk entries are verified before use. New reusable state is recorded according to the prefix-cache policy and retained within the configured soft budgets and free-space reserve.

Failure policy

Invalid cache configuration fails startup so a typo cannot silently change the operator’s retention policy. Once a valid configuration is running, storage unavailability fails open for inference: Skippy logs the problem and serves with cold prefill rather than making the model unavailable.

prefix_cache_ram enables an optional host-RAM tier and requires an active disk tier:

[cache] prefix_cache_ram = "4GiB" kv_cache_disk = "32GiB"

Legacy SKIPPY_KV_CACHE=off and SKIPPY_PREFIX_CACHE=off remain emergency kill switches. Unset them before explicitly enabling prefix caching.

Disk-tier reference

SettingValuesPurpose
kv_cache_diskoff, auto, or IEC sizeEnable and bound durable storage
kv_cache_disk_dirLocal directoryChoose the cache root
kv_cache_min_freeIEC sizeReserve free space on the volume
prefix_cache_ramoff or IEC sizeAdd the host-RAM tier above disk

Invalid startup configuration is fatal. Runtime storage failure is not: inference continues with cold prefill and Skippy reports the cache error.