Skip to Content
Configuration ReferenceSettings files

Serving configuration

Serving configuration defines how Skippy loads a model, allocates resources, schedules requests, exposes APIs, and records reusable state. Every supported operator control is available through skippy serve --help.

Use CLI options for temporary overrides and experiments. Use environment variables for process-level deployment settings. Use --settings serve.toml for a complete, reviewable serving configuration.

Create a settings file

version = 1 [model] model = "Qwen3-0.6B-Q4_K_M" mmap = true [execution] n_gpu_layers = -1 threads = 4 batch_size = 512 [kv] ctx_size = 8192 kv_cache_type_k = "f16" kv_cache_type_v = "f16" [cache] prefix_cache = "auto" kv_cache_disk = "off" [scheduling] generation_concurrency = 2 continuous_batching = true [api] bind_addr = "127.0.0.1:9337" default_max_tokens = 2048

Run with an override and inspect the resolved launch:

skippy serve --settings serve.toml --temperature 0.2 skippy serve --settings serve.toml --print-effective-config

How settings resolve

Settings resolve in this order:

  1. Explicit command-line options.
  2. SKIPPY_SERVE_<OPTION_NAME> environment variables.
  3. Values in the serving settings file.
  4. Admitted model/stage policy and built-in defaults.

Paths in a settings file are relative to that file. Paths supplied on the CLI remain relative to the working directory. Unknown keys, wrong sections, invalid values, and conflicting model sources are errors.

Environment variables use the CLI representation and the SKIPPY_SERVE_<OPTION_NAME> form. For example, SKIPPY_SERVE_THREADS=8 sets [execution].threads; --threads 4 overrides it.

Configuration surfaces

SurfaceBest forExample
CLIOne run or a measured override--temperature 0.2
EnvironmentDeployment-level process settingsSKIPPY_SERVE_THREADS=8
TOMLVersioned complete serving policy--settings serve.toml
Admitted stage planPlanner-owned split identity and tensor closure--config stage-0.json

Continue to the option reference.