Making inference requests
Skippy exposes OpenAI-compatible and Anthropic Messages APIs on the same address and model lifecycle. Both APIs submit work to the same loaded model, scheduler, cache, and sampling pipeline.
Use the OpenAI-compatible API for clients built around Chat Completions. Use the Anthropic-compatible API for clients built around Messages. Wait for the server ready event before using either endpoint.
OpenAI chat completions
curl http://127.0.0.1:9337/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "Qwen3-0.6B-Q4_K_M",
"messages": [{"role": "user", "content": "Reply with OK"}],
"max_tokens": 32
}'Anthropic Messages
curl http://127.0.0.1:9337/v1/messages \
-H 'Content-Type: application/json' \
-H 'anthropic-version: 2023-06-01' \
-d '{
"model": "Qwen3-0.6B-Q4_K_M",
"messages": [{"role": "user", "content": "Reply with OK"}],
"max_tokens": 32
}'Use skippy serve --model ... --prompt when you want an interactive prompt and
the HTTP API in the same process. Use skippy prompt --endpoint URL to connect
to an already-running server.
How requests are processed
The model value must match an identifier returned by GET /v1/models. Skippy
renders the selected chat template, tokenizes the complete conversation,
applies the request or server sampling defaults, and admits the generation to
an available lane. Explicit request values take precedence over serving
defaults.
The response reports prompt and generated token usage. Streaming requests use the same lifecycle but emit incremental events. Invalid model IDs, unsupported request features, conflicting token limits, and context overflow return an API error rather than silently changing the request.
Endpoint reference
| API | Endpoint | Required request fields |
|---|---|---|
| OpenAI Chat Completions | POST /v1/chat/completions | model, messages |
| Anthropic Messages | POST /v1/messages | model, messages, max_tokens |
| Model discovery | GET /v1/models | None |
Send Content-Type: application/json for request bodies. Anthropic-compatible
clients should also send an anthropic-version header. Use the model-specific
request fields for sampling, reasoning, tools, structured output, streaming,
and output limits.
Deterministic caching
Skippy can reuse the model state produced for a shared prompt prefix. This avoids recomputing repeated system prompts, tool definitions, documents, and conversation history.
prompt_cache_key is a stable identifier chosen by your application. It places
related requests in the same cache namespace. A key can isolate tenants or
agent configurations, keep compatible requests together, and deliberately
start a fresh namespace when a prompt or toolset changes.
The key does not identify the cached prompt and does not guarantee a hit. Requests must still use a compatible model and runtime and share an identical leading token sequence. Reuse ends at the first token that differs.
You can omit prompt_cache_key for normal prefix caching. Requests without one
use the default cache namespace and can still reuse matching prefixes.
OpenAI-compatible requests
Add prompt_cache_key to a chat completion request:
{
"model": "Qwen3-0.6B-Q4_K_M",
"prompt_cache_key": "support-agent:v1",
"messages": [
{
"role": "system",
"content": "You are a support agent. Follow the company support policy."
},
{
"role": "user",
"content": "How do I reset my password?"
}
]
}A later request can use the same key and system prompt with a different user message:
{
"model": "Qwen3-0.6B-Q4_K_M",
"prompt_cache_key": "support-agent:v1",
"messages": [
{
"role": "system",
"content": "You are a support agent. Follow the company support policy."
},
{
"role": "user",
"content": "How do I update my email address?"
}
]
}Skippy can reuse the matching system-prompt prefix and prefill only the changed tail.
Anthropic-compatible requests
The Anthropic-compatible API accepts the same top-level field:
{
"model": "Qwen3-0.6B-Q4_K_M",
"prompt_cache_key": "support-agent:v1",
"max_tokens": 256,
"system": "You are a support agent. Follow the company support policy.",
"messages": [
{
"role": "user",
"content": "How do I reset my password?"
}
]
}Anthropic cache_control controls retention behavior. It does not select the
cache namespace or replace prompt_cache_key.
Choose a stable key
| Use case | Example |
|---|---|
| One agent configuration | support-agent:v1 |
| Tenant and agent isolation | tenant-42:support-agent:v1 |
| Versioned tool definitions | research-agent:tools-v3 |
| Default shared namespace | Omit prompt_cache_key |
Change the key when the isolation boundary or stable prompt configuration changes. Do not use a unique request ID, timestamp, or random value because a new value prevents reuse from earlier requests. Do not include secrets in the key.
Skippy trims surrounding whitespace and treats an empty key as absent.