Skip to Content
Overview

Introducing Skippy

Skippy is an AI inference engine for local and distributed workloads. It is the inference engine that powers Mesh LLM .

Built on llama.cpp, Skippy adds production-focused model and runtime management, OpenAI- and Anthropic-compatible APIs, explicit pipeline parallelism, multi-tier KV caching, memory-aware scheduling, and hardware-specific optimizations.

Skippy supports CPU inference and GPU acceleration through Metal, CUDA, ROCm, and Vulkan. Run it on a single machine or divide one verified model across an ordered set of workers. A single CLI and configuration model takes you from local inference to multi-machine serving.

Serve your first model

Build the development product and start a recommended model:

just build ./target/debug/skippy models recommended ./target/debug/skippy serve --model Qwen3-0.6B-Q4_K_M --prompt

The prompt and HTTP API are available together. From another terminal:

curl http://127.0.0.1:9337/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"Qwen3-0.6B-Q4_K_M","messages":[{"role":"user","content":"Hello"}]}'

How Skippy serves a request

Skippy resolves the model and a compatible native runtime, plans memory for weights and live context state, and opens one serving lifecycle for both API styles. Each request is tokenized, admitted by the scheduler, evaluated by the local runtime or ordered pipeline stages, and sampled by stage zero.

The same lifecycle owns model identity, prefix-cache compatibility, runtime selection, request limits, and telemetry. That keeps local and distributed serving behavior consistent even when the model is divided across machines.

Learn

Developed by