● local-first · self-hosted · CPU-only

Run an OpenAI-compatible
inference API on your own hardware.

A production-shaped FastAPI service that speaks the OpenAI wire format — chat completions, streaming, tool calling and embeddings — powered by real local models. No cloud, no keys, no GPU required.

Explore the interactive API docs → /docs · OpenAPI · try every endpoint live

Models models/qwen2.5-0.5b-instruct-q4_k_m.ggufloaded models/nomic-embed-text-v1.5.Q8_0.ggufloaded

Backend local llama.cpp ctx 512 threads 2

🐚

Client

any OpenAI SDK

→

⚡

FastAPI

auth · rate-limit

→

🧮

Tokenizer

real token counts

→

🧠

Model

Qwen2.5 · CPU

→

📡

Stream

SSE responses

Endpoints

Every route is fully OpenAI-compatible, protected by Bearer API-key auth and rate limiting.

POST /v1/chat/completions

Multi-turn chat completions from the local Qwen2.5 model, with real token usage accounting.

stream = true · SSE chunks

POST /v1/embeddings

Semantic embeddings from a dedicated local model (nomic-embed-text, 768-dim vectors).

single string or batch

POST /v1/chat/completions

Tool / function calling — the model decides to call get_weather or add, then reasons over the result.

tools = [...]

GET /v1/models

List the models currently served by this instance.

qwen2.5 · nomic-embed

GET /health

Readiness probe for uptime monitors and orchestrators.

{"status":"ok"}

Why it matters

The same concerns every real inference API has to solve, tackled deliberately and minimally.

🔐

Auth

Bearer API-key validation on every protected route.

⏱️

Rate limiting

Fixed-window limits per key with Retry-After headers.

📶

Streaming

Token-level SSE with familiar OpenAI chunk shapes.

🧰

Tool calling

Model-driven function selection + execution loop.

🏠

Self-hosted

Runs entirely on CPU with quantized local weights.

📐

Compatible

Swap the base URL in any OpenAI client — nothing else changes.