Own your AI.
Your models, your hardware, your rules.
Zallama is a small daemon that turns any .gguf model into a real, standing service on your machine — chat, vision, speech, embeddings, reranking, image generation — all speaking the OpenAI API dialect your tools already know. Point your app at http://localhost:11435 instead of api.openai.com, and nothing else has to change.
$ git clone https://github.com/rzafiamy/zallama.git
$ cd zallama && sudo bash install.sh
$ zallama serve
Stop renting
intelligence.
Every time you send a prompt to a cloud API, you're renting intelligence — and handing over your data, your context, and your habits along with it. Your hardware is idle. Your bill keeps growing. And the model behind the curtain can change, get deprecated, or get more expensive overnight.
Zallama flips that. You decide which models load, how much RAM/VRAM they get, when they sleep, and who's allowed to talk to them. Nobody else's usage policy, nobody else's outage, nobody else's log retention.
This is what it looks like to own your AI.
Your data stays home
Prompts, documents, audio, images — everything is processed on your machine and never leaves it. The daemon binds to 127.0.0.1 unless you say otherwise.
Your hardware finally works
That GPU sitting idle becomes a chat, vision, speech and retrieval server. Models load on demand and unload when idle, so your RAM and VRAM go back to you.
Nothing else has to change
A real OpenAI /v1 surface — chat, embeddings, audio, images, streaming included. Any OpenAI SDK, LangChain or existing integration works with a one-line base-URL swap.
You set the rules
Context size, GPU offload, memory budget, idle timeout, who gets a key — all YAML, all per-model, all hot-reloadable. No usage policy but yours, and no model that vanishes overnight.
A router and a process manager — that's it.
Under the hood, Zallama is a dynamic router and process manager for your local GGUF models. Ask for a model, and it starts the right engine, routes your request to it, and unloads it after a period of inactivity.
registry.yaml — one entry per model: file, modality, backend, params. Hot-reloaded on every request.
Reads the model field, checks the modality matches the endpoint, proxies to the right backend.
Spawns engines on ports 8100+, health-checks startup, caps concurrency, idle-unloads.
Set mem_budget_gb; LRU models of the same group are evicted so the next one fits. Pinned ones stay.
Register, don't discover
Zallama never scans a folder and silently loads what it finds. A model has to be registered before it exists. Nothing runs, and no memory is claimed, until you say so.
Config, not code
Two YAML layers: a global config read at startup (ports, budgets, defaults), and a per-model registry reloaded on every request. zallama set edits it for you.
Modality + backend seam
Each model declares a modality and a backend. New engines arrive as a new Backend subclass and an endpoint — the process manager never changes.
Everything a local AI server should do
Five commands cover most of your day. The rest is there when you need it.
A CLI that gets out of your way
serve, pull, run, set, ps — five commands cover most of your day. Tab completion for subcommands and model names included.
Fast model acquisition
Friendly shorthands like llama3.2:3b or any raw HuggingFace path. Downloads accelerate through aria2c (8 parallel connections) with a pure-Python fallback.
Reasoning models, rendered live
DeepSeek-R1, Qwen 3.5 and other thinking models stream their reasoning in dim gray inside the interactive chat. Turn it off per model with reasoning=false.
A real OpenAI /v1 surface
Chat, completions, embeddings, audio, images, rerank — streaming included — so existing SDKs and tools just work.
Vision, speech & images
Attach an mmproj projector and send images. Transcribe any audio (auto-transcoded via ffmpeg). Synthesize 54 voices. Generate images with stable-diffusion.cpp.
RAG, built in
A reranker at /v1/rerank plus zvec, an embedded HNSW vector store. Embed, store, search and rerank — no external vector database to run.
Memory awareness
Set a mem_budget_gb and Zallama evicts least-recently-used models to make room — automatically. Pin the small always-on ones so they never get kicked out.
Measure, don't guess
calibrate finds the largest context that fits your VRAM. bench sweeps parameters and reports tok/s. probe checks thinking, tool calls and vision with real requests.
Live monitoring
zallama monitor is an htop for your stack: GPU, VRAM, loaded models, in-flight requests with TTFT, prefill and decode rates. Plus a Prometheus /metrics endpoint.
Pluggable backends
Each model declares a modality + backend. New engines slot in behind one abstraction without touching the core.
Config, not code
Context size, GPU offload, batching, KV-cache quantization, speculative decoding, sampling defaults — all YAML, all per-model, all hot-reloadable.
Locked down by default
Binds to 127.0.0.1, optional Bearer-token auth with hashed keys and expiry, sane timeouts, and a separate admin port you can firewall on its own.
Start the daemon, pull a model, talk to it.
This is the whole loop. Everything else on this site is a deeper look at one of these four steps.
Wake the daemon
zallama serveStarts the controller on http://localhost:11435. It sits quietly until a model is requested.
Pull a model — it's now yours
# A friendly shorthand (from Unsloth's repo)
zallama pull llama3.2:3b
# Or any GGUF straight from HuggingFace
zallama pull unsloth/Qwen2.5-Coder-7B-Instruct-GGUF/Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf
# Speech-to-text, auto-registered as modality=asr
zallama pull parakeet:0.6bTune it to your hardware
zallama set llama3.2:3b n_gpu_layers=99 ctx_size=8192
zallama set qwen3.5-4b-q4_k_m reasoning=false # skip thinking blocksTalk to it
zallama run llama3.2:3bThat's it. A model running on hardware you control, answering only to you.
Replays are reconstructed from real sessions; numbers are illustrative.
One endpoint, every modality.
Each model declares what it is. Zallama picks the engine and the route. A request to the wrong endpoint gets a clear 400, never a crash.
Text & reasoning
Any instruct or thinking GGUF. Streaming, tool calling, and reasoning blocks pass straight through. Speculative decoding via a baked-in MTP head roughly doubles decode speed.
zallama pull qwen2.5-coder:7b
zallama pull deepseek-r1:8bVision
Register the base GGUF with its projector and send image_url content parts exactly as you would to OpenAI. Zallama passes --mmproj to the backend for you.
zallama add qwen2.5-vl-7b \
~/.zallama/models/qwen2.5-vl-7b-q4_k_m.gguf \
--mmproj ~/.zallama/models/mmproj-qwen2.5-vl-7b-f16.ggufSpeech-to-text
Upload any audio format — Zallama transcodes to WAV via ffmpeg. Use the multilingual Parakeet TDT v3 for 25 European languages with auto-detection.
zallama pull parakeet:0.6b # English
zallama pull mudler/parakeet-cpp-gguf/tdt-0.6b-v3-q8_0.gguf # multilingualText-to-speech
Kokoro-82M: 54 voices across 8 languages, returned as a WAV stream. Leave voice out and Zallama picks one matching the detected language of your text.
zallama pull kokoro:82mImage generation
Stable Diffusion 1.5, SDXL Turbo, FLUX and Qwen-Image via stable-diffusion.cpp. Set steps, cfg_scale and sampler once per model; the daemon fills them in.
zallama pull flux:klein
zallama generate flux:klein "a lighthouse at dawn" --output dawn.pngEmbeddings, rerank & zvec
Everything for local RAG: an embedding model, a cross-encoder reranker, and zvec — an in-process HNSW vector store that embeds and searches by calling Zallama's own /v1/embeddings.
zallama pull nomic-embed:v1.5
zallama pull bge-reranker:v2-m3
zallama zvec create notes
Your RAM and VRAM
go back to you.
Three mechanisms keep your machine usable. An idle sweep unloads a model after idle_timeout seconds of silence. A count cap (max_loaded_models) limits how many run at once. And a memory budget (mem_budget_gb) evicts least-recently-used models until the incoming one fits — only ever from the incoming model's own eviction group, so a chat model is never bumped by an embedding request.
Each model's cost comes from its declared mem_gb. If undeclared, it's estimated from file size — an estimate that is frequently off by 100% or more. So measure it:
# Predict the largest ctx_size that fits — writes nothing
zallama calibrate qwen3.5-4b-q4_k_m
# Measure by actually loading, then write ctx_size + mem_gb
zallama calibrate qwen3.5-4b-q4_k_m --probe --applySmall, latency-sensitive services (ASR, TTS) can be pinned: true — pre-warmed at startup and exempt from both the idle sweep and eviction. A ~1 GB model turns a 780 ms round-trip into an 80 ms one.
📚 Eviction, pinning & VRAM planning in the docsLoaded
0
Headroom
12.0 GB
Evictions
0
Loaded models (LRU at the top)
📌 = pinned, never evictedKnow what fits, what it costs, and what's happening now.
calibrate tells you what fits. bench tells you what it costs. monitor shows you what's happening.
zallama bench
$ zallama bench qwen3.5-4b-q4_k_m --sweep ctx_size=4096,16384
┌────────┬─────┬────────┬─────────┬─────────┬─────────────┬─────────────┐
│ PROMPT │ GEN │ LOAD s │ VRAM GB │ TTFT ms │ PREFILL t/s │ DECODE t/s │
├────────┼─────┼────────┼─────────┼─────────┼─────────────┼─────────────┤
│ 469 │ 128 │ 1.5 │ 4.8 │ 118 ±2 │ 8925 ±20 │ 179.8 ±0.2 │
│ 469 │ 128 │ 1.6 │ 5.2 │ 121 ±3 │ 8810 ±35 │ 178.9 ±0.4 │
└────────┴─────┴────────┴─────────┴─────────┴─────────────┴─────────────┘Sweeps restart the model with each combination and restore your original params when done — even on Ctrl-C. Export .json and --compare two machines side by side.
zallama monitor
GPU RTX 4090 util ████░░░░░░ 38% VRAM ██████████████████░░ 22.1/24.0 GiB
CPU load ██░░░░░░░░ 14% RAM ██░░░░░░░░░░░░░░░░░░ 4.9/62.5 GiB
MODELS 3 loaded/4 • declared 22.2/23.8 GB • measured 22.1 GiB
NAME STATE VRAM ACTIVE IDLE UP BACKEND
nex:n2.5-mini busy 21.4 GiB 1 2s 6s llama-server
granite-embedding… idle 0.7 GiB 0 40s 2m embedding-server
IN FLIGHT 1
# MODEL ENDPOINT ELAPSED TTFT TOKENS
7 nex:n2.5-mini chat/completions 1.9s 158ms 337TTFT is stamped on the first streamed delta; prefill/decode come off llama.cpp's own timings. The same data — plus lifecycle counters and histograms — is exposed for Prometheus at GET /metrics on the admin port — see Operate.
Built to be run, not just tried.
A separate admin port, a management API, Prometheus metrics, and eviction rules that keep a big model from being bumped by a small one.
Two ports, one process
The inference port serves only /v1/*. Everything that manages the daemon — /api/* for ps, load, pull, requests — and /metrics sit on the admin port, which you can firewall separately or bind to a VPN interface with admin_host.
# what the CLI calls under the hood
curl :11436/api/ps
curl :11436/api/requests | jq '.[0]'
curl -X POST :11436/api/models/llama3.2:3b/loadPrometheus /metrics
Capacity vs. declared vs. measured VRAM, per-model lifecycle counters (starts, startup failures, crashes, evictions), token and timing totals, TTFT/duration histograms, whole-card GPU numbers — and one zallama_model_param series per effective launch parameter, so a graph can be joined to the ctx_size it was produced under.
# free VRAM — what every 503 comes down to
zallama_gpu_memory_total_bytes - zallama_gpu_memory_used_bytes
# restart loop
increase(zallama_model_start_failure_total[5m]) > 2
# honest decode tok/s over 5 min, per model
rate(zallama_completion_tokens_total[5m])
/ rate(zallama_decode_seconds_total[5m])Eviction groups
A model only ever evicts a model from its own group. Text and image models are primary; ASR, embedding, rerank and TTS are services. So a 20 GB chat model is never bumped because an embedding request came in — and small services trade one shared slot among themselves. Give a group its own memory cap, and let a busy backend finish streaming before it's reclaimed.
llama_server:
evict_group_mem_budgets:
services: 1.5 # ASR + embed + rerank share 1.5 GB
evict_drain_timeout: 30 # don't kill a mid-stream backendChange the base URL. Keep everything else.
Name the model in the request body — Zallama loads it if needed and routes to the right engine.
Owning it means running it safely.
Safe for single-user local use out of the box — and more deliberate the further you push it outward.
Localhost by default
The daemon binds to 127.0.0.1. Nothing on your network can reach it until you change host — and if you do, set a key at the same time.
Hashed, expiring API keys
zallama apikey generates a 256-bit key, stores only its SHA-256 hash with an expiry, chmods the config to 600, and prints the key once. Verified with a constant-time compare.
VPN first, proxy second
For remote access, prefer WireGuard/Tailscale — zero public surface. If it must face the internet, keep Zallama on localhost and put a TLS proxy in front that only forwards /v1/*.
$ zallama apikey --expires 90d # prints the key once — store it in your password manager
$ ZALLAMA_HOST=0.0.0.0 zallama serve
$ curl http://llm-box:11435/v1/models -H "Authorization: Bearer $YOUR_KEY"Measured write-ups, not folklore.
Guides from the repository with real numbers on real hardware.
Fitting Your Models on One GPU
KV quantization, ctx_size, MoE expert offload and mmproj — each lever with its measured cost.
Doubling Decode Speed with MTP
Switch on a baked-in multi-token-prediction head with spec_type: draft-mtp for ~1 GiB of VRAM.
Tuning Image Generation
diffusion_fa, vae_tiling, quantized text encoders — what each switch buys and costs.
What Qwen3-0.6B Can Actually Do
A tiny model inside an agentic client — what works, what doesn't, and the settings that matter.
MiniCPM5-2B as an Agentic Model
Tool calling and reasoning with a 2B model — measured behaviour, not marketing.
A Small Model Beside a Big One
Running two models on one card: the memory arithmetic and where the budget breaks.
Model Tuning Log
A running log across models: settings tried, tok/s and VRAM measured — so nothing gets re-benchmarked from scratch.
Monitoring with Prometheus
Every metric on /metrics, the PromQL worth graphing, and alert suggestions.
Registry Parameter Reference
Every params key per modality and backend, in one place.
Own your AI ⭐️
Free, open source, MIT licensed, 100% local. Clone the repo, run the installer, and start serving text, vision, speech, images and retrieval from a single endpoint on hardware you control.
git clone https://github.com/rzafiamy/zallama.git
cd zallama && sudo bash install.sh