Run Zallama your way.

Zallama is a local, multimodal, memory-aware LLM server. It is a small daemon that turns any .gguf model into a real, standing service on your machine — chat, vision, speech, embeddings, reranking, retrieval and image generation — all speaking the OpenAI API dialect your tools already know. Point your existing app at http://localhost:11435 instead of api.openai.com, and nothing else has to change.

Under the hood it is a dynamic router and process manager for your GGUF models. Ask for a model, and Zallama starts the right backend (llama-server, parakeet-server, kokoro-server, sd-server), routes your request to it, and unloads it after a period of inactivity so your RAM/VRAM goes back to you. Each model declares a modality (text, embedding, rerank, asr, tts, image); new modalities are added as new backends, not as changes scattered across the codebase.

Requirements

• Python 3.10+.
• A llama-server binary — built from llama.cpp or placed in ./bin/llama-server.
• aria2c — optional but recommended for fast downloads.
• For ASR and TTS: their engines built (below) and ffmpeg installed.
• For image generation: sd-server from stable-diffusion.cpp.

1 · Clone the repository

git clone https://github.com/rzafiamy/zallama.git
cd zallama

2 · Build the inference engines

Helper scripts build each engine and install the binaries into ./bin/. The clone and build happen in a temp directory, keeping your repo tree clean. Each takes a release tag or branch name:

# llama.cpp (text / chat / embeddings / vision)
./build-ggml-llama.cpp.sh b4600

# parakeet.cpp (ASR / speech-to-text)
./build-ggml-parakeet.cpp.sh master

# kokoro.cpp (TTS / voice synthesis)
./build-ggml-kokoro.cpp.sh v0.3.0

# stable-diffusion.cpp (image generation)
./build-ggml-stable-diffusion.cpp.sh master

Optional · Use prebuilt packages

To skip building from source, prebuilt packages are available from the shared Google Drive folder: prebuilt packages. Download the one matching your platform/accelerator stack and copy the binaries into ./bin/:

mkdir -p ./bin
cp /path/to/extracted/package/llama-server ./bin/

# Optional modality backends
cp /path/to/extracted/package/parakeet-server ./bin/   # ASR
cp /path/to/extracted/package/kokoro-server ./bin/     # TTS
cp /path/to/extracted/package/sd-server ./bin/         # images

At minimum, Zallama needs llama-server in ./bin/ or on your PATH. The installer checks ./bin/ first, so locally copied binaries are used automatically. See PREBUILT.md for package naming rules and the prebuilt-cli.sh helper, which detects your Ubuntu/CUDA setup, lists compatible packages, and installs the selected archive.

3 · Run the installer

sudo bash install.sh

The installer:

1. Verifies Python 3 and installs requirements.txt into a project-local .venv (required on modern Debian/Ubuntu under PEP 668).
2. Checks for a llama-server binary (./bin, or on PATH).
3. Makes the zallama CLI executable and creates ~/.zallama/{models,logs,bin}.
4. Symlinks zallama into /usr/local/bin (or ~/.local/bin) when possible.
5. Installs a systemd service — only when run as root. See Deployment.

To run zallama from anywhere without the symlink, add the repo to your PATH:

export PATH="$PWD:$PATH"   # from the zallama checkout

This is the whole loop — start the daemon, pull a model, talk to it. Everything else on this page is a deeper look at one of these four steps.

zallama serve

# Friendly shorthand (Unsloth repo)
zallama pull llama3.2:3b

# Any GGUF from HuggingFace
zallama pull unsloth/Qwen2.5-Coder-7B-Instruct-GGUF/Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf

# Speech-to-text (auto-registered as asr)
zallama pull parakeet:0.6b
zallama pull mudler/parakeet-cpp-gguf/tdt-0.6b-v3-q8_0.gguf

zallama set qwen3.5-4b-q4_k_m reasoning=false ctx_size=8192
zallama set llama3.2:1b n_gpu_layers=99

zallama run llama3.2:3b

Shorthand presets

These names resolve to well-known repositories and register the right modality for you:

Zallama can serve GGUF files you already downloaded, but it won't auto-discover files sitting in a directory — a model has to be registered before it shows up in zallama list or /v1/models. This is deliberate: nothing runs, and no memory is claimed, until you say so.

Check where Zallama looks for models (default is ~/.zallama/models):

zallama config

Copy a model there and register it:

cp ~/Downloads/mistral-7b-q4.gguf ~/.zallama/models/
zallama add mistral-7b ~/.zallama/models/mistral-7b-q4.gguf

# Or register a file outside models_dir by absolute path
zallama add my-model /data/models/my-model-q4_k_m.gguf

# Then run it like anything you pulled
zallama run mistral-7b

For manual registry edits, relative paths resolve under models_dir:

models:
  - name: "mistral-7b"
    file: "mistral-7b-q4.gguf"
    params:
      ctx_size: 8192
      n_gpu_layers: 99

Vision models need one extra file

Copy the base GGUF and its projector, then register both together:

cp ~/Downloads/qwen2.5-vl-7b-q4_k_m.gguf ~/.zallama/models/
cp ~/Downloads/mmproj-qwen2.5-vl-7b-f16.gguf ~/.zallama/models/

zallama add qwen2.5-vl-7b \
  ~/.zallama/models/qwen2.5-vl-7b-q4_k_m.gguf \
  --mmproj ~/.zallama/models/mmproj-qwen2.5-vl-7b-f16.gguf

Or by hand:

models:
  - name: "qwen2.5-vl-7b"
    file: "qwen2.5-vl-7b-q4_k_m.gguf"
    modality: "text"
    backend: "llama-server"
    artifacts:
      mmproj: "mmproj-qwen2.5-vl-7b-f16.gguf"
    params:
      ctx_size: 8192
      n_gpu_layers: 99

When the model loads, Zallama resolves both paths relative to models_dir and passes --mmproj <file> to llama-server. If mmproj is missing or not listed under artifacts, the model may still load for text, but image input won't work.

Tab completion

./install.sh installs bash and zsh completion automatically when it can write to the system completion directories. Once enabled, TAB completes both subcommands and registered model names:

zallama r<TAB>          # → reload  remove  rm  run
zallama run <TAB>       # → completes from your registered models

Completion reads the model list straight from registry.yaml, so it works even when the daemon isn't running. To install manually:

# bash
sudo cp completions/zallama.bash /usr/share/bash-completion/completions/zallama
# zsh — copy into a dir on your $fpath, then run compinit
cp completions/_zallama ~/.zsh/completions/_zallama

There are two layers, and they change on different timescales:

See what's active

zallama config

Shows the effective configuration, where it was loaded from, and the available keys.

Change global settings

mkdir -p ~/.zallama
cp config/config.example.yaml ~/.zallama/config.yaml
$EDITOR ~/.zallama/config.yaml

Zallama looks for the first existing config file, in order:

1. <repo>/config/config.yaml
2. ~/.zallama/config.yaml
3. /etc/zallama/config.yaml

Anything omitted falls back to built-in defaults. Restart the daemon after editing config.yaml — global config is read at startup:

systemctl restart zallama

Change per-model settings

Prefer the CLI over hand-editing the registry:

zallama set <model> <key>=<value> [<key>=<value> ...]
zallama set llama3.2:1b ctx_size=8192 n_gpu_layers=99
zallama set my-reranker modality=rerank
zallama show llama3.2:1b

Per-model changes are saved to the registry and apply on the model's next load. If it's already running, its process keeps the params it launched with — reload it to pick up the change:

zallama reload <model>

Environment overrides

For a one-off shell/session, without touching any file:

ZALLAMA_HOST=0.0.0.0 ZALLAMA_PORT=11435 zallama serve
ZALLAMA_ADMIN_PORT=11436 zallama serve      # /api/* + /metrics listener
ZALLAMA_MODELS_DIR=/data/models zallama serve
LLAMA_GPU_LAYERS=99 zallama serve
ZALLAMA_EMBEDDING_MODEL=my-embed zallama serve

Global settings reference

zallama:
  host: "127.0.0.1"      # localhost only; set "0.0.0.0" to expose on the network
  port: 11435            # inference: /v1/*
  admin_port: 0          # admin: /api/* + /metrics (0 = port + 1, i.e. 11436)
  admin_host: ""         # admin bind address ("" = same as host)
  models_dir: "~/.zallama/models"
  logs_dir: "~/.zallama/logs"
  log_level: "info"
  api_key: ""            # plaintext Bearer token — prefer the hashed form below
  api_key_sha256: ""     # SHA-256 hex of the key; manage with `zallama apikey`
  api_key_expires: ""    # ISO-8601 UTC expiry written by `zallama apikey` ("" = never)
  request_timeout: 600   # seconds for non-streaming upstream proxy calls

llama_server:
  binary: ""             # Auto-detects in ./bin/llama-server or PATH
  port_start: 8100       # Backends spawn on ports 8100, 8101, etc.
  startup_timeout: 60    # Seconds to wait for a backend's /health
  idle_timeout: 300      # Auto-unload model after 300s of inactivity (0 to disable)
  max_loaded_models: 0   # Max concurrently loaded models; evicts LRU (0 = unlimited)
  mem_budget_gb: 0       # Memory budget (GB); evicts LRU to fit a model's mem_gb (0 = unlimited)
  mem_init_gb: 2         # Fallback per-model cost when mem_gb is undeclared & file size unknown
  evict_group_mem_budgets: {}   # per-group cap, e.g. { services: 1.5 } — see Memory-aware eviction
  evict_drain_timeout: 30       # wait for a busy victim to finish streaming before killing it (0 = never)
  default_params:
    ctx_size: 8192
    n_gpu_layers: 99     # Attempt full GPU offload by default
    threads: 8
    flash_attn: true
    cont_batching: true
    parallel: 1          # Single-session slot allocation (avoids context limits)
    mlock: false
    no_mmap: false

rag:
  embedding_model: ""    # default model for /v1/embeddings and zvec
  rerank_model: ""       # default reranker
  zvec_dir: "~/.zallama/zvec"
  default_top_k: 5

The daemon listens on two ports: the inference API (/v1/*) on port, and the management API (/api/* — ps, list, load, pull…) plus the Prometheus /metrics endpoint on admin_port. Both share the same API key; /metrics and /health are the only unauthenticated paths. The split lets you expose inference to clients while keeping management and scraping on a port you firewall separately (or bind to a different interface with admin_host). The CLI finds the admin port from the same config; override with ZALLAMA_ADMIN_HOST=http://host:port if it runs elsewhere.

Model registry reference

models:
  # Standard text/chat model
  - name: "qwen3.5-4b-q4_k_m"
    file: "/home/user/.zallama/models/Qwen3.5-4B-Q4_K_M.gguf"
    description: "Downloaded from unsloth/Qwen3.5-4B-GGUF"
    params:
      ctx_size: 8192
      n_gpu_layers: 99
      reasoning: false   # Bypasses thinking blocks for immediate responses

  # Vision (multimodal) model — attach an mmproj projector via `artifacts`
  - name: "qwen2-vl-7b"
    file: "Qwen2-VL-7B-Instruct-Q4_K_M.gguf"   # relative paths resolve under models_dir
    modality: "text"     # vision is served on /v1/chat/completions (default modality)
    backend: "llama-server"
    artifacts:
      mmproj: "mmproj-Qwen2-VL-7B-Instruct-f16.gguf"
    params:
      ctx_size: 8192
      n_gpu_layers: 99

  # ASR (speech-to-text) model — runs on parakeet-server
  - name: "tdt-0.6b-v3-q8_0"
    file: "tdt-0.6b-v3-q8_0.gguf"
    modality: "asr"
    backend: "parakeet-server"
    description: "Parakeet TDT 0.6B v3 (multilingual ASR)"
    mem_gb: 1.3
    pinned: true
    params:
      threads: 4

Each entry may declare:

Useful params for making a model fit on a busy GPU — cache_type_k/cache_type_v (quantize the KV cache), ctx_size, n_cpu_moe (keep the expert weights of the first N layers in system RAM, MoE models only) and n_gpu_layers — are covered with measured trade-offs in Fitting Your Models on One GPU. For diffusion models, see Tuning Image Generation.

If your GGUF carries a baked-in MTP head — llama.cpp logs its tensors as unused tensor blk.N.nextn.* -- ignoring until you switch it on — spec_type: draft-mtp roughly doubles decode speed for about 1 GiB of VRAM. See Doubling Decode Speed with MTP. For a model with no baked-in head but a genuinely separate draft checkpoint, register it as the draft artifact and set spec_type: draft-simple instead; spec_draft_ngl caps how much of the draft model rides on VRAM.

Sampling knobs — temperature, top_p, top_k, min_p, presence_penalty, repeat_penalty — are also settable per-model. They become the launch defaults, so a chat request that sets its own value always wins; the registry value only applies when the request leaves the field out.

zallama calibrate tells you what fits. zallama bench tells you what it costs — in tokens/sec, VRAM, and latency — for any combination of parameters you care to compare.

$ zallama bench qwen3.5-4b-q4_k_m

┌───────────────────────────────────────────────────────────────────────┐
│ PROMPT │ GEN │ LOAD s │ VRAM GB │ TTFT ms │ PREFILL t/s │ DECODE t/s  │
├────────┼─────┼────────┼─────────┼─────────┼─────────────┼─────────────┤
│    469 │ 128 │    1.5 │     4.8 │  118 ±2 │    8925 ±20 │  179.8 ±0.2 │
└───────────────────────────────────────────────────────────────────────┘

Three numbers, three different questions:

LOAD s and VRAM GB come along for free — which is how you find out that doubling ctx_size cost you 400 MB of VRAM and bought nothing.

Sweeping parameters

--sweep <param>=<v1>,<v2> is repeatable and builds a full grid. Each combination is written to the registry, the model is restarted so the backend actually launches with it, and your original parameters are restored when the run ends — including on Ctrl-C.

# Does a bigger context slow generation down? What does it cost in VRAM?
zallama bench qwen3.5-4b-q4_k_m --sweep ctx_size=4096,16384,65536

# What does thinking cost, at short and long prompts?
zallama bench qwen3.5-4b-q4_k_m --sweep reasoning=on,off -p 512,8192

# Find the offload cliff on a small GPU
zallama bench qwen3.5-9b-q4_k_m --sweep n_gpu_layers=0,20,40,99

# Quantized KV cache: how much ctx does the VRAM buy back?
zallama bench qwen3.5-4b-q4_k_m --sweep cache_type_k=f16,q8_0,q4_0
zallama bench flux:klein --image-size 1024x1024 --sweep vae_tiling=true,false

# Compare models head to head, and keep the numbers
zallama bench --all --runs 5 --out bench.md

Anything llama-server takes is fair game: ctx_size, reasoning, reasoning_effort, chat_template_kwargs, n_gpu_layers, threads, batch_size, ubatch_size, flash_attn, parallel, cache_type_k, cache_type_v, spec_type, spec_draft_n_max, spec_draft_ngl, image_min_tokens, image_max_tokens, temperature, top_p, top_k, min_p, presence_penalty, repeat_penalty.

Measured results from past sweeps are kept in the Model Tuning Log, so a config doesn't get re-benchmarked from scratch every session.

Options

Comparing two machines

A tokens/sec number means nothing without the GPU, driver and llama.cpp build behind it, so every .json export carries them. Run the same command on each box, then compare:

# on each machine — identical workload, one file each
zallama bench Qwen3.6-35B-A3B-UD-Q4_K_M \
  --sweep ctx_size=32768 --sweep reasoning=on,off \
  -p 32000 -n 256 --runs 3 --out $(hostname).json

zallama bench --compare rtx4090.json rtx3090.json

 Machines
│ LABEL   │ GPU                     │ VRAM GiB │ DRIVER     │ CPU                │ POINTS │
│ rtx4090 │ NVIDIA GeForce RTX 4090 │ 24.0     │ 570.172.08 │ Core(TM) i9-14900K │ 2      │
│ rtx3090 │ NVIDIA GeForce RTX 3090 │ 24.0     │ 570.172.08 │ Core(TM) i9-14900K │ 2      │

 DECODE t/s  (higher is better, % vs rtx4090)
│ POINT                                     │ rtx4090 (base) │ rtx3090      │
│ …· reasoning=on · prompt 32000 · gen 256  │          182.2 │ 113.0 (-38%) │
│ …· reasoning=off · prompt 32000 · gen 256 │          181.6 │ 112.6 (-38%) │

The first file is the baseline; every other column shows its delta. Only points with identical model, swept params, prompt size, max-tokens and concurrency are put side by side — anything unmatched is reported, never silently averaged in. If the exports came from different llama.cpp builds you get a warning, because part of the gap is then the engine rather than the hardware. --metric picks which tables to print: prefill_tps, decode_tps, throughput_tps, ttft_ms, load_s, vram_gib, gen_tokens.

Reading the results

• PREFILL / DECODE come off llama.cpp's own clock (its timings block), so they measure the engine, not your network. TTFT and TOTAL are measured at the client and include queueing and proxy overhead — which is what you want when the question is "how does this feel".
• A combination that fails to load (a ctx_size that won't fit, say) is reported inline and skipped; the sweep carries on and the failure is recorded in the export with its error.
• --concurrency above the model's parallel slots just queues. Sweep them together — --sweep parallel=1,4 -c 4 — to see what batching actually buys.
• ACCEPT % appears for models running speculative decoding: the share of drafted tokens the target accepted. Read a spec_draft_n_max sweep off this column. Anything near 100% means the model is copying its prompt, not generating.

Three views of the same data, at three timescales: zallama monitor for right now, GET /api/requests for the last 200 requests, and Prometheus GET /metrics for history, graphs and alerts.

Live dashboard (zallama monitor)

An htop-style view of the whole stack, refreshed every second (q quits, r refreshes, --interval S, --once prints a single frame for scripts; alias top):

 GPU  RTX 4090     util ████░░░░░░  38%   VRAM ██████████████████░░ 22.1/24.0 GiB  61°C  212W/450W
 CPU               load ██░░░░░░░░  14%   RAM  ██░░░░░░░░░░░░░░░░░░  4.9/62.5 GiB

 MODELS  3 loaded/4  •  declared 22.2/23.8 GB  •  measured 22.1 GiB
   NAME                 STATE     VRAM  ACTIVE  IDLE   UP   BACKEND
   nex:n2.5-mini        busy  21.4 GiB       1    2s   6s   llama-server
   granite-embedding…   idle   0.7 GiB       0   40s   2m   embedding-server

 IN FLIGHT  1
       #  MODEL          ENDPOINT          ELAPSED    TTFT  TOKENS
       7  nex:n2.5-mini  chat/completions     1.9s   158ms     337

 RECENT  last 20  •  since 42m: 61 req, 1 err, 48,120 in / 9,312 out tokens
   TIME     MODEL          ENDPOINT          PROMPT CACHE  OUT   TTFT  PREFILL  DECODE  ACC  TOTAL  ST
   08:00:01 nex:n2.5-mini  chat/completions     13     0   18  221ms    248/s 183.9/s    —  221ms  ok
   07:59:57 nex:n2.5-mini  chat/completions     25     0  400  158ms    194/s 194.8/s    —   2.2s  ok

The daemon records every proxied /v1 request (GET /api/requests on the admin port): TTFT is stamped on the first content delta of a stream, PREFILL/DECODE/CACHE come off the timings block llama.cpp appends to its response, ACC is draft acceptance for speculative models, and an in-flight streamed request shows how many tokens have gone by so far. Records also carry the raw prefill_ms, decode_ms, draft_n and draft_n_accepted. The last 200 requests are kept in memory; nothing is written to disk.

Prometheus (GET /metrics)

Everything monitor shows — and the history that tool can't keep — is exposed as a Prometheus text endpoint on the admin port:

http://<host>:<admin_port>/metrics        # default admin_port = port + 1 → 11436

It is unauthenticated (like /health) so scrapers on other machines don't need the API key. The body contains model names, counters and GPU numbers, nothing else. Restrict who can reach it with the admin port's bind address (zallama.admin_host) or a firewall rule, not with the key. Rendering is cheap (one nvidia-smi call and a walk over in-memory structs), so 5–15 s intervals are fine:

scrape_configs:
  - job_name: zallama
    scrape_interval: 15s
    static_configs:
      - targets: ["llm-box:11436"]
        labels: { gpu_host: llm-box }

What's exposed

Base units throughout: bytes, seconds, ratios 0–1. model is the registry name. Anything named _declared_ is the daemon's launch-time estimate (mem_gb); everything else is measured.

Daemon / capacity

Registry — every model, loaded or not

zallama_model_param is the "what was it configured as when that happened" series: a re-tune (zallama set … ctx_size=24576) shows up as the old label set going stale and a new one appearing, exactly at the timestamp the tok/s or VRAM graph changed. Join it onto anything per-model with group_left:

zallama_last_decode_tokens_per_second
  * on (model) group_left (value)
    zallama_model_param{param="spec_type"}

Per loaded instance

Lifecycle counters (survive the instance)

Requests (per model)

Only /v1 endpoints that go through the proxy are counted: chat/completions, completions, embeddings, rerank, audio, images. Token and timing fields are filled for the llama-server family (they come off llama.cpp's timings block); embeddings / audio / images contribute only counts and duration.

GPU (whole card, nvidia-smi)

zallama_gpu_memory_used_bytes, zallama_gpu_memory_total_bytes, zallama_gpu_utilization_ratio, zallama_gpu_temperature_celsius, zallama_gpu_power_watts, all labelled {gpu, name}. Used memory is all processes, not just Zallama's — it's the number that decides whether the next model loads. If a dcgm-exporter / node_exporter already scrapes the card, ignore these; they exist so a Zallama-only setup still sees VRAM.

Queries worth having

# Free VRAM — the value every 503 comes down to
zallama_gpu_memory_total_bytes - zallama_gpu_memory_used_bytes

# Restart loop (evict/respawn ping-pong, or a model that no longer fits)
increase(zallama_model_start_failure_total[5m]) > 2

# Average decode tok/s over 5 minutes, per model — the honest number, not the last request
rate(zallama_completion_tokens_total[5m]) / rate(zallama_decode_seconds_total[5m])

# Draft acceptance over a window
rate(zallama_draft_accepted_tokens_total[15m]) / rate(zallama_draft_tokens_total[15m])

# p95 TTFT
histogram_quantile(0.95, sum by (model, le) (rate(zallama_request_ttft_seconds_bucket[10m])))

# Which models run with speculative decoding, and at what context
zallama_model_param{param="spec_type"}
zallama_model_param_value{param="ctx_size"}

# How far the daemon's accounting is from reality
# (positive = mem_gb is under-declared and the model will one day fail to load next to the services)
zallama_model_vram_bytes - zallama_model_mem_declared_bytes

Implementation notes

• server/metrics.py renders the text format by hand — gauges, counters and two histograms from numbers that already exist in ProcessManager and RequestLog. No prometheus_client: a registry would just be a second copy of that state to keep in sync.
• Lifecycle counters live in ProcessManager.lifecycle and deliberately outlive the instances they describe. Request aggregates live in RequestLog.aggregates, next to the 200-entry deque zallama monitor reads; the deque is a window, the aggregates are monotonic.
• Everything resets when the daemon restarts, which is what Prometheus expects from counters (rate()/increase() handle the reset).
• Histogram buckets: TTFT 50 ms … 30 s, duration 100 ms … 600 s. Change TTFT_BUCKETS / DURATION_BUCKETS in server/request_log.py if your workload is far from that.

Beyond the time-based idle sweep and the max_loaded_models count cap, Zallama can keep total loaded-model memory within a budget. Set mem_budget_gb and, before starting a model, Zallama evicts least-recently-used instances — from the same eviction group only — until the incoming model fits.

Each model's cost is taken from its declared mem_gb; if undeclared, it's estimated from the GGUF file size (≈ size × 1.2 for KV-cache overhead), falling back to mem_init_gb when the size is unknown. Declaring mem_gb is recommended for accuracy:

zallama set qwen3.5-4b-q4_k_m mem_gb=4

Calibrate: predict, then measure

For a starting number without loading anything, zallama calibrate <model> reads the GGUF's own dimensions — counting only the layers that actually hold a KV cache, which on hybrid and sliding-window architectures is a fraction of the block count — and reports the largest ctx_size that still fits alongside the weights, the artifacts and a compute reserve. It writes nothing; it prints the zallama set line.

For the real number, zallama calibrate <model> --probe loads the model instead of predicting: it bisects ctx_size between 4096 and the trained context (~6–10 loads, a minute or two), reads the card's actual free VRAM after each, and reports the largest value that still leaves --margin GiB free — by default the services group's memory budget, i.e. room for the embedding/ASR/TTS models this one has to coexist with. Everything the static estimate has to guess — compute buffers, an MTP or DFlash draft context's own KV cache, whether the mmproj sits on the GPU, a MoE's expert layout — is simply measured. Add --apply to write ctx_size and the measured mem_gb straight into the registry:

zallama calibrate nemotron:30b-a3b --probe --apply
zallama calibrate Qwen3.8-27B-UD-Q5_K_M --probe --margin 1.5 --max 131072

And before trusting any of it, zallama probe <model> sends three short real requests — a trivial question, a tool definition, and (if the entry has an mmproj) a solid red image — and reports whether the model thinks before answering and what that costs, whether tool calls come back structured, and whether it sees. It also reads the chat template to say which thinking switch this model actually understands, because reasoning: false only works on templates with a <think> toggle; others read reasoning_effort, enable_thinking or their own variable via chat_template_kwargs.

Eviction groups: evict_group

When a model needs to load and capacity is tight, Zallama evicts the least-recently-used non-pinned instance that shares the incoming model's eviction group — never one outside it. Every modality has a built-in default group, so this works with no config at all:

In other words: a large, slow-to-reload text or image model only ever gets evicted to make room for another text/image model — never bumped just because a small ASR or embedding request came in. Conversely, ASR and embedding trade a shared slot back and forth between themselves without ever reaching into the text model's slot. Override it per model from the CLI:

zallama set my-reranker evict_group=custom-slot   # put it in its own group
zallama set my-reranker evict_group=              # opt out of grouping entirely (any non-pinned victim)
zallama set my-reranker evict_group=none           # reset to the modality default

If capacity is tight and no same-group victim is loaded, Zallama does not fall back to evicting across groups — it logs a warning and admits the incoming model over budget instead, the same fallback used when every loaded model is pinned.

Per-group budgets: evict_group_mem_budgets

max_loaded_models and mem_budget_gb only pressure eviction once the global count/budget is exceeded. If those were sized for two primary models and only one is loaded, the leftover slack lets several services models pile up side by side. A per-group memory budget in config.yaml closes that gap, checked regardless of global slack:

llama_server:
  evict_group_mem_budgets:
    services: 1.5   # ASR + embedding + rerank share 1.5 GB between them

It is a memory cap, not an instance-count cap: two small models coexist as long as their combined mem_gb fits, and the group's LRU member is evicted only when a new arrival would push the group over its own budget — while primary stays untouched. As with the global budget, it's only meaningful once the members carry measured mem_gb values.

Don't kill a backend mid-response: evict_drain_timeout

When two models don't fit together and clients alternate between them, a request for model B can land while model A is still streaming — eviction would kill A's backend under the in-flight proxy and A's client gets a truncated stream / 502. Zallama tracks in-flight requests per backend: eviction prefers a victim with nothing in flight, and when the only eligible victim is busy it waits up to evict_drain_timeout seconds (default 30; 0 = evict immediately) for it to go idle. The idle sweep also skips a backend that's still serving, so a generation longer than idle_timeout won't be cut off.

Keeping a model resident: pinned

A model with pinned: true in its registry entry is pre-loaded when the daemon starts and never evicted. It exists for small, latency-sensitive backends that would otherwise be evicted by every chat request and reloaded on the next call — ASR and TTS in particular, where a ~1 GB model turns a 780 ms round trip into an 80 ms one.

- name: tdt-0.6b-v3-q8_0
  modality: asr
  backend: parakeet-server
  mem_gb: 1.3
  pinned: true

Two things to know. Pinned models still count against max_loaded_models, so size that cap as (pinned services) + (concurrent big models) — with one pinned TTS and max_loaded_models: 1, the only slot is permanently taken and every other model thrashes. And when the cap is reached but everything loaded is pinned, Zallama logs a warning and admits the incoming model over budget rather than killing a warm pinned instance.

Pinning a small model can make your largest one unloadable. Fitting Your Models on One GPU works through that arithmetic and the levers — KV quantization, ctx_size, MoE expert offload (n_cpu_moe), mmproj — with measured numbers for each.

Inspect current usage and headroom any time with zallama ps (or GET /api/ps):

NAME                      PORT     MEM      UPTIME       LAST USED
─────────────────────────────────────────────────────────────────────────
qwen3.5-4b-q4_k_m         8100     4.0GB    3m12s        8s ago

Memory: 4.0GB / 12.0GB used  •  8.0GB free  •  1 loaded

Vision models run on llama-server with a multimodal projector (mmproj). Register the base GGUF and point an mmproj artifact at the projector file:

zallama add qwen2-vl-7b ~/.zallama/models/Qwen2-VL-7B-Instruct-Q4_K_M.gguf \
  --mmproj ~/.zallama/models/mmproj-Qwen2-VL-7B-Instruct-f16.gguf

Zallama passes --mmproj <file> to the backend automatically, and image input flows through the standard OpenAI /v1/chat/completions endpoint (image_url message content):

curl http://localhost:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2-vl-7b",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "What is in this image?"},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}}
      ]
    }]
  }'

Run zallama probe qwen2-vl-7b to confirm the model actually sees — it sends a solid red image and checks the answer.

Audio transcription runs on the parakeet-server backend (parakeet.cpp) and is exposed at the OpenAI-compatible POST /v1/audio/transcriptions endpoint.

1. Build the binary (installs parakeet-server into ./bin/):

   ./build-ggml-parakeet.cpp.sh master

2. Pull a model (auto-registered as modality: asr, backend: parakeet-server):

   zallama pull parakeet:0.6b                                   # English (TDT 0.6B v2)
   zallama pull mudler/parakeet-cpp-gguf/tdt-0.6b-v3-q8_0.gguf  # Multilingual v3 (incl. French)

3. Transcribe — upload any audio format; Zallama transcodes it to WAV via ffmpeg before forwarding:

   curl http://localhost:11435/v1/audio/transcriptions \
     -F model=tdt-0.6b-v3-q8_0 \
     -F file=@speech.mp3 \
     -F response_format=text

response_format accepts text, json, or verbose_json (with -F 'timestamp_granularities[]=word' for per-word timing).

Speech synthesis runs on the kokoro-server backend (kokoro.cpp) and is exposed at the OpenAI-compatible POST /v1/audio/speech endpoint.

1. Build the binary (installs kokoro-server and kokoro-cli into ./bin/):

   ./build-ggml-kokoro.cpp.sh v0.3.0

2. Pull a model (auto-registered as modality: tts, backend: kokoro-server):

   zallama pull kokoro:82m     # Kokoro-82M, 54 voices across 8 languages

3. Synthesize — the response is a WAV stream:

   curl http://localhost:11435/v1/audio/speech \
     -H "Content-Type: application/json" \
     -d '{"model":"kokoro:82m","input":"Bonjour, comment allez-vous ?"}' \
     -o speech.wav

Voice selection

Kokoro takes no language argument — it phonemizes according to the voice prefix (ff_siwis → French, af_heart → American English, if_sara → Italian, …). Sending French text with an English voice therefore reads it with English sounds. So when a request names no voice, Zallama guesses the language from the text and picks that language's voice. Precedence:

Detection is a small built-in heuristic, not a language identifier: CJK and Devanagari are settled by script, the Latin languages by function-word frequency. It deliberately answers "unknown" for very short inputs — "Merci" and "Mercy" are not distinguishable in five letters — and falls through to your registry default there. Pin voice in the request whenever you need a guaranteed result:

curl http://localhost:11435/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"kokoro:82m","input":"Bonjour !","voice":"ff_siwis"}' \
  -o speech.wav

Text-to-image runs on the sd-server backend (stable-diffusion.cpp) and is exposed at the OpenAI-compatible POST /v1/images/generations endpoint.

1. Build the binary (installs sd-server and the sd CLI into ./bin/). Builds with CUDA when nvcc is found and falls back to a CPU build otherwise:

   ./build-ggml-stable-diffusion.cpp.sh master

2. Pull a model (auto-registered as modality: image, backend: sd-server):

   zallama pull sd:1.5         # Stable Diffusion v1.5
   zallama pull sdxl:turbo     # SDXL Turbo — few-step, near real-time
   zallama pull flux:klein     # FLUX Klein (compact 4-bit, fast 4-step generation)
   zallama pull qwen-image:20b # Qwen-Image 20B (MMDiT image generation)
   
   # Already have weights locally? Register them directly:
   zallama add sd15 /path/to/v1-5-pruned-emaonly.safetensors --modality image

3. Generate — from the CLI or over HTTP:

   zallama generate sd:1.5 "a lighthouse at dawn, cinematic" --output dawn.png --size 512x512 --steps 20
   
   curl http://localhost:11435/v1/images/generations \
     -H "Content-Type: application/json" \
     -d '{"model":"sd:1.5","prompt":"a lighthouse at dawn","size":"512x512","response_format":"b64_json"}'

Diffusion weights ship as .safetensors / .ckpt rather than GGUF, and the downloader accepts those extensions for image repos. Generation knobs (steps, cfg_scale, sampler, negative_prompt) can be set once per model and reused for every request — the daemon applies those registry values to any request that does not specify them. Auxiliary weights (vae, taesd, control_net, clip_l, clip_g, t5xxl) are passed to sd-server at launch when registered as artifacts on the model.

zallama set sd:1.5 steps=25 cfg_scale=7.5

zallama set flux:klein diffusion_fa=true vae_tiling=false
zallama bench flux:klein --image-size 1024x1024 --sweep vae_tiling=true,false

The other memory switches are max_vram + stream_layers (graph-cut segmented execution — prefer these over offload_to_cpu), auto_fit, taesd (tiny autoencoder, far faster decode), eager_load, mmap, and backend for per-component placement (e.g. backend=te=cpu). Tile geometry is tunable with vae_tile_size, vae_tile_overlap and vae_relative_tile_size. Full measured write-up: Tuning Image Generation.

Zallama separates the generic process lifecycle (spawn, health-check, port assignment, LRU eviction, kill) from engine-specific logic (which binary to run, how to build its arguments, which health path to poll). The latter lives behind a Backend abstraction in server/backends.py.

This is the seam for new modalities. Each one arrived as a new Backend subclass plus a matching endpoint proxy — no changes to the process manager or registry schema.

Zallama ships everything needed for retrieval-augmented generation locally: an embedding model (via /v1/embeddings), a reranker, and zvec — an in-process vector store. No external vector database server to run.

One-command setup

zallama pull has shorthands for an embedding model and a reranker that write the correct registry entries (the embedding-server and rerank-server backends) for you:

zallama pull nomic-embed:v1.5      # embedding model → /v1/embeddings
zallama pull bge-reranker:v2-m3    # reranker        → /v1/rerank

Then point the rag config block at them (or use the matching env vars):

rag:
  embedding_model: "nomic-embed:v1.5"
  rerank_model: "bge-reranker:v2-m3"

Reranking — POST /v1/rerank

Reranking scores how relevant each document is to a query using a cross-encoder model. It runs on llama-server in --reranking mode via the rerank-server backend:

curl http://localhost:11435/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bge-reranker-v2-m3",
    "query": "How do I unload a model?",
    "documents": ["zallama unload <name> stops a model", "zallama pull fetches a model"],
    "top_n": 2,
    "return_documents": true
  }'

Returns a Cohere/Jina-style { "results": [{ "index", "relevance_score", "document"? }] }, sorted by score.

zvec vector store

zvec stores documents and their embeddings (HNSW index, persisted under rag.zvec_dir). It embeds and searches by calling Zallama's own /v1/embeddings, so it just needs a default embedding model — set rag.embedding_model (or ZALLAMA_EMBEDDING_MODEL). A query can optionally rerank its candidates.

# From the CLI
zallama zvec create notes                       # uses rag.embedding_model
zallama zvec upsert notes ./docs.txt            # one document per line (or a JSON array)
zallama zvec query notes "how to unload a model" --top-k 3 --rerank bge-reranker-v2-m3
zallama zvec collections

# Or over HTTP
curl http://localhost:11435/v1/zvec/notes/query \
  -H "Content-Type: application/json" \
  -d '{"query": "how to unload a model", "top_k": 3, "rerank_model": "bge-reranker-v2-m3"}'

Configure defaults in the rag block of config/config.yaml (embedding_model, rerank_model, zvec_dir, default_top_k).

Zallama acts as a standard OpenAI-compatible API gateway. Specify the model you want in the request body, and Zallama handles loading and routing automatically — meaning any OpenAI SDK, LangChain, or existing integration can point here with a one-line base-URL change.

curl http://localhost:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5-4b-q4_k_m",
    "messages": [
      {"role": "user", "content": "Tell me a joke."}
    ],
    "stream": true
  }'

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11435/v1",
    api_key="not-needed-on-localhost",
)

stream = client.chat.completions.create(
    model="qwen3.5-4b-q4_k_m",
    messages=[{"role": "user", "content": "Tell me a joke."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:11435/v1",
  apiKey: "not-needed-on-localhost",
});

const stream = await client.chat.completions.create({
  model: "qwen3.5-4b-q4_k_m",
  messages: [{ role: "user", content: "Tell me a joke." }],
  stream: true,
});
for await (const chunk of stream)
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");

With an API key set, add Authorization: Bearer <key> to every request (see Security). Interactive examples for every endpoint live on the home page playground.

For an always-on daemon, install Zallama as a systemd service. Running the installer as root writes the unit automatically; otherwise install it manually:

sudo tee /etc/systemd/system/zallama.service >/dev/null <<EOF
[Unit]
Description=Zallama — Local LLM Server
After=network.target

[Service]
Type=simple
User=$USER
WorkingDirectory=$PWD
ExecStart=$PWD/zallama serve
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now zallama
systemctl status zallama

The daemon runs two listeners in one process. The inference port (11435) serves only /v1/*, /health and /. Everything that manages the daemon — the calls the CLI makes for ps, list, load, pull, monitor — lives on the admin port (admin_port, default port + 1 → 11436), together with Prometheus /metrics. Both share the same API key; /health and /metrics are the only unauthenticated paths.

# What the CLI does under the hood
curl http://localhost:11436/api/ps -H "Authorization: Bearer $KEY"
curl http://localhost:11436/api/requests -H "Authorization: Bearer $KEY" | jq '.[0]'
curl -X POST http://localhost:11436/api/models/qwen3.5-4b-q4_k_m/load -H "Authorization: Bearer $KEY"

# Register a vision model over HTTP
curl -X POST http://localhost:11436/api/models/add \
  -H "Content-Type: application/json" -H "Authorization: Bearer $KEY" \
  -d '{"name":"qwen2-vl-7b","file":"Qwen2-VL-7B-Instruct-Q4_K_M.gguf",
       "artifacts":{"mmproj":"mmproj-Qwen2-VL-7B-Instruct-f16.gguf"},
       "params":{"ctx_size":8192,"n_gpu_layers":99}}'

The CLI finds the admin port from the same config file; point it elsewhere with ZALLAMA_ADMIN_HOST=http://host:port. Anything else that called /api/* on the inference port before v1.16 — scripts, dashboards, scripts/agentic_benchmark.py --admin-url — needs the new port.

Owning your AI means owning the responsibility for it running safely. Zallama defaults to localhost-only (host: 127.0.0.1) with no authentication, which is safe for single-user local use — and gets more deliberate the further you push it outward.

API key

Set host: "0.0.0.0" only together with an API key. The easiest and safest way is the CLI:

zallama apikey                     # generate a key valid 30 days (default)
zallama apikey --expires 90d       # or 12h / 45m / 2026-09-01 / never
zallama apikey hash <your-key>     # keep a key you chose yourself
zallama apikey clear               # disable auth

zallama apikey generates a 256-bit random key, writes only its SHA-256 hash (plus the expiry instant) into config.yaml, chmods the file to 600, and prints the key once — it cannot be recovered afterwards, so store it in your password manager. Restart the daemon to apply. Expired keys are rejected with a 401 until you issue a new one.

With a key set, everything except / and /health — including the API docs — requires an Authorization: Bearer <key> header, verified hash-to-hash with a constant-time comparison:

curl http://localhost:11435/v1/models -H "Authorization: Bearer $YOUR_KEY"

A plaintext api_key config value is still honored for backward compatibility, but api_key_sha256 is preferred and wins if both are set. CORS allows all origins but does not send credentials.

Remote access: prefer a VPN

For personal remote access, the most secure option is to not expose Zallama publicly at all: keep host: 127.0.0.1 and bind to a WireGuard/Tailscale interface, reaching it over the VPN. Zero public attack surface, no TLS or auth to configure.

Public exposure: reverse proxy

If it must face the internet, keep Zallama on 127.0.0.1, set an API key anyway (defense in depth), and put a TLS-terminating proxy in front. Example with Caddy (automatic Let's Encrypt certificates), exposing only the OpenAI-compatible inference surface:

llm.example.com {
    # Only /v1 passes through; the management API and /metrics live on the
    # separate admin port (11436) and never reach the proxy at all
    @api path /v1/*
    reverse_proxy @api 127.0.0.1:11435
    respond 403

    request_body {
        max_size 25MB   # enough for audio uploads; tune down if text-only
    }
}

Additional layers worth adding:

• Firewall: allow only 80/443 inbound (ufw allow 80,443/tcp then ufw enable).
• Ban abusive clients: fail2ban or CrowdSec keyed on repeated 401/403 responses.
• mTLS: if your clients support client certificates, require them at the proxy — anonymous scanners can't even complete a TLS handshake.
• systemd sandboxing: run as an unprivileged user with the hardening options below.

[Service]
User=zallama
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/home/zallama/.zallama
PrivateTmp=yes

Contributions are welcome! A good shape for a PR:

1. Fork and branch from main.
2. Keep changes focused; match the surrounding code style.
3. For a new backend/modality, add a Backend subclass in server/backends.py and the matching endpoint proxy — the process manager and registry schema should not need changes.
4. Update CHANGELOG.md under [Unreleased].
5. Open a PR describing the change and how you tested it.

Zallama is released under the MIT License. © 2026 Rija Z.

Measured write-ups from the repository: