Serve Olala-v1.0-mini-7A1B in an sbx sandbox
This guide creates a sandbox that serves an Olala GGUF model with llama-server (OpenAI-compatible API), reachable from the host. llama-server comes from OVHai's llama.cpp fork (CPU only), shipped by the kits/llama-cpp-olala kit.
Upstream llama.cpp, Ollama and LM Studio do not support the
olalaarchitecture yet: the OVHai-LLM/llama.cpp fork is required (see the GGUF model card).
Create the sandbox
On the host, at the project root:
sbx env create sbxenv.yaml
The development sandbox (Claude Code + Go/Python/mini-me toolchains,
houla-houla) is described separately insbxenv.dev.yaml:sbx env run sbxenv.dev.yaml.
The postCreate hook runs scripts/olala.sh up in the sandbox:
- uses the kit's
llama-server(without the kit, it builds the fork: a few minutes); - downloads
Olala-v1.0-mini-7A1B-q8_0-safe2.gguf(6.9 GB) into./models, in the repository; - starts
llama-serveron0.0.0.0:8080, published on host port 8080.
Each step is skipped when already done (llama-server found, model present, server running). Models stay in ./models (ignored by git) when the sandbox is removed: a new sandbox does not download them again.
The repository folder is on kDrive, so the models are synced too. Exclude
models/from kDrive sync if you do not want that.
Options (pass them again to every sbx env command of this sandbox):
| Argument | Default | Purpose |
|---|---|---|
model_repo |
OVHaiLLM/Olala-v1.0-mini-7A1B-GGUF |
Hugging Face repository |
model_file |
Olala-v1.0-mini-7A1B-q8_0-safe2.gguf |
GGUF file |
models_dir |
./models |
Models folder (relative to the repository root, or absolute) |
host_port |
8080 |
Host port |
threads |
16 |
CPU threads (keep 2 CPUs free: using all of them is ~30% slower) |
ctx |
16384 |
Context size (shared by the 4 slots) |
sbx env create sbxenv.yaml --env-arg model_file=Olala-v1.0-mini-7A1B-q5_k_m.gguf --env-arg host_port=9090
In an existing sandbox
Inside the sandbox (or through sbx env exec sbxenv.yaml -- bash -lc '$SBX_WS/scripts/olala.sh <command>'):
scripts/olala.sh up # build + download + serve
scripts/olala.sh status # state and /health
scripts/olala.sh restart # after a model change
scripts/olala.sh stop
scripts/olala.sh logs
Switch model:
OLALA_MODEL_FILE=Olala-v1.0-mini-7A1B-q6_k.gguf scripts/olala.sh download
scripts/olala.sh stop
OLALA_MODEL_FILE=Olala-v1.0-mini-7A1B-q6_k.gguf scripts/olala.sh serve
The same settings exist as environment variables: OLALA_MODELS_DIR, OLALA_THREADS, OLALA_CTX, OLALA_PORT, OLALA_LOAD_MODE, OLALA_EXTRA_ARGS… (listed at the top of scripts/olala.sh).
Model loading: the script runs
llama-server --load-mode none(the model is read into RAM, ~7 GB). Withmmap(llama.cpp's default) the file stays on the repository mount and generation drops to ~2 tokens/s.
A local sandbox stops as soon as no session keeps it running, and llama-server with it. The postCreate hook therefore switches the sandbox to detached mode (sbx run -d --name houla-olala, permanent for this sandbox): it keeps running until sbx stop. After sbx stop or a reboot, start it again, then the server:
sbx run -d --name houla-olala
sbx env exec --no-input sbxenv.yaml -- bash -lc "$PWD/scripts/olala.sh serve"
For a sandbox created without sbxenv.yaml (e.g. houla-houla, the dev sandbox of sbxenv.dev.yaml), publish the port from the host:
sbx ports houla-houla --publish 8080:8080
Test with curl
From the host (or the sandbox):
# Health
curl -s http://localhost:8080/health
# Served models (name = file name without .gguf)
curl -s http://localhost:8080/v1/models
# Chat
curl -s http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "Olala-v1.0-mini-7A1B-q8_0-safe2",
"messages": [{"role": "user", "content": "Explain how a heat pump works in two sentences."}],
"temperature": 0.8, "top_p": 0.9, "top_k": 40,
"reasoning_effort": "low"
}' | jq -r '.choices[0].message.content'
# Streaming, no reasoning
curl -sN http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "Olala-v1.0-mini-7A1B-q8_0-safe2",
"stream": true,
"reasoning_effort": "disabled",
"messages": [{"role": "user", "content": "Say hello"}]
}'
reasoning_effort is disabled, low, medium (default) or high. The reasoning comes back in choices[0].message.reasoning_content.
Measured in an aarch64 sandbox (18 CPUs, 31 GB) with q8_0-safe2, the kit's llama-server, 16 threads: 35 to 44 tokens/s generation.
The llama-cpp-olala kit
An sbx mixin (kits v3) that builds the fork once, when the kit is built, and puts llama-server and llama-cli in /opt/llama-cpp-olala (links in /usr/local/bin). Portable build (GGML_NATIVE=OFF, every CPU variant loaded at runtime), arm64 (tested) and amd64 (not tested yet). The ref argument pins the fork commit.
To use it in another sandbox (syntax not yet tried with sbx):
sbx run claude --kit ./kits/llama-cpp-olala
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 |
|