bots-garden/houlapublic Fork
⑂ main
Commits
⬇ Clone ▾
git clone https://git.rickub.com/bots-garden/houla.git
git clone ssh://git@rickub.com/bots-garden/houla.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Download ZIP
🛟 Updated. 6dcdaeb · on main · k33g · 1h ago
olala-sbx.en.md · 123 lines · 5.3 KBmarkdown
Blame HistoryOpen raw

Serve Olala-v1.0-mini-7A1B in an sbx sandbox

This guide creates a sandbox that serves an Olala GGUF model with llama-server (OpenAI-compatible API), reachable from the host. llama-server comes from OVHai's llama.cpp fork (CPU only), shipped by the kits/llama-cpp-olala kit.

Upstream llama.cpp, Ollama and LM Studio do not support the olala architecture yet: the OVHai-LLM/llama.cpp fork is required (see the GGUF model card).

Create the sandbox

On the host, at the project root:

sbx env create sbxenv.yaml

The development sandbox (Claude Code + Go/Python/mini-me toolchains, houla-houla) is described separately in sbxenv.dev.yaml: sbx env run sbxenv.dev.yaml.

The postCreate hook runs scripts/olala.sh up in the sandbox:

  1. uses the kit's llama-server (without the kit, it builds the fork: a few minutes);
  2. downloads Olala-v1.0-mini-7A1B-q8_0-safe2.gguf (6.9 GB) into ./models, in the repository;
  3. starts llama-server on 0.0.0.0:8080, published on host port 8080.

Each step is skipped when already done (llama-server found, model present, server running). Models stay in ./models (ignored by git) when the sandbox is removed: a new sandbox does not download them again.

The repository folder is on kDrive, so the models are synced too. Exclude models/ from kDrive sync if you do not want that.

Options (pass them again to every sbx env command of this sandbox):

Argument Default Purpose
model_repo OVHaiLLM/Olala-v1.0-mini-7A1B-GGUF Hugging Face repository
model_file Olala-v1.0-mini-7A1B-q8_0-safe2.gguf GGUF file
models_dir ./models Models folder (relative to the repository root, or absolute)
host_port 8080 Host port
threads 16 CPU threads (keep 2 CPUs free: using all of them is ~30% slower)
ctx 16384 Context size (shared by the 4 slots)
sbx env create sbxenv.yaml --env-arg model_file=Olala-v1.0-mini-7A1B-q5_k_m.gguf --env-arg host_port=9090

In an existing sandbox

Inside the sandbox (or through sbx env exec sbxenv.yaml -- bash -lc '$SBX_WS/scripts/olala.sh <command>'):

scripts/olala.sh up        # build + download + serve
scripts/olala.sh status    # state and /health
scripts/olala.sh restart   # after a model change
scripts/olala.sh stop
scripts/olala.sh logs

Switch model:

OLALA_MODEL_FILE=Olala-v1.0-mini-7A1B-q6_k.gguf scripts/olala.sh download
scripts/olala.sh stop
OLALA_MODEL_FILE=Olala-v1.0-mini-7A1B-q6_k.gguf scripts/olala.sh serve

The same settings exist as environment variables: OLALA_MODELS_DIR, OLALA_THREADS, OLALA_CTX, OLALA_PORT, OLALA_LOAD_MODE, OLALA_EXTRA_ARGS… (listed at the top of scripts/olala.sh).

Model loading: the script runs llama-server --load-mode none (the model is read into RAM, ~7 GB). With mmap (llama.cpp's default) the file stays on the repository mount and generation drops to ~2 tokens/s.

A local sandbox stops as soon as no session keeps it running, and llama-server with it. The postCreate hook therefore switches the sandbox to detached mode (sbx run -d --name houla-olala, permanent for this sandbox): it keeps running until sbx stop. After sbx stop or a reboot, start it again, then the server:

sbx run -d --name houla-olala
sbx env exec --no-input sbxenv.yaml -- bash -lc "$PWD/scripts/olala.sh serve"

For a sandbox created without sbxenv.yaml (e.g. houla-houla, the dev sandbox of sbxenv.dev.yaml), publish the port from the host:

sbx ports houla-houla --publish 8080:8080

Test with curl

From the host (or the sandbox):

# Health
curl -s http://localhost:8080/health

# Served models (name = file name without .gguf)
curl -s http://localhost:8080/v1/models

# Chat
curl -s http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Olala-v1.0-mini-7A1B-q8_0-safe2",
    "messages": [{"role": "user", "content": "Explain how a heat pump works in two sentences."}],
    "temperature": 0.8, "top_p": 0.9, "top_k": 40,
    "reasoning_effort": "low"
  }' | jq -r '.choices[0].message.content'

# Streaming, no reasoning
curl -sN http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Olala-v1.0-mini-7A1B-q8_0-safe2",
    "stream": true,
    "reasoning_effort": "disabled",
    "messages": [{"role": "user", "content": "Say hello"}]
  }'

reasoning_effort is disabled, low, medium (default) or high. The reasoning comes back in choices[0].message.reasoning_content.

Measured in an aarch64 sandbox (18 CPUs, 31 GB) with q8_0-safe2, the kit's llama-server, 16 threads: 35 to 44 tokens/s generation.

The llama-cpp-olala kit

An sbx mixin (kits v3) that builds the fork once, when the kit is built, and puts llama-server and llama-cli in /opt/llama-cpp-olala (links in /usr/local/bin). Portable build (GGML_NATIVE=OFF, every CPU variant loaded at runtime), arm64 (tested) and amd64 (not tested yet). The ref argument pins the fork commit.

To use it in another sandbox (syntax not yet tried with sbx):

sbx run claude --kit ./kits/llama-cpp-olala
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
# Serve Olala-v1.0-mini-7A1B in an sbx sandbox

This guide creates a sandbox that serves an Olala GGUF model with `llama-server` (OpenAI-compatible API), reachable from the host. `llama-server` comes from OVHai's llama.cpp fork (CPU only), shipped by the [`kits/llama-cpp-olala`](../kits/llama-cpp-olala/) kit.

> Upstream llama.cpp, Ollama and LM Studio do not support the `olala` architecture yet: the [OVHai-LLM/llama.cpp](https://github.com/OVHai-LLM/llama.cpp) fork is required (see the [GGUF model card](https://huggingface.co/OVHaiLLM/Olala-v1.0-mini-7A1B-GGUF)).

## Create the sandbox

On the host, at the project root:

```bash
sbx env create sbxenv.yaml
```

> The development sandbox (Claude Code + Go/Python/mini-me toolchains, `houla-houla`) is described separately in `sbxenv.dev.yaml`: `sbx env run sbxenv.dev.yaml`.

The `postCreate` hook runs [`scripts/olala.sh up`](../scripts/olala.sh) in the sandbox:

1. uses the kit's `llama-server` (without the kit, it builds the fork: a few minutes);
2. downloads `Olala-v1.0-mini-7A1B-q8_0-safe2.gguf` (6.9 GB) into `./models`, in the repository;
3. starts `llama-server` on `0.0.0.0:8080`, published on host port 8080.

Each step is skipped when already done (`llama-server` found, model present, server running). Models stay in `./models` (ignored by git) when the sandbox is removed: a new sandbox does not download them again.

> The repository folder is on kDrive, so the models are synced too. Exclude `models/` from kDrive sync if you do not want that.

Options (pass them again to every `sbx env` command of this sandbox):

| Argument | Default | Purpose |
| --- | --- | --- |
| `model_repo` | `OVHaiLLM/Olala-v1.0-mini-7A1B-GGUF` | Hugging Face repository |
| `model_file` | `Olala-v1.0-mini-7A1B-q8_0-safe2.gguf` | GGUF file |
| `models_dir` | `./models` | Models folder (relative to the repository root, or absolute) |
| `host_port` | `8080` | Host port |
| `threads` | `16` | CPU threads (keep 2 CPUs free: using all of them is ~30% slower) |
| `ctx` | `16384` | Context size (shared by the 4 slots) |

```bash
sbx env create sbxenv.yaml --env-arg model_file=Olala-v1.0-mini-7A1B-q5_k_m.gguf --env-arg host_port=9090
```

## In an existing sandbox

Inside the sandbox (or through `sbx env exec sbxenv.yaml -- bash -lc '$SBX_WS/scripts/olala.sh <command>'`):

```bash
scripts/olala.sh up        # build + download + serve
scripts/olala.sh status    # state and /health
scripts/olala.sh restart   # after a model change
scripts/olala.sh stop
scripts/olala.sh logs
```

Switch model:

```bash
OLALA_MODEL_FILE=Olala-v1.0-mini-7A1B-q6_k.gguf scripts/olala.sh download
scripts/olala.sh stop
OLALA_MODEL_FILE=Olala-v1.0-mini-7A1B-q6_k.gguf scripts/olala.sh serve
```

The same settings exist as environment variables: `OLALA_MODELS_DIR`, `OLALA_THREADS`, `OLALA_CTX`, `OLALA_PORT`, `OLALA_LOAD_MODE`, `OLALA_EXTRA_ARGS`… (listed at the top of [`scripts/olala.sh`](../scripts/olala.sh)).

> **Model loading:** the script runs `llama-server --load-mode none` (the model is read into RAM, ~7 GB). With `mmap` (llama.cpp's default) the file stays on the repository mount and generation drops to ~2 tokens/s.

A local sandbox stops as soon as no session keeps it running, and `llama-server` with it. The `postCreate` hook therefore switches the sandbox to detached mode (`sbx run -d --name houla-olala`, permanent for this sandbox): it keeps running until `sbx stop`. After `sbx stop` or a reboot, start it again, then the server:

```bash
sbx run -d --name houla-olala
sbx env exec --no-input sbxenv.yaml -- bash -lc "$PWD/scripts/olala.sh serve"
```

For a sandbox created without `sbxenv.yaml` (e.g. `houla-houla`, the dev sandbox of `sbxenv.dev.yaml`), publish the port from the host:

```bash
sbx ports houla-houla --publish 8080:8080
```

## Test with curl

From the host (or the sandbox):

```bash
# Health
curl -s http://localhost:8080/health

# Served models (name = file name without .gguf)
curl -s http://localhost:8080/v1/models

# Chat
curl -s http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Olala-v1.0-mini-7A1B-q8_0-safe2",
    "messages": [{"role": "user", "content": "Explain how a heat pump works in two sentences."}],
    "temperature": 0.8, "top_p": 0.9, "top_k": 40,
    "reasoning_effort": "low"
  }' | jq -r '.choices[0].message.content'

# Streaming, no reasoning
curl -sN http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Olala-v1.0-mini-7A1B-q8_0-safe2",
    "stream": true,
    "reasoning_effort": "disabled",
    "messages": [{"role": "user", "content": "Say hello"}]
  }'
```

`reasoning_effort` is `disabled`, `low`, `medium` (default) or `high`. The reasoning comes back in `choices[0].message.reasoning_content`.

Measured in an aarch64 sandbox (18 CPUs, 31 GB) with q8_0-safe2, the kit's `llama-server`, 16 threads: 35 to 44 tokens/s generation.

## The `llama-cpp-olala` kit

An sbx mixin (kits v3) that builds the fork once, when the kit is built, and puts `llama-server` and `llama-cli` in `/opt/llama-cpp-olala` (links in `/usr/local/bin`). Portable build (`GGML_NATIVE=OFF`, every CPU variant loaded at runtime), arm64 (tested) and amd64 (not tested yet). The `ref` argument pins the fork commit.

To use it in another sandbox (syntax not yet tried with `sbx`):

```bash
sbx run claude --kit ./kits/llama-cpp-olala
```