How to run mini-me on llama.cpp instead of Docker Model Runner

This guide shows how to point mm at a llama-server you start yourself. It assumes you already know how to build mm and how to obtain a GGUF model.

Steps

Start llama-server with a chat template and tool calling enabled. --jinja is mandatory: without it the server refuses the tools parameter and the agent cannot run a single command. Give the served model an alias with -a, because llama-server routes requests on the model field:
```
llama-server -hf jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M \
             -a jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M \
             --jinja -c 32768 --port 8080
```
Use the shipped llama.cpp configuration file. Its model: key must match the -a alias above:
```
./mm agent.llamacpp.yaml
```
Check the banner. provider: llamacpp confirms the provider, and ctx: 32768 (/props) confirms the agent read the served context size from the server's /props endpoint.

Variants

Different port or host. Set baseUrl in the YAML to http://<host>:<port>/v1, or override it for one run with AGENT_BASE_URL=http://<host>:<port>/v1 ./mm agent.llamacpp.yaml. An environment override is taken as-is, without the fallback probe.
Server started with --api-key. Export the key in LLAMA_API_KEY (the provider's default variable) or name another variable under apiKeyEnv. The key itself never goes in the YAML.
Keep agent.yaml and switch provider for one run. AGENT_PROVIDER=llamacpp ./mm works as long as agent.yaml leaves baseUrl empty; an explicit DMR URL in the file would still be honoured and point at port 12434.
The server hides /props (a reverse proxy, for instance). Set contextWindow to the value you started the server with, otherwise the banner shows ctx: unknown.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


# How to run mini-me on llama.cpp instead of Docker Model Runner

This guide shows how to point `mm` at a `llama-server` you start yourself. It assumes you already know how to build `mm` and how to obtain a GGUF model.

## Steps

1. Start `llama-server` with a chat template and tool calling enabled. `--jinja` is mandatory: without it the server refuses the `tools` parameter and the agent cannot run a single command. Give the served model an alias with `-a`, because `llama-server` routes requests on the `model` field:

   ```bash
   llama-server -hf jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M \
                -a jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M \
                --jinja -c 32768 --port 8080
   ```

2. Use the shipped llama.cpp configuration file. Its `model:` key must match the `-a` alias above:

   ```bash
   ./mm agent.llamacpp.yaml
   ```

3. Check the banner. `provider: llamacpp` confirms the provider, and `ctx: 32768 (/props)` confirms the agent read the served context size from the server's `/props` endpoint.

## Variants

- **Different port or host.** Set `baseUrl` in the YAML to `http://<host>:<port>/v1`, or override it for one run with `AGENT_BASE_URL=http://<host>:<port>/v1 ./mm agent.llamacpp.yaml`. An environment override is taken as-is, without the fallback probe.
- **Server started with `--api-key`.** Export the key in `LLAMA_API_KEY` (the provider's default variable) or name another variable under `apiKeyEnv`. The key itself never goes in the YAML.
- **Keep `agent.yaml` and switch provider for one run.** `AGENT_PROVIDER=llamacpp ./mm` works as long as `agent.yaml` leaves `baseUrl` empty; an explicit DMR URL in the file would still be honoured and point at port 12434.
- **The server hides `/props`** (a reverse proxy, for instance). Set `contextWindow` to the value you started the server with, otherwise the banner shows `ctx: unknown`.

## See also

- Every key of the file: [configuration reference](../reference/configuration.md)
- Why one provider implementation serves both servers: [providers](../explanation/providers.md)