bots-garden/mini-me · run-with-llama-cpp.md

⑂ main

run-with-llama-cpp.md · 33 lines · 2.1 KBmarkdown    Blame HistoryRaw
   1# How to run mini-me on llama.cpp instead of Docker Model Runner
3This guide shows how to point `mm` at a `llama-server` you start yourself. It assumes you already know how to build `mm` and how to obtain a GGUF model.
71. Start `llama-server` with a chat template and tool calling enabled. `--jinja` is mandatory: without it the server refuses the `tools` parameter and the agent cannot run a single command. Give the served model an alias with `-a`, because `llama-server` routes requests on the `model` field:
9
10   llama-server -hf jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M \
11                -a jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M \
12                --jinja -c 32768 --port 8080
152. Use the shipped llama.cpp configuration file. Its `model:` key must match the `-a` alias above:
17
18   ./mm agent.llamacpp.yaml
213. Check the banner. `provider: llamacpp` confirms the provider, and `ctx: 32768 (/props)` confirms the agent read the served context size from the server's `/props` endpoint.
23## Variants
25- **Different port or host.** Set `baseUrl` in the YAML to `http://<host>:<port>/v1`, or override it for one run with `AGENT_BASE_URL=http://<host>:<port>/v1 ./mm agent.llamacpp.yaml`. An environment override is taken as-is, without the fallback probe.
26- **Server started with `--api-key`.** Export the key in `LLAMA_API_KEY` (the provider's default variable) or name another variable under `apiKeyEnv`. The key itself never goes in the YAML.
27- **Keep `agent.yaml` and switch provider for one run.** `AGENT_PROVIDER=llamacpp ./mm` works as long as `agent.yaml` leaves `baseUrl` empty; an explicit DMR URL in the file would still be honoured and point at port 12434.
28- **The server hides `/props`** (a reverse proxy, for instance). Set `contextWindow` to the value you started the server with, otherwise the banner shows `ctx: unknown`.
30## See also
32- Every key of the file: [configuration reference](../reference/configuration.md)
33- Why one provider implementation serves both servers: [providers](../explanation/providers.md)

💾 Saved. d722711 k33g 6h ago	1	# How to run mini-me on llama.cpp instead of Docker Model Runner
	2
	3	This guide shows how to point `mm` at a `llama-server` you start yourself. It assumes you already know how to build `mm` and how to obtain a GGUF model.
	4
	5	## Steps
	6
	7	1. Start `llama-server` with a chat template and tool calling enabled. `--jinja` is mandatory: without it the server refuses the `tools` parameter and the agent cannot run a single command. Give the served model an alias with `-a`, because `llama-server` routes requests on the `model` field:
	8
	9	```bash
	10	llama-server -hf jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M \
	11	-a jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M \
	12	--jinja -c 32768 --port 8080
	13	```
	14
	15	2. Use the shipped llama.cpp configuration file. Its `model:` key must match the `-a` alias above:
	16
	17	```bash
	18	./mm agent.llamacpp.yaml
	19	```
	20
	21	3. Check the banner. `provider: llamacpp` confirms the provider, and `ctx: 32768 (/props)` confirms the agent read the served context size from the server's `/props` endpoint.
	22
	23	## Variants
	24
	25	- Different port or host. Set `baseUrl` in the YAML to `http://<host>:<port>/v1`, or override it for one run with `AGENT_BASE_URL=http://<host>:<port>/v1 ./mm agent.llamacpp.yaml`. An environment override is taken as-is, without the fallback probe.
	26	- Server started with `--api-key`. Export the key in `LLAMA_API_KEY` (the provider's default variable) or name another variable under `apiKeyEnv`. The key itself never goes in the YAML.
	27	- Keep `agent.yaml` and switch provider for one run. `AGENT_PROVIDER=llamacpp ./mm` works as long as `agent.yaml` leaves `baseUrl` empty; an explicit DMR URL in the file would still be honoured and point at port 12434.
	28	- The server hides `/props` (a reverse proxy, for instance). Set `contextWindow` to the value you started the server with, otherwise the banner shows `ctx: unknown`.
	29
	30	## See also
	31
	32	- Every key of the file: [configuration reference](../reference/configuration.md)
	33	- Why one provider implementation serves both servers: [providers](../explanation/providers.md)