How to run mini-me on Kronk

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88

# How to run mini-me on Kronk

This guide shows how to point `mm` at [Kronk](https://www.kronkai.com), the Go inference server of Ardan Labs, through its OpenAI-compatible endpoint. It assumes Kronk is installed (`brew install ardanlabs/kronk/kronk`, or `go install github.com/ardanlabs/kronk/cmd/kronk@latest`) and that you know how to build `mm`.

## Steps

1. Pull the model. Kronk loads an installed model on the first request, but a model that is not installed is a `model not found` error:

   ```bash
   kronk model pull mradermacher/Qwopus3.5-4B-Coder.Q8_0
   ```

2. Start the server. It listens on `127.0.0.1:11435`:

   ```bash
   kronk server start
   ```

3. Run the demo configuration from its directory (its `skillsDir` is `../../skills`):

   ```bash
   cd demos/kronk
   ../../mm -config agent.kronk.yaml
   ```

4. Check the banner. `provider: kronk` confirms the provider, and `ctx: 131072 (/v1/kronk/models)` is the window Kronk serves the model with, read at start-up whether the model is loaded or not.

## Variants

- **Another model.** Set `model:` to the id exactly as Kronk lists it on `GET /v1/models`: `provider/modelID`. An id without its provider is refused by Kronk with `must use provider/modelID`. **The id you pull with is not always the id Kronk serves.** For example, Qwen3.5 2B, a smaller model:

  ```bash
  kronk model pull unsloth/Qwen3.5-2B-GGUF:Q4_K_M --local   # --local: with the server stopped
  curl -s localhost:11435/v1/models                          # lists it as unsloth/Qwen3.5-2B-Q4_K_M
  ```

  ```yaml
  model: unsloth/Qwen3.5-2B-Q4_K_M    # not unsloth/Qwen3.5-2B-GGUF:Q4_K_M, which is a 404
  ```

  Measured on Kronk 1.33.2: `ctx: 131072 (/v1/kronk/models)`, and tool calls work.
- **A Kronk profile.** Kronk also lists configuration profiles, `provider/modelID/PROFILE`, for example `mradermacher/Qwopus3.5-4B-Coder.Q8_0/AGENT`. A profile is the same GGUF file, served with the settings of that key in `~/.kronk/models/model_config.yaml`. Nothing is downloaded twice. The `AGENT` profiles come with Kronk's default `model_config.yaml`; you can add your own there, then restart Kronk. The one for Qwopus reads:

  ```yaml
  mradermacher/Qwopus3.5-4B-Coder.Q8_0/AGENT:
    context-window: 65536
    sampling-parameters:
      max_tokens: 8192
      temperature: 1
      top_k: 20
      top_p: 0.95
  ```

  A profile is a valid `model:`. What it changes for `mm`:

  - **The window: yes.** It is the profile's own, here 65536 instead of the 131072 Kronk picks for the plain model. The banner shows it (`ctx: 65536 (/v1/kronk/models)`), and compaction triggers at 75 % of it. A smaller window also takes less memory on the server.
  - **Sampling: only for the keys `mm` does not send.** A value in the request wins over the profile's: measured on Kronk 1.33.2, `temperature: 0` in the request gave the same answer three times in a row under the profile's `temperature: 1`, and `max_tokens: 5` cut the answer at 5 tokens under the profile's 8192. `mm` always sends `temperature`, `top_p` and `max_tokens`: its defaults, merged with the file's `sampling` (see [`sampling`](../reference/configuration.md)). So the profile's `temperature`, `top_p` and `max_tokens` never apply, and its other keys apply only when `sampling` leaves them out, here `top_k: 20`. This holds even for a file with no `sampling:` block: `mm` then sends `temperature: 0`, `top_p: 0.9`, `max_tokens: 4096` (checked in a trace). To get a profile's values, write them in `sampling`; `mm init` does it for you (below).
  - **Reasoning: no.** `enable_thinking` and `chat_template_kwargs` in `sampling` are per request too.

  `mm init` lists profiles with a line that says what they are, and the window shows them as `Qwopus3.5-4B-Coder.Q8…/AGENT`. When you pick a profile, `mm init` reads its sampling from Kronk and copies it into the file's `sampling`. The copied values are `temperature`, `top_p`, `top_k`, `min_p`, and `max_tokens` when the profile caps it. They replace the family table's and the GGUF's values, each marked `[server] Kronk profile AGENT`:

  ```yaml
  sampling:
    # [server] Kronk profile AGENT (model_config.yaml), read from the server:
    # mm's values replace a profile's, so they are copied here
    temperature: 1.0
    top_p: 0.95
    top_k: 20
    min_p: 0.0
    max_tokens: 8192
  ```
- **Write the file with the wizard.** `mm init --provider kronk` lists Kronk's models and reads the chosen one's window, sampling and chat template from Kronk. See [How to create an agent](create-an-agent.md).
- **Let mm pull the model.** With `autoPull: true`, mm pulls a model Kronk does not list at start-up, through `POST /v1/kronk/models/pull`, and shows the progress. See [`autoPull`](../reference/configuration.md).
- **From a Docker sandbox.** `kronk server start` binds the loopback only, which a sandbox cannot reach. On the host, start it with `kronk server start --api-host 0.0.0.0:11435`, and allow `localhost:11435` in the sandbox's network policy. With no `baseUrl` answering, `mm` then falls back to `http://host.docker.internal:11435/v1` by itself.
- **A larger or smaller window.** The window is Kronk's, per model: set `context-window` for the model in `~/.kronk/models/model_config.yaml` and restart Kronk. `mm` reads the new value at start-up.
- **A protected Kronk.** In the `authenticated` or `full-protected` modes, export a token in `KRONK_TOKEN` (or name another variable under `apiKeyEnv`). A refused token is a start-up warning naming the variable.
- **Free the memory.** `/unload` and the `unload_model` tool unload the model through `POST /v1/kronk/models/unload`. See [How to free a model's memory](unload-a-model.md).
- **Keep your own `agent.yaml`.** `AGENT_PROVIDER=kronk ./mm` works as long as the file leaves `baseUrl` empty.

## What differs from the other servers

- No `Reading the prompt …` line: Kronk sends no progress while it reads, so `firstTokenTimeout` has to cover the whole prompt and the model's loading.
- Reasoning is turned off with `chat_template_kwargs: {enable_thinking: false}`, as on llama.cpp; Kronk ignores `reasoning_effort: none`. The summary of a compaction does it by itself. See [How to turn a model's reasoning off](turn-reasoning-off.md).

## See also

- [Providers: one implementation per protocol](../explanation/providers.md) — what was measured on Kronk.
- [Configuration reference](../reference/configuration.md) — provider defaults.