Agent settings for small models (in plain words)

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217

# Agent settings for small models (in plain words)

*[Version française](reglages-agents-petits-modeles.md)*

> This guide explains, without jargon, what each setting of a mini-me agent does, why it exists, what its default value is, and how to set it **when you run a small language model locally** (one that fits on your laptop, not a giant model in the cloud).

## Why these settings exist

A small local model (a few billion parameters, running on a consumer CPU/GPU) does not have the same weaknesses as a large cloud model:

- it is **slow**, above all when it reads a long prompt ("prompt processing");
- it has a **limited memory (context window)**, often 8k to 32k tokens, sometimes less;
- it **loops or drifts** more easily (repeats the same command, forgets what it already read, spins off in its reasoning);
- it can **hallucinate** the result of a command it never actually ran.

The settings below are the safeguards that make up for this. They tell the agent "stop if you are too slow", "do not keep everything in memory", "do not go round in circles forever", "do not send back a wall of text that clogs your own memory".

They go in the agent's YAML file (`agent.*.yaml`), under the same key names.

---

## 1. `contextWindow`: the size of the model's memory

**In plain words**: the size of the model's "working memory", in tokens (one token ≈ 0.75 word). Everything the agent said, read and ran must fit in it. When it is full, room must be made (see the `context` section below).

- **Default value**: `0`, meaning "I don't know, ask the server". If the server (llama.cpp, Docker Model Runner…) reports the context size it was configured with, mini-me uses it automatically.
- **How to set it**: usually, **do not set it yourself** when your server can report it: leave `0`. Set it by hand only when your server does not report it, and then use the real value you configured on the server (e.g. `-c 8192` on `llama-server` → `contextWindow: 8192`).
- **For a small model**: small models often come with an advertised window of 32k or more, but **in practice their quality drops well before** that limit. Knowing the real usable value keeps the context-compression calculations (section 8) reliable.

---

## 2. `firstTokenTimeout`: how long the model may take to *start* answering

**In plain words**: before the model produces its first word, the server must *read* the whole prompt (the question + the history + the instructions). On a small local model, reading a 20-30k-token prompt can take several minutes. This setting says: "how long do I wait before deciding that it crashed?"

- **Default value**: `3 minutes`.
- **Special feature**: when the server (llama.cpp) reports its reading progress meanwhile, the clock **restarts** at each report. So in practice it will not cut as long as the server is making progress: this timeout mostly protects against a real, silent hang.
- **How to set it**:
  - Small model (≤ 8B) with a good GPU: the default (3 min) is usually enough.
  - Larger model (15-30B), modest machine (CPU only, little VRAM), or very long prompts (big `AGENTS.md`, long history): raise it to `5m` or even `10m`.
  - Golden rule: **never shorter than `watchdogTimeout`** (see just below), or the protection makes no sense.

---

## 3. `watchdogTimeout`: the watchdog while the model *generates*

**In plain words**: once the model has started answering (or using a tool), this setting checks that it keeps producing something regularly. If it stays silent longer than this, *with nothing new coming out*, mini-me decides the connection is stuck and cuts it.

- **Default value**: `20 seconds`.
- **Difference from `firstTokenTimeout`**: `firstTokenTimeout` watches the **start** (reading the prompt), `watchdogTimeout` watches **what follows** (token-by-token generation, or a tool running). That is why `watchdogTimeout` is much shorter.
- **How to set it**:
  - If the agent has the `bash` tool and commands may legitimately take time (build, tests, downloads), **raise this timeout together with `bashTimeout`**: while a bash command runs, the watchdog keeps counting. If `bashTimeout` is `2m`, `watchdogTimeout` must be at least `2m` too, or mini-me would cut the command before it ends.
  - For a simple "chat" use without heavy tools: the default is fine, even on a slow small model (token-by-token generation is what counts, not overall speed).

---

## 4. `bashTimeout`: how long a shell command may run

**In plain words**: when the agent runs a command (`bash`), this is the longest it may run before mini-me kills it, even if it has printed nothing.

- **Default value**: `30 seconds`.
- **How to set it**:
  - Quick tasks (reading a file, `git status`, `ls`): the 30 s default is plenty.
  - Heavy tasks you expect (compiling, tests, installing dependencies, a long search): raise it generously, for example `2m`, `5m`, or more depending on the project.
  - **Always raise `watchdogTimeout` at the same time** (see section 3). It is the most common trap: a command is killed by the watchdog although `bashTimeout` gave it time.
  - This setting depends less on the size of the model than on the kind of project the agent works on.

---

## 5. `maxTurns`: how many model ↔ tool round trips one question may take

**In plain words**: to answer one question, the agent may go round several times: "I run a command" → "I look at the result" → "I run another one" → and so on. This setting caps the number of rounds before mini-me stops the agent (so that it cannot loop forever).

- **Default value**: `10`.
- **Value actually used in the shipped configurations**: `40`. Experience shows that 10 is often too low for realistic coding tasks with a small model, which needs more iterations (re-reading, fixing, trying again) than a large one.
- **How to set it**:
  - Narrow task, a single action expected (a factual question, one tool call): `8` to `15` is enough, and it protects better against loops.
  - General coding agent (exploring code, editing, testing): `40` to `60` is reasonable, especially with a small model that needs several tries.
  - The smaller or weaker the model, the more likely it is to loop without ever concluding: pair a generous `maxTurns` with good loop detection rather than setting a huge value "just in case".

---

## 6. `maxOutput`: the most a tool may send back to the model

**In plain words**: if a `bash` command returns 500,000 characters (a big log file, for example), we do not want it to fill the model's whole memory at once. This setting truncates the result before the model sees it.

- **Default value**: `16000` characters.
- **How it truncates**: not just by cutting the end. mini-me keeps the **beginning** of the output and up to 2000 characters of the **end**, because the useful information (an error, a summary) is often at the end of a long output.
- **How to set it**:
  - Small model with a small context window (8k-16k tokens): **lower** it, for example to `4000`-`8000`, so that one command cannot use up a third of the available memory.
  - Model with a large context window, or an agent that really needs to see whole outputs (e.g. analysing a big diff): you can raise it to `20000`-`30000`, but remember that every output "costs" tokens and brings forward the moment the context compression (section 8) kicks in.

---

## 7. `agentsMd`: how the agent reads its project instructions

**In plain words**: `AGENTS.md` (or an equivalent file) holds the project's own instructions, which the agent must follow. This block controls whether and how that file is loaded.

```yaml
agentsMd:
  enabled: true
  path: "AGENTS.md"
  maxChars: 0
```

- **`enabled`**, default `true`: reads the file automatically.
- **`path`**, default `"AGENTS.md"`: the name or path of the file to read.
- **`maxChars`**, default `0` (= no limit).

### The important point about `maxChars`

It is **not** a truncation. When `AGENTS.md` is longer than `maxChars` characters, it is **not cut**: it is **left out entirely** (the source code says why: cutting in the middle could drop the very rule that matters). The start-up banner says so (`skipped (too long)`) rather than pretending the whole file was read.

- **How to set it**:
  - **Even without `maxChars`, mini-me warns you**: the start-up banner warns when `AGENTS.md` takes **20 % or more** of a known context window (estimate: number of characters ÷ 3.5). With a small model, that is the sign to shorten the file.
  - **`maxChars` is a hard limit on top of that warning.** With a small model and a small window, a long `AGENTS.md` can take a large share of the memory before the conversation even starts. Setting `maxChars` guarantees it never goes beyond: above it, the file is left out, the banner says so, and you know it must be shortened.
  - To compute a value: 20 % of the window × 3.5. Example: an 8192-token window → 20 % = 1638 tokens → about `maxChars: 5700`.
  - If your `AGENTS.md` is short (a few hundred lines at most) and your model has a comfortable window (32k+), you can leave `0`.

---

## 8. `context`: automatic compression of the history

**In plain words**: as the conversation goes on, the history (questions, answers, command results) grows. On a small model with limited memory, there comes a point where it no longer fits. This block decides **when** and **how** the older exchanges are summarised automatically to make room, without losing the thread.

```yaml
context:
  enabled: false
  threshold: 75
  maxMessages: 80
  keepLastTurns: 3
  summaryMaxTokens: 1200
  prompt: ""
  showStats: true
```

| Sub-setting | Default | What it does | How to set it |
|---|---|---|---|
| `enabled` | `false` | Turns automatic compression on. Even when it is off, you can still compress by hand (`/compact`). | **Turn it on (`true`) for any small local model**: it is the main protection against a "full context" that makes the agent fail or degrade. |
| `threshold` | `75` (%) | Share of the context window beyond which compression starts. 75 % deliberately leaves a quarter of the memory free for the next question, the tool results and the answer. | Not too low (you would compress too often and lose continuity); not too high (you risk filling up before compression can act). 70-80 % is a good range. |
| `maxMessages` | `80` | Fallback trigger based on the *number* of messages, used when the window size is unknown (`contextWindow: 0` and a server that does not say). One command = 2 messages (the call + the result), so 80 ≈ 30 commands of history. | The code justifies 80 as roughly the point where a 12B local model starts to slow down. On a smaller model, you can lower it if the agent gets slower or confused before 80 messages. Set `0` to turn this trigger off when you are sure of `contextWindow`. |
| `keepLastTurns` | `3` | Number of most recent turns kept **as they are** (not summarised): that is where the agent is actively working, and summarising it would make the agent redo what it has just done. | Keep at least `2`-`3`. Raise it a little (`4`-`5`) if the agent "forgets" a very recent action right after a compression. |
| `summaryMaxTokens` | `1200` | Largest summary generated to replace the old exchanges. The cap stops a runaway model from writing a summary that itself fills the window it was meant to empty. | For a small model, do not raise it much: a summary must stay compact. `1200` suits most cases; go down to `600`-`800` to save even more room. |
| `prompt` | `""` (built-in prompt) | Replaces the prompt used to ask for the summary. | Leave it empty unless you have a specific need (e.g. a summary that keeps a particular format). |
| `showStats` | `true` | Prints an information line after each compression (how many tokens were saved, etc.). | Leave it on: it shows that compression does its job. Turn it off only for a quieter output. |

**Practical tip**: the summary is requested **without the tools and without "reasoning" mode**, precisely so that `summaryMaxTokens` is not wasted on thinking instead of a useful summary.

---

## Summary: typical profiles

| Situation | `firstTokenTimeout` | `watchdogTimeout` | `bashTimeout` | `maxTurns` | `maxOutput` | `context.enabled` |
|---|---|---|---|---|---|---|
| Small model (≤ 8B), short tasks/chat, no `bash` or commands under 20 s | `3m` (default) | `20s` (default) | `30s` (default) | `15` | `8000` | `true` |
| Small/medium model, general coding agent | `5m` | `2m` (aligned on bashTimeout) | `2m` | `40` | `16000` | `true` |
| Larger model or modest machine (CPU) | `10m` | `5m` | `5m` | `40`-`60` | `16000`-`24000` | `true` |

In the first row, `watchdogTimeout` (20 s) is shorter than `bashTimeout` (30 s): those are the defaults, and they only fit when no command runs longer than 20 s. As soon as the agent runs longer commands, align the two as in the other rows (section 3).

These profiles are starting points, not absolute truths: adjust them by watching how your model actually behaves (does it loop, is it cut too early, does the context fill up too fast).

---

## 9. Sampling: how the model picks its words

**In plain words**: "sampling" is the set of settings that control how the model picks the next word among all the ones it finds plausible. It decides whether the model is **precise and predictable** or **creative and varied**. Unlike the settings above (timeouts, quotas), sampling has no fixed structure in mini-me: it is a free list of keys (`sampling:` in the YAML) passed almost as is to the model's server.

```yaml
sampling:
  temperature: 0.0
  top_p: 0.9
  max_tokens: 4096
```

These are the only three keys mini-me sets by default. The others (`top_k`, `min_p`, `repeat_penalty`…) have no default: add them yourself if your server and model support them and you need them.

### `temperature`: the "precise ↔ creative" dial

**In plain words**: the closer it is to `0`, the more the model always picks the word it finds most likely (stable, reproducible answers, sometimes a little rigid). The higher it goes towards `1` (or beyond), the more it accepts less obvious words (more varied answers, but also more risk of mistakes and drift).

- **mini-me default**: `0.0`, deliberately very low, because mini-me is designed as a **coding agent**, where precision and reproducibility matter, not creativity.
- **How to set it for a small model**:
  - An agent that runs commands, writes code, follows strict instructions: **keep `0.0`-`0.2`**. A small model already makes more mistakes and loops more than a large one; a low temperature reduces that risk.
  - A conversational or creative agent, or one that must rephrase or brainstorm: go up to `0.5`-`0.8`. Beyond `0.8`, a small model more easily turns incoherent.

### `top_p` (nucleus sampling): how wide the choice is

**In plain words**: instead of considering every possible word, the model only looks at the smallest group of words whose combined probability reaches this share. `top_p: 0.9` means "only consider the words that together cover 90 % of the probability; ignore the long tail of unlikely ones".

- **mini-me default**: `0.9`.
- **How to set it**: together with a low `temperature`, `0.9` is already a good "safe" compromise for a small model. You can go down to `0.8` to cut odd detours further; there is no point going higher for a coding agent.

### `top_k`: limiting the number of candidates (optional, no default in mini-me)

**In plain words**: keep only the `k` most likely words at each step and ignore the rest, whatever `top_p` says.

- **How to set it**: small models (often small Qwen, Gemma or Llama) do not all react to `top_k` the same way, and some model families explicitly recommend a value (for example `20`-`40`) in their model card. **Check the metadata or recommendations of the GGUF model you use** rather than guessing: a poorly chosen `top_k` can make a small model more repetitive or, on the contrary, more unstable.

### `repeat_penalty` / `min_p`: against repetition (optional, no default in mini-me)

**In plain words**: `repeat_penalty` penalises the model for reusing a word it used recently (useful against repetition loops, a classic weakness of small models). `min_p` is an alternative to `top_p` that drops the words whose probability is too low compared with the most likely word at that moment.

- **How to set them**: these are typically the settings that **vary most from one model family to another**; there is no universal value. Good practice is to use the values the model card recommends (often in the GGUF metadata or the model's documentation) rather than improvising. If the agent repeats sentences or actions in a loop, this is often the first setting to adjust, together with mini-me's loop detector and `maxTurns` (section 5).

### `max_tokens`: the longest an answer may be

**In plain words**: how many tokens the model may generate in one answer before it is cut.

- **mini-me default**: `4096`.
- **How to set it**: for a small model with a small context window, check that `max_tokens` leaves enough room in the window for the prompt itself: `max_tokens` + prompt size must not exceed `contextWindow`. Lower it (`1024`-`2048`) if your answers must stay short (e.g. classification, extraction); keep the default for an agent that writes code or longer explanations.

### In short: mini-me's sampling philosophy

mini-me assumes that a **small local model is a coding agent first**, so its default sampling favours **precision and reproducibility** (`temperature: 0.0`) over creativity. It is the safest starting point to limit mistakes, hallucinated command results and loops; loosen it yourself if your use is more conversational or creative.

> mini-me has no automatic "sampling profile" per model family today: the defaults are generic and deliberately conservative. When you switch models, check these settings again rather than keeping another model's out of habit.