How to check that the tools go out in a stable order

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131

# How to check that the tools go out in a stable order

Since ticket 53, `mm` sends the tools of every request sorted by name, so that a llama.cpp server can reuse the prompt it cached for the previous question. This guide shows how to check that change: that nothing regressed, that the order is stable, and how much prompt reading it saves on your own server and model.

It assumes a checkout of `mm`, the Go toolchain, `python3`, and a running Docker Model Runner or `llama-server` with a model that handles tool calls.

## 1. Run the test suite

From the repository root:

```bash
go test ./...
```

Every package must report `ok`. Then run the tests of this change on their own, several times — the bug was random, so a single green run proves little:

```bash
go test ./internal/engine -run 'TestToolsGoOut|TestStableTools' -count=10 -v
```

| Test | What it checks |
|------|----------------|
| `TestToolsGoOutInTheSameOrderOnEveryQuestion` | Twelve questions through the real `llamacpp` provider, against a fake server that records the requests: each one carries the seven tools sorted by name. |
| `TestStableToolsSortsByName` | The list comes out sorted. |
| `TestStableToolsLeavesTheOriginalAlone` | The slice Genkit owns is not reordered in place. |
| `TestStableToolsIgnoresTheIncomingOrder` | Every rotation of the same tools gives the same list. |
| `TestStableToolsWithNoneOrOne` | A request without tools (the summary request of `/compact`) is left untouched. |

To see the first test catch the bug, remove the line `params.Request.Tools = stableTools(params.Request.Tools)` from `internal/engine/engine.go` and run it again: it fails with lines like `question 2 sent the tools as [plan fetch_content bash …], want [bash edit_file …]`. Put the line back.

## 2. Build the binary before and after the change

```bash
go build -o /tmp/mm-after .
git worktree add /tmp/mm-before-src 1fa0c42
(cd /tmp/mm-before-src && go build -o /tmp/mm-before .)
git worktree remove /tmp/mm-before-src
```

`1fa0c42` is the last commit without the fix; any older commit works too.

## 3. Start the recording proxy

In a second terminal, start `tests/tool-order/proxy.py` in front of the model server:

```bash
python3 tests/tool-order/proxy.py --port 18770 --upstream http://localhost:12434
```

It prints `proxy on http://127.0.0.1:18770 -> http://localhost:12434`, then one line per request once `mm` starts asking.

## 4. Make a configuration that goes through the proxy

```bash
sed -e 's#^baseUrl:.*#baseUrl: http://127.0.0.1:18770/engines/v1#' \
    -e 's#^fallback:.*#fallback: ""#' \
    -e "s#^skillsDir:.*#skillsDir: $PWD/skills#" \
    agent.dmr.yaml > /tmp/agent.proxy.yaml
```

`fallback: ""` keeps `mm` on the proxy even when the probe is slow. The skills directory is made absolute because, in editor mode, a relative one is read from the configuration file's directory — without it `read_skill` would be missing and the request would not be the one you use every day.

## 5. Ask the same questions with each binary

`tests/tool-order/session.py` asks several questions in one session, through the editor protocol (`mm -acp`):

```bash
python3 tests/tool-order/session.py /tmp/mm-before /tmp/agent.proxy.yaml \
  "Answer in one short sentence: what is Go?" \
  "Answer in one short sentence: what is Rust?" \
  "Answer in one short sentence: what is Zig?" \
  "Answer in one short sentence: what is Odin?"
```

Each question prints `question N: {'stopReason': 'end_turn', …}`. Run the same command with `/tmp/mm-after`.

Keep the questions short and free of file changes: a question that makes the model write a file asks the editor for permission, and the script has no one to answer it.

## 6. Read the proxy's lines

Each line is one request:

```
#6 tools=f2f7a1fa ['bash', 'edit_file', 'fetch_content', …] cache_n=3381 prompt_n=20 prompt_ms=253
```

| Field | Meaning |
|-------|---------|
| `tools=` | A fingerprint of the tools block exactly as it was sent. |
| `[…]` | The tool names, in the order they were sent. |
| `cache_n` | Prompt tokens the server took from its cache. |
| `prompt_n` | Prompt tokens it had to read again. |
| `prompt_ms` | The time it spent reading them. |

What to look for:

- **After the change**, `tools=` has the same value on every line, and from the second question on `prompt_n` stays small (the new question and the previous answer) while `cache_n` covers the rest of the prompt.
- **Before the change**, `tools=` changes from one question to the next, and each time it changes `prompt_n` jumps back to the size of the whole prompt.

The first question of a session reads the whole prompt in both cases, unless the server still holds the same prefix from an earlier session.

Measured on 2026-09-30 with Mellum 12B on Docker Model Runner, four questions, `agent.dmr.yaml` (six tools, skills not loaded):

| Question | Before: `prompt_n` / `prompt_ms` | After: `prompt_n` / `prompt_ms` |
|----------|------------------|------------------|
| 1 | 3355 / 6776 | 3355 / 7141 |
| 2 | 3401 / 7186 | 20 / 253 |
| 3 | 107 / 713 (the order happened to match question 1) | 20 / 257 |
| 4 | 110 / 707 | 20 / 252 |

The gain grows with the session: before the change, a question whose tool order differs re-reads the whole history, not only the tools.

## 7. Check the rest by hand

The change touches every request, so check once that ordinary use still works with the new binary and your usual configuration:

- A question that needs a tool: `/tmp/mm-after -config agent.dmr.yaml -p "list the files of this directory"` runs `bash` and answers.
- A long session that compresses: `/compact` still writes a summary (its request has no tools and is left alone).
- An MCP server, if you use one: its tools are sorted with the built-in ones and still called by their names.

## Variants

- **llama-server**: start the proxy with `--upstream http://localhost:8080`, and build the configuration from `agent.llamacpp.yaml` with `baseUrl: http://127.0.0.1:18770/v1`.
- **`mm` or the server in a container**: use `--upstream http://host.docker.internal:12434` for a Docker Model Runner on the host.
- **Another model**: the prompt sizes change, the pattern should not. The numbers above are an example, not a target.

## See also

- Why the order matters: [providers — explanation](../explanation/providers.md)
- The configuration keys used here: [configuration reference](../reference/configuration.md)
- Running a single question: [run a single prompt](run-a-single-prompt.md)