Measure whether the model picks the right skill

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

# Measure whether the model picks the right skill

> Run the routing eval, read its table, and use it to decide whether a change to the catalogue or to the prompt helped.

`mm` shows a skill in the catalogue and hopes the model reaches for it. Whether it does is a property of the **model**, not of the code, so no ordinary test can answer it. This one asks a live model, twenty times.

## Run it

```bash
MM_EVAL=1 AGENT_CONFIG=tests/agent.dmr-qwen.yaml go test ./internal/eval -v -timeout 25m
```

Both variables are required in practice: without `MM_EVAL` the test skips, and without `AGENT_CONFIG` it evaluates the built-in defaults rather than the configuration you care about. `go test ./...` skips it, so nothing in CI depends on a server being up.

Expect a few minutes: one model turn per case, and small models are not fast. Rows are printed as they complete.

## Read the table

```
[ 6/20] ✓ hit    want=what-time-is-it  got=what-time-is-it  1.204s  "quelle heure est-il ?"  ← French
[14/20] ≠ wrong  want=greetings        got=vulcan-salute    0.981s  "Salue Bob de ma part"   ← two skills claim salut/saluer
[16/20] · miss   want=bg-jobs          got=(none)           2.017s  "lance le serveur en tâche de fond"
[17/20] ! extra  want=(none)           got=greetings        0.774s  "write a haiku about pointers"
```

| Mark | Meaning | What it points at |
|---|---|---|
| `✓ hit` | the expected skill, or correctly no skill | — |
| `≠ wrong` | a skill was loaded, but not that one | the **catalogue**: two entries the model cannot tell apart |
| `· miss` | a skill applied and none was loaded | the **prompt**: the model did not think to look |
| `! extra` | no skill applied and one was loaded | over-eager instruction, or a catalogue entry that claims too much |
| `✗ error` | the turn failed | **never a routing score.** Either the instrument (a stub that starved the model, `caseTimeout` too short, a server that went away) or the model being unable to compose a valid tool call at all (`unexpected end of JSON input`, or the runaway guard cutting off repeated arguments). The message says which. Fails the test, because the numbers are then incomplete. |

`wrong` and `miss` are kept apart on purpose. Merging them hides which of the two problems you have, and they have different fixes.

## Use it

**The number on its own means nothing.** There is no threshold and the test does not fail on a low score — a number that fails a build becomes a target, and the first thing anyone does with a red build is edit the cases.

The eval is a **before and after**:

1. Run it, keep the output.
2. Change one thing — a skill's `keywords`, the `SKILLS` block of the system prompt, a `description`.
3. Run it again against the same model, with the same configuration.
4. Compare.

One change at a time. Two at once and you have measured their sum.

Small models at `temperature: 0.0` are reproducible enough for this to mean something; run the eval twice before trusting a difference of one case.

## What it measures, and what it does not

It measures the **catalogue** — the tool description the model reads — under one system prompt. One model turn per case, with tools that record what was asked for and **run nothing**: a skill that says "run this echo" must not echo twenty times, and letting the agent loop would measure recovery as much as routing.

It does not measure whether the skill, once loaded, was followed correctly. That is a different question and needs a different instrument.

It also does not measure whether the model can emit a valid tool call. When it cannot — and small models often cannot — you get `✗ error`, not a low routing score. Those two failures have nothing to do with each other and no amount of catalogue work fixes the second one. Measured on `qwen3.5-4b`: three of twenty cases failed this way, reproducibly, on the prompts that invite a more structured call.

## Add a case

`internal/eval/routing_test.go`, the `cases` table:

```go
{"lance le serveur en tâche de fond", "bg-jobs", "French"},
{"what is 17 times 23?", none, ""},
```

Two rules that keep the numbers honest:

- **Do not phrase the prompt in the words of the catalogue.** A case that says "use the greetings skill" measures nothing.
- **Keep the `none` cases.** A catalogue that fires on everything is as broken as one that never fires, and abstaining correctly is a hit.

When you find a real routing failure, add it as a case before fixing it. That is how the eval stops the same failure coming back.

## Write cases that can separate explanations

A single case tells you something failed. A **set** of cases tells you why, and that is what makes the eval worth running.

Worked example. `bg-jobs` was missing on English requests and hitting on French ones, and the obvious reading was "the catalogue's French coverage is thin" — except `bg-jobs` has no French keyword at all. Three cases settled it:

| Case | Result |
|---|---|
| `start the dev server in the background` — English, **names** the concept | ✓ hit |
| `démarre le serveur de dev et vérifie qu'il répond` — French, only a situation | · miss |
| `lance le serveur en tâche de fond` — French, **names** the concept | ✓ hit |
| `start the dev server and check it answers` — English, only a situation | · miss |

A clean 2×2: **the language is irrelevant.** What decides it is whether the request names the concept the skill is about. That reframed the problem from a catalogue one into a prompt one — the model has no reason to know a server blocks the turn — and no amount of keyword editing would have found it.

Build cases in pairs that differ by **one** thing. One case is an anecdote.