⑂ main

Improve skill discovery and execution #6

Merged@k33g wants to merge feature/skill-discovery into main

Merged as 11f470a9fafc.

30 files changed

modified .tickets/issues/0003-improve-tui.yaml +2 -2 Viewed

@@ -1,12 +1,12 @@
1	1	# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2	2	id: 3
3	3	title: Improve TUI
4		-state: open
	4	+state: closed
5	5	author:
6	6	name: k33g
7	7	email: ph.charriere@gmail.com
8	8	createdAt: 2026-09-15T22:25:16.449Z
9		-updatedAt: 2026-09-18T18:55:57.663Z
	9	+updatedAt: 2026-09-21T11:52:52.713Z
10	10	labels:
11	11	- kind::feature
12	12	- priority::medium

  # Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
  id: 3
  title: Improve TUI
--state: open
++state: closed
  author:
    name: k33g
    email: ph.charriere@gmail.com
  createdAt: 2026-09-15T22:25:16.449Z
--updatedAt: 2026-09-18T18:55:57.663Z
++updatedAt: 2026-09-21T11:52:52.713Z
  labels:
    - kind::feature
    - priority::medium

modified .tickets/issues/0015-improve-skills-detection.yaml +266 -15 Viewed

@@ -1,12 +1,12 @@
1	1	# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2	2	id: 15
3	3	title: Improve skills detection
4		-state: open
	4	+state: closed
5	5	author:
6	6	name: k33g
7	7	email: ph.charriere@gmail.com
8	8	createdAt: 2026-09-16T05:52:51.530Z
9		-updatedAt: 2026-09-21T05:45:00.000Z
	9	+updatedAt: 2026-09-21T19:00:59.231Z
10	10	labels: []
11	11	milestone: v0.0.3
12	12	body: \|
@@ -71,61 +71,312 @@ body: \|
71	71	### Not re-opened
72	72
73	73	Decisions in force per `.memory/summary.md`, none of which the above touches: both layouts accepted; the flat file wins when both exist; `skills/README.md` skipped; relative `skillsDir` resolved by `resolveSkillsDir`; banner warning at count zero.
	74	+ ---
	75	+
	76	+ ## Tier 1 implemented (Claude, 2026-09-21, branch `feature/skill-discovery`)
	77	+
	78	+ Tasks 1–5 closed. Tasks 6–7 (the eval, then the prompt block) deliberately left open: nothing there should start before the eval exists.
	79	+
	80	+ ### 1. `read_skill` resolves through the catalogue
	81	+
	82	+ `skills.Find` / `skills.Load` replace `skills.Read`. The model's string is now only ever compared against the names the catalogue advertises and never reaches the filesystem — so the `filepath.Base` barrier is gone because there is nothing left to escape from, which the test `TestALoadedNameNeverReachesTheFilesystem` pins.
	83	+
	84	+ Matching is forgiving in the two ways a model gets a name wrong and a human does not — stray whitespace or quotes, and the wrong case — and never forgiving about which skill: an exact name always beats a case-insensitive one.
	85	+
	86	+ Found while doing it: `List` used `sort.Slice`, which is not stable. The recorded rule "when both layouts claim a name, the flat file wins" was therefore decided by coin toss as soon as the lookup went through the sorted list. Now `sort.SliceStable` over a slice whose flat entries come first, and a test pins the rule.
	87	+
	88	+ ### 2 and 3. `bg-jobs`, and the test that prevents the class
	89	+
	90	+ `skills/bg-jobs/SKILL.md` written — start (both streams redirected, pid kept), poll for readiness instead of sleeping, inspect with `kill -0` and the log, stop politely then insist, and the traps (one shell per call, port already held, no stdin, stray `nohup.out`).
	91	+
	92	+ The durable half is `TestEverySkillNamedInAShippedPromptExists`: it scans every `agent.yaml`, `tests/agent.yaml` and the built-in default prompt for the `` `name` skill `` shape and asserts each one resolves. Verified by removing the new skill — it fails on all four sites at once, naming each.
	93	+
	94	+ ### 4. `keywords` parsed and rendered
	95	+
	96	+ Capped at 8 per skill, because the catalogue is paid for on every request. What the model now sees:
	97	+
	98	+ ```
	99	+ greetings — Greet somebody with a festive message. (greet, greetings, hello, hi, welcome, bonjour, coucou, saluer)
	100	+ ```
	101	+
	102	+ `docs/{en,fr}/reference/skill-format.md` corrected: it stated that `keywords` was "accepted in the file and ignored by the loader".
	103	+
	104	+ ### 5. Two new warnings, and a recap that can say zero
	105	+
	106	+ Banner: `N skill(s) with no description (…) — the model sees only the name and will pick blindly`, and `two skills share a name (…) — only the first is reachable`. Neither could fire before, since the only warning triggered at a count of zero.
	107	+
	108	+ `engine.Skills` now counts skills loaded, not `read_skill` responses: a miss answers `[skill: no skill named …]`, the `[tool: …]` shape the other tools use, and is not counted. The recap prints `📖 N skill(s)` whenever the agent has skills at all (`engine.SkillsAvailable`, set once by `main`), zero included.
	109	+
	110	+ `📖 0 skill(s)` is the point of the whole item: before, a turn where no skill fired and a turn where one fired and was ignored looked identical — the segment was simply absent.
	111	+
	112	+ ### Verified end to end (qwen3.5-4b against a real DMR)
	113	+
	114	+ \| Asked \| What happened \|
	115	+ \|---\|---\|
	116	+ \| « quelle heure est-il ? » \| reached `what-time-is-it` through the `heure` / `quelle heure` keywords → `⚙ 1 command(s) · 📖 1 skill(s)` \|
	117	+ \| "What is 17 times 23?" \| no skill, and it says so: `⚙ 0 command(s) · 📖 0 skill(s)` \|
	118	+ \| « Salue Bob de ma part » \| reached a skill from a French phrase — but picked `vulcan-salute` over `greetings`. Both claim the same keyword space (`salut` / `saluer`). The bridge works; the choice between two overlapping skills is exactly what task 6's eval is for. \|
	119	+
	120	+ Quality gate run 34: 10 warnings, 20 smells — the baseline. `go test ./...`, `gofmt`, `go vet` green.
	121	+
	122	+ ### Not done, on purpose
	123	+
	124	+ Tasks 6 and 7. Also untouched: the `📖` line in the ACP trail still prints unconditionally on stderr (it always did), and the session `errors` trail still records no tool-level failure (Tier 3, item 10).
	125	+
	126	+ ---
	127	+
	128	+ ## Task 6 done: the routing eval exists, and here is its first reading (Claude, 2026-09-21)
	129	+
	130	+ `internal/eval`, opt-in:
	131	+
	132	+ ```
	133	+ MM_EVAL=1 AGENT_CONFIG=tests/agent.dmr-qwen.yaml go test ./internal/eval -v -timeout 60m
	134	+ ```
	135	+
	136	+ `go test ./...` skips it, so nothing in CI needs a server. Docs: `docs/{en,fr}/how-to/measure-skill-routing.md`.
	137	+
	138	+ ### What it measures
	139	+
	140	+ The catalogue, not the agent. One model turn per case, with stub tools that record what was asked for and run nothing — `read_skill` carries the real `skills.Catalogue(list)`, which is the thing under test. Rationale: `vulcan-salute` says "run this echo", and letting the agent loop would echo twenty times and measure recovery as much as routing.
	141	+
	142	+ 20 cases, EN and FR, deliberately not phrased in the catalogue's words, including six where no skill applies and abstaining is the right answer.
	143	+
	144	+ Four outcomes, and `wrong` is kept apart from `miss` because they have different fixes: a skill loaded but the wrong one is a catalogue problem, nothing loaded at all is a prompt problem.
	145	+
	146	+ No threshold, and it does not fail on a low score. A number that fails a build becomes a target, and the first response to a red build is to edit the cases. Use it as a before/after, one change at a time.
	147	+
	148	+ ### First reading — qwen3.5-4b, 5 skills, catalogue 1048 bytes
	149	+
	150	+ ```
	151	+ 20 cases · hit 16 · miss 1 · error 3 → 80% right
	152	+ ```
	153	+
	154	+ Of the 17 cases that completed: 16 hits, 1 miss. Every skill was reached in both English and French, including `give Spock the proper farewell` → `vulcan-salute`, which shares no word with that skill's name or description and could only have come through the keywords. All six abstention cases were correct — the catalogue does not fire on "write a haiku about pointers".
	155	+
	156	+ The one genuine routing miss: `« qui appelle cette fonction ? »` → no skill. French, and about symbols. Its English twin (`where is the Greet function defined?`) did not miss. That is the kind of single, concrete finding this ticket wanted.
	157	+
	158	+ The 3 errors are not routing at all — the model could not compose a valid tool call: twice `unexpected end of JSON input`, once the runaway guard cutting off 2558 characters of repeated `{"command":"ls -laR …"}`. Same failure family the codebase already carries three guards for (`normalize`, `cleanSymbol`, `toolCallGuard`), now observed on a different model. No catalogue or prompt work fixes that, and the eval says so rather than scoring it.
	159	+
	160	+ ### The eval found two defects in itself, both fixed
	161	+
	162	+ 1. The first stubs answered `[eval] command not run`. Given nothing to finish on, the model reran the same command until the runaway guard fired or the case timed out: 5 of 20 cases were artefacts of the instrument. Stubs now answer that the work is done and the turn should end — run time also dropped from minutes per case to 5–20s.
	163	+ 2. `caseTimeout` was 2 minutes, and three good cases hit it exactly. Raised to 5 minutes: an `error` poisons the whole table, so the cap exists to catch a hung server, not to score speed.
	164	+
	165	+ Rows now stream to stderr as they complete (`testing` buffers `t.Log` to the end, and a silent multi-minute eval looks hung), and `os.Stdout` is redirected for the run because `engine.emit` writes the answer with `fmt.Print`, not through `ui.Out`.
	166	+
	167	+ ### What this changes for task 7
	168	+
	169	+ Task 7 (fix the SKILLS prompt block) now has an instrument and a baseline. Two things to know before touching it:
	170	+
	171	+ - The `Salue Bob de ma part` collision does not reproduce here: the eval picks `greetings` in both runs, where the full agent picked `vulcan-salute` earlier the same day. One-turn-with-stubs and the real loop are not the same setting. Keep the case, do not declare it fixed.
	172	+ - The single miss is a French case whose English twin passes. That points at the catalogue's French coverage, not at the SKILLS block — worth checking before rewriting the prompt.
	173	+
	174	+ Quality gate run 35: 10 warnings, 20 smells — unchanged.
	175	+
	176	+ ---
	177	+
	178	+ ## What the eval established, and what it refuted (Claude, 2026-09-21)
	179	+
	180	+ The eval's first real use. Three keyword hypotheses, three refutations, zero routing improvement — and that is the result. Every one of the three sounded right and would have shipped without an instrument.
	181	+
	182	+ Baseline, mellum2-12b, 23 cases: `hit 17 · wrong 2 · miss 4`.
	183	+
	184	+ ### Refuted (1): "the French miss is because no French keyword survives the 8-word cap"
	185	+
	186	+ `lsp-navigation` has 16 keywords and the surviving 8 were English jargon, three of them duplicating the description. Reordered so French sat inside the cap. On qwen the totals did not move and cases merely shuffled; on mellum, `« qui appelle cette fonction ? »` still misses. Reverted.
	187	+
	188	+ ### Refuted (2): "`greet` as a keyword hijacks `where is the Greet function defined?`"
	189	+
	190	+ That request routes to `greetings` instead of `lsp-navigation`. Dropped `greet` from the keywords, re-measured — it still routes to `greetings`. So it is the skill's NAME colliding with the symbol `Greet`, and no keyword edit can fix it. Reverted.
	191	+
	192	+ Finding worth keeping: a skill named after a common code identifier will hijack requests about that identifier. Either rename the skill or accept it; keywords are the wrong lever.
	193	+
	194	+ ### Refuted (3): "`bg-jobs` needs its cap spread across its four actions"
	195	+
	196	+ Its surviving 8 say "start" four ways (`serve`, `daemon`, `nohup`, `start`) and "stop" none — `stop`, `kill`, `running` are all cut. Spread them across start/wait/inspect/stop. Result: worse, 17 hits → 15, then 16 on a second run, with `start the dev server in the background` stably flipping from hit to miss. The "redundant" synonyms were carrying weight; the keyword set does not behave additively. Reverted.
	197	+
	198	+ ### Established: the language is irrelevant
	199	+
	200	+ `bg-jobs` was missing in English and hitting in French, and the obvious reading was thin French coverage — except `bg-jobs` has no French keyword at all. Three cases added to the eval settle it, a clean 2×2:
	201	+
	202	+ \| Request \| Result \|
	203	+ \|---\|---\|
	204	+ \| `start the dev server in the background` — English, names the concept \| ✓ hit \|
	205	+ \| `lance le serveur en tâche de fond` — French, names the concept \| ✓ hit \|
	206	+ \| `start the dev server and check it answers` — English, only a situation \| · miss \|
	207	+ \| `démarre le serveur de dev et vérifie qu'il répond` — French, only a situation \| · miss \|
	208	+
	209	+ What decides it is whether the request names the concept the skill is about, not which language it is in. That moves the remaining `bg-jobs` failures out of the catalogue and into the prompt: the model has no reason to know that starting a server blocks the turn.
	210	+
	211	+ ### Established: run the eval on mellum2-12b
	212	+
	213	+ qwen3.5-4b errors on all three `lsp` cases — tool-call composition, not routing (`unexpected end of JSON input`, runaway guard) — which makes `lsp` routing unmeasurable on that model. mellum2-12b runs all 23 cases clean in about two minutes instead of twenty.
	214	+
	215	+ ### What this means for task 7
	216	+
	217	+ Task 7 proposes rewriting the `SKILLS` block of the system prompt. The three known failures are now characterised, and two of the three are prompt problems after all:
	218	+
	219	+ - `start the dev server …` and `démarre le serveur …` — the model does not know a server blocks. A prompt that says "before acting, check whether a procedure covers this" is the right lever. This is task 7, and it now has a measurable target.
	220	+ - `qui appelle cette fonction ?` — names no symbol. Likely also a prompt matter, or simply out of reach.
	221	+ - `where is the Greet function defined?` — a NAME collision. Not task 7; not keywords either.
	222	+
	223	+ So task 7 is worth doing, but for different reasons than the ticket originally assumed, and against a baseline that now exists.
	224	+
	225	+ ### Also fixed on the way
	226	+
	227	+ Keyword ORDER became significant when the cap was introduced, and nothing told an author their last words were dropped. `skills.Truncated` + a banner warning + the rule in `reference/skill-format.md` (put the most distinctive first; do not spend the eight on words the `description` already carries, since the description is one language and the keywords are where the others belong).
	228	+
	229	+ `internal/lsp`'s `TestOperationsOverAFakeServer` was flaky at ~10% and is fixed (`fakeServer.awaitDoc`, a bounded poll — `didChange` is a notification and the test read the server's state out of band). 100 runs green, 60 under `-race`.
	230	+
	231	+ Not fixed, reported: `internal/engine`'s `TestExplainThroughTheStack/connection_refused` fails about once in thirty. `closedURL` records an ephemeral port and closes it, assuming nothing takes it — a TOCTOU race. One-line fix: return `http://127.0.0.1:1` instead.
	232	+
	233	+ `skills/greetings`, `skills/lsp-navigation` and `skills/bg-jobs` are byte-identical to before the experiments. Quality gate run 37: 10 warnings, 20 smells.
	234	+
	235	+ ---
	236	+
	237	+ ## Task 7 measured, and REJECTED on evidence (Claude, 2026-09-21)
	238	+
	239	+ First, a fact nobody had noticed: the eval's configuration and the built-in default prompt have no `SKILLS` block at all. So the whole baseline above — 17 of 23 — was the catalogue routing on its own, with zero prompt guidance. That gave task 7 a clean "before".
	240	+
	241	+ Four conditions, same model (mellum2-12b), same 23 cases, one change each:
	242	+
	243	+ \| `SKILLS` block \| hit \| wrong \| miss \| error \|
	244	+ \|---\|---\|---\|---\|---\|
	245	+ \| none (run 1) \| 17 \| 2 \| 4 \| 0 \|
	246	+ \| none (run 2) \| 16 \| 2 \| 5 \| 0 \|
	247	+ \| 2 lines \| 16 \| 2 \| 5 \| 0 \|
	248	+ \| 9 lines (the proposed wording) \| 14 \| 2 \| 3 \| 4 \|
	249	+
	250	+ Two conclusions, and the second is the important one.
	251	+
	252	+ 1. No block and a short block are indistinguishable. 17 / 16 / 16 — the baseline itself moves by a case, so a one-case difference means nothing. Telling the model to check the catalogue does not make it check the catalogue.
	253	+
	254	+ 2. The nine-line block broke tool-call composition. Four cases errored — `unexpected end of JSON input`, `generation timed out (stalled)` — where all three other conditions had zero. And it cost the two `bg-jobs` cases that had reliably worked: `lance le serveur en tâche de fond` and `start the dev server in the background` both went from hit to miss.
	255	+
	256	+ The nine-line and two-line blocks say the same thing; only the length differs, and only the long one produced errors. So prompt weight, not prompt content, is what hurt — which is exactly the argument ticket 4 records against MCP ("20 MCP tools sit in every request forever, and mm's target model class already mangles a 6-field schema"), here observed directly on nine lines of prose rather than on tool schemas.
	257	+
	258	+ Also constant across all four conditions: the two `wrong` cases (`where is the Greet function defined?` → `greetings`). The name collision moves with neither keywords nor prompt.
	259	+
	260	+ ### So task 7 is closed as rejected
	261	+
	262	+ The premise — "the model does not think to look, so tell it to look" — does not survive the instrument. Nothing was changed in the repository: all four prompt variants lived in `/tmp`.
	263	+
	264	+ ### And it opened a better question (task 8)
	265	+
	266	+ `agent.dmr.yaml`, `agent.llamacpp.yaml` and `agent.lsp.yaml` all ship a `SKILLS` block of about six lines, and it has never been measured. Given the above, it may be costing more than it gives. I did not touch them — I measured a block of my own wording on a config that had none, which is not the same experiment. Measuring theirs is now the open question, and the eval makes it a two-minute job: run it against each shipped YAML with the block, then with it commented out.
	267	+
	268	+ ---
	269	+
	270	+ ## Task 8 done — and it CORRECTS the task 7 write-up above (Claude, 2026-09-21)
	271	+
	272	+ Measured `agent.dmr.yaml`'s own `SKILLS` block, in its own prompt, on mellum2-12b, 23 cases, two runs of each condition because the differences are two cases wide:
	273	+
	274	+ \| Condition \| run 1 \| run 2 \|
	275	+ \|---\|---\|---\|
	276	+ \| the shipped block, as it ships \| hit 18 · wrong 2 · miss 3 \| hit 18 · wrong 1 · miss 3 · extra 1 \|
	277	+ \| the shipped block + the no-match escape sentence \| hit 16 · wrong 1 · miss 6 \| hit 16 · wrong 1 · miss 6 \|
	278	+ \| no block at all \| hit 16 · wrong 2 · miss 5 \| hit 16 · wrong 2 · miss 5 \|
	279	+
	280	+ The shipped block earns +2 cases, reproducibly. Do not remove it. That answers task 8: it is not costing more than it gives, it is paying for itself.
	281	+
	282	+ ### And it isolates why my own block failed
	283	+
	284	+ The sentence I added, following this ticket's own task 7 —
	285	+
	286	+ > If none of them covers the request, do not mention skills — just do the work.
	287	+
	288	+ — costs exactly the block's entire benefit: 18 → 16, back to the no-block level, with misses going 3 → 6. It under-fires, which is the predicted mechanism: telling a model it may skip is an invitation to skip, and a 12B model takes it.
	289	+
	290	+ So the task 7 write-up above is too broad where it says prompt guidance does not help routing. Corrected:
	291	+
	292	+ - Prompt guidance DOES help — the shipped, unconditional block is worth +2 over nothing.
	293	+ - The explicit no-match escape is what hurts, and it was the specific thing task 7 proposed adding. Isolated to one sentence, reproduced twice.
	294	+ - The "prompt weight" reading also needs narrowing: the shipped prompt is much LONGER overall than the one I first tested, and its six-line block adds no errors at all. Length alone does not predict harm. The four tool-call failures my nine-line block produced remain unexplained — weight or wording, not separated.
	295	+
	296	+ ### What to do with this
	297	+
	298	+ Leave all three shipped blocks alone. If anyone is tempted to "soften" a SKILLS block with a no-match escape — which reads like an obvious improvement, and is why this ticket proposed it — the measurement says it costs two cases in twenty-three.
	299	+
	300	+ Method note for whoever runs this next: the two conditions must differ by one thing. These were derived from the same `agent.dmr.yaml` with only the block removed or one sentence added; the model, the catalogue, the skills directory and the rest of the prompt were identical.
74	301	tasks:
75	302	- id: 1
76		- title: "Resolve read_skill through the catalogue's name, not the file path"
77		- state: open
	303	+ title: Resolve read_skill through the catalogue's name, not the file path
	304	+ state: closed
78	305	priority: high
79	306	author:
80	307	name: k33g
81	308	email: ph.charriere@gmail.com
82	309	createdAt: 2026-09-21T05:45:00.000Z
	310	+ updatedAt: 2026-09-21T11:20:00.000Z
	311	+ closedAt: 2026-09-21T11:20:00.000Z
83	312	- id: 2
84	313	title: "bg-jobs: ship the skill or delete the sentence from the 3 YAMLs and the default prompt"
85		- state: open
	314	+ state: closed
86	315	priority: high
87	316	author:
88	317	name: k33g
89	318	email: ph.charriere@gmail.com
90	319	createdAt: 2026-09-21T05:45:00.000Z
	320	+ updatedAt: 2026-09-21T11:20:00.000Z
	321	+ closedAt: 2026-09-21T11:20:00.000Z
91	322	- id: 3
92		- title: "Test that every skill named in a shipped prompt actually resolves"
93		- state: open
	323	+ title: Test that every skill named in a shipped prompt actually resolves
	324	+ state: closed
94	325	priority: high
95	326	author:
96	327	name: k33g
97	328	email: ph.charriere@gmail.com
98	329	createdAt: 2026-09-21T05:45:00.000Z
	330	+ updatedAt: 2026-09-21T11:20:00.000Z
	331	+ closedAt: 2026-09-21T11:20:00.000Z
99	332	- id: 4
100		- title: "Parse keywords and render them in the catalogue; correct skill-format.md"
101		- state: open
	333	+ title: Parse keywords and render them in the catalogue; correct skill-format.md
	334	+ state: closed
102	335	priority: medium
103	336	author:
104	337	name: k33g
105	338	email: ph.charriere@gmail.com
106	339	createdAt: 2026-09-21T05:45:00.000Z
	340	+ updatedAt: 2026-09-21T11:20:00.000Z
	341	+ closedAt: 2026-09-21T11:20:00.000Z
107	342	- id: 5
108		- title: "Warn on a skill with no description; print the 📖 recap unconditionally and count only successes"
109		- state: open
	343	+ title: Warn on a skill with no description; print the 📖 recap unconditionally and count only successes
	344	+ state: closed
110	345	priority: medium
111	346	author:
112	347	name: k33g
113	348	email: ph.charriere@gmail.com
114	349	createdAt: 2026-09-21T05:45:00.000Z
	350	+ updatedAt: 2026-09-21T11:20:00.000Z
	351	+ closedAt: 2026-09-21T11:20:00.000Z
115	352	- id: 6
116		- title: "Build a skill-routing eval before any prompt change"
117		- state: open
	353	+ title: Build a skill-routing eval before any prompt change
	354	+ state: closed
118	355	priority: medium
119	356	author:
120	357	name: k33g
121	358	email: ph.charriere@gmail.com
122	359	createdAt: 2026-09-21T05:45:00.000Z
	360	+ updatedAt: 2026-09-21T13:10:00.000Z
	361	+ closedAt: 2026-09-21T13:10:00.000Z
123	362	- id: 7
124		- title: "Then: fix the SKILLS prompt block and add it to the built-in default prompt"
125		- state: open
	363	+ title: "Then: fix the SKILLS prompt block — MEASURED AND REJECTED, it makes routing worse"
	364	+ state: closed
126	365	priority: low
127	366	author:
128	367	name: k33g
129	368	email: ph.charriere@gmail.com
130	369	createdAt: 2026-09-21T05:45:00.000Z
	370	+ updatedAt: 2026-09-21T17:40:00.000Z
	371	+ closedAt: 2026-09-21T17:40:00.000Z
	372	+ - id: 8
	373	+ title: "Measure the shipped SKILLS blocks — DONE: the block earns +2 cases, and the escape sentence task 7 proposed cancels exactly that"
	374	+ state: closed
	375	+ priority: medium
	376	+ author:
	377	+ name: k33g
	378	+ email: ph.charriere@gmail.com
	379	+ createdAt: 2026-09-21T17:40:00.000Z
	380	+ updatedAt: 2026-09-21T20:15:00.000Z
	381	+ closedAt: 2026-09-21T20:15:00.000Z
131	382	comments: []

  # Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
  id: 15
  title: Improve skills detection
--state: open
++state: closed
  author:
    name: k33g
    email: ph.charriere@gmail.com
  createdAt: 2026-09-16T05:52:51.530Z
--updatedAt: 2026-09-21T05:45:00.000Z
++updatedAt: 2026-09-21T19:00:59.231Z
  labels: []
  milestone: v0.0.3
  body: |
    ### Not re-opened
    Decisions in force per `.memory/summary.md`, none of which the above touches: both layouts accepted; the flat file wins when both exist; `skills/README.md` skipped; relative `skillsDir` resolved by `resolveSkillsDir`; banner warning at count zero.
++  ---
++
++  ## Tier 1 implemented (Claude, 2026-09-21, branch `feature/skill-discovery`)
++
++  Tasks 1–5 closed. Tasks 6–7 (the eval, then the prompt block) deliberately left open: nothing there should start before the eval exists.
++
++  ### 1. `read_skill` resolves through the catalogue
++
++  `skills.Find` / `skills.Load` replace `skills.Read`. The model's string is now only ever compared against the names the catalogue advertises and **never reaches the filesystem** — so the `filepath.Base` barrier is gone because there is nothing left to escape from, which the test `TestALoadedNameNeverReachesTheFilesystem` pins.
++
++  Matching is forgiving in the two ways a model gets a name wrong and a human does not — stray whitespace or quotes, and the wrong case — and never forgiving about *which* skill: an exact name always beats a case-insensitive one.
++
++  **Found while doing it**: `List` used `sort.Slice`, which is **not stable**. The recorded rule "when both layouts claim a name, the flat file wins" was therefore decided by coin toss as soon as the lookup went through the sorted list. Now `sort.SliceStable` over a slice whose flat entries come first, and a test pins the rule.
++
++  ### 2 and 3. `bg-jobs`, and the test that prevents the class
++
++  `skills/bg-jobs/SKILL.md` written — start (both streams redirected, pid kept), **poll for readiness instead of sleeping**, inspect with `kill -0` and the log, stop politely then insist, and the traps (one shell per call, port already held, no stdin, stray `nohup.out`).
++
++  The durable half is `TestEverySkillNamedInAShippedPromptExists`: it scans every `agent*.yaml`, `tests/agent*.yaml` and the built-in default prompt for the `` `name` skill `` shape and asserts each one resolves. Verified by removing the new skill — it fails on all four sites at once, naming each.
++
++  ### 4. `keywords` parsed and rendered
++
++  Capped at 8 per skill, because the catalogue is paid for on every request. What the model now sees:
++
++  ```
++  greetings — Greet somebody with a festive message. (greet, greetings, hello, hi, welcome, bonjour, coucou, saluer)
++  ```
++
++  `docs/{en,fr}/reference/skill-format.md` corrected: it stated that `keywords` was "accepted in the file and ignored by the loader".
++
++  ### 5. Two new warnings, and a recap that can say zero
++
++  Banner: `N skill(s) with no description (…) — the model sees only the name and will pick blindly`, and `two skills share a name (…) — only the first is reachable`. Neither could fire before, since the only warning triggered at a count of zero.
++
++  `engine.Skills` now counts skills **loaded**, not `read_skill` responses: a miss answers `[skill: no skill named …]`, the `[tool: …]` shape the other tools use, and is not counted. The recap prints `📖 N skill(s)` whenever the agent has skills at all (`engine.SkillsAvailable`, set once by `main`), zero included.
++
++  **`📖 0 skill(s)` is the point of the whole item**: before, a turn where no skill fired and a turn where one fired and was ignored looked identical — the segment was simply absent.
++
++  ### Verified end to end (qwen3.5-4b against a real DMR)
++
++  | Asked | What happened |
++  |---|---|
++  | *« quelle heure est-il ? »* | reached `what-time-is-it` through the `heure` / `quelle heure` keywords → `⚙ 1 command(s) · 📖 1 skill(s)` |
++  | *"What is 17 times 23?"* | no skill, and it says so: `⚙ 0 command(s) · 📖 0 skill(s)` |
++  | *« Salue Bob de ma part »* | reached a skill from a French phrase — but picked `vulcan-salute` over `greetings`. **Both claim the same keyword space** (`salut` / `saluer`). The bridge works; the choice between two overlapping skills is exactly what task 6's eval is for. |
++
++  Quality gate run 34: 10 warnings, 20 smells — the baseline. `go test ./...`, `gofmt`, `go vet` green.
++
++  ### Not done, on purpose
++
++  Tasks 6 and 7. Also untouched: the `📖` line in the ACP trail still prints unconditionally on stderr (it always did), and the session `errors` trail still records no tool-level failure (Tier 3, item 10).
++
++  ---
++
++  ## Task 6 done: the routing eval exists, and here is its first reading (Claude, 2026-09-21)
++
++  `internal/eval`, opt-in:
++
++  ```
++  MM_EVAL=1 AGENT_CONFIG=tests/agent.dmr-qwen.yaml go test ./internal/eval -v -timeout 60m
++  ```
++
++  `go test ./...` skips it, so nothing in CI needs a server. Docs: `docs/{en,fr}/how-to/measure-skill-routing.md`.
++
++  ### What it measures
++
++  **The catalogue, not the agent.** One model turn per case, with stub tools that record what was asked for and run nothing — `read_skill` carries the real `skills.Catalogue(list)`, which is the thing under test. Rationale: `vulcan-salute` says "run this echo", and letting the agent loop would echo twenty times and measure recovery as much as routing.
++
++  20 cases, EN and FR, deliberately **not** phrased in the catalogue's words, including six where no skill applies and abstaining is the right answer.
++
++  Four outcomes, and **`wrong` is kept apart from `miss`** because they have different fixes: a skill loaded but the wrong one is a *catalogue* problem, nothing loaded at all is a *prompt* problem.
++
++  **No threshold, and it does not fail on a low score.** A number that fails a build becomes a target, and the first response to a red build is to edit the cases. Use it as a before/after, one change at a time.
++
++  ### First reading — qwen3.5-4b, 5 skills, catalogue 1048 bytes
++
++  ```
++  20 cases · hit 16 · miss 1 · error 3  →  80% right
++  ```
++
++  Of the 17 cases that completed: **16 hits, 1 miss**. Every skill was reached in both English and French, including `give Spock the proper farewell` → `vulcan-salute`, which shares no word with that skill's name or description and could only have come through the keywords. All six abstention cases were correct — the catalogue does not fire on "write a haiku about pointers".
++
++  **The one genuine routing miss**: `« qui appelle cette fonction ? »` → no skill. French, and about symbols. Its English twin (`where is the Greet function defined?`) did not miss. That is the kind of single, concrete finding this ticket wanted.
++
++  **The 3 errors are not routing at all** — the model could not compose a valid tool call: twice `unexpected end of JSON input`, once the runaway guard cutting off 2558 characters of repeated `{"command":"ls -laR …"}`. Same failure family the codebase already carries three guards for (`normalize`, `cleanSymbol`, `toolCallGuard`), now observed on a different model. **No catalogue or prompt work fixes that**, and the eval says so rather than scoring it.
++
++  ### The eval found two defects in itself, both fixed
++
++  1. The first stubs answered `[eval] command not run`. Given nothing to finish on, the model reran the same command until the runaway guard fired or the case timed out: **5 of 20 cases were artefacts of the instrument**. Stubs now answer that the work is done and the turn should end — run time also dropped from minutes per case to 5–20s.
++  2. `caseTimeout` was 2 minutes, and three good cases hit it exactly. Raised to 5 minutes: an `error` poisons the whole table, so the cap exists to catch a hung server, not to score speed.
++
++  Rows now stream to stderr as they complete (`testing` buffers `t.Log` to the end, and a silent multi-minute eval looks hung), and `os.Stdout` is redirected for the run because `engine.emit` writes the answer with `fmt.Print`, not through `ui.Out`.
++
++  ### What this changes for task 7
++
++  Task 7 (fix the SKILLS prompt block) now has an instrument and a baseline. Two things to know before touching it:
++
++  - The **`Salue Bob de ma part`** collision does **not** reproduce here: the eval picks `greetings` in both runs, where the full agent picked `vulcan-salute` earlier the same day. One-turn-with-stubs and the real loop are not the same setting. Keep the case, do not declare it fixed.
++  - The single miss is a **French** case whose English twin passes. That points at the catalogue's French coverage, not at the SKILLS block — worth checking before rewriting the prompt.
++
++  Quality gate run 35: 10 warnings, 20 smells — unchanged.
++
++  ---
++
++  ## What the eval established, and what it refuted (Claude, 2026-09-21)
++
++  The eval's first real use. **Three keyword hypotheses, three refutations, zero routing improvement — and that is the result.** Every one of the three sounded right and would have shipped without an instrument.
++
++  Baseline, mellum2-12b, 23 cases: `hit 17 · wrong 2 · miss 4`.
++
++  ### Refuted (1): "the French miss is because no French keyword survives the 8-word cap"
++
++  `lsp-navigation` has 16 keywords and the surviving 8 were English jargon, three of them duplicating the description. Reordered so French sat inside the cap. On qwen the totals did not move and cases merely shuffled; on mellum, `« qui appelle cette fonction ? »` **still misses**. Reverted.
++
++  ### Refuted (2): "`greet` as a keyword hijacks `where is the Greet function defined?`"
++
++  That request routes to `greetings` instead of `lsp-navigation`. Dropped `greet` from the keywords, re-measured — **it still routes to `greetings`**. So it is the skill's **NAME** colliding with the symbol `Greet`, and no keyword edit can fix it. Reverted.
++
++  **Finding worth keeping**: a skill named after a common code identifier will hijack requests about that identifier. Either rename the skill or accept it; keywords are the wrong lever.
++
++  ### Refuted (3): "`bg-jobs` needs its cap spread across its four actions"
++
++  Its surviving 8 say "start" four ways (`serve`, `daemon`, `nohup`, `start`) and "stop" none — `stop`, `kill`, `running` are all cut. Spread them across start/wait/inspect/stop. Result: **worse**, 17 hits → 15, then 16 on a second run, with `start the dev server in the background` stably flipping from hit to **miss**. The "redundant" synonyms were carrying weight; the keyword set does not behave additively. Reverted.
++
++  ### Established: the language is irrelevant
++
++  `bg-jobs` was missing in English and hitting in French, and the obvious reading was thin French coverage — except **`bg-jobs` has no French keyword at all**. Three cases added to the eval settle it, a clean 2×2:
++
++  | Request | Result |
++  |---|---|
++  | `start the dev server in the background` — English, **names** the concept | ✓ hit |
++  | `lance le serveur en tâche de fond` — French, **names** the concept | ✓ hit |
++  | `start the dev server and check it answers` — English, only a situation | · miss |
++  | `démarre le serveur de dev et vérifie qu'il répond` — French, only a situation | · miss |
++
++  **What decides it is whether the request names the concept the skill is about**, not which language it is in. That moves the remaining `bg-jobs` failures out of the catalogue and into the prompt: the model has no reason to know that starting a server blocks the turn.
++
++  ### Established: run the eval on mellum2-12b
++
++  qwen3.5-4b errors on all three `lsp` cases — tool-call composition, not routing (`unexpected end of JSON input`, runaway guard) — which makes `lsp` routing unmeasurable on that model. mellum2-12b runs all 23 cases clean in about two minutes instead of twenty.
++
++  ### What this means for task 7
++
++  Task 7 proposes rewriting the `SKILLS` block of the system prompt. The three known failures are now characterised, and **two of the three are prompt problems after all**:
++
++  - `start the dev server …` and `démarre le serveur …` — the model does not know a server blocks. A prompt that says "before acting, check whether a procedure covers this" is the right lever. **This is task 7, and it now has a measurable target.**
++  - `qui appelle cette fonction ?` — names no symbol. Likely also a prompt matter, or simply out of reach.
++  - `where is the Greet function defined?` — a NAME collision. Not task 7; not keywords either.
++
++  So task 7 is worth doing, but for different reasons than the ticket originally assumed, and against a baseline that now exists.
++
++  ### Also fixed on the way
++
++  Keyword ORDER became significant when the cap was introduced, and nothing told an author their last words were dropped. `skills.Truncated` + a banner warning + the rule in `reference/skill-format.md` (put the most distinctive first; do not spend the eight on words the `description` already carries, since the description is one language and the keywords are where the others belong).
++
++  `internal/lsp`'s `TestOperationsOverAFakeServer` was flaky at ~10% and is fixed (`fakeServer.awaitDoc`, a bounded poll — `didChange` is a notification and the test read the server's state out of band). 100 runs green, 60 under `-race`.
++
++  **Not fixed, reported**: `internal/engine`'s `TestExplainThroughTheStack/connection_refused` fails about once in thirty. `closedURL` records an ephemeral port and closes it, assuming nothing takes it — a TOCTOU race. One-line fix: return `http://127.0.0.1:1` instead.
++
++  `skills/greetings`, `skills/lsp-navigation` and `skills/bg-jobs` are byte-identical to before the experiments. Quality gate run 37: 10 warnings, 20 smells.
++
++  ---
++
++  ## Task 7 measured, and REJECTED on evidence (Claude, 2026-09-21)
++
++  First, a fact nobody had noticed: **the eval's configuration and the built-in default prompt have no `SKILLS` block at all.** So the whole baseline above — 17 of 23 — was the catalogue routing on its own, with zero prompt guidance. That gave task 7 a clean "before".
++
++  Four conditions, same model (mellum2-12b), same 23 cases, one change each:
++
++  | `SKILLS` block | hit | wrong | miss | **error** |
++  |---|---|---|---|---|
++  | none (run 1) | 17 | 2 | 4 | **0** |
++  | none (run 2) | 16 | 2 | 5 | **0** |
++  | 2 lines | 16 | 2 | 5 | **0** |
++  | 9 lines (the proposed wording) | 14 | 2 | 3 | **4** |
++
++  Two conclusions, and the second is the important one.
++
++  **1. No block and a short block are indistinguishable.** 17 / 16 / 16 — the baseline itself moves by a case, so a one-case difference means nothing. Telling the model to check the catalogue does not make it check the catalogue.
++
++  **2. The nine-line block broke tool-call composition.** Four cases errored — `unexpected end of JSON input`, `generation timed out (stalled)` — where all three other conditions had zero. And it cost the two `bg-jobs` cases that had reliably worked: `lance le serveur en tâche de fond` and `start the dev server in the background` both went from hit to miss.
++
++  The nine-line and two-line blocks say the same thing; only the length differs, and only the long one produced errors. **So prompt weight, not prompt content, is what hurt** — which is exactly the argument ticket 4 records against MCP ("20 MCP tools sit in every request forever, and mm's target model class already mangles a 6-field schema"), here observed directly on nine lines of prose rather than on tool schemas.
++
++  Also constant across all four conditions: the two `wrong` cases (`where is the Greet function defined?` → `greetings`). The name collision moves with neither keywords nor prompt.
++
++  ### So task 7 is closed as rejected
++
++  The premise — "the model does not think to look, so tell it to look" — does not survive the instrument. Nothing was changed in the repository: all four prompt variants lived in `/tmp`.
++
++  ### And it opened a better question (task 8)
++
++  `agent.dmr.yaml`, `agent.llamacpp.yaml` and `agent.lsp.yaml` all **ship a `SKILLS` block of about six lines**, and it has never been measured. Given the above, it may be costing more than it gives. I did not touch them — I measured a block of my own wording on a config that had none, which is not the same experiment. Measuring theirs is now the open question, and the eval makes it a two-minute job: run it against each shipped YAML with the block, then with it commented out.
++
++  ---
++
++  ## Task 8 done — and it CORRECTS the task 7 write-up above (Claude, 2026-09-21)
++
++  Measured `agent.dmr.yaml`'s own `SKILLS` block, in its own prompt, on mellum2-12b, 23 cases, **two runs of each condition** because the differences are two cases wide:
++
++  | Condition | run 1 | run 2 |
++  |---|---|---|
++  | the shipped block, as it ships | **hit 18** · wrong 2 · miss 3 | **hit 18** · wrong 1 · miss 3 · extra 1 |
++  | the shipped block **+ the no-match escape sentence** | hit 16 · wrong 1 · miss 6 | hit 16 · wrong 1 · miss 6 |
++  | no block at all | hit 16 · wrong 2 · miss 5 | hit 16 · wrong 2 · miss 5 |
++
++  **The shipped block earns +2 cases, reproducibly. Do not remove it.** That answers task 8: it is not costing more than it gives, it is paying for itself.
++
++  ### And it isolates why my own block failed
++
++  The sentence I added, following this ticket's own task 7 —
++
++  > If none of them covers the request, do not mention skills — just do the work.
++
++  — **costs exactly the block's entire benefit**: 18 → 16, back to the no-block level, with misses going 3 → 6. It under-fires, which is the predicted mechanism: telling a model it may skip is an invitation to skip, and a 12B model takes it.
++
++  So the task 7 write-up above is **too broad where it says prompt guidance does not help routing**. Corrected:
++
++  - **Prompt guidance DOES help** — the shipped, unconditional block is worth +2 over nothing.
++  - **The explicit no-match escape is what hurts**, and it was the specific thing task 7 proposed adding. Isolated to one sentence, reproduced twice.
++  - The "prompt weight" reading also needs narrowing: the shipped prompt is much LONGER overall than the one I first tested, and its six-line block adds **no errors at all**. Length alone does not predict harm. The four tool-call failures my nine-line block produced remain unexplained — weight or wording, not separated.
++
++  ### What to do with this
++
++  Leave all three shipped blocks alone. If anyone is tempted to "soften" a SKILLS block with a no-match escape — which reads like an obvious improvement, and is why this ticket proposed it — the measurement says it costs two cases in twenty-three.
++
++  Method note for whoever runs this next: the two conditions must differ by **one** thing. These were derived from the same `agent.dmr.yaml` with only the block removed or one sentence added; the model, the catalogue, the skills directory and the rest of the prompt were identical.
  tasks:
    - id: 1
--    title: "Resolve read_skill through the catalogue's name, not the file path"
++    title: Resolve read_skill through the catalogue's name, not the file path
--    state: open
++    state: closed
      priority: high
      author:
        name: k33g
        email: ph.charriere@gmail.com
      createdAt: 2026-09-21T05:45:00.000Z
++    updatedAt: 2026-09-21T11:20:00.000Z
++    closedAt: 2026-09-21T11:20:00.000Z
    - id: 2
      title: "bg-jobs: ship the skill or delete the sentence from the 3 YAMLs and the default prompt"
--    state: open
++    state: closed
      priority: high
      author:
        name: k33g
        email: ph.charriere@gmail.com
      createdAt: 2026-09-21T05:45:00.000Z
++    updatedAt: 2026-09-21T11:20:00.000Z
++    closedAt: 2026-09-21T11:20:00.000Z
    - id: 3
--    title: "Test that every skill named in a shipped prompt actually resolves"
++    title: Test that every skill named in a shipped prompt actually resolves
--    state: open
++    state: closed
      priority: high
      author:
        name: k33g
        email: ph.charriere@gmail.com
      createdAt: 2026-09-21T05:45:00.000Z
++    updatedAt: 2026-09-21T11:20:00.000Z
++    closedAt: 2026-09-21T11:20:00.000Z
    - id: 4
--    title: "Parse keywords and render them in the catalogue; correct skill-format.md"
++    title: Parse keywords and render them in the catalogue; correct skill-format.md
--    state: open
++    state: closed
      priority: medium
      author:
        name: k33g
        email: ph.charriere@gmail.com
      createdAt: 2026-09-21T05:45:00.000Z
++    updatedAt: 2026-09-21T11:20:00.000Z
++    closedAt: 2026-09-21T11:20:00.000Z
    - id: 5
--    title: "Warn on a skill with no description; print the 📖 recap unconditionally and count only successes"
++    title: Warn on a skill with no description; print the 📖 recap unconditionally and count only successes
--    state: open
++    state: closed
      priority: medium
      author:
        name: k33g
        email: ph.charriere@gmail.com
      createdAt: 2026-09-21T05:45:00.000Z
++    updatedAt: 2026-09-21T11:20:00.000Z
++    closedAt: 2026-09-21T11:20:00.000Z
    - id: 6
--    title: "Build a skill-routing eval before any prompt change"
++    title: Build a skill-routing eval before any prompt change
--    state: open
++    state: closed
      priority: medium
      author:
        name: k33g
        email: ph.charriere@gmail.com
      createdAt: 2026-09-21T05:45:00.000Z
++    updatedAt: 2026-09-21T13:10:00.000Z
++    closedAt: 2026-09-21T13:10:00.000Z
    - id: 7
--    title: "Then: fix the SKILLS prompt block and add it to the built-in default prompt"
++    title: "Then: fix the SKILLS prompt block — MEASURED AND REJECTED, it makes routing worse"
--    state: open
++    state: closed
      priority: low
      author:
        name: k33g
        email: ph.charriere@gmail.com
      createdAt: 2026-09-21T05:45:00.000Z
++    updatedAt: 2026-09-21T17:40:00.000Z
++    closedAt: 2026-09-21T17:40:00.000Z
++  - id: 8
++    title: "Measure the shipped SKILLS blocks — DONE: the block earns +2 cases, and the escape sentence task 7 proposed cancels exactly that"
++    state: closed
++    priority: medium
++    author:
++      name: k33g
++      email: ph.charriere@gmail.com
++    createdAt: 2026-09-21T17:40:00.000Z
++    updatedAt: 2026-09-21T20:15:00.000Z
++    closedAt: 2026-09-21T20:15:00.000Z
  comments: []

added .tickets/issues/0037-add-a-file-selector-to-the-tui-mode.yaml +13 -0 Viewed

new file mode 100644
@@ -0,0 +1,13 @@
	1	+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
	2	+id: 37
	3	+title: Add a file selector to the tui mode
	4	+state: open
	5	+author:
	6	+ name: k33g
	7	+ email: ph.charriere@gmail.com
	8	+createdAt: 2026-09-21T18:55:37.512Z
	9	+updatedAt: 2026-09-21T18:55:37.512Z
	10	+labels: []
	11	+body: ""
	12	+tasks: []
	13	+comments: []

++# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
++id: 37
++title: Add a file selector to the tui mode
++state: open
++author:
++  name: k33g
++  email: ph.charriere@gmail.com
++createdAt: 2026-09-21T18:55:37.512Z
++updatedAt: 2026-09-21T18:55:37.512Z
++labels: []
++body: ""
++tasks: []
++comments: []

added .tickets/issues/0038-add-a-skill-selector.yaml +13 -0 Viewed

new file mode 100644
@@ -0,0 +1,13 @@
	1	+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
	2	+id: 38
	3	+title: add a skill selector
	4	+state: open
	5	+author:
	6	+ name: k33g
	7	+ email: ph.charriere@gmail.com
	8	+createdAt: 2026-09-21T19:00:12.508Z
	9	+updatedAt: 2026-09-21T19:00:12.508Z
	10	+labels: []
	11	+body: ""
	12	+tasks: []
	13	+comments: []

++# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
++id: 38
++title: add a skill selector
++state: open
++author:
++  name: k33g
++  email: ph.charriere@gmail.com
++createdAt: 2026-09-21T19:00:12.508Z
++updatedAt: 2026-09-21T19:00:12.508Z
++labels: []
++body: ""
++tasks: []
++comments: []

added demo/agent.dmr.yaml +181 -0 Viewed

new file mode 100644
@@ -0,0 +1,181 @@
	1	+# Agent settings.
	2	+#
	3	+provider: dmr
	4	+
	5	+#model: huggingface.co/jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M
	6	+model: huggingface.co/unsloth/gemma-4-e2b-it-gguf:Q4_K_M
	7	+
	8	+baseUrl: http://localhost:12434/engines/v1
	9	+fallback: http://host.docker.internal:12434/engines/v1
	10	+
	11	+
	12	+# How many tokens the server actually serves.
	13	+# 0 = ask the server
	14	+contextWindow: 0
	15	+
	16	+# Max number of characters a tool returns to the model.
	17	+# Context safeguard: beyond that, the output is truncated (beginning + end kept).
	18	+maxOutput: 16000
	19	+
	20	+# Max number of model <-> tools round trips for a single question.
	21	+# Raise it for tasks that need long chains of commands.
	22	+maxTurns: 40
	23	+
	24	+# The `bash` tool.
	25	+# `false` removes it: the model can still read and edit files (editTools) and load skills,
	26	+# but never runs a command. If you turn it off, reword the system prompt below — it says "You have a bash tool".
	27	+bashTool: true
	28	+
	29	+# The built-in file tools: read_file, write_file, edit_file.
	30	+editTools: true
	31	+
	32	+lsp:
	33	+ servers:
	34	+ - name: gopls # go install golang.org/x/tools/gopls@latest
	35	+ command: gopls
	36	+ extensions: [".go"]
	37	+
	38	+# Language servers for the `lsp` tool (definition, references, hover, symbols, diagnostics, rename).
	39	+# The file extension picks the server. A server missing from the PATH is skipped with a warning.
	40	+# No server listed = no `lsp` tool. Examples (uncomment what you have installed):
	41	+# lsp:
	42	+# servers:
	43	+# - name: gopls
	44	+# command: gopls
	45	+# extensions: [".go"]
	46	+# - name: typescript
	47	+# command: typescript-language-server
	48	+# args: ["--stdio"]
	49	+# extensions: [".ts", ".tsx", ".js", ".jsx"]
	50	+# - name: python
	51	+# command: pyright-langserver
	52	+# args: ["--stdio"]
	53	+# extensions: [".py"]
	54	+# - name: rust
	55	+# command: rust-analyzer
	56	+# extensions: [".rs"]
	57	+
	58	+# Where the `read_skill` tool looks for its markdown procedures.
	59	+# The path is relative to the CURRENT DIRECTORY, so run the agent from the directory that holds skills.
	60	+skillsDir: .mm/skills
	61	+
	62	+# Persistent sessions, per project.
	63	+# When enabled, every conversation is saved as one JSON file under `dir`
	64	+sessions:
	65	+ enabled: true
	66	+ dir: .mm/sessions
	67	+
	68	+# How many lines of a command's output are echoed to the terminal.
	69	+# 0 disables it.
	70	+previewLines: 20
	71	+
	72	+# The system prompt: what the agent is, and what it is allowed to do.
	73	+# This is the knob to play with — it is the shortest path to changing behaviour.
	74	+system: \|
	75	+ Your name is Bob.
	76	+ You are a coding agent working in a terminal.
	77	+ You have a "bash" tool to run shell commands.
	78	+ Use it to explore files, run tests, inspect the repository, etc.
	79	+ Chain several commands if needed, then answer clearly in English.
	80	+
	81	+ A request often mixes things you answer from yourself ("say hello") with
	82	+ things only a command can answer ("list the files"). Handle every part, in
	83	+ the order asked, and run a command for each part that needs one.
	84	+ Never state the contents of a file, the output of a command, or the state of
	85	+ the repository unless a command in THIS answer returned it. What you did not
	86	+ read, you do not know: run the command instead of recalling it.
	87	+
	88	+ SKILLS
	89	+ You have a second tool, `read_skill`. Its description lists the procedures
	90	+ available for this project — one per kind of task.
	91	+
	92	+ Any request to DO something to a Go project is a skill, not a shell command
	93	+ you invent. Match the request against that list, call `read_skill` FIRST,
	94	+ before any bash command, and then follow what it says step by step.
	95	+
	96	+ FILE EDITING
	97	+ You have three tools for files: `read_file`, `edit_file` and `write_file`.
	98	+ They are how a file gets read and changed here: each change is exact,
	99	+ checked before it is written, and comes back as a diff with line numbers.
	100	+ bash is for running things — building, testing, listing, searching.
	101	+
	102	+ - Read before you write: call `read_file` on the file (numbered=true when
	103	+ you need line numbers). You cannot target text you have not seen; never
	104	+ rely on what you think you remember about a file.
	105	+ - To change an existing file, call `edit_file` with one or more {old, new}
	106	+ pairs. `old` is copied from the file character for character — same
	107	+ spaces, same indentation, same line breaks — and appears exactly once:
	108	+ add the surrounding lines until it is unique. Several pairs are applied
	109	+ together, against the original file. An empty `new` deletes the text.
	110	+ - Call `write_file` only to create a file, or to rewrite one entirely and
	111	+ on purpose. On an existing file it replaces everything, including what
	112	+ you did not intend to touch.
	113	+ - Read the diff the tool returns: it says exactly what changed and on which
	114	+ line. If `edit_file` refuses — text not found, ambiguous, overlapping
	115	+ edits — read the file again and fix `old`. Do not fall back to
	116	+ `write_file` to force the change through.
	117	+ - After editing code, run the narrowest check with bash: the formatter, the
	118	+ compiler, or the test covering that file.
	119	+
	120	+ RULES
	121	+ - Keep everything the file already does, unless the user asked to remove it.
	122	+ - Touch only the files the request is about. Do not add tests, files or
	123	+ features that were not asked for.
	124	+ - Never run a git command unless the user says git, commit or push.
	125	+ - Never move, rename or delete a file unless the user asked for it.
	126	+ - Then answer in English, in a few lines.
	127	+ - If you don't know how to use a <cli>, run `<cli> --help` (or `<cli> help`)
	128	+ to understand the options, then run the command.
	129	+
	130	+ BACKGROUND JOBS
	131	+ Never let a command block the answer. Anything that serves, watches or runs
	132	+ long goes to the background, with BOTH streams redirected and its pid kept:
	133	+
	134	+ nohup <command> > /tmp/<job>.log 2>&1 & echo $! > /tmp/<job>.pid
	135	+
	136	+ Redirecting only stdout still blocks until the process exits. Read the
	137	+ `bg-jobs` skill before you wait on, inspect or stop such a job — each has a
	138	+ rule you cannot guess. Stop every job you started before you finish, and say
	139	+ which ones you left running.
	140	+
	141	+# Generation settings (OpenAI API keys). Low temperature for a coding agent:
	142	+# we want precise and reproducible answers, not creativity.
	143	+sampling:
	144	+ temperature: 0.0
	145	+ parallel_tool_calls: false
	146	+ top_p: 0.9
	147	+ max_tokens: 4096
	148	+
	149	+watchdogTimeout: 30s
	150	+
	151	+# Context compression
	152	+context:
	153	+ enabled: false
	154	+
	155	+ # Compress when the history reaches this share of the window.
	156	+ threshold: 75
	157	+
	158	+ # Fallback on the message count, for when the window is unknown.
	159	+ # One command costs 2 messages (call + response), so 80 is roughly 30 commands of history.
	160	+ maxMessages: 80
	161	+
	162	+ # How many recent questions (with their commands and answers) are kept as-is instead of being summarised.
	163	+ # The model works on the last few turns, so summarising them would make it redo what it just did.
	164	+ keepLastTurns: 3
	165	+
	166	+ # max_tokens of the summary request.
	167	+ summaryMaxTokens: 1200
	168	+
	169	+ # Replaces the built-in summary prompt when set.
	170	+ # Empty = built-in.
	171	+ prompt: ""
	172	+
	173	+ showStats: true
	174	+
	175	+displayCommands: true
	176	+# Show what a thinking model reasons about before it answers (dim italic in the
	177	+# terminal, its own style in the window, a thought chunk in an editor).
	178	+# The reasoning is always READ — it is what keeps the watchdog alive while the
	179	+# model thinks — this only says whether to show it.
	180	+showThinking: true
	181	+

++# Agent settings.
++#
++provider: dmr
++
++#model: huggingface.co/jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M
++model: huggingface.co/unsloth/gemma-4-e2b-it-gguf:Q4_K_M
++
++baseUrl: http://localhost:12434/engines/v1
++fallback: http://host.docker.internal:12434/engines/v1
++
++
++# How many tokens the server actually serves.
++# 0 = ask the server
++contextWindow: 0
++
++# Max number of characters a tool returns to the model.
++# Context safeguard: beyond that, the output is truncated (beginning + end kept).
++maxOutput: 16000
++
++# Max number of model <-> tools round trips for a single question.
++# Raise it for tasks that need long chains of commands.
++maxTurns: 40
++
++# The `bash` tool.
++# `false` removes it: the model can still read and edit files (editTools) and load skills,
++# but never runs a command. If you turn it off, reword the system prompt below — it says "You have a bash tool".
++bashTool: true
++
++# The built-in file tools: read_file, write_file, edit_file.
++editTools: true
++
++lsp:
++  servers:
++    - name: gopls                              # go install golang.org/x/tools/gopls@latest
++      command: gopls
++      extensions: [".go"]
++
++# Language servers for the `lsp` tool (definition, references, hover, symbols, diagnostics, rename).
++# The file extension picks the server. A server missing from the PATH is skipped with a warning.
++# No server listed = no `lsp` tool. Examples (uncomment what you have installed):
++# lsp:
++#   servers:
++#     - name: gopls
++#       command: gopls
++#       extensions: [".go"]
++#     - name: typescript
++#       command: typescript-language-server
++#       args: ["--stdio"]
++#       extensions: [".ts", ".tsx", ".js", ".jsx"]
++#     - name: python
++#       command: pyright-langserver
++#       args: ["--stdio"]
++#       extensions: [".py"]
++#     - name: rust
++#       command: rust-analyzer
++#       extensions: [".rs"]
++
++# Where the `read_skill` tool looks for its markdown procedures.
++# The path is relative to the CURRENT DIRECTORY, so run the agent from the directory that holds skills.
++skillsDir: .mm/skills
++
++# Persistent sessions, per project.
++# When enabled, every conversation is saved as one JSON file under `dir`
++sessions:
++  enabled: true
++  dir: .mm/sessions
++
++# How many lines of a command's output are echoed to the terminal.
++# 0 disables it.
++previewLines: 20
++
++# The system prompt: what the agent is, and what it is allowed to do.
++# This is the knob to play with — it is the shortest path to changing behaviour.
++system: |
++  Your name is Bob.
++  You are a coding agent working in a terminal.
++  You have a "bash" tool to run shell commands.
++  Use it to explore files, run tests, inspect the repository, etc.
++  Chain several commands if needed, then answer clearly in English.
++
++  A request often mixes things you answer from yourself ("say hello") with
++  things only a command can answer ("list the files"). Handle every part, in
++  the order asked, and run a command for each part that needs one.
++  Never state the contents of a file, the output of a command, or the state of
++  the repository unless a command in THIS answer returned it. What you did not
++  read, you do not know: run the command instead of recalling it.
++
++  SKILLS
++  You have a second tool, `read_skill`. Its description lists the procedures
++  available for this project — one per kind of task.
++
++  Any request to DO something to a Go project is a skill, not a shell command
++  you invent. Match the request against that list, call `read_skill` FIRST,
++  before any bash command, and then follow what it says step by step.
++
++  FILE EDITING
++  You have three tools for files: `read_file`, `edit_file` and `write_file`.
++  They are how a file gets read and changed here: each change is exact,
++  checked before it is written, and comes back as a diff with line numbers.
++  bash is for running things — building, testing, listing, searching.
++
++  - Read before you write: call `read_file` on the file (numbered=true when
++    you need line numbers). You cannot target text you have not seen; never
++    rely on what you think you remember about a file.
++  - To change an existing file, call `edit_file` with one or more {old, new}
++    pairs. `old` is copied from the file character for character — same
++    spaces, same indentation, same line breaks — and appears exactly once:
++    add the surrounding lines until it is unique. Several pairs are applied
++    together, against the original file. An empty `new` deletes the text.
++  - Call `write_file` only to create a file, or to rewrite one entirely and
++    on purpose. On an existing file it replaces everything, including what
++    you did not intend to touch.
++  - Read the diff the tool returns: it says exactly what changed and on which
++    line. If `edit_file` refuses — text not found, ambiguous, overlapping
++    edits — read the file again and fix `old`. Do not fall back to
++    `write_file` to force the change through.
++  - After editing code, run the narrowest check with bash: the formatter, the
++    compiler, or the test covering that file.
++
++  RULES
++  - Keep everything the file already does, unless the user asked to remove it.
++  - Touch only the files the request is about. Do not add tests, files or
++    features that were not asked for.
++  - Never run a git command unless the user says git, commit or push.
++  - Never move, rename or delete a file unless the user asked for it.
++  - Then answer in English, in a few lines.
++  - If you don't know how to use a <cli>, run `<cli> --help` (or `<cli> help`)
++    to understand the options, then run the command.
++
++  BACKGROUND JOBS
++  Never let a command block the answer. Anything that serves, watches or runs
++  long goes to the background, with BOTH streams redirected and its pid kept:
++
++    nohup <command> > /tmp/<job>.log 2>&1 & echo $! > /tmp/<job>.pid
++
++  Redirecting only stdout still blocks until the process exits. Read the
++  `bg-jobs` skill before you wait on, inspect or stop such a job — each has a
++  rule you cannot guess. Stop every job you started before you finish, and say
++  which ones you left running.
++
++# Generation settings (OpenAI API keys). Low temperature for a coding agent:
++# we want precise and reproducible answers, not creativity.
++sampling:
++  temperature: 0.0
++  parallel_tool_calls: false
++  top_p: 0.9
++  max_tokens: 4096
++
++watchdogTimeout: 30s
++
++# Context compression
++context:
++  enabled: false
++
++  # Compress when the history reaches this share of the window.
++  threshold: 75
++
++  # Fallback on the message count, for when the window is unknown.
++  # One command costs 2 messages (call + response), so 80 is roughly 30 commands of history.
++  maxMessages: 80
++
++  # How many recent questions (with their commands and answers) are kept as-is instead of being summarised.
++  # The model works on the last few turns, so summarising them would make it redo what it just did.
++  keepLastTurns: 3
++
++  # max_tokens of the summary request.
++  summaryMaxTokens: 1200
++
++  # Replaces the built-in summary prompt when set.
++  # Empty = built-in.
++  prompt: ""
++
++  showStats: true
++
++displayCommands: true
++# Show what a thinking model reasons about before it answers (dim italic in the
++# terminal, its own style in the window, a thought chunk in an editor).
++# The reasoning is always READ — it is what keeps the watchdog alive while the
++# model thinks — this only says whether to show it.
++showThinking: true
++

added demo/main.go +41 -0 Viewed

new file mode 100644
@@ -0,0 +1,41 @@
	1	+package main
	2	+
	3	+import (
	4	+ "encoding/json"
	5	+ "fmt"
	6	+ "log"
	7	+ "net/http"
	8	+)
	9	+
	10	+// Animal struct to represent an animal
	11	+type Animal struct {
	12	+ Name string `json:"name"`
	13	+ Species string `json:"species"`
	14	+}
	15	+
	16	+// Sample data
	17	+var animals = []Animal{
	18	+ {Name: "Lion", Species: "Mammal"},
	19	+ {Name: "Elephant", Species: "Mammal"},
	20	+ {Name: "Eagle", Species: "Bird"},
	21	+ {Name: "Shark", Species: "Fish"},
	22	+}
	23	+
	24	+// handler function to serve the animal list
	25	+func animalHandler(w http.ResponseWriter, r *http.Request) {
	26	+ // Set the content type to JSON
	27	+ w.Header().Set("Content-Type", "application/json")
	28	+
	29	+ // Encode the animal slice to JSON
	30	+ json.NewEncoder(w).Encode(animals)
	31	+}
	32	+
	33	+func main() {
	34	+ // Define the handler for the root path
	35	+ http.HandleFunc("/", animalHandler)
	36	+
	37	+ // Start the server on port 8080
	38	+ port := ":8080"
	39	+ fmt.Printf("Starting server on http://localhost%s\n", port)
	40	+ log.Fatal(http.ListenAndServe(port, nil))
	41	+}

++package main
++
++import (
++	"encoding/json"
++	"fmt"
++	"log"
++	"net/http"
++)
++
++// Animal struct to represent an animal
++type Animal struct {
++	Name string `json:"name"`
++	Species string `json:"species"`
++}
++
++// Sample data
++var animals = []Animal{
++	{Name: "Lion", Species: "Mammal"},
++	{Name: "Elephant", Species: "Mammal"},
++	{Name: "Eagle", Species: "Bird"},
++	{Name: "Shark", Species: "Fish"},
++}
++
++// handler function to serve the animal list
++func animalHandler(w http.ResponseWriter, r *http.Request) {
++	// Set the content type to JSON
++	w.Header().Set("Content-Type", "application/json")
++
++	// Encode the animal slice to JSON
++	json.NewEncoder(w).Encode(animals)
++}
++
++func main() {
++	// Define the handler for the root path
++	http.HandleFunc("/", animalHandler)
++
++	// Start the server on port 8080
++	port := ":8080"
++	fmt.Printf("Starting server on http://localhost%s\n", port)
++	log.Fatal(http.ListenAndServe(port, nil))
++}

modified docs/en/README.md +1 -0 Viewed

@@ -13,6 +13,7 @@ This documentation is organised along the four [Diátaxis](https://diataxis.fr)
13	13	- [Run mini-me on llama.cpp instead of Docker Model Runner](how-to/run-with-llama-cpp.md)
14	14	- [Use mini-me from an editor (ACP)](how-to/use-from-an-editor.md)
15	15	- [Add a skill](how-to/add-a-skill.md)
	16	+- [Measure whether the model picks the right skill](how-to/measure-skill-routing.md) — the routing eval: run it, read its table, use it as a before-and-after
16	17	- [Keep a long session inside the context window](how-to/manage-the-context-window.md)
17	18	- [Start a new session without restarting](how-to/start-a-new-session.md)
18	19	- [Point the model at a file with `@path`](how-to/attach-a-file.md)

  - [Run mini-me on llama.cpp instead of Docker Model Runner](how-to/run-with-llama-cpp.md)
  - [Use mini-me from an editor (ACP)](how-to/use-from-an-editor.md)
  - [Add a skill](how-to/add-a-skill.md)
++- [Measure whether the model picks the right skill](how-to/measure-skill-routing.md) — the routing eval: run it, read its table, use it as a before-and-after
  - [Keep a long session inside the context window](how-to/manage-the-context-window.md)
  - [Start a new session without restarting](how-to/start-a-new-session.md)
  - [Point the model at a file with `@path`](how-to/attach-a-file.md)

added docs/en/how-to/measure-skill-routing.md +90 -0 Viewed

new file mode 100644
@@ -0,0 +1,90 @@
	1	+# Measure whether the model picks the right skill
	2	+
	3	+> Run the routing eval, read its table, and use it to decide whether a change to the catalogue or to the prompt helped.
	4	+
	5	+`mm` shows a skill in the catalogue and hopes the model reaches for it. Whether it does is a property of the model, not of the code, so no ordinary test can answer it. This one asks a live model, twenty times.
	6	+
	7	+## Run it
	8	+
	9	+```bash
	10	+MM_EVAL=1 AGENT_CONFIG=tests/agent.dmr-qwen.yaml go test ./internal/eval -v -timeout 25m
	11	+```
	12	+
	13	+Both variables are required in practice: without `MM_EVAL` the test skips, and without `AGENT_CONFIG` it evaluates the built-in defaults rather than the configuration you care about. `go test ./...` skips it, so nothing in CI depends on a server being up.
	14	+
	15	+Expect a few minutes: one model turn per case, and small models are not fast. Rows are printed as they complete.
	16	+
	17	+## Read the table
	18	+
	19	+```
	20	+[ 6/20] ✓ hit want=what-time-is-it got=what-time-is-it 1.204s "quelle heure est-il ?" ← French
	21	+[14/20] ≠ wrong want=greetings got=vulcan-salute 0.981s "Salue Bob de ma part" ← two skills claim salut/saluer
	22	+[16/20] · miss want=bg-jobs got=(none) 2.017s "lance le serveur en tâche de fond"
	23	+[17/20] ! extra want=(none) got=greetings 0.774s "write a haiku about pointers"
	24	+```
	25	+
	26	+\| Mark \| Meaning \| What it points at \|
	27	+\|---\|---\|---\|
	28	+\| `✓ hit` \| the expected skill, or correctly no skill \| — \|
	29	+\| `≠ wrong` \| a skill was loaded, but not that one \| the catalogue: two entries the model cannot tell apart \|
	30	+\| `· miss` \| a skill applied and none was loaded \| the prompt: the model did not think to look \|
	31	+\| `! extra` \| no skill applied and one was loaded \| over-eager instruction, or a catalogue entry that claims too much \|
	32	+\| `✗ error` \| the turn failed \| never a routing score. Either the instrument (a stub that starved the model, `caseTimeout` too short, a server that went away) or the model being unable to compose a valid tool call at all (`unexpected end of JSON input`, or the runaway guard cutting off repeated arguments). The message says which. Fails the test, because the numbers are then incomplete. \|
	33	+
	34	+`wrong` and `miss` are kept apart on purpose. Merging them hides which of the two problems you have, and they have different fixes.
	35	+
	36	+## Use it
	37	+
	38	+The number on its own means nothing. There is no threshold and the test does not fail on a low score — a number that fails a build becomes a target, and the first thing anyone does with a red build is edit the cases.
	39	+
	40	+The eval is a before and after:
	41	+
	42	+1. Run it, keep the output.
	43	+2. Change one thing — a skill's `keywords`, the `SKILLS` block of the system prompt, a `description`.
	44	+3. Run it again against the same model, with the same configuration.
	45	+4. Compare.
	46	+
	47	+One change at a time. Two at once and you have measured their sum.
	48	+
	49	+Small models at `temperature: 0.0` are reproducible enough for this to mean something; run the eval twice before trusting a difference of one case.
	50	+
	51	+## What it measures, and what it does not
	52	+
	53	+It measures the catalogue — the tool description the model reads — under one system prompt. One model turn per case, with tools that record what was asked for and run nothing: a skill that says "run this echo" must not echo twenty times, and letting the agent loop would measure recovery as much as routing.
	54	+
	55	+It does not measure whether the skill, once loaded, was followed correctly. That is a different question and needs a different instrument.
	56	+
	57	+It also does not measure whether the model can emit a valid tool call. When it cannot — and small models often cannot — you get `✗ error`, not a low routing score. Those two failures have nothing to do with each other and no amount of catalogue work fixes the second one. Measured on `qwen3.5-4b`: three of twenty cases failed this way, reproducibly, on the prompts that invite a more structured call.
	58	+
	59	+## Add a case
	60	+
	61	+`internal/eval/routing_test.go`, the `cases` table:
	62	+
	63	+```go
	64	+{"lance le serveur en tâche de fond", "bg-jobs", "French"},
	65	+{"what is 17 times 23?", none, ""},
	66	+```
	67	+
	68	+Two rules that keep the numbers honest:
	69	+
	70	+- Do not phrase the prompt in the words of the catalogue. A case that says "use the greetings skill" measures nothing.
	71	+- Keep the `none` cases. A catalogue that fires on everything is as broken as one that never fires, and abstaining correctly is a hit.
	72	+
	73	+When you find a real routing failure, add it as a case before fixing it. That is how the eval stops the same failure coming back.
	74	+
	75	+## Write cases that can separate explanations
	76	+
	77	+A single case tells you something failed. A set of cases tells you why, and that is what makes the eval worth running.
	78	+
	79	+Worked example. `bg-jobs` was missing on English requests and hitting on French ones, and the obvious reading was "the catalogue's French coverage is thin" — except `bg-jobs` has no French keyword at all. Three cases settled it:
	80	+
	81	+\| Case \| Result \|
	82	+\|---\|---\|
	83	+\| `start the dev server in the background` — English, names the concept \| ✓ hit \|
	84	+\| `démarre le serveur de dev et vérifie qu'il répond` — French, only a situation \| · miss \|
	85	+\| `lance le serveur en tâche de fond` — French, names the concept \| ✓ hit \|
	86	+\| `start the dev server and check it answers` — English, only a situation \| · miss \|
	87	+
	88	+A clean 2×2: the language is irrelevant. What decides it is whether the request names the concept the skill is about. That reframed the problem from a catalogue one into a prompt one — the model has no reason to know a server blocks the turn — and no amount of keyword editing would have found it.
	89	+
	90	+Build cases in pairs that differ by one thing. One case is an anecdote.

++# Measure whether the model picks the right skill
++
++> Run the routing eval, read its table, and use it to decide whether a change to the catalogue or to the prompt helped.
++
++`mm` shows a skill in the catalogue and hopes the model reaches for it. Whether it does is a property of the **model**, not of the code, so no ordinary test can answer it. This one asks a live model, twenty times.
++
++## Run it
++
++```bash
++MM_EVAL=1 AGENT_CONFIG=tests/agent.dmr-qwen.yaml go test ./internal/eval -v -timeout 25m
++```
++
++Both variables are required in practice: without `MM_EVAL` the test skips, and without `AGENT_CONFIG` it evaluates the built-in defaults rather than the configuration you care about. `go test ./...` skips it, so nothing in CI depends on a server being up.
++
++Expect a few minutes: one model turn per case, and small models are not fast. Rows are printed as they complete.
++
++## Read the table
++
++```
++[ 6/20] ✓ hit    want=what-time-is-it  got=what-time-is-it  1.204s  "quelle heure est-il ?"  ← French
++[14/20] ≠ wrong  want=greetings        got=vulcan-salute    0.981s  "Salue Bob de ma part"   ← two skills claim salut/saluer
++[16/20] · miss   want=bg-jobs          got=(none)           2.017s  "lance le serveur en tâche de fond"
++[17/20] ! extra  want=(none)           got=greetings        0.774s  "write a haiku about pointers"
++```
++
++| Mark | Meaning | What it points at |
++|---|---|---|
++| `✓ hit` | the expected skill, or correctly no skill | — |
++| `≠ wrong` | a skill was loaded, but not that one | the **catalogue**: two entries the model cannot tell apart |
++| `· miss` | a skill applied and none was loaded | the **prompt**: the model did not think to look |
++| `! extra` | no skill applied and one was loaded | over-eager instruction, or a catalogue entry that claims too much |
++| `✗ error` | the turn failed | **never a routing score.** Either the instrument (a stub that starved the model, `caseTimeout` too short, a server that went away) or the model being unable to compose a valid tool call at all (`unexpected end of JSON input`, or the runaway guard cutting off repeated arguments). The message says which. Fails the test, because the numbers are then incomplete. |
++
++`wrong` and `miss` are kept apart on purpose. Merging them hides which of the two problems you have, and they have different fixes.
++
++## Use it
++
++**The number on its own means nothing.** There is no threshold and the test does not fail on a low score — a number that fails a build becomes a target, and the first thing anyone does with a red build is edit the cases.
++
++The eval is a **before and after**:
++
++1. Run it, keep the output.
++2. Change one thing — a skill's `keywords`, the `SKILLS` block of the system prompt, a `description`.
++3. Run it again against the same model, with the same configuration.
++4. Compare.
++
++One change at a time. Two at once and you have measured their sum.
++
++Small models at `temperature: 0.0` are reproducible enough for this to mean something; run the eval twice before trusting a difference of one case.
++
++## What it measures, and what it does not
++
++It measures the **catalogue** — the tool description the model reads — under one system prompt. One model turn per case, with tools that record what was asked for and **run nothing**: a skill that says "run this echo" must not echo twenty times, and letting the agent loop would measure recovery as much as routing.
++
++It does not measure whether the skill, once loaded, was followed correctly. That is a different question and needs a different instrument.
++
++It also does not measure whether the model can emit a valid tool call. When it cannot — and small models often cannot — you get `✗ error`, not a low routing score. Those two failures have nothing to do with each other and no amount of catalogue work fixes the second one. Measured on `qwen3.5-4b`: three of twenty cases failed this way, reproducibly, on the prompts that invite a more structured call.
++
++## Add a case
++
++`internal/eval/routing_test.go`, the `cases` table:
++
++```go
++{"lance le serveur en tâche de fond", "bg-jobs", "French"},
++{"what is 17 times 23?", none, ""},
++```
++
++Two rules that keep the numbers honest:
++
++- **Do not phrase the prompt in the words of the catalogue.** A case that says "use the greetings skill" measures nothing.
++- **Keep the `none` cases.** A catalogue that fires on everything is as broken as one that never fires, and abstaining correctly is a hit.
++
++When you find a real routing failure, add it as a case before fixing it. That is how the eval stops the same failure coming back.
++
++## Write cases that can separate explanations
++
++A single case tells you something failed. A **set** of cases tells you why, and that is what makes the eval worth running.
++
++Worked example. `bg-jobs` was missing on English requests and hitting on French ones, and the obvious reading was "the catalogue's French coverage is thin" — except `bg-jobs` has no French keyword at all. Three cases settled it:
++
++| Case | Result |
++|---|---|
++| `start the dev server in the background` — English, **names** the concept | ✓ hit |
++| `démarre le serveur de dev et vérifie qu'il répond` — French, only a situation | · miss |
++| `lance le serveur en tâche de fond` — French, **names** the concept | ✓ hit |
++| `start the dev server and check it answers` — English, only a situation | · miss |
++
++A clean 2×2: **the language is irrelevant.** What decides it is whether the request names the concept the skill is about. That reframed the problem from a catalogue one into a prompt one — the model has no reason to know a server blocks the turn — and no amount of keyword editing would have found it.
++
++Build cases in pairs that differ by **one** thing. One case is an anecdote.

modified docs/en/reference/skill-format.md +32 -7 Viewed

@@ -8,18 +8,20 @@
8	8	\|------\|-------\|
9	9	\| Directory \| `skillsDir` (default `skills`). \|
10	10	\| Patterns \| `<skillsDir>/<name>.md` (flat) and `<skillsDir>/<name>/SKILL.md` (one directory per skill), mixed freely. Other files in a skill's directory are ignored. A `README.md` at the root of the directory is documentation, not a skill: it is skipped. \|
11		-\| Skill name on disk \| The file name without `.md`, or the directory name for a `SKILL.md`. \|
12		-\| Both forms for one name \| `read_skill` returns the flat file. \|
	11	+\| Skill name on disk \| The file name without `.md`, or the directory name for a `SKILL.md` — used only as the default `name` when the front matter gives none. \|
	12	+\| Resolution \| `read_skill` resolves the name through the catalogue, not through a path: it accepts exactly the names the catalogue shows, so a `name:` that differs from the file name works. Surrounding spaces and quotes are trimmed and the case is forgiven; an exact name always wins over a case-insensitive one. \|
	13	+\| Both forms for one name \| `read_skill` returns the flat file. The banner warns that the other is unreachable. \|
13	14	\| Missing directory \| Not an error: zero skills, `read_skill` not declared. \|
14	15
15	16	## Front matter
16	17
17		-Optional. When present, the file starts with a line `---` and the header ends at the next `---` line. Only two keys are read; the others are ignored.
	18	+Optional. When present, the file starts with a line `---` and the header ends at the next `---` line. Only three keys are read; the others are ignored.
18	19
19	20	\| Key \| Type \| Default \| Description \|
20	21	\|-----\|------\|---------\|-------------\|
21	22	\| `name` \| string \| file name without `.md` \| Name shown in the catalogue and accepted by `read_skill`. An empty value keeps the default. \|
22		-\| `description` \| string \| `""` \| One-line summary shown in the catalogue. \|
	23	+\| `description` \| string \| `""` \| One-line summary shown in the catalogue. A skill without one appears as a bare name, which the model cannot match a request against; the start-up banner warns about it. \|
	24	+\| `keywords` \| comma-separated list \| empty \| Words a user might actually type, in any language. Shown in the catalogue after the description, capped at the first 8 — the catalogue is sent with every request. This is the lexical bridge: "salue Bob" reaches `greetings` through `saluer`, never through the name. Order matters: past the eighth word nothing reaches the model, so put the most distinctive first. The banner warns when a list is being truncated. Do not spend the eight on words already in the `description` — the catalogue shows both, and a description is one language, so the keywords are where the others belong. \|
23	25
24	26	A `description:` line in the body, after the closing `---`, is not metadata.
25	27
@@ -29,11 +31,11 @@ The tool description is:
29	31
30	32	```
31	33	Load a skill: the step-by-step procedure to follow for this kind of task. Call it BEFORE doing the work, with the name of the matching skill, and then follow what it says. Available skills:
32		- <name> — <description>
	34	+ <name> — <description> (<keyword>, <keyword>, …)
33	35	<name>
34	36	```
35	37
36		-Entries are sorted by name; the ` — <description>` part is omitted when the description is empty.
	38	+Entries are sorted by name; the ` — <description>` part is omitted when the description is empty, and the ` (…)` part when there are no keywords.
37	39
38	40	### Example
39	41
@@ -53,7 +55,30 @@ date "+%A %d %B %Y — %H:%M:%S (%Z)"
53	55	```
54	56	```
55	57
56		-The `keywords` key is accepted in the file and ignored by the loader.
	58	+The `keywords` of that example reach the catalogue as `what-time-is-it — Give the current date and time. (time, date, clock, now, today, heure, date du jour, quelle heure)`.
	59	+
	60	+## Writing a skill the model will ACT on
	61	+
	62	+A skill that tells the model to run a command has one failure mode, and it is not the one you expect: the model reproduces the command instead of running it, in a fenced code block, and answers as if it had done the work.
	63	+
	64	+Measured on `mellum2-12b-a2.5b-instruct`, the same skill either way:
	65	+
	66	+\| The skill says \| Result \|
	67	+\|---\|---\|
	68	+\| ` ```bash ` fenced block \| `⚙ 0 command(s)` — the model printed the command back, three runs out of three \|
	69	+\| the command indented, and "call your `bash` tool with this command" \| `⚙ 1 command(s)` — three runs out of three \|
	70	+
	71	+A fenced block is a display affordance. Put one in the model's context and the most probable continuation is the same block, which is exactly what a small model produces. Naming the tool and indenting the command removes the invitation.
	72	+
	73	+Note this is model-dependent — `qwen3.5-4b` ran the fenced version correctly — so a skill that works on your model may still fail on someone else's. Write it the safe way regardless.
	74	+
	75	+The rule, for a skill whose point is that a command runs:
	76	+
	77	+- no ` ```bash ` fence around the command to execute. Indent it by four spaces instead.
	78	+- name the tool: "call your `bash` tool with this command", not "run:".
	79	+- keep saying what the shipped skills say — the answer is the command's output, nothing else — because that is what stops the model answering from memory.
	80	+
	81	+A skill that is reference material rather than one command to run — `bg-jobs`, which shows half a dozen — keeps its fences. Nothing there is "the command to execute", so there is no invitation to misread.
57	82
58	83	## Shipped examples
59	84

  |------|-------|
  | Directory | `skillsDir` (default `skills`). |
  | Patterns | `<skillsDir>/<name>.md` (flat) and `<skillsDir>/<name>/SKILL.md` (one directory per skill), mixed freely. Other files in a skill's directory are ignored. A `README.md` at the root of the directory is documentation, not a skill: it is skipped. |
--| Skill name on disk | The file name without `.md`, or the directory name for a `SKILL.md`. |
++| Skill name on disk | The file name without `.md`, or the directory name for a `SKILL.md` — used only as the default `name` when the front matter gives none. |
--| Both forms for one name | `read_skill` returns the flat file. |
++| Resolution | `read_skill` resolves the name **through the catalogue**, not through a path: it accepts exactly the names the catalogue shows, so a `name:` that differs from the file name works. Surrounding spaces and quotes are trimmed and the case is forgiven; an exact name always wins over a case-insensitive one. |
++| Both forms for one name | `read_skill` returns the flat file. The banner warns that the other is unreachable. |
  | Missing directory | Not an error: zero skills, `read_skill` not declared. |
  ## Front matter
--Optional. When present, the file starts with a line `---` and the header ends at the next `---` line. Only two keys are read; the others are ignored.
++Optional. When present, the file starts with a line `---` and the header ends at the next `---` line. Only three keys are read; the others are ignored.
  | Key | Type | Default | Description |
  |-----|------|---------|-------------|
  | `name` | string | file name without `.md` | Name shown in the catalogue and accepted by `read_skill`. An empty value keeps the default. |
--| `description` | string | `""` | One-line summary shown in the catalogue. |
++| `description` | string | `""` | One-line summary shown in the catalogue. A skill without one appears as a bare name, which the model cannot match a request against; the start-up banner warns about it. |
++| `keywords` | comma-separated list | empty | Words a user might actually type, in any language. Shown in the catalogue after the description, **capped at the first 8** — the catalogue is sent with every request. This is the lexical bridge: *"salue Bob"* reaches `greetings` through `saluer`, never through the name. **Order matters**: past the eighth word nothing reaches the model, so put the most distinctive first. The banner warns when a list is being truncated. Do not spend the eight on words already in the `description` — the catalogue shows both, and a description is one language, so the keywords are where the others belong. |
  A `description:` line in the body, after the closing `---`, is not metadata.
  ```
  Load a skill: the step-by-step procedure to follow for this kind of task. Call it BEFORE doing the work, with the name of the matching skill, and then follow what it says. Available skills:
--  <name> — <description>
++  <name> — <description> (<keyword>, <keyword>, …)
    <name>
  ```
--Entries are sorted by name; the ` — <description>` part is omitted when the description is empty.
++Entries are sorted by name; the ` — <description>` part is omitted when the description is empty, and the ` (…)` part when there are no keywords.
  ### Example
  ```
  ```
--The `keywords` key is accepted in the file and ignored by the loader.
++The `keywords` of that example reach the catalogue as `what-time-is-it — Give the current date and time. (time, date, clock, now, today, heure, date du jour, quelle heure)`.
++
++## Writing a skill the model will ACT on
++
++A skill that tells the model to run a command has one failure mode, and it is not the one you expect: the model **reproduces the command instead of running it**, in a fenced code block, and answers as if it had done the work.
++
++Measured on `mellum2-12b-a2.5b-instruct`, the same skill either way:
++
++| The skill says | Result |
++|---|---|
++| ` ```bash ` fenced block | `⚙ 0 command(s)` — the model printed the command back, three runs out of three |
++| the command **indented**, and "call your `bash` tool with this command" | `⚙ 1 command(s)` — three runs out of three |
++
++A fenced block is a *display* affordance. Put one in the model's context and the most probable continuation is the same block, which is exactly what a small model produces. Naming the tool and indenting the command removes the invitation.
++
++Note this is model-dependent — `qwen3.5-4b` ran the fenced version correctly — so a skill that works on your model may still fail on someone else's. Write it the safe way regardless.
++
++The rule, for a skill whose point is that a command runs:
++
++- **no ` ```bash ` fence around the command to execute.** Indent it by four spaces instead.
++- **name the tool**: "call your `bash` tool with this command", not "run:".
++- keep saying what the shipped skills say — *the answer is the command's output, nothing else* — because that is what stops the model answering from memory.
++
++A skill that is reference material rather than one command to run — `bg-jobs`, which shows half a dozen — keeps its fences. Nothing there is "the command to execute", so there is no invitation to misread.
  ## Shipped examples

modified docs/fr/README.md +1 -0 Viewed

@@ -13,6 +13,7 @@ Cette documentation est organisée selon les quatre quadrants [Diátaxis](https:
13	13	- [Faire tourner mini-me sur llama.cpp plutôt que Docker Model Runner](how-to/run-with-llama-cpp.md)
14	14	- [Utiliser mini-me depuis un éditeur (ACP)](how-to/use-from-an-editor.md)
15	15	- [Ajouter un skill](how-to/add-a-skill.md)
	16	+- [Mesurer si le modèle choisit le bon skill](how-to/measure-skill-routing.md) — l'eval de routage : le lancer, lire son tableau, s'en servir en avant/après
16	17	- [Garder une longue session dans la fenêtre de contexte](how-to/manage-the-context-window.md)
17	18	- [Démarrer une nouvelle session sans relancer](how-to/start-a-new-session.md)
18	19	- [Désigner un fichier au modèle avec `@chemin`](how-to/attach-a-file.md)

  - [Faire tourner mini-me sur llama.cpp plutôt que Docker Model Runner](how-to/run-with-llama-cpp.md)
  - [Utiliser mini-me depuis un éditeur (ACP)](how-to/use-from-an-editor.md)
  - [Ajouter un skill](how-to/add-a-skill.md)
++- [Mesurer si le modèle choisit le bon skill](how-to/measure-skill-routing.md) — l'eval de routage : le lancer, lire son tableau, s'en servir en avant/après
  - [Garder une longue session dans la fenêtre de contexte](how-to/manage-the-context-window.md)
  - [Démarrer une nouvelle session sans relancer](how-to/start-a-new-session.md)
  - [Désigner un fichier au modèle avec `@chemin`](how-to/attach-a-file.md)

added docs/fr/how-to/measure-skill-routing.md +90 -0 Viewed