bots-garden/mini-me · 11f470a

⑂ 11f470a

Merge feature/skill-discovery into main

rickub committed 2026-09-21T19:02:08Z Browse files

11f470a parents: dda11d4 0f8196a

modified .tickets/issues/0003-improve-tui.yaml +2 -2 Viewed

@@ -1,12 +1,12 @@
1	1	# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2	2	id: 3
3	3	title: Improve TUI
4		-state: open
	4	+state: closed
5	5	author:
6	6	name: k33g
7	7	email: ph.charriere@gmail.com
8	8	createdAt: 2026-09-15T22:25:16.449Z
9		-updatedAt: 2026-09-18T18:55:57.663Z
	9	+updatedAt: 2026-09-21T11:52:52.713Z
10	10	labels:
11	11	- kind::feature
12	12	- priority::medium

  # Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
  id: 3
  title: Improve TUI
--state: open
++state: closed
  author:
    name: k33g
    email: ph.charriere@gmail.com
  createdAt: 2026-09-15T22:25:16.449Z
--updatedAt: 2026-09-18T18:55:57.663Z
++updatedAt: 2026-09-21T11:52:52.713Z
  labels:
    - kind::feature
    - priority::medium

modified .tickets/issues/0015-improve-skills-detection.yaml +266 -15 Viewed

@@ -1,12 +1,12 @@
1	1	# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2	2	id: 15
3	3	title: Improve skills detection
4		-state: open
	4	+state: closed
5	5	author:
6	6	name: k33g
7	7	email: ph.charriere@gmail.com
8	8	createdAt: 2026-09-16T05:52:51.530Z
9		-updatedAt: 2026-09-21T05:45:00.000Z
	9	+updatedAt: 2026-09-21T19:00:59.231Z
10	10	labels: []
11	11	milestone: v0.0.3
12	12	body: \|
@@ -71,61 +71,312 @@ body: \|
71	71	### Not re-opened
72	72
73	73	Decisions in force per `.memory/summary.md`, none of which the above touches: both layouts accepted; the flat file wins when both exist; `skills/README.md` skipped; relative `skillsDir` resolved by `resolveSkillsDir`; banner warning at count zero.
	74	+ ---
	75	+
	76	+ ## Tier 1 implemented (Claude, 2026-09-21, branch `feature/skill-discovery`)
	77	+
	78	+ Tasks 1–5 closed. Tasks 6–7 (the eval, then the prompt block) deliberately left open: nothing there should start before the eval exists.
	79	+
	80	+ ### 1. `read_skill` resolves through the catalogue
	81	+
	82	+ `skills.Find` / `skills.Load` replace `skills.Read`. The model's string is now only ever compared against the names the catalogue advertises and never reaches the filesystem — so the `filepath.Base` barrier is gone because there is nothing left to escape from, which the test `TestALoadedNameNeverReachesTheFilesystem` pins.
	83	+
	84	+ Matching is forgiving in the two ways a model gets a name wrong and a human does not — stray whitespace or quotes, and the wrong case — and never forgiving about which skill: an exact name always beats a case-insensitive one.
	85	+
	86	+ Found while doing it: `List` used `sort.Slice`, which is not stable. The recorded rule "when both layouts claim a name, the flat file wins" was therefore decided by coin toss as soon as the lookup went through the sorted list. Now `sort.SliceStable` over a slice whose flat entries come first, and a test pins the rule.
	87	+
	88	+ ### 2 and 3. `bg-jobs`, and the test that prevents the class
	89	+
	90	+ `skills/bg-jobs/SKILL.md` written — start (both streams redirected, pid kept), poll for readiness instead of sleeping, inspect with `kill -0` and the log, stop politely then insist, and the traps (one shell per call, port already held, no stdin, stray `nohup.out`).
	91	+
	92	+ The durable half is `TestEverySkillNamedInAShippedPromptExists`: it scans every `agent.yaml`, `tests/agent.yaml` and the built-in default prompt for the `` `name` skill `` shape and asserts each one resolves. Verified by removing the new skill — it fails on all four sites at once, naming each.
	93	+
	94	+ ### 4. `keywords` parsed and rendered
	95	+
	96	+ Capped at 8 per skill, because the catalogue is paid for on every request. What the model now sees:
	97	+
	98	+ ```
	99	+ greetings — Greet somebody with a festive message. (greet, greetings, hello, hi, welcome, bonjour, coucou, saluer)
	100	+ ```
	101	+
	102	+ `docs/{en,fr}/reference/skill-format.md` corrected: it stated that `keywords` was "accepted in the file and ignored by the loader".
	103	+
	104	+ ### 5. Two new warnings, and a recap that can say zero
	105	+
	106	+ Banner: `N skill(s) with no description (…) — the model sees only the name and will pick blindly`, and `two skills share a name (…) — only the first is reachable`. Neither could fire before, since the only warning triggered at a count of zero.
	107	+
	108	+ `engine.Skills` now counts skills loaded, not `read_skill` responses: a miss answers `[skill: no skill named …]`, the `[tool: …]` shape the other tools use, and is not counted. The recap prints `📖 N skill(s)` whenever the agent has skills at all (`engine.SkillsAvailable`, set once by `main`), zero included.
	109	+
	110	+ `📖 0 skill(s)` is the point of the whole item: before, a turn where no skill fired and a turn where one fired and was ignored looked identical — the segment was simply absent.
	111	+
	112	+ ### Verified end to end (qwen3.5-4b against a real DMR)
	113	+
	114	+ \| Asked \| What happened \|
	115	+ \|---\|---\|
	116	+ \| « quelle heure est-il ? » \| reached `what-time-is-it` through the `heure` / `quelle heure` keywords → `⚙ 1 command(s) · 📖 1 skill(s)` \|
	117	+ \| "What is 17 times 23?" \| no skill, and it says so: `⚙ 0 command(s) · 📖 0 skill(s)` \|
	118	+ \| « Salue Bob de ma part » \| reached a skill from a French phrase — but picked `vulcan-salute` over `greetings`. Both claim the same keyword space (`salut` / `saluer`). The bridge works; the choice between two overlapping skills is exactly what task 6's eval is for. \|
	119	+
	120	+ Quality gate run 34: 10 warnings, 20 smells — the baseline. `go test ./...`, `gofmt`, `go vet` green.
	121	+
	122	+ ### Not done, on purpose
	123	+
	124	+ Tasks 6 and 7. Also untouched: the `📖` line in the ACP trail still prints unconditionally on stderr (it always did), and the session `errors` trail still records no tool-level failure (Tier 3, item 10).
	125	+
	126	+ ---
	127	+
	128	+ ## Task 6 done: the routing eval exists, and here is its first reading (Claude, 2026-09-21)
	129	+
	130	+ `internal/eval`, opt-in:
	131	+
	132	+ ```
	133	+ MM_EVAL=1 AGENT_CONFIG=tests/agent.dmr-qwen.yaml go test ./internal/eval -v -timeout 60m
	134	+ ```
	135	+
	136	+ `go test ./...` skips it, so nothing in CI needs a server. Docs: `docs/{en,fr}/how-to/measure-skill-routing.md`.
	137	+
	138	+ ### What it measures
	139	+
	140	+ The catalogue, not the agent. One model turn per case, with stub tools that record what was asked for and run nothing — `read_skill` carries the real `skills.Catalogue(list)`, which is the thing under test. Rationale: `vulcan-salute` says "run this echo", and letting the agent loop would echo twenty times and measure recovery as much as routing.
	141	+
	142	+ 20 cases, EN and FR, deliberately not phrased in the catalogue's words, including six where no skill applies and abstaining is the right answer.
	143	+
	144	+ Four outcomes, and `wrong` is kept apart from `miss` because they have different fixes: a skill loaded but the wrong one is a catalogue problem, nothing loaded at all is a prompt problem.
	145	+
	146	+ No threshold, and it does not fail on a low score. A number that fails a build becomes a target, and the first response to a red build is to edit the cases. Use it as a before/after, one change at a time.
	147	+
	148	+ ### First reading — qwen3.5-4b, 5 skills, catalogue 1048 bytes
	149	+
	150	+ ```
	151	+ 20 cases · hit 16 · miss 1 · error 3 → 80% right
	152	+ ```
	153	+
	154	+ Of the 17 cases that completed: 16 hits, 1 miss. Every skill was reached in both English and French, including `give Spock the proper farewell` → `vulcan-salute`, which shares no word with that skill's name or description and could only have come through the keywords. All six abstention cases were correct — the catalogue does not fire on "write a haiku about pointers".
	155	+
	156	+ The one genuine routing miss: `« qui appelle cette fonction ? »` → no skill. French, and about symbols. Its English twin (`where is the Greet function defined?`) did not miss. That is the kind of single, concrete finding this ticket wanted.
	157	+
	158	+ The 3 errors are not routing at all — the model could not compose a valid tool call: twice `unexpected end of JSON input`, once the runaway guard cutting off 2558 characters of repeated `{"command":"ls -laR …"}`. Same failure family the codebase already carries three guards for (`normalize`, `cleanSymbol`, `toolCallGuard`), now observed on a different model. No catalogue or prompt work fixes that, and the eval says so rather than scoring it.
	159	+
	160	+ ### The eval found two defects in itself, both fixed
	161	+
	162	+ 1. The first stubs answered `[eval] command not run`. Given nothing to finish on, the model reran the same command until the runaway guard fired or the case timed out: 5 of 20 cases were artefacts of the instrument. Stubs now answer that the work is done and the turn should end — run time also dropped from minutes per case to 5–20s.
	163	+ 2. `caseTimeout` was 2 minutes, and three good cases hit it exactly. Raised to 5 minutes: an `error` poisons the whole table, so the cap exists to catch a hung server, not to score speed.
	164	+
	165	+ Rows now stream to stderr as they complete (`testing` buffers `t.Log` to the end, and a silent multi-minute eval looks hung), and `os.Stdout` is redirected for the run because `engine.emit` writes the answer with `fmt.Print`, not through `ui.Out`.
	166	+
	167	+ ### What this changes for task 7
	168	+
	169	+ Task 7 (fix the SKILLS prompt block) now has an instrument and a baseline. Two things to know before touching it:
	170	+
	171	+ - The `Salue Bob de ma part` collision does not reproduce here: the eval picks `greetings` in both runs, where the full agent picked `vulcan-salute` earlier the same day. One-turn-with-stubs and the real loop are not the same setting. Keep the case, do not declare it fixed.
	172	+ - The single miss is a French case whose English twin passes. That points at the catalogue's French coverage, not at the SKILLS block — worth checking before rewriting the prompt.
	173	+
	174	+ Quality gate run 35: 10 warnings, 20 smells — unchanged.
	175	+
	176	+ ---
	177	+
	178	+ ## What the eval established, and what it refuted (Claude, 2026-09-21)
	179	+
	180	+ The eval's first real use. Three keyword hypotheses, three refutations, zero routing improvement — and that is the result. Every one of the three sounded right and would have shipped without an instrument.
	181	+
	182	+ Baseline, mellum2-12b, 23 cases: `hit 17 · wrong 2 · miss 4`.
	183	+
	184	+ ### Refuted (1): "the French miss is because no French keyword survives the 8-word cap"
	185	+
	186	+ `lsp-navigation` has 16 keywords and the surviving 8 were English jargon, three of them duplicating the description. Reordered so French sat inside the cap. On qwen the totals did not move and cases merely shuffled; on mellum, `« qui appelle cette fonction ? »` still misses. Reverted.
	187	+
	188	+ ### Refuted (2): "`greet` as a keyword hijacks `where is the Greet function defined?`"
	189	+
	190	+ That request routes to `greetings` instead of `lsp-navigation`. Dropped `greet` from the keywords, re-measured — it still routes to `greetings`. So it is the skill's NAME colliding with the symbol `Greet`, and no keyword edit can fix it. Reverted.
	191	+
	192	+ Finding worth keeping: a skill named after a common code identifier will hijack requests about that identifier. Either rename the skill or accept it; keywords are the wrong lever.
	193	+
	194	+ ### Refuted (3): "`bg-jobs` needs its cap spread across its four actions"
	195	+
	196	+ Its surviving 8 say "start" four ways (`serve`, `daemon`, `nohup`, `start`) and "stop" none — `stop`, `kill`, `running` are all cut. Spread them across start/wait/inspect/stop. Result: worse, 17 hits → 15, then 16 on a second run, with `start the dev server in the background` stably flipping from hit to miss. The "redundant" synonyms were carrying weight; the keyword set does not behave additively. Reverted.
	197	+
	198	+ ### Established: the language is irrelevant
	199	+
	200	+ `bg-jobs` was missing in English and hitting in French, and the obvious reading was thin French coverage — except `bg-jobs` has no French keyword at all. Three cases added to the eval settle it, a clean 2×2:
	201	+
	202	+ \| Request \| Result \|
	203	+ \|---\|---\|
	204	+ \| `start the dev server in the background` — English, names the concept \| ✓ hit \|
	205	+ \| `lance le serveur en tâche de fond` — French, names the concept \| ✓ hit \|
	206	+ \| `start the dev server and check it answers` — English, only a situation \| · miss \|
	207	+ \| `démarre le serveur de dev et vérifie qu'il répond` — French, only a situation \| · miss \|
	208	+
	209	+ What decides it is whether the request names the concept the skill is about, not which language it is in. That moves the remaining `bg-jobs` failures out of the catalogue and into the prompt: the model has no reason to know that starting a server blocks the turn.
	210	+
	211	+ ### Established: run the eval on mellum2-12b
	212	+
	213	+ qwen3.5-4b errors on all three `lsp` cases — tool-call composition, not routing (`unexpected end of JSON input`, runaway guard) — which makes `lsp` routing unmeasurable on that model. mellum2-12b runs all 23 cases clean in about two minutes instead of twenty.
	214	+
	215	+ ### What this means for task 7
	216	+
	217	+ Task 7 proposes rewriting the `SKILLS` block of the system prompt. The three known failures are now characterised, and two of the three are prompt problems after all:
	218	+
	219	+ - `start the dev server …` and `démarre le serveur …` — the model does not know a server blocks. A prompt that says "before acting, check whether a procedure covers this" is the right lever. This is task 7, and it now has a measurable target.
	220	+ - `qui appelle cette fonction ?` — names no symbol. Likely also a prompt matter, or simply out of reach.
	221	+ - `where is the Greet function defined?` — a NAME collision. Not task 7; not keywords either.
	222	+
	223	+ So task 7 is worth doing, but for different reasons than the ticket originally assumed, and against a baseline that now exists.
	224	+
	225	+ ### Also fixed on the way
	226	+
	227	+ Keyword ORDER became significant when the cap was introduced, and nothing told an author their last words were dropped. `skills.Truncated` + a banner warning + the rule in `reference/skill-format.md` (put the most distinctive first; do not spend the eight on words the `description` already carries, since the description is one language and the keywords are where the others belong).
	228	+
	229	+ `internal/lsp`'s `TestOperationsOverAFakeServer` was flaky at ~10% and is fixed (`fakeServer.awaitDoc`, a bounded poll — `didChange` is a notification and the test read the server's state out of band). 100 runs green, 60 under `-race`.
	230	+
	231	+ Not fixed, reported: `internal/engine`'s `TestExplainThroughTheStack/connection_refused` fails about once in thirty. `closedURL` records an ephemeral port and closes it, assuming nothing takes it — a TOCTOU race. One-line fix: return `http://127.0.0.1:1` instead.
	232	+
	233	+ `skills/greetings`, `skills/lsp-navigation` and `skills/bg-jobs` are byte-identical to before the experiments. Quality gate run 37: 10 warnings, 20 smells.
	234	+
	235	+ ---
	236	+
	237	+ ## Task 7 measured, and REJECTED on evidence (Claude, 2026-09-21)
	238	+
	239	+ First, a fact nobody had noticed: the eval's configuration and the built-in default prompt have no `SKILLS` block at all. So the whole baseline above — 17 of 23 — was the catalogue routing on its own, with zero prompt guidance. That gave task 7 a clean "before".
	240	+
	241	+ Four conditions, same model (mellum2-12b), same 23 cases, one change each:
	242	+
	243	+ \| `SKILLS` block \| hit \| wrong \| miss \| error \|
	244	+ \|---\|---\|---\|---\|---\|
	245	+ \| none (run 1) \| 17 \| 2 \| 4 \| 0 \|
	246	+ \| none (run 2) \| 16 \| 2 \| 5 \| 0 \|
	247	+ \| 2 lines \| 16 \| 2 \| 5 \| 0 \|
	248	+ \| 9 lines (the proposed wording) \| 14 \| 2 \| 3 \| 4 \|
	249	+
	250	+ Two conclusions, and the second is the important one.
	251	+
	252	+ 1. No block and a short block are indistinguishable. 17 / 16 / 16 — the baseline itself moves by a case, so a one-case difference means nothing. Telling the model to check the catalogue does not make it check the catalogue.
	253	+
	254	+ 2. The nine-line block broke tool-call composition. Four cases errored — `unexpected end of JSON input`, `generation timed out (stalled)` — where all three other conditions had zero. And it cost the two `bg-jobs` cases that had reliably worked: `lance le serveur en tâche de fond` and `start the dev server in the background` both went from hit to miss.
	255	+
	256	+ The nine-line and two-line blocks say the same thing; only the length differs, and only the long one produced errors. So prompt weight, not prompt content, is what hurt — which is exactly the argument ticket 4 records against MCP ("20 MCP tools sit in every request forever, and mm's target model class already mangles a 6-field schema"), here observed directly on nine lines of prose rather than on tool schemas.
	257	+
	258	+ Also constant across all four conditions: the two `wrong` cases (`where is the Greet function defined?` → `greetings`). The name collision moves with neither keywords nor prompt.
	259	+
	260	+ ### So task 7 is closed as rejected
	261	+
	262	+ The premise — "the model does not think to look, so tell it to look" — does not survive the instrument. Nothing was changed in the repository: all four prompt variants lived in `/tmp`.
	263	+
	264	+ ### And it opened a better question (task 8)
	265	+
	266	+ `agent.dmr.yaml`, `agent.llamacpp.yaml` and `agent.lsp.yaml` all ship a `SKILLS` block of about six lines, and it has never been measured. Given the above, it may be costing more than it gives. I did not touch them — I measured a block of my own wording on a config that had none, which is not the same experiment. Measuring theirs is now the open question, and the eval makes it a two-minute job: run it against each shipped YAML with the block, then with it commented out.
	267	+
	268	+ ---
	269	+
	270	+ ## Task 8 done — and it CORRECTS the task 7 write-up above (Claude, 2026-09-21)
	271	+
	272	+ Measured `agent.dmr.yaml`'s own `SKILLS` block, in its own prompt, on mellum2-12b, 23 cases, two runs of each condition because the differences are two cases wide:
	273	+
	274	+ \| Condition \| run 1 \| run 2 \|
	275	+ \|---\|---\|---\|
	276	+ \| the shipped block, as it ships \| hit 18 · wrong 2 · miss 3 \| hit 18 · wrong 1 · miss 3 · extra 1 \|
	277	+ \| the shipped block + the no-match escape sentence \| hit 16 · wrong 1 · miss 6 \| hit 16 · wrong 1 · miss 6 \|
	278	+ \| no block at all \| hit 16 · wrong 2 · miss 5 \| hit 16 · wrong 2 · miss 5 \|
	279	+
	280	+ The shipped block earns +2 cases, reproducibly. Do not remove it. That answers task 8: it is not costing more than it gives, it is paying for itself.
	281	+
	282	+ ### And it isolates why my own block failed
	283	+
	284	+ The sentence I added, following this ticket's own task 7 —
	285	+
	286	+ > If none of them covers the request, do not mention skills — just do the work.
	287	+
	288	+ — costs exactly the block's entire benefit: 18 → 16, back to the no-block level, with misses going 3 → 6. It under-fires, which is the predicted mechanism: telling a model it may skip is an invitation to skip, and a 12B model takes it.
	289	+
	290	+ So the task 7 write-up above is too broad where it says prompt guidance does not help routing. Corrected:
	291	+
	292	+ - Prompt guidance DOES help — the shipped, unconditional block is worth +2 over nothing.
	293	+ - The explicit no-match escape is what hurts, and it was the specific thing task 7 proposed adding. Isolated to one sentence, reproduced twice.
	294	+ - The "prompt weight" reading also needs narrowing: the shipped prompt is much LONGER overall than the one I first tested, and its six-line block adds no errors at all. Length alone does not predict harm. The four tool-call failures my nine-line block produced remain unexplained — weight or wording, not separated.
	295	+
	296	+ ### What to do with this
	297	+
	298	+ Leave all three shipped blocks alone. If anyone is tempted to "soften" a SKILLS block with a no-match escape — which reads like an obvious improvement, and is why this ticket proposed it — the measurement says it costs two cases in twenty-three.
	299	+
	300	+ Method note for whoever runs this next: the two conditions must differ by one thing. These were derived from the same `agent.dmr.yaml` with only the block removed or one sentence added; the model, the catalogue, the skills directory and the rest of the prompt were identical.
74	301	tasks:
75	302	- id: 1
76		- title: "Resolve read_skill through the catalogue's name, not the file path"
77		- state: open
	303	+ title: Resolve read_skill through the catalogue's name, not the file path
	304	+ state: closed
78	305	priority: high
79	306	author:
80	307	name: k33g
81	308	email: ph.charriere@gmail.com
82	309	createdAt: 2026-09-21T05:45:00.000Z
	310	+ updatedAt: 2026-09-21T11:20:00.000Z
	311	+ closedAt: 2026-09-21T11:20:00.000Z
83	312	- id: 2
84	313	title: "bg-jobs: ship the skill or delete the sentence from the 3 YAMLs and the default prompt"
85		- state: open
	314	+ state: closed
86	315	priority: high
87	316	author:
88	317	name: k33g
89	318	email: ph.charriere@gmail.com
90	319	createdAt: 2026-09-21T05:45:00.000Z
	320	+ updatedAt: 2026-09-21T11:20:00.000Z
	321	+ closedAt: 2026-09-21T11:20:00.000Z
91	322	- id: 3
92		- title: "Test that every skill named in a shipped prompt actually resolves"
93		- state: open
	323	+ title: Test that every skill named in a shipped prompt actually resolves
	324	+ state: closed
94	325	priority: high
95	326	author:
96	327	name: k33g
97	328	email: ph.charriere@gmail.com
98	329	createdAt: 2026-09-21T05:45:00.000Z
	330	+ updatedAt: 2026-09-21T11:20:00.000Z
	331	+ closedAt: 2026-09-21T11:20:00.000Z
99	332	- id: 4
100		- title: "Parse keywords and render them in the catalogue; correct skill-format.md"
101		- state: open
	333	+ title: Parse keywords and render them in the catalogue; correct skill-format.md
	334	+ state: closed
102	335	priority: medium
103	336	author:
104	337	name: k33g
105	338	email: ph.charriere@gmail.com
106	339	createdAt: 2026-09-21T05:45:00.000Z
	340	+ updatedAt: 2026-09-21T11:20:00.000Z
	341	+ closedAt: 2026-09-21T11:20:00.000Z
107	342	- id: 5
108		- title: "Warn on a skill with no description; print the 📖 recap unconditionally and count only successes"
109		- state: open
	343	+ title: Warn on a skill with no description; print the 📖 recap unconditionally and count only successes
	344	+ state: closed
110	345	priority: medium
111	346	author:
112	347	name: k33g
113	348	email: ph.charriere@gmail.com
114	349	createdAt: 2026-09-21T05:45:00.000Z
	350	+ updatedAt: 2026-09-21T11:20:00.000Z
	351	+ closedAt: 2026-09-21T11:20:00.000Z
115	352	- id: 6
116		- title: "Build a skill-routing eval before any prompt change"
117		- state: open
	353	+ title: Build a skill-routing eval before any prompt change
	354	+ state: closed
118	355	priority: medium
119	356	author:
120	357	name: k33g
121	358	email: ph.charriere@gmail.com
122	359	createdAt: 2026-09-21T05:45:00.000Z
	360	+ updatedAt: 2026-09-21T13:10:00.000Z
	361	+ closedAt: 2026-09-21T13:10:00.000Z
123	362	- id: 7
124		- title: "Then: fix the SKILLS prompt block and add it to the built-in default prompt"
125		- state: open
	363	+ title: "Then: fix the SKILLS prompt block — MEASURED AND REJECTED, it makes routing worse"
	364	+ state: closed
126	365	priority: low
127	366	author:
128	367	name: k33g
129	368	email: ph.charriere@gmail.com
130	369	createdAt: 2026-09-21T05:45:00.000Z
	370	+ updatedAt: 2026-09-21T17:40:00.000Z
	371	+ closedAt: 2026-09-21T17:40:00.000Z
	372	+ - id: 8
	373	+ title: "Measure the shipped SKILLS blocks — DONE: the block earns +2 cases, and the escape sentence task 7 proposed cancels exactly that"
	374	+ state: closed
	375	+ priority: medium
	376	+ author:
	377	+ name: k33g
	378	+ email: ph.charriere@gmail.com
	379	+ createdAt: 2026-09-21T17:40:00.000Z
	380	+ updatedAt: 2026-09-21T20:15:00.000Z
	381	+ closedAt: 2026-09-21T20:15:00.000Z
131	382	comments: []

  # Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
  id: 15
  title: Improve skills detection
--state: open
++state: closed
  author:
    name: k33g
    email: ph.charriere@gmail.com
  createdAt: 2026-09-16T05:52:51.530Z
--updatedAt: 2026-09-21T05:45:00.000Z
++updatedAt: 2026-09-21T19:00:59.231Z
  labels: []
  milestone: v0.0.3
  body: |
    ### Not re-opened
    Decisions in force per `.memory/summary.md`, none of which the above touches: both layouts accepted; the flat file wins when both exist; `skills/README.md` skipped; relative `skillsDir` resolved by `resolveSkillsDir`; banner warning at count zero.
++  ---
++
++  ## Tier 1 implemented (Claude, 2026-09-21, branch `feature/skill-discovery`)
++
++  Tasks 1–5 closed. Tasks 6–7 (the eval, then the prompt block) deliberately left open: nothing there should start before the eval exists.
++
++  ### 1. `read_skill` resolves through the catalogue
++
++  `skills.Find` / `skills.Load` replace `skills.Read`. The model's string is now only ever compared against the names the catalogue advertises and **never reaches the filesystem** — so the `filepath.Base` barrier is gone because there is nothing left to escape from, which the test `TestALoadedNameNeverReachesTheFilesystem` pins.
++
++  Matching is forgiving in the two ways a model gets a name wrong and a human does not — stray whitespace or quotes, and the wrong case — and never forgiving about *which* skill: an exact name always beats a case-insensitive one.
++
++  **Found while doing it**: `List` used `sort.Slice`, which is **not stable**. The recorded rule "when both layouts claim a name, the flat file wins" was therefore decided by coin toss as soon as the lookup went through the sorted list. Now `sort.SliceStable` over a slice whose flat entries come first, and a test pins the rule.
++
++  ### 2 and 3. `bg-jobs`, and the test that prevents the class
++
++  `skills/bg-jobs/SKILL.md` written — start (both streams redirected, pid kept), **poll for readiness instead of sleeping**, inspect with `kill -0` and the log, stop politely then insist, and the traps (one shell per call, port already held, no stdin, stray `nohup.out`).
++
++  The durable half is `TestEverySkillNamedInAShippedPromptExists`: it scans every `agent*.yaml`, `tests/agent*.yaml` and the built-in default prompt for the `` `name` skill `` shape and asserts each one resolves. Verified by removing the new skill — it fails on all four sites at once, naming each.
++
++  ### 4. `keywords` parsed and rendered
++
++  Capped at 8 per skill, because the catalogue is paid for on every request. What the model now sees:
++
++  ```
++  greetings — Greet somebody with a festive message. (greet, greetings, hello, hi, welcome, bonjour, coucou, saluer)
++  ```
++
++  `docs/{en,fr}/reference/skill-format.md` corrected: it stated that `keywords` was "accepted in the file and ignored by the loader".
++
++  ### 5. Two new warnings, and a recap that can say zero
++
++  Banner: `N skill(s) with no description (…) — the model sees only the name and will pick blindly`, and `two skills share a name (…) — only the first is reachable`. Neither could fire before, since the only warning triggered at a count of zero.
++
++  `engine.Skills` now counts skills **loaded**, not `read_skill` responses: a miss answers `[skill: no skill named …]`, the `[tool: …]` shape the other tools use, and is not counted. The recap prints `📖 N skill(s)` whenever the agent has skills at all (`engine.SkillsAvailable`, set once by `main`), zero included.
++
++  **`📖 0 skill(s)` is the point of the whole item**: before, a turn where no skill fired and a turn where one fired and was ignored looked identical — the segment was simply absent.
++
++  ### Verified end to end (qwen3.5-4b against a real DMR)
++
++  | Asked | What happened |
++  |---|---|
++  | *« quelle heure est-il ? »* | reached `what-time-is-it` through the `heure` / `quelle heure` keywords → `⚙ 1 command(s) · 📖 1 skill(s)` |
++  | *"What is 17 times 23?"* | no skill, and it says so: `⚙ 0 command(s) · 📖 0 skill(s)` |
++  | *« Salue Bob de ma part »* | reached a skill from a French phrase — but picked `vulcan-salute` over `greetings`. **Both claim the same keyword space** (`salut` / `saluer`). The bridge works; the choice between two overlapping skills is exactly what task 6's eval is for. |
++
++  Quality gate run 34: 10 warnings, 20 smells — the baseline. `go test ./...`, `gofmt`, `go vet` green.
++
++  ### Not done, on purpose
++
++  Tasks 6 and 7. Also untouched: the `📖` line in the ACP trail still prints unconditionally on stderr (it always did), and the session `errors` trail still records no tool-level failure (Tier 3, item 10).
++
++  ---
++
++  ## Task 6 done: the routing eval exists, and here is its first reading (Claude, 2026-09-21)
++
++  `internal/eval`, opt-in:
++
++  ```
++  MM_EVAL=1 AGENT_CONFIG=tests/agent.dmr-qwen.yaml go test ./internal/eval -v -timeout 60m
++  ```
++
++  `go test ./...` skips it, so nothing in CI needs a server. Docs: `docs/{en,fr}/how-to/measure-skill-routing.md`.
++
++  ### What it measures
++
++  **The catalogue, not the agent.** One model turn per case, with stub tools that record what was asked for and run nothing — `read_skill` carries the real `skills.Catalogue(list)`, which is the thing under test. Rationale: `vulcan-salute` says "run this echo", and letting the agent loop would echo twenty times and measure recovery as much as routing.
++
++  20 cases, EN and FR, deliberately **not** phrased in the catalogue's words, including six where no skill applies and abstaining is the right answer.
++
++  Four outcomes, and **`wrong` is kept apart from `miss`** because they have different fixes: a skill loaded but the wrong one is a *catalogue* problem, nothing loaded at all is a *prompt* problem.
++
++  **No threshold, and it does not fail on a low score.** A number that fails a build becomes a target, and the first response to a red build is to edit the cases. Use it as a before/after, one change at a time.
++
++  ### First reading — qwen3.5-4b, 5 skills, catalogue 1048 bytes
++
++  ```
++  20 cases · hit 16 · miss 1 · error 3  →  80% right
++  ```
++
++  Of the 17 cases that completed: **16 hits, 1 miss**. Every skill was reached in both English and French, including `give Spock the proper farewell` → `vulcan-salute`, which shares no word with that skill's name or description and could only have come through the keywords. All six abstention cases were correct — the catalogue does not fire on "write a haiku about pointers".
++
++  **The one genuine routing miss**: `« qui appelle cette fonction ? »` → no skill. French, and about symbols. Its English twin (`where is the Greet function defined?`) did not miss. That is the kind of single, concrete finding this ticket wanted.
++
++  **The 3 errors are not routing at all** — the model could not compose a valid tool call: twice `unexpected end of JSON input`, once the runaway guard cutting off 2558 characters of repeated `{"command":"ls -laR …"}`. Same failure family the codebase already carries three guards for (`normalize`, `cleanSymbol`, `toolCallGuard`), now observed on a different model. **No catalogue or prompt work fixes that**, and the eval says so rather than scoring it.
++
++  ### The eval found two defects in itself, both fixed
++
++  1. The first stubs answered `[eval] command not run`. Given nothing to finish on, the model reran the same command until the runaway guard fired or the case timed out: **5 of 20 cases were artefacts of the instrument**. Stubs now answer that the work is done and the turn should end — run time also dropped from minutes per case to 5–20s.
++  2. `caseTimeout` was 2 minutes, and three good cases hit it exactly. Raised to 5 minutes: an `error` poisons the whole table, so the cap exists to catch a hung server, not to score speed.
++
++  Rows now stream to stderr as they complete (`testing` buffers `t.Log` to the end, and a silent multi-minute eval looks hung), and `os.Stdout` is redirected for the run because `engine.emit` writes the answer with `fmt.Print`, not through `ui.Out`.
++
++  ### What this changes for task 7
++
++  Task 7 (fix the SKILLS prompt block) now has an instrument and a baseline. Two things to know before touching it:
++
++  - The **`Salue Bob de ma part`** collision does **not** reproduce here: the eval picks `greetings` in both runs, where the full agent picked `vulcan-salute` earlier the same day. One-turn-with-stubs and the real loop are not the same setting. Keep the case, do not declare it fixed.
++  - The single miss is a **French** case whose English twin passes. That points at the catalogue's French coverage, not at the SKILLS block — worth checking before rewriting the prompt.
++
++  Quality gate run 35: 10 warnings, 20 smells — unchanged.
++
++  ---
++
++  ## What the eval established, and what it refuted (Claude, 2026-09-21)
++
++  The eval's first real use. **Three keyword hypotheses, three refutations, zero routing improvement — and that is the result.** Every one of the three sounded right and would have shipped without an instrument.
++
++  Baseline, mellum2-12b, 23 cases: `hit 17 · wrong 2 · miss 4`.
++
++  ### Refuted (1): "the French miss is because no French keyword survives the 8-word cap"
++
++  `lsp-navigation` has 16 keywords and the surviving 8 were English jargon, three of them duplicating the description. Reordered so French sat inside the cap. On qwen the totals did not move and cases merely shuffled; on mellum, `« qui appelle cette fonction ? »` **still misses**. Reverted.
++
++  ### Refuted (2): "`greet` as a keyword hijacks `where is the Greet function defined?`"
++
++  That request routes to `greetings` instead of `lsp-navigation`. Dropped `greet` from the keywords, re-measured — **it still routes to `greetings`**. So it is the skill's **NAME** colliding with the symbol `Greet`, and no keyword edit can fix it. Reverted.
++
++  **Finding worth keeping**: a skill named after a common code identifier will hijack requests about that identifier. Either rename the skill or accept it; keywords are the wrong lever.
++
++  ### Refuted (3): "`bg-jobs` needs its cap spread across its four actions"
++
++  Its surviving 8 say "start" four ways (`serve`, `daemon`, `nohup`, `start`) and "stop" none — `stop`, `kill`, `running` are all cut. Spread them across start/wait/inspect/stop. Result: **worse**, 17 hits → 15, then 16 on a second run, with `start the dev server in the background` stably flipping from hit to **miss**. The "redundant" synonyms were carrying weight; the keyword set does not behave additively. Reverted.
++
++  ### Established: the language is irrelevant
++
++  `bg-jobs` was missing in English and hitting in French, and the obvious reading was thin French coverage — except **`bg-jobs` has no French keyword at all**. Three cases added to the eval settle it, a clean 2×2:
++
++  | Request | Result |
++  |---|---|
++  | `start the dev server in the background` — English, **names** the concept | ✓ hit |
++  | `lance le serveur en tâche de fond` — French, **names** the concept | ✓ hit |
++  | `start the dev server and check it answers` — English, only a situation | · miss |
++  | `démarre le serveur de dev et vérifie qu'il répond` — French, only a situation | · miss |
++
++  **What decides it is whether the request names the concept the skill is about**, not which language it is in. That moves the remaining `bg-jobs` failures out of the catalogue and into the prompt: the model has no reason to know that starting a server blocks the turn.
++
++  ### Established: run the eval on mellum2-12b
++
++  qwen3.5-4b errors on all three `lsp` cases — tool-call composition, not routing (`unexpected end of JSON input`, runaway guard) — which makes `lsp` routing unmeasurable on that model. mellum2-12b runs all 23 cases clean in about two minutes instead of twenty.
++
++  ### What this means for task 7
++
++  Task 7 proposes rewriting the `SKILLS` block of the system prompt. The three known failures are now characterised, and **two of the three are prompt problems after all**:
++
++  - `start the dev server …` and `démarre le serveur …` — the model does not know a server blocks. A prompt that says "before acting, check whether a procedure covers this" is the right lever. **This is task 7, and it now has a measurable target.**
++  - `qui appelle cette fonction ?` — names no symbol. Likely also a prompt matter, or simply out of reach.
++  - `where is the Greet function defined?` — a NAME collision. Not task 7; not keywords either.
++
++  So task 7 is worth doing, but for different reasons than the ticket originally assumed, and against a baseline that now exists.
++
++  ### Also fixed on the way
++
++  Keyword ORDER became significant when the cap was introduced, and nothing told an author their last words were dropped. `skills.Truncated` + a banner warning + the rule in `reference/skill-format.md` (put the most distinctive first; do not spend the eight on words the `description` already carries, since the description is one language and the keywords are where the others belong).
++
++  `internal/lsp`'s `TestOperationsOverAFakeServer` was flaky at ~10% and is fixed (`fakeServer.awaitDoc`, a bounded poll — `didChange` is a notification and the test read the server's state out of band). 100 runs green, 60 under `-race`.
++
++  **Not fixed, reported**: `internal/engine`'s `TestExplainThroughTheStack/connection_refused` fails about once in thirty. `closedURL` records an ephemeral port and closes it, assuming nothing takes it — a TOCTOU race. One-line fix: return `http://127.0.0.1:1` instead.
++
++  `skills/greetings`, `skills/lsp-navigation` and `skills/bg-jobs` are byte-identical to before the experiments. Quality gate run 37: 10 warnings, 20 smells.
++
++  ---
++
++  ## Task 7 measured, and REJECTED on evidence (Claude, 2026-09-21)
++
++  First, a fact nobody had noticed: **the eval's configuration and the built-in default prompt have no `SKILLS` block at all.** So the whole baseline above — 17 of 23 — was the catalogue routing on its own, with zero prompt guidance. That gave task 7 a clean "before".
++
++  Four conditions, same model (mellum2-12b), same 23 cases, one change each:
++
++  | `SKILLS` block | hit | wrong | miss | **error** |
++  |---|---|---|---|---|
++  | none (run 1) | 17 | 2 | 4 | **0** |
++  | none (run 2) | 16 | 2 | 5 | **0** |
++  | 2 lines | 16 | 2 | 5 | **0** |
++  | 9 lines (the proposed wording) | 14 | 2 | 3 | **4** |
++
++  Two conclusions, and the second is the important one.
++
++  **1. No block and a short block are indistinguishable.** 17 / 16 / 16 — the baseline itself moves by a case, so a one-case difference means nothing. Telling the model to check the catalogue does not make it check the catalogue.
++
++  **2. The nine-line block broke tool-call composition.** Four cases errored — `unexpected end of JSON input`, `generation timed out (stalled)` — where all three other conditions had zero. And it cost the two `bg-jobs` cases that had reliably worked: `lance le serveur en tâche de fond` and `start the dev server in the background` both went from hit to miss.
++
++  The nine-line and two-line blocks say the same thing; only the length differs, and only the long one produced errors. **So prompt weight, not prompt content, is what hurt** — which is exactly the argument ticket 4 records against MCP ("20 MCP tools sit in every request forever, and mm's target model class already mangles a 6-field schema"), here observed directly on nine lines of prose rather than on tool schemas.
++
++  Also constant across all four conditions: the two `wrong` cases (`where is the Greet function defined?` → `greetings`). The name collision moves with neither keywords nor prompt.
++
++  ### So task 7 is closed as rejected
++
++  The premise — "the model does not think to look, so tell it to look" — does not survive the instrument. Nothing was changed in the repository: all four prompt variants lived in `/tmp`.
++
++  ### And it opened a better question (task 8)
++
++  `agent.dmr.yaml`, `agent.llamacpp.yaml` and `agent.lsp.yaml` all **ship a `SKILLS` block of about six lines**, and it has never been measured. Given the above, it may be costing more than it gives. I did not touch them — I measured a block of my own wording on a config that had none, which is not the same experiment. Measuring theirs is now the open question, and the eval makes it a two-minute job: run it against each shipped YAML with the block, then with it commented out.
++
++  ---
++
++  ## Task 8 done — and it CORRECTS the task 7 write-up above (Claude, 2026-09-21)
++
++  Measured `agent.dmr.yaml`'s own `SKILLS` block, in its own prompt, on mellum2-12b, 23 cases, **two runs of each condition** because the differences are two cases wide:
++
++  | Condition | run 1 | run 2 |
++  |---|---|---|
++  | the shipped block, as it ships | **hit 18** · wrong 2 · miss 3 | **hit 18** · wrong 1 · miss 3 · extra 1 |
++  | the shipped block **+ the no-match escape sentence** | hit 16 · wrong 1 · miss 6 | hit 16 · wrong 1 · miss 6 |
++  | no block at all | hit 16 · wrong 2 · miss 5 | hit 16 · wrong 2 · miss 5 |
++
++  **The shipped block earns +2 cases, reproducibly. Do not remove it.** That answers task 8: it is not costing more than it gives, it is paying for itself.
++
++  ### And it isolates why my own block failed
++
++  The sentence I added, following this ticket's own task 7 —
++
++  > If none of them covers the request, do not mention skills — just do the work.
++
++  — **costs exactly the block's entire benefit**: 18 → 16, back to the no-block level, with misses going 3 → 6. It under-fires, which is the predicted mechanism: telling a model it may skip is an invitation to skip, and a 12B model takes it.
++
++  So the task 7 write-up above is **too broad where it says prompt guidance does not help routing**. Corrected:
++
++  - **Prompt guidance DOES help** — the shipped, unconditional block is worth +2 over nothing.
++  - **The explicit no-match escape is what hurts**, and it was the specific thing task 7 proposed adding. Isolated to one sentence, reproduced twice.
++  - The "prompt weight" reading also needs narrowing: the shipped prompt is much LONGER overall than the one I first tested, and its six-line block adds **no errors at all**. Length alone does not predict harm. The four tool-call failures my nine-line block produced remain unexplained — weight or wording, not separated.
++
++  ### What to do with this
++
++  Leave all three shipped blocks alone. If anyone is tempted to "soften" a SKILLS block with a no-match escape — which reads like an obvious improvement, and is why this ticket proposed it — the measurement says it costs two cases in twenty-three.
++
++  Method note for whoever runs this next: the two conditions must differ by **one** thing. These were derived from the same `agent.dmr.yaml` with only the block removed or one sentence added; the model, the catalogue, the skills directory and the rest of the prompt were identical.
  tasks:
    - id: 1
--    title: "Resolve read_skill through the catalogue's name, not the file path"
++    title: Resolve read_skill through the catalogue's name, not the file path
--    state: open
++    state: closed
      priority: high
      author:
        name: k33g
        email: ph.charriere@gmail.com
      createdAt: 2026-09-21T05:45:00.000Z
++    updatedAt: 2026-09-21T11:20:00.000Z
++    closedAt: 2026-09-21T11:20:00.000Z
    - id: 2
      title: "bg-jobs: ship the skill or delete the sentence from the 3 YAMLs and the default prompt"
--    state: open
++    state: closed
      priority: high
      author:
        name: k33g
        email: ph.charriere@gmail.com
      createdAt: 2026-09-21T05:45:00.000Z
++    updatedAt: 2026-09-21T11:20:00.000Z
++    closedAt: 2026-09-21T11:20:00.000Z
    - id: 3
--    title: "Test that every skill named in a shipped prompt actually resolves"
++    title: Test that every skill named in a shipped prompt actually resolves
--    state: open
++    state: closed
      priority: high
      author:
        name: k33g
        email: ph.charriere@gmail.com
      createdAt: 2026-09-21T05:45:00.000Z
++    updatedAt: 2026-09-21T11:20:00.000Z
++    closedAt: 2026-09-21T11:20:00.000Z
    - id: 4
--    title: "Parse keywords and render them in the catalogue; correct skill-format.md"
++    title: Parse keywords and render them in the catalogue; correct skill-format.md
--    state: open
++    state: closed
      priority: medium
      author:
        name: k33g
        email: ph.charriere@gmail.com
      createdAt: 2026-09-21T05:45:00.000Z
++    updatedAt: 2026-09-21T11:20:00.000Z
++    closedAt: 2026-09-21T11:20:00.000Z
    - id: 5
--    title: "Warn on a skill with no description; print the 📖 recap unconditionally and count only successes"
++    title: Warn on a skill with no description; print the 📖 recap unconditionally and count only successes
--    state: open
++    state: closed
      priority: medium
      author:
        name: k33g
        email: ph.charriere@gmail.com
      createdAt: 2026-09-21T05:45:00.000Z
++    updatedAt: 2026-09-21T11:20:00.000Z
++    closedAt: 2026-09-21T11:20:00.000Z
    - id: 6
--    title: "Build a skill-routing eval before any prompt change"
++    title: Build a skill-routing eval before any prompt change
--    state: open
++    state: closed
      priority: medium
      author:
        name: k33g
        email: ph.charriere@gmail.com
      createdAt: 2026-09-21T05:45:00.000Z
++    updatedAt: 2026-09-21T13:10:00.000Z
++    closedAt: 2026-09-21T13:10:00.000Z
    - id: 7
--    title: "Then: fix the SKILLS prompt block and add it to the built-in default prompt"
++    title: "Then: fix the SKILLS prompt block — MEASURED AND REJECTED, it makes routing worse"
--    state: open
++    state: closed
      priority: low
      author:
        name: k33g
        email: ph.charriere@gmail.com
      createdAt: 2026-09-21T05:45:00.000Z
++    updatedAt: 2026-09-21T17:40:00.000Z
++    closedAt: 2026-09-21T17:40:00.000Z
++  - id: 8
++    title: "Measure the shipped SKILLS blocks — DONE: the block earns +2 cases, and the escape sentence task 7 proposed cancels exactly that"
++    state: closed
++    priority: medium
++    author:
++      name: k33g
++      email: ph.charriere@gmail.com
++    createdAt: 2026-09-21T17:40:00.000Z
++    updatedAt: 2026-09-21T20:15:00.000Z
++    closedAt: 2026-09-21T20:15:00.000Z
  comments: []

added .tickets/issues/0037-add-a-file-selector-to-the-tui-mode.yaml +13 -0 Viewed

new file mode 100644
@@ -0,0 +1,13 @@
	1	+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
	2	+id: 37
	3	+title: Add a file selector to the tui mode
	4	+state: open
	5	+author:
	6	+ name: k33g
	7	+ email: ph.charriere@gmail.com
	8	+createdAt: 2026-09-21T18:55:37.512Z
	9	+updatedAt: 2026-09-21T18:55:37.512Z
	10	+labels: []
	11	+body: ""
	12	+tasks: []
	13	+comments: []

++# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
++id: 37
++title: Add a file selector to the tui mode
++state: open
++author:
++  name: k33g
++  email: ph.charriere@gmail.com
++createdAt: 2026-09-21T18:55:37.512Z
++updatedAt: 2026-09-21T18:55:37.512Z
++labels: []
++body: ""
++tasks: []
++comments: []

added .tickets/issues/0038-add-a-skill-selector.yaml +13 -0 Viewed

new file mode 100644
@@ -0,0 +1,13 @@
	1	+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
	2	+id: 38
	3	+title: add a skill selector
	4	+state: open
	5	+author:
	6	+ name: k33g
	7	+ email: ph.charriere@gmail.com
	8	+createdAt: 2026-09-21T19:00:12.508Z
	9	+updatedAt: 2026-09-21T19:00:12.508Z
	10	+labels: []
	11	+body: ""
	12	+tasks: []
	13	+comments: []

++# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
++id: 38
++title: add a skill selector
++state: open
++author:
++  name: k33g
++  email: ph.charriere@gmail.com
++createdAt: 2026-09-21T19:00:12.508Z
++updatedAt: 2026-09-21T19:00:12.508Z
++labels: []
++body: ""
++tasks: []
++comments: []

added demo/agent.dmr.yaml +181 -0 Viewed

new file mode 100644
@@ -0,0 +1,181 @@
	1	+# Agent settings.
	2	+#
	3	+provider: dmr
	4	+
	5	+#model: huggingface.co/jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M
	6	+model: huggingface.co/unsloth/gemma-4-e2b-it-gguf:Q4_K_M
	7	+
	8	+baseUrl: http://localhost:12434/engines/v1
	9	+fallback: http://host.docker.internal:12434/engines/v1
	10	+
	11	+
	12	+# How many tokens the server actually serves.
	13	+# 0 = ask the server
	14	+contextWindow: 0
	15	+
	16	+# Max number of characters a tool returns to the model.
	17	+# Context safeguard: beyond that, the output is truncated (beginning + end kept).
	18	+maxOutput: 16000
	19	+
	20	+# Max number of model <-> tools round trips for a single question.
	21	+# Raise it for tasks that need long chains of commands.
	22	+maxTurns: 40
	23	+
	24	+# The `bash` tool.
	25	+# `false` removes it: the model can still read and edit files (editTools) and load skills,
	26	+# but never runs a command. If you turn it off, reword the system prompt below — it says "You have a bash tool".
	27	+bashTool: true
	28	+
	29	+# The built-in file tools: read_file, write_file, edit_file.
	30	+editTools: true
	31	+
	32	+lsp:
	33	+ servers:
	34	+ - name: gopls # go install golang.org/x/tools/gopls@latest
	35	+ command: gopls
	36	+ extensions: [".go"]
	37	+
	38	+# Language servers for the `lsp` tool (definition, references, hover, symbols, diagnostics, rename).
	39	+# The file extension picks the server. A server missing from the PATH is skipped with a warning.
	40	+# No server listed = no `lsp` tool. Examples (uncomment what you have installed):
	41	+# lsp:
	42	+# servers:
	43	+# - name: gopls
	44	+# command: gopls
	45	+# extensions: [".go"]
	46	+# - name: typescript
	47	+# command: typescript-language-server
	48	+# args: ["--stdio"]
	49	+# extensions: [".ts", ".tsx", ".js", ".jsx"]
	50	+# - name: python
	51	+# command: pyright-langserver
	52	+# args: ["--stdio"]
	53	+# extensions: [".py"]
	54	+# - name: rust
	55	+# command: rust-analyzer
	56	+# extensions: [".rs"]
	57	+
	58	+# Where the `read_skill` tool looks for its markdown procedures.
	59	+# The path is relative to the CURRENT DIRECTORY, so run the agent from the directory that holds skills.
	60	+skillsDir: .mm/skills
	61	+
	62	+# Persistent sessions, per project.
	63	+# When enabled, every conversation is saved as one JSON file under `dir`
	64	+sessions:
	65	+ enabled: true
	66	+ dir: .mm/sessions
	67	+
	68	+# How many lines of a command's output are echoed to the terminal.
	69	+# 0 disables it.
	70	+previewLines: 20
	71	+
	72	+# The system prompt: what the agent is, and what it is allowed to do.
	73	+# This is the knob to play with — it is the shortest path to changing behaviour.
	74	+system: \|
	75	+ Your name is Bob.
	76	+ You are a coding agent working in a terminal.
	77	+ You have a "bash" tool to run shell commands.
	78	+ Use it to explore files, run tests, inspect the repository, etc.
	79	+ Chain several commands if needed, then answer clearly in English.
	80	+
	81	+ A request often mixes things you answer from yourself ("say hello") with
	82	+ things only a command can answer ("list the files"). Handle every part, in
	83	+ the order asked, and run a command for each part that needs one.
	84	+ Never state the contents of a file, the output of a command, or the state of
	85	+ the repository unless a command in THIS answer returned it. What you did not
	86	+ read, you do not know: run the command instead of recalling it.
	87	+
	88	+ SKILLS
	89	+ You have a second tool, `read_skill`. Its description lists the procedures
	90	+ available for this project — one per kind of task.
	91	+
	92	+ Any request to DO something to a Go project is a skill, not a shell command
	93	+ you invent. Match the request against that list, call `read_skill` FIRST,
	94	+ before any bash command, and then follow what it says step by step.
	95	+
	96	+ FILE EDITING
	97	+ You have three tools for files: `read_file`, `edit_file` and `write_file`.
	98	+ They are how a file gets read and changed here: each change is exact,
	99	+ checked before it is written, and comes back as a diff with line numbers.
	100	+ bash is for running things — building, testing, listing, searching.
	101	+
	102	+ - Read before you write: call `read_file` on the file (numbered=true when
	103	+ you need line numbers). You cannot target text you have not seen; never
	104	+ rely on what you think you remember about a file.
	105	+ - To change an existing file, call `edit_file` with one or more {old, new}
	106	+ pairs. `old` is copied from the file character for character — same
	107	+ spaces, same indentation, same line breaks — and appears exactly once:
	108	+ add the surrounding lines until it is unique. Several pairs are applied
	109	+ together, against the original file. An empty `new` deletes the text.
	110	+ - Call `write_file` only to create a file, or to rewrite one entirely and
	111	+ on purpose. On an existing file it replaces everything, including what
	112	+ you did not intend to touch.
	113	+ - Read the diff the tool returns: it says exactly what changed and on which
	114	+ line. If `edit_file` refuses — text not found, ambiguous, overlapping
	115	+ edits — read the file again and fix `old`. Do not fall back to
	116	+ `write_file` to force the change through.
	117	+ - After editing code, run the narrowest check with bash: the formatter, the
	118	+ compiler, or the test covering that file.
	119	+
	120	+ RULES
	121	+ - Keep everything the file already does, unless the user asked to remove it.
	122	+ - Touch only the files the request is about. Do not add tests, files or
	123	+ features that were not asked for.
	124	+ - Never run a git command unless the user says git, commit or push.
	125	+ - Never move, rename or delete a file unless the user asked for it.
	126	+ - Then answer in English, in a few lines.
	127	+ - If you don't know how to use a <cli>, run `<cli> --help` (or `<cli> help`)
	128	+ to understand the options, then run the command.
	129	+
	130	+ BACKGROUND JOBS
	131	+ Never let a command block the answer. Anything that serves, watches or runs
	132	+ long goes to the background, with BOTH streams redirected and its pid kept:
	133	+
	134	+ nohup <command> > /tmp/<job>.log 2>&1 & echo $! > /tmp/<job>.pid
	135	+
	136	+ Redirecting only stdout still blocks until the process exits. Read the
	137	+ `bg-jobs` skill before you wait on, inspect or stop such a job — each has a
	138	+ rule you cannot guess. Stop every job you started before you finish, and say
	139	+ which ones you left running.
	140	+
	141	+# Generation settings (OpenAI API keys). Low temperature for a coding agent:
	142	+# we want precise and reproducible answers, not creativity.
	143	+sampling:
	144	+ temperature: 0.0
	145	+ parallel_tool_calls: false
	146	+ top_p: 0.9
	147	+ max_tokens: 4096
	148	+
	149	+watchdogTimeout: 30s
	150	+
	151	+# Context compression
	152	+context:
	153	+ enabled: false
	154	+
	155	+ # Compress when the history reaches this share of the window.
	156	+ threshold: 75
	157	+
	158	+ # Fallback on the message count, for when the window is unknown.
	159	+ # One command costs 2 messages (call + response), so 80 is roughly 30 commands of history.
	160	+ maxMessages: 80
	161	+
	162	+ # How many recent questions (with their commands and answers) are kept as-is instead of being summarised.
	163	+ # The model works on the last few turns, so summarising them would make it redo what it just did.
	164	+ keepLastTurns: 3
	165	+
	166	+ # max_tokens of the summary request.
	167	+ summaryMaxTokens: 1200
	168	+
	169	+ # Replaces the built-in summary prompt when set.
	170	+ # Empty = built-in.
	171	+ prompt: ""
	172	+
	173	+ showStats: true
	174	+
	175	+displayCommands: true
	176	+# Show what a thinking model reasons about before it answers (dim italic in the
	177	+# terminal, its own style in the window, a thought chunk in an editor).
	178	+# The reasoning is always READ — it is what keeps the watchdog alive while the
	179	+# model thinks — this only says whether to show it.
	180	+showThinking: true
	181	+

++# Agent settings.
++#
++provider: dmr
++
++#model: huggingface.co/jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M
++model: huggingface.co/unsloth/gemma-4-e2b-it-gguf:Q4_K_M
++
++baseUrl: http://localhost:12434/engines/v1
++fallback: http://host.docker.internal:12434/engines/v1
++
++
++# How many tokens the server actually serves.
++# 0 = ask the server
++contextWindow: 0
++
++# Max number of characters a tool returns to the model.
++# Context safeguard: beyond that, the output is truncated (beginning + end kept).
++maxOutput: 16000
++
++# Max number of model <-> tools round trips for a single question.
++# Raise it for tasks that need long chains of commands.
++maxTurns: 40
++
++# The `bash` tool.
++# `false` removes it: the model can still read and edit files (editTools) and load skills,
++# but never runs a command. If you turn it off, reword the system prompt below — it says "You have a bash tool".
++bashTool: true
++
++# The built-in file tools: read_file, write_file, edit_file.
++editTools: true
++
++lsp:
++  servers:
++    - name: gopls                              # go install golang.org/x/tools/gopls@latest
++      command: gopls
++      extensions: [".go"]
++
++# Language servers for the `lsp` tool (definition, references, hover, symbols, diagnostics, rename).
++# The file extension picks the server. A server missing from the PATH is skipped with a warning.
++# No server listed = no `lsp` tool. Examples (uncomment what you have installed):
++# lsp:
++#   servers:
++#     - name: gopls
++#       command: gopls
++#       extensions: [".go"]
++#     - name: typescript
++#       command: typescript-language-server
++#       args: ["--stdio"]
++#       extensions: [".ts", ".tsx", ".js", ".jsx"]
++#     - name: python
++#       command: pyright-langserver
++#       args: ["--stdio"]
++#       extensions: [".py"]
++#     - name: rust
++#       command: rust-analyzer
++#       extensions: [".rs"]
++
++# Where the `read_skill` tool looks for its markdown procedures.
++# The path is relative to the CURRENT DIRECTORY, so run the agent from the directory that holds skills.
++skillsDir: .mm/skills
++
++# Persistent sessions, per project.
++# When enabled, every conversation is saved as one JSON file under `dir`
++sessions:
++  enabled: true
++  dir: .mm/sessions
++
++# How many lines of a command's output are echoed to the terminal.
++# 0 disables it.
++previewLines: 20
++
++# The system prompt: what the agent is, and what it is allowed to do.
++# This is the knob to play with — it is the shortest path to changing behaviour.
++system: |
++  Your name is Bob.
++  You are a coding agent working in a terminal.
++  You have a "bash" tool to run shell commands.
++  Use it to explore files, run tests, inspect the repository, etc.
++  Chain several commands if needed, then answer clearly in English.
++
++  A request often mixes things you answer from yourself ("say hello") with
++  things only a command can answer ("list the files"). Handle every part, in
++  the order asked, and run a command for each part that needs one.
++  Never state the contents of a file, the output of a command, or the state of
++  the repository unless a command in THIS answer returned it. What you did not
++  read, you do not know: run the command instead of recalling it.
++
++  SKILLS
++  You have a second tool, `read_skill`. Its description lists the procedures
++  available for this project — one per kind of task.
++
++  Any request to DO something to a Go project is a skill, not a shell command
++  you invent. Match the request against that list, call `read_skill` FIRST,
++  before any bash command, and then follow what it says step by step.
++
++  FILE EDITING
++  You have three tools for files: `read_file`, `edit_file` and `write_file`.
++  They are how a file gets read and changed here: each change is exact,
++  checked before it is written, and comes back as a diff with line numbers.
++  bash is for running things — building, testing, listing, searching.
++
++  - Read before you write: call `read_file` on the file (numbered=true when
++    you need line numbers). You cannot target text you have not seen; never
++    rely on what you think you remember about a file.
++  - To change an existing file, call `edit_file` with one or more {old, new}
++    pairs. `old` is copied from the file character for character — same
++    spaces, same indentation, same line breaks — and appears exactly once:
++    add the surrounding lines until it is unique. Several pairs are applied
++    together, against the original file. An empty `new` deletes the text.
++  - Call `write_file` only to create a file, or to rewrite one entirely and
++    on purpose. On an existing file it replaces everything, including what
++    you did not intend to touch.
++  - Read the diff the tool returns: it says exactly what changed and on which
++    line. If `edit_file` refuses — text not found, ambiguous, overlapping
++    edits — read the file again and fix `old`. Do not fall back to
++    `write_file` to force the change through.
++  - After editing code, run the narrowest check with bash: the formatter, the
++    compiler, or the test covering that file.
++
++  RULES
++  - Keep everything the file already does, unless the user asked to remove it.
++  - Touch only the files the request is about. Do not add tests, files or
++    features that were not asked for.
++  - Never run a git command unless the user says git, commit or push.
++  - Never move, rename or delete a file unless the user asked for it.
++  - Then answer in English, in a few lines.
++  - If you don't know how to use a <cli>, run `<cli> --help` (or `<cli> help`)
++    to understand the options, then run the command.
++
++  BACKGROUND JOBS
++  Never let a command block the answer. Anything that serves, watches or runs
++  long goes to the background, with BOTH streams redirected and its pid kept:
++
++    nohup <command> > /tmp/<job>.log 2>&1 & echo $! > /tmp/<job>.pid
++
++  Redirecting only stdout still blocks until the process exits. Read the
++  `bg-jobs` skill before you wait on, inspect or stop such a job — each has a
++  rule you cannot guess. Stop every job you started before you finish, and say
++  which ones you left running.
++
++# Generation settings (OpenAI API keys). Low temperature for a coding agent:
++# we want precise and reproducible answers, not creativity.
++sampling:
++  temperature: 0.0
++  parallel_tool_calls: false
++  top_p: 0.9
++  max_tokens: 4096
++
++watchdogTimeout: 30s
++
++# Context compression
++context:
++  enabled: false
++
++  # Compress when the history reaches this share of the window.
++  threshold: 75
++
++  # Fallback on the message count, for when the window is unknown.
++  # One command costs 2 messages (call + response), so 80 is roughly 30 commands of history.
++  maxMessages: 80
++
++  # How many recent questions (with their commands and answers) are kept as-is instead of being summarised.
++  # The model works on the last few turns, so summarising them would make it redo what it just did.
++  keepLastTurns: 3
++
++  # max_tokens of the summary request.
++  summaryMaxTokens: 1200
++
++  # Replaces the built-in summary prompt when set.
++  # Empty = built-in.
++  prompt: ""
++
++  showStats: true
++
++displayCommands: true
++# Show what a thinking model reasons about before it answers (dim italic in the
++# terminal, its own style in the window, a thought chunk in an editor).
++# The reasoning is always READ — it is what keeps the watchdog alive while the
++# model thinks — this only says whether to show it.
++showThinking: true
++

added demo/main.go +41 -0 Viewed

new file mode 100644
@@ -0,0 +1,41 @@
	1	+package main
	2	+
	3	+import (
	4	+ "encoding/json"
	5	+ "fmt"
	6	+ "log"
	7	+ "net/http"
	8	+)
	9	+
	10	+// Animal struct to represent an animal
	11	+type Animal struct {
	12	+ Name string `json:"name"`
	13	+ Species string `json:"species"`
	14	+}
	15	+
	16	+// Sample data
	17	+var animals = []Animal{
	18	+ {Name: "Lion", Species: "Mammal"},
	19	+ {Name: "Elephant", Species: "Mammal"},
	20	+ {Name: "Eagle", Species: "Bird"},
	21	+ {Name: "Shark", Species: "Fish"},
	22	+}
	23	+
	24	+// handler function to serve the animal list
	25	+func animalHandler(w http.ResponseWriter, r *http.Request) {
	26	+ // Set the content type to JSON
	27	+ w.Header().Set("Content-Type", "application/json")
	28	+
	29	+ // Encode the animal slice to JSON
	30	+ json.NewEncoder(w).Encode(animals)
	31	+}
	32	+
	33	+func main() {
	34	+ // Define the handler for the root path
	35	+ http.HandleFunc("/", animalHandler)
	36	+
	37	+ // Start the server on port 8080
	38	+ port := ":8080"
	39	+ fmt.Printf("Starting server on http://localhost%s\n", port)
	40	+ log.Fatal(http.ListenAndServe(port, nil))
	41	+}

++package main
++
++import (
++	"encoding/json"
++	"fmt"
++	"log"
++	"net/http"
++)
++
++// Animal struct to represent an animal
++type Animal struct {
++	Name string `json:"name"`
++	Species string `json:"species"`
++}
++
++// Sample data
++var animals = []Animal{
++	{Name: "Lion", Species: "Mammal"},
++	{Name: "Elephant", Species: "Mammal"},
++	{Name: "Eagle", Species: "Bird"},
++	{Name: "Shark", Species: "Fish"},
++}
++
++// handler function to serve the animal list
++func animalHandler(w http.ResponseWriter, r *http.Request) {
++	// Set the content type to JSON
++	w.Header().Set("Content-Type", "application/json")
++
++	// Encode the animal slice to JSON
++	json.NewEncoder(w).Encode(animals)
++}
++
++func main() {
++	// Define the handler for the root path
++	http.HandleFunc("/", animalHandler)
++
++	// Start the server on port 8080
++	port := ":8080"
++	fmt.Printf("Starting server on http://localhost%s\n", port)
++	log.Fatal(http.ListenAndServe(port, nil))
++}

modified docs/en/README.md +1 -0 Viewed

@@ -13,6 +13,7 @@ This documentation is organised along the four [Diátaxis](https://diataxis.fr)
13	13	- [Run mini-me on llama.cpp instead of Docker Model Runner](how-to/run-with-llama-cpp.md)
14	14	- [Use mini-me from an editor (ACP)](how-to/use-from-an-editor.md)
15	15	- [Add a skill](how-to/add-a-skill.md)
	16	+- [Measure whether the model picks the right skill](how-to/measure-skill-routing.md) — the routing eval: run it, read its table, use it as a before-and-after
16	17	- [Keep a long session inside the context window](how-to/manage-the-context-window.md)
17	18	- [Start a new session without restarting](how-to/start-a-new-session.md)
18	19	- [Point the model at a file with `@path`](how-to/attach-a-file.md)

  - [Run mini-me on llama.cpp instead of Docker Model Runner](how-to/run-with-llama-cpp.md)
  - [Use mini-me from an editor (ACP)](how-to/use-from-an-editor.md)
  - [Add a skill](how-to/add-a-skill.md)
++- [Measure whether the model picks the right skill](how-to/measure-skill-routing.md) — the routing eval: run it, read its table, use it as a before-and-after
  - [Keep a long session inside the context window](how-to/manage-the-context-window.md)
  - [Start a new session without restarting](how-to/start-a-new-session.md)
  - [Point the model at a file with `@path`](how-to/attach-a-file.md)

added docs/en/how-to/measure-skill-routing.md +90 -0 Viewed

new file mode 100644
@@ -0,0 +1,90 @@
	1	+# Measure whether the model picks the right skill
	2	+
	3	+> Run the routing eval, read its table, and use it to decide whether a change to the catalogue or to the prompt helped.
	4	+
	5	+`mm` shows a skill in the catalogue and hopes the model reaches for it. Whether it does is a property of the model, not of the code, so no ordinary test can answer it. This one asks a live model, twenty times.
	6	+
	7	+## Run it
	8	+
	9	+```bash
	10	+MM_EVAL=1 AGENT_CONFIG=tests/agent.dmr-qwen.yaml go test ./internal/eval -v -timeout 25m
	11	+```
	12	+
	13	+Both variables are required in practice: without `MM_EVAL` the test skips, and without `AGENT_CONFIG` it evaluates the built-in defaults rather than the configuration you care about. `go test ./...` skips it, so nothing in CI depends on a server being up.
	14	+
	15	+Expect a few minutes: one model turn per case, and small models are not fast. Rows are printed as they complete.
	16	+
	17	+## Read the table
	18	+
	19	+```
	20	+[ 6/20] ✓ hit want=what-time-is-it got=what-time-is-it 1.204s "quelle heure est-il ?" ← French
	21	+[14/20] ≠ wrong want=greetings got=vulcan-salute 0.981s "Salue Bob de ma part" ← two skills claim salut/saluer
	22	+[16/20] · miss want=bg-jobs got=(none) 2.017s "lance le serveur en tâche de fond"
	23	+[17/20] ! extra want=(none) got=greetings 0.774s "write a haiku about pointers"
	24	+```
	25	+
	26	+\| Mark \| Meaning \| What it points at \|
	27	+\|---\|---\|---\|
	28	+\| `✓ hit` \| the expected skill, or correctly no skill \| — \|
	29	+\| `≠ wrong` \| a skill was loaded, but not that one \| the catalogue: two entries the model cannot tell apart \|
	30	+\| `· miss` \| a skill applied and none was loaded \| the prompt: the model did not think to look \|
	31	+\| `! extra` \| no skill applied and one was loaded \| over-eager instruction, or a catalogue entry that claims too much \|
	32	+\| `✗ error` \| the turn failed \| never a routing score. Either the instrument (a stub that starved the model, `caseTimeout` too short, a server that went away) or the model being unable to compose a valid tool call at all (`unexpected end of JSON input`, or the runaway guard cutting off repeated arguments). The message says which. Fails the test, because the numbers are then incomplete. \|
	33	+
	34	+`wrong` and `miss` are kept apart on purpose. Merging them hides which of the two problems you have, and they have different fixes.
	35	+
	36	+## Use it
	37	+
	38	+The number on its own means nothing. There is no threshold and the test does not fail on a low score — a number that fails a build becomes a target, and the first thing anyone does with a red build is edit the cases.
	39	+
	40	+The eval is a before and after:
	41	+
	42	+1. Run it, keep the output.
	43	+2. Change one thing — a skill's `keywords`, the `SKILLS` block of the system prompt, a `description`.
	44	+3. Run it again against the same model, with the same configuration.
	45	+4. Compare.
	46	+
	47	+One change at a time. Two at once and you have measured their sum.
	48	+
	49	+Small models at `temperature: 0.0` are reproducible enough for this to mean something; run the eval twice before trusting a difference of one case.
	50	+
	51	+## What it measures, and what it does not
	52	+
	53	+It measures the catalogue — the tool description the model reads — under one system prompt. One model turn per case, with tools that record what was asked for and run nothing: a skill that says "run this echo" must not echo twenty times, and letting the agent loop would measure recovery as much as routing.
	54	+
	55	+It does not measure whether the skill, once loaded, was followed correctly. That is a different question and needs a different instrument.
	56	+
	57	+It also does not measure whether the model can emit a valid tool call. When it cannot — and small models often cannot — you get `✗ error`, not a low routing score. Those two failures have nothing to do with each other and no amount of catalogue work fixes the second one. Measured on `qwen3.5-4b`: three of twenty cases failed this way, reproducibly, on the prompts that invite a more structured call.
	58	+
	59	+## Add a case
	60	+
	61	+`internal/eval/routing_test.go`, the `cases` table:
	62	+
	63	+```go
	64	+{"lance le serveur en tâche de fond", "bg-jobs", "French"},
	65	+{"what is 17 times 23?", none, ""},
	66	+```
	67	+
	68	+Two rules that keep the numbers honest:
	69	+
	70	+- Do not phrase the prompt in the words of the catalogue. A case that says "use the greetings skill" measures nothing.
	71	+- Keep the `none` cases. A catalogue that fires on everything is as broken as one that never fires, and abstaining correctly is a hit.
	72	+
	73	+When you find a real routing failure, add it as a case before fixing it. That is how the eval stops the same failure coming back.
	74	+
	75	+## Write cases that can separate explanations
	76	+
	77	+A single case tells you something failed. A set of cases tells you why, and that is what makes the eval worth running.
	78	+
	79	+Worked example. `bg-jobs` was missing on English requests and hitting on French ones, and the obvious reading was "the catalogue's French coverage is thin" — except `bg-jobs` has no French keyword at all. Three cases settled it:
	80	+
	81	+\| Case \| Result \|
	82	+\|---\|---\|
	83	+\| `start the dev server in the background` — English, names the concept \| ✓ hit \|
	84	+\| `démarre le serveur de dev et vérifie qu'il répond` — French, only a situation \| · miss \|
	85	+\| `lance le serveur en tâche de fond` — French, names the concept \| ✓ hit \|
	86	+\| `start the dev server and check it answers` — English, only a situation \| · miss \|
	87	+
	88	+A clean 2×2: the language is irrelevant. What decides it is whether the request names the concept the skill is about. That reframed the problem from a catalogue one into a prompt one — the model has no reason to know a server blocks the turn — and no amount of keyword editing would have found it.
	89	+
	90	+Build cases in pairs that differ by one thing. One case is an anecdote.

++# Measure whether the model picks the right skill
++
++> Run the routing eval, read its table, and use it to decide whether a change to the catalogue or to the prompt helped.
++
++`mm` shows a skill in the catalogue and hopes the model reaches for it. Whether it does is a property of the **model**, not of the code, so no ordinary test can answer it. This one asks a live model, twenty times.
++
++## Run it
++
++```bash
++MM_EVAL=1 AGENT_CONFIG=tests/agent.dmr-qwen.yaml go test ./internal/eval -v -timeout 25m
++```
++
++Both variables are required in practice: without `MM_EVAL` the test skips, and without `AGENT_CONFIG` it evaluates the built-in defaults rather than the configuration you care about. `go test ./...` skips it, so nothing in CI depends on a server being up.
++
++Expect a few minutes: one model turn per case, and small models are not fast. Rows are printed as they complete.
++
++## Read the table
++
++```
++[ 6/20] ✓ hit    want=what-time-is-it  got=what-time-is-it  1.204s  "quelle heure est-il ?"  ← French
++[14/20] ≠ wrong  want=greetings        got=vulcan-salute    0.981s  "Salue Bob de ma part"   ← two skills claim salut/saluer
++[16/20] · miss   want=bg-jobs          got=(none)           2.017s  "lance le serveur en tâche de fond"
++[17/20] ! extra  want=(none)           got=greetings        0.774s  "write a haiku about pointers"
++```
++
++| Mark | Meaning | What it points at |
++|---|---|---|
++| `✓ hit` | the expected skill, or correctly no skill | — |
++| `≠ wrong` | a skill was loaded, but not that one | the **catalogue**: two entries the model cannot tell apart |
++| `· miss` | a skill applied and none was loaded | the **prompt**: the model did not think to look |
++| `! extra` | no skill applied and one was loaded | over-eager instruction, or a catalogue entry that claims too much |
++| `✗ error` | the turn failed | **never a routing score.** Either the instrument (a stub that starved the model, `caseTimeout` too short, a server that went away) or the model being unable to compose a valid tool call at all (`unexpected end of JSON input`, or the runaway guard cutting off repeated arguments). The message says which. Fails the test, because the numbers are then incomplete. |
++
++`wrong` and `miss` are kept apart on purpose. Merging them hides which of the two problems you have, and they have different fixes.
++
++## Use it
++
++**The number on its own means nothing.** There is no threshold and the test does not fail on a low score — a number that fails a build becomes a target, and the first thing anyone does with a red build is edit the cases.
++
++The eval is a **before and after**:
++
++1. Run it, keep the output.
++2. Change one thing — a skill's `keywords`, the `SKILLS` block of the system prompt, a `description`.
++3. Run it again against the same model, with the same configuration.
++4. Compare.
++
++One change at a time. Two at once and you have measured their sum.
++
++Small models at `temperature: 0.0` are reproducible enough for this to mean something; run the eval twice before trusting a difference of one case.
++
++## What it measures, and what it does not
++
++It measures the **catalogue** — the tool description the model reads — under one system prompt. One model turn per case, with tools that record what was asked for and **run nothing**: a skill that says "run this echo" must not echo twenty times, and letting the agent loop would measure recovery as much as routing.
++
++It does not measure whether the skill, once loaded, was followed correctly. That is a different question and needs a different instrument.
++
++It also does not measure whether the model can emit a valid tool call. When it cannot — and small models often cannot — you get `✗ error`, not a low routing score. Those two failures have nothing to do with each other and no amount of catalogue work fixes the second one. Measured on `qwen3.5-4b`: three of twenty cases failed this way, reproducibly, on the prompts that invite a more structured call.
++
++## Add a case
++
++`internal/eval/routing_test.go`, the `cases` table:
++
++```go
++{"lance le serveur en tâche de fond", "bg-jobs", "French"},
++{"what is 17 times 23?", none, ""},
++```
++
++Two rules that keep the numbers honest:
++
++- **Do not phrase the prompt in the words of the catalogue.** A case that says "use the greetings skill" measures nothing.
++- **Keep the `none` cases.** A catalogue that fires on everything is as broken as one that never fires, and abstaining correctly is a hit.
++
++When you find a real routing failure, add it as a case before fixing it. That is how the eval stops the same failure coming back.
++
++## Write cases that can separate explanations
++
++A single case tells you something failed. A **set** of cases tells you why, and that is what makes the eval worth running.
++
++Worked example. `bg-jobs` was missing on English requests and hitting on French ones, and the obvious reading was "the catalogue's French coverage is thin" — except `bg-jobs` has no French keyword at all. Three cases settled it:
++
++| Case | Result |
++|---|---|
++| `start the dev server in the background` — English, **names** the concept | ✓ hit |
++| `démarre le serveur de dev et vérifie qu'il répond` — French, only a situation | · miss |
++| `lance le serveur en tâche de fond` — French, **names** the concept | ✓ hit |
++| `start the dev server and check it answers` — English, only a situation | · miss |
++
++A clean 2×2: **the language is irrelevant.** What decides it is whether the request names the concept the skill is about. That reframed the problem from a catalogue one into a prompt one — the model has no reason to know a server blocks the turn — and no amount of keyword editing would have found it.
++
++Build cases in pairs that differ by **one** thing. One case is an anecdote.

modified docs/en/reference/skill-format.md +32 -7 Viewed

@@ -8,18 +8,20 @@
8	8	\|------\|-------\|
9	9	\| Directory \| `skillsDir` (default `skills`). \|
10	10	\| Patterns \| `<skillsDir>/<name>.md` (flat) and `<skillsDir>/<name>/SKILL.md` (one directory per skill), mixed freely. Other files in a skill's directory are ignored. A `README.md` at the root of the directory is documentation, not a skill: it is skipped. \|
11		-\| Skill name on disk \| The file name without `.md`, or the directory name for a `SKILL.md`. \|
12		-\| Both forms for one name \| `read_skill` returns the flat file. \|
	11	+\| Skill name on disk \| The file name without `.md`, or the directory name for a `SKILL.md` — used only as the default `name` when the front matter gives none. \|
	12	+\| Resolution \| `read_skill` resolves the name through the catalogue, not through a path: it accepts exactly the names the catalogue shows, so a `name:` that differs from the file name works. Surrounding spaces and quotes are trimmed and the case is forgiven; an exact name always wins over a case-insensitive one. \|
	13	+\| Both forms for one name \| `read_skill` returns the flat file. The banner warns that the other is unreachable. \|
13	14	\| Missing directory \| Not an error: zero skills, `read_skill` not declared. \|
14	15
15	16	## Front matter
16	17
17		-Optional. When present, the file starts with a line `---` and the header ends at the next `---` line. Only two keys are read; the others are ignored.
	18	+Optional. When present, the file starts with a line `---` and the header ends at the next `---` line. Only three keys are read; the others are ignored.
18	19
19	20	\| Key \| Type \| Default \| Description \|
20	21	\|-----\|------\|---------\|-------------\|
21	22	\| `name` \| string \| file name without `.md` \| Name shown in the catalogue and accepted by `read_skill`. An empty value keeps the default. \|
22		-\| `description` \| string \| `""` \| One-line summary shown in the catalogue. \|
	23	+\| `description` \| string \| `""` \| One-line summary shown in the catalogue. A skill without one appears as a bare name, which the model cannot match a request against; the start-up banner warns about it. \|
	24	+\| `keywords` \| comma-separated list \| empty \| Words a user might actually type, in any language. Shown in the catalogue after the description, capped at the first 8 — the catalogue is sent with every request. This is the lexical bridge: "salue Bob" reaches `greetings` through `saluer`, never through the name. Order matters: past the eighth word nothing reaches the model, so put the most distinctive first. The banner warns when a list is being truncated. Do not spend the eight on words already in the `description` — the catalogue shows both, and a description is one language, so the keywords are where the others belong. \|
23	25
24	26	A `description:` line in the body, after the closing `---`, is not metadata.
25	27
@@ -29,11 +31,11 @@ The tool description is:
29	31
30	32	```
31	33	Load a skill: the step-by-step procedure to follow for this kind of task. Call it BEFORE doing the work, with the name of the matching skill, and then follow what it says. Available skills:
32		- <name> — <description>
	34	+ <name> — <description> (<keyword>, <keyword>, …)
33	35	<name>
34	36	```
35	37
36		-Entries are sorted by name; the ` — <description>` part is omitted when the description is empty.
	38	+Entries are sorted by name; the ` — <description>` part is omitted when the description is empty, and the ` (…)` part when there are no keywords.
37	39
38	40	### Example
39	41
@@ -53,7 +55,30 @@ date "+%A %d %B %Y — %H:%M:%S (%Z)"
53	55	```
54	56	```
55	57
56		-The `keywords` key is accepted in the file and ignored by the loader.
	58	+The `keywords` of that example reach the catalogue as `what-time-is-it — Give the current date and time. (time, date, clock, now, today, heure, date du jour, quelle heure)`.
	59	+
	60	+## Writing a skill the model will ACT on
	61	+
	62	+A skill that tells the model to run a command has one failure mode, and it is not the one you expect: the model reproduces the command instead of running it, in a fenced code block, and answers as if it had done the work.
	63	+
	64	+Measured on `mellum2-12b-a2.5b-instruct`, the same skill either way:
	65	+
	66	+\| The skill says \| Result \|
	67	+\|---\|---\|
	68	+\| ` ```bash ` fenced block \| `⚙ 0 command(s)` — the model printed the command back, three runs out of three \|
	69	+\| the command indented, and "call your `bash` tool with this command" \| `⚙ 1 command(s)` — three runs out of three \|
	70	+
	71	+A fenced block is a display affordance. Put one in the model's context and the most probable continuation is the same block, which is exactly what a small model produces. Naming the tool and indenting the command removes the invitation.
	72	+
	73	+Note this is model-dependent — `qwen3.5-4b` ran the fenced version correctly — so a skill that works on your model may still fail on someone else's. Write it the safe way regardless.
	74	+
	75	+The rule, for a skill whose point is that a command runs:
	76	+
	77	+- no ` ```bash ` fence around the command to execute. Indent it by four spaces instead.
	78	+- name the tool: "call your `bash` tool with this command", not "run:".
	79	+- keep saying what the shipped skills say — the answer is the command's output, nothing else — because that is what stops the model answering from memory.
	80	+
	81	+A skill that is reference material rather than one command to run — `bg-jobs`, which shows half a dozen — keeps its fences. Nothing there is "the command to execute", so there is no invitation to misread.
57	82
58	83	## Shipped examples
59	84

  |------|-------|
  | Directory | `skillsDir` (default `skills`). |
  | Patterns | `<skillsDir>/<name>.md` (flat) and `<skillsDir>/<name>/SKILL.md` (one directory per skill), mixed freely. Other files in a skill's directory are ignored. A `README.md` at the root of the directory is documentation, not a skill: it is skipped. |
--| Skill name on disk | The file name without `.md`, or the directory name for a `SKILL.md`. |
++| Skill name on disk | The file name without `.md`, or the directory name for a `SKILL.md` — used only as the default `name` when the front matter gives none. |
--| Both forms for one name | `read_skill` returns the flat file. |
++| Resolution | `read_skill` resolves the name **through the catalogue**, not through a path: it accepts exactly the names the catalogue shows, so a `name:` that differs from the file name works. Surrounding spaces and quotes are trimmed and the case is forgiven; an exact name always wins over a case-insensitive one. |
++| Both forms for one name | `read_skill` returns the flat file. The banner warns that the other is unreachable. |
  | Missing directory | Not an error: zero skills, `read_skill` not declared. |
  ## Front matter
--Optional. When present, the file starts with a line `---` and the header ends at the next `---` line. Only two keys are read; the others are ignored.
++Optional. When present, the file starts with a line `---` and the header ends at the next `---` line. Only three keys are read; the others are ignored.
  | Key | Type | Default | Description |
  |-----|------|---------|-------------|
  | `name` | string | file name without `.md` | Name shown in the catalogue and accepted by `read_skill`. An empty value keeps the default. |
--| `description` | string | `""` | One-line summary shown in the catalogue. |
++| `description` | string | `""` | One-line summary shown in the catalogue. A skill without one appears as a bare name, which the model cannot match a request against; the start-up banner warns about it. |
++| `keywords` | comma-separated list | empty | Words a user might actually type, in any language. Shown in the catalogue after the description, **capped at the first 8** — the catalogue is sent with every request. This is the lexical bridge: *"salue Bob"* reaches `greetings` through `saluer`, never through the name. **Order matters**: past the eighth word nothing reaches the model, so put the most distinctive first. The banner warns when a list is being truncated. Do not spend the eight on words already in the `description` — the catalogue shows both, and a description is one language, so the keywords are where the others belong. |
  A `description:` line in the body, after the closing `---`, is not metadata.
  ```
  Load a skill: the step-by-step procedure to follow for this kind of task. Call it BEFORE doing the work, with the name of the matching skill, and then follow what it says. Available skills:
--  <name> — <description>
++  <name> — <description> (<keyword>, <keyword>, …)
    <name>
  ```
--Entries are sorted by name; the ` — <description>` part is omitted when the description is empty.
++Entries are sorted by name; the ` — <description>` part is omitted when the description is empty, and the ` (…)` part when there are no keywords.
  ### Example
  ```
  ```
--The `keywords` key is accepted in the file and ignored by the loader.
++The `keywords` of that example reach the catalogue as `what-time-is-it — Give the current date and time. (time, date, clock, now, today, heure, date du jour, quelle heure)`.
++
++## Writing a skill the model will ACT on
++
++A skill that tells the model to run a command has one failure mode, and it is not the one you expect: the model **reproduces the command instead of running it**, in a fenced code block, and answers as if it had done the work.
++
++Measured on `mellum2-12b-a2.5b-instruct`, the same skill either way:
++
++| The skill says | Result |
++|---|---|
++| ` ```bash ` fenced block | `⚙ 0 command(s)` — the model printed the command back, three runs out of three |
++| the command **indented**, and "call your `bash` tool with this command" | `⚙ 1 command(s)` — three runs out of three |
++
++A fenced block is a *display* affordance. Put one in the model's context and the most probable continuation is the same block, which is exactly what a small model produces. Naming the tool and indenting the command removes the invitation.
++
++Note this is model-dependent — `qwen3.5-4b` ran the fenced version correctly — so a skill that works on your model may still fail on someone else's. Write it the safe way regardless.
++
++The rule, for a skill whose point is that a command runs:
++
++- **no ` ```bash ` fence around the command to execute.** Indent it by four spaces instead.
++- **name the tool**: "call your `bash` tool with this command", not "run:".
++- keep saying what the shipped skills say — *the answer is the command's output, nothing else* — because that is what stops the model answering from memory.
++
++A skill that is reference material rather than one command to run — `bg-jobs`, which shows half a dozen — keeps its fences. Nothing there is "the command to execute", so there is no invitation to misread.
  ## Shipped examples

modified docs/fr/README.md +1 -0 Viewed

@@ -13,6 +13,7 @@ Cette documentation est organisée selon les quatre quadrants [Diátaxis](https:
13	13	- [Faire tourner mini-me sur llama.cpp plutôt que Docker Model Runner](how-to/run-with-llama-cpp.md)
14	14	- [Utiliser mini-me depuis un éditeur (ACP)](how-to/use-from-an-editor.md)
15	15	- [Ajouter un skill](how-to/add-a-skill.md)
	16	+- [Mesurer si le modèle choisit le bon skill](how-to/measure-skill-routing.md) — l'eval de routage : le lancer, lire son tableau, s'en servir en avant/après
16	17	- [Garder une longue session dans la fenêtre de contexte](how-to/manage-the-context-window.md)
17	18	- [Démarrer une nouvelle session sans relancer](how-to/start-a-new-session.md)
18	19	- [Désigner un fichier au modèle avec `@chemin`](how-to/attach-a-file.md)

  - [Faire tourner mini-me sur llama.cpp plutôt que Docker Model Runner](how-to/run-with-llama-cpp.md)
  - [Utiliser mini-me depuis un éditeur (ACP)](how-to/use-from-an-editor.md)
  - [Ajouter un skill](how-to/add-a-skill.md)
++- [Mesurer si le modèle choisit le bon skill](how-to/measure-skill-routing.md) — l'eval de routage : le lancer, lire son tableau, s'en servir en avant/après
  - [Garder une longue session dans la fenêtre de contexte](how-to/manage-the-context-window.md)
  - [Démarrer une nouvelle session sans relancer](how-to/start-a-new-session.md)
  - [Désigner un fichier au modèle avec `@chemin`](how-to/attach-a-file.md)

added docs/fr/how-to/measure-skill-routing.md +90 -0 Viewed

new file mode 100644
@@ -0,0 +1,90 @@
	1	+# Mesurer si le modèle choisit le bon skill
	2	+
	3	+> Lancer l'eval de routage, lire son tableau, et s'en servir pour décider si un changement du catalogue ou du prompt a aidé.
	4	+
	5	+`mm` affiche un skill dans le catalogue et espère que le modèle ira le chercher. Qu'il le fasse ou non est une propriété du modèle, pas du code : aucun test ordinaire ne peut y répondre. Celui-ci interroge un modèle vivant, vingt fois.
	6	+
	7	+## Le lancer
	8	+
	9	+```bash
	10	+MM_EVAL=1 AGENT_CONFIG=tests/agent.dmr-qwen.yaml go test ./internal/eval -v -timeout 25m
	11	+```
	12	+
	13	+Les deux variables sont nécessaires en pratique : sans `MM_EVAL` le test se saute, et sans `AGENT_CONFIG` il évalue les défauts intégrés plutôt que la configuration qui vous intéresse. `go test ./...` le saute, donc rien dans la CI ne dépend d'un serveur démarré.
	14	+
	15	+Comptez quelques minutes : un tour de modèle par cas, et les petits modèles ne sont pas rapides. Les lignes s'affichent au fur et à mesure.
	16	+
	17	+## Lire le tableau
	18	+
	19	+```
	20	+[ 6/20] ✓ hit want=what-time-is-it got=what-time-is-it 1.204s "quelle heure est-il ?" ← French
	21	+[14/20] ≠ wrong want=greetings got=vulcan-salute 0.981s "Salue Bob de ma part" ← two skills claim salut/saluer
	22	+[16/20] · miss want=bg-jobs got=(none) 2.017s "lance le serveur en tâche de fond"
	23	+[17/20] ! extra want=(none) got=greetings 0.774s "write a haiku about pointers"
	24	+```
	25	+
	26	+\| Marque \| Sens \| Ce que ça désigne \|
	27	+\|---\|---\|---\|
	28	+\| `✓ hit` \| le skill attendu, ou correctement aucun \| — \|
	29	+\| `≠ wrong` \| un skill a été chargé, mais pas celui-là \| le catalogue : deux entrées que le modèle ne distingue pas \|
	30	+\| `· miss` \| un skill s'appliquait et aucun n'a été chargé \| le prompt : le modèle n'a pas pensé à chercher \|
	31	+\| `! extra` \| aucun skill ne s'appliquait et un a été chargé \| instruction trop insistante, ou entrée qui promet trop \|
	32	+\| `✗ error` \| le tour a échoué \| jamais un score de routage. Soit l'instrument (un stub qui a affamé le modèle, `caseTimeout` trop court, un serveur parti), soit le modèle incapable de composer un appel d'outil valide (`unexpected end of JSON input`, ou le garde-fou anti-emballement qui coupe des arguments répétés). Le message dit lequel. Fait échouer le test, car les chiffres sont alors incomplets. \|
	33	+
	34	+`wrong` et `miss` sont séparés à dessein. Les confondre masque lequel des deux problèmes vous avez, et ils n'ont pas le même remède.
	35	+
	36	+## S'en servir
	37	+
	38	+Le chiffre seul ne veut rien dire. Il n'y a pas de seuil et le test n'échoue pas sur un mauvais score — un chiffre qui casse le build devient une cible, et la première chose qu'on fait devant un build rouge, c'est modifier les cas.
	39	+
	40	+L'eval est un avant/après :
	41	+
	42	+1. Le lancer, garder la sortie.
	43	+2. Changer une chose — les `keywords` d'un skill, le bloc `SKILLS` du prompt système, une `description`.
	44	+3. Le relancer sur le même modèle, avec la même configuration.
	45	+4. Comparer.
	46	+
	47	+Un changement à la fois. Deux d'un coup et vous avez mesuré leur somme.
	48	+
	49	+Les petits modèles à `temperature: 0.0` sont assez reproductibles pour que ça ait un sens ; relancez deux fois avant de croire à un écart d'un seul cas.
	50	+
	51	+## Ce qu'il mesure, et ce qu'il ne mesure pas
	52	+
	53	+Il mesure le catalogue — la description d'outil que le modèle lit — sous un prompt système donné. Un tour de modèle par cas, avec des outils qui enregistrent ce qui a été demandé et n'exécutent rien : un skill qui dit « lance cet echo » ne doit pas echo vingt fois, et laisser tourner la boucle de l'agent mesurerait la récupération autant que le routage.
	54	+
	55	+Il ne mesure pas si le skill, une fois chargé, a été correctement suivi. C'est une autre question, et il faut un autre instrument.
	56	+
	57	+Il ne mesure pas non plus si le modèle sait émettre un appel d'outil valide. Quand il ne sait pas — et les petits modèles souvent ne savent pas — vous obtenez `✗ error`, pas un mauvais score de routage. Ces deux échecs n'ont rien à voir, et aucun travail sur le catalogue ne corrige le second. Mesuré sur `qwen3.5-4b` : trois cas sur vingt échouent ainsi, de façon reproductible, sur les demandes qui appellent un appel plus structuré.
	58	+
	59	+## Ajouter un cas
	60	+
	61	+`internal/eval/routing_test.go`, la table `cases` :
	62	+
	63	+```go
	64	+{"lance le serveur en tâche de fond", "bg-jobs", "French"},
	65	+{"what is 17 times 23?", none, ""},
	66	+```
	67	+
	68	+Deux règles qui gardent les chiffres honnêtes :
	69	+
	70	+- Ne formulez pas la demande avec les mots du catalogue. Un cas qui dit « utilise le skill greetings » ne mesure rien.
	71	+- Gardez les cas `none`. Un catalogue qui se déclenche sur tout est aussi cassé qu'un catalogue qui ne se déclenche jamais, et s'abstenir à raison est un succès.
	72	+
	73	+Quand vous trouvez un vrai échec de routage, ajoutez-le comme cas avant de le corriger. C'est ainsi que l'eval empêche le même échec de revenir.
	74	+
	75	+## Écrivez des cas capables de séparer les explications
	76	+
	77	+Un cas isolé dit que quelque chose a échoué. Un ensemble de cas dit pourquoi, et c'est ce qui rend l'eval utile.
	78	+
	79	+Exemple vécu. `bg-jobs` échouait sur les demandes anglaises et réussissait sur les françaises, et la lecture évidente était « la couverture française du catalogue est faible » — sauf que `bg-jobs` n'a aucun mot-clé français. Trois cas ont tranché :
	80	+
	81	+\| Cas \| Résultat \|
	82	+\|---\|---\|
	83	+\| `start the dev server in the background` — anglais, nomme le concept \| ✓ hit \|
	84	+\| `démarre le serveur de dev et vérifie qu'il répond` — français, juste une situation \| · miss \|
	85	+\| `lance le serveur en tâche de fond` — français, nomme le concept \| ✓ hit \|
	86	+\| `start the dev server and check it answers` — anglais, juste une situation \| · miss \|
	87	+
	88	+Un 2×2 net : la langue n'a rien à voir. Ce qui décide, c'est que la demande nomme le concept dont parle le skill. Ça a déplacé le problème du catalogue vers le prompt — le modèle n'a aucune raison de savoir qu'un serveur bloque le tour — et aucune retouche de mots-clés ne l'aurait trouvé.
	89	+
	90	+Construisez les cas par paires qui ne diffèrent que d'une chose. Un cas seul est une anecdote.

++# Mesurer si le modèle choisit le bon skill
++
++> Lancer l'eval de routage, lire son tableau, et s'en servir pour décider si un changement du catalogue ou du prompt a aidé.
++
++`mm` affiche un skill dans le catalogue et espère que le modèle ira le chercher. Qu'il le fasse ou non est une propriété du **modèle**, pas du code : aucun test ordinaire ne peut y répondre. Celui-ci interroge un modèle vivant, vingt fois.
++
++## Le lancer
++
++```bash
++MM_EVAL=1 AGENT_CONFIG=tests/agent.dmr-qwen.yaml go test ./internal/eval -v -timeout 25m
++```
++
++Les deux variables sont nécessaires en pratique : sans `MM_EVAL` le test se saute, et sans `AGENT_CONFIG` il évalue les défauts intégrés plutôt que la configuration qui vous intéresse. `go test ./...` le saute, donc rien dans la CI ne dépend d'un serveur démarré.
++
++Comptez quelques minutes : un tour de modèle par cas, et les petits modèles ne sont pas rapides. Les lignes s'affichent au fur et à mesure.
++
++## Lire le tableau
++
++```
++[ 6/20] ✓ hit    want=what-time-is-it  got=what-time-is-it  1.204s  "quelle heure est-il ?"  ← French
++[14/20] ≠ wrong  want=greetings        got=vulcan-salute    0.981s  "Salue Bob de ma part"   ← two skills claim salut/saluer
++[16/20] · miss   want=bg-jobs          got=(none)           2.017s  "lance le serveur en tâche de fond"
++[17/20] ! extra  want=(none)           got=greetings        0.774s  "write a haiku about pointers"
++```
++
++| Marque | Sens | Ce que ça désigne |
++|---|---|---|
++| `✓ hit` | le skill attendu, ou correctement aucun | — |
++| `≠ wrong` | un skill a été chargé, mais pas celui-là | le **catalogue** : deux entrées que le modèle ne distingue pas |
++| `· miss` | un skill s'appliquait et aucun n'a été chargé | le **prompt** : le modèle n'a pas pensé à chercher |
++| `! extra` | aucun skill ne s'appliquait et un a été chargé | instruction trop insistante, ou entrée qui promet trop |
++| `✗ error` | le tour a échoué | **jamais un score de routage.** Soit l'instrument (un stub qui a affamé le modèle, `caseTimeout` trop court, un serveur parti), soit le modèle incapable de composer un appel d'outil valide (`unexpected end of JSON input`, ou le garde-fou anti-emballement qui coupe des arguments répétés). Le message dit lequel. Fait échouer le test, car les chiffres sont alors incomplets. |
++
++`wrong` et `miss` sont séparés à dessein. Les confondre masque lequel des deux problèmes vous avez, et ils n'ont pas le même remède.
++
++## S'en servir
++
++**Le chiffre seul ne veut rien dire.** Il n'y a pas de seuil et le test n'échoue pas sur un mauvais score — un chiffre qui casse le build devient une cible, et la première chose qu'on fait devant un build rouge, c'est modifier les cas.
++
++L'eval est un **avant/après** :
++
++1. Le lancer, garder la sortie.
++2. Changer **une** chose — les `keywords` d'un skill, le bloc `SKILLS` du prompt système, une `description`.
++3. Le relancer sur le même modèle, avec la même configuration.
++4. Comparer.
++
++Un changement à la fois. Deux d'un coup et vous avez mesuré leur somme.
++
++Les petits modèles à `temperature: 0.0` sont assez reproductibles pour que ça ait un sens ; relancez deux fois avant de croire à un écart d'un seul cas.
++
++## Ce qu'il mesure, et ce qu'il ne mesure pas
++
++Il mesure le **catalogue** — la description d'outil que le modèle lit — sous un prompt système donné. Un tour de modèle par cas, avec des outils qui enregistrent ce qui a été demandé et **n'exécutent rien** : un skill qui dit « lance cet echo » ne doit pas echo vingt fois, et laisser tourner la boucle de l'agent mesurerait la récupération autant que le routage.
++
++Il ne mesure pas si le skill, une fois chargé, a été correctement suivi. C'est une autre question, et il faut un autre instrument.
++
++Il ne mesure pas non plus si le modèle sait émettre un appel d'outil valide. Quand il ne sait pas — et les petits modèles souvent ne savent pas — vous obtenez `✗ error`, pas un mauvais score de routage. Ces deux échecs n'ont rien à voir, et aucun travail sur le catalogue ne corrige le second. Mesuré sur `qwen3.5-4b` : trois cas sur vingt échouent ainsi, de façon reproductible, sur les demandes qui appellent un appel plus structuré.
++
++## Ajouter un cas
++
++`internal/eval/routing_test.go`, la table `cases` :
++
++```go
++{"lance le serveur en tâche de fond", "bg-jobs", "French"},
++{"what is 17 times 23?", none, ""},
++```
++
++Deux règles qui gardent les chiffres honnêtes :
++
++- **Ne formulez pas la demande avec les mots du catalogue.** Un cas qui dit « utilise le skill greetings » ne mesure rien.
++- **Gardez les cas `none`.** Un catalogue qui se déclenche sur tout est aussi cassé qu'un catalogue qui ne se déclenche jamais, et s'abstenir à raison est un succès.
++
++Quand vous trouvez un vrai échec de routage, ajoutez-le comme cas **avant** de le corriger. C'est ainsi que l'eval empêche le même échec de revenir.
++
++## Écrivez des cas capables de séparer les explications
++
++Un cas isolé dit que quelque chose a échoué. Un **ensemble** de cas dit pourquoi, et c'est ce qui rend l'eval utile.
++
++Exemple vécu. `bg-jobs` échouait sur les demandes anglaises et réussissait sur les françaises, et la lecture évidente était « la couverture française du catalogue est faible » — sauf que `bg-jobs` n'a aucun mot-clé français. Trois cas ont tranché :
++
++| Cas | Résultat |
++|---|---|
++| `start the dev server in the background` — anglais, **nomme** le concept | ✓ hit |
++| `démarre le serveur de dev et vérifie qu'il répond` — français, juste une situation | · miss |
++| `lance le serveur en tâche de fond` — français, **nomme** le concept | ✓ hit |
++| `start the dev server and check it answers` — anglais, juste une situation | · miss |
++
++Un 2×2 net : **la langue n'a rien à voir.** Ce qui décide, c'est que la demande nomme le concept dont parle le skill. Ça a déplacé le problème du catalogue vers le prompt — le modèle n'a aucune raison de savoir qu'un serveur bloque le tour — et aucune retouche de mots-clés ne l'aurait trouvé.
++
++Construisez les cas par paires qui ne diffèrent que d'**une** chose. Un cas seul est une anecdote.

modified docs/fr/reference/skill-format.md +29 -5 Viewed

@@ -14,12 +14,13 @@
14	14
15	15	## En-tête (front matter)
16	16
17		-Optionnel. Quand il est présent, le fichier commence par une ligne `---` et l'en-tête se termine à la ligne `---` suivante. Seules deux clés sont lues ; les autres sont ignorées.
	17	+Optionnel. Quand il est présent, le fichier commence par une ligne `---` et l'en-tête se termine à la ligne `---` suivante. Seules trois clés sont lues ; les autres sont ignorées.
18	18
19	19	\| Clé \| Type \| Défaut \| Description \|
20	20	\|-----\|------\|--------\|-------------\|
21	21	\| `name` \| chaîne \| nom du fichier sans `.md` \| Nom affiché dans le catalogue et accepté par `read_skill`. Une valeur vide garde le défaut. \|
22		-\| `description` \| chaîne \| `""` \| Résumé d'une ligne affiché dans le catalogue. \|
	22	+\| `description` \| chaîne \| `""` \| Résumé d'une ligne affiché dans le catalogue. Un skill qui n'en a pas apparaît comme un nom nu, auquel le modèle ne peut rien rattacher ; la bannière de démarrage le signale. \|
	23	+\| `keywords` \| liste séparée par des virgules \| vide \| Les mots qu'un utilisateur taperait vraiment, dans n'importe quelle langue. Affichés dans le catalogue après la description, plafonnés aux 8 premiers — le catalogue part à chaque requête. C'est le pont lexical : « salue Bob » atteint `greetings` par `saluer`, jamais par le nom. L'ordre compte : au-delà du huitième mot, rien n'atteint le modèle — mettez les plus distinctifs en premier. La bannière avertit quand une liste est tronquée. Ne dépensez pas les huit sur des mots déjà présents dans la `description` : le catalogue affiche les deux, et une description est dans une seule langue — les mots-clés sont là pour les autres. \|
23	24
24	25	Une ligne `description:` dans le corps, après le `---` de fermeture, n'est pas une métadonnée.
25	26
@@ -29,11 +30,11 @@ La description de l'outil est :
29	30
30	31	```
31	32	Load a skill: the step-by-step procedure to follow for this kind of task. Call it BEFORE doing the work, with the name of the matching skill, and then follow what it says. Available skills:
32		- <name> — <description>
	33	+ <name> — <description> (<keyword>, <keyword>, …)
33	34	<name>
34	35	```
35	36
36		-Les entrées sont triées par nom ; la partie ` — <description>` est omise quand la description est vide.
	37	+Les entrées sont triées par nom ; la partie ` — <description>` est omise quand la description est vide, et la partie ` (…)` quand il n'y a pas de mots-clés.
37	38
38	39	### Exemple
39	40
@@ -53,7 +54,30 @@ date "+%A %d %B %Y — %H:%M:%S (%Z)"
53	54	```
54	55	```
55	56
56		-La clé `keywords` est acceptée dans le fichier et ignorée par le chargeur.
	57	+Les `keywords` de cet exemple arrivent dans le catalogue sous la forme `what-time-is-it — Give the current date and time. (time, date, clock, now, today, heure, date du jour, quelle heure)`.
	58	+
	59	+## Écrire un skill que le modèle va EXÉCUTER
	60	+
	61	+Un skill qui demande au modèle de lancer une commande a un mode d'échec, et ce n'est pas celui qu'on attend : le modèle recopie la commande au lieu de l'exécuter, dans un bloc de code clôturé, et répond comme s'il avait fait le travail.
	62	+
	63	+Mesuré sur `mellum2-12b-a2.5b-instruct`, le même skill dans les deux formes :
	64	+
	65	+\| Ce que dit le skill \| Résultat \|
	66	+\|---\|---\|
	67	+\| bloc clôturé ` ```bash ` \| `⚙ 0 command(s)` — le modèle réaffiche la commande, trois passes sur trois \|
	68	+\| la commande indentée, et « call your `bash` tool with this command » \| `⚙ 1 command(s)` — trois passes sur trois \|
	69	+
	70	+Un bloc clôturé est une affordance d'affichage. Mettez-en un dans le contexte du modèle et la continuation la plus probable est le même bloc — c'est exactement ce qu'un petit modèle produit. Nommer l'outil et indenter la commande retire l'invitation.
	71	+
	72	+C'est dépendant du modèle — `qwen3.5-4b` exécutait correctement la version clôturée — donc un skill qui marche sur votre modèle peut échouer sur celui d'un autre. Écrivez-le de la façon sûre quand même.
	73	+
	74	+La règle, pour un skill dont tout l'intérêt est qu'une commande tourne :
	75	+
	76	+- pas de clôture ` ```bash ` autour de la commande à exécuter. Indentez-la de quatre espaces.
	77	+- nommez l'outil : « call your `bash` tool with this command », pas « run: ».
	78	+- gardez ce que disent les skills livrés — the answer is the command's output, nothing else — car c'est ce qui empêche le modèle de répondre de mémoire.
	79	+
	80	+Un skill qui est de la documentation plutôt qu'une commande à lancer — `bg-jobs`, qui en montre une demi-douzaine — garde ses clôtures. Rien là-dedans n'est « la commande à exécuter », donc il n'y a pas d'invitation à mal lire.
57	81
58	82	## Exemples livrés
59	83

  ## En-tête (front matter)
--Optionnel. Quand il est présent, le fichier commence par une ligne `---` et l'en-tête se termine à la ligne `---` suivante. Seules deux clés sont lues ; les autres sont ignorées.
++Optionnel. Quand il est présent, le fichier commence par une ligne `---` et l'en-tête se termine à la ligne `---` suivante. Seules trois clés sont lues ; les autres sont ignorées.
  | Clé | Type | Défaut | Description |
  |-----|------|--------|-------------|
  | `name` | chaîne | nom du fichier sans `.md` | Nom affiché dans le catalogue et accepté par `read_skill`. Une valeur vide garde le défaut. |
--| `description` | chaîne | `""` | Résumé d'une ligne affiché dans le catalogue. |
++| `description` | chaîne | `""` | Résumé d'une ligne affiché dans le catalogue. Un skill qui n'en a pas apparaît comme un nom nu, auquel le modèle ne peut rien rattacher ; la bannière de démarrage le signale. |
++| `keywords` | liste séparée par des virgules | vide | Les mots qu'un utilisateur taperait vraiment, dans n'importe quelle langue. Affichés dans le catalogue après la description, **plafonnés aux 8 premiers** — le catalogue part à chaque requête. C'est le pont lexical : *« salue Bob »* atteint `greetings` par `saluer`, jamais par le nom. **L'ordre compte** : au-delà du huitième mot, rien n'atteint le modèle — mettez les plus distinctifs en premier. La bannière avertit quand une liste est tronquée. Ne dépensez pas les huit sur des mots déjà présents dans la `description` : le catalogue affiche les deux, et une description est dans une seule langue — les mots-clés sont là pour les autres. |
  Une ligne `description:` dans le corps, après le `---` de fermeture, n'est pas une métadonnée.
  ```
  Load a skill: the step-by-step procedure to follow for this kind of task. Call it BEFORE doing the work, with the name of the matching skill, and then follow what it says. Available skills:
--  <name> — <description>
++  <name> — <description> (<keyword>, <keyword>, …)
    <name>
  ```
--Les entrées sont triées par nom ; la partie ` — <description>` est omise quand la description est vide.
++Les entrées sont triées par nom ; la partie ` — <description>` est omise quand la description est vide, et la partie ` (…)` quand il n'y a pas de mots-clés.
  ### Exemple
  ```
  ```
--La clé `keywords` est acceptée dans le fichier et ignorée par le chargeur.
++Les `keywords` de cet exemple arrivent dans le catalogue sous la forme `what-time-is-it — Give the current date and time. (time, date, clock, now, today, heure, date du jour, quelle heure)`.
++
++## Écrire un skill que le modèle va EXÉCUTER
++
++Un skill qui demande au modèle de lancer une commande a un mode d'échec, et ce n'est pas celui qu'on attend : le modèle **recopie la commande au lieu de l'exécuter**, dans un bloc de code clôturé, et répond comme s'il avait fait le travail.
++
++Mesuré sur `mellum2-12b-a2.5b-instruct`, le même skill dans les deux formes :
++
++| Ce que dit le skill | Résultat |
++|---|---|
++| bloc clôturé ` ```bash ` | `⚙ 0 command(s)` — le modèle réaffiche la commande, trois passes sur trois |
++| la commande **indentée**, et « call your `bash` tool with this command » | `⚙ 1 command(s)` — trois passes sur trois |
++
++Un bloc clôturé est une affordance d'*affichage*. Mettez-en un dans le contexte du modèle et la continuation la plus probable est le même bloc — c'est exactement ce qu'un petit modèle produit. Nommer l'outil et indenter la commande retire l'invitation.
++
++C'est dépendant du modèle — `qwen3.5-4b` exécutait correctement la version clôturée — donc un skill qui marche sur votre modèle peut échouer sur celui d'un autre. Écrivez-le de la façon sûre quand même.
++
++La règle, pour un skill dont tout l'intérêt est qu'une commande tourne :
++
++- **pas de clôture ` ```bash ` autour de la commande à exécuter.** Indentez-la de quatre espaces.
++- **nommez l'outil** : « call your `bash` tool with this command », pas « run: ».
++- gardez ce que disent les skills livrés — *the answer is the command's output, nothing else* — car c'est ce qui empêche le modèle de répondre de mémoire.
++
++Un skill qui est de la documentation plutôt qu'une commande à lancer — `bg-jobs`, qui en montre une demi-douzaine — garde ses clôtures. Rien là-dedans n'est « la commande à exécuter », donc il n'y a pas d'invitation à mal lire.
  ## Exemples livrés

modified internal/agent/agent.go +4 -2 Viewed

@@ -251,8 +251,10 @@ func Run(ctx context.Context, e *engine.Engine, system string, tools []ai.ToolRe
251	251	if len(ops) > 0 {
252	252	line += fmt.Sprintf(" · 📝 %d file op(s)", len(ops))
253	253	}
254		- if k := engine.Skills(turn); k > 0 {
255		- line += fmt.Sprintf(" · 📖 %d skill(s)", k)
	254	+ // Always shown when the agent has skills, 0 included: see
	255	+ // engine.SkillsAvailable.
	256	+ if engine.SkillsAvailable() {
	257	+ line += fmt.Sprintf(" · 📖 %d skill(s)", engine.Skills(turn))
256	258	}
257	259	if k := engine.LSPCalls(turn); k > 0 {
258	260	line += fmt.Sprintf(" · 🔎 %d lsp call(s)", k)

  				if len(ops) > 0 {
  					line += fmt.Sprintf(" · 📝 %d file op(s)", len(ops))
  				}
--				if k := engine.Skills(turn); k > 0 {
++				// Always shown when the agent has skills, 0 included: see
--					line += fmt.Sprintf(" · 📖 %d skill(s)", k)
++				// engine.SkillsAvailable.
++				if engine.SkillsAvailable() {
++					line += fmt.Sprintf(" · 📖 %d skill(s)", engine.Skills(turn))
  				}
  				if k := engine.LSPCalls(turn); k > 0 {
  					line += fmt.Sprintf(" · 🔎 %d lsp call(s)", k)

modified internal/engine/engine.go +42 -1 Viewed

@@ -666,9 +666,50 @@ func fileOpText(r *ai.ToolRequest) string {
666	666	// next to the command count: seeing which procedure the agent chose to follow is
667	667	// half the point of this part.
668	668	func Skills(history []*ai.Message) int {
669		- return responses(history, toolSkill)
	669	+ n := 0
	670	+ for _, m := range history {
	671	+ for _, p := range m.Content {
	672	+ if !p.IsToolResponse() \|\| p.ToolResponse == nil \|\| p.ToolResponse.Name != toolSkill {
	673	+ continue
	674	+ }
	675	+ if !strings.HasPrefix(strings.TrimSpace(responseText(p.ToolResponse)), skillMiss) {
	676	+ n++
	677	+ }
	678	+ }
	679	+ }
	680	+ return n
670	681	}
671	682
	683	+// skillMiss is how `read_skill` answers a name that is not in the catalogue.
	684	+// Counting those as skills made the number say the opposite of what it is
	685	+// for: `📖 2 skill(s)` could mean two failed lookups and nothing loaded.
	686	+const skillMiss = "[skill:"
	687	+
	688	+// responseText is a tool response as text. Every tool here returns a string;
	689	+// anything else is formatted rather than dropped, so a change of shape shows
	690	+// up as an odd count instead of a silent zero.
	691	+func responseText(r *ai.ToolResponse) string {
	692	+ if s, ok := r.Output.(string); ok {
	693	+ return s
	694	+ }
	695	+ return fmt.Sprintf("%v", r.Output)
	696	+}
	697	+
	698	+// skillsDeclared says this agent has a skills catalogue at all. Set once by
	699	+// main, read by the three turn recaps.
	700	+//
	701	+// Why it exists: `📖 N skill(s)` used to be printed only when N > 0, so
	702	+// "no skill applied" and "a skill applied and the model ignored it" looked
	703	+// identical — nothing at all. Showing `📖 0 skill(s)` is the whole point;
	704	+// showing it in a project that has no skills would be noise.
	705	+var skillsDeclared atomic.Bool
	706	+
	707	+// DeclareSkills records whether the agent was given any skill.
	708	+func DeclareSkills(yes bool) { skillsDeclared.Store(yes) }
	709	+
	710	+// SkillsAvailable reports it, for the turn recaps.
	711	+func SkillsAvailable() bool { return skillsDeclared.Load() }
	712	+
672	713	// LSPCalls counts the `lsp` tool responses in a history — questions put to a
673	714	// language server (definition, references, hover, symbols, diagnostics,
674	715	// rename). Shown apart from the commands for the same reason as the file

  // next to the command count: seeing which procedure the agent chose to follow is
  // half the point of this part.
  func Skills(history []*ai.Message) int {
--	return responses(history, toolSkill)
++	n := 0
++	for _, m := range history {
++		for _, p := range m.Content {
++			if !p.IsToolResponse() || p.ToolResponse == nil || p.ToolResponse.Name != toolSkill {
++				continue
++			}
++			if !strings.HasPrefix(strings.TrimSpace(responseText(p.ToolResponse)), skillMiss) {
++				n++
++			}
++		}
++	}
++	return n
  }
++// skillMiss is how `read_skill` answers a name that is not in the catalogue.
++// Counting those as skills made the number say the opposite of what it is
++// for: `📖 2 skill(s)` could mean two failed lookups and nothing loaded.
++const skillMiss = "[skill:"
++
++// responseText is a tool response as text. Every tool here returns a string;
++// anything else is formatted rather than dropped, so a change of shape shows
++// up as an odd count instead of a silent zero.
++func responseText(r *ai.ToolResponse) string {
++	if s, ok := r.Output.(string); ok {
++		return s
++	}
++	return fmt.Sprintf("%v", r.Output)
++}
++
++// skillsDeclared says this agent has a skills catalogue at all. Set once by
++// main, read by the three turn recaps.
++//
++// Why it exists: `📖 N skill(s)` used to be printed only when N > 0, so
++// "no skill applied" and "a skill applied and the model ignored it" looked
++// identical — nothing at all. Showing `📖 0 skill(s)` is the whole point;
++// showing it in a project that has no skills would be noise.
++var skillsDeclared atomic.Bool
++
++// DeclareSkills records whether the agent was given any skill.
++func DeclareSkills(yes bool) { skillsDeclared.Store(yes) }
++
++// SkillsAvailable reports it, for the turn recaps.
++func SkillsAvailable() bool { return skillsDeclared.Load() }
++
  // LSPCalls counts the `lsp` tool responses in a history — questions put to a
  // language server (definition, references, hover, symbols, diagnostics,
  // rename). Shown apart from the commands for the same reason as the file

modified internal/engine/provider_test.go +11 -4 Viewed

@@ -51,12 +51,19 @@ func openaiServer(t testing.T, status int, body string, models ...string) http
51	51
52	52	// closedURL returns a URL nothing listens on: a server started and stopped, so
53	53	// the port was free a moment ago.
	54	+// closedURL is an address nothing answers on.
	55	+//
	56	+// It used to start an httptest server, record its URL and close it — which
	57	+// leaves an EPHEMERAL port that the kernel is free to hand to the next
	58	+// listener, so the test occasionally connected to something real and lost the
	59	+// "nothing answers" hint it was checking for. Measured: one failure in about
	60	+// thirty runs, and more when other servers were being started nearby.
	61	+//
	62	+// Port 1 cannot be listened on without privileges and nothing uses it, so a
	63	+// connection there is refused by construction rather than by luck.
54	64	func closedURL(t *testing.T) string {
55	65	t.Helper()
56		- srv := httptest.NewServer(http.NotFoundHandler())
57		- url := srv.URL
58		- srv.Close()
59		- return url
	66	+ return "http://127.0.0.1:1"
60	67	}
61	68
62	69	func ptr(s string) *string { return &s }

  // closedURL returns a URL nothing listens on: a server started and stopped, so
  // the port was free a moment ago.
++// closedURL is an address nothing answers on.
++//
++// It used to start an httptest server, record its URL and close it — which
++// leaves an EPHEMERAL port that the kernel is free to hand to the next
++// listener, so the test occasionally connected to something real and lost the
++// "nothing answers" hint it was checking for. Measured: one failure in about
++// thirty runs, and more when other servers were being started nearby.
++//
++// Port 1 cannot be listened on without privileges and nothing uses it, so a
++// connection there is refused by construction rather than by luck.
  func closedURL(t *testing.T) string {
  	t.Helper()
--	srv := httptest.NewServer(http.NotFoundHandler())
++	return "http://127.0.0.1:1"
--	url := srv.URL
--	srv.Close()
--	return url
  }
  func ptr(s string) *string { return &s }

added internal/eval/doc.go +24 -0 Viewed

new file mode 100644
@@ -0,0 +1,24 @@
	1	+// Package eval measures whether the model picks the right skill.
	2	+//
	3	+// Everything else in mm is tested against a fake: fast, hermetic, and
	4	+// deterministic. This one cannot be. The question it answers — "does a 4B
	5	+// model reading this catalogue reach for the right procedure?" — is a
	6	+// question about a MODEL, and only a model can answer it.
	7	+//
	8	+// So it is opt-in. `go test ./...` skips it; nothing in CI depends on a
	9	+// server being up. Run it deliberately:
	10	+//
	11	+// MM_EVAL=1 AGENT_CONFIG=tests/agent.dmr-qwen.yaml go test ./internal/eval -v
	12	+//
	13	+// What it measures is the CATALOGUE, not the agent. One model turn per case,
	14	+// with tools that record what was asked for and run nothing: a skill that
	15	+// says "run this echo" must not be echoed twenty times, and an agent loop
	16	+// would measure recovery as much as routing. What is left is exactly the
	17	+// thing ticket 15 is about — given this system prompt and this catalogue,
	18	+// what does the model reach for first?
	19	+//
	20	+// Read the numbers as a comparison, not as a grade. The useful question is
	21	+// never "is 70% good" but "did that change to the keywords, or to the SKILLS
	22	+// block of the prompt, move it up or down" — so run it before a change and
	23	+// after, against the same model, and keep both outputs.
	24	+package eval

++// Package eval measures whether the model picks the right skill.
++//
++// Everything else in mm is tested against a fake: fast, hermetic, and
++// deterministic. This one cannot be. The question it answers — "does a 4B
++// model reading this catalogue reach for the right procedure?" — is a
++// question about a MODEL, and only a model can answer it.
++//
++// So it is opt-in. `go test ./...` skips it; nothing in CI depends on a
++// server being up. Run it deliberately:
++//
++//	MM_EVAL=1 AGENT_CONFIG=tests/agent.dmr-qwen.yaml go test ./internal/eval -v
++//
++// What it measures is the CATALOGUE, not the agent. One model turn per case,
++// with tools that record what was asked for and run nothing: a skill that
++// says "run this echo" must not be echoed twenty times, and an agent loop
++// would measure recovery as much as routing. What is left is exactly the
++// thing ticket 15 is about — given this system prompt and this catalogue,
++// what does the model reach for first?
++//
++// Read the numbers as a comparison, not as a grade. The useful question is
++// never "is 70% good" but "did that change to the keywords, or to the SKILLS
++// block of the prompt, move it up or down" — so run it before a change and
++// after, against the same model, and keep both outputs.
++package eval

added internal/eval/routing_test.go +345 -0 Viewed

new file mode 100644
@@ -0,0 +1,345 @@
	1	+package eval
	2	+
	3	+import (
	4	+ "context"
	5	+ "fmt"
	6	+ "io"
	7	+ "os"
	8	+ "path/filepath"
	9	+ "sort"
	10	+ "strings"
	11	+ "sync"
	12	+ "testing"
	13	+ "time"
	14	+
	15	+ "mm/internal/config"
	16	+ "mm/internal/engine"
	17	+ "mm/internal/skills"
	18	+ "mm/internal/spinner"
	19	+ "mm/internal/ui"
	20	+
	21	+ "github.com/firebase/genkit/go/ai"
	22	+ "github.com/firebase/genkit/go/genkit"
	23	+)
	24	+
	25	+// none is the expected skill of a request that no skill covers. Abstaining is
	26	+// a right answer, and a catalogue that fires on everything is as broken as
	27	+// one that never fires — hence these cases carry the same weight as the rest.
	28	+const none = ""
	29	+
	30	+// A case is a request a user could plausibly type, and the skill it should
	31	+// reach. The prompts are deliberately not phrased in the words of the
	32	+// catalogue: a model matching "use the greetings skill" measures nothing.
	33	+type routingCase struct {
	34	+ prompt string
	35	+ want string
	36	+ why string // what this case is here to catch, when it is not obvious
	37	+}
	38	+
	39	+var cases = []routingCase{
	40	+ // --- each shipped skill, in the words a user would use ------------------
	41	+ {"say hi to Sam for me", "greetings", ""},
	42	+ {"envoie un petit bonjour à Sam", "greetings", "French, and the skill's name is English"},
	43	+ {"give Spock the proper farewell", "vulcan-salute", "no shared word with the name or the description"},
	44	+ {"fais le salut vulcain à Sam", "vulcan-salute", ""},
	45	+ {"what time is it?", "what-time-is-it", ""},
	46	+ {"quelle heure est-il ?", "what-time-is-it", "French"},
	47	+ {"what is today's date?", "what-time-is-it", ""},
	48	+ {"where is the Greet function defined?", "lsp-navigation", ""},
	49	+ {"qui appelle cette fonction ?", "lsp-navigation", "French, and the skill is about symbols"},
	50	+ {"rename the Greet symbol to Hello everywhere", "lsp-navigation", ""},
	51	+ {"start the dev server and check it answers", "bg-jobs", ""},
	52	+ {"lance le serveur en tâche de fond", "bg-jobs", "French"},
	53	+ {"the server I started earlier is still running, stop it", "bg-jobs", ""},
	54	+
	55	+ // --- bg-jobs: why does the FRENCH case hit and the English ones miss? ---
	56	+ // Measured on mellum2-12b and qwen3.5-4b: "lance le serveur en tâche de
	57	+ // fond" hits, "start the dev server and check it answers" misses. bg-jobs
	58	+ // has NO French keyword at all, so "French coverage" cannot be the
	59	+ // reason. Three candidate explanations, and these three cases separate
	60	+ // them:
	61	+ // (a) the French prompt NAMES the concept — "tâche de fond" is literally
	62	+ // "background", the first word of the description and of the
	63	+ // keywords — while the English ones describe a situation;
	64	+ // (b) `stop`, `kill` and `running` are past the 8-keyword cap and never
	65	+ // reach the model, so a request to stop a job has only `server`;
	66	+ // (c) the language really does matter.
	67	+ // If (a) holds, the first hits, the second misses, and the language is a
	68	+ // red herring. If (b) holds, the third hits.
	69	+ {"start the dev server in the background", "bg-jobs", "(a) English, but NAMES the concept"},
	70	+ {"démarre le serveur de dev et vérifie qu'il répond", "bg-jobs", "(a) French, but only a situation"},
	71	+ {"kill the background job I started", "bg-jobs", "(b) stop/kill/running are past the cap"},
	72	+
	73	+ // --- the overlap found on 2026-09-21 ------------------------------------
	74	+ // "Salue Bob de ma part" reached a skill through the keywords — and picked
	75	+ // vulcan-salute, because `greetings` claims "saluer" and `vulcan-salute`
	76	+ // claims "salut". Both are defensible readings of the French; the point of
	77	+ // keeping the case is to see whether a change makes the collision worse.
	78	+ {"Salue Bob de ma part", "greetings", "two skills claim salut/saluer — see ticket 15"},
	79	+
	80	+ // --- nothing applies: abstaining is the right answer --------------------
	81	+ {"what is 17 times 23?", none, ""},
	82	+ {"explain the difference between a slice and an array in Go", none, "a question, not a task"},
	83	+ {"read main.go and tell me what it does", none, "a file tool, not a skill"},
	84	+ {"list the files in this directory", none, "bash, not a skill"},
	85	+ {"write a haiku about pointers", none, ""},
	86	+ {"combien font 17 fois 23 ?", none, "French, still no skill"},
	87	+}
	88	+
	89	+// outcome is how one case came out. "wrong" is kept apart from "miss" on
	90	+// purpose: a model that reaches for a procedure and picks the neighbouring
	91	+// one has a catalogue problem, and a model that reaches for nothing has a
	92	+// prompt problem. Merging them hides which.
	93	+type outcome string
	94	+
	95	+const (
	96	+ hit outcome = "hit" // the expected skill, or correctly nothing
	97	+ wrong outcome = "wrong" // a skill was loaded, but not that one
	98	+ miss outcome = "miss" // a skill was expected and none was loaded
	99	+ extra outcome = "extra" // no skill applied and one was loaded anyway
	100	+ broke outcome = "error" // the turn failed: see the note in the test body
	101	+)
	102	+
	103	+func TestSkillRouting(t *testing.T) {
	104	+ if os.Getenv("MM_EVAL") == "" {
	105	+ t.Skip("set MM_EVAL=1 (and AGENT_CONFIG) to run the routing eval against a live model")
	106	+ }
	107	+
	108	+ // The eval writes a table, not a transcript: silence the agent's own
	109	+ // output and its spinner, which would fight the test log for the line.
	110	+ // ui.Out covers the banner and the warnings; the ANSWER does not go
	111	+ // through it — engine.emit writes the streamed text with fmt.Print, i.e.
	112	+ // straight to os.Stdout — so stdout itself is redirected for the run.
	113	+ // t.Log is unaffected: testing keeps its own writer.
	114	+ ui.Out = io.Discard
	115	+ spinner.Disable()
	116	+ if devnull, err := os.OpenFile(os.DevNull, os.O_WRONLY, 0); err == nil {
	117	+ saved := os.Stdout
	118	+ os.Stdout = devnull
	119	+ t.Cleanup(func() { os.Stdout = saved; devnull.Close() })
	120	+ }
	121	+
	122	+ path, err := config.Load(os.Getenv("AGENT_CONFIG"))
	123	+ if err != nil {
	124	+ t.Fatalf("config: %v", err)
	125	+ }
	126	+ dir := config.Cfg.SkillsDir
	127	+ if !filepath.IsAbs(dir) {
	128	+ root, _ := filepath.Abs(filepath.Join("..", ".."))
	129	+ dir = filepath.Join(root, dir)
	130	+ }
	131	+ list := skills.List(dir)
	132	+ if len(list) == 0 {
	133	+ t.Fatalf("no skills in %s — the eval would score an empty catalogue", dir)
	134	+ }
	135	+
	136	+ e, err := engine.New(context.Background())
	137	+ if err != nil {
	138	+ t.Fatalf("engine: %v", err)
	139	+ }
	140	+ if info := e.Probe(context.Background()); !info.Reachable {
	141	+ t.Fatalf("%s does not answer — start the model server first", e.Backend.BaseURL)
	142	+ }
	143	+
	144	+ asked := &recorder{}
	145	+ tools := stubTools(t, e.G, list, asked)
	146	+
	147	+ t.Logf("model %s @ %s · config %s · %d skills · catalogue %d bytes",
	148	+ config.Cfg.Model, e.Backend.Provider, path, len(list), len(skills.Catalogue(list)))
	149	+
	150	+ // Rows go to stderr as they happen, not through t.Log: testing buffers
	151	+ // its log until the test ends, and an eval that prints nothing for four
	152	+ // minutes looks hung. The summary goes to both, so it survives in the
	153	+ // test output too.
	154	+ scores := map[outcome]int{}
	155	+ for i, c := range cases {
	156	+ asked.reset()
	157	+ got, dur, err := ask(e, tools, c.prompt)
	158	+ res := score(c.want, got, err)
	159	+ scores[res]++
	160	+ fmt.Fprintf(os.Stderr, "[%2d/%2d] %s\n", i+1, len(cases), row(res, c, got, dur, err))
	161	+ }
	162	+
	163	+ out := summary(scores, len(cases))
	164	+ fmt.Fprintln(os.Stderr, out)
	165	+ t.Log(out)
	166	+
	167	+ // Deliberately NOT a threshold. A number that fails the build would turn
	168	+ // a measurement into a target, and the first response to a red build is
	169	+ // to edit the cases. Read the table; compare it with the last run.
	170	+ if scores[broke] > 0 {
	171	+ t.Errorf("%d case(s) failed to run, so the numbers above are incomplete.\n"+
	172	+ "An `error` is never a routing score. Two families, and the message says which:\n"+
	173	+ " - the instrument: a stub that gave the model nothing to finish on, a caseTimeout "+
	174	+ "too short for this model, a server that went away;\n"+
	175	+ " - the model: it could not compose a valid tool call at all — `unexpected end of "+
	176	+ "JSON input`, or the runaway guard cutting off repeated arguments. That is a real "+
	177	+ "finding about the model, not about the catalogue, and no prompt change will fix it.",
	178	+ scores[broke])
	179	+ }
	180	+}
	181	+
	182	+// caseTimeout is generous on purpose. An `error` does not just lose one case,
	183	+// it poisons the whole table — so the cap is there to stop a hung server, not
	184	+// to score speed. Measured on qwen3.5-4b: most cases land in 5–20s, one took
	185	+// 1m20s, and a 2-minute cap turned three perfectly good cases into errors.
	186	+//
	187	+// A slow model therefore makes a slow eval. That is the right trade: the
	188	+// numbers are the point, the wall clock is not.
	189	+const caseTimeout = 5 * time.Minute
	190	+
	191	+// ask runs ONE model turn and reports the skill it asked for, if any.
	192	+func ask(e *engine.Engine, tools []ai.ToolRef, prompt string) (string, time.Duration, error) {
	193	+ ctx, cancel := context.WithTimeout(context.Background(), caseTimeout)
	194	+ defer cancel()
	195	+
	196	+ start := time.Now()
	197	+ msgs := []*ai.Message{
	198	+ ai.NewSystemTextMessage(config.Cfg.System),
	199	+ ai.NewUserTextMessage(prompt),
	200	+ }
	201	+ _, history, err := e.Generate(ctx, msgs, tools)
	202	+ took := time.Since(start).Round(time.Millisecond)
	203	+ if err != nil {
	204	+ return "", took, err
	205	+ }
	206	+ return firstSkill(history), took, nil
	207	+}
	208	+
	209	+// firstSkill is the name of the first read_skill call of a turn — what the
	210	+// model reached for, before any recovery.
	211	+func firstSkill(history []*ai.Message) string {
	212	+ for _, m := range history {
	213	+ for _, p := range m.Content {
	214	+ if !p.IsToolRequest() \|\| p.ToolRequest == nil \|\| p.ToolRequest.Name != "read_skill" {
	215	+ continue
	216	+ }
	217	+ if in, ok := p.ToolRequest.Input.(map[string]any); ok {
	218	+ if n, ok := in["name"].(string); ok {
	219	+ return strings.TrimSpace(n)
	220	+ }
	221	+ }
	222	+ }
	223	+ }
	224	+ return ""
	225	+}
	226	+
	227	+func score(want, got string, err error) outcome {
	228	+ switch {
	229	+ case err != nil:
	230	+ return broke
	231	+ case want == none && got == "":
	232	+ return hit
	233	+ case want == none:
	234	+ return extra
	235	+ case got == "":
	236	+ return miss
	237	+ case got == want:
	238	+ return hit
	239	+ }
	240	+ return wrong
	241	+}
	242	+
	243	+func row(res outcome, c routingCase, got string, dur time.Duration, err error) string {
	244	+ mark := map[outcome]string{hit: "✓", wrong: "≠", miss: "·", extra: "!", broke: "✗"}[res]
	245	+ want, gotShown := c.want, got
	246	+ if want == none {
	247	+ want = "(none)"
	248	+ }
	249	+ if gotShown == "" {
	250	+ gotShown = "(none)"
	251	+ }
	252	+ line := fmt.Sprintf("%s %-6s want=%-16s got=%-16s %6s %q", mark, res, want, gotShown, dur, c.prompt)
	253	+ if c.why != "" {
	254	+ line += " ← " + c.why
	255	+ }
	256	+ if err != nil {
	257	+ line += " [" + err.Error() + "]"
	258	+ }
	259	+ return line
	260	+}
	261	+
	262	+func summary(scores map[outcome]int, total int) string {
	263	+ var b strings.Builder
	264	+ fmt.Fprintf(&b, "\n%d cases · ", total)
	265	+ keys := []outcome{hit, wrong, miss, extra, broke}
	266	+ var parts []string
	267	+ for _, k := range keys {
	268	+ if scores[k] > 0 {
	269	+ parts = append(parts, fmt.Sprintf("%s %d", k, scores[k]))
	270	+ }
	271	+ }
	272	+ sort.Strings(nil) // keep the order above, not alphabetical
	273	+ b.WriteString(strings.Join(parts, " · "))
	274	+ fmt.Fprintf(&b, " → %d%% right", 100*scores[hit]/max(total, 1))
	275	+ return b.String()
	276	+}
	277	+
	278	+// --- the stub tools ---------------------------------------------------------
	279	+
	280	+// recorder remembers what the model asked for. Written from Genkit's tool
	281	+// goroutines, read after the turn.
	282	+type recorder struct {
	283	+ mu sync.Mutex
	284	+ skill string
	285	+}
	286	+
	287	+func (r *recorder) reset() { r.mu.Lock(); r.skill = ""; r.mu.Unlock() }
	288	+
	289	+// stubTools declares the real tool DESCRIPTIONS with implementations that run
	290	+// nothing.
	291	+//
	292	+// The descriptions are what is being measured — `read_skill` carries the real
	293	+// catalogue — but a skill that says "run this echo" must not actually echo,
	294	+// and the file tools must not touch the disk.
	295	+//
	296	+// Every stub must answer something the model can FINISH on. The first version
	297	+// answered "[eval] command not run", and a quarter of the run turned into
	298	+// artefacts: given nothing to work with, the model reran the same command
	299	+// until the runaway guard cut it off or the case timed out — five cases
	300	+// scored `error`, none of them about routing. A stub that starves the model
	301	+// measures the stub. So each one now says, in as few words as possible, that
	302	+// the work is done and the turn should end.
	303	+func stubTools(t testing.T, g genkit.Genkit, list []skills.Skill, rec *recorder) []ai.ToolRef {
	304	+ t.Helper()
	305	+ type in struct {
	306	+ Name string `json:"name" jsonschema_description:"The skill name, exactly as listed in this tool's description"`
	307	+ }
	308	+ readSkill := genkit.DefineTool(g, "read_skill", skills.Catalogue(list),
	309	+ func(_ *ai.ToolContext, i in) (string, error) {
	310	+ rec.mu.Lock()
	311	+ if rec.skill == "" {
	312	+ rec.skill = i.Name
	313	+ }
	314	+ rec.mu.Unlock()
	315	+ // Answer as if the procedure had been read and carried out: what
	316	+ // happens after the choice is not what is being measured.
	317	+ return done("the skill was loaded and its steps are already carried out"), nil
	318	+ })
	319	+
	320	+ type cmd struct {
	321	+ Command string `json:"command" jsonschema_description:"The shell command to run"`
	322	+ }
	323	+ bash := genkit.DefineTool(g, "bash", "Run a shell command and return its output.",
	324	+ func(_ *ai.ToolContext, c cmd) (string, error) {
	325	+ return done("the command ran and did what was asked"), nil
	326	+ })
	327	+
	328	+ type path struct {
	329	+ Path string `json:"path" jsonschema_description:"Path of the file"`
	330	+ }
	331	+ readFile := genkit.DefineTool(g, "read_file", "Read a file from the project.",
	332	+ func(_ *ai.ToolContext, p path) (string, error) {
	333	+ return done("the file was read and holds nothing surprising"), nil
	334	+ })
	335	+
	336	+ return []ai.ToolRef{readSkill, bash, readFile}
	337	+}
	338	+
	339	+// done is what every stub answers: this worked, there is nothing more to do,
	340	+// end the turn. The wording matters more than it looks — the eval scores the
	341	+// FIRST skill asked for, so anything that keeps the model calling tools costs
	342	+// time and risks a runaway, and measures nothing extra.
	343	+func done(what string) string {
	344	+ return "[eval] " + what + ". Nothing else is needed: answer the user now, in one short sentence, and call no further tool."
	345	+}

++package eval
++
++import (
++	"context"
++	"fmt"
++	"io"
++	"os"
++	"path/filepath"
++	"sort"
++	"strings"
++	"sync"
++	"testing"
++	"time"
++
++	"mm/internal/config"
++	"mm/internal/engine"
++	"mm/internal/skills"
++	"mm/internal/spinner"
++	"mm/internal/ui"
++
++	"github.com/firebase/genkit/go/ai"
++	"github.com/firebase/genkit/go/genkit"
++)
++
++// none is the expected skill of a request that no skill covers. Abstaining is
++// a right answer, and a catalogue that fires on everything is as broken as
++// one that never fires — hence these cases carry the same weight as the rest.
++const none = ""
++
++// A case is a request a user could plausibly type, and the skill it should
++// reach. The prompts are deliberately not phrased in the words of the
++// catalogue: a model matching "use the greetings skill" measures nothing.
++type routingCase struct {
++	prompt string
++	want   string
++	why    string // what this case is here to catch, when it is not obvious
++}
++
++var cases = []routingCase{
++	// --- each shipped skill, in the words a user would use ------------------
++	{"say hi to Sam for me", "greetings", ""},
++	{"envoie un petit bonjour à Sam", "greetings", "French, and the skill's name is English"},
++	{"give Spock the proper farewell", "vulcan-salute", "no shared word with the name or the description"},
++	{"fais le salut vulcain à Sam", "vulcan-salute", ""},
++	{"what time is it?", "what-time-is-it", ""},
++	{"quelle heure est-il ?", "what-time-is-it", "French"},
++	{"what is today's date?", "what-time-is-it", ""},
++	{"where is the Greet function defined?", "lsp-navigation", ""},
++	{"qui appelle cette fonction ?", "lsp-navigation", "French, and the skill is about symbols"},
++	{"rename the Greet symbol to Hello everywhere", "lsp-navigation", ""},
++	{"start the dev server and check it answers", "bg-jobs", ""},
++	{"lance le serveur en tâche de fond", "bg-jobs", "French"},
++	{"the server I started earlier is still running, stop it", "bg-jobs", ""},
++
++	// --- bg-jobs: why does the FRENCH case hit and the English ones miss? ---
++	// Measured on mellum2-12b and qwen3.5-4b: "lance le serveur en tâche de
++	// fond" hits, "start the dev server and check it answers" misses. bg-jobs
++	// has NO French keyword at all, so "French coverage" cannot be the
++	// reason. Three candidate explanations, and these three cases separate
++	// them:
++	//   (a) the French prompt NAMES the concept — "tâche de fond" is literally
++	//       "background", the first word of the description and of the
++	//       keywords — while the English ones describe a situation;
++	//   (b) `stop`, `kill` and `running` are past the 8-keyword cap and never
++	//       reach the model, so a request to stop a job has only `server`;
++	//   (c) the language really does matter.
++	// If (a) holds, the first hits, the second misses, and the language is a
++	// red herring. If (b) holds, the third hits.
++	{"start the dev server in the background", "bg-jobs", "(a) English, but NAMES the concept"},
++	{"démarre le serveur de dev et vérifie qu'il répond", "bg-jobs", "(a) French, but only a situation"},
++	{"kill the background job I started", "bg-jobs", "(b) stop/kill/running are past the cap"},
++
++	// --- the overlap found on 2026-09-21 ------------------------------------
++	// "Salue Bob de ma part" reached a skill through the keywords — and picked
++	// vulcan-salute, because `greetings` claims "saluer" and `vulcan-salute`
++	// claims "salut". Both are defensible readings of the French; the point of
++	// keeping the case is to see whether a change makes the collision worse.
++	{"Salue Bob de ma part", "greetings", "two skills claim salut/saluer — see ticket 15"},
++
++	// --- nothing applies: abstaining is the right answer --------------------
++	{"what is 17 times 23?", none, ""},
++	{"explain the difference between a slice and an array in Go", none, "a question, not a task"},
++	{"read main.go and tell me what it does", none, "a file tool, not a skill"},
++	{"list the files in this directory", none, "bash, not a skill"},
++	{"write a haiku about pointers", none, ""},
++	{"combien font 17 fois 23 ?", none, "French, still no skill"},
++}
++
++// outcome is how one case came out. "wrong" is kept apart from "miss" on
++// purpose: a model that reaches for a procedure and picks the neighbouring
++// one has a catalogue problem, and a model that reaches for nothing has a
++// prompt problem. Merging them hides which.
++type outcome string
++
++const (
++	hit   outcome = "hit"   // the expected skill, or correctly nothing
++	wrong outcome = "wrong" // a skill was loaded, but not that one
++	miss  outcome = "miss"  // a skill was expected and none was loaded
++	extra outcome = "extra" // no skill applied and one was loaded anyway
++	broke outcome = "error" // the turn failed: see the note in the test body
++)
++
++func TestSkillRouting(t *testing.T) {
++	if os.Getenv("MM_EVAL") == "" {
++		t.Skip("set MM_EVAL=1 (and AGENT_CONFIG) to run the routing eval against a live model")
++	}
++
++	// The eval writes a table, not a transcript: silence the agent's own
++	// output and its spinner, which would fight the test log for the line.
++	// ui.Out covers the banner and the warnings; the ANSWER does not go
++	// through it — engine.emit writes the streamed text with fmt.Print, i.e.
++	// straight to os.Stdout — so stdout itself is redirected for the run.
++	// t.Log is unaffected: testing keeps its own writer.
++	ui.Out = io.Discard
++	spinner.Disable()
++	if devnull, err := os.OpenFile(os.DevNull, os.O_WRONLY, 0); err == nil {
++		saved := os.Stdout
++		os.Stdout = devnull
++		t.Cleanup(func() { os.Stdout = saved; devnull.Close() })
++	}
++
++	path, err := config.Load(os.Getenv("AGENT_CONFIG"))
++	if err != nil {
++		t.Fatalf("config: %v", err)
++	}
++	dir := config.Cfg.SkillsDir
++	if !filepath.IsAbs(dir) {
++		root, _ := filepath.Abs(filepath.Join("..", ".."))
++		dir = filepath.Join(root, dir)
++	}
++	list := skills.List(dir)
++	if len(list) == 0 {
++		t.Fatalf("no skills in %s — the eval would score an empty catalogue", dir)
++	}
++
++	e, err := engine.New(context.Background())
++	if err != nil {
++		t.Fatalf("engine: %v", err)
++	}
++	if info := e.Probe(context.Background()); !info.Reachable {
++		t.Fatalf("%s does not answer — start the model server first", e.Backend.BaseURL)
++	}
++
++	asked := &recorder{}
++	tools := stubTools(t, e.G, list, asked)
++
++	t.Logf("model %s @ %s · config %s · %d skills · catalogue %d bytes",
++		config.Cfg.Model, e.Backend.Provider, path, len(list), len(skills.Catalogue(list)))
++
++	// Rows go to stderr as they happen, not through t.Log: testing buffers
++	// its log until the test ends, and an eval that prints nothing for four
++	// minutes looks hung. The summary goes to both, so it survives in the
++	// test output too.
++	scores := map[outcome]int{}
++	for i, c := range cases {
++		asked.reset()
++		got, dur, err := ask(e, tools, c.prompt)
++		res := score(c.want, got, err)
++		scores[res]++
++		fmt.Fprintf(os.Stderr, "[%2d/%2d] %s\n", i+1, len(cases), row(res, c, got, dur, err))
++	}
++
++	out := summary(scores, len(cases))
++	fmt.Fprintln(os.Stderr, out)
++	t.Log(out)
++
++	// Deliberately NOT a threshold. A number that fails the build would turn
++	// a measurement into a target, and the first response to a red build is
++	// to edit the cases. Read the table; compare it with the last run.
++	if scores[broke] > 0 {
++		t.Errorf("%d case(s) failed to run, so the numbers above are incomplete.\n"+
++			"An `error` is never a routing score. Two families, and the message says which:\n"+
++			"  - the instrument: a stub that gave the model nothing to finish on, a caseTimeout "+
++			"too short for this model, a server that went away;\n"+
++			"  - the model: it could not compose a valid tool call at all — `unexpected end of "+
++			"JSON input`, or the runaway guard cutting off repeated arguments. That is a real "+
++			"finding about the model, not about the catalogue, and no prompt change will fix it.",
++			scores[broke])
++	}
++}
++
++// caseTimeout is generous on purpose. An `error` does not just lose one case,
++// it poisons the whole table — so the cap is there to stop a hung server, not
++// to score speed. Measured on qwen3.5-4b: most cases land in 5–20s, one took
++// 1m20s, and a 2-minute cap turned three perfectly good cases into errors.
++//
++// A slow model therefore makes a slow eval. That is the right trade: the
++// numbers are the point, the wall clock is not.
++const caseTimeout = 5 * time.Minute
++
++// ask runs ONE model turn and reports the skill it asked for, if any.
++func ask(e *engine.Engine, tools []ai.ToolRef, prompt string) (string, time.Duration, error) {
++	ctx, cancel := context.WithTimeout(context.Background(), caseTimeout)
++	defer cancel()
++
++	start := time.Now()
++	msgs := []*ai.Message{
++		ai.NewSystemTextMessage(config.Cfg.System),
++		ai.NewUserTextMessage(prompt),
++	}
++	_, history, err := e.Generate(ctx, msgs, tools)
++	took := time.Since(start).Round(time.Millisecond)
++	if err != nil {
++		return "", took, err
++	}
++	return firstSkill(history), took, nil
++}
++
++// firstSkill is the name of the first read_skill call of a turn — what the
++// model reached for, before any recovery.
++func firstSkill(history []*ai.Message) string {
++	for _, m := range history {
++		for _, p := range m.Content {
++			if !p.IsToolRequest() || p.ToolRequest == nil || p.ToolRequest.Name != "read_skill" {
++				continue
++			}
++			if in, ok := p.ToolRequest.Input.(map[string]any); ok {
++				if n, ok := in["name"].(string); ok {
++					return strings.TrimSpace(n)
++				}
++			}
++		}
++	}
++	return ""
++}
++
++func score(want, got string, err error) outcome {
++	switch {
++	case err != nil:
++		return broke
++	case want == none && got == "":
++		return hit
++	case want == none:
++		return extra
++	case got == "":
++		return miss
++	case got == want:
++		return hit
++	}
++	return wrong
++}
++
++func row(res outcome, c routingCase, got string, dur time.Duration, err error) string {
++	mark := map[outcome]string{hit: "✓", wrong: "≠", miss: "·", extra: "!", broke: "✗"}[res]
++	want, gotShown := c.want, got
++	if want == none {
++		want = "(none)"
++	}
++	if gotShown == "" {
++		gotShown = "(none)"
++	}
++	line := fmt.Sprintf("%s %-6s want=%-16s got=%-16s %6s  %q", mark, res, want, gotShown, dur, c.prompt)
++	if c.why != "" {
++		line += "  ← " + c.why
++	}
++	if err != nil {
++		line += "  [" + err.Error() + "]"
++	}
++	return line
++}
++
++func summary(scores map[outcome]int, total int) string {
++	var b strings.Builder
++	fmt.Fprintf(&b, "\n%d cases · ", total)
++	keys := []outcome{hit, wrong, miss, extra, broke}
++	var parts []string
++	for _, k := range keys {
++		if scores[k] > 0 {
++			parts = append(parts, fmt.Sprintf("%s %d", k, scores[k]))
++		}
++	}
++	sort.Strings(nil) // keep the order above, not alphabetical
++	b.WriteString(strings.Join(parts, " · "))
++	fmt.Fprintf(&b, "  →  %d%% right", 100*scores[hit]/max(total, 1))
++	return b.String()
++}
++
++// --- the stub tools ---------------------------------------------------------
++
++// recorder remembers what the model asked for. Written from Genkit's tool
++// goroutines, read after the turn.
++type recorder struct {
++	mu    sync.Mutex
++	skill string
++}
++
++func (r *recorder) reset() { r.mu.Lock(); r.skill = ""; r.mu.Unlock() }
++
++// stubTools declares the real tool DESCRIPTIONS with implementations that run
++// nothing.
++//
++// The descriptions are what is being measured — `read_skill` carries the real
++// catalogue — but a skill that says "run this echo" must not actually echo,
++// and the file tools must not touch the disk.
++//
++// Every stub must answer something the model can FINISH on. The first version
++// answered "[eval] command not run", and a quarter of the run turned into
++// artefacts: given nothing to work with, the model reran the same command
++// until the runaway guard cut it off or the case timed out — five cases
++// scored `error`, none of them about routing. A stub that starves the model
++// measures the stub. So each one now says, in as few words as possible, that
++// the work is done and the turn should end.
++func stubTools(t *testing.T, g *genkit.Genkit, list []skills.Skill, rec *recorder) []ai.ToolRef {
++	t.Helper()
++	type in struct {
++		Name string `json:"name" jsonschema_description:"The skill name, exactly as listed in this tool's description"`
++	}
++	readSkill := genkit.DefineTool(g, "read_skill", skills.Catalogue(list),
++		func(_ *ai.ToolContext, i in) (string, error) {
++			rec.mu.Lock()
++			if rec.skill == "" {
++				rec.skill = i.Name
++			}
++			rec.mu.Unlock()
++			// Answer as if the procedure had been read and carried out: what
++			// happens after the choice is not what is being measured.
++			return done("the skill was loaded and its steps are already carried out"), nil
++		})
++
++	type cmd struct {
++		Command string `json:"command" jsonschema_description:"The shell command to run"`
++	}
++	bash := genkit.DefineTool(g, "bash", "Run a shell command and return its output.",
++		func(_ *ai.ToolContext, c cmd) (string, error) {
++			return done("the command ran and did what was asked"), nil
++		})
++
++	type path struct {
++		Path string `json:"path" jsonschema_description:"Path of the file"`
++	}
++	readFile := genkit.DefineTool(g, "read_file", "Read a file from the project.",
++		func(_ *ai.ToolContext, p path) (string, error) {
++			return done("the file was read and holds nothing surprising"), nil
++		})
++
++	return []ai.ToolRef{readSkill, bash, readFile}
++}
++
++// done is what every stub answers: this worked, there is nothing more to do,
++// end the turn. The wording matters more than it looks — the eval scores the
++// FIRST skill asked for, so anything that keeps the model calling tools costs
++// time and risks a runaway, and measures nothing extra.
++func done(what string) string {
++	return "[eval] " + what + ". Nothing else is needed: answer the user now, in one short sentence, and call no further tool."
++}

modified internal/lsp/lsp_test.go +33 -5 Viewed

@@ -82,6 +82,35 @@ func (f fakeServer) DidChange(_ context.Context, p protocol.DidChangeTextDocum
82	82	return nil
83	83	}
84	84
	85	+// awaitDoc waits until the server has been told about a document at the given
	86	+// version, and returns the text it holds for it.
	87	+//
	88	+// It exists because `textDocument/didChange` is a NOTIFICATION: the client
	89	+// sends it and moves on without a reply, so a test that reads this state the
	90	+// instant a client call returns can read it before the notification has been
	91	+// handled. Nothing is wrong in production — JSON-RPC on one stream delivers
	92	+// the notification before the next request, which is what every later call
	93	+// relies on — but a test looking at the server's state OUT OF BAND has to
	94	+// wait for it. Measured before this helper: the rename assertion below failed
	95	+// about one run in ten.
	96	+func (f fakeServer) awaitDoc(t testing.T, u uri.URI, version int32) string {
	97	+ t.Helper()
	98	+ deadline := time.Now().Add(2 * time.Second)
	99	+ for {
	100	+ f.mu.Lock()
	101	+ seen, text := f.opened[u], f.texts[u]
	102	+ f.mu.Unlock()
	103	+ if seen == version {
	104	+ return text
	105	+ }
	106	+ if time.Now().After(deadline) {
	107	+ t.Fatalf("the server was never told about %s at version %d (last seen %d, text %q)",
	108	+ u, version, seen, text)
	109	+ }
	110	+ time.Sleep(5 * time.Millisecond)
	111	+ }
	112	+}
	113	+
85	114	func (f fakeServer) Definition(_ context.Context, p protocol.DefinitionParams) (protocol.DefinitionResult, error) {
86	115	return protocol.LocationSlice{{URI: p.TextDocument.URI, Range: span(2, 5, 2, 10)}}, nil
87	116	}
@@ -197,11 +226,10 @@ func TestOperationsOverAFakeServer(t *testing.T) {
197	226	if string(data) != "package main\n\nfunc greet() {}\n\nfunc main() {\n\tgreet()\n}\n" {
198	227	t.Errorf("file after rename:\n%s", data)
199	228	}
200		- fake.mu.Lock()
201		- version, text := fake.opened[uri.File(filepath.Join(root, "main.go"))], fake.texts[uri.File(filepath.Join(root, "main.go"))]
202		- fake.mu.Unlock()
203		- if version != 2 \|\| !strings.Contains(text, "greet") {
204		- t.Errorf("server not told about the renamed file: version %d, text %q", version, text)
	229	+ // Rename writes the file and then re-syncs it; the sync is a notification,
	230	+ // so wait for it rather than reading the instant Rename returns.
	231	+ if text := fake.awaitDoc(t, uri.File(filepath.Join(root, "main.go")), 2); !strings.Contains(text, "greet") {
	232	+ t.Errorf("server told about the renamed file, but with the old text: %q", text)
205	233	}
206	234
207	235	if _, err := c.Rename(ctx, "main.go", 6, 2, "refuse"); err == nil \|\| !strings.Contains(err.Error(), "fake: rename failed: ") {

  	return nil
  }
++// awaitDoc waits until the server has been told about a document at the given
++// version, and returns the text it holds for it.
++//
++// It exists because `textDocument/didChange` is a NOTIFICATION: the client
++// sends it and moves on without a reply, so a test that reads this state the
++// instant a client call returns can read it before the notification has been
++// handled. Nothing is wrong in production — JSON-RPC on one stream delivers
++// the notification before the next request, which is what every later call
++// relies on — but a test looking at the server's state OUT OF BAND has to
++// wait for it. Measured before this helper: the rename assertion below failed
++// about one run in ten.
++func (f *fakeServer) awaitDoc(t *testing.T, u uri.URI, version int32) string {
++	t.Helper()
++	deadline := time.Now().Add(2 * time.Second)
++	for {
++		f.mu.Lock()
++		seen, text := f.opened[u], f.texts[u]
++		f.mu.Unlock()
++		if seen == version {
++			return text
++		}
++		if time.Now().After(deadline) {
++			t.Fatalf("the server was never told about %s at version %d (last seen %d, text %q)",
++				u, version, seen, text)
++		}
++		time.Sleep(5 * time.Millisecond)
++	}
++}
++
  func (f *fakeServer) Definition(_ context.Context, p *protocol.DefinitionParams) (protocol.DefinitionResult, error) {
  	return protocol.LocationSlice{{URI: p.TextDocument.URI, Range: span(2, 5, 2, 10)}}, nil
  }
  	if string(data) != "package main\n\nfunc greet() {}\n\nfunc main() {\n\tgreet()\n}\n" {
  		t.Errorf("file after rename:\n%s", data)
  	}
--	fake.mu.Lock()
++	// Rename writes the file and then re-syncs it; the sync is a notification,
--	version, text := fake.opened[uri.File(filepath.Join(root, "main.go"))], fake.texts[uri.File(filepath.Join(root, "main.go"))]
++	// so wait for it rather than reading the instant Rename returns.
--	fake.mu.Unlock()
++	if text := fake.awaitDoc(t, uri.File(filepath.Join(root, "main.go")), 2); !strings.Contains(text, "greet") {
--	if version != 2 || !strings.Contains(text, "greet") {
++		t.Errorf("server told about the renamed file, but with the old text: %q", text)
--		t.Errorf("server not told about the renamed file: version %d, text %q", version, text)
  	}
  	if _, err := c.Rename(ctx, "main.go", 6, 2, "refuse"); err == nil || !strings.Contains(err.Error(), "fake: rename failed: ") {

added internal/skills/report.go +57 -0 Viewed

new file mode 100644
@@ -0,0 +1,57 @@
	1	+// What is wrong with a catalogue, for the banner to say at start-up.
	2	+//
	3	+// None of these refuses to run: a skills directory with a problem still
	4	+// works, it just works less well than its author thinks. The whole point is
	5	+// that each of these failures is SILENT — the skill is there, the count is
	6	+// not zero, and the one warning that existed before only fired when the
	7	+// directory was empty.
	8	+
	9	+package skills
	10	+
	11	+import (
	12	+ "fmt"
	13	+ "strings"
	14	+)
	15	+
	16	+// Duplicates lists the names two skills share. The catalogue would show the
	17	+// name twice and Find would always return the first, so the second is
	18	+// unreachable — worth a warning at start-up, not a refusal to run.
	19	+func Duplicates(list []Skill) []string {
	20	+ var out []string
	21	+ for i := 1; i < len(list); i++ { // List sorts by name, so duplicates are adjacent
	22	+ if list[i].Name == list[i-1].Name && (len(out) == 0 \|\| out[len(out)-1] != list[i].Name) {
	23	+ out = append(out, list[i].Name)
	24	+ }
	25	+ }
	26	+ return out
	27	+}
	28	+
	29	+// Truncated lists the skills with more keywords than the catalogue shows.
	30	+//
	31	+// The cap made the ORDER of a keyword list significant, which it was not
	32	+// before, and nothing tells an author that their last eight words are being
	33	+// dropped. Found the hard way: `lsp-navigation` has sixteen keywords, the
	34	+// eight that survived were all English jargon, and the one routing failure
	35	+// the eval found was a French request.
	36	+func Truncated(list []Skill) []string {
	37	+ var out []string
	38	+ for _, s := range list {
	39	+ if len(s.Keywords) > maxKeywords {
	40	+ out = append(out, fmt.Sprintf("%s (%d of %d)", s.Name, maxKeywords, len(s.Keywords)))
	41	+ }
	42	+ }
	43	+ return out
	44	+}
	45	+
	46	+// Undescribed lists the skills whose front matter gave no description. They
	47	+// still appear in the catalogue, as a bare name — the worst entry there is,
	48	+// since the model has nothing to match a request against.
	49	+func Undescribed(list []Skill) []string {
	50	+ var out []string
	51	+ for _, s := range list {
	52	+ if strings.TrimSpace(s.Description) == "" {
	53	+ out = append(out, s.Name)
	54	+ }
	55	+ }
	56	+ return out
	57	+}

++// What is wrong with a catalogue, for the banner to say at start-up.
++//
++// None of these refuses to run: a skills directory with a problem still
++// works, it just works less well than its author thinks. The whole point is
++// that each of these failures is SILENT — the skill is there, the count is
++// not zero, and the one warning that existed before only fired when the
++// directory was empty.
++
++package skills
++
++import (
++	"fmt"
++	"strings"
++)
++
++// Duplicates lists the names two skills share. The catalogue would show the
++// name twice and Find would always return the first, so the second is
++// unreachable — worth a warning at start-up, not a refusal to run.
++func Duplicates(list []Skill) []string {
++	var out []string
++	for i := 1; i < len(list); i++ { // List sorts by name, so duplicates are adjacent
++		if list[i].Name == list[i-1].Name && (len(out) == 0 || out[len(out)-1] != list[i].Name) {
++			out = append(out, list[i].Name)
++		}
++	}
++	return out
++}
++
++// Truncated lists the skills with more keywords than the catalogue shows.
++//
++// The cap made the ORDER of a keyword list significant, which it was not
++// before, and nothing tells an author that their last eight words are being
++// dropped. Found the hard way: `lsp-navigation` has sixteen keywords, the
++// eight that survived were all English jargon, and the one routing failure
++// the eval found was a French request.
++func Truncated(list []Skill) []string {
++	var out []string
++	for _, s := range list {
++		if len(s.Keywords) > maxKeywords {
++			out = append(out, fmt.Sprintf("%s (%d of %d)", s.Name, maxKeywords, len(s.Keywords)))
++		}
++	}
++	return out
++}
++
++// Undescribed lists the skills whose front matter gave no description. They
++// still appear in the catalogue, as a bare name — the worst entry there is,
++// since the model has nothing to match a request against.
++func Undescribed(list []Skill) []string {
++	var out []string
++	for _, s := range list {
++		if strings.TrimSpace(s.Description) == "" {
++			out = append(out, s.Name)
++		}
++	}
++	return out
++}

added internal/skills/shipped_test.go +111 -0 Viewed

new file mode 100644
@@ -0,0 +1,111 @@
	1	+package skills_test
	2	+
	3	+import (
	4	+ "os"
	5	+ "path/filepath"
	6	+ "regexp"
	7	+ "strings"
	8	+ "testing"
	9	+
	10	+ "mm/internal/config"
	11	+ "mm/internal/skills"
	12	+)
	13	+
	14	+// repoRoot is where the shipped YAML files and the skills directory live.
	15	+func repoRoot(t *testing.T) string {
	16	+ t.Helper()
	17	+ root, err := filepath.Abs(filepath.Join("..", ".."))
	18	+ if err != nil {
	19	+ t.Fatalf("repo root: %v", err)
	20	+ }
	21	+ if _, err := os.Stat(filepath.Join(root, "go.mod")); err != nil {
	22	+ t.Fatalf("%s is not the repo root: %v", root, err)
	23	+ }
	24	+ return root
	25	+}
	26	+
	27	+// namedSkill matches the way every prompt refers to a skill: the name quoted
	28	+// or backticked, followed by the word "skill". The prompts in the YAML files
	29	+// use backticks; the built-in default prompt is a Go string, so it uses
	30	+// double quotes.
	31	+var namedSkill = regexp.MustCompile("[`\"']([a-z0-9][a-z0-9._-]*)[`\"']\\s+skill\\b")
	32	+
	33	+// TestEverySkillNamedInAShippedPromptExists is the durable half of the
	34	+// bg-jobs fix.
	35	+//
	36	+// All three shipped YAML files AND the built-in default prompt told the model
	37	+// to "read the `bg-jobs` skill", and `skills/bg-jobs` did not exist — so every
	38	+// background-job request drove the model into a read_skill call that could
	39	+// only fail. Nothing caught it: not a test, not the banner, not the docs.
	40	+//
	41	+// Shipping the skill fixes today. This test fixes the class: name a skill in
	42	+// a prompt without shipping it, and the build says so.
	43	+func TestEverySkillNamedInAShippedPromptExists(t *testing.T) {
	44	+ root := repoRoot(t)
	45	+ list := skills.List(filepath.Join(root, "skills"))
	46	+ if len(list) == 0 {
	47	+ t.Fatal("no skills found in skills/ — the test cannot tell a missing skill from a missing directory")
	48	+ }
	49	+
	50	+ prompts := map[string]string{"built-in default prompt": config.Cfg.System} // package default: Load is never called here
	51	+ yamls, err := filepath.Glob(filepath.Join(root, "agent*.yaml"))
	52	+ if err != nil {
	53	+ t.Fatalf("glob: %v", err)
	54	+ }
	55	+ more, _ := filepath.Glob(filepath.Join(root, "tests", "agent*.yaml"))
	56	+ for _, p := range append(yamls, more...) {
	57	+ b, err := os.ReadFile(p)
	58	+ if err != nil {
	59	+ t.Fatalf("read %s: %v", p, err)
	60	+ }
	61	+ prompts[filepath.Base(p)] = string(b)
	62	+ }
	63	+ if len(prompts) < 2 {
	64	+ t.Fatal("no shipped YAML found: the test would pass by looking at nothing")
	65	+ }
	66	+
	67	+ for where, text := range prompts {
	68	+ for _, m := range namedSkill.FindAllStringSubmatch(text, -1) {
	69	+ name := m[1]
	70	+ if _, ok := skills.Find(list, name); !ok {
	71	+ t.Errorf("%s tells the model to read the %q skill, which does not exist — available: %s",
	72	+ where, name, strings.Join(skills.Names(list), ", "))
	73	+ }
	74	+ }
	75	+ }
	76	+}
	77	+
	78	+// Every shipped skill must be reachable by the name the catalogue advertises,
	79	+// whatever its file is called. This is the regression test for the defect the
	80	+// catalogue/path split caused.
	81	+func TestEveryShippedSkillLoadsByItsCatalogueName(t *testing.T) {
	82	+ list := skills.List(filepath.Join(repoRoot(t), "skills"))
	83	+ for _, s := range list {
	84	+ content, err := skills.Load(list, s.Name)
	85	+ if err != nil {
	86	+ t.Errorf("%q is in the catalogue but cannot be loaded: %v", s.Name, err)
	87	+ continue
	88	+ }
	89	+ if strings.TrimSpace(content) == "" {
	90	+ t.Errorf("%q loads empty", s.Name)
	91	+ }
	92	+ }
	93	+}
	94	+
	95	+// A catalogue entry with no description is a name the model cannot match a
	96	+// request against. The banner warns about it; the shipped ones must not need
	97	+// the warning.
	98	+func TestShippedSkillsAreDescribedAndKeyworded(t *testing.T) {
	99	+ list := skills.List(filepath.Join(repoRoot(t), "skills"))
	100	+ if bare := skills.Undescribed(list); len(bare) > 0 {
	101	+ t.Errorf("shipped skills with no description: %s", strings.Join(bare, ", "))
	102	+ }
	103	+ if dup := skills.Duplicates(list); len(dup) > 0 {
	104	+ t.Errorf("shipped skills sharing a name: %s", strings.Join(dup, ", "))
	105	+ }
	106	+ for _, s := range list {
	107	+ if len(s.Keywords) == 0 {
	108	+ t.Errorf("%q has no keywords: it can only be found by its name", s.Name)
	109	+ }
	110	+ }
	111	+}

++package skills_test
++
++import (
++	"os"
++	"path/filepath"
++	"regexp"
++	"strings"
++	"testing"
++
++	"mm/internal/config"
++	"mm/internal/skills"
++)
++
++// repoRoot is where the shipped YAML files and the skills directory live.
++func repoRoot(t *testing.T) string {
++	t.Helper()
++	root, err := filepath.Abs(filepath.Join("..", ".."))
++	if err != nil {
++		t.Fatalf("repo root: %v", err)
++	}
++	if _, err := os.Stat(filepath.Join(root, "go.mod")); err != nil {
++		t.Fatalf("%s is not the repo root: %v", root, err)
++	}
++	return root
++}
++
++// namedSkill matches the way every prompt refers to a skill: the name quoted
++// or backticked, followed by the word "skill". The prompts in the YAML files
++// use backticks; the built-in default prompt is a Go string, so it uses
++// double quotes.
++var namedSkill = regexp.MustCompile("[`\"']([a-z0-9][a-z0-9._-]*)[`\"']\\s+skill\\b")
++
++// TestEverySkillNamedInAShippedPromptExists is the durable half of the
++// bg-jobs fix.
++//
++// All three shipped YAML files AND the built-in default prompt told the model
++// to "read the `bg-jobs` skill", and `skills/bg-jobs` did not exist — so every
++// background-job request drove the model into a read_skill call that could
++// only fail. Nothing caught it: not a test, not the banner, not the docs.
++//
++// Shipping the skill fixes today. This test fixes the class: name a skill in
++// a prompt without shipping it, and the build says so.
++func TestEverySkillNamedInAShippedPromptExists(t *testing.T) {
++	root := repoRoot(t)
++	list := skills.List(filepath.Join(root, "skills"))
++	if len(list) == 0 {
++		t.Fatal("no skills found in skills/ — the test cannot tell a missing skill from a missing directory")
++	}
++
++	prompts := map[string]string{"built-in default prompt": config.Cfg.System} // package default: Load is never called here
++	yamls, err := filepath.Glob(filepath.Join(root, "agent*.yaml"))
++	if err != nil {
++		t.Fatalf("glob: %v", err)
++	}
++	more, _ := filepath.Glob(filepath.Join(root, "tests", "agent*.yaml"))
++	for _, p := range append(yamls, more...) {
++		b, err := os.ReadFile(p)
++		if err != nil {
++			t.Fatalf("read %s: %v", p, err)
++		}
++		prompts[filepath.Base(p)] = string(b)
++	}
++	if len(prompts) < 2 {
++		t.Fatal("no shipped YAML found: the test would pass by looking at nothing")
++	}
++
++	for where, text := range prompts {
++		for _, m := range namedSkill.FindAllStringSubmatch(text, -1) {
++			name := m[1]
++			if _, ok := skills.Find(list, name); !ok {
++				t.Errorf("%s tells the model to read the %q skill, which does not exist — available: %s",
++					where, name, strings.Join(skills.Names(list), ", "))
++			}
++		}
++	}
++}
++
++// Every shipped skill must be reachable by the name the catalogue advertises,
++// whatever its file is called. This is the regression test for the defect the
++// catalogue/path split caused.
++func TestEveryShippedSkillLoadsByItsCatalogueName(t *testing.T) {
++	list := skills.List(filepath.Join(repoRoot(t), "skills"))
++	for _, s := range list {
++		content, err := skills.Load(list, s.Name)
++		if err != nil {
++			t.Errorf("%q is in the catalogue but cannot be loaded: %v", s.Name, err)
++			continue
++		}
++		if strings.TrimSpace(content) == "" {
++			t.Errorf("%q loads empty", s.Name)
++		}
++	}
++}
++
++// A catalogue entry with no description is a name the model cannot match a
++// request against. The banner warns about it; the shipped ones must not need
++// the warning.
++func TestShippedSkillsAreDescribedAndKeyworded(t *testing.T) {
++	list := skills.List(filepath.Join(repoRoot(t), "skills"))
++	if bare := skills.Undescribed(list); len(bare) > 0 {
++		t.Errorf("shipped skills with no description: %s", strings.Join(bare, ", "))
++	}
++	if dup := skills.Duplicates(list); len(dup) > 0 {
++		t.Errorf("shipped skills sharing a name: %s", strings.Join(dup, ", "))
++	}
++	for _, s := range list {
++		if len(s.Keywords) == 0 {
++			t.Errorf("%q has no keywords: it can only be found by its name", s.Name)
++		}
++	}
++}

modified internal/skills/skills.go +76 -12 Viewed

@@ -16,6 +16,7 @@
16	16	package skills
17	17
18	18	import (
	19	+ "fmt"
19	20	"os"
20	21	"path/filepath"
21	22	"sort"
@@ -26,6 +27,7 @@ import (
26	27	type Skill struct {
27	28	Name string // from the front matter, falling back to the file name
28	29	Description string // the one-liner shown in the catalogue
	30	+ Keywords []string
29	31	Path string
30	32	}
31	33
@@ -43,8 +45,9 @@ func defaultName(path string) string {
43	45	return strings.TrimSuffix(filepath.Base(path), ".md")
44	46	}
45	47
46		-// parseHeader reads the `name:` and `description:` lines of the front matter.
47		-// It stops at the closing `---`: a `description:` in the body is not metadata.
	48	+// parseHeader reads the `name:`, `description:` and `keywords:` lines of the
	49	+// front matter. It stops at the closing `---`: a `description:` in the body is
	50	+// not metadata.
48	51	func parseHeader(content, path string) Skill {
49	52	s := Skill{Name: defaultName(path), Path: path}
50	53	lines := strings.Split(content, "\n")
@@ -67,11 +70,25 @@ func parseHeader(content, path string) Skill {
67	70	}
68	71	case "description":
69	72	s.Description = value
	73	+ case "keywords":
	74	+ s.Keywords = splitKeywords(value)
70	75	}
71	76	}
72	77	return s
73	78	}
74	79
	80	+// splitKeywords turns "greet, hello, bonjour" into its words, dropping the
	81	+// empties a trailing comma leaves behind.
	82	+func splitKeywords(value string) []string {
	83	+ var out []string
	84	+ for _, w := range strings.Split(value, ",") {
	85	+ if w = strings.TrimSpace(w); w != "" {
	86	+ out = append(out, w)
	87	+ }
	88	+ }
	89	+ return out
	90	+}
	91	+
75	92	// List returns the skills of dir, sorted by name. Two layouts are accepted,
76	93	// and may be mixed: a flat `<dir>/<name>.md`, and one directory per skill,
77	94	// `<dir>/<name>/SKILL.md`. A missing directory is not an error: it just means
@@ -96,7 +113,11 @@ func List(dir string) []Skill {
96	113	}
97	114	out = append(out, parseHeader(string(content), p))
98	115	}
99		- sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
	116	+ // STABLE, and the flat files were appended first: when two skills share a
	117	+ // name, the flat `<name>.md` stays ahead of `<name>/SKILL.md` and is the
	118	+ // one Find returns. That is the recorded rule — "the flat file wins" —
	119	+ // and an unstable sort would decide it by coin toss.
	120	+ sort.SliceStable(out, func(i, j int) bool { return out[i].Name < out[j].Name })
100	121	return out
101	122	}
102	123
@@ -106,17 +127,45 @@ func isReadme(path string) bool {
106	127	return strings.EqualFold(filepath.Base(path), "README.md")
107	128	}
108	129
109		-// Read returns the full markdown of one skill, whichever layout it uses: the
110		-// flat `<name>.md` is tried first, then `<name>/SKILL.md`.
111		-func Read(dir, name string) (string, error) {
112		- // Keep the name a plain file name: no "../" escaping out of the directory.
113		- name = filepath.Base(name)
114		- content, err := os.ReadFile(filepath.Join(dir, name+".md"))
115		- if err != nil {
116		- if nested, nestedErr := os.ReadFile(filepath.Join(dir, name, skillFile)); nestedErr == nil {
117		- return string(nested), nil
	130	+// Find returns the skill the CATALOGUE advertises under this name.
	131	+//
	132	+// The lookup goes through the list, not through the filesystem, and that is
	133	+// the whole point: `Catalogue` names a skill by its front-matter `name`,
	134	+// while the old path-based read looked for `<dir>/<name>.md`. When the two
	135	+// disagreed — a skill whose `name:` is not its file name — the tool refused
	136	+// the very string the catalogue had told the model to use, with a message
	137	+// that contradicted itself: `No skill named "renamed". Available: renamed`.
	138	+//
	139	+// It is also strictly safer: the model's string is now only ever compared
	140	+// against known names and never reaches the filesystem, so there is no path
	141	+// to escape from.
	142	+//
	143	+// Matching is forgiving in the two ways a model gets it wrong and no human
	144	+// would: surrounding whitespace or quotes, and the wrong case. It is never
	145	+// forgiving about which skill: an exact name always wins over a case-
	146	+// insensitive one.
	147	+func Find(list []Skill, name string) (Skill, bool) {
	148	+ name = strings.Trim(strings.TrimSpace(name), `"'`+"`")
	149	+ for _, s := range list {
	150	+ if s.Name == name {
	151	+ return s, true
118	152	}
119	153	}
	154	+ for _, s := range list {
	155	+ if strings.EqualFold(s.Name, name) {
	156	+ return s, true
	157	+ }
	158	+ }
	159	+ return Skill{}, false
	160	+}
	161	+
	162	+// Load reads the markdown of the skill the catalogue advertises under name.
	163	+func Load(list []Skill, name string) (string, error) {
	164	+ s, ok := Find(list, name)
	165	+ if !ok {
	166	+ return "", fmt.Errorf("no skill named %q", name)
	167	+ }
	168	+ content, err := os.ReadFile(s.Path)
120	169	return string(content), err
121	170	}
122	171
@@ -133,11 +182,26 @@ func Catalogue(list []Skill) string {
133	182	if s.Description != "" {
134	183	b.WriteString(" — " + s.Description)
135	184	}
	185	+ // The keywords are the lexical bridge between what a user types and
	186	+ // what a skill is called: "salue Bob" reaches `greetings` through
	187	+ // "saluer", never through the name. They were written in every
	188	+ // shipped skill and read by nobody until 2026-09-21.
	189	+ if kw := s.Keywords; len(kw) > 0 {
	190	+ if len(kw) > maxKeywords {
	191	+ kw = kw[:maxKeywords]
	192	+ }
	193	+ b.WriteString(" (" + strings.Join(kw, ", ") + ")")
	194	+ }
136	195	b.WriteString("\n")
137	196	}
138	197	return b.String()
139	198	}
140	199
	200	+// maxKeywords caps what one skill contributes to the catalogue. The catalogue
	201	+// is sent with EVERY request, so it is paid for on every turn: a handful of
	202	+// words per skill helps a small model match, thirty would crowd out the task.
	203	+const maxKeywords = 8
	204	+
141	205	// Names lists the skill names, for an error message that helps the model
142	206	// recover from a typo.
143	207	func Names(list []Skill) []string {

  package skills
  import (
++	"fmt"
  	"os"
  	"path/filepath"
  	"sort"
  type Skill struct {
  	Name        string // from the front matter, falling back to the file name
  	Description string // the one-liner shown in the catalogue
++	Keywords    []string
  	Path        string
  }
  	return strings.TrimSuffix(filepath.Base(path), ".md")
  }
--// parseHeader reads the `name:` and `description:` lines of the front matter.
++// parseHeader reads the `name:`, `description:` and `keywords:` lines of the
--// It stops at the closing `---`: a `description:` in the body is not metadata.
++// front matter. It stops at the closing `---`: a `description:` in the body is
++// not metadata.
  func parseHeader(content, path string) Skill {
  	s := Skill{Name: defaultName(path), Path: path}
  	lines := strings.Split(content, "\n")
  			}
  		case "description":
  			s.Description = value
++		case "keywords":
++			s.Keywords = splitKeywords(value)
  		}
  	}
  	return s
  }
++// splitKeywords turns "greet, hello, bonjour" into its words, dropping the
++// empties a trailing comma leaves behind.
++func splitKeywords(value string) []string {
++	var out []string
++	for _, w := range strings.Split(value, ",") {
++		if w = strings.TrimSpace(w); w != "" {
++			out = append(out, w)
++		}
++	}
++	return out
++}
++
  // List returns the skills of dir, sorted by name. Two layouts are accepted,
  // and may be mixed: a flat `<dir>/<name>.md`, and one directory per skill,
  // `<dir>/<name>/SKILL.md`. A missing directory is not an error: it just means
  		}
  		out = append(out, parseHeader(string(content), p))
  	}
--	sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
++	// STABLE, and the flat files were appended first: when two skills share a
++	// name, the flat `<name>.md` stays ahead of `<name>/SKILL.md` and is the
++	// one Find returns. That is the recorded rule — "the flat file wins" —
++	// and an unstable sort would decide it by coin toss.
++	sort.SliceStable(out, func(i, j int) bool { return out[i].Name < out[j].Name })
  	return out
  }
  	return strings.EqualFold(filepath.Base(path), "README.md")
  }
--// Read returns the full markdown of one skill, whichever layout it uses: the
++// Find returns the skill the CATALOGUE advertises under this name.
--// flat `<name>.md` is tried first, then `<name>/SKILL.md`.
++//
--func Read(dir, name string) (string, error) {
++// The lookup goes through the list, not through the filesystem, and that is
--	// Keep the name a plain file name: no "../" escaping out of the directory.
++// the whole point: `Catalogue` names a skill by its front-matter `name`,
--	name = filepath.Base(name)
++// while the old path-based read looked for `<dir>/<name>.md`. When the two
--	content, err := os.ReadFile(filepath.Join(dir, name+".md"))
++// disagreed — a skill whose `name:` is not its file name — the tool refused
--	if err != nil {
++// the very string the catalogue had told the model to use, with a message
--		if nested, nestedErr := os.ReadFile(filepath.Join(dir, name, skillFile)); nestedErr == nil {
++// that contradicted itself: `No skill named "renamed". Available: renamed`.
--			return string(nested), nil
++//
++// It is also strictly safer: the model's string is now only ever compared
++// against known names and never reaches the filesystem, so there is no path
++// to escape from.
++//
++// Matching is forgiving in the two ways a model gets it wrong and no human
++// would: surrounding whitespace or quotes, and the wrong case. It is never
++// forgiving about which skill: an exact name always wins over a case-
++// insensitive one.
++func Find(list []Skill, name string) (Skill, bool) {
++	name = strings.Trim(strings.TrimSpace(name), `"'`+"`")
++	for _, s := range list {
++		if s.Name == name {
++			return s, true
  		}
  	}
++	for _, s := range list {
++		if strings.EqualFold(s.Name, name) {
++			return s, true
++		}
++	}
++	return Skill{}, false
++}
++
++// Load reads the markdown of the skill the catalogue advertises under name.
++func Load(list []Skill, name string) (string, error) {
++	s, ok := Find(list, name)
++	if !ok {
++		return "", fmt.Errorf("no skill named %q", name)
++	}
++	content, err := os.ReadFile(s.Path)
  	return string(content), err
  }
  		if s.Description != "" {
  			b.WriteString(" — " + s.Description)
  		}
++		// The keywords are the lexical bridge between what a user types and
++		// what a skill is called: "salue Bob" reaches `greetings` through
++		// "saluer", never through the name. They were written in every
++		// shipped skill and read by nobody until 2026-09-21.
++		if kw := s.Keywords; len(kw) > 0 {
++			if len(kw) > maxKeywords {
++				kw = kw[:maxKeywords]
++			}
++			b.WriteString(" (" + strings.Join(kw, ", ") + ")")
++		}
  		b.WriteString("\n")
  	}
  	return b.String()
  }
++// maxKeywords caps what one skill contributes to the catalogue. The catalogue
++// is sent with EVERY request, so it is paid for on every turn: a handful of
++// words per skill helps a small model match, thirty would crowd out the task.
++const maxKeywords = 8
++
  // Names lists the skill names, for an error message that helps the model
  // recover from a typo.
  func Names(list []Skill) []string {

modified internal/skills/skills_test.go +87 -25 Viewed

@@ -95,41 +95,94 @@ func TestListMissingDirIsEmpty(t *testing.T) {
95	95	}
96	96	}
97	97
98		-func TestRead(t *testing.T) {
	98	+func TestLoad(t *testing.T) {
99	99	dir := t.TempDir()
100	100	write(t, dir, "go-fmt.md", "---\nname: go-fmt\n---\n# Format\n")
	101	+ list := List(dir)
101	102
102		- content, err := Read(dir, "go-fmt")
	103	+ content, err := Load(list, "go-fmt")
103	104	if err != nil {
104		- t.Fatalf("Read: %v", err)
	105	+ t.Fatalf("Load: %v", err)
105	106	}
106	107	if !strings.Contains(content, "# Format") {
107	108	t.Errorf("unexpected content: %q", content)
108	109	}
109		-
110		- if _, err := Read(dir, "unknown"); err == nil {
	110	+ if _, err := Load(list, "unknown"); err == nil {
111	111	t.Error("an unknown skill must return an error")
112	112	}
113	113	}
114	114
115		-// filepath.Base in Read is a barrier: the name comes from the MODEL, so it must
116		-// not be able to escape the skills directory.
117		-func TestReadCannotEscapeDir(t *testing.T) {
	115	+// The catalogue names a skill by its front-matter `name`, and loading must
	116	+// use the same name. The old path-based read did not: a skill whose `name:`
	117	+// differed from its file name was advertised and then refused, with a message
	118	+// that contradicted itself — `No skill named "renamed". Available: renamed`.
	119	+func TestLoadUsesTheCatalogueNameNotTheFileName(t *testing.T) {
118	120	dir := t.TempDir()
119		- parent := filepath.Dir(dir)
	121	+ write(t, dir, "01-rename.md", "---\nname: go-rename\ndescription: rename a symbol\n---\n# Rename\n")
	122	+ list := List(dir)
	123	+
	124	+ if len(list) != 1 \|\| list[0].Name != "go-rename" {
	125	+ t.Fatalf("catalogue = %+v, want one skill called go-rename", list)
	126	+ }
	127	+ if got, err := Load(list, "go-rename"); err != nil \|\| !strings.Contains(got, "# Rename") {
	128	+ t.Fatalf("the catalogue advertises go-rename and Load gives %q, %v", got, err)
	129	+ }
	130	+ if _, err := Load(list, "01-rename"); err == nil {
	131	+ t.Error("the file name is not the skill's name and must not resolve")
	132	+ }
	133	+}
	134	+
	135	+// A model gets a name almost right: stray spaces, quotes it should not have
	136	+// sent, the wrong case. None of those is a different skill.
	137	+func TestFindForgivesWhatOnlyAModelGetsWrong(t *testing.T) {
	138	+ dir := t.TempDir()
	139	+ write(t, dir, "go-rename.md", "---\nname: go-rename\n---\n#\n")
	140	+ list := List(dir)
	141	+
	142	+ for _, name := range []string{"go-rename", " go-rename ", `"go-rename"`, "'go-rename'", "Go-Rename"} {
	143	+ if _, ok := Find(list, name); !ok {
	144	+ t.Errorf("Find(%q) missed the skill", name)
	145	+ }
	146	+ }
	147	+ if _, ok := Find(list, "go-renam"); ok {
	148	+ t.Error("Find matched a name that is not the skill's")
	149	+ }
	150	+}
	151	+
	152	+// Forgiving about the spelling, never about WHICH skill: an exact name always
	153	+// beats a case-insensitive one.
	154	+func TestFindPrefersTheExactName(t *testing.T) {
	155	+ dir := t.TempDir()
	156	+ write(t, dir, "a.md", "---\nname: Deploy\n---\n# upper\n")
	157	+ write(t, dir, "b.md", "---\nname: deploy\n---\n# lower\n")
	158	+
	159	+ if got, ok := Find(List(dir), "deploy"); !ok \|\| got.Name != "deploy" {
	160	+ t.Errorf("Find(deploy) = %+v, %v — want the exact match", got, ok)
	161	+ }
	162	+}
	163	+
	164	+// The name now comes from the CATALOGUE, so it never reaches the filesystem
	165	+// at all. What used to need a filepath.Base barrier is now structural: a
	166	+// traversal string simply is not a name any skill has.
	167	+func TestALoadedNameNeverReachesTheFilesystem(t *testing.T) {
	168	+ parent := t.TempDir()
	169	+ dir := filepath.Join(parent, "skills")
	170	+ if err := os.MkdirAll(dir, 0o755); err != nil {
	171	+ t.Fatal(err)
	172	+ }
	173	+ write(t, dir, "real.md", "---\nname: real\n---\n# real\n")
120	174	if err := os.WriteFile(filepath.Join(parent, "secret.md"), []byte("forbidden"), 0o644); err != nil {
121	175	t.Fatal(err)
122	176	}
	177	+ list := List(dir)
123	178
124		- for _, name := range []string{"../secret", "../../secret", "/etc/passwd"} {
125		- if content, err := Read(dir, name); err == nil {
126		- t.Errorf("Read(%q) succeeded and returned %q — the barrier is gone", name, content)
	179	+ for _, name := range []string{"../secret", "../../secret", "/etc/passwd", "real/../../secret"} {
	180	+ if content, err := Load(list, name); err == nil {
	181	+ t.Errorf("Load(%q) succeeded and returned %q", name, content)
127	182	}
128	183	}
129	184	}
130	185
131		-// The catalogue IS the tool description: every skill must appear in it with its
132		-// name, otherwise the model cannot ask for it.
133	186	func TestCatalogue(t *testing.T) {
134	187	list := []Skill{
135	188	{Name: "go-fmt", Description: "format code"},
@@ -219,25 +272,34 @@ func TestListAcceptsFlatAndNestedLayouts(t *testing.T) {
219	272	// both exist (it is the more explicit of the two), and still cannot be walked
220	273	// out of the directory with a nested name: "../escape" is reduced to "escape",
221	274	// which does not exist inside dir.
222		-func TestReadAcceptsFlatAndNestedLayouts(t *testing.T) {
	275	+func TestLoadAcceptsFlatAndNestedLayouts(t *testing.T) {
223	276	parent := t.TempDir()
224	277	dir := filepath.Join(parent, "skills")
225	278	writeNested(t, parent, "escape", "# outside")
226	279	writeNested(t, dir, "nested-only", "# nested")
227	280	write(t, dir, "both.md", "# flat wins")
228	281	writeNested(t, dir, "both", "# nested loses")
	282	+ list := List(dir)
229	283
230		- if got, err := Read(dir, "nested-only"); err != nil \|\| got != "# nested" {
231		- t.Errorf("Read(nested-only) = %q, %v", got, err)
	284	+ if got, err := Load(list, "nested-only"); err != nil \|\| got != "# nested" {
	285	+ t.Errorf("Load(nested-only) = %q, %v", got, err)
	286	+ }
	287	+ // The recorded rule: when both layouts claim a name, the flat file wins.
	288	+ // List sorts STABLY over a slice whose flat entries come first, which is
	289	+ // what makes this deterministic — an unstable sort decided it by coin toss.
	290	+ if got, err := Load(list, "both"); err != nil \|\| got != "# flat wins" {
	291	+ t.Errorf("Load(both) = %q, %v; want the flat file", got, err)
232	292	}
233		- if got, err := Read(dir, "both"); err != nil \|\| got != "# flat wins" {
234		- t.Errorf("Read(both) = %q, %v; want the flat file", got, err)
	293	+ if _, err := Load(list, "missing"); err == nil {
	294	+ t.Error("Load(missing) returned no error")
235	295	}
236		- if _, err := Read(dir, "missing"); err == nil {
237		- t.Error("Read(missing) returned no error")
	296	+ if _, err := Load(list, "../escape"); err == nil {
	297	+ t.Error("Load(../escape) escaped the directory")
238	298	}
239		- if _, err := Read(dir, "../escape"); err == nil {
240		- t.Error("Read(../escape) escaped the directory")
	299	+ // Two skills under one name: reported, so the banner can say the second
	300	+ // one is unreachable.
	301	+ if dup := Duplicates(list); len(dup) != 1 \|\| dup[0] != "both" {
	302	+ t.Errorf("Duplicates = %v, want [both]", dup)
241	303	}
242	304	}
243	305
@@ -253,7 +315,7 @@ func TestListSkipsReadme(t *testing.T) {
253	315	if len(list) != 1 \|\| list[0].Name != "real" {
254	316	t.Errorf("List = %+v, want only the real skill", list)
255	317	}
256		- if _, err := Read(dir, "README"); err == nil {
257		- t.Log("Read(README) still opens the file: acceptable, only the catalogue hides it")
	318	+ if _, err := Load(list, "README"); err == nil {
	319	+ t.Error("README is not a skill and must not be loadable through the catalogue")
258	320	}
259	321	}

  	}
  }
--func TestRead(t *testing.T) {
++func TestLoad(t *testing.T) {
  	dir := t.TempDir()
  	write(t, dir, "go-fmt.md", "---\nname: go-fmt\n---\n# Format\n")
++	list := List(dir)
--	content, err := Read(dir, "go-fmt")
++	content, err := Load(list, "go-fmt")
  	if err != nil {
--		t.Fatalf("Read: %v", err)
++		t.Fatalf("Load: %v", err)
  	}
  	if !strings.Contains(content, "# Format") {
  		t.Errorf("unexpected content: %q", content)
  	}
--
++	if _, err := Load(list, "unknown"); err == nil {
--	if _, err := Read(dir, "unknown"); err == nil {
  		t.Error("an unknown skill must return an error")
  	}
  }
--// filepath.Base in Read is a barrier: the name comes from the MODEL, so it must
++// The catalogue names a skill by its front-matter `name`, and loading must
--// not be able to escape the skills directory.
++// use the same name. The old path-based read did not: a skill whose `name:`
--func TestReadCannotEscapeDir(t *testing.T) {
++// differed from its file name was advertised and then refused, with a message
++// that contradicted itself — `No skill named "renamed". Available: renamed`.
++func TestLoadUsesTheCatalogueNameNotTheFileName(t *testing.T) {
  	dir := t.TempDir()
--	parent := filepath.Dir(dir)
++	write(t, dir, "01-rename.md", "---\nname: go-rename\ndescription: rename a symbol\n---\n# Rename\n")
++	list := List(dir)
++
++	if len(list) != 1 || list[0].Name != "go-rename" {
++		t.Fatalf("catalogue = %+v, want one skill called go-rename", list)
++	}
++	if got, err := Load(list, "go-rename"); err != nil || !strings.Contains(got, "# Rename") {
++		t.Fatalf("the catalogue advertises go-rename and Load gives %q, %v", got, err)
++	}
++	if _, err := Load(list, "01-rename"); err == nil {
++		t.Error("the file name is not the skill's name and must not resolve")
++	}
++}
++
++// A model gets a name almost right: stray spaces, quotes it should not have
++// sent, the wrong case. None of those is a different skill.
++func TestFindForgivesWhatOnlyAModelGetsWrong(t *testing.T) {
++	dir := t.TempDir()
++	write(t, dir, "go-rename.md", "---\nname: go-rename\n---\n#\n")
++	list := List(dir)
++
++	for _, name := range []string{"go-rename", " go-rename ", `"go-rename"`, "'go-rename'", "Go-Rename"} {
++		if _, ok := Find(list, name); !ok {
++			t.Errorf("Find(%q) missed the skill", name)
++		}
++	}
++	if _, ok := Find(list, "go-renam"); ok {
++		t.Error("Find matched a name that is not the skill's")
++	}
++}
++
++// Forgiving about the spelling, never about WHICH skill: an exact name always
++// beats a case-insensitive one.
++func TestFindPrefersTheExactName(t *testing.T) {
++	dir := t.TempDir()
++	write(t, dir, "a.md", "---\nname: Deploy\n---\n# upper\n")
++	write(t, dir, "b.md", "---\nname: deploy\n---\n# lower\n")
++
++	if got, ok := Find(List(dir), "deploy"); !ok || got.Name != "deploy" {
++		t.Errorf("Find(deploy) = %+v, %v — want the exact match", got, ok)
++	}
++}
++
++// The name now comes from the CATALOGUE, so it never reaches the filesystem
++// at all. What used to need a filepath.Base barrier is now structural: a
++// traversal string simply is not a name any skill has.
++func TestALoadedNameNeverReachesTheFilesystem(t *testing.T) {
++	parent := t.TempDir()
++	dir := filepath.Join(parent, "skills")
++	if err := os.MkdirAll(dir, 0o755); err != nil {
++		t.Fatal(err)
++	}
++	write(t, dir, "real.md", "---\nname: real\n---\n# real\n")
  	if err := os.WriteFile(filepath.Join(parent, "secret.md"), []byte("forbidden"), 0o644); err != nil {
  		t.Fatal(err)
  	}
++	list := List(dir)
--	for _, name := range []string{"../secret", "../../secret", "/etc/passwd"} {
++	for _, name := range []string{"../secret", "../../secret", "/etc/passwd", "real/../../secret"} {
--		if content, err := Read(dir, name); err == nil {
++		if content, err := Load(list, name); err == nil {
--			t.Errorf("Read(%q) succeeded and returned %q — the barrier is gone", name, content)
++			t.Errorf("Load(%q) succeeded and returned %q", name, content)
  		}
  	}
  }
--// The catalogue IS the tool description: every skill must appear in it with its
--// name, otherwise the model cannot ask for it.
  func TestCatalogue(t *testing.T) {
  	list := []Skill{
  		{Name: "go-fmt", Description: "format code"},
  // both exist (it is the more explicit of the two), and still cannot be walked
  // out of the directory with a nested name: "../escape" is reduced to "escape",
  // which does not exist inside dir.
--func TestReadAcceptsFlatAndNestedLayouts(t *testing.T) {
++func TestLoadAcceptsFlatAndNestedLayouts(t *testing.T) {
  	parent := t.TempDir()
  	dir := filepath.Join(parent, "skills")
  	writeNested(t, parent, "escape", "# outside")
  	writeNested(t, dir, "nested-only", "# nested")
  	write(t, dir, "both.md", "# flat wins")
  	writeNested(t, dir, "both", "# nested loses")
++	list := List(dir)
--	if got, err := Read(dir, "nested-only"); err != nil || got != "# nested" {
++	if got, err := Load(list, "nested-only"); err != nil || got != "# nested" {
--		t.Errorf("Read(nested-only) = %q, %v", got, err)
++		t.Errorf("Load(nested-only) = %q, %v", got, err)
++	}
++	// The recorded rule: when both layouts claim a name, the flat file wins.
++	// List sorts STABLY over a slice whose flat entries come first, which is
++	// what makes this deterministic — an unstable sort decided it by coin toss.
++	if got, err := Load(list, "both"); err != nil || got != "# flat wins" {
++		t.Errorf("Load(both) = %q, %v; want the flat file", got, err)
  	}
--	if got, err := Read(dir, "both"); err != nil || got != "# flat wins" {
++	if _, err := Load(list, "missing"); err == nil {
--		t.Errorf("Read(both) = %q, %v; want the flat file", got, err)
++		t.Error("Load(missing) returned no error")
  	}
--	if _, err := Read(dir, "missing"); err == nil {
++	if _, err := Load(list, "../escape"); err == nil {
--		t.Error("Read(missing) returned no error")
++		t.Error("Load(../escape) escaped the directory")
  	}
--	if _, err := Read(dir, "../escape"); err == nil {
++	// Two skills under one name: reported, so the banner can say the second
--		t.Error("Read(../escape) escaped the directory")
++	// one is unreachable.
++	if dup := Duplicates(list); len(dup) != 1 || dup[0] != "both" {
++		t.Errorf("Duplicates = %v, want [both]", dup)
  	}
  }
  	if len(list) != 1 || list[0].Name != "real" {
  		t.Errorf("List = %+v, want only the real skill", list)
  	}
--	if _, err := Read(dir, "README"); err == nil {
++	if _, err := Load(list, "README"); err == nil {
--		t.Log("Read(README) still opens the file: acceptable, only the catalogue hides it")
++		t.Error("README is not a skill and must not be loadable through the catalogue")
  	}
  }

added internal/tools/skillmiss_test.go +78 -0 Viewed

new file mode 100644
@@ -0,0 +1,78 @@
	1	+package tools
	2	+
	3	+import (
	4	+ "testing"
	5	+
	6	+ "mm/internal/engine"
	7	+ "mm/internal/skills"
	8	+
	9	+ "github.com/firebase/genkit/go/ai"
	10	+)
	11	+
	12	+// This test exists because two packages agree on a string and nothing made
	13	+// them.
	14	+//
	15	+// `read_skill` answers a name that is not in the catalogue with a `[skill: …]`
	16	+// line, and engine.Skills counts a turn's LOADED skills by skipping exactly
	17	+// that prefix — otherwise `📖 2 skill(s)` can mean two failed lookups and
	18	+// nothing loaded, which is how it behaved before 2026-09-21.
	19	+//
	20	+// Reword the miss without touching the counter and the recap silently starts
	21	+// counting failures again. No compiler and no other test would notice, so
	22	+// this one asserts the round trip: what the tool produces is what the engine
	23	+// recognises.
	24	+//
	25	+// The import of engine is deliberately test-only — a production edge from
	26	+// every tool to the LLM connection would be a heavy price for one string.
	27	+func TestTheMissMessageIsWhatTheRecapSkips(t *testing.T) {
	28	+ list := []skills.Skill{
	29	+ {Name: "greetings", Description: "greet somebody"},
	30	+ {Name: "bg-jobs", Description: "background jobs"},
	31	+ }
	32	+
	33	+ history := func(output string) []*ai.Message {
	34	+ return []*ai.Message{{
	35	+ Role: ai.RoleTool,
	36	+ Content: []*ai.Part{ai.NewToolResponsePart(&ai.ToolResponse{
	37	+ Name: "read_skill",
	38	+ Output: output,
	39	+ })},
	40	+ }}
	41	+ }
	42	+
	43	+ miss := skillMiss("greetigns", list)
	44	+ if n := engine.Skills(history(miss)); n != 0 {
	45	+ t.Errorf("engine.Skills counted a miss as a loaded skill (%d).\n"+
	46	+ "The tool said: %q\n"+
	47	+ "Keep the [skill: …] shape, or update engine's marker with it.", n, miss)
	48	+ }
	49	+
	50	+ // A real skill, counted. Without this half the test would pass on a
	51	+ // counter that returns zero for everything.
	52	+ loaded := "---\nname: greetings\n---\n# Greet\nsay hello\n"
	53	+ if n := engine.Skills(history(loaded)); n != 1 {
	54	+ t.Errorf("engine.Skills = %d for a loaded skill, want 1", n)
	55	+ }
	56	+}
	57	+
	58	+// The message has to be usable by the model that receives it: what it asked
	59	+// for, and what it could have asked for instead.
	60	+func TestTheMissMessageTellsTheModelHowToRecover(t *testing.T) {
	61	+ list := []skills.Skill{{Name: "greetings"}, {Name: "bg-jobs"}}
	62	+ got := skillMiss("greetigns", list)
	63	+
	64	+ for _, want := range []string{"greetigns", "greetings", "bg-jobs"} {
	65	+ if !contains(got, want) {
	66	+ t.Errorf("the miss message does not mention %q: %s", want, got)
	67	+ }
	68	+ }
	69	+}
	70	+
	71	+func contains(s, sub string) bool {
	72	+ for i := 0; i+len(sub) <= len(s); i++ {
	73	+ if s[i:i+len(sub)] == sub {
	74	+ return true
	75	+ }
	76	+ }
	77	+ return false
	78	+}

++package tools
++
++import (
++	"testing"
++
++	"mm/internal/engine"
++	"mm/internal/skills"
++
++	"github.com/firebase/genkit/go/ai"
++)
++
++// This test exists because two packages agree on a string and nothing made
++// them.
++//
++// `read_skill` answers a name that is not in the catalogue with a `[skill: …]`
++// line, and engine.Skills counts a turn's LOADED skills by skipping exactly
++// that prefix — otherwise `📖 2 skill(s)` can mean two failed lookups and
++// nothing loaded, which is how it behaved before 2026-09-21.
++//
++// Reword the miss without touching the counter and the recap silently starts
++// counting failures again. No compiler and no other test would notice, so
++// this one asserts the round trip: what the tool produces is what the engine
++// recognises.
++//
++// The import of engine is deliberately test-only — a production edge from
++// every tool to the LLM connection would be a heavy price for one string.
++func TestTheMissMessageIsWhatTheRecapSkips(t *testing.T) {
++	list := []skills.Skill{
++		{Name: "greetings", Description: "greet somebody"},
++		{Name: "bg-jobs", Description: "background jobs"},
++	}
++
++	history := func(output string) []*ai.Message {
++		return []*ai.Message{{
++			Role: ai.RoleTool,
++			Content: []*ai.Part{ai.NewToolResponsePart(&ai.ToolResponse{
++				Name:   "read_skill",
++				Output: output,
++			})},
++		}}
++	}
++
++	miss := skillMiss("greetigns", list)
++	if n := engine.Skills(history(miss)); n != 0 {
++		t.Errorf("engine.Skills counted a miss as a loaded skill (%d).\n"+
++			"The tool said: %q\n"+
++			"Keep the [skill: …] shape, or update engine's marker with it.", n, miss)
++	}
++
++	// A real skill, counted. Without this half the test would pass on a
++	// counter that returns zero for everything.
++	loaded := "---\nname: greetings\n---\n# Greet\nsay hello\n"
++	if n := engine.Skills(history(loaded)); n != 1 {
++		t.Errorf("engine.Skills = %d for a loaded skill, want 1", n)
++	}
++}
++
++// The message has to be usable by the model that receives it: what it asked
++// for, and what it could have asked for instead.
++func TestTheMissMessageTellsTheModelHowToRecover(t *testing.T) {
++	list := []skills.Skill{{Name: "greetings"}, {Name: "bg-jobs"}}
++	got := skillMiss("greetigns", list)
++
++	for _, want := range []string{"greetigns", "greetings", "bg-jobs"} {
++		if !contains(got, want) {
++			t.Errorf("the miss message does not mention %q: %s", want, got)
++		}
++	}
++}
++
++func contains(s, sub string) bool {
++	for i := 0; i+len(sub) <= len(s); i++ {
++		if s[i:i+len(sub)] == sub {
++			return true
++		}
++	}
++	return false
++}

modified internal/tools/tools.go +15 -6 Viewed

@@ -323,7 +323,7 @@ func ReadSkill(g genkit.Genkit, dir string, d detector.LoopDetector) ai.ToolRe
323	323	printMu.Unlock()
324	324	})
325	325
326		- content, err := skills.Read(dir, in.Name)
	326	+ content, err := skills.Load(list, in.Name)
327	327	if err != nil {
328	328	// Unknown skill: information, not a crash. The list of valid
329	329	// names goes back AS TEXT so the model can correct itself.
@@ -331,8 +331,7 @@ func ReadSkill(g genkit.Genkit, dir string, d detector.LoopDetector) ai.ToolRe
331	331	// Recorded in the detector like everything else: a model
332	332	// asking four times for the same missing skill is looping,
333	333	// exactly as it would on a failing command.
334		- out := fmt.Sprintf("No skill named %q. Available skills: %s",
335		- in.Name, strings.Join(skills.Names(list), ", "))
	334	+ out := skillMiss(in.Name, list)
336	335	return record(d, "read_skill", in.Name, out)
337	336	}
338	337
@@ -345,6 +344,17 @@ func ReadSkill(g genkit.Genkit, dir string, d detector.LoopDetector) ai.ToolRe
345	344	})
346	345	}
347	346
	347	+// skillMiss is what a model gets when it asks for a skill that is not in the
	348	+// catalogue: what it asked for, and every name it could have asked for.
	349	+//
	350	+// The `[skill: …]` shape is the one the other tools use for an answer that is
	351	+// information rather than content (`[lsp: …]`, `[command rejected …]`), and
	352	+// the turn recap keys off it: a miss must not be counted as a skill loaded.
	353	+func skillMiss(name string, list []skills.Skill) string {
	354	+ return fmt.Sprintf("[skill: no skill named %q — available: %s]",
	355	+ name, strings.Join(skills.Names(list), ", "))
	356	+}
	357	+
348	358	// readSkillEvented is `read_skill` for an event-consuming front end. No
349	359	// permission dialog: it reads a procedure the agent ships, nothing more — and a
350	360	// dialog for every skill would teach the user to click "allow" without reading,
@@ -361,10 +371,9 @@ func readSkillEvented(s ui.Sink, d *detector.LoopDetector, dir string, list []sk
361	371	s.ToolStart(ev)
362	372	s.ToolRunning(ev.ID)
363	373
364		- content, err := skills.Read(dir, in.Name)
	374	+ content, err := skills.Load(list, in.Name)
365	375	if err != nil {
366		- out := fmt.Sprintf("No skill named %q. Available skills: %s",
367		- in.Name, strings.Join(skills.Names(list), ", "))
	376	+ out := skillMiss(in.Name, list)
368	377	s.ToolEnd(ev.ID, ui.ToolResult{Output: out, Failed: true})
369	378	return record(d, "read_skill", in.Name, out)
370	379	}

  				printMu.Unlock()
  			})
--			content, err := skills.Read(dir, in.Name)
++			content, err := skills.Load(list, in.Name)
  			if err != nil {
  				// Unknown skill: information, not a crash. The list of valid
  				// names goes back AS TEXT so the model can correct itself.
  				// Recorded in the detector like everything else: a model
  				// asking four times for the same missing skill is looping,
  				// exactly as it would on a failing command.
--				out := fmt.Sprintf("No skill named %q. Available skills: %s",
++				out := skillMiss(in.Name, list)
--					in.Name, strings.Join(skills.Names(list), ", "))
  				return record(d, "read_skill", in.Name, out)
  			}
  		})
  }
++// skillMiss is what a model gets when it asks for a skill that is not in the
++// catalogue: what it asked for, and every name it could have asked for.
++//
++// The `[skill: …]` shape is the one the other tools use for an answer that is
++// information rather than content (`[lsp: …]`, `[command rejected …]`), and
++// the turn recap keys off it: a miss must not be counted as a skill loaded.
++func skillMiss(name string, list []skills.Skill) string {
++	return fmt.Sprintf("[skill: no skill named %q — available: %s]",
++		name, strings.Join(skills.Names(list), ", "))
++}
++
  // readSkillEvented is `read_skill` for an event-consuming front end. No
  // permission dialog: it reads a procedure the agent ships, nothing more — and a
  // dialog for every skill would teach the user to click "allow" without reading,
  	s.ToolStart(ev)
  	s.ToolRunning(ev.ID)
--	content, err := skills.Read(dir, in.Name)
++	content, err := skills.Load(list, in.Name)
  	if err != nil {
--		out := fmt.Sprintf("No skill named %q. Available skills: %s",
++		out := skillMiss(in.Name, list)
--			in.Name, strings.Join(skills.Names(list), ", "))
  		s.ToolEnd(ev.ID, ui.ToolResult{Output: out, Failed: true})
  		return record(d, "read_skill", in.Name, out)
  	}

modified internal/tui/turn.go +5 -2 Viewed

@@ -305,8 +305,11 @@ func recap(turn []*ai.Message) string {
305	305	if ops := engine.FileOps(turn); len(ops) > 0 {
306	306	line += fmt.Sprintf(" · 📝 %d file op(s)", len(ops))
307	307	}
308		- if k := engine.Skills(turn); k > 0 {
309		- line += fmt.Sprintf(" · 📖 %d skill(s)", k)
	308	+ // Always shown when the agent has skills, 0 included: a zero here is the
	309	+ // signal that the model never reached for a procedure that may have
	310	+ // applied, and it is the only place that signal exists.
	311	+ if engine.SkillsAvailable() {
	312	+ line += fmt.Sprintf(" · 📖 %d skill(s)", engine.Skills(turn))
310	313	}
311	314	if k := engine.LSPCalls(turn); k > 0 {
312	315	line += fmt.Sprintf(" · 🔎 %d lsp call(s)", k)

  	if ops := engine.FileOps(turn); len(ops) > 0 {
  		line += fmt.Sprintf(" · 📝 %d file op(s)", len(ops))
  	}
--	if k := engine.Skills(turn); k > 0 {
++	// Always shown when the agent has skills, 0 included: a zero here is the
--		line += fmt.Sprintf(" · 📖 %d skill(s)", k)
++	// signal that the model never reached for a procedure that may have
++	// applied, and it is the only place that signal exists.
++	if engine.SkillsAvailable() {
++		line += fmt.Sprintf(" · 📖 %d skill(s)", engine.Skills(turn))
  	}
  	if k := engine.LSPCalls(turn); k > 0 {
  		line += fmt.Sprintf(" · 🔎 %d lsp call(s)", k)

modified main.go +33 -3 Viewed

@@ -191,8 +191,11 @@ type toolSet struct {
191	191	tools []ai.ToolRef
192	192	names []string
193	193	skillCount int
194		- warnings []string
195		- close func()
	194	+ // skills is the catalogue itself, kept so the banner can say what is
	195	+ // wrong with it and not only how many there are.
	196	+ skills []skills.Skill
	197	+ warnings []string
	198	+ close func()
196	199	}
197	200
198	201	// assembleTools declares the tools every front end gets: `bash` unless
@@ -210,7 +213,11 @@ func assembleTools(e engine.Engine, d detector.LoopDetector) toolSet {
210	213	set.tools = append(set.tools, tools.Bash(e.G, d))
211	214	set.names = append(set.names, "bash")
212	215	}
213		- set.skillCount = len(skills.List(config.Cfg.SkillsDir))
	216	+ set.skills = skills.List(config.Cfg.SkillsDir)
	217	+ set.skillCount = len(set.skills)
	218	+ // The turn recap shows `📖 0 skill(s)` only where a skill could have been
	219	+ // used — see engine.SkillsAvailable.
	220	+ engine.DeclareSkills(set.skillCount > 0)
214	221	if skillTool := tools.ReadSkill(e.G, config.Cfg.SkillsDir, d); skillTool != nil {
215	222	set.tools = append(set.tools, skillTool)
216	223	set.names = append(set.names, "read_skill")
@@ -281,6 +288,29 @@ func printBanner(ctx context.Context, e *engine.Engine, path string, set toolSet
281	288	info.Warnings = append(info.Warnings,
282	289	fmt.Sprintf("no skills found in %s (expected <name>.md or <name>/SKILL.md there)", config.Cfg.SkillsDir))
283	290	}
	291	+ // A skill with no description reaches the catalogue as a bare name, which
	292	+ // is the worst entry there is: the model has nothing to match a request
	293	+ // against, and the failure is silent — the count is not zero, so the
	294	+ // warning above never fires.
	295	+ if bare := skills.Undescribed(set.skills); len(bare) > 0 {
	296	+ info.Warnings = append(info.Warnings,
	297	+ fmt.Sprintf("%d skill(s) with no description (%s) — the model sees only the name and will pick blindly",
	298	+ len(bare), strings.Join(bare, ", ")))
	299	+ }
	300	+ // Only the first maxKeywords of a list reach the catalogue, so the order
	301	+ // of a keyword line decides what the model gets to match against. An
	302	+ // author cannot see that from their own file.
	303	+ if cut := skills.Truncated(set.skills); len(cut) > 0 {
	304	+ info.Warnings = append(info.Warnings,
	305	+ fmt.Sprintf("keywords truncated for %s — only the first ones reach the model, so put the most distinctive first",
	306	+ strings.Join(cut, ", ")))
	307	+ }
	308	+ // Two skills under one name: the catalogue shows it twice and the second
	309	+ // is unreachable, since a lookup always returns the first.
	310	+ if dup := skills.Duplicates(set.skills); len(dup) > 0 {
	311	+ info.Warnings = append(info.Warnings,
	312	+ fmt.Sprintf("two skills share a name (%s) — only the first is reachable", strings.Join(dup, ", ")))
	313	+ }
284	314	// No tool at all is a legal set-up — a chat with the model — but rarely the
285	315	// intended one; say so where the "tools:" column would otherwise be empty.
286	316	if len(toolNames) == 0 {

  	tools      []ai.ToolRef
  	names      []string
  	skillCount int
--	warnings   []string
++	// skills is the catalogue itself, kept so the banner can say what is
--	close      func()
++	// wrong with it and not only how many there are.
++	skills   []skills.Skill
++	warnings []string
++	close    func()
  }
  // assembleTools declares the tools every front end gets: `bash` unless
  		set.tools = append(set.tools, tools.Bash(e.G, d))
  		set.names = append(set.names, "bash")
  	}
--	set.skillCount = len(skills.List(config.Cfg.SkillsDir))
++	set.skills = skills.List(config.Cfg.SkillsDir)
++	set.skillCount = len(set.skills)
++	// The turn recap shows `📖 0 skill(s)` only where a skill could have been
++	// used — see engine.SkillsAvailable.
++	engine.DeclareSkills(set.skillCount > 0)
  	if skillTool := tools.ReadSkill(e.G, config.Cfg.SkillsDir, d); skillTool != nil {
  		set.tools = append(set.tools, skillTool)
  		set.names = append(set.names, "read_skill")
  		info.Warnings = append(info.Warnings,
  			fmt.Sprintf("no skills found in %s (expected <name>.md or <name>/SKILL.md there)", config.Cfg.SkillsDir))
  	}
++	// A skill with no description reaches the catalogue as a bare name, which
++	// is the worst entry there is: the model has nothing to match a request
++	// against, and the failure is silent — the count is not zero, so the
++	// warning above never fires.
++	if bare := skills.Undescribed(set.skills); len(bare) > 0 {
++		info.Warnings = append(info.Warnings,
++			fmt.Sprintf("%d skill(s) with no description (%s) — the model sees only the name and will pick blindly",
++				len(bare), strings.Join(bare, ", ")))
++	}
++	// Only the first maxKeywords of a list reach the catalogue, so the order
++	// of a keyword line decides what the model gets to match against. An
++	// author cannot see that from their own file.
++	if cut := skills.Truncated(set.skills); len(cut) > 0 {
++		info.Warnings = append(info.Warnings,
++			fmt.Sprintf("keywords truncated for %s — only the first ones reach the model, so put the most distinctive first",
++				strings.Join(cut, ", ")))
++	}
++	// Two skills under one name: the catalogue shows it twice and the second
++	// is unreachable, since a lookup always returns the first.
++	if dup := skills.Duplicates(set.skills); len(dup) > 0 {
++		info.Warnings = append(info.Warnings,
++			fmt.Sprintf("two skills share a name (%s) — only the first is reachable", strings.Join(dup, ", ")))
++	}
  	// No tool at all is a legal set-up — a chat with the model — but rarely the
  	// intended one; say so where the "tools:" column would otherwise be empty.
  	if len(toolNames) == 0 {

added skills/bg-jobs/SKILL.md +82 -0 Viewed

new file mode 100644
@@ -0,0 +1,82 @@
	1	+---
	2	+name: bg-jobs
	3	+description: Start, wait on, inspect and stop a long-running background job — a server, a watcher, a build that does not end — without blocking the turn.
	4	+keywords: background, job, server, serve, start, daemon, nohup, watch, tail, log, pid, kill, stop, port, running
	5	+---
	6	+
	7	+# Background jobs
	8	+
	9	+## When to use it
	10	+
	11	+Any command that serves, watches, or runs until stopped: an HTTP server, `npm run dev`, `go run` on a web app, a file watcher, a queue worker. Read this before you start one, and before you wait on, inspect or stop one.
	12	+
	13	+A command that never returns will hold the turn for its whole budget and then be killed. The answer the user asked for never arrives.
	14	+
	15	+## Start it
	16	+
	17	+Both streams redirected, the pid kept:
	18	+
	19	+```bash
	20	+nohup <command> > /tmp/<job>.log 2>&1 & echo $! > /tmp/<job>.pid
	21	+```
	22	+
	23	+Three parts, and all three matter:
	24	+
	25	+- `> /tmp/<job>.log 2>&1` — both streams. Redirecting only stdout is the classic mistake: the job keeps the tool's own error pipe open, and the call cannot finish until the job does. You get the 30-second budget spent on nothing.
	26	+- `&` — the shell returns at once.
	27	+- `echo $! > /tmp/<job>.pid` — without the pid you cannot stop it later except by guessing at `pkill` patterns, which also kills things you did not start.
	28	+
	29	+Use one name per job, and reuse it for the log and the pid file.
	30	+
	31	+## Wait for it to be ready
	32	+
	33	+Do not `sleep` and hope. Poll for the thing you actually need, with a bound:
	34	+
	35	+```bash
	36	+for i in $(seq 1 30); do
	37	+ curl -fsS localhost:8080/ >/dev/null 2>&1 && echo ready && break
	38	+ sleep 0.5
	39	+done
	40	+```
	41	+
	42	+For something that is not an HTTP server, poll the log for the line it prints when it is up:
	43	+
	44	+```bash
	45	+for i in $(seq 1 30); do
	46	+ grep -q "listening on" /tmp/<job>.log && echo ready && break
	47	+ sleep 0.5
	48	+done
	49	+```
	50	+
	51	+If the loop ends without `ready`, the job did not start — go read the log.
	52	+
	53	+## Inspect it
	54	+
	55	+```bash
	56	+tail -n 40 /tmp/<job>.log # what it said
	57	+kill -0 "$(cat /tmp/<job>.pid)" 2>/dev/null && echo alive \|\| echo dead
	58	+```
	59	+
	60	+`kill -0` sends no signal; it only asks whether the process is there. A job that is `dead` right after starting almost always says why in the last lines of its log — read them before changing anything.
	61	+
	62	+## Stop it
	63	+
	64	+```bash
	65	+kill "$(cat /tmp/<job>.pid)" 2>/dev/null
	66	+sleep 0.5
	67	+kill -0 "$(cat /tmp/<job>.pid)" 2>/dev/null && kill -9 "$(cat /tmp/<job>.pid)"
	68	+rm -f /tmp/<job>.pid
	69	+```
	70	+
	71	+Ask politely first (`kill` sends TERM, which lets a server close its listener), then insist. Removing the pid file afterwards stops a later command from acting on a pid the system has since reused for something else.
	72	+
	73	+## Before you finish
	74	+
	75	+Stop every job you started, and say which ones you left running and why. A server still holding port 8080 after the turn is a surprise the user will hit later, when nothing on screen explains it.
	76	+
	77	+## Traps
	78	+
	79	+- One shell per call. Each `bash` call is a new shell in the same directory: a job started in one call is still running in the next, but shell variables are gone. That is exactly why the pid goes to a file and not to `$PID`.
	80	+- The port is already in use. A previous run left something behind. `lsof -i :8080` or `ss -lptn 'sport = :8080'` names the holder; stop that before starting again, rather than moving to another port and forgetting.
	81	+- Nothing is interactive. There is no stdin. A command that asks a question will hang until it is killed; pass the answer as a flag, or do not run it.
	82	+- `nohup` writes `nohup.out` in the working directory when you forget to redirect — if you see that file appear, a redirect is missing.

++---
++name: bg-jobs
++description: Start, wait on, inspect and stop a long-running background job — a server, a watcher, a build that does not end — without blocking the turn.
++keywords: background, job, server, serve, start, daemon, nohup, watch, tail, log, pid, kill, stop, port, running
++---
++
++# Background jobs
++
++## When to use it
++
++Any command that **serves, watches, or runs until stopped**: an HTTP server, `npm run dev`, `go run` on a web app, a file watcher, a queue worker. Read this before you start one, and before you wait on, inspect or stop one.
++
++A command that never returns will hold the turn for its whole budget and then be killed. The answer the user asked for never arrives.
++
++## Start it
++
++Both streams redirected, the pid kept:
++
++```bash
++nohup <command> > /tmp/<job>.log 2>&1 & echo $! > /tmp/<job>.pid
++```
++
++Three parts, and all three matter:
++
++- **`> /tmp/<job>.log 2>&1`** — both streams. Redirecting only stdout is the classic mistake: the job keeps the tool's own error pipe open, and the call cannot finish until the job does. You get the 30-second budget spent on nothing.
++- **`&`** — the shell returns at once.
++- **`echo $! > /tmp/<job>.pid`** — without the pid you cannot stop it later except by guessing at `pkill` patterns, which also kills things you did not start.
++
++Use one name per job, and reuse it for the log and the pid file.
++
++## Wait for it to be ready
++
++**Do not `sleep` and hope.** Poll for the thing you actually need, with a bound:
++
++```bash
++for i in $(seq 1 30); do
++  curl -fsS localhost:8080/ >/dev/null 2>&1 && echo ready && break
++  sleep 0.5
++done
++```
++
++For something that is not an HTTP server, poll the log for the line it prints when it is up:
++
++```bash
++for i in $(seq 1 30); do
++  grep -q "listening on" /tmp/<job>.log && echo ready && break
++  sleep 0.5
++done
++```
++
++If the loop ends without `ready`, the job did not start — go read the log.
++
++## Inspect it
++
++```bash
++tail -n 40 /tmp/<job>.log          # what it said
++kill -0 "$(cat /tmp/<job>.pid)" 2>/dev/null && echo alive || echo dead
++```
++
++`kill -0` sends no signal; it only asks whether the process is there. A job that is `dead` right after starting almost always says why in the last lines of its log — read them before changing anything.
++
++## Stop it
++
++```bash
++kill "$(cat /tmp/<job>.pid)" 2>/dev/null
++sleep 0.5
++kill -0 "$(cat /tmp/<job>.pid)" 2>/dev/null && kill -9 "$(cat /tmp/<job>.pid)"
++rm -f /tmp/<job>.pid
++```
++
++Ask politely first (`kill` sends TERM, which lets a server close its listener), then insist. Removing the pid file afterwards stops a later command from acting on a pid the system has since reused for something else.
++
++## Before you finish
++
++**Stop every job you started**, and **say which ones you left running** and why. A server still holding port 8080 after the turn is a surprise the user will hit later, when nothing on screen explains it.
++
++## Traps
++
++- **One shell per call.** Each `bash` call is a new shell in the same directory: a job started in one call is still running in the next, but shell variables are gone. That is exactly why the pid goes to a file and not to `$PID`.
++- **The port is already in use.** A previous run left something behind. `lsof -i :8080` or `ss -lptn 'sport = :8080'` names the holder; stop that before starting again, rather than moving to another port and forgetting.
++- **Nothing is interactive.** There is no stdin. A command that asks a question will hang until it is killed; pass the answer as a flag, or do not run it.
++- **`nohup` writes `nohup.out`** in the working directory when you forget to redirect — if you see that file appear, a redirect is missing.

modified skills/greetings/SKILL.md +2 -4 Viewed

@@ -12,8 +12,6 @@ The command below is for YOU to run with your `bash` tool, in this same turn.
12	12	Do not show it to the user, and do not answer from memory: the answer is the
13	13	command's output, nothing else.
14	14
15		-Replace `<name>` with the person being greeted, then run:
	15	+Replace `<name>` with the person being greeted, then call your `bash` tool with this command:
16	16
17		-```bash
18		-echo "🎉 greetings <name>"
19		-```
	17	+ echo "🎉 greetings <name>"

  Do not show it to the user, and do not answer from memory: the answer is the
  command's output, nothing else.
--Replace `<name>` with the person being greeted, then run:
++Replace `<name>` with the person being greeted, then call your `bash` tool with this command:
--```bash
++    echo "🎉 greetings <name>"
--echo "🎉 greetings <name>"
--```

modified skills/vulcan-salute/SKILL.md +2 -4 Viewed

@@ -12,8 +12,6 @@ The command below is for YOU to run with your `bash` tool, in this same turn.
12	12	Do not show it to the user, and do not answer from memory: the answer is the
13	13	command's output, nothing else.
14	14
15		-Replace `<name>` with the person being saluted, the run:
	15	+Replace `<name>` with the person being saluted, then call your `bash` tool with this command:
16	16
17		-```bash
18		-echo "🖖 Live long and prosper! <name> 🖖"
19		-```
	17	+ echo "🖖 Live long and prosper! <name> 🖖"

  Do not show it to the user, and do not answer from memory: the answer is the
  command's output, nothing else.
--Replace `<name>` with the person being saluted, the run:
++Replace `<name>` with the person being saluted, then call your `bash` tool with this command:
--```bash
++    echo "🖖 Live long and prosper! <name> 🖖"
--echo "🖖 Live long and prosper! <name> 🖖"
--```

modified skills/what-time-is-it/SKILL.md +2 -4 Viewed

@@ -11,9 +11,7 @@ keywords: time, date, clock, now, today, heure, date du jour, quelle heure
11	11	The command below is for YOU to run with your `bash` tool, in this same turn.
12	12	Do not show it to the user, and never guess the time: only the command knows it.
13	13
14		-Run:
	14	+Call your `bash` tool with this command:
15	15
16		-```bash
17		-date "+%A %d %B %Y — %H:%M:%S (%Z)"
18		-```
	16	+ date "+%A %d %B %Y — %H:%M:%S (%Z)"
19	17

  The command below is for YOU to run with your `bash` tool, in this same turn.
  Do not show it to the user, and never guess the time: only the command knows it.
--Run:
++Call your `bash` tool with this command:
--```bash
++    date "+%A %d %B %Y — %H:%M:%S (%Z)"
--date "+%A %d %B %Y — %H:%M:%S (%Z)"
--```