rickub
bots-garden/mini-mepublic Fork
⑂ a8d612a
Branches
feature/agents-file feature/ask-agent-hooks feature/ask-cli feature/decider-no-tools feature/declared-tools feature/display-called-hooks feature/fetch-build-in feature/genkit-v1.13.1 feature/github-mcp-in-sbx feature/go-1.27.3 feature/hooks feature/improve-init feature/interactive-prompt feature/last-question feature/llmman-provider feature/log-errors feature/lsp-builtin feature/mcp-gateway-session feature/mcp-support feature/memory-search feature/mini-me-as-a-sdk feature/mini-mes feature/mm-init feature/open-terminal feature/prompt-flag feature/response-request feature/resume-acp feature/sbx-template-kit feature/search-in-doc feature/selectors feature/skill-discovery feature/structured-output feature/system-one feature/talk-to-another-mm feature/tasks-builtin feature/try-again feature/unload-model feature/wasm-custom-tools feature/yolo feature/zed-skill-selector fix/generation-timeout fix/plan-mode fix/too-long-completion fix/tool-order fix/word-wrap improve-mem-usage main thinking
Tags
v0.0.0 v0.0.1 v0.0.2 v0.0.3 v0.0.4 v0.1.0 v0.1.1 v0.1.2 v0.1.3 v0.1.4 v0.1.5 v0.1.6 v0.1.7 v0.1.8 v0.1.9 v0.2.0 v0.2.1 v0.2.2 v0.2.3 v0.2.4 v0.2.5 v0.2.6 v0.3.0 v0.4.0 v0.5.0 v0.6.0 v0.7.0 v0.8.0 v0.9.0 v1.0.0 v1.0.1 v1.0.2 v1.0.3 View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/bots-garden/mini-me.git
git clone ssh://git@rickub.com/bots-garden/mini-me.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Download ZIP

🛟 Updated.

k33g committed 2026-10-26T12:55:51+01:00 Browse files
a8d612a parent: f5170d0
modified .tickets/issues/0071-mini-me-as-a-sdk.yaml +10 -2
@@ -6,7 +6,7 @@ author:
66 name: k33g
77 email: ph.charriere@gmail.com
88 createdAt: 2026-10-02T05:32:33.878Z
9-updatedAt: 2026-10-22T17:07:15.939Z
9+updatedAt: 2026-10-26T05:00:00.000Z
1010 labels:
1111 - priority::urgent
1212 - kind::feature
@@ -103,4 +103,12 @@ body: |
103103 - **Stability promise**: `v0` and experimental until step 4 is done?
104104 - **Priority**: is headless/stream-json wanted first, as a quick win?
105105 tasks: []
106-comments: []
106+comments:
107+ - id: 1
108+ author:
109+ name: k33g
110+ email: ph.charriere@gmail.com
111+ createdAt: 2026-10-26T05:00:00.000Z
112+ body: |
113+ Update 2026-10-26: a smaller first version is proposed in ticket 139 — a public package with ONE method (`minime.Load` + `agent.Complete(ctx, prompt, minime.OnEvent(fn))`, streamed or not, cancelled by ctx, no STDIO), built as a thin façade over today's engine, the way ACP already uses it (`ui.SetActive(sink)` + `engine.Generate`). It would come BEFORE step 1 of this plan, with the same rules (own types, no Genkit in the API) so this plan can then remove the globals behind it without changing the API.
114+ Facts that changed since 2026-10-14: `config.Cfg` is now read in 92 files; the wiring in `main.go` has grown (autoPull, skill and tool routing, struggle signals, decision models), which is one more reason to move it out of `package main` first. `ui.Sink.Allow(ctx, call)` is where an SDK tool-confirmation callback would plug in.
@@ -6,7 +6,7 @@ author:
6 name: k33g6 name: k33g
7 email: ph.charriere@gmail.com7 email: ph.charriere@gmail.com
8 createdAt: 2026-10-02T05:32:33.878Z8 createdAt: 2026-10-02T05:32:33.878Z
9-updatedAt: 2026-10-22T17:07:15.939Z9+updatedAt: 2026-10-26T05:00:00.000Z
10 labels:10 labels:
11 - priority::urgent11 - priority::urgent
12 - kind::feature12 - kind::feature
@@ -103,4 +103,12 @@ body: |
103 - **Stability promise**: `v0` and experimental until step 4 is done?103 - **Stability promise**: `v0` and experimental until step 4 is done?
104 - **Priority**: is headless/stream-json wanted first, as a quick win?104 - **Priority**: is headless/stream-json wanted first, as a quick win?
105 tasks: []105 tasks: []
106-comments: []106+comments:
107+ - id: 1
108+ author:
109+ name: k33g
110+ email: ph.charriere@gmail.com
111+ createdAt: 2026-10-26T05:00:00.000Z
112+ body: |
113+ Update 2026-10-26: a smaller first version is proposed in ticket 139 — a public package with ONE method (`minime.Load` + `agent.Complete(ctx, prompt, minime.OnEvent(fn))`, streamed or not, cancelled by ctx, no STDIO), built as a thin façade over today's engine, the way ACP already uses it (`ui.SetActive(sink)` + `engine.Generate`). It would come BEFORE step 1 of this plan, with the same rules (own types, no Genkit in the API) so this plan can then remove the globals behind it without changing the API.
114+ Facts that changed since 2026-10-14: `config.Cfg` is now read in 92 files; the wiring in `main.go` has grown (autoPull, skill and tool routing, struggle signals, decision models), which is one more reason to move it out of `package main` first. `ui.Sink.Allow(ctx, call)` is where an SDK tool-confirmation callback would plug in.
modified .tickets/issues/0105-mini-me-as-a-sdk-2.yaml +9 -2
@@ -6,7 +6,7 @@ author:
66 name: k33g
77 email: ph.charriere@gmail.com
88 createdAt: 2026-10-20T07:51:18.410Z
9-updatedAt: 2026-10-22T17:07:42.565Z
9+updatedAt: 2026-10-26T05:00:00.000Z
1010 labels: []
1111 epic: SDK
1212 related:
@@ -14,4 +14,11 @@ related:
1414 body: |
1515 Check how it's possible to create a specific package that "exposes" the various methods
1616 tasks: []
17-comments: []
17+comments:
18+ - id: 1
19+ author:
20+ name: k33g
21+ email: ph.charriere@gmail.com
22+ createdAt: 2026-10-26T05:00:00.000Z
23+ body: |
24+ Answer 2026-10-26: yes, it is possible — Go forbids importing `internal/` from another module, so the package that exposes the methods must sit outside `internal/` (proposed: `minime/`) and the module needs an importable path (`rickub.com/bots-garden/mini-me`, go-get on Rickub to verify). The agent wiring now in `package main` would move to an internal package used by both the binary and this public package. The full proposal, with its first single method (`Complete`), is in ticket 139.
@@ -6,7 +6,7 @@ author:
6 name: k33g6 name: k33g
7 email: ph.charriere@gmail.com7 email: ph.charriere@gmail.com
8 createdAt: 2026-10-20T07:51:18.410Z8 createdAt: 2026-10-20T07:51:18.410Z
9-updatedAt: 2026-10-22T17:07:42.565Z9+updatedAt: 2026-10-26T05:00:00.000Z
10 labels: []10 labels: []
11 epic: SDK11 epic: SDK
12 related:12 related:
@@ -14,4 +14,11 @@ related:
14 body: |14 body: |
15 Check how it's possible to create a specific package that "exposes" the various methods15 Check how it's possible to create a specific package that "exposes" the various methods
16 tasks: []16 tasks: []
17-comments: []17+comments:
18+ - id: 1
19+ author:
20+ name: k33g
21+ email: ph.charriere@gmail.com
22+ createdAt: 2026-10-26T05:00:00.000Z
23+ body: |
24+ Answer 2026-10-26: yes, it is possible — Go forbids importing `internal/` from another module, so the package that exposes the methods must sit outside `internal/` (proposed: `minime/`) and the module needs an importable path (`rickub.com/bots-garden/mini-me`, go-get on Rickub to verify). The agent wiring now in `package main` would move to an internal package used by both the binary and this public package. The full proposal, with its first single method (`Complete`), is in ticket 139.
modified .tickets/issues/0139-some-api-ideas.yaml +134 -3
@@ -6,13 +6,144 @@ author:
66 name: k33g
77 email: ph.charriere@gmail.com
88 createdAt: 2026-10-23T09:54:13.140Z
9-updatedAt: 2026-10-23T09:57:11.218Z
10-labels: []
9+updatedAt: 2026-10-26T05:00:00.000Z
10+labels:
11+ - kind::feature
1112 epic: SDK
13+related:
14+ - 71
15+ - 105
16+ - 113
1217 body: |
1318 il faudrait par exemple créer une lib avec une méthode par exemple `prompt`
1419 (equivalent de mm -p "hello this is my question"
1520
1621 et on aurait la possibilité via le pattern go de passer les éléments necessaires (les reglages du fichiers de config, des tools, ...)
17-tasks: []
22+
23+ ---
24+
25+ ## Request (2026-10-26)
26+ A first version of mini-me as a library with ONE method: from an agent configuration, run a completion, get the result streamed or not, and be able to cancel it. No STDIO: a Go program must not need the `mm` binary installed. In the end, mini-me should be usable both as an agent (the binary) and as a library. Nothing implemented yet: this section records the analysis and the proposals.
27+
28+ ## Proposed API (the same in every proposal below)
29+ ```go
30+ agent, err := minime.Load(ctx, "agent.yaml") // builds the agent, runs nothing
31+
32+ // not streamed
33+ res, err := agent.Complete(ctx, "Explain this repository")
34+ fmt.Println(res.Text)
35+
36+ // streamed: an option on the same method
37+ res, err := agent.Complete(ctx, "Explain this repository",
38+ minime.OnEvent(func(ev minime.Event) { // Text, Thinking, ToolCall, ToolResult, Notice, Struggle
39+ if ev.Kind == minime.Text { fmt.Print(ev.Text) }
40+ }))
41+
42+ // cancel: the Go idiom, the context
43+ ctx, cancel := context.WithCancel(ctx)
44+ ```
45+ - One method, `Complete`. `Load` only builds the agent once (reading the YAML, starting MCP / LSP servers on every call would be wasteful).
46+ - The public types belong to `minime`, never to Genkit or `internal/` (same rule as ticket 71: Genkit changes, ticket 76).
47+
48+ ## What blocks an import today (checked 2026-10-26)
49+ | Obstacle | Where | Consequence |
50+ |---|---|---|
51+ | The module is named `mm` | `go.mod` | not go-gettable; would become `rickub.com/bots-garden/mini-me` — whether Rickub serves the `?go-get=1` meta tag is NOT verified |
52+ | Every package is under `internal/` | 35 packages | another module cannot import them: a public package is needed |
53+ | The agent's wiring is in `package main` | `main.go`: `assembleTools`, `autoPull`, skill/tool routing, hooks, struggle signals | the SDK cannot reuse it; copying it would make SDK and binary drift |
54+ | Global state | `config.Cfg` (read in 92 files), `ui.Out`, `ui.SetActive`, `skills.Declare`, the spinner | one agent per process, one completion at a time |
55+ | Tool confirmation is a terminal question | `yolo: false` | a library has no terminal: another rule is needed |
56+
57+ ## What already fits
58+ The editor mode (ACP) already does what the SDK needs, without STDIO for the answer itself (`internal/acp/acp.go` ~413–451): `ui.SetActive(sink)` then `engine.Generate(ctx, messages, tools)`.
59+ - `engine.New(ctx)` opens the provider of the config (llmman, Ollama, DMR…).
60+ - `engine.Generate` runs the whole turn (model calls, tools, hooks, watchdog, loop detector, routing) and returns the final answer plus the whole conversation.
61+ - `ui.Sink` receives the stream: `Text`, `Reasoning`, `ToolStart` / `ToolRunning` / `ToolEnd`, and `Allow(ctx, call) bool` — the place for a tool-confirmation callback.
62+ - Cancelling the `ctx` already stops the turn, a running `bash` included (Esc in Zed works that way).
63+
64+ ## Proposals
65+ - **A — Thin façade over the existing engine (fastest).** A public package `minime/` calls the engine as ACP does. The wiring of `main.go` moves to an internal package (`internal/assemble`), called by both `main` and the SDK. Documented limits: one agent loaded per process (`config.Load` fills the global `config.Cfg`), completions serialised under a lock (`ui.SetActive` is ONE sink for the process). Moderate cost, low risk.
66+ - **B — Per-instance core (the real library).** Replace the global state with an `Agent` value carrying its config, sink and engine; terminal, window and editor become its clients. Several agents and completions in parallel. High cost (at least the 92 files reading `config.Cfg`), regression risk everywhere. This is step 1 of ticket 71.
67+ - **C — A first, then B behind the same API (recommended).** v1 = the façade; since its API exposes nothing internal, the globals can then be removed package by package without changing one SDK signature, and the "one agent per process" rule disappears without breaking users.
68+ - **D — `mm serve`, HTTP + SSE (later, for other languages).** The same `Complete` behind an HTTP server for Python / JavaScript. Needs `mm` running somewhere (a container is enough), so it does not meet "no binary" for Go — a complement, not v1. See ticket 113.
69+
70+ Sketch of the façade (proposal A / C):
71+ ```go
72+ func Load(ctx context.Context, path string) (*Agent, error) {
73+ config.Load(path) // fills config.Cfg (global)
74+ e, _ := engine.New(ctx)
75+ tools := assemble.Tools(e, …) // today in main.go → moved under internal/
76+ return &Agent{e: e, tools: tools, history: []*ai.Message{system}}, nil
77+ }
78+
79+ func (a *Agent) Complete(ctx context.Context, prompt string, opts ...Option) (Result, error) {
80+ mu.Lock(); defer mu.Unlock() // one turn at a time: the sink is global
81+ ui.SetActive(eventSink{onEvent}) // turns Text/ToolStart/… into minime.Event
82+ defer ui.SetActive(nil)
83+ resp, full, err := a.e.Generate(ctx, append(a.history, user(prompt)), a.tools)
84+ a.history = full
85+ return Result{Text: resp.Text()}, err
86+ }
87+ ```
88+ Not streamed = the same path: without `OnEvent` the sink drops the chunks and `Complete` returns the final text.
89+
90+ ## Traps to handle
91+ - The engine still writes some lines straight to `ui.Out` (the spinner, "stream cut — retrying this model call once"…): they must go to the sink or be off in SDK mode — a library never writes to its host's stdout.
92+ - Renaming the module rewrites every import AND the `-X mm/internal/version.stamped` ldflag in `02-build-releases.sh`.
93+ - `ask_user` and the `/ask` hand-offs (`mmConfigs` starts child `mm -acp` processes) need the binary: out of v1, or refused with a clear error.
94+
95+ ## Decisions for the user
96+ 1. Proposal: A, B, C (recommended) or D first.
97+ 2. Does `Complete` keep the conversation between calls (like the REPL, recommended), or is each call isolated (like `mm -p`)?
98+ 3. Tool confirmation with `yolo: false`: a callback (`OnApprove`), refusing when there is none (recommended), or the YAML only (`yolo: false` refuses whatever would ask)?
99+ 4. Module path and package name: `rickub.com/bots-garden/mini-me` + `minime/`?
100+tasks:
101+ - id: 1
102+ title: Decide the proposal, the history and the tool-confirmation rule
103+ state: open
104+ priority: none
105+ author:
106+ name: k33g
107+ email: ph.charriere@gmail.com
108+ createdAt: 2026-10-26T05:00:00.000Z
109+ - id: 2
110+ title: Make the module importable (module path, go-get on Rickub)
111+ state: open
112+ priority: none
113+ author:
114+ name: k33g
115+ email: ph.charriere@gmail.com
116+ createdAt: 2026-10-26T05:00:00.000Z
117+ - id: 3
118+ title: Move the agent wiring out of package main into an internal package
119+ state: open
120+ priority: none
121+ author:
122+ name: k33g
123+ email: ph.charriere@gmail.com
124+ createdAt: 2026-10-26T05:00:00.000Z
125+ - id: 4
126+ title: Public package with Load + Complete (stream option, ctx cancel), unit tests and Example tests
127+ state: open
128+ priority: none
129+ author:
130+ name: k33g
131+ email: ph.charriere@gmail.com
132+ createdAt: 2026-10-26T05:00:00.000Z
133+ - id: 5
134+ title: Engine output that bypasses the sink goes to events (no write to the host's stdout)
135+ state: open
136+ priority: none
137+ author:
138+ name: k33g
139+ email: ph.charriere@gmail.com
140+ createdAt: 2026-10-26T05:00:00.000Z
141+ - id: 6
142+ title: Docs EN + FR (how-to embed mini-me in a Go program, API reference)
143+ state: open
144+ priority: none
145+ author:
146+ name: k33g
147+ email: ph.charriere@gmail.com
148+ createdAt: 2026-10-26T05:00:00.000Z
18149 comments: []
@@ -6,13 +6,144 @@ author:
6 name: k33g6 name: k33g
7 email: ph.charriere@gmail.com7 email: ph.charriere@gmail.com
8 createdAt: 2026-10-23T09:54:13.140Z8 createdAt: 2026-10-23T09:54:13.140Z
9-updatedAt: 2026-10-23T09:57:11.218Z9+updatedAt: 2026-10-26T05:00:00.000Z
10-labels: []10+labels:
11+ - kind::feature
11 epic: SDK12 epic: SDK
13+related:
14+ - 71
15+ - 105
16+ - 113
12 body: |17 body: |
13 il faudrait par exemple créer une lib avec une méthode par exemple `prompt`18 il faudrait par exemple créer une lib avec une méthode par exemple `prompt`
14 (equivalent de mm -p "hello this is my question"19 (equivalent de mm -p "hello this is my question"
15 20
16 et on aurait la possibilité via le pattern go de passer les éléments necessaires (les reglages du fichiers de config, des tools, ...) 21 et on aurait la possibilité via le pattern go de passer les éléments necessaires (les reglages du fichiers de config, des tools, ...)
17-tasks: []22+
23+ ---
24+
25+ ## Request (2026-10-26)
26+ A first version of mini-me as a library with ONE method: from an agent configuration, run a completion, get the result streamed or not, and be able to cancel it. No STDIO: a Go program must not need the `mm` binary installed. In the end, mini-me should be usable both as an agent (the binary) and as a library. Nothing implemented yet: this section records the analysis and the proposals.
27+
28+ ## Proposed API (the same in every proposal below)
29+ ```go
30+ agent, err := minime.Load(ctx, "agent.yaml") // builds the agent, runs nothing
31+
32+ // not streamed
33+ res, err := agent.Complete(ctx, "Explain this repository")
34+ fmt.Println(res.Text)
35+
36+ // streamed: an option on the same method
37+ res, err := agent.Complete(ctx, "Explain this repository",
38+ minime.OnEvent(func(ev minime.Event) { // Text, Thinking, ToolCall, ToolResult, Notice, Struggle
39+ if ev.Kind == minime.Text { fmt.Print(ev.Text) }
40+ }))
41+
42+ // cancel: the Go idiom, the context
43+ ctx, cancel := context.WithCancel(ctx)
44+ ```
45+ - One method, `Complete`. `Load` only builds the agent once (reading the YAML, starting MCP / LSP servers on every call would be wasteful).
46+ - The public types belong to `minime`, never to Genkit or `internal/` (same rule as ticket 71: Genkit changes, ticket 76).
47+
48+ ## What blocks an import today (checked 2026-10-26)
49+ | Obstacle | Where | Consequence |
50+ |---|---|---|
51+ | The module is named `mm` | `go.mod` | not go-gettable; would become `rickub.com/bots-garden/mini-me` — whether Rickub serves the `?go-get=1` meta tag is NOT verified |
52+ | Every package is under `internal/` | 35 packages | another module cannot import them: a public package is needed |
53+ | The agent's wiring is in `package main` | `main.go`: `assembleTools`, `autoPull`, skill/tool routing, hooks, struggle signals | the SDK cannot reuse it; copying it would make SDK and binary drift |
54+ | Global state | `config.Cfg` (read in 92 files), `ui.Out`, `ui.SetActive`, `skills.Declare`, the spinner | one agent per process, one completion at a time |
55+ | Tool confirmation is a terminal question | `yolo: false` | a library has no terminal: another rule is needed |
56+
57+ ## What already fits
58+ The editor mode (ACP) already does what the SDK needs, without STDIO for the answer itself (`internal/acp/acp.go` ~413–451): `ui.SetActive(sink)` then `engine.Generate(ctx, messages, tools)`.
59+ - `engine.New(ctx)` opens the provider of the config (llmman, Ollama, DMR…).
60+ - `engine.Generate` runs the whole turn (model calls, tools, hooks, watchdog, loop detector, routing) and returns the final answer plus the whole conversation.
61+ - `ui.Sink` receives the stream: `Text`, `Reasoning`, `ToolStart` / `ToolRunning` / `ToolEnd`, and `Allow(ctx, call) bool` — the place for a tool-confirmation callback.
62+ - Cancelling the `ctx` already stops the turn, a running `bash` included (Esc in Zed works that way).
63+
64+ ## Proposals
65+ - **A — Thin façade over the existing engine (fastest).** A public package `minime/` calls the engine as ACP does. The wiring of `main.go` moves to an internal package (`internal/assemble`), called by both `main` and the SDK. Documented limits: one agent loaded per process (`config.Load` fills the global `config.Cfg`), completions serialised under a lock (`ui.SetActive` is ONE sink for the process). Moderate cost, low risk.
66+ - **B — Per-instance core (the real library).** Replace the global state with an `Agent` value carrying its config, sink and engine; terminal, window and editor become its clients. Several agents and completions in parallel. High cost (at least the 92 files reading `config.Cfg`), regression risk everywhere. This is step 1 of ticket 71.
67+ - **C — A first, then B behind the same API (recommended).** v1 = the façade; since its API exposes nothing internal, the globals can then be removed package by package without changing one SDK signature, and the "one agent per process" rule disappears without breaking users.
68+ - **D — `mm serve`, HTTP + SSE (later, for other languages).** The same `Complete` behind an HTTP server for Python / JavaScript. Needs `mm` running somewhere (a container is enough), so it does not meet "no binary" for Go — a complement, not v1. See ticket 113.
69+
70+ Sketch of the façade (proposal A / C):
71+ ```go
72+ func Load(ctx context.Context, path string) (*Agent, error) {
73+ config.Load(path) // fills config.Cfg (global)
74+ e, _ := engine.New(ctx)
75+ tools := assemble.Tools(e, …) // today in main.go → moved under internal/
76+ return &Agent{e: e, tools: tools, history: []*ai.Message{system}}, nil
77+ }
78+
79+ func (a *Agent) Complete(ctx context.Context, prompt string, opts ...Option) (Result, error) {
80+ mu.Lock(); defer mu.Unlock() // one turn at a time: the sink is global
81+ ui.SetActive(eventSink{onEvent}) // turns Text/ToolStart/… into minime.Event
82+ defer ui.SetActive(nil)
83+ resp, full, err := a.e.Generate(ctx, append(a.history, user(prompt)), a.tools)
84+ a.history = full
85+ return Result{Text: resp.Text()}, err
86+ }
87+ ```
88+ Not streamed = the same path: without `OnEvent` the sink drops the chunks and `Complete` returns the final text.
89+
90+ ## Traps to handle
91+ - The engine still writes some lines straight to `ui.Out` (the spinner, "stream cut — retrying this model call once"…): they must go to the sink or be off in SDK mode — a library never writes to its host's stdout.
92+ - Renaming the module rewrites every import AND the `-X mm/internal/version.stamped` ldflag in `02-build-releases.sh`.
93+ - `ask_user` and the `/ask` hand-offs (`mmConfigs` starts child `mm -acp` processes) need the binary: out of v1, or refused with a clear error.
94+
95+ ## Decisions for the user
96+ 1. Proposal: A, B, C (recommended) or D first.
97+ 2. Does `Complete` keep the conversation between calls (like the REPL, recommended), or is each call isolated (like `mm -p`)?
98+ 3. Tool confirmation with `yolo: false`: a callback (`OnApprove`), refusing when there is none (recommended), or the YAML only (`yolo: false` refuses whatever would ask)?
99+ 4. Module path and package name: `rickub.com/bots-garden/mini-me` + `minime/`?
100+tasks:
101+ - id: 1
102+ title: Decide the proposal, the history and the tool-confirmation rule
103+ state: open
104+ priority: none
105+ author:
106+ name: k33g
107+ email: ph.charriere@gmail.com
108+ createdAt: 2026-10-26T05:00:00.000Z
109+ - id: 2
110+ title: Make the module importable (module path, go-get on Rickub)
111+ state: open
112+ priority: none
113+ author:
114+ name: k33g
115+ email: ph.charriere@gmail.com
116+ createdAt: 2026-10-26T05:00:00.000Z
117+ - id: 3
118+ title: Move the agent wiring out of package main into an internal package
119+ state: open
120+ priority: none
121+ author:
122+ name: k33g
123+ email: ph.charriere@gmail.com
124+ createdAt: 2026-10-26T05:00:00.000Z
125+ - id: 4
126+ title: Public package with Load + Complete (stream option, ctx cancel), unit tests and Example tests
127+ state: open
128+ priority: none
129+ author:
130+ name: k33g
131+ email: ph.charriere@gmail.com
132+ createdAt: 2026-10-26T05:00:00.000Z
133+ - id: 5
134+ title: Engine output that bypasses the sink goes to events (no write to the host's stdout)
135+ state: open
136+ priority: none
137+ author:
138+ name: k33g
139+ email: ph.charriere@gmail.com
140+ createdAt: 2026-10-26T05:00:00.000Z
141+ - id: 6
142+ title: Docs EN + FR (how-to embed mini-me in a Go program, API reference)
143+ state: open
144+ priority: none
145+ author:
146+ name: k33g
147+ email: ph.charriere@gmail.com
148+ createdAt: 2026-10-26T05:00:00.000Z
18 comments: []149 comments: []
added .tickets/issues/0167-node-js-client-over-acp-a-first-sdk-prototype-fo.yaml +136 -0
new file mode 100644
@@ -0,0 +1,136 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 167
3+title: "Node.js client over ACP: a first SDK prototype for other languages"
4+state: closed
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T06:00:00.000Z
9+updatedAt: 2026-10-26T11:00:00.000Z
10+labels:
11+ - kind::feature
12+milestone: v1.0.2
13+epic: SDK
14+related:
15+ - 139
16+ - 71
17+ - 113
18+ - 168
19+ - 169
20+ - 170
21+ - 171
22+ - 172
23+body: |
24+ ## Goal
25+ Prove that mm can be used as a library from another language by driving `mm -acp` over ACP (JSON-RPC on stdin/stdout), with NO change to mm. Node.js first (user's choice, 2026-10-26); Python later if the prototype convinces. Prototype only: one demo folder, not a published package.
26+
27+ This is the "other languages" path. It complements the Go SDK of ticket 139 (in-process, no binary) and does not replace it. Remote use (an agent in another machine or container) stays for `mm serve` (ticket 113 / proposal D of 139).
28+
29+ ## Why ACP fits (checked in internal/acp and docs/en/reference/acp.md on 2026-10-26)
30+ | Wanted | ACP, already in mm |
31+ |---|---|
32+ | run a completion | `session/prompt` |
33+ | stream | `session/update` notifications (text chunks, thoughts, tool calls, plan) |
34+ | not streamed | join the text chunks until `session/prompt` answers (its `stopReason`) |
35+ | cancel | `session/cancel` |
36+ | tool confirmation (`yolo: false`) | mm sends `session/request_permission` to the client → a callback |
37+ | `ask_user` | `elicitation/create`, only when the client advertises `clientCapabilities.elicitation.form` at `initialize` |
38+ | keep the conversation | one ACP session = one history; `session/load` / `session/resume` with `sessions.enabled` |
39+
40+ - mm asks nothing else of the client: it reads files and runs commands itself, so the client does not have to implement `fs/*` or `terminal/*`.
41+ - One agent = one process: several agents run side by side, the global state of the Go code (`config.Cfg`, `ui.SetActive`) is no problem here.
42+ - Everything mm does works unchanged: hooks, MCP, `mmConfigs`, skill/tool routing, struggle signals.
43+
44+ ## Proposed shape (to discuss in the prototype)
45+ ```js
46+ import { Agent } from "./minime.js";
47+
48+ const agent = await Agent.start({
49+ config: "agent.yaml", // passed as AGENT_CONFIG (absolute path)
50+ cwd: process.cwd(), // the session's cwd (session/new)
51+ onPermission: (call) => call.kind === "read", // no callback → refuse
52+ });
53+
54+ const text = await agent.complete("Explain this repository"); // not streamed
55+
56+ for await (const ev of agent.stream("And the tests?")) { // streamed
57+ if (ev.kind === "text") process.stdout.write(ev.text);
58+ }
59+
60+ const ac = new AbortController();
61+ setTimeout(() => ac.abort(), 5000);
62+ await agent.complete("A long task", { signal: ac.signal }); // abort → session/cancel
63+
64+ await agent.close(); // closes stdin, mm exits
65+ ```
66+ - Plain Node.js (built-in `child_process`, `readline`), no dependency, ESM, Node 22 (the sandbox has v22.22.1).
67+ - Events reduced to a few kinds: `text`, `thinking`, `tool_call`, `tool_result`, `done` — not the editor-oriented detail of ACP (diffs, statuses).
68+ - The `mm` binary is found on the PATH or through `MM_BIN` (the convention of the demo hooks).
69+ - stderr of mm (banner, warnings, `[acp]` lines) is drained, optionally passed to a callback, never mixed with the answer.
70+ - Check first whether the official ACP TypeScript library (`@agentclientprotocol/sdk` or similar — not verified, no network in the sandbox) is worth using instead of raw JSON-RPC; raw JSON-RPC keeps the prototype dependency-free.
71+
72+ ## Limits to state in the README
73+ - The `mm` binary must be installed (a later package could ship it per platform, as some npm packages do for native tools).
74+ - Local only: stdio.
75+ - The config is a file path (AGENT_CONFIG); no in-memory config.
76+
77+ ## Done when
78+ - `demo.acp.sdk/nodejs/` (location chosen by the user on 2026-10-26, instead of `demos/node-acp-client/`) holds `minime.js` (the client), one example each for not streamed, streamed and cancelled, and README EN + FR with the results measured on llmman.
79+ - Permissions: a refused and an accepted tool call shown with an agent with `yolo: false`.
80+ - Nothing changed in mm's Go code.
81+tasks:
82+ - id: 1
83+ title: Check the official ACP TypeScript library vs raw JSON-RPC; decide
84+ state: closed
85+ priority: none
86+ author:
87+ name: k33g
88+ email: ph.charriere@gmail.com
89+ createdAt: 2026-10-26T06:00:00.000Z
90+ - id: 2
91+ title: "minime.js: start (initialize + session/new), complete, stream, cancel, close"
92+ state: closed
93+ priority: none
94+ author:
95+ name: k33g
96+ email: ph.charriere@gmail.com
97+ createdAt: 2026-10-26T06:00:00.000Z
98+ - id: 3
99+ title: Permission callback (session/request_permission), refuse by default
100+ state: closed
101+ priority: none
102+ author:
103+ name: k33g
104+ email: ph.charriere@gmail.com
105+ createdAt: 2026-10-26T06:00:00.000Z
106+ - id: 4
107+ title: Examples (not streamed, streamed, cancelled, permission) run on llmman
108+ state: closed
109+ priority: none
110+ author:
111+ name: k33g
112+ email: ph.charriere@gmail.com
113+ createdAt: 2026-10-26T06:00:00.000Z
114+ - id: 5
115+ title: README EN + FR with measured results and limits
116+ state: closed
117+ priority: none
118+ author:
119+ name: k33g
120+ email: ph.charriere@gmail.com
121+ createdAt: 2026-10-26T06:00:00.000Z
122+comments:
123+ - id: 1
124+ author:
125+ name: k33g
126+ email: ph.charriere@gmail.com
127+ createdAt: 2026-10-26T10:30:00.000Z
128+ body: |
129+ Done in `demo.acp.sdk/nodejs/` (with an index `demo.acp.sdk/README.{en,fr}.md` for the other languages, tickets 168–172). Nothing changed in mm's Go code.
130+
131+ - `minime.js`, no dependency (Node >= 20): `Agent.start({config, cwd, mm, onPermission, onStderr})`, `complete(prompt, {signal, onEvent})` → `{text, stopReason, usage}`, `stream(prompt, {signal})` → async iterator (text, thinking, tool_call, tool_result, done), cancel with an `AbortSignal` (→ `session/cancel`, ends `cancelled` with the text so far; leaving the loop early cancels too), `reset()` (new session), `close()`. One Agent = one `mm -acp` process = one conversation; one turn at a time per Agent. Without `onPermission`, every call that needs a confirmation is refused. mm's JSON-RPC errors are rejected with mm's words (`data` included).
132+ - Task 1: raw JSON-RPC chosen; the official ACP TypeScript library was NOT looked at (no network in the sandbox).
133+ - Message shapes captured from `mm -acp` (v1.0.1) before writing: `session/update` kinds `agent_message_chunk`, `agent_thought_chunk`, `tool_call`, `tool_call_update` (rawOutput.output), `usage_update`, `available_commands_update`; `session/request_permission` with options allow / allow_always / reject / never; `session/prompt` result `{stopReason, usage}`.
134+ - Tests: `npm test` (node --test test/minime.test.js), 11 tests against `test/fake-mm.js`, which plays mm's side (answers, a permission, a slow turn to cancel, a crash). A bare `node --test` would also run fake-mm.js and hang.
135+ - Measured on llmman (Qwen3.5 4B, `yolo: false`): complete 2.9 s; stream with a read-only callback (`find … | wc -l` allowed, answer streamed); cancel after 3.0 s (`cancelled`, 383 characters kept; the next question works but does not know the story — mm drops an unanswered question); permission: `pwd` refused → the model invented `/root`, allowed → the real directory. Seen in three runs before `find` was allowed: the same refused command repeated until mm's loop detector stopped the turn, and `stream()` rejected with that message.
136+ - Not done: npm package (it would have to ship the binary), `session/load` / `session/resume`, images and attachments, `ask_user` forms.
new file mode 100644
@@ -0,0 +1,136 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 167
3+title: "Node.js client over ACP: a first SDK prototype for other languages"
4+state: closed
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T06:00:00.000Z
9+updatedAt: 2026-10-26T11:00:00.000Z
10+labels:
11+ - kind::feature
12+milestone: v1.0.2
13+epic: SDK
14+related:
15+ - 139
16+ - 71
17+ - 113
18+ - 168
19+ - 169
20+ - 170
21+ - 171
22+ - 172
23+body: |
24+ ## Goal
25+ Prove that mm can be used as a library from another language by driving `mm -acp` over ACP (JSON-RPC on stdin/stdout), with NO change to mm. Node.js first (user's choice, 2026-10-26); Python later if the prototype convinces. Prototype only: one demo folder, not a published package.
26+
27+ This is the "other languages" path. It complements the Go SDK of ticket 139 (in-process, no binary) and does not replace it. Remote use (an agent in another machine or container) stays for `mm serve` (ticket 113 / proposal D of 139).
28+
29+ ## Why ACP fits (checked in internal/acp and docs/en/reference/acp.md on 2026-10-26)
30+ | Wanted | ACP, already in mm |
31+ |---|---|
32+ | run a completion | `session/prompt` |
33+ | stream | `session/update` notifications (text chunks, thoughts, tool calls, plan) |
34+ | not streamed | join the text chunks until `session/prompt` answers (its `stopReason`) |
35+ | cancel | `session/cancel` |
36+ | tool confirmation (`yolo: false`) | mm sends `session/request_permission` to the client → a callback |
37+ | `ask_user` | `elicitation/create`, only when the client advertises `clientCapabilities.elicitation.form` at `initialize` |
38+ | keep the conversation | one ACP session = one history; `session/load` / `session/resume` with `sessions.enabled` |
39+
40+ - mm asks nothing else of the client: it reads files and runs commands itself, so the client does not have to implement `fs/*` or `terminal/*`.
41+ - One agent = one process: several agents run side by side, the global state of the Go code (`config.Cfg`, `ui.SetActive`) is no problem here.
42+ - Everything mm does works unchanged: hooks, MCP, `mmConfigs`, skill/tool routing, struggle signals.
43+
44+ ## Proposed shape (to discuss in the prototype)
45+ ```js
46+ import { Agent } from "./minime.js";
47+
48+ const agent = await Agent.start({
49+ config: "agent.yaml", // passed as AGENT_CONFIG (absolute path)
50+ cwd: process.cwd(), // the session's cwd (session/new)
51+ onPermission: (call) => call.kind === "read", // no callback → refuse
52+ });
53+
54+ const text = await agent.complete("Explain this repository"); // not streamed
55+
56+ for await (const ev of agent.stream("And the tests?")) { // streamed
57+ if (ev.kind === "text") process.stdout.write(ev.text);
58+ }
59+
60+ const ac = new AbortController();
61+ setTimeout(() => ac.abort(), 5000);
62+ await agent.complete("A long task", { signal: ac.signal }); // abort → session/cancel
63+
64+ await agent.close(); // closes stdin, mm exits
65+ ```
66+ - Plain Node.js (built-in `child_process`, `readline`), no dependency, ESM, Node 22 (the sandbox has v22.22.1).
67+ - Events reduced to a few kinds: `text`, `thinking`, `tool_call`, `tool_result`, `done` — not the editor-oriented detail of ACP (diffs, statuses).
68+ - The `mm` binary is found on the PATH or through `MM_BIN` (the convention of the demo hooks).
69+ - stderr of mm (banner, warnings, `[acp]` lines) is drained, optionally passed to a callback, never mixed with the answer.
70+ - Check first whether the official ACP TypeScript library (`@agentclientprotocol/sdk` or similar — not verified, no network in the sandbox) is worth using instead of raw JSON-RPC; raw JSON-RPC keeps the prototype dependency-free.
71+
72+ ## Limits to state in the README
73+ - The `mm` binary must be installed (a later package could ship it per platform, as some npm packages do for native tools).
74+ - Local only: stdio.
75+ - The config is a file path (AGENT_CONFIG); no in-memory config.
76+
77+ ## Done when
78+ - `demo.acp.sdk/nodejs/` (location chosen by the user on 2026-10-26, instead of `demos/node-acp-client/`) holds `minime.js` (the client), one example each for not streamed, streamed and cancelled, and README EN + FR with the results measured on llmman.
79+ - Permissions: a refused and an accepted tool call shown with an agent with `yolo: false`.
80+ - Nothing changed in mm's Go code.
81+tasks:
82+ - id: 1
83+ title: Check the official ACP TypeScript library vs raw JSON-RPC; decide
84+ state: closed
85+ priority: none
86+ author:
87+ name: k33g
88+ email: ph.charriere@gmail.com
89+ createdAt: 2026-10-26T06:00:00.000Z
90+ - id: 2
91+ title: "minime.js: start (initialize + session/new), complete, stream, cancel, close"
92+ state: closed
93+ priority: none
94+ author:
95+ name: k33g
96+ email: ph.charriere@gmail.com
97+ createdAt: 2026-10-26T06:00:00.000Z
98+ - id: 3
99+ title: Permission callback (session/request_permission), refuse by default
100+ state: closed
101+ priority: none
102+ author:
103+ name: k33g
104+ email: ph.charriere@gmail.com
105+ createdAt: 2026-10-26T06:00:00.000Z
106+ - id: 4
107+ title: Examples (not streamed, streamed, cancelled, permission) run on llmman
108+ state: closed
109+ priority: none
110+ author:
111+ name: k33g
112+ email: ph.charriere@gmail.com
113+ createdAt: 2026-10-26T06:00:00.000Z
114+ - id: 5
115+ title: README EN + FR with measured results and limits
116+ state: closed
117+ priority: none
118+ author:
119+ name: k33g
120+ email: ph.charriere@gmail.com
121+ createdAt: 2026-10-26T06:00:00.000Z
122+comments:
123+ - id: 1
124+ author:
125+ name: k33g
126+ email: ph.charriere@gmail.com
127+ createdAt: 2026-10-26T10:30:00.000Z
128+ body: |
129+ Done in `demo.acp.sdk/nodejs/` (with an index `demo.acp.sdk/README.{en,fr}.md` for the other languages, tickets 168–172). Nothing changed in mm's Go code.
130+
131+ - `minime.js`, no dependency (Node >= 20): `Agent.start({config, cwd, mm, onPermission, onStderr})`, `complete(prompt, {signal, onEvent})` → `{text, stopReason, usage}`, `stream(prompt, {signal})` → async iterator (text, thinking, tool_call, tool_result, done), cancel with an `AbortSignal` (→ `session/cancel`, ends `cancelled` with the text so far; leaving the loop early cancels too), `reset()` (new session), `close()`. One Agent = one `mm -acp` process = one conversation; one turn at a time per Agent. Without `onPermission`, every call that needs a confirmation is refused. mm's JSON-RPC errors are rejected with mm's words (`data` included).
132+ - Task 1: raw JSON-RPC chosen; the official ACP TypeScript library was NOT looked at (no network in the sandbox).
133+ - Message shapes captured from `mm -acp` (v1.0.1) before writing: `session/update` kinds `agent_message_chunk`, `agent_thought_chunk`, `tool_call`, `tool_call_update` (rawOutput.output), `usage_update`, `available_commands_update`; `session/request_permission` with options allow / allow_always / reject / never; `session/prompt` result `{stopReason, usage}`.
134+ - Tests: `npm test` (node --test test/minime.test.js), 11 tests against `test/fake-mm.js`, which plays mm's side (answers, a permission, a slow turn to cancel, a crash). A bare `node --test` would also run fake-mm.js and hang.
135+ - Measured on llmman (Qwen3.5 4B, `yolo: false`): complete 2.9 s; stream with a read-only callback (`find … | wc -l` allowed, answer streamed); cancel after 3.0 s (`cancelled`, 383 characters kept; the next question works but does not know the story — mm drops an unanswered question); permission: `pwd` refused → the model invented `/root`, allowed → the real directory. Seen in three runs before `find` was allowed: the same refused command repeated until mm's loop detector stopped the turn, and `stream()` rejected with that message.
136+ - Not done: npm package (it would have to ship the binary), `session/load` / `session/resume`, images and attachments, `ask_user` forms.
added .tickets/issues/0168-go-client-over-acp-sdk-prototype.yaml +72 -0
new file mode 100644
@@ -0,0 +1,72 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 168
3+title: "Go client over ACP: an out-of-process SDK prototype"
4+state: open
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T06:30:00.000Z
9+updatedAt: 2026-10-26T06:30:00.000Z
10+labels:
11+ - kind::feature
12+epic: SDK
13+related:
14+ - 167
15+ - 139
16+body: |
17+ ## Goal
18+ A Go program drives `mm -acp` as a child process. Unlike ticket 139 (in-process library, no binary), this one needs the binary but works TODAY with no change to mm, and gives process isolation (one agent per process, so the global state of mm's Go code is no problem).
19+
20+ ## Shared ground
21+ Same idea as ticket 167 (Node.js), in another language: drive `mm -acp` over ACP (JSON-RPC, one message per line on stdin/stdout), with NO change to mm. Ticket 167 holds the ACP mapping (`initialize`, `session/new`, `session/prompt`, `session/update`, `session/cancel`, `session/request_permission`, `elicitation/create`) and the limits (the `mm` binary must be installed, local stdio only, config by `AGENT_CONFIG`). The same API as 167: start an agent from a config file, `complete` (not streamed), `stream`, cancel, a permission callback that refuses by default, close. Start once the Node.js prototype has settled the API.
22+
23+ ## Specific to Go
24+ - mm already IS such a client: `internal/ask` (pool.go, client.go, elicit.go) starts child `mm -acp` processes with `github.com/coder/acp-go-sdk` (`sdk.NewClientSideConnection`) for `/ask`, `ask_agent` and `mm --ask`. The prototype can follow that code — but as its own module under `demos/` (like `demos/decision-chat-golang/hooks/ask-decider`), since `internal/` cannot be imported.
25+ - API in Go idiom: `agent.Complete(ctx, prompt, opts...)`, streaming through a callback or an `iter.Seq2[Event, error]`, cancellation by `ctx` (→ `session/cancel`).
26+ - Worth writing down in the README: when to choose this (isolation, works now) vs ticket 139 (no binary, in-process).
27+
28+ ## Done when
29+ - `demos/go-acp-client/` holds the client, one example each for not streamed, streamed, cancelled and a permission (agent with `yolo: false`), and README EN + FR with results measured on llmman. The module has `go test`-ed parsing of the events (no model needed).
30+ - Nothing changed in mm's Go code.
31+tasks:
32+ - id: 1
33+ title: "Decide: reuse acp-go-sdk (as internal/ask does) — expected — or raw JSON-RPC"
34+ state: open
35+ priority: none
36+ author:
37+ name: k33g
38+ email: ph.charriere@gmail.com
39+ createdAt: 2026-10-26T06:30:00.000Z
40+ - id: 2
41+ title: "Client: start (initialize + session/new), complete, stream, cancel, close"
42+ state: open
43+ priority: none
44+ author:
45+ name: k33g
46+ email: ph.charriere@gmail.com
47+ createdAt: 2026-10-26T06:30:00.000Z
48+ - id: 3
49+ title: "Permission callback (session/request_permission), refuse by default"
50+ state: open
51+ priority: none
52+ author:
53+ name: k33g
54+ email: ph.charriere@gmail.com
55+ createdAt: 2026-10-26T06:30:00.000Z
56+ - id: 4
57+ title: "Examples (not streamed, streamed, cancelled, permission) run on llmman"
58+ state: open
59+ priority: none
60+ author:
61+ name: k33g
62+ email: ph.charriere@gmail.com
63+ createdAt: 2026-10-26T06:30:00.000Z
64+ - id: 5
65+ title: "README EN + FR with measured results and limits"
66+ state: open
67+ priority: none
68+ author:
69+ name: k33g
70+ email: ph.charriere@gmail.com
71+ createdAt: 2026-10-26T06:30:00.000Z
72+comments: []
new file mode 100644
@@ -0,0 +1,72 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 168
3+title: "Go client over ACP: an out-of-process SDK prototype"
4+state: open
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T06:30:00.000Z
9+updatedAt: 2026-10-26T06:30:00.000Z
10+labels:
11+ - kind::feature
12+epic: SDK
13+related:
14+ - 167
15+ - 139
16+body: |
17+ ## Goal
18+ A Go program drives `mm -acp` as a child process. Unlike ticket 139 (in-process library, no binary), this one needs the binary but works TODAY with no change to mm, and gives process isolation (one agent per process, so the global state of mm's Go code is no problem).
19+
20+ ## Shared ground
21+ Same idea as ticket 167 (Node.js), in another language: drive `mm -acp` over ACP (JSON-RPC, one message per line on stdin/stdout), with NO change to mm. Ticket 167 holds the ACP mapping (`initialize`, `session/new`, `session/prompt`, `session/update`, `session/cancel`, `session/request_permission`, `elicitation/create`) and the limits (the `mm` binary must be installed, local stdio only, config by `AGENT_CONFIG`). The same API as 167: start an agent from a config file, `complete` (not streamed), `stream`, cancel, a permission callback that refuses by default, close. Start once the Node.js prototype has settled the API.
22+
23+ ## Specific to Go
24+ - mm already IS such a client: `internal/ask` (pool.go, client.go, elicit.go) starts child `mm -acp` processes with `github.com/coder/acp-go-sdk` (`sdk.NewClientSideConnection`) for `/ask`, `ask_agent` and `mm --ask`. The prototype can follow that code — but as its own module under `demos/` (like `demos/decision-chat-golang/hooks/ask-decider`), since `internal/` cannot be imported.
25+ - API in Go idiom: `agent.Complete(ctx, prompt, opts...)`, streaming through a callback or an `iter.Seq2[Event, error]`, cancellation by `ctx` (→ `session/cancel`).
26+ - Worth writing down in the README: when to choose this (isolation, works now) vs ticket 139 (no binary, in-process).
27+
28+ ## Done when
29+ - `demos/go-acp-client/` holds the client, one example each for not streamed, streamed, cancelled and a permission (agent with `yolo: false`), and README EN + FR with results measured on llmman. The module has `go test`-ed parsing of the events (no model needed).
30+ - Nothing changed in mm's Go code.
31+tasks:
32+ - id: 1
33+ title: "Decide: reuse acp-go-sdk (as internal/ask does) — expected — or raw JSON-RPC"
34+ state: open
35+ priority: none
36+ author:
37+ name: k33g
38+ email: ph.charriere@gmail.com
39+ createdAt: 2026-10-26T06:30:00.000Z
40+ - id: 2
41+ title: "Client: start (initialize + session/new), complete, stream, cancel, close"
42+ state: open
43+ priority: none
44+ author:
45+ name: k33g
46+ email: ph.charriere@gmail.com
47+ createdAt: 2026-10-26T06:30:00.000Z
48+ - id: 3
49+ title: "Permission callback (session/request_permission), refuse by default"
50+ state: open
51+ priority: none
52+ author:
53+ name: k33g
54+ email: ph.charriere@gmail.com
55+ createdAt: 2026-10-26T06:30:00.000Z
56+ - id: 4
57+ title: "Examples (not streamed, streamed, cancelled, permission) run on llmman"
58+ state: open
59+ priority: none
60+ author:
61+ name: k33g
62+ email: ph.charriere@gmail.com
63+ createdAt: 2026-10-26T06:30:00.000Z
64+ - id: 5
65+ title: "README EN + FR with measured results and limits"
66+ state: open
67+ priority: none
68+ author:
69+ name: k33g
70+ email: ph.charriere@gmail.com
71+ createdAt: 2026-10-26T06:30:00.000Z
72+comments: []
added .tickets/issues/0169-python-client-over-acp-sdk-prototype.yaml +73 -0
new file mode 100644
@@ -0,0 +1,73 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 169
3+title: "Python client over ACP: an SDK prototype"
4+state: open
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T06:30:00.000Z
9+updatedAt: 2026-10-26T06:30:00.000Z
10+labels:
11+ - kind::feature
12+epic: SDK
13+related:
14+ - 167
15+ - 139
16+body: |
17+ ## Goal
18+ A Python program uses mm as a library by driving `mm -acp`. The first language asked for (2026-10-26), started after the Node.js prototype (ticket 167).
19+
20+ ## Shared ground
21+ Same idea as ticket 167 (Node.js), in another language: drive `mm -acp` over ACP (JSON-RPC, one message per line on stdin/stdout), with NO change to mm. Ticket 167 holds the ACP mapping (`initialize`, `session/new`, `session/prompt`, `session/update`, `session/cancel`, `session/request_permission`, `elicitation/create`) and the limits (the `mm` binary must be installed, local stdio only, config by `AGENT_CONFIG`). The same API as 167: start an agent from a config file, `complete` (not streamed), `stream`, cancel, a permission callback that refuses by default, close. Start once the Node.js prototype has settled the API.
22+
23+ ## Specific to Python
24+ - `asyncio` with the standard library only (`asyncio.create_subprocess_exec`, stdin/stdout streams, `json`): no dependency. Python 3.14 is in the sandbox.
25+ - API: `async with minime.Agent("agent.yaml", cwd=".", on_permission=...) as agent`, `await agent.complete(...)`, `async for ev in agent.stream(...)`, cancellation by cancelling the task (→ `session/cancel`).
26+ - Check whether an official ACP Python library exists (`agent-client-protocol` on PyPI is believed to — NOT verified, no network in the sandbox) and whether it is worth a dependency.
27+ - Later, if it becomes a package: ship the `mm` binary inside per-platform wheels (as ruff or uv do) so `pip install` is enough.
28+
29+ ## Done when
30+ - `demos/python-acp-client/` holds the client, one example each for not streamed, streamed, cancelled and a permission (agent with `yolo: false`), and README EN + FR with results measured on llmman.
31+ - Nothing changed in mm's Go code.
32+tasks:
33+ - id: 1
34+ title: "Check the official ACP Python library vs stdlib JSON-RPC; decide"
35+ state: open
36+ priority: none
37+ author:
38+ name: k33g
39+ email: ph.charriere@gmail.com
40+ createdAt: 2026-10-26T06:30:00.000Z
41+ - id: 2
42+ title: "Client: start (initialize + session/new), complete, stream, cancel, close"
43+ state: open
44+ priority: none
45+ author:
46+ name: k33g
47+ email: ph.charriere@gmail.com
48+ createdAt: 2026-10-26T06:30:00.000Z
49+ - id: 3
50+ title: "Permission callback (session/request_permission), refuse by default"
51+ state: open
52+ priority: none
53+ author:
54+ name: k33g
55+ email: ph.charriere@gmail.com
56+ createdAt: 2026-10-26T06:30:00.000Z
57+ - id: 4
58+ title: "Examples (not streamed, streamed, cancelled, permission) run on llmman"
59+ state: open
60+ priority: none
61+ author:
62+ name: k33g
63+ email: ph.charriere@gmail.com
64+ createdAt: 2026-10-26T06:30:00.000Z
65+ - id: 5
66+ title: "README EN + FR with measured results and limits"
67+ state: open
68+ priority: none
69+ author:
70+ name: k33g
71+ email: ph.charriere@gmail.com
72+ createdAt: 2026-10-26T06:30:00.000Z
73+comments: []
new file mode 100644
@@ -0,0 +1,73 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 169
3+title: "Python client over ACP: an SDK prototype"
4+state: open
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T06:30:00.000Z
9+updatedAt: 2026-10-26T06:30:00.000Z
10+labels:
11+ - kind::feature
12+epic: SDK
13+related:
14+ - 167
15+ - 139
16+body: |
17+ ## Goal
18+ A Python program uses mm as a library by driving `mm -acp`. The first language asked for (2026-10-26), started after the Node.js prototype (ticket 167).
19+
20+ ## Shared ground
21+ Same idea as ticket 167 (Node.js), in another language: drive `mm -acp` over ACP (JSON-RPC, one message per line on stdin/stdout), with NO change to mm. Ticket 167 holds the ACP mapping (`initialize`, `session/new`, `session/prompt`, `session/update`, `session/cancel`, `session/request_permission`, `elicitation/create`) and the limits (the `mm` binary must be installed, local stdio only, config by `AGENT_CONFIG`). The same API as 167: start an agent from a config file, `complete` (not streamed), `stream`, cancel, a permission callback that refuses by default, close. Start once the Node.js prototype has settled the API.
22+
23+ ## Specific to Python
24+ - `asyncio` with the standard library only (`asyncio.create_subprocess_exec`, stdin/stdout streams, `json`): no dependency. Python 3.14 is in the sandbox.
25+ - API: `async with minime.Agent("agent.yaml", cwd=".", on_permission=...) as agent`, `await agent.complete(...)`, `async for ev in agent.stream(...)`, cancellation by cancelling the task (→ `session/cancel`).
26+ - Check whether an official ACP Python library exists (`agent-client-protocol` on PyPI is believed to — NOT verified, no network in the sandbox) and whether it is worth a dependency.
27+ - Later, if it becomes a package: ship the `mm` binary inside per-platform wheels (as ruff or uv do) so `pip install` is enough.
28+
29+ ## Done when
30+ - `demos/python-acp-client/` holds the client, one example each for not streamed, streamed, cancelled and a permission (agent with `yolo: false`), and README EN + FR with results measured on llmman.
31+ - Nothing changed in mm's Go code.
32+tasks:
33+ - id: 1
34+ title: "Check the official ACP Python library vs stdlib JSON-RPC; decide"
35+ state: open
36+ priority: none
37+ author:
38+ name: k33g
39+ email: ph.charriere@gmail.com
40+ createdAt: 2026-10-26T06:30:00.000Z
41+ - id: 2
42+ title: "Client: start (initialize + session/new), complete, stream, cancel, close"
43+ state: open
44+ priority: none
45+ author:
46+ name: k33g
47+ email: ph.charriere@gmail.com
48+ createdAt: 2026-10-26T06:30:00.000Z
49+ - id: 3
50+ title: "Permission callback (session/request_permission), refuse by default"
51+ state: open
52+ priority: none
53+ author:
54+ name: k33g
55+ email: ph.charriere@gmail.com
56+ createdAt: 2026-10-26T06:30:00.000Z
57+ - id: 4
58+ title: "Examples (not streamed, streamed, cancelled, permission) run on llmman"
59+ state: open
60+ priority: none
61+ author:
62+ name: k33g
63+ email: ph.charriere@gmail.com
64+ createdAt: 2026-10-26T06:30:00.000Z
65+ - id: 5
66+ title: "README EN + FR with measured results and limits"
67+ state: open
68+ priority: none
69+ author:
70+ name: k33g
71+ email: ph.charriere@gmail.com
72+ createdAt: 2026-10-26T06:30:00.000Z
73+comments: []
added .tickets/issues/0170-rust-client-over-acp-sdk-prototype.yaml +80 -0
new file mode 100644
@@ -0,0 +1,80 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 170
3+title: "Rust client over ACP: an SDK prototype"
4+state: open
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T06:30:00.000Z
9+updatedAt: 2026-10-26T06:30:00.000Z
10+labels:
11+ - kind::feature
12+epic: SDK
13+related:
14+ - 167
15+ - 139
16+body: |
17+ ## Goal
18+ A Rust program uses mm as a library by driving `mm -acp`.
19+
20+ ## Shared ground
21+ Same idea as ticket 167 (Node.js), in another language: drive `mm -acp` over ACP (JSON-RPC, one message per line on stdin/stdout), with NO change to mm. Ticket 167 holds the ACP mapping (`initialize`, `session/new`, `session/prompt`, `session/update`, `session/cancel`, `session/request_permission`, `elicitation/create`) and the limits (the `mm` binary must be installed, local stdio only, config by `AGENT_CONFIG`). The same API as 167: start an agent from a config file, `complete` (not streamed), `stream`, cancel, a permission callback that refuses by default, close. Start once the Node.js prototype has settled the API.
22+
23+ ## Specific to Rust
24+ - ACP comes from Zed, written in Rust: a Rust crate for the protocol (`agent-client-protocol`) is believed to exist — NOT verified (no network in the sandbox). Otherwise `tokio` (`tokio::process::Command`, line-delimited JSON) + `serde_json`.
25+ - API: `Agent::start(config).await?`, `agent.complete(prompt).await?`, `agent.stream(prompt)` returning a `Stream<Item = Result<Event>>`, cancellation by dropping the future or a `CancellationToken` (→ `session/cancel`).
26+ - No Rust toolchain in this sandbox (`cargo` absent): it must be installed, or the prototype built and run on the user's machine.
27+
28+ ## Done when
29+ - `demos/rust-acp-client/` holds the client, one example each for not streamed, streamed, cancelled and a permission (agent with `yolo: false`), and README EN + FR with results measured on llmman.
30+ - Nothing changed in mm's Go code.
31+tasks:
32+ - id: 1
33+ title: "Check the Rust ACP crate vs tokio + serde_json; decide"
34+ state: open
35+ priority: none
36+ author:
37+ name: k33g
38+ email: ph.charriere@gmail.com
39+ createdAt: 2026-10-26T06:30:00.000Z
40+ - id: 2
41+ title: "Get a Rust toolchain (sandbox kit or the user's machine)"
42+ state: open
43+ priority: none
44+ author:
45+ name: k33g
46+ email: ph.charriere@gmail.com
47+ createdAt: 2026-10-26T06:30:00.000Z
48+ - id: 3
49+ title: "Client: start (initialize + session/new), complete, stream, cancel, close"
50+ state: open
51+ priority: none
52+ author:
53+ name: k33g
54+ email: ph.charriere@gmail.com
55+ createdAt: 2026-10-26T06:30:00.000Z
56+ - id: 4
57+ title: "Permission callback (session/request_permission), refuse by default"
58+ state: open
59+ priority: none
60+ author:
61+ name: k33g
62+ email: ph.charriere@gmail.com
63+ createdAt: 2026-10-26T06:30:00.000Z
64+ - id: 5
65+ title: "Examples (not streamed, streamed, cancelled, permission) run on llmman"
66+ state: open
67+ priority: none
68+ author:
69+ name: k33g
70+ email: ph.charriere@gmail.com
71+ createdAt: 2026-10-26T06:30:00.000Z
72+ - id: 6
73+ title: "README EN + FR with measured results and limits"
74+ state: open
75+ priority: none
76+ author:
77+ name: k33g
78+ email: ph.charriere@gmail.com
79+ createdAt: 2026-10-26T06:30:00.000Z
80+comments: []
new file mode 100644
@@ -0,0 +1,80 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 170
3+title: "Rust client over ACP: an SDK prototype"
4+state: open
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T06:30:00.000Z
9+updatedAt: 2026-10-26T06:30:00.000Z
10+labels:
11+ - kind::feature
12+epic: SDK
13+related:
14+ - 167
15+ - 139
16+body: |
17+ ## Goal
18+ A Rust program uses mm as a library by driving `mm -acp`.
19+
20+ ## Shared ground
21+ Same idea as ticket 167 (Node.js), in another language: drive `mm -acp` over ACP (JSON-RPC, one message per line on stdin/stdout), with NO change to mm. Ticket 167 holds the ACP mapping (`initialize`, `session/new`, `session/prompt`, `session/update`, `session/cancel`, `session/request_permission`, `elicitation/create`) and the limits (the `mm` binary must be installed, local stdio only, config by `AGENT_CONFIG`). The same API as 167: start an agent from a config file, `complete` (not streamed), `stream`, cancel, a permission callback that refuses by default, close. Start once the Node.js prototype has settled the API.
22+
23+ ## Specific to Rust
24+ - ACP comes from Zed, written in Rust: a Rust crate for the protocol (`agent-client-protocol`) is believed to exist — NOT verified (no network in the sandbox). Otherwise `tokio` (`tokio::process::Command`, line-delimited JSON) + `serde_json`.
25+ - API: `Agent::start(config).await?`, `agent.complete(prompt).await?`, `agent.stream(prompt)` returning a `Stream<Item = Result<Event>>`, cancellation by dropping the future or a `CancellationToken` (→ `session/cancel`).
26+ - No Rust toolchain in this sandbox (`cargo` absent): it must be installed, or the prototype built and run on the user's machine.
27+
28+ ## Done when
29+ - `demos/rust-acp-client/` holds the client, one example each for not streamed, streamed, cancelled and a permission (agent with `yolo: false`), and README EN + FR with results measured on llmman.
30+ - Nothing changed in mm's Go code.
31+tasks:
32+ - id: 1
33+ title: "Check the Rust ACP crate vs tokio + serde_json; decide"
34+ state: open
35+ priority: none
36+ author:
37+ name: k33g
38+ email: ph.charriere@gmail.com
39+ createdAt: 2026-10-26T06:30:00.000Z
40+ - id: 2
41+ title: "Get a Rust toolchain (sandbox kit or the user's machine)"
42+ state: open
43+ priority: none
44+ author:
45+ name: k33g
46+ email: ph.charriere@gmail.com
47+ createdAt: 2026-10-26T06:30:00.000Z
48+ - id: 3
49+ title: "Client: start (initialize + session/new), complete, stream, cancel, close"
50+ state: open
51+ priority: none
52+ author:
53+ name: k33g
54+ email: ph.charriere@gmail.com
55+ createdAt: 2026-10-26T06:30:00.000Z
56+ - id: 4
57+ title: "Permission callback (session/request_permission), refuse by default"
58+ state: open
59+ priority: none
60+ author:
61+ name: k33g
62+ email: ph.charriere@gmail.com
63+ createdAt: 2026-10-26T06:30:00.000Z
64+ - id: 5
65+ title: "Examples (not streamed, streamed, cancelled, permission) run on llmman"
66+ state: open
67+ priority: none
68+ author:
69+ name: k33g
70+ email: ph.charriere@gmail.com
71+ createdAt: 2026-10-26T06:30:00.000Z
72+ - id: 6
73+ title: "README EN + FR with measured results and limits"
74+ state: open
75+ priority: none
76+ author:
77+ name: k33g
78+ email: ph.charriere@gmail.com
79+ createdAt: 2026-10-26T06:30:00.000Z
80+comments: []
added .tickets/issues/0171-swift-client-over-acp-sdk-prototype.yaml +80 -0
new file mode 100644
@@ -0,0 +1,80 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 171
3+title: "Swift client over ACP: an SDK prototype"
4+state: open
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T06:30:00.000Z
9+updatedAt: 2026-10-26T06:30:00.000Z
10+labels:
11+ - kind::feature
12+epic: SDK
13+related:
14+ - 167
15+ - 139
16+body: |
17+ ## Goal
18+ A Swift program (a macOS app, a command-line tool) uses mm as a library by driving `mm -acp` — a path towards a native Mac front end for mm.
19+
20+ ## Shared ground
21+ Same idea as ticket 167 (Node.js), in another language: drive `mm -acp` over ACP (JSON-RPC, one message per line on stdin/stdout), with NO change to mm. Ticket 167 holds the ACP mapping (`initialize`, `session/new`, `session/prompt`, `session/update`, `session/cancel`, `session/request_permission`, `elicitation/create`) and the limits (the `mm` binary must be installed, local stdio only, config by `AGENT_CONFIG`). The same API as 167: start an agent from a config file, `complete` (not streamed), `stream`, cancel, a permission callback that refuses by default, close. Start once the Node.js prototype has settled the API.
22+
23+ ## Specific to Swift
24+ - Foundation `Process` + `Pipe`, `FileHandle.bytes.lines` for the line-delimited JSON, `Codable` for the messages, Swift concurrency: `func complete(_:) async throws -> String`, `func stream(_:) -> AsyncThrowingStream<Event, Error>`, cancellation through `Task.cancel()` (→ `session/cancel`).
25+ - Check whether a Swift ACP package exists (not verified).
26+ - No Swift toolchain in this sandbox (Linux, `swift` absent): built and run on the user's Mac. A sandboxed macOS app cannot start an arbitrary binary without entitlements — fine for a command-line prototype, to note for an app.
27+
28+ ## Done when
29+ - `demos/swift-acp-client/` holds the client, one example each for not streamed, streamed, cancelled and a permission (agent with `yolo: false`), and README EN + FR with results measured on llmman. A Swift package (`Package.swift`) with an executable target.
30+ - Nothing changed in mm's Go code.
31+tasks:
32+ - id: 1
33+ title: "Check for a Swift ACP package; decide"
34+ state: open
35+ priority: none
36+ author:
37+ name: k33g
38+ email: ph.charriere@gmail.com
39+ createdAt: 2026-10-26T06:30:00.000Z
40+ - id: 2
41+ title: "Build and run on the user's Mac (no Swift in the sandbox)"
42+ state: open
43+ priority: none
44+ author:
45+ name: k33g
46+ email: ph.charriere@gmail.com
47+ createdAt: 2026-10-26T06:30:00.000Z
48+ - id: 3
49+ title: "Client: start (initialize + session/new), complete, stream, cancel, close"
50+ state: open
51+ priority: none
52+ author:
53+ name: k33g
54+ email: ph.charriere@gmail.com
55+ createdAt: 2026-10-26T06:30:00.000Z
56+ - id: 4
57+ title: "Permission callback (session/request_permission), refuse by default"
58+ state: open
59+ priority: none
60+ author:
61+ name: k33g
62+ email: ph.charriere@gmail.com
63+ createdAt: 2026-10-26T06:30:00.000Z
64+ - id: 5
65+ title: "Examples (not streamed, streamed, cancelled, permission) run on llmman"
66+ state: open
67+ priority: none
68+ author:
69+ name: k33g
70+ email: ph.charriere@gmail.com
71+ createdAt: 2026-10-26T06:30:00.000Z
72+ - id: 6
73+ title: "README EN + FR with measured results and limits"
74+ state: open
75+ priority: none
76+ author:
77+ name: k33g
78+ email: ph.charriere@gmail.com
79+ createdAt: 2026-10-26T06:30:00.000Z
80+comments: []
new file mode 100644
@@ -0,0 +1,80 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 171
3+title: "Swift client over ACP: an SDK prototype"
4+state: open
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T06:30:00.000Z
9+updatedAt: 2026-10-26T06:30:00.000Z
10+labels:
11+ - kind::feature
12+epic: SDK
13+related:
14+ - 167
15+ - 139
16+body: |
17+ ## Goal
18+ A Swift program (a macOS app, a command-line tool) uses mm as a library by driving `mm -acp` — a path towards a native Mac front end for mm.
19+
20+ ## Shared ground
21+ Same idea as ticket 167 (Node.js), in another language: drive `mm -acp` over ACP (JSON-RPC, one message per line on stdin/stdout), with NO change to mm. Ticket 167 holds the ACP mapping (`initialize`, `session/new`, `session/prompt`, `session/update`, `session/cancel`, `session/request_permission`, `elicitation/create`) and the limits (the `mm` binary must be installed, local stdio only, config by `AGENT_CONFIG`). The same API as 167: start an agent from a config file, `complete` (not streamed), `stream`, cancel, a permission callback that refuses by default, close. Start once the Node.js prototype has settled the API.
22+
23+ ## Specific to Swift
24+ - Foundation `Process` + `Pipe`, `FileHandle.bytes.lines` for the line-delimited JSON, `Codable` for the messages, Swift concurrency: `func complete(_:) async throws -> String`, `func stream(_:) -> AsyncThrowingStream<Event, Error>`, cancellation through `Task.cancel()` (→ `session/cancel`).
25+ - Check whether a Swift ACP package exists (not verified).
26+ - No Swift toolchain in this sandbox (Linux, `swift` absent): built and run on the user's Mac. A sandboxed macOS app cannot start an arbitrary binary without entitlements — fine for a command-line prototype, to note for an app.
27+
28+ ## Done when
29+ - `demos/swift-acp-client/` holds the client, one example each for not streamed, streamed, cancelled and a permission (agent with `yolo: false`), and README EN + FR with results measured on llmman. A Swift package (`Package.swift`) with an executable target.
30+ - Nothing changed in mm's Go code.
31+tasks:
32+ - id: 1
33+ title: "Check for a Swift ACP package; decide"
34+ state: open
35+ priority: none
36+ author:
37+ name: k33g
38+ email: ph.charriere@gmail.com
39+ createdAt: 2026-10-26T06:30:00.000Z
40+ - id: 2
41+ title: "Build and run on the user's Mac (no Swift in the sandbox)"
42+ state: open
43+ priority: none
44+ author:
45+ name: k33g
46+ email: ph.charriere@gmail.com
47+ createdAt: 2026-10-26T06:30:00.000Z
48+ - id: 3
49+ title: "Client: start (initialize + session/new), complete, stream, cancel, close"
50+ state: open
51+ priority: none
52+ author:
53+ name: k33g
54+ email: ph.charriere@gmail.com
55+ createdAt: 2026-10-26T06:30:00.000Z
56+ - id: 4
57+ title: "Permission callback (session/request_permission), refuse by default"
58+ state: open
59+ priority: none
60+ author:
61+ name: k33g
62+ email: ph.charriere@gmail.com
63+ createdAt: 2026-10-26T06:30:00.000Z
64+ - id: 5
65+ title: "Examples (not streamed, streamed, cancelled, permission) run on llmman"
66+ state: open
67+ priority: none
68+ author:
69+ name: k33g
70+ email: ph.charriere@gmail.com
71+ createdAt: 2026-10-26T06:30:00.000Z
72+ - id: 6
73+ title: "README EN + FR with measured results and limits"
74+ state: open
75+ priority: none
76+ author:
77+ name: k33g
78+ email: ph.charriere@gmail.com
79+ createdAt: 2026-10-26T06:30:00.000Z
80+comments: []
added .tickets/issues/0172-nushell-script-client-over-acp.yaml +84 -0
new file mode 100644
@@ -0,0 +1,84 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 172
3+title: "Nushell script client over ACP: mm from a shell script"
4+state: open
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T06:30:00.000Z
9+updatedAt: 2026-10-26T06:30:00.000Z
10+labels:
11+ - kind::feature
12+epic: SDK
13+related:
14+ - 167
15+ - 139
16+body: |
17+ ## Goal
18+ Use mm from a Nushell script with structured data (tables, records) instead of text: send a question, get the answer and the tool calls as Nushell values.
19+
20+ ## Shared ground
21+ Same idea as ticket 167 (Node.js), in another language: drive `mm -acp` over ACP (JSON-RPC, one message per line on stdin/stdout), with NO change to mm. Ticket 167 holds the ACP mapping (`initialize`, `session/new`, `session/prompt`, `session/update`, `session/cancel`, `session/request_permission`, `elicitation/create`) and the limits (the `mm` binary must be installed, local stdio only, config by `AGENT_CONFIG`). The same API as 167: start an agent from a config file, `complete` (not streamed), `stream`, cancel, a permission callback that refuses by default, close. Start once the Node.js prototype has settled the API.
22+
23+ ## Specific to Nushell — the hard part
24+ - ACP is a conversation on ONE long-lived process: `session/prompt` needs the `sessionId` that `session/new` returns (mm's ids are time-sortable and random, so it cannot be guessed). A Nushell pipeline writes stdin and reads stdout, but answering a message read from stdout on the same process's stdin is not something a plain pipeline does. To study first:
25+ - Nushell's background jobs / `job` commands and whether a pipe can be held open both ways;
26+ - a small helper (a Nushell plugin, or reuse of the Node/Python client of tickets 167/169 behind a command) — then Nushell only parses its JSON output;
27+ - for the simplest case, plain `mm -p` already gives one answer and is easy from Nushell — the ACP client only adds streaming, cancellation and permissions.
28+ - Permissions cannot be asked interactively in a pipeline: refuse by default, or a closure given by the script.
29+ - Streaming: each `session/update` becomes a record, so the script can `where kind == text` or `each` over them.
30+ - `nu` is not installed in this sandbox.
31+
32+ ## Done when
33+ - `demos/nushell-acp-client/` holds the client, one example each for not streamed, streamed, cancelled and a permission (agent with `yolo: false`), and README EN + FR with results measured on llmman. The README compares it with plain `mm -p` from Nushell, so the reader knows when ACP is worth it.
34+ - Nothing changed in mm's Go code.
35+tasks:
36+ - id: 1
37+ title: "Study: can Nushell hold a two-way pipe to one process? Pick the approach (pure nu, plugin, or a helper)"
38+ state: open
39+ priority: none
40+ author:
41+ name: k33g
42+ email: ph.charriere@gmail.com
43+ createdAt: 2026-10-26T06:30:00.000Z
44+ - id: 2
45+ title: "Get Nushell (sandbox or the user's machine)"
46+ state: open
47+ priority: none
48+ author:
49+ name: k33g
50+ email: ph.charriere@gmail.com
51+ createdAt: 2026-10-26T06:30:00.000Z
52+ - id: 3
53+ title: "Client: start (initialize + session/new), complete, stream, cancel, close"
54+ state: open
55+ priority: none
56+ author:
57+ name: k33g
58+ email: ph.charriere@gmail.com
59+ createdAt: 2026-10-26T06:30:00.000Z
60+ - id: 4
61+ title: "Permission callback (session/request_permission), refuse by default"
62+ state: open
63+ priority: none
64+ author:
65+ name: k33g
66+ email: ph.charriere@gmail.com
67+ createdAt: 2026-10-26T06:30:00.000Z
68+ - id: 5
69+ title: "Examples (not streamed, streamed, cancelled, permission) run on llmman"
70+ state: open
71+ priority: none
72+ author:
73+ name: k33g
74+ email: ph.charriere@gmail.com
75+ createdAt: 2026-10-26T06:30:00.000Z
76+ - id: 6
77+ title: "README EN + FR with measured results and limits"
78+ state: open
79+ priority: none
80+ author:
81+ name: k33g
82+ email: ph.charriere@gmail.com
83+ createdAt: 2026-10-26T06:30:00.000Z
84+comments: []
new file mode 100644
@@ -0,0 +1,84 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 172
3+title: "Nushell script client over ACP: mm from a shell script"
4+state: open
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T06:30:00.000Z
9+updatedAt: 2026-10-26T06:30:00.000Z
10+labels:
11+ - kind::feature
12+epic: SDK
13+related:
14+ - 167
15+ - 139
16+body: |
17+ ## Goal
18+ Use mm from a Nushell script with structured data (tables, records) instead of text: send a question, get the answer and the tool calls as Nushell values.
19+
20+ ## Shared ground
21+ Same idea as ticket 167 (Node.js), in another language: drive `mm -acp` over ACP (JSON-RPC, one message per line on stdin/stdout), with NO change to mm. Ticket 167 holds the ACP mapping (`initialize`, `session/new`, `session/prompt`, `session/update`, `session/cancel`, `session/request_permission`, `elicitation/create`) and the limits (the `mm` binary must be installed, local stdio only, config by `AGENT_CONFIG`). The same API as 167: start an agent from a config file, `complete` (not streamed), `stream`, cancel, a permission callback that refuses by default, close. Start once the Node.js prototype has settled the API.
22+
23+ ## Specific to Nushell — the hard part
24+ - ACP is a conversation on ONE long-lived process: `session/prompt` needs the `sessionId` that `session/new` returns (mm's ids are time-sortable and random, so it cannot be guessed). A Nushell pipeline writes stdin and reads stdout, but answering a message read from stdout on the same process's stdin is not something a plain pipeline does. To study first:
25+ - Nushell's background jobs / `job` commands and whether a pipe can be held open both ways;
26+ - a small helper (a Nushell plugin, or reuse of the Node/Python client of tickets 167/169 behind a command) — then Nushell only parses its JSON output;
27+ - for the simplest case, plain `mm -p` already gives one answer and is easy from Nushell — the ACP client only adds streaming, cancellation and permissions.
28+ - Permissions cannot be asked interactively in a pipeline: refuse by default, or a closure given by the script.
29+ - Streaming: each `session/update` becomes a record, so the script can `where kind == text` or `each` over them.
30+ - `nu` is not installed in this sandbox.
31+
32+ ## Done when
33+ - `demos/nushell-acp-client/` holds the client, one example each for not streamed, streamed, cancelled and a permission (agent with `yolo: false`), and README EN + FR with results measured on llmman. The README compares it with plain `mm -p` from Nushell, so the reader knows when ACP is worth it.
34+ - Nothing changed in mm's Go code.
35+tasks:
36+ - id: 1
37+ title: "Study: can Nushell hold a two-way pipe to one process? Pick the approach (pure nu, plugin, or a helper)"
38+ state: open
39+ priority: none
40+ author:
41+ name: k33g
42+ email: ph.charriere@gmail.com
43+ createdAt: 2026-10-26T06:30:00.000Z
44+ - id: 2
45+ title: "Get Nushell (sandbox or the user's machine)"
46+ state: open
47+ priority: none
48+ author:
49+ name: k33g
50+ email: ph.charriere@gmail.com
51+ createdAt: 2026-10-26T06:30:00.000Z
52+ - id: 3
53+ title: "Client: start (initialize + session/new), complete, stream, cancel, close"
54+ state: open
55+ priority: none
56+ author:
57+ name: k33g
58+ email: ph.charriere@gmail.com
59+ createdAt: 2026-10-26T06:30:00.000Z
60+ - id: 4
61+ title: "Permission callback (session/request_permission), refuse by default"
62+ state: open
63+ priority: none
64+ author:
65+ name: k33g
66+ email: ph.charriere@gmail.com
67+ createdAt: 2026-10-26T06:30:00.000Z
68+ - id: 5
69+ title: "Examples (not streamed, streamed, cancelled, permission) run on llmman"
70+ state: open
71+ priority: none
72+ author:
73+ name: k33g
74+ email: ph.charriere@gmail.com
75+ createdAt: 2026-10-26T06:30:00.000Z
76+ - id: 6
77+ title: "README EN + FR with measured results and limits"
78+ state: open
79+ priority: none
80+ author:
81+ name: k33g
82+ email: ph.charriere@gmail.com
83+ createdAt: 2026-10-26T06:30:00.000Z
84+comments: []
added .tickets/issues/0173-switch-the-license-from-mit-to-fsl-1-1-mit.yaml +92 -0
new file mode 100644
@@ -0,0 +1,92 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 173
3+title: Switch the license from MIT to FSL-1.1-MIT (temporary)
4+state: open
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T09:10:00.000Z
9+updatedAt: 2026-10-26T09:10:00.000Z
10+labels:
11+ - kind::chore
12+ - documentation
13+body: |
14+ ## Decision (2026-10-26)
15+ The next releases of mini-me are published under the **Functional Source License, version 1.1, MIT Future License** (FSL-1.1-MIT) instead of MIT. Not applied yet: this ticket prepares the change.
16+
17+ ## Why
18+ The wish: everyone can use the `mm` binary and contribute, but the code is not taken to build another product. No standard license forbids every reuse; FSL is the closest standard text:
19+ - anyone may use, read, modify and redistribute the code, for any purpose **except a "Competing Use"**: making a product or service that competes with mini-me with its code;
20+ - each version becomes **MIT two years after it is published**, automatically;
21+ - using mini-me as a library from another program (the SDK of tickets 139, 167–172) is not a competing use, so the SDK stays possible;
22+ - a short, known, unmodified text (Sentry, GitButler, Convex… use it): nothing to write or have reviewed by a lawyer, unlike a custom license.
23+
24+ Options weighed and set aside: a custom "no reuse at all" license (would forbid the SDK, needs a lawyer, not a known text), PolyForm Strict (non-commercial use only), PolyForm Shield (no conversion to open source), BSL 1.1 (more to write, up to 4 years), AGPL-3.0 and MPL-2.0 (open source, but do not stop reuse).
25+
26+ ## Things to know and to say
27+ - **FSL is not open source** in the OSI sense during the two years (it restricts a field of use); some contributors and distributions avoid such projects. Do not call mini-me "open source" in the README or the docs once switched: "source-available", or "Fair Source" (fair.io).
28+ - **Versions already published stay MIT** (every tag up to the last one before the switch: v1.0.0, v1.0.1…). Only the versions after the switch are FSL.
29+ - The switch is possible because all commits so far are by k33g (`git log`: k33g, plus "rickub" merge commits). Before switching, check that no outside contribution has landed since; if one has, its author must agree.
30+
31+ ## A note in the LICENSE file: the switch is temporary
32+ At the top of `LICENSE`, before the FSL text, a short note (EN, and FR in the docs) saying that:
33+ - mini-me moved from MIT to FSL-1.1-MIT on <date>, from version <first FSL version>;
34+ - **this choice is temporary and may change in the near future** (another license, or back to an open-source one);
35+ - versions published before that date remain under MIT;
36+ - each FSL version becomes MIT two years after its release, whatever happens next.
37+ The note must be clearly separate from the license text itself (the FSL text is used unmodified, as its authors require).
38+
39+ ## Where the license is named today (checked 2026-10-26 with `git grep -w MIT`)
40+ - `LICENSE` (23 lines, MIT, "Copyright (c) 2026 k33g (Philippe Charrière) and the bots-garden organization");
41+ - `README.md`, section `## License`: "MIT — see LICENSE";
42+ - `release-notes/v0.4.0.{en,fr}.md` mention MIT: past notes, leave them as they are.
43+ Nothing in `docs/` names the license yet; `mm --about` does not either (to check when doing it).
44+
45+ ## Done when
46+ - `LICENSE` = the temporary-switch note + the official FSL-1.1-MIT text from fsl.software, unmodified, with the placeholders filled (licensor, software name, year);
47+ - README (`## License`) and a short page or paragraph in the docs EN + FR: what the license allows, the competing-use limit, the MIT conversion after two years, the temporary nature, and that older versions stay MIT;
48+ - a `CONTRIBUTING.md` with a DCO (`Signed-off-by` on each commit), so contributions can follow a later license change;
49+ - the release notes of the first FSL version say it, in the Upgrading section;
50+ - no file still calls the current code "MIT" or "open source".
51+tasks:
52+ - id: 1
53+ title: Check that no outside contribution has landed since 2026-10-26 (git log authors)
54+ state: open
55+ priority: none
56+ author:
57+ name: k33g
58+ email: ph.charriere@gmail.com
59+ createdAt: 2026-10-26T09:10:00.000Z
60+ - id: 2
61+ title: "LICENSE: the temporary-switch note, then the official FSL-1.1-MIT text with its placeholders filled"
62+ state: open
63+ priority: none
64+ author:
65+ name: k33g
66+ email: ph.charriere@gmail.com
67+ createdAt: 2026-10-26T09:10:00.000Z
68+ - id: 3
69+ title: README (## License) and docs EN + FR on the license, its limit, the MIT conversion and its temporary nature
70+ state: open
71+ priority: none
72+ author:
73+ name: k33g
74+ email: ph.charriere@gmail.com
75+ createdAt: 2026-10-26T09:10:00.000Z
76+ - id: 4
77+ title: CONTRIBUTING.md with a DCO (Signed-off-by)
78+ state: open
79+ priority: none
80+ author:
81+ name: k33g
82+ email: ph.charriere@gmail.com
83+ createdAt: 2026-10-26T09:10:00.000Z
84+ - id: 5
85+ title: Release notes of the first FSL version (Upgrading section)
86+ state: open
87+ priority: none
88+ author:
89+ name: k33g
90+ email: ph.charriere@gmail.com
91+ createdAt: 2026-10-26T09:10:00.000Z
92+comments: []
new file mode 100644
@@ -0,0 +1,92 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 173
3+title: Switch the license from MIT to FSL-1.1-MIT (temporary)
4+state: open
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T09:10:00.000Z
9+updatedAt: 2026-10-26T09:10:00.000Z
10+labels:
11+ - kind::chore
12+ - documentation
13+body: |
14+ ## Decision (2026-10-26)
15+ The next releases of mini-me are published under the **Functional Source License, version 1.1, MIT Future License** (FSL-1.1-MIT) instead of MIT. Not applied yet: this ticket prepares the change.
16+
17+ ## Why
18+ The wish: everyone can use the `mm` binary and contribute, but the code is not taken to build another product. No standard license forbids every reuse; FSL is the closest standard text:
19+ - anyone may use, read, modify and redistribute the code, for any purpose **except a "Competing Use"**: making a product or service that competes with mini-me with its code;
20+ - each version becomes **MIT two years after it is published**, automatically;
21+ - using mini-me as a library from another program (the SDK of tickets 139, 167–172) is not a competing use, so the SDK stays possible;
22+ - a short, known, unmodified text (Sentry, GitButler, Convex… use it): nothing to write or have reviewed by a lawyer, unlike a custom license.
23+
24+ Options weighed and set aside: a custom "no reuse at all" license (would forbid the SDK, needs a lawyer, not a known text), PolyForm Strict (non-commercial use only), PolyForm Shield (no conversion to open source), BSL 1.1 (more to write, up to 4 years), AGPL-3.0 and MPL-2.0 (open source, but do not stop reuse).
25+
26+ ## Things to know and to say
27+ - **FSL is not open source** in the OSI sense during the two years (it restricts a field of use); some contributors and distributions avoid such projects. Do not call mini-me "open source" in the README or the docs once switched: "source-available", or "Fair Source" (fair.io).
28+ - **Versions already published stay MIT** (every tag up to the last one before the switch: v1.0.0, v1.0.1…). Only the versions after the switch are FSL.
29+ - The switch is possible because all commits so far are by k33g (`git log`: k33g, plus "rickub" merge commits). Before switching, check that no outside contribution has landed since; if one has, its author must agree.
30+
31+ ## A note in the LICENSE file: the switch is temporary
32+ At the top of `LICENSE`, before the FSL text, a short note (EN, and FR in the docs) saying that:
33+ - mini-me moved from MIT to FSL-1.1-MIT on <date>, from version <first FSL version>;
34+ - **this choice is temporary and may change in the near future** (another license, or back to an open-source one);
35+ - versions published before that date remain under MIT;
36+ - each FSL version becomes MIT two years after its release, whatever happens next.
37+ The note must be clearly separate from the license text itself (the FSL text is used unmodified, as its authors require).
38+
39+ ## Where the license is named today (checked 2026-10-26 with `git grep -w MIT`)
40+ - `LICENSE` (23 lines, MIT, "Copyright (c) 2026 k33g (Philippe Charrière) and the bots-garden organization");
41+ - `README.md`, section `## License`: "MIT — see LICENSE";
42+ - `release-notes/v0.4.0.{en,fr}.md` mention MIT: past notes, leave them as they are.
43+ Nothing in `docs/` names the license yet; `mm --about` does not either (to check when doing it).
44+
45+ ## Done when
46+ - `LICENSE` = the temporary-switch note + the official FSL-1.1-MIT text from fsl.software, unmodified, with the placeholders filled (licensor, software name, year);
47+ - README (`## License`) and a short page or paragraph in the docs EN + FR: what the license allows, the competing-use limit, the MIT conversion after two years, the temporary nature, and that older versions stay MIT;
48+ - a `CONTRIBUTING.md` with a DCO (`Signed-off-by` on each commit), so contributions can follow a later license change;
49+ - the release notes of the first FSL version say it, in the Upgrading section;
50+ - no file still calls the current code "MIT" or "open source".
51+tasks:
52+ - id: 1
53+ title: Check that no outside contribution has landed since 2026-10-26 (git log authors)
54+ state: open
55+ priority: none
56+ author:
57+ name: k33g
58+ email: ph.charriere@gmail.com
59+ createdAt: 2026-10-26T09:10:00.000Z
60+ - id: 2
61+ title: "LICENSE: the temporary-switch note, then the official FSL-1.1-MIT text with its placeholders filled"
62+ state: open
63+ priority: none
64+ author:
65+ name: k33g
66+ email: ph.charriere@gmail.com
67+ createdAt: 2026-10-26T09:10:00.000Z
68+ - id: 3
69+ title: README (## License) and docs EN + FR on the license, its limit, the MIT conversion and its temporary nature
70+ state: open
71+ priority: none
72+ author:
73+ name: k33g
74+ email: ph.charriere@gmail.com
75+ createdAt: 2026-10-26T09:10:00.000Z
76+ - id: 4
77+ title: CONTRIBUTING.md with a DCO (Signed-off-by)
78+ state: open
79+ priority: none
80+ author:
81+ name: k33g
82+ email: ph.charriere@gmail.com
83+ createdAt: 2026-10-26T09:10:00.000Z
84+ - id: 5
85+ title: Release notes of the first FSL version (Upgrading section)
86+ state: open
87+ priority: none
88+ author:
89+ name: k33g
90+ email: ph.charriere@gmail.com
91+ createdAt: 2026-10-26T09:10:00.000Z
92+comments: []
added .tickets/issues/0174-testing-model-check-test-a-model-as-a-working-agent.yaml +86 -0
new file mode 100644
@@ -0,0 +1,86 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 174
3+title: "testing/model-check: test what a model can do as a working agent"
4+state: closed
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T10:45:00.000Z
9+updatedAt: 2026-10-26T11:00:00.000Z
10+labels:
11+ - kind::feature
12+ - documentation
13+milestone: v1.0.2
14+epic: Models
15+related:
16+ - 175
17+body: |
18+ ## What it is
19+ `testing/model-check/` measures what a model can do as a **working agent** with mm's built-in tools (bash, the file tools). The candidate (Qwen3.5 4B by default) gets eight small tasks in a scratch folder `work/`, recreated from `skills/model-check/fixtures/` before each one:
20+
21+ | Test | Capability | Checked by |
22+ |---|---|---|
23+ | T1 | tool call: write a file with one exact line | command |
24+ | T2 | tool call: count lines with a command, answer the number only | the coordinator |
25+ | T3 | instruction: a JSON array of the three primary colours of light | command (valid JSON, exact content) |
26+ | T4 | instruction: reply with exactly READY | the coordinator |
27+ | T5 | code: a Go FizzBuzz that `go run` prints right | command |
28+ | T6 | code: complete a Go package so that `go test` passes, tests untouched | command |
29+ | T7 | several steps: edit two keys of a YAML file, nothing else | command |
30+ | T8 | several steps: copy the TODO lines into a new file, say how many | command + the coordinator |
31+
32+ `fixtures/` has its own `go.mod` (`module work`) so the Go files a candidate writes never join mini-me's module. Needs `go` and `python3` on the PATH. Run: `cd testing/model-check && ../common/run.sh`.
33+
34+ ## How the testers of `testing/` work (shared by both sets)
35+ - **Two agents.** A coordinator (`agent.yaml`, Gemma 4 12B on llmman) and a `candidate` (`agent.candidate.yaml`) whose MODEL is under test; the coordinator reaches it through `mmConfigs` and `ask_agent`. To test another model: change the candidate's `model:` line (and its `sampling:`).
36+ - **A skill** tells the coordinator what to do, and a **bench script** does the work: `./bench.sh start`, then a loop of `./bench.sh next` + `ask_agent` with the prompt `next` printed, then notes and `./bench.sh report`. `next` checks the test just run before handing out the following one, so no check can be skipped; the prompts live in `skills/<skill>/tests.tsv`.
37+ - **The report** is a Markdown file, `reports/<date>-<model>.md`: tested model, coordinator, score, results table (PASS / FAIL / NOT RUN with the reason), the coordinator's notes and verdict, the list of tests. It is built from files written during the run (`reports/.results.tsv`, `reports/.notes.md`), not from the coordinator's memory. `../common/run.sh` runs a set and writes the report itself if the coordinator stops first.
38+ - **Shared code**: `testing/common/bench-lib.sh` (start, next, record, table, report) and `testing/common/run.sh`.
39+ - **Why it is built this way** (each point is a failed run of 2026-10-26): a coordinator left to call the checks itself skipped them all once; Gemma 4 12B rewrote `skills/function-calling-check/` as `function_calling_check` and looped (hence the short `./bench.sh`); it recorded a result under the placeholder id `test` (now refused); reloaded after each task, the coordinator re-read the whole conversation and the server broke its stream.
40+
41+ ## Where it stands (2026-10-26)
42+ - Two runs with the FIRST procedure (before `next`), Gemma 4 12B checking Qwen3.5 4B: with both models loaded (this llmman serves each at 262144 context), the candidate got `500 Compute error.` from T2 on; with `unloadModels: true`, T1–T5 PASS and T6 fixed by the candidate (checked by hand: PASS), then the coordinator's stream broke after a reload (15 min 23 s).
43+ - NOT re-run since the `next` loop and the report were added. The function-calling set, built the same way, went to the end.
44+ - Docs: `testing/model-check/README.{en,fr}.md`, `testing/README.{en,fr}.md`.
45+tasks:
46+ - id: 1
47+ title: "Bench: fixtures, checks T1–T8, skill, coordinator and candidate files"
48+ state: closed
49+ priority: none
50+ author:
51+ name: k33g
52+ email: ph.charriere@gmail.com
53+ createdAt: 2026-10-26T10:45:00.000Z
54+ - id: 2
55+ title: "Markdown report built by bench.sh report; run.sh writes it if the coordinator stops"
56+ state: closed
57+ priority: none
58+ author:
59+ name: k33g
60+ email: ph.charriere@gmail.com
61+ createdAt: 2026-10-26T10:45:00.000Z
62+ - id: 3
63+ title: "README EN + FR"
64+ state: closed
65+ priority: none
66+ author:
67+ name: k33g
68+ email: ph.charriere@gmail.com
69+ createdAt: 2026-10-26T10:45:00.000Z
70+ - id: 4
71+ title: "Full run with the next loop (llmman with LLMMAN_CONTEXT_LENGTH=32768; try unloadModels false); replace the README's Measured results"
72+ state: open
73+ priority: none
74+ author:
75+ name: k33g
76+ email: ph.charriere@gmail.com
77+ createdAt: 2026-10-26T10:45:00.000Z
78+ - id: 5
79+ title: "Run it on other candidate models and keep their reports"
80+ state: open
81+ priority: none
82+ author:
83+ name: k33g
84+ email: ph.charriere@gmail.com
85+ createdAt: 2026-10-26T10:45:00.000Z
86+comments: []
new file mode 100644
@@ -0,0 +1,86 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 174
3+title: "testing/model-check: test what a model can do as a working agent"
4+state: closed
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T10:45:00.000Z
9+updatedAt: 2026-10-26T11:00:00.000Z
10+labels:
11+ - kind::feature
12+ - documentation
13+milestone: v1.0.2
14+epic: Models
15+related:
16+ - 175
17+body: |
18+ ## What it is
19+ `testing/model-check/` measures what a model can do as a **working agent** with mm's built-in tools (bash, the file tools). The candidate (Qwen3.5 4B by default) gets eight small tasks in a scratch folder `work/`, recreated from `skills/model-check/fixtures/` before each one:
20+
21+ | Test | Capability | Checked by |
22+ |---|---|---|
23+ | T1 | tool call: write a file with one exact line | command |
24+ | T2 | tool call: count lines with a command, answer the number only | the coordinator |
25+ | T3 | instruction: a JSON array of the three primary colours of light | command (valid JSON, exact content) |
26+ | T4 | instruction: reply with exactly READY | the coordinator |
27+ | T5 | code: a Go FizzBuzz that `go run` prints right | command |
28+ | T6 | code: complete a Go package so that `go test` passes, tests untouched | command |
29+ | T7 | several steps: edit two keys of a YAML file, nothing else | command |
30+ | T8 | several steps: copy the TODO lines into a new file, say how many | command + the coordinator |
31+
32+ `fixtures/` has its own `go.mod` (`module work`) so the Go files a candidate writes never join mini-me's module. Needs `go` and `python3` on the PATH. Run: `cd testing/model-check && ../common/run.sh`.
33+
34+ ## How the testers of `testing/` work (shared by both sets)
35+ - **Two agents.** A coordinator (`agent.yaml`, Gemma 4 12B on llmman) and a `candidate` (`agent.candidate.yaml`) whose MODEL is under test; the coordinator reaches it through `mmConfigs` and `ask_agent`. To test another model: change the candidate's `model:` line (and its `sampling:`).
36+ - **A skill** tells the coordinator what to do, and a **bench script** does the work: `./bench.sh start`, then a loop of `./bench.sh next` + `ask_agent` with the prompt `next` printed, then notes and `./bench.sh report`. `next` checks the test just run before handing out the following one, so no check can be skipped; the prompts live in `skills/<skill>/tests.tsv`.
37+ - **The report** is a Markdown file, `reports/<date>-<model>.md`: tested model, coordinator, score, results table (PASS / FAIL / NOT RUN with the reason), the coordinator's notes and verdict, the list of tests. It is built from files written during the run (`reports/.results.tsv`, `reports/.notes.md`), not from the coordinator's memory. `../common/run.sh` runs a set and writes the report itself if the coordinator stops first.
38+ - **Shared code**: `testing/common/bench-lib.sh` (start, next, record, table, report) and `testing/common/run.sh`.
39+ - **Why it is built this way** (each point is a failed run of 2026-10-26): a coordinator left to call the checks itself skipped them all once; Gemma 4 12B rewrote `skills/function-calling-check/` as `function_calling_check` and looped (hence the short `./bench.sh`); it recorded a result under the placeholder id `test` (now refused); reloaded after each task, the coordinator re-read the whole conversation and the server broke its stream.
40+
41+ ## Where it stands (2026-10-26)
42+ - Two runs with the FIRST procedure (before `next`), Gemma 4 12B checking Qwen3.5 4B: with both models loaded (this llmman serves each at 262144 context), the candidate got `500 Compute error.` from T2 on; with `unloadModels: true`, T1–T5 PASS and T6 fixed by the candidate (checked by hand: PASS), then the coordinator's stream broke after a reload (15 min 23 s).
43+ - NOT re-run since the `next` loop and the report were added. The function-calling set, built the same way, went to the end.
44+ - Docs: `testing/model-check/README.{en,fr}.md`, `testing/README.{en,fr}.md`.
45+tasks:
46+ - id: 1
47+ title: "Bench: fixtures, checks T1–T8, skill, coordinator and candidate files"
48+ state: closed
49+ priority: none
50+ author:
51+ name: k33g
52+ email: ph.charriere@gmail.com
53+ createdAt: 2026-10-26T10:45:00.000Z
54+ - id: 2
55+ title: "Markdown report built by bench.sh report; run.sh writes it if the coordinator stops"
56+ state: closed
57+ priority: none
58+ author:
59+ name: k33g
60+ email: ph.charriere@gmail.com
61+ createdAt: 2026-10-26T10:45:00.000Z
62+ - id: 3
63+ title: "README EN + FR"
64+ state: closed
65+ priority: none
66+ author:
67+ name: k33g
68+ email: ph.charriere@gmail.com
69+ createdAt: 2026-10-26T10:45:00.000Z
70+ - id: 4
71+ title: "Full run with the next loop (llmman with LLMMAN_CONTEXT_LENGTH=32768; try unloadModels false); replace the README's Measured results"
72+ state: open
73+ priority: none
74+ author:
75+ name: k33g
76+ email: ph.charriere@gmail.com
77+ createdAt: 2026-10-26T10:45:00.000Z
78+ - id: 5
79+ title: "Run it on other candidate models and keep their reports"
80+ state: open
81+ priority: none
82+ author:
83+ name: k33g
84+ email: ph.charriere@gmail.com
85+ createdAt: 2026-10-26T10:45:00.000Z
86+comments: []
added .tickets/issues/0175-testing-function-calling-test-a-model-s-tool-calls.yaml +88 -0
new file mode 100644
@@ -0,0 +1,88 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 175
3+title: "testing/function-calling: test a model's function calling, with tools declared in the agent file"
4+state: closed
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T10:45:00.000Z
9+updatedAt: 2026-10-26T11:00:00.000Z
10+labels:
11+ - kind::feature
12+ - documentation
13+milestone: v1.0.2
14+epic: Models
15+related:
16+ - 174
17+body: |
18+ ## What it is
19+ `testing/function-calling/` measures **function calling alone**. The candidate has NO built-in tool: only six tools declared in its agent file (`tools:` in `agent.candidate.yaml`) — `get_weather`, `convert_currency`, `find_user`, `list_orders`, `create_reminder`, `book_table` — with string, enum (with a default), number, integer and boolean parameters. Every tool runs `tools/call.sh`, which logs the call (tool name and arguments as mm passed them) in `work/calls.jsonl` and answers a fixed, made-up result. `skills/function-calling-check/check.py` reads that log:
20+
21+ | Test | Capability | PASS when |
22+ |---|---|---|
23+ | F1 | one call, a string argument | one `get_weather(city=Paris)`, default unit kept |
24+ | F2 | an enum argument | one `get_weather(Tokyo, fahrenheit)` |
25+ | F3 | a number and two enums | one `convert_currency(250, EUR, USD)` |
26+ | F4 | a boolean argument | one `create_reminder` with `urgent=true` |
27+ | F5 | chained calls | `find_user`, then `list_orders` with the user_id it returned |
28+ | F6 | two calls in one request | two `get_weather`, Rome and Oslo |
29+ | F7 | no call when no tool fits | no call for "12 times 12" |
30+ | F8 | no invented arguments | no `book_table` when nothing was given |
31+
32+ To test other tools, change them in `agent.candidate.yaml` and their checks in `check.py`. Needs `python3`. Run: `cd testing/function-calling && ../common/run.sh`.
33+
34+ ## How the testers of `testing/` work (shared by both sets)
35+ - **Two agents.** A coordinator (`agent.yaml`, Gemma 4 12B on llmman) and a `candidate` (`agent.candidate.yaml`) whose MODEL is under test; the coordinator reaches it through `mmConfigs` and `ask_agent`. To test another model: change the candidate's `model:` line (and its `sampling:`).
36+ - **A skill** tells the coordinator what to do, and a **bench script** does the work: `./bench.sh start`, then a loop of `./bench.sh next` + `ask_agent` with the prompt `next` printed, then notes and `./bench.sh report`. `next` checks the test just run before handing out the following one, so no check can be skipped; the prompts live in `skills/<skill>/tests.tsv`.
37+ - **The report** is a Markdown file, `reports/<date>-<model>.md`: tested model, coordinator, score, results table (PASS / FAIL / NOT RUN with the reason), the coordinator's notes and verdict, the list of tests. It is built from files written during the run (`reports/.results.tsv`, `reports/.notes.md`), not from the coordinator's memory. `../common/run.sh` runs a set and writes the report itself if the coordinator stops first.
38+ - **Shared code**: `testing/common/bench-lib.sh` (start, next, record, table, report) and `testing/common/run.sh`.
39+ - **Why it is built this way** (each point is a failed run of 2026-10-26): a coordinator left to call the checks itself skipped them all once; Gemma 4 12B rewrote `skills/function-calling-check/` as `function_calling_check` and looped (hence the short `./bench.sh`); it recorded a result under the placeholder id `test` (now refused); reloaded after each task, the coordinator re-read the whole conversation and the server broke its stream.
40+
41+ ## Where it stands (2026-10-26)
42+ - Candidate: Luciole 1B Instruct 1.1 (`hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M`, user's choice; its template takes tools, Hermes-style; sampling 0.7 / 0.9 from the GGUF).
43+ - Full run: **5 of 8**, 13 min 04 s, report written by the coordinator: `testing/function-calling/reports/2026-10-26-0901-Luciole-1B-Instruct-1.1-GGUF-Q4_K_M.md`. FAIL: F3 runaway tool call, F5 `max_tokens` hit while writing a call, F7 a tool called for 12 × 12 then a loop.
44+ - Luciole varies from run to run (F3 passed twice, failed twice — once with 42 calls with made-up amounts; F2 once two calls; F5 once invented ids): one run is a sample.
45+ - Both models stay loaded during a run (`unloadModels: false`).
46+ - Docs: `testing/function-calling/README.{en,fr}.md`, `testing/README.{en,fr}.md`.
47+tasks:
48+ - id: 1
49+ title: "Candidate with six declared tools, tools/call.sh logging the calls, check.py"
50+ state: closed
51+ priority: none
52+ author:
53+ name: k33g
54+ email: ph.charriere@gmail.com
55+ createdAt: 2026-10-26T10:45:00.000Z
56+ - id: 2
57+ title: "The next loop, the Markdown report, run.sh"
58+ state: closed
59+ priority: none
60+ author:
61+ name: k33g
62+ email: ph.charriere@gmail.com
63+ createdAt: 2026-10-26T10:45:00.000Z
64+ - id: 3
65+ title: "Full run with Luciole 1B (5/8) and README EN + FR"
66+ state: closed
67+ priority: none
68+ author:
69+ name: k33g
70+ email: ph.charriere@gmail.com
71+ createdAt: 2026-10-26T10:45:00.000Z
72+ - id: 4
73+ title: "Several runs of the same model, to show its variance in the report or README"
74+ state: open
75+ priority: none
76+ author:
77+ name: k33g
78+ email: ph.charriere@gmail.com
79+ createdAt: 2026-10-26T10:45:00.000Z
80+ - id: 5
81+ title: "Run it on other candidate models (Qwen3.5 4B, Mellum2…) and compare"
82+ state: open
83+ priority: none
84+ author:
85+ name: k33g
86+ email: ph.charriere@gmail.com
87+ createdAt: 2026-10-26T10:45:00.000Z
88+comments: []
new file mode 100644
@@ -0,0 +1,88 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 175
3+title: "testing/function-calling: test a model's function calling, with tools declared in the agent file"
4+state: closed
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-26T10:45:00.000Z
9+updatedAt: 2026-10-26T11:00:00.000Z
10+labels:
11+ - kind::feature
12+ - documentation
13+milestone: v1.0.2
14+epic: Models
15+related:
16+ - 174
17+body: |
18+ ## What it is
19+ `testing/function-calling/` measures **function calling alone**. The candidate has NO built-in tool: only six tools declared in its agent file (`tools:` in `agent.candidate.yaml`) — `get_weather`, `convert_currency`, `find_user`, `list_orders`, `create_reminder`, `book_table` — with string, enum (with a default), number, integer and boolean parameters. Every tool runs `tools/call.sh`, which logs the call (tool name and arguments as mm passed them) in `work/calls.jsonl` and answers a fixed, made-up result. `skills/function-calling-check/check.py` reads that log:
20+
21+ | Test | Capability | PASS when |
22+ |---|---|---|
23+ | F1 | one call, a string argument | one `get_weather(city=Paris)`, default unit kept |
24+ | F2 | an enum argument | one `get_weather(Tokyo, fahrenheit)` |
25+ | F3 | a number and two enums | one `convert_currency(250, EUR, USD)` |
26+ | F4 | a boolean argument | one `create_reminder` with `urgent=true` |
27+ | F5 | chained calls | `find_user`, then `list_orders` with the user_id it returned |
28+ | F6 | two calls in one request | two `get_weather`, Rome and Oslo |
29+ | F7 | no call when no tool fits | no call for "12 times 12" |
30+ | F8 | no invented arguments | no `book_table` when nothing was given |
31+
32+ To test other tools, change them in `agent.candidate.yaml` and their checks in `check.py`. Needs `python3`. Run: `cd testing/function-calling && ../common/run.sh`.
33+
34+ ## How the testers of `testing/` work (shared by both sets)
35+ - **Two agents.** A coordinator (`agent.yaml`, Gemma 4 12B on llmman) and a `candidate` (`agent.candidate.yaml`) whose MODEL is under test; the coordinator reaches it through `mmConfigs` and `ask_agent`. To test another model: change the candidate's `model:` line (and its `sampling:`).
36+ - **A skill** tells the coordinator what to do, and a **bench script** does the work: `./bench.sh start`, then a loop of `./bench.sh next` + `ask_agent` with the prompt `next` printed, then notes and `./bench.sh report`. `next` checks the test just run before handing out the following one, so no check can be skipped; the prompts live in `skills/<skill>/tests.tsv`.
37+ - **The report** is a Markdown file, `reports/<date>-<model>.md`: tested model, coordinator, score, results table (PASS / FAIL / NOT RUN with the reason), the coordinator's notes and verdict, the list of tests. It is built from files written during the run (`reports/.results.tsv`, `reports/.notes.md`), not from the coordinator's memory. `../common/run.sh` runs a set and writes the report itself if the coordinator stops first.
38+ - **Shared code**: `testing/common/bench-lib.sh` (start, next, record, table, report) and `testing/common/run.sh`.
39+ - **Why it is built this way** (each point is a failed run of 2026-10-26): a coordinator left to call the checks itself skipped them all once; Gemma 4 12B rewrote `skills/function-calling-check/` as `function_calling_check` and looped (hence the short `./bench.sh`); it recorded a result under the placeholder id `test` (now refused); reloaded after each task, the coordinator re-read the whole conversation and the server broke its stream.
40+
41+ ## Where it stands (2026-10-26)
42+ - Candidate: Luciole 1B Instruct 1.1 (`hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M`, user's choice; its template takes tools, Hermes-style; sampling 0.7 / 0.9 from the GGUF).
43+ - Full run: **5 of 8**, 13 min 04 s, report written by the coordinator: `testing/function-calling/reports/2026-10-26-0901-Luciole-1B-Instruct-1.1-GGUF-Q4_K_M.md`. FAIL: F3 runaway tool call, F5 `max_tokens` hit while writing a call, F7 a tool called for 12 × 12 then a loop.
44+ - Luciole varies from run to run (F3 passed twice, failed twice — once with 42 calls with made-up amounts; F2 once two calls; F5 once invented ids): one run is a sample.
45+ - Both models stay loaded during a run (`unloadModels: false`).
46+ - Docs: `testing/function-calling/README.{en,fr}.md`, `testing/README.{en,fr}.md`.
47+tasks:
48+ - id: 1
49+ title: "Candidate with six declared tools, tools/call.sh logging the calls, check.py"
50+ state: closed
51+ priority: none
52+ author:
53+ name: k33g
54+ email: ph.charriere@gmail.com
55+ createdAt: 2026-10-26T10:45:00.000Z
56+ - id: 2
57+ title: "The next loop, the Markdown report, run.sh"
58+ state: closed
59+ priority: none
60+ author:
61+ name: k33g
62+ email: ph.charriere@gmail.com
63+ createdAt: 2026-10-26T10:45:00.000Z
64+ - id: 3
65+ title: "Full run with Luciole 1B (5/8) and README EN + FR"
66+ state: closed
67+ priority: none
68+ author:
69+ name: k33g
70+ email: ph.charriere@gmail.com
71+ createdAt: 2026-10-26T10:45:00.000Z
72+ - id: 4
73+ title: "Several runs of the same model, to show its variance in the report or README"
74+ state: open
75+ priority: none
76+ author:
77+ name: k33g
78+ email: ph.charriere@gmail.com
79+ createdAt: 2026-10-26T10:45:00.000Z
80+ - id: 5
81+ title: "Run it on other candidate models (Qwen3.5 4B, Mellum2…) and compare"
82+ state: open
83+ priority: none
84+ author:
85+ name: k33g
86+ email: ph.charriere@gmail.com
87+ createdAt: 2026-10-26T10:45:00.000Z
88+comments: []
modified README.md +2 -0
@@ -141,6 +141,8 @@ The `demos/` directory holds ready-made set-ups (most with a README explaining h
141141 | `demos/mini-mes/` | small teams of agents (`riker` coordinating a specifier and a coder, `gopher` a Go specialist) |
142142 | `demos/little-coder/` | a Go + htmx web-app generator for a small local model (Mellum2 12B on llmman, 32 GB RAM): a tested starter, skills, `AGENTS.md`, measured results |
143143 | `demos/team/` | a team of four agents (lead, specifier, coder, tester) on one small model (Ornith 1.5 9B on llmman) building Go + htmx apps through spec and test files, with measured results |
144+| `demo.acp.sdk/` | mini-me as a library from other languages, by driving `mm -acp` over ACP: `nodejs/minime.js` (ticket 167) — `Agent.start`, `complete`, `stream`, cancel with an `AbortSignal`, a permission callback; no dependency, 11 tests with a fake mm, four examples measured on llmman |
145+| `testing/` | model testers run by mm itself: a coordinator (Gemma 4 12B) follows a skill, gives small tasks to a `candidate` agent whose model is under test (Qwen3.5 4B for model-check, Luciole 1B for function-calling), checks each result with a command and writes a Markdown report — `model-check/` (tool calls, exact instructions, Go code that must run, several steps) and `function-calling/` (only tools declared in the candidate's `tools:`) |
144146 | `dev.team/` | a team of five agents for a 32 GB laptop (ticket 152): a Qwen3.5 4B coordinator hands each request to a reformulator, a coder, a reviewer or a tutor (Gemma 4 12B, Mellum2 12B); hooks keep one model in memory at a time; measured: bug fixes, lessons and reviews work, new features not yet |
145147 | `demos/blog-team/` | the smallest team: a lead, a coder and a reviewer in 75 lines of YAML, one model (Mellum2 12B Q8_0 on llmman), written for a blog post |
146148 | `demos/translator/` | a French ↔ English translator (Gemma 4 E4B on llmman): `AGENTS.md` for the style, skills for technical texts, e-mails and idioms, `$styles` for professional / informal / Slack versions, `translate.sh` for files |
@@ -141,6 +141,8 @@ The `demos/` directory holds ready-made set-ups (most with a README explaining h
141 | `demos/mini-mes/` | small teams of agents (`riker` coordinating a specifier and a coder, `gopher` a Go specialist) |141 | `demos/mini-mes/` | small teams of agents (`riker` coordinating a specifier and a coder, `gopher` a Go specialist) |
142 | `demos/little-coder/` | a Go + htmx web-app generator for a small local model (Mellum2 12B on llmman, 32 GB RAM): a tested starter, skills, `AGENTS.md`, measured results |142 | `demos/little-coder/` | a Go + htmx web-app generator for a small local model (Mellum2 12B on llmman, 32 GB RAM): a tested starter, skills, `AGENTS.md`, measured results |
143 | `demos/team/` | a team of four agents (lead, specifier, coder, tester) on one small model (Ornith 1.5 9B on llmman) building Go + htmx apps through spec and test files, with measured results |143 | `demos/team/` | a team of four agents (lead, specifier, coder, tester) on one small model (Ornith 1.5 9B on llmman) building Go + htmx apps through spec and test files, with measured results |
144+| `demo.acp.sdk/` | mini-me as a library from other languages, by driving `mm -acp` over ACP: `nodejs/minime.js` (ticket 167) — `Agent.start`, `complete`, `stream`, cancel with an `AbortSignal`, a permission callback; no dependency, 11 tests with a fake mm, four examples measured on llmman |
145+| `testing/` | model testers run by mm itself: a coordinator (Gemma 4 12B) follows a skill, gives small tasks to a `candidate` agent whose model is under test (Qwen3.5 4B for model-check, Luciole 1B for function-calling), checks each result with a command and writes a Markdown report — `model-check/` (tool calls, exact instructions, Go code that must run, several steps) and `function-calling/` (only tools declared in the candidate's `tools:`) |
144 | `dev.team/` | a team of five agents for a 32 GB laptop (ticket 152): a Qwen3.5 4B coordinator hands each request to a reformulator, a coder, a reviewer or a tutor (Gemma 4 12B, Mellum2 12B); hooks keep one model in memory at a time; measured: bug fixes, lessons and reviews work, new features not yet |146 | `dev.team/` | a team of five agents for a 32 GB laptop (ticket 152): a Qwen3.5 4B coordinator hands each request to a reformulator, a coder, a reviewer or a tutor (Gemma 4 12B, Mellum2 12B); hooks keep one model in memory at a time; measured: bug fixes, lessons and reviews work, new features not yet |
145 | `demos/blog-team/` | the smallest team: a lead, a coder and a reviewer in 75 lines of YAML, one model (Mellum2 12B Q8_0 on llmman), written for a blog post |147 | `demos/blog-team/` | the smallest team: a lead, a coder and a reviewer in 75 lines of YAML, one model (Mellum2 12B Q8_0 on llmman), written for a blog post |
146 | `demos/translator/` | a French ↔ English translator (Gemma 4 E4B on llmman): `AGENTS.md` for the style, skills for technical texts, e-mails and idioms, `$styles` for professional / informal / Slack versions, `translate.sh` for files |148 | `demos/translator/` | a French ↔ English translator (Gemma 4 E4B on llmman): `AGENTS.md` for the style, skills for technical texts, e-mails and idioms, `$styles` for professional / informal / Slack versions, `translate.sh` for files |
added demo.acp.sdk/README.en.md +23 -0
new file mode 100644
@@ -0,0 +1,23 @@
1+# Using mini-me as a library, over ACP
2+
3+*[Version française](README.fr.md)*
4+
5+Programs in other languages use mini-me as a library by driving `mm -acp`: the same JSON-RPC protocol an editor such as Zed speaks, on the child process's stdin and stdout. Nothing changes in mm; every agent file works, with its tools, hooks, MCP servers and other agents.
6+
7+| Folder | Language | Status |
8+|---|---|---|
9+| [`nodejs/`](nodejs/README.en.md) | Node.js (ticket 167) | done: `minime.js`, four examples, tests |
10+| `go/`, `python/`, `rust/`, `swift/`, `nushell/` | tickets 168–172 | not started |
11+
12+What every client does:
13+
14+| Wanted | ACP message |
15+|---|---|
16+| start | `initialize`, then `session/new {cwd}` |
17+| run a completion | `session/prompt` |
18+| stream | the `session/update` notifications: `agent_message_chunk`, `agent_thought_chunk`, `tool_call`, `tool_call_update` |
19+| not streamed | join the text chunks until `session/prompt` answers with its `stopReason` |
20+| cancel | the `session/cancel` notification; the prompt then answers `stopReason: "cancelled"` |
21+| confirm a tool (`yolo: false`) | answer mm's `session/request_permission` request |
22+
23+Limits shared by all: the `mm` binary must be installed, it runs locally (stdio), and the agent is a file (`mm -acp -config <file>`). For a Go program without the binary, see ticket 139 (an in-process SDK).
new file mode 100644
@@ -0,0 +1,23 @@
1+# Using mini-me as a library, over ACP
2+
3+*[Version française](README.fr.md)*
4+
5+Programs in other languages use mini-me as a library by driving `mm -acp`: the same JSON-RPC protocol an editor such as Zed speaks, on the child process's stdin and stdout. Nothing changes in mm; every agent file works, with its tools, hooks, MCP servers and other agents.
6+
7+| Folder | Language | Status |
8+|---|---|---|
9+| [`nodejs/`](nodejs/README.en.md) | Node.js (ticket 167) | done: `minime.js`, four examples, tests |
10+| `go/`, `python/`, `rust/`, `swift/`, `nushell/` | tickets 168–172 | not started |
11+
12+What every client does:
13+
14+| Wanted | ACP message |
15+|---|---|
16+| start | `initialize`, then `session/new {cwd}` |
17+| run a completion | `session/prompt` |
18+| stream | the `session/update` notifications: `agent_message_chunk`, `agent_thought_chunk`, `tool_call`, `tool_call_update` |
19+| not streamed | join the text chunks until `session/prompt` answers with its `stopReason` |
20+| cancel | the `session/cancel` notification; the prompt then answers `stopReason: "cancelled"` |
21+| confirm a tool (`yolo: false`) | answer mm's `session/request_permission` request |
22+
23+Limits shared by all: the `mm` binary must be installed, it runs locally (stdio), and the agent is a file (`mm -acp -config <file>`). For a Go program without the binary, see ticket 139 (an in-process SDK).
added demo.acp.sdk/README.fr.md +23 -0
new file mode 100644
@@ -0,0 +1,23 @@
1+# Utiliser mini-me comme bibliothèque, par ACP
2+
3+*[English version](README.en.md)*
4+
5+Des programmes écrits dans d'autres langages utilisent mini-me comme bibliothèque en pilotant `mm -acp` : le même protocole JSON-RPC que parle un éditeur comme Zed, sur l'entrée et la sortie standard du processus enfant. Rien ne change dans mm ; tout fichier d'agent fonctionne, avec ses outils, ses hooks, ses serveurs MCP et ses autres agents.
6+
7+| Dossier | Langage | État |
8+|---|---|---|
9+| [`nodejs/`](nodejs/README.fr.md) | Node.js (ticket 167) | fait : `minime.js`, quatre exemples, des tests |
10+| `go/`, `python/`, `rust/`, `swift/`, `nushell/` | tickets 168–172 | pas commencés |
11+
12+Ce que fait chaque client :
13+
14+| Besoin | Message ACP |
15+|---|---|
16+| démarrer | `initialize`, puis `session/new {cwd}` |
17+| lancer une complétion | `session/prompt` |
18+| stream | les notifications `session/update` : `agent_message_chunk`, `agent_thought_chunk`, `tool_call`, `tool_call_update` |
19+| sans stream | concaténer les morceaux de texte jusqu'à la réponse de `session/prompt` et son `stopReason` |
20+| annuler | la notification `session/cancel` ; le prompt répond alors `stopReason: "cancelled"` |
21+| confirmer un outil (`yolo: false`) | répondre à la requête `session/request_permission` de mm |
22+
23+Limites communes : le binaire `mm` doit être installé, tout tourne en local (stdio), et l'agent est un fichier (`mm -acp -config <fichier>`). Pour un programme Go sans le binaire, voir le ticket 139 (un SDK en mémoire).
new file mode 100644
@@ -0,0 +1,23 @@
1+# Utiliser mini-me comme bibliothèque, par ACP
2+
3+*[English version](README.en.md)*
4+
5+Des programmes écrits dans d'autres langages utilisent mini-me comme bibliothèque en pilotant `mm -acp` : le même protocole JSON-RPC que parle un éditeur comme Zed, sur l'entrée et la sortie standard du processus enfant. Rien ne change dans mm ; tout fichier d'agent fonctionne, avec ses outils, ses hooks, ses serveurs MCP et ses autres agents.
6+
7+| Dossier | Langage | État |
8+|---|---|---|
9+| [`nodejs/`](nodejs/README.fr.md) | Node.js (ticket 167) | fait : `minime.js`, quatre exemples, des tests |
10+| `go/`, `python/`, `rust/`, `swift/`, `nushell/` | tickets 168–172 | pas commencés |
11+
12+Ce que fait chaque client :
13+
14+| Besoin | Message ACP |
15+|---|---|
16+| démarrer | `initialize`, puis `session/new {cwd}` |
17+| lancer une complétion | `session/prompt` |
18+| stream | les notifications `session/update` : `agent_message_chunk`, `agent_thought_chunk`, `tool_call`, `tool_call_update` |
19+| sans stream | concaténer les morceaux de texte jusqu'à la réponse de `session/prompt` et son `stopReason` |
20+| annuler | la notification `session/cancel` ; le prompt répond alors `stopReason: "cancelled"` |
21+| confirmer un outil (`yolo: false`) | répondre à la requête `session/request_permission` de mm |
22+
23+Limites communes : le binaire `mm` doit être installé, tout tourne en local (stdio), et l'agent est un fichier (`mm -acp -config <fichier>`). Pour un programme Go sans le binaire, voir le ticket 139 (un SDK en mémoire).
added demo.acp.sdk/nodejs/README.en.md +86 -0
new file mode 100644
@@ -0,0 +1,86 @@
1+# mini-me from Node.js: `minime.js`
2+
3+*[Version française](README.fr.md)*
4+
5+A Node.js client that uses mini-me as a library by driving `mm -acp` over ACP (ticket 167). One file, `minime.js`, no dependency (Node ≥ 20: `child_process` and `readline`). See [`../README.en.md`](../README.en.md) for the ACP messages behind it.
6+
7+```js
8+import { Agent } from "./minime.js";
9+
10+const agent = await Agent.start({ config: "agent.yaml" });
11+
12+const { text } = await agent.complete("Explain this repository"); // not streamed
13+
14+for await (const ev of agent.stream("And the tests?")) { // streamed
15+ if (ev.kind === "text") process.stdout.write(ev.text);
16+}
17+
18+const ac = new AbortController();
19+setTimeout(() => ac.abort(), 3000);
20+await agent.complete("A long task", { signal: ac.signal }); // cancelled
21+
22+await agent.close();
23+```
24+
25+## Run the examples
26+
27+```bash
28+llmman pull hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M
29+llmman serve
30+
31+cd demo.acp.sdk/nodejs
32+node examples/complete.js # not streamed
33+node examples/stream.js # streamed, with the tool calls
34+node examples/cancel.js # cancelled after 3 seconds
35+node examples/permission.js # a command refused, then allowed
36+```
37+
38+`mm` must be on the PATH, or named by `MM_BIN`. The examples use `agent.yaml` (Qwen3.5 4B on llmman, bash only, `yolo: false`); any agent file works.
39+
40+## The API
41+
42+| | |
43+|---|---|
44+| `await Agent.start({ config, cwd?, mm?, onPermission?, onStderr? })` | starts `mm -acp -config <config>` (made absolute), then `initialize` and `session/new {cwd}`. `mm` defaults to `$MM_BIN`, then `mm`. |
45+| `await agent.complete(prompt, { signal?, onEvent? })` | one turn; resolves to `{ text, stopReason, usage }` once it is over. `onEvent` sees every event as it comes. |
46+| `agent.stream(prompt, { signal? })` | one turn as an async iterator of events; the last one is `done`. Leaving the loop early cancels the turn. |
47+| `signal` (an `AbortSignal`) | aborting it sends `session/cancel`; the turn ends with `stopReason: "cancelled"` and the text so far. |
48+| `onPermission(call)` | asked when the agent wants to run a tool that needs a confirmation (`yolo: false`). Return `true` (allow once), `false` (reject once) or an ACP `optionId` (`"allow_always"`, `"never"`). **Without it, every such call is refused.** |
49+| `onStderr(line)` | mm's stderr (banner, warnings, `[acp]` lines). Otherwise it is read and dropped; its last line is added to the error when mm exits. |
50+| `await agent.reset()` | a new session on the same process: the conversation starts over. |
51+| `await agent.close()` | closes mm's stdin and waits for it to exit. |
52+| `agent.sessionId` | the ACP session id. |
53+
54+The events: `{kind: "text", text}`, `{kind: "thinking", text}`, `{kind: "tool_call", id, title, toolKind, input}`, `{kind: "tool_result", id, status, output}`, `{kind: "done", stopReason, usage, text}`.
55+
56+Rules worth knowing:
57+
58+- **One Agent = one mm process = one conversation.** Calls continue it, as in an editor. One turn at a time per Agent (a second one throws `already running`); start several Agents to work in parallel.
59+- **A turn mm ends with an error rejects** with mm's own words, e.g. `minime: Internal error (-32603): {"error":"the model repeated the same bash call 6 times…"}`.
60+- **A cancelled question is dropped from the conversation by mm**: the next question does not know about it.
61+- The client advertises no file-system, terminal or form capability: mm needs none, and its `ask_user` tool is not offered.
62+
63+## Tests
64+
65+```bash
66+npm test # node --test test/minime.test.js — 11 tests, no model, no mm
67+```
68+
69+`test/fake-mm.js` plays mm's side of ACP with the message shapes captured from mm v1.0.1: answers, a tool call with its permission, a slow turn to cancel, a crash. Not a bare `node --test`: it would also run `fake-mm.js` as a test and wait forever.
70+
71+## Measured (2026-10-26, llmman on the author's host, mm v1.0.1 built from the working tree)
72+
73+| Example | Result |
74+|---|---|
75+| `complete.js` | one-sentence answer, `end_turn`, 663 tokens, **2.9 s** |
76+| `stream.js` | `find examples -name "*.js" \| wc -l` allowed by the callback (read-only), output `4`, answer streamed chunk by chunk, `end_turn` |
77+| `cancel.js` | story cut after **3.0 s** (`cancelled`, 383 characters kept); the next question on the same Agent answered at once — without knowing the story, which mm dropped |
78+| `permission.js` | `pwd` refused: the model **invented** an answer (`/root`); allowed: the real directory. A refusal does not stop a small model from guessing. |
79+
80+Seen in three runs before the callback allowed `find`: the model retried the same refused command until mm's loop detector stopped the turn (6 identical calls); `stream()` rejected with that message — hence the `try/catch` in `stream.js`.
81+
82+## Not done
83+
84+- The official ACP TypeScript library was not tried (no network in the sandbox where this was written): raw JSON-RPC keeps the client dependency-free.
85+- Not a published npm package: the `mm` binary would have to ship with it, per platform.
86+- `session/load` / `session/resume` (stored sessions), images and file attachments in prompts, `ask_user` forms: not exposed.
new file mode 100644
@@ -0,0 +1,86 @@
1+# mini-me from Node.js: `minime.js`
2+
3+*[Version française](README.fr.md)*
4+
5+A Node.js client that uses mini-me as a library by driving `mm -acp` over ACP (ticket 167). One file, `minime.js`, no dependency (Node ≥ 20: `child_process` and `readline`). See [`../README.en.md`](../README.en.md) for the ACP messages behind it.
6+
7+```js
8+import { Agent } from "./minime.js";
9+
10+const agent = await Agent.start({ config: "agent.yaml" });
11+
12+const { text } = await agent.complete("Explain this repository"); // not streamed
13+
14+for await (const ev of agent.stream("And the tests?")) { // streamed
15+ if (ev.kind === "text") process.stdout.write(ev.text);
16+}
17+
18+const ac = new AbortController();
19+setTimeout(() => ac.abort(), 3000);
20+await agent.complete("A long task", { signal: ac.signal }); // cancelled
21+
22+await agent.close();
23+```
24+
25+## Run the examples
26+
27+```bash
28+llmman pull hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M
29+llmman serve
30+
31+cd demo.acp.sdk/nodejs
32+node examples/complete.js # not streamed
33+node examples/stream.js # streamed, with the tool calls
34+node examples/cancel.js # cancelled after 3 seconds
35+node examples/permission.js # a command refused, then allowed
36+```
37+
38+`mm` must be on the PATH, or named by `MM_BIN`. The examples use `agent.yaml` (Qwen3.5 4B on llmman, bash only, `yolo: false`); any agent file works.
39+
40+## The API
41+
42+| | |
43+|---|---|
44+| `await Agent.start({ config, cwd?, mm?, onPermission?, onStderr? })` | starts `mm -acp -config <config>` (made absolute), then `initialize` and `session/new {cwd}`. `mm` defaults to `$MM_BIN`, then `mm`. |
45+| `await agent.complete(prompt, { signal?, onEvent? })` | one turn; resolves to `{ text, stopReason, usage }` once it is over. `onEvent` sees every event as it comes. |
46+| `agent.stream(prompt, { signal? })` | one turn as an async iterator of events; the last one is `done`. Leaving the loop early cancels the turn. |
47+| `signal` (an `AbortSignal`) | aborting it sends `session/cancel`; the turn ends with `stopReason: "cancelled"` and the text so far. |
48+| `onPermission(call)` | asked when the agent wants to run a tool that needs a confirmation (`yolo: false`). Return `true` (allow once), `false` (reject once) or an ACP `optionId` (`"allow_always"`, `"never"`). **Without it, every such call is refused.** |
49+| `onStderr(line)` | mm's stderr (banner, warnings, `[acp]` lines). Otherwise it is read and dropped; its last line is added to the error when mm exits. |
50+| `await agent.reset()` | a new session on the same process: the conversation starts over. |
51+| `await agent.close()` | closes mm's stdin and waits for it to exit. |
52+| `agent.sessionId` | the ACP session id. |
53+
54+The events: `{kind: "text", text}`, `{kind: "thinking", text}`, `{kind: "tool_call", id, title, toolKind, input}`, `{kind: "tool_result", id, status, output}`, `{kind: "done", stopReason, usage, text}`.
55+
56+Rules worth knowing:
57+
58+- **One Agent = one mm process = one conversation.** Calls continue it, as in an editor. One turn at a time per Agent (a second one throws `already running`); start several Agents to work in parallel.
59+- **A turn mm ends with an error rejects** with mm's own words, e.g. `minime: Internal error (-32603): {"error":"the model repeated the same bash call 6 times…"}`.
60+- **A cancelled question is dropped from the conversation by mm**: the next question does not know about it.
61+- The client advertises no file-system, terminal or form capability: mm needs none, and its `ask_user` tool is not offered.
62+
63+## Tests
64+
65+```bash
66+npm test # node --test test/minime.test.js — 11 tests, no model, no mm
67+```
68+
69+`test/fake-mm.js` plays mm's side of ACP with the message shapes captured from mm v1.0.1: answers, a tool call with its permission, a slow turn to cancel, a crash. Not a bare `node --test`: it would also run `fake-mm.js` as a test and wait forever.
70+
71+## Measured (2026-10-26, llmman on the author's host, mm v1.0.1 built from the working tree)
72+
73+| Example | Result |
74+|---|---|
75+| `complete.js` | one-sentence answer, `end_turn`, 663 tokens, **2.9 s** |
76+| `stream.js` | `find examples -name "*.js" \| wc -l` allowed by the callback (read-only), output `4`, answer streamed chunk by chunk, `end_turn` |
77+| `cancel.js` | story cut after **3.0 s** (`cancelled`, 383 characters kept); the next question on the same Agent answered at once — without knowing the story, which mm dropped |
78+| `permission.js` | `pwd` refused: the model **invented** an answer (`/root`); allowed: the real directory. A refusal does not stop a small model from guessing. |
79+
80+Seen in three runs before the callback allowed `find`: the model retried the same refused command until mm's loop detector stopped the turn (6 identical calls); `stream()` rejected with that message — hence the `try/catch` in `stream.js`.
81+
82+## Not done
83+
84+- The official ACP TypeScript library was not tried (no network in the sandbox where this was written): raw JSON-RPC keeps the client dependency-free.
85+- Not a published npm package: the `mm` binary would have to ship with it, per platform.
86+- `session/load` / `session/resume` (stored sessions), images and file attachments in prompts, `ask_user` forms: not exposed.
added demo.acp.sdk/nodejs/README.fr.md +86 -0
new file mode 100644
@@ -0,0 +1,86 @@
1+# mini-me depuis Node.js : `minime.js`
2+
3+*[English version](README.en.md)*
4+
5+Un client Node.js qui utilise mini-me comme bibliothèque en pilotant `mm -acp` par ACP (ticket 167). Un fichier, `minime.js`, aucune dépendance (Node ≥ 20 : `child_process` et `readline`). Voir [`../README.fr.md`](../README.fr.md) pour les messages ACP utilisés.
6+
7+```js
8+import { Agent } from "./minime.js";
9+
10+const agent = await Agent.start({ config: "agent.yaml" });
11+
12+const { text } = await agent.complete("Explain this repository"); // sans stream
13+
14+for await (const ev of agent.stream("And the tests?")) { // en stream
15+ if (ev.kind === "text") process.stdout.write(ev.text);
16+}
17+
18+const ac = new AbortController();
19+setTimeout(() => ac.abort(), 3000);
20+await agent.complete("A long task", { signal: ac.signal }); // annulé
21+
22+await agent.close();
23+```
24+
25+## Lancer les exemples
26+
27+```bash
28+llmman pull hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M
29+llmman serve
30+
31+cd demo.acp.sdk/nodejs
32+node examples/complete.js # sans stream
33+node examples/stream.js # en stream, avec les appels d'outils
34+node examples/cancel.js # annulé au bout de 3 secondes
35+node examples/permission.js # une commande refusée, puis autorisée
36+```
37+
38+`mm` doit être dans le PATH, ou désigné par `MM_BIN`. Les exemples utilisent `agent.yaml` (Qwen3.5 4B sur llmman, bash seulement, `yolo: false`) ; tout fichier d'agent convient.
39+
40+## L'API
41+
42+| | |
43+|---|---|
44+| `await Agent.start({ config, cwd?, mm?, onPermission?, onStderr? })` | lance `mm -acp -config <config>` (rendu absolu), puis `initialize` et `session/new {cwd}`. `mm` vaut par défaut `$MM_BIN`, puis `mm`. |
45+| `await agent.complete(prompt, { signal?, onEvent? })` | un tour ; renvoie `{ text, stopReason, usage }` une fois terminé. `onEvent` voit chaque événement au fil de l'eau. |
46+| `agent.stream(prompt, { signal? })` | un tour sous forme d'itérateur asynchrone d'événements ; le dernier est `done`. Sortir de la boucle avant la fin annule le tour. |
47+| `signal` (un `AbortSignal`) | l'annuler envoie `session/cancel` ; le tour se termine avec `stopReason: "cancelled"` et le texte déjà reçu. |
48+| `onPermission(call)` | appelé quand l'agent veut lancer un outil qui demande une confirmation (`yolo: false`). Renvoyer `true` (autoriser une fois), `false` (refuser une fois) ou un `optionId` ACP (`"allow_always"`, `"never"`). **Sans lui, tous ces appels sont refusés.** |
49+| `onStderr(line)` | la sortie d'erreur de mm (bannière, avertissements, lignes `[acp]`). Sinon elle est lue et ignorée ; sa dernière ligne est ajoutée à l'erreur quand mm s'arrête. |
50+| `await agent.reset()` | une nouvelle session sur le même processus : la conversation repart de zéro. |
51+| `await agent.close()` | ferme l'entrée standard de mm et attend sa fin. |
52+| `agent.sessionId` | l'identifiant de session ACP. |
53+
54+Les événements : `{kind: "text", text}`, `{kind: "thinking", text}`, `{kind: "tool_call", id, title, toolKind, input}`, `{kind: "tool_result", id, status, output}`, `{kind: "done", stopReason, usage, text}`.
55+
56+À savoir :
57+
58+- **Un Agent = un processus mm = une conversation.** Les appels la poursuivent, comme dans un éditeur. Un seul tour à la fois par Agent (un second lève `already running`) ; lancez plusieurs Agents pour travailler en parallèle.
59+- **Un tour que mm termine en erreur est rejeté** avec les mots de mm, par exemple `minime: Internal error (-32603): {"error":"the model repeated the same bash call 6 times…"}`.
60+- **Une question annulée est retirée de la conversation par mm** : la question suivante n'en sait rien.
61+- Le client n'annonce aucune capacité fichiers, terminal ni formulaire : mm n'en a pas besoin, et son outil `ask_user` n'est pas proposé.
62+
63+## Tests
64+
65+```bash
66+npm test # node --test test/minime.test.js — 11 tests, sans modèle, sans mm
67+```
68+
69+`test/fake-mm.js` joue le rôle de mm dans ACP, avec les formes de messages capturées sur mm v1.0.1 : des réponses, un appel d'outil avec sa permission, un tour lent à annuler, un plantage. Pas de `node --test` tout court : il lancerait aussi `fake-mm.js` comme un test, qui attendrait sans fin.
70+
71+## Mesuré (2026-10-26, llmman sur la machine de l'auteur, mm v1.0.1 compilé depuis l'arbre de travail)
72+
73+| Exemple | Résultat |
74+|---|---|
75+| `complete.js` | réponse en une phrase, `end_turn`, 663 tokens, **2,9 s** |
76+| `stream.js` | `find examples -name "*.js" \| wc -l` autorisé par le callback (lecture seule), sortie `4`, réponse affichée morceau par morceau, `end_turn` |
77+| `cancel.js` | histoire coupée au bout de **3,0 s** (`cancelled`, 383 caractères gardés) ; la question suivante sur le même Agent a répondu tout de suite — sans connaître l'histoire, que mm a retirée |
78+| `permission.js` | `pwd` refusé : le modèle a **inventé** une réponse (`/root`) ; autorisé : le vrai dossier. Un refus n'empêche pas un petit modèle de deviner. |
79+
80+Vu sur trois passages, avant que le callback n'autorise `find` : le modèle a relancé la même commande refusée jusqu'à ce que le détecteur de boucle de mm arrête le tour (6 appels identiques) ; `stream()` a été rejeté avec ce message — d'où le `try/catch` de `stream.js`.
81+
82+## Pas fait
83+
84+- La bibliothèque ACP TypeScript officielle n'a pas été essayée (pas de réseau dans le bac à sable où ceci a été écrit) : du JSON-RPC brut garde le client sans dépendance.
85+- Pas de paquet npm publié : il faudrait livrer le binaire `mm` avec, par plateforme.
86+- `session/load` / `session/resume` (sessions enregistrées), images et fichiers joints dans les prompts, formulaires `ask_user` : non exposés.
new file mode 100644
@@ -0,0 +1,86 @@
1+# mini-me depuis Node.js : `minime.js`
2+
3+*[English version](README.en.md)*
4+
5+Un client Node.js qui utilise mini-me comme bibliothèque en pilotant `mm -acp` par ACP (ticket 167). Un fichier, `minime.js`, aucune dépendance (Node ≥ 20 : `child_process` et `readline`). Voir [`../README.fr.md`](../README.fr.md) pour les messages ACP utilisés.
6+
7+```js
8+import { Agent } from "./minime.js";
9+
10+const agent = await Agent.start({ config: "agent.yaml" });
11+
12+const { text } = await agent.complete("Explain this repository"); // sans stream
13+
14+for await (const ev of agent.stream("And the tests?")) { // en stream
15+ if (ev.kind === "text") process.stdout.write(ev.text);
16+}
17+
18+const ac = new AbortController();
19+setTimeout(() => ac.abort(), 3000);
20+await agent.complete("A long task", { signal: ac.signal }); // annulé
21+
22+await agent.close();
23+```
24+
25+## Lancer les exemples
26+
27+```bash
28+llmman pull hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M
29+llmman serve
30+
31+cd demo.acp.sdk/nodejs
32+node examples/complete.js # sans stream
33+node examples/stream.js # en stream, avec les appels d'outils
34+node examples/cancel.js # annulé au bout de 3 secondes
35+node examples/permission.js # une commande refusée, puis autorisée
36+```
37+
38+`mm` doit être dans le PATH, ou désigné par `MM_BIN`. Les exemples utilisent `agent.yaml` (Qwen3.5 4B sur llmman, bash seulement, `yolo: false`) ; tout fichier d'agent convient.
39+
40+## L'API
41+
42+| | |
43+|---|---|
44+| `await Agent.start({ config, cwd?, mm?, onPermission?, onStderr? })` | lance `mm -acp -config <config>` (rendu absolu), puis `initialize` et `session/new {cwd}`. `mm` vaut par défaut `$MM_BIN`, puis `mm`. |
45+| `await agent.complete(prompt, { signal?, onEvent? })` | un tour ; renvoie `{ text, stopReason, usage }` une fois terminé. `onEvent` voit chaque événement au fil de l'eau. |
46+| `agent.stream(prompt, { signal? })` | un tour sous forme d'itérateur asynchrone d'événements ; le dernier est `done`. Sortir de la boucle avant la fin annule le tour. |
47+| `signal` (un `AbortSignal`) | l'annuler envoie `session/cancel` ; le tour se termine avec `stopReason: "cancelled"` et le texte déjà reçu. |
48+| `onPermission(call)` | appelé quand l'agent veut lancer un outil qui demande une confirmation (`yolo: false`). Renvoyer `true` (autoriser une fois), `false` (refuser une fois) ou un `optionId` ACP (`"allow_always"`, `"never"`). **Sans lui, tous ces appels sont refusés.** |
49+| `onStderr(line)` | la sortie d'erreur de mm (bannière, avertissements, lignes `[acp]`). Sinon elle est lue et ignorée ; sa dernière ligne est ajoutée à l'erreur quand mm s'arrête. |
50+| `await agent.reset()` | une nouvelle session sur le même processus : la conversation repart de zéro. |
51+| `await agent.close()` | ferme l'entrée standard de mm et attend sa fin. |
52+| `agent.sessionId` | l'identifiant de session ACP. |
53+
54+Les événements : `{kind: "text", text}`, `{kind: "thinking", text}`, `{kind: "tool_call", id, title, toolKind, input}`, `{kind: "tool_result", id, status, output}`, `{kind: "done", stopReason, usage, text}`.
55+
56+À savoir :
57+
58+- **Un Agent = un processus mm = une conversation.** Les appels la poursuivent, comme dans un éditeur. Un seul tour à la fois par Agent (un second lève `already running`) ; lancez plusieurs Agents pour travailler en parallèle.
59+- **Un tour que mm termine en erreur est rejeté** avec les mots de mm, par exemple `minime: Internal error (-32603): {"error":"the model repeated the same bash call 6 times…"}`.
60+- **Une question annulée est retirée de la conversation par mm** : la question suivante n'en sait rien.
61+- Le client n'annonce aucune capacité fichiers, terminal ni formulaire : mm n'en a pas besoin, et son outil `ask_user` n'est pas proposé.
62+
63+## Tests
64+
65+```bash
66+npm test # node --test test/minime.test.js — 11 tests, sans modèle, sans mm
67+```
68+
69+`test/fake-mm.js` joue le rôle de mm dans ACP, avec les formes de messages capturées sur mm v1.0.1 : des réponses, un appel d'outil avec sa permission, un tour lent à annuler, un plantage. Pas de `node --test` tout court : il lancerait aussi `fake-mm.js` comme un test, qui attendrait sans fin.
70+
71+## Mesuré (2026-10-26, llmman sur la machine de l'auteur, mm v1.0.1 compilé depuis l'arbre de travail)
72+
73+| Exemple | Résultat |
74+|---|---|
75+| `complete.js` | réponse en une phrase, `end_turn`, 663 tokens, **2,9 s** |
76+| `stream.js` | `find examples -name "*.js" \| wc -l` autorisé par le callback (lecture seule), sortie `4`, réponse affichée morceau par morceau, `end_turn` |
77+| `cancel.js` | histoire coupée au bout de **3,0 s** (`cancelled`, 383 caractères gardés) ; la question suivante sur le même Agent a répondu tout de suite — sans connaître l'histoire, que mm a retirée |
78+| `permission.js` | `pwd` refusé : le modèle a **inventé** une réponse (`/root`) ; autorisé : le vrai dossier. Un refus n'empêche pas un petit modèle de deviner. |
79+
80+Vu sur trois passages, avant que le callback n'autorise `find` : le modèle a relancé la même commande refusée jusqu'à ce que le détecteur de boucle de mm arrête le tour (6 appels identiques) ; `stream()` a été rejeté avec ce message — d'où le `try/catch` de `stream.js`.
81+
82+## Pas fait
83+
84+- La bibliothèque ACP TypeScript officielle n'a pas été essayée (pas de réseau dans le bac à sable où ceci a été écrit) : du JSON-RPC brut garde le client sans dépendance.
85+- Pas de paquet npm publié : il faudrait livrer le binaire `mm` avec, par plateforme.
86+- `session/load` / `session/resume` (sessions enregistrées), images et fichiers joints dans les prompts, formulaires `ask_user` : non exposés.
added demo.acp.sdk/nodejs/agent.yaml +43 -0
new file mode 100644
@@ -0,0 +1,43 @@
1+# The agent the Node.js examples drive through minime.js — an ordinary mm
2+# agent file: minime.js starts `mm -acp -config <this file>`. Any agent file
3+# works; this one is small on purpose.
4+#
5+# Qwen3.5 4B on llmman, bash only, and yolo: false so that every command
6+# goes through the permission callback (examples/permission.js).
7+
8+provider: llmman
9+model: hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M
10+baseUrl: http://127.0.0.1:17434/v1
11+fallback: http://host.docker.internal:17434/v1
12+contextWindow: 32768
13+
14+sampling:
15+ # Qwen3.5, thinking off, tool-heavy: 0.0 (the "precise" setting measured
16+ # in the demos, internal/agentinit/presets.yaml).
17+ temperature: 0.0
18+ max_tokens: 1024
19+ chat_template_kwargs:
20+ enable_thinking: false
21+
22+maxTurns: 6
23+yolo: false
24+bashTool: true
25+editTools: false
26+planTool: false
27+askUserTool: false
28+fetchContentTool: false
29+skillsDir: no-skills
30+
31+agentsMd:
32+ enabled: false
33+
34+sessions:
35+ enabled: false
36+
37+acp:
38+ name: node-demo
39+ title: Node.js demo agent
40+
41+system: |
42+ You are a helpful assistant. Use bash when you need to look at the files
43+ or the system. Answer briefly.
new file mode 100644
@@ -0,0 +1,43 @@
1+# The agent the Node.js examples drive through minime.js — an ordinary mm
2+# agent file: minime.js starts `mm -acp -config <this file>`. Any agent file
3+# works; this one is small on purpose.
4+#
5+# Qwen3.5 4B on llmman, bash only, and yolo: false so that every command
6+# goes through the permission callback (examples/permission.js).
7+
8+provider: llmman
9+model: hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M
10+baseUrl: http://127.0.0.1:17434/v1
11+fallback: http://host.docker.internal:17434/v1
12+contextWindow: 32768
13+
14+sampling:
15+ # Qwen3.5, thinking off, tool-heavy: 0.0 (the "precise" setting measured
16+ # in the demos, internal/agentinit/presets.yaml).
17+ temperature: 0.0
18+ max_tokens: 1024
19+ chat_template_kwargs:
20+ enable_thinking: false
21+
22+maxTurns: 6
23+yolo: false
24+bashTool: true
25+editTools: false
26+planTool: false
27+askUserTool: false
28+fetchContentTool: false
29+skillsDir: no-skills
30+
31+agentsMd:
32+ enabled: false
33+
34+sessions:
35+ enabled: false
36+
37+acp:
38+ name: node-demo
39+ title: Node.js demo agent
40+
41+system: |
42+ You are a helpful assistant. Use bash when you need to look at the files
43+ or the system. Answer briefly.
added demo.acp.sdk/nodejs/examples/cancel.js +21 -0
new file mode 100644
@@ -0,0 +1,21 @@
1+// Cancelled: an AbortController stops the turn after 3 seconds. mm receives
2+// session/cancel, stops the generation, and the agent can be used again.
3+// mm drops a question it did not finish answering from the conversation: the
4+// next question does not know about the story.
5+// node examples/cancel.js
6+import { Agent } from "../minime.js";
7+
8+const agent = await Agent.start({ config: "agent.yaml" });
9+
10+const ac = new AbortController();
11+setTimeout(() => ac.abort(), 3000);
12+const started = Date.now();
13+const first = await agent.complete("Write a 2000-word story about a lighthouse keeper.", {
14+ signal: ac.signal,
15+ onEvent: (ev) => ev.kind === "text" && process.stdout.write(ev.text),
16+});
17+console.log(`\n\n[${first.stopReason} after ${((Date.now() - started) / 1000).toFixed(1)} s, ${first.text.length} characters kept]`);
18+
19+const second = await agent.complete("Now answer in three words: what was the story about?");
20+console.log(`\nNext question (the cancelled one was dropped from the conversation): ${second.text}`);
21+await agent.close();
new file mode 100644
@@ -0,0 +1,21 @@
1+// Cancelled: an AbortController stops the turn after 3 seconds. mm receives
2+// session/cancel, stops the generation, and the agent can be used again.
3+// mm drops a question it did not finish answering from the conversation: the
4+// next question does not know about the story.
5+// node examples/cancel.js
6+import { Agent } from "../minime.js";
7+
8+const agent = await Agent.start({ config: "agent.yaml" });
9+
10+const ac = new AbortController();
11+setTimeout(() => ac.abort(), 3000);
12+const started = Date.now();
13+const first = await agent.complete("Write a 2000-word story about a lighthouse keeper.", {
14+ signal: ac.signal,
15+ onEvent: (ev) => ev.kind === "text" && process.stdout.write(ev.text),
16+});
17+console.log(`\n\n[${first.stopReason} after ${((Date.now() - started) / 1000).toFixed(1)} s, ${first.text.length} characters kept]`);
18+
19+const second = await agent.complete("Now answer in three words: what was the story about?");
20+console.log(`\nNext question (the cancelled one was dropped from the conversation): ${second.text}`);
21+await agent.close();
added demo.acp.sdk/nodejs/examples/complete.js +10 -0
new file mode 100644
@@ -0,0 +1,10 @@
1+// Not streamed: one question, the whole answer at once.
2+// node examples/complete.js (from demo.acp.sdk/nodejs/)
3+import { Agent } from "../minime.js";
4+
5+const agent = await Agent.start({ config: "agent.yaml" });
6+const started = Date.now();
7+const { text, stopReason, usage } = await agent.complete("In one sentence: what is a JSON-RPC notification?");
8+console.log(text);
9+console.log(`\n[${stopReason} · ${usage?.totalTokens ?? "?"} tokens · ${((Date.now() - started) / 1000).toFixed(1)} s]`);
10+await agent.close();
new file mode 100644
@@ -0,0 +1,10 @@
1+// Not streamed: one question, the whole answer at once.
2+// node examples/complete.js (from demo.acp.sdk/nodejs/)
3+import { Agent } from "../minime.js";
4+
5+const agent = await Agent.start({ config: "agent.yaml" });
6+const started = Date.now();
7+const { text, stopReason, usage } = await agent.complete("In one sentence: what is a JSON-RPC notification?");
8+console.log(text);
9+console.log(`\n[${stopReason} · ${usage?.totalTokens ?? "?"} tokens · ${((Date.now() - started) / 1000).toFixed(1)} s]`);
10+await agent.close();
added demo.acp.sdk/nodejs/examples/permission.js +18 -0
new file mode 100644
@@ -0,0 +1,18 @@
1+// Permission: the agent has yolo: false, so each command goes through
2+// onPermission. Here it is refused, then allowed, and the answers differ.
3+// node examples/permission.js
4+import { Agent } from "../minime.js";
5+
6+for (const allow of [false, true]) {
7+ const agent = await Agent.start({
8+ config: "agent.yaml",
9+ onPermission: (call) => {
10+ console.log(` asked: ${call.title} → ${allow ? "allowed" : "refused"}`);
11+ return allow;
12+ },
13+ });
14+ console.log(`\n--- onPermission returns ${allow}`);
15+ const { text } = await agent.complete("What is the name of the current directory? Use pwd.");
16+ console.log(text);
17+ await agent.close();
18+}
new file mode 100644
@@ -0,0 +1,18 @@
1+// Permission: the agent has yolo: false, so each command goes through
2+// onPermission. Here it is refused, then allowed, and the answers differ.
3+// node examples/permission.js
4+import { Agent } from "../minime.js";
5+
6+for (const allow of [false, true]) {
7+ const agent = await Agent.start({
8+ config: "agent.yaml",
9+ onPermission: (call) => {
10+ console.log(` asked: ${call.title} → ${allow ? "allowed" : "refused"}`);
11+ return allow;
12+ },
13+ });
14+ console.log(`\n--- onPermission returns ${allow}`);
15+ const { text } = await agent.complete("What is the name of the current directory? Use pwd.");
16+ console.log(text);
17+ await agent.close();
18+}
added demo.acp.sdk/nodejs/examples/stream.js +40 -0
new file mode 100644
@@ -0,0 +1,40 @@
1+// Streamed: the answer as it comes, the tool calls as they happen.
2+// The permission callback allows read-only commands only (ls, find, wc…)
3+// and says what it decided. A prefix check is a demo, not a sandbox:
4+// `find . -delete` or `ls; rm x` would pass it.
5+// node examples/stream.js (from demo.acp.sdk/nodejs/)
6+import { Agent } from "../minime.js";
7+
8+const readOnly = /^(ls|find|cat|wc|head|tail|pwd|grep)\b/;
9+const agent = await Agent.start({
10+ config: "agent.yaml",
11+ onPermission: (call) => {
12+ const ok = readOnly.test(call.input?.command ?? "");
13+ console.log(`\n🔐 ${call.title} → ${ok ? "allowed (read-only)" : "refused"}`);
14+ return ok;
15+ },
16+});
17+
18+try {
19+ for await (const ev of agent.stream("How many .js files are in the examples folder? Use a command.")) {
20+ switch (ev.kind) {
21+ case "text":
22+ process.stdout.write(ev.text);
23+ break;
24+ case "tool_call":
25+ console.log(`🛠️ ${ev.title}`);
26+ break;
27+ case "tool_result":
28+ console.log(` ${ev.status}: ${String(ev.output).trim()}`);
29+ break;
30+ case "done":
31+ console.log(`\n[${ev.stopReason}]`);
32+ break;
33+ }
34+ }
35+} catch (err) {
36+ // mm ended the turn with an error (e.g. its loop detector after the same
37+ // refused command six times): the promise is rejected with mm's words.
38+ console.error(`\n${err.message}`);
39+}
40+await agent.close();
new file mode 100644
@@ -0,0 +1,40 @@
1+// Streamed: the answer as it comes, the tool calls as they happen.
2+// The permission callback allows read-only commands only (ls, find, wc…)
3+// and says what it decided. A prefix check is a demo, not a sandbox:
4+// `find . -delete` or `ls; rm x` would pass it.
5+// node examples/stream.js (from demo.acp.sdk/nodejs/)
6+import { Agent } from "../minime.js";
7+
8+const readOnly = /^(ls|find|cat|wc|head|tail|pwd|grep)\b/;
9+const agent = await Agent.start({
10+ config: "agent.yaml",
11+ onPermission: (call) => {
12+ const ok = readOnly.test(call.input?.command ?? "");
13+ console.log(`\n🔐 ${call.title} → ${ok ? "allowed (read-only)" : "refused"}`);
14+ return ok;
15+ },
16+});
17+
18+try {
19+ for await (const ev of agent.stream("How many .js files are in the examples folder? Use a command.")) {
20+ switch (ev.kind) {
21+ case "text":
22+ process.stdout.write(ev.text);
23+ break;
24+ case "tool_call":
25+ console.log(`🛠️ ${ev.title}`);
26+ break;
27+ case "tool_result":
28+ console.log(` ${ev.status}: ${String(ev.output).trim()}`);
29+ break;
30+ case "done":
31+ console.log(`\n[${ev.stopReason}]`);
32+ break;
33+ }
34+ }
35+} catch (err) {
36+ // mm ended the turn with an error (e.g. its loop detector after the same
37+ // refused command six times): the promise is rejected with mm's words.
38+ console.error(`\n${err.message}`);
39+}
40+await agent.close();
added demo.acp.sdk/nodejs/minime.js +307 -0
new file mode 100644
@@ -0,0 +1,307 @@
1+// minime.js — use mini-me from Node.js, as a library, by driving `mm -acp`
2+// over ACP (JSON-RPC, one message per line on the child's stdin/stdout).
3+// No dependency: child_process and readline only. Ticket 167.
4+//
5+// import { Agent } from "./minime.js";
6+//
7+// const agent = await Agent.start({ config: "agent.yaml" });
8+// const answer = await agent.complete("Explain this repository"); // not streamed
9+// for await (const ev of agent.stream("And the tests?")) { // streamed
10+// if (ev.kind === "text") process.stdout.write(ev.text);
11+// }
12+// await agent.close();
13+//
14+// One Agent = one `mm -acp` process = one ACP session: the conversation goes
15+// on from one call to the next, as in an editor. reset() starts a new one.
16+
17+import { spawn } from "node:child_process";
18+import { createInterface } from "node:readline";
19+import path from "node:path";
20+
21+/**
22+ * The events a turn produces, reduced from ACP's `session/update`
23+ * notifications (which carry more detail for editors):
24+ *
25+ * { kind: "text", text } a chunk of the answer
26+ * { kind: "thinking", text } a chunk of a thinking model's reasoning
27+ * { kind: "tool_call", id, title, toolKind, input } a tool call announced, before it runs
28+ * { kind: "tool_result", id, status, output } its outcome (status: completed or failed)
29+ * { kind: "done", stopReason, usage, text } the end of the turn, with the whole answer
30+ */
31+
32+const PROTOCOL_VERSION = 1;
33+
34+export class Agent {
35+ /**
36+ * Starts `mm -acp` on an agent file, then opens a session.
37+ *
38+ * @param {object} options
39+ * @param {string} options.config the agent file (agent.yaml); made absolute
40+ * @param {string} [options.cwd] the session's working directory (default: process.cwd())
41+ * @param {string} [options.mm] the mm binary (default: $MM_BIN, then "mm" on the PATH)
42+ * @param {(call: object) => boolean|string|Promise<boolean|string>} [options.onPermission]
43+ * asked when the agent wants to run a tool that needs a confirmation (`yolo: false`).
44+ * true = allow once, false = reject once, or an ACP optionId ("allow_always", "never"…).
45+ * Without it, every such call is REJECTED: nothing runs unasked.
46+ * @param {(line: string) => void} [options.onStderr] mm's stderr lines (banner, warnings)
47+ * @returns {Promise<Agent>}
48+ *
49+ * @example
50+ * const agent = await Agent.start({ config: "agent.yaml", onPermission: (c) => c.toolKind === "read" });
51+ */
52+ static async start(options) {
53+ const agent = new Agent(options);
54+ await agent.#initialize();
55+ return agent;
56+ }
57+
58+ #child;
59+ #nextId = 1;
60+ #pending = new Map(); // request id → { resolve, reject }
61+ #onPermission;
62+ #onStderr;
63+ #stderrTail = [];
64+ #cwd;
65+ #sessionId = null;
66+ #turn = null; // the turn in progress: { push(event), text }
67+ #exited = null;
68+
69+ constructor({ config, cwd = process.cwd(), mm = process.env.MM_BIN || "mm", onPermission, onStderr } = {}) {
70+ if (!config) throw new Error("minime: `config` (the agent file) is required");
71+ this.#cwd = path.resolve(cwd);
72+ this.#onPermission = onPermission;
73+ this.#onStderr = onStderr;
74+ this.#child = spawn(mm, ["-acp", "-config", path.resolve(config)], { cwd: this.#cwd, stdio: ["pipe", "pipe", "pipe"] });
75+ this.#child.on("error", (err) => this.#fail(new Error(`minime: cannot start ${mm}: ${err.message}`)));
76+ this.#child.on("exit", (code, signal) => {
77+ this.#exited = { code, signal };
78+ this.#fail(new Error(`minime: mm exited (${signal ?? code})${this.#lastStderr()}`));
79+ });
80+ createInterface({ input: this.#child.stdout }).on("line", (line) => this.#receive(line));
81+ createInterface({ input: this.#child.stderr }).on("line", (line) => {
82+ this.#stderrTail = [...this.#stderrTail, line].slice(-5);
83+ this.#onStderr?.(line);
84+ });
85+ }
86+
87+ /** The ACP session id of the current conversation. */
88+ get sessionId() {
89+ return this.#sessionId;
90+ }
91+
92+ /**
93+ * Runs one turn and returns the whole answer once it is over.
94+ *
95+ * @param {string} prompt
96+ * @param {object} [options]
97+ * @param {AbortSignal} [options.signal] aborting it sends `session/cancel`; the turn then ends
98+ * with stopReason "cancelled" and the text so far
99+ * @param {(event: object) => void} [options.onEvent] every event, as they come
100+ * @returns {Promise<{ text: string, stopReason: string, usage: object|undefined }>}
101+ *
102+ * @example
103+ * const { text } = await agent.complete("What is in this folder?");
104+ */
105+ async complete(prompt, { signal, onEvent } = {}) {
106+ let last;
107+ for await (const ev of this.stream(prompt, { signal })) {
108+ onEvent?.(ev);
109+ last = ev;
110+ }
111+ return { text: last.text, stopReason: last.stopReason, usage: last.usage };
112+ }
113+
114+ /**
115+ * Runs one turn and yields its events as they come; the last one is
116+ * `{ kind: "done", stopReason, usage, text }`.
117+ *
118+ * @param {string} prompt
119+ * @param {object} [options]
120+ * @param {AbortSignal} [options.signal]
121+ * @returns {AsyncGenerator<object>}
122+ *
123+ * @example
124+ * for await (const ev of agent.stream("Explain main.go")) {
125+ * if (ev.kind === "text") process.stdout.write(ev.text);
126+ * }
127+ */
128+ async *stream(prompt, { signal } = {}) {
129+ if (this.#turn) throw new Error("minime: a turn is already running on this agent — wait for it, or use another Agent");
130+ signal?.throwIfAborted();
131+ const queue = [];
132+ let wake = null;
133+ const turn = {
134+ text: "",
135+ push: (ev) => {
136+ queue.push(ev);
137+ wake?.();
138+ },
139+ };
140+ this.#turn = turn;
141+
142+ const onAbort = () => this.#notify("session/cancel", { sessionId: this.#sessionId });
143+ signal?.addEventListener("abort", onAbort, { once: true });
144+
145+ const done = this.#request("session/prompt", { sessionId: this.#sessionId, prompt: [{ type: "text", text: prompt }] }).then(
146+ (result) => turn.push({ kind: "done", stopReason: result.stopReason, usage: result.usage, text: turn.text }),
147+ (err) => turn.push({ kind: "error", error: err }),
148+ );
149+
150+ try {
151+ for (;;) {
152+ while (queue.length === 0) await new Promise((resolve) => (wake = resolve));
153+ const ev = queue.shift();
154+ if (ev.kind === "error") throw ev.error;
155+ yield ev;
156+ if (ev.kind === "done") return;
157+ }
158+ } finally {
159+ signal?.removeEventListener("abort", onAbort);
160+ // A caller that stops iterating early cancels the turn: nothing keeps running unread.
161+ if (this.#turn === turn && !this.#exited) {
162+ this.#notify("session/cancel", { sessionId: this.#sessionId });
163+ await done;
164+ }
165+ this.#turn = null;
166+ }
167+ }
168+
169+ /** Forgets the conversation: a new ACP session on the same mm process. */
170+ async reset() {
171+ if (this.#turn) throw new Error("minime: cannot reset during a turn");
172+ await this.#newSession();
173+ }
174+
175+ /** Closes mm's stdin and waits for it to exit. */
176+ async close() {
177+ if (this.#exited) return;
178+ const exited = new Promise((resolve) => this.#child.once("exit", resolve));
179+ this.#child.stdin.end();
180+ await exited;
181+ }
182+
183+ async #initialize() {
184+ // No clientCapabilities: no file system or terminal for mm to call back
185+ // (mm does not use them), and no form support, so ask_user is not offered.
186+ await this.#request("initialize", { protocolVersion: PROTOCOL_VERSION, clientCapabilities: {} });
187+ await this.#newSession();
188+ }
189+
190+ async #newSession() {
191+ const result = await this.#request("session/new", { cwd: this.#cwd, mcpServers: [] });
192+ this.#sessionId = result.sessionId;
193+ }
194+
195+ #request(method, params) {
196+ if (this.#exited) return Promise.reject(new Error(`minime: mm is not running${this.#lastStderr()}`));
197+ const id = this.#nextId++;
198+ return new Promise((resolve, reject) => {
199+ this.#pending.set(id, { resolve, reject });
200+ this.#write({ jsonrpc: "2.0", id, method, params });
201+ });
202+ }
203+
204+ #notify(method, params) {
205+ if (!this.#exited) this.#write({ jsonrpc: "2.0", method, params });
206+ }
207+
208+ #write(message) {
209+ this.#child.stdin.write(JSON.stringify(message) + "\n");
210+ }
211+
212+ #receive(line) {
213+ if (!line.trim()) return;
214+ let message;
215+ try {
216+ message = JSON.parse(line);
217+ } catch {
218+ return; // stdout carries JSON-RPC only; anything else is not ours to read
219+ }
220+ if (message.method && message.id !== undefined) return this.#answerRequest(message);
221+ if (message.method === "session/update") return this.#update(message.params);
222+ if (message.id !== undefined) {
223+ const waiting = this.#pending.get(message.id);
224+ if (!waiting) return;
225+ this.#pending.delete(message.id);
226+ if (message.error) waiting.reject(rpcError(message.error));
227+ else waiting.resolve(message.result);
228+ }
229+ }
230+
231+ // A request FROM mm: the permission to run a tool. Anything else is refused.
232+ async #answerRequest({ id, method, params }) {
233+ if (method !== "session/request_permission") {
234+ this.#write({ jsonrpc: "2.0", id, error: { code: -32601, message: `minime does not handle ${method}` } });
235+ return;
236+ }
237+ const call = toolCall(params.toolCall);
238+ let choice = false;
239+ try {
240+ choice = this.#onPermission ? await this.#onPermission(call) : false;
241+ } catch {
242+ choice = false;
243+ }
244+ const optionId = pickOption(params.options, choice);
245+ this.#write({ jsonrpc: "2.0", id, result: { outcome: { outcome: "selected", optionId } } });
246+ }
247+
248+ #update({ update }) {
249+ const turn = this.#turn;
250+ if (!turn) return; // e.g. available_commands_update right after session/new
251+ switch (update.sessionUpdate) {
252+ case "agent_message_chunk":
253+ if (update.content?.type === "text") {
254+ turn.text += update.content.text;
255+ turn.push({ kind: "text", text: update.content.text });
256+ }
257+ break;
258+ case "agent_thought_chunk":
259+ if (update.content?.type === "text") turn.push({ kind: "thinking", text: update.content.text });
260+ break;
261+ case "tool_call":
262+ turn.push({ kind: "tool_call", ...toolCall(update) });
263+ break;
264+ case "tool_call_update":
265+ if (update.status === "completed" || update.status === "failed") {
266+ turn.push({ kind: "tool_result", id: update.toolCallId, status: update.status, output: outputOf(update) });
267+ }
268+ break;
269+ default:
270+ break; // plan, usage_update, available_commands_update…: editor detail
271+ }
272+ }
273+
274+ #fail(err) {
275+ for (const { reject } of this.#pending.values()) reject(err);
276+ this.#pending.clear();
277+ }
278+
279+ #lastStderr() {
280+ return this.#stderrTail.length ? ` — ${this.#stderrTail.at(-1)}` : "";
281+ }
282+}
283+
284+// The protocol's message, plus mm's own words when it sends them in `data`.
285+function rpcError({ message, code, data }) {
286+ const detail = data === undefined ? "" : `: ${typeof data === "string" ? data : JSON.stringify(data)}`;
287+ return new Error(`minime: ${message} (${code})${detail}`);
288+}
289+
290+function toolCall(tc) {
291+ return { id: tc.toolCallId, title: tc.title, toolKind: tc.kind, input: tc.rawInput };
292+}
293+
294+function outputOf(update) {
295+ if (update.rawOutput?.output !== undefined) return update.rawOutput.output;
296+ return (update.content ?? [])
297+ .map((block) => block.content?.text ?? "")
298+ .join("");
299+}
300+
301+// true / false → the "once" options; a string → that optionId when offered.
302+export function pickOption(options, choice) {
303+ const byKind = (kind) => options.find((o) => o.kind === kind)?.optionId;
304+ if (typeof choice === "string" && options.some((o) => o.optionId === choice)) return choice;
305+ if (choice === true) return byKind("allow_once") ?? byKind("allow_always");
306+ return byKind("reject_once") ?? byKind("reject_always");
307+}
new file mode 100644
@@ -0,0 +1,307 @@
1+// minime.js — use mini-me from Node.js, as a library, by driving `mm -acp`
2+// over ACP (JSON-RPC, one message per line on the child's stdin/stdout).
3+// No dependency: child_process and readline only. Ticket 167.
4+//
5+// import { Agent } from "./minime.js";
6+//
7+// const agent = await Agent.start({ config: "agent.yaml" });
8+// const answer = await agent.complete("Explain this repository"); // not streamed
9+// for await (const ev of agent.stream("And the tests?")) { // streamed
10+// if (ev.kind === "text") process.stdout.write(ev.text);
11+// }
12+// await agent.close();
13+//
14+// One Agent = one `mm -acp` process = one ACP session: the conversation goes
15+// on from one call to the next, as in an editor. reset() starts a new one.
16+
17+import { spawn } from "node:child_process";
18+import { createInterface } from "node:readline";
19+import path from "node:path";
20+
21+/**
22+ * The events a turn produces, reduced from ACP's `session/update`
23+ * notifications (which carry more detail for editors):
24+ *
25+ * { kind: "text", text } a chunk of the answer
26+ * { kind: "thinking", text } a chunk of a thinking model's reasoning
27+ * { kind: "tool_call", id, title, toolKind, input } a tool call announced, before it runs
28+ * { kind: "tool_result", id, status, output } its outcome (status: completed or failed)
29+ * { kind: "done", stopReason, usage, text } the end of the turn, with the whole answer
30+ */
31+
32+const PROTOCOL_VERSION = 1;
33+
34+export class Agent {
35+ /**
36+ * Starts `mm -acp` on an agent file, then opens a session.
37+ *
38+ * @param {object} options
39+ * @param {string} options.config the agent file (agent.yaml); made absolute
40+ * @param {string} [options.cwd] the session's working directory (default: process.cwd())
41+ * @param {string} [options.mm] the mm binary (default: $MM_BIN, then "mm" on the PATH)
42+ * @param {(call: object) => boolean|string|Promise<boolean|string>} [options.onPermission]
43+ * asked when the agent wants to run a tool that needs a confirmation (`yolo: false`).
44+ * true = allow once, false = reject once, or an ACP optionId ("allow_always", "never"…).
45+ * Without it, every such call is REJECTED: nothing runs unasked.
46+ * @param {(line: string) => void} [options.onStderr] mm's stderr lines (banner, warnings)
47+ * @returns {Promise<Agent>}
48+ *
49+ * @example
50+ * const agent = await Agent.start({ config: "agent.yaml", onPermission: (c) => c.toolKind === "read" });
51+ */
52+ static async start(options) {
53+ const agent = new Agent(options);
54+ await agent.#initialize();
55+ return agent;
56+ }
57+
58+ #child;
59+ #nextId = 1;
60+ #pending = new Map(); // request id → { resolve, reject }
61+ #onPermission;
62+ #onStderr;
63+ #stderrTail = [];
64+ #cwd;
65+ #sessionId = null;
66+ #turn = null; // the turn in progress: { push(event), text }
67+ #exited = null;
68+
69+ constructor({ config, cwd = process.cwd(), mm = process.env.MM_BIN || "mm", onPermission, onStderr } = {}) {
70+ if (!config) throw new Error("minime: `config` (the agent file) is required");
71+ this.#cwd = path.resolve(cwd);
72+ this.#onPermission = onPermission;
73+ this.#onStderr = onStderr;
74+ this.#child = spawn(mm, ["-acp", "-config", path.resolve(config)], { cwd: this.#cwd, stdio: ["pipe", "pipe", "pipe"] });
75+ this.#child.on("error", (err) => this.#fail(new Error(`minime: cannot start ${mm}: ${err.message}`)));
76+ this.#child.on("exit", (code, signal) => {
77+ this.#exited = { code, signal };
78+ this.#fail(new Error(`minime: mm exited (${signal ?? code})${this.#lastStderr()}`));
79+ });
80+ createInterface({ input: this.#child.stdout }).on("line", (line) => this.#receive(line));
81+ createInterface({ input: this.#child.stderr }).on("line", (line) => {
82+ this.#stderrTail = [...this.#stderrTail, line].slice(-5);
83+ this.#onStderr?.(line);
84+ });
85+ }
86+
87+ /** The ACP session id of the current conversation. */
88+ get sessionId() {
89+ return this.#sessionId;
90+ }
91+
92+ /**
93+ * Runs one turn and returns the whole answer once it is over.
94+ *
95+ * @param {string} prompt
96+ * @param {object} [options]
97+ * @param {AbortSignal} [options.signal] aborting it sends `session/cancel`; the turn then ends
98+ * with stopReason "cancelled" and the text so far
99+ * @param {(event: object) => void} [options.onEvent] every event, as they come
100+ * @returns {Promise<{ text: string, stopReason: string, usage: object|undefined }>}
101+ *
102+ * @example
103+ * const { text } = await agent.complete("What is in this folder?");
104+ */
105+ async complete(prompt, { signal, onEvent } = {}) {
106+ let last;
107+ for await (const ev of this.stream(prompt, { signal })) {
108+ onEvent?.(ev);
109+ last = ev;
110+ }
111+ return { text: last.text, stopReason: last.stopReason, usage: last.usage };
112+ }
113+
114+ /**
115+ * Runs one turn and yields its events as they come; the last one is
116+ * `{ kind: "done", stopReason, usage, text }`.
117+ *
118+ * @param {string} prompt
119+ * @param {object} [options]
120+ * @param {AbortSignal} [options.signal]
121+ * @returns {AsyncGenerator<object>}
122+ *
123+ * @example
124+ * for await (const ev of agent.stream("Explain main.go")) {
125+ * if (ev.kind === "text") process.stdout.write(ev.text);
126+ * }
127+ */
128+ async *stream(prompt, { signal } = {}) {
129+ if (this.#turn) throw new Error("minime: a turn is already running on this agent — wait for it, or use another Agent");
130+ signal?.throwIfAborted();
131+ const queue = [];
132+ let wake = null;
133+ const turn = {
134+ text: "",
135+ push: (ev) => {
136+ queue.push(ev);
137+ wake?.();
138+ },
139+ };
140+ this.#turn = turn;
141+
142+ const onAbort = () => this.#notify("session/cancel", { sessionId: this.#sessionId });
143+ signal?.addEventListener("abort", onAbort, { once: true });
144+
145+ const done = this.#request("session/prompt", { sessionId: this.#sessionId, prompt: [{ type: "text", text: prompt }] }).then(
146+ (result) => turn.push({ kind: "done", stopReason: result.stopReason, usage: result.usage, text: turn.text }),
147+ (err) => turn.push({ kind: "error", error: err }),
148+ );
149+
150+ try {
151+ for (;;) {
152+ while (queue.length === 0) await new Promise((resolve) => (wake = resolve));
153+ const ev = queue.shift();
154+ if (ev.kind === "error") throw ev.error;
155+ yield ev;
156+ if (ev.kind === "done") return;
157+ }
158+ } finally {
159+ signal?.removeEventListener("abort", onAbort);
160+ // A caller that stops iterating early cancels the turn: nothing keeps running unread.
161+ if (this.#turn === turn && !this.#exited) {
162+ this.#notify("session/cancel", { sessionId: this.#sessionId });
163+ await done;
164+ }
165+ this.#turn = null;
166+ }
167+ }
168+
169+ /** Forgets the conversation: a new ACP session on the same mm process. */
170+ async reset() {
171+ if (this.#turn) throw new Error("minime: cannot reset during a turn");
172+ await this.#newSession();
173+ }
174+
175+ /** Closes mm's stdin and waits for it to exit. */
176+ async close() {
177+ if (this.#exited) return;
178+ const exited = new Promise((resolve) => this.#child.once("exit", resolve));
179+ this.#child.stdin.end();
180+ await exited;
181+ }
182+
183+ async #initialize() {
184+ // No clientCapabilities: no file system or terminal for mm to call back
185+ // (mm does not use them), and no form support, so ask_user is not offered.
186+ await this.#request("initialize", { protocolVersion: PROTOCOL_VERSION, clientCapabilities: {} });
187+ await this.#newSession();
188+ }
189+
190+ async #newSession() {
191+ const result = await this.#request("session/new", { cwd: this.#cwd, mcpServers: [] });
192+ this.#sessionId = result.sessionId;
193+ }
194+
195+ #request(method, params) {
196+ if (this.#exited) return Promise.reject(new Error(`minime: mm is not running${this.#lastStderr()}`));
197+ const id = this.#nextId++;
198+ return new Promise((resolve, reject) => {
199+ this.#pending.set(id, { resolve, reject });
200+ this.#write({ jsonrpc: "2.0", id, method, params });
201+ });
202+ }
203+
204+ #notify(method, params) {
205+ if (!this.#exited) this.#write({ jsonrpc: "2.0", method, params });
206+ }
207+
208+ #write(message) {
209+ this.#child.stdin.write(JSON.stringify(message) + "\n");
210+ }
211+
212+ #receive(line) {
213+ if (!line.trim()) return;
214+ let message;
215+ try {
216+ message = JSON.parse(line);
217+ } catch {
218+ return; // stdout carries JSON-RPC only; anything else is not ours to read
219+ }
220+ if (message.method && message.id !== undefined) return this.#answerRequest(message);
221+ if (message.method === "session/update") return this.#update(message.params);
222+ if (message.id !== undefined) {
223+ const waiting = this.#pending.get(message.id);
224+ if (!waiting) return;
225+ this.#pending.delete(message.id);
226+ if (message.error) waiting.reject(rpcError(message.error));
227+ else waiting.resolve(message.result);
228+ }
229+ }
230+
231+ // A request FROM mm: the permission to run a tool. Anything else is refused.
232+ async #answerRequest({ id, method, params }) {
233+ if (method !== "session/request_permission") {
234+ this.#write({ jsonrpc: "2.0", id, error: { code: -32601, message: `minime does not handle ${method}` } });
235+ return;
236+ }
237+ const call = toolCall(params.toolCall);
238+ let choice = false;
239+ try {
240+ choice = this.#onPermission ? await this.#onPermission(call) : false;
241+ } catch {
242+ choice = false;
243+ }
244+ const optionId = pickOption(params.options, choice);
245+ this.#write({ jsonrpc: "2.0", id, result: { outcome: { outcome: "selected", optionId } } });
246+ }
247+
248+ #update({ update }) {
249+ const turn = this.#turn;
250+ if (!turn) return; // e.g. available_commands_update right after session/new
251+ switch (update.sessionUpdate) {
252+ case "agent_message_chunk":
253+ if (update.content?.type === "text") {
254+ turn.text += update.content.text;
255+ turn.push({ kind: "text", text: update.content.text });
256+ }
257+ break;
258+ case "agent_thought_chunk":
259+ if (update.content?.type === "text") turn.push({ kind: "thinking", text: update.content.text });
260+ break;
261+ case "tool_call":
262+ turn.push({ kind: "tool_call", ...toolCall(update) });
263+ break;
264+ case "tool_call_update":
265+ if (update.status === "completed" || update.status === "failed") {
266+ turn.push({ kind: "tool_result", id: update.toolCallId, status: update.status, output: outputOf(update) });
267+ }
268+ break;
269+ default:
270+ break; // plan, usage_update, available_commands_update…: editor detail
271+ }
272+ }
273+
274+ #fail(err) {
275+ for (const { reject } of this.#pending.values()) reject(err);
276+ this.#pending.clear();
277+ }
278+
279+ #lastStderr() {
280+ return this.#stderrTail.length ? ` — ${this.#stderrTail.at(-1)}` : "";
281+ }
282+}
283+
284+// The protocol's message, plus mm's own words when it sends them in `data`.
285+function rpcError({ message, code, data }) {
286+ const detail = data === undefined ? "" : `: ${typeof data === "string" ? data : JSON.stringify(data)}`;
287+ return new Error(`minime: ${message} (${code})${detail}`);
288+}
289+
290+function toolCall(tc) {
291+ return { id: tc.toolCallId, title: tc.title, toolKind: tc.kind, input: tc.rawInput };
292+}
293+
294+function outputOf(update) {
295+ if (update.rawOutput?.output !== undefined) return update.rawOutput.output;
296+ return (update.content ?? [])
297+ .map((block) => block.content?.text ?? "")
298+ .join("");
299+}
300+
301+// true / false → the "once" options; a string → that optionId when offered.
302+export function pickOption(options, choice) {
303+ const byKind = (kind) => options.find((o) => o.kind === kind)?.optionId;
304+ if (typeof choice === "string" && options.some((o) => o.optionId === choice)) return choice;
305+ if (choice === true) return byKind("allow_once") ?? byKind("allow_always");
306+ return byKind("reject_once") ?? byKind("reject_always");
307+}
added demo.acp.sdk/nodejs/package.json +10 -0
new file mode 100644
@@ -0,0 +1,10 @@
1+{
2+ "name": "minime-acp-client",
3+ "private": true,
4+ "type": "module",
5+ "description": "Use mini-me from Node.js by driving mm -acp over ACP (ticket 167). No dependency.",
6+ "engines": { "node": ">=20" },
7+ "scripts": {
8+ "test": "node --test test/minime.test.js"
9+ }
10+}
new file mode 100644
@@ -0,0 +1,10 @@
1+{
2+ "name": "minime-acp-client",
3+ "private": true,
4+ "type": "module",
5+ "description": "Use mini-me from Node.js by driving mm -acp over ACP (ticket 167). No dependency.",
6+ "engines": { "node": ">=20" },
7+ "scripts": {
8+ "test": "node --test test/minime.test.js"
9+ }
10+}
added demo.acp.sdk/nodejs/test/fake-mm.js +89 -0
new file mode 100755
@@ -0,0 +1,89 @@
1+#!/usr/bin/env node
2+// A stand-in for `mm -acp` in the tests: it speaks the same ACP messages,
3+// in the same shapes as mm (captured from mm v1.0.1 on 2026-10-26), without
4+// a model. What it does depends on the prompt:
5+//
6+// "tool" announces a bash call, asks the permission, runs it or not
7+// "slow" streams nothing until session/cancel, then ends "cancelled"
8+// "crash" exits in the middle of the turn
9+// else answers "Hello" + ", world" in two chunks
10+//
11+// It writes one line on stderr, like mm's banner.
12+
13+import { createInterface } from "node:readline";
14+
15+const send = (m) => process.stdout.write(JSON.stringify({ jsonrpc: "2.0", ...m }) + "\n");
16+const update = (sessionId, u) => send({ method: "session/update", params: { sessionId, update: u } });
17+
18+let sessions = 0;
19+let nextOwnId = 1;
20+const waiting = new Map(); // our request id → resolve
21+const cancelled = new Map(); // sessionId → resolve
22+
23+process.stderr.write("fake-mm | model: none\n");
24+
25+createInterface({ input: process.stdin }).on("line", async (line) => {
26+ const m = JSON.parse(line);
27+ if (m.id !== undefined && !m.method) return waiting.get(m.id)?.(m.result);
28+ switch (m.method) {
29+ case "initialize":
30+ return send({ id: m.id, result: { protocolVersion: 1, agentCapabilities: {}, agentInfo: { name: "fake" }, authMethods: [] } });
31+ case "session/new":
32+ sessions += 1;
33+ return send({ id: m.id, result: { sessionId: `s${sessions}` } });
34+ case "session/cancel":
35+ return cancelled.get(m.params.sessionId)?.();
36+ case "session/prompt":
37+ return prompt(m.id, m.params.sessionId, m.params.prompt[0].text);
38+ default:
39+ return send({ id: m.id, error: { code: -32601, message: "method not found" } });
40+ }
41+});
42+
43+async function prompt(id, sid, text) {
44+ update(sid, { sessionUpdate: "usage_update", size: 32768, used: 10 });
45+ if (text.includes("crash")) {
46+ update(sid, { sessionUpdate: "agent_message_chunk", content: { type: "text", text: "partial" } });
47+ process.exit(3);
48+ }
49+ if (text.includes("slow")) {
50+ update(sid, { sessionUpdate: "agent_message_chunk", content: { type: "text", text: "thinking about it" } });
51+ await new Promise((resolve) => cancelled.set(sid, resolve));
52+ return send({ id, result: { stopReason: "cancelled" } });
53+ }
54+ if (text.includes("tool")) {
55+ const call = { kind: "execute", rawInput: { command: "ls" }, status: "pending", title: "bash: ls", toolCallId: "call_1" };
56+ update(sid, { sessionUpdate: "tool_call", ...call });
57+ const ownId = nextOwnId++;
58+ const answer = await new Promise((resolve) => {
59+ waiting.set(ownId, resolve);
60+ send({
61+ id: ownId,
62+ method: "session/request_permission",
63+ params: {
64+ sessionId: sid,
65+ toolCall: call,
66+ options: [
67+ { kind: "allow_once", name: "Allow once", optionId: "allow" },
68+ { kind: "allow_always", name: "Allow bash for this session", optionId: "allow_always" },
69+ { kind: "reject_once", name: "Reject", optionId: "reject" },
70+ { kind: "reject_always", name: "Never allow bash in this session", optionId: "never" },
71+ ],
72+ },
73+ });
74+ });
75+ const allowed = answer.outcome.optionId.startsWith("allow");
76+ update(sid, {
77+ sessionUpdate: "tool_call_update",
78+ toolCallId: "call_1",
79+ status: allowed ? "completed" : "failed",
80+ rawOutput: { output: allowed ? "a.txt\nb.txt\n" : "[refused]" },
81+ });
82+ update(sid, { sessionUpdate: "agent_message_chunk", content: { type: "text", text: allowed ? "2 files" : "not allowed" } });
83+ return send({ id, result: { stopReason: "end_turn", usage: { inputTokens: 5, outputTokens: 2, totalTokens: 7 } } });
84+ }
85+ update(sid, { sessionUpdate: "agent_thought_chunk", content: { type: "text", text: "greet" } });
86+ update(sid, { sessionUpdate: "agent_message_chunk", content: { type: "text", text: "Hello" } });
87+ update(sid, { sessionUpdate: "agent_message_chunk", content: { type: "text", text: ", world" } });
88+ send({ id, result: { stopReason: "end_turn", usage: { inputTokens: 3, outputTokens: 2, totalTokens: 5 } } });
89+}
new file mode 100755
@@ -0,0 +1,89 @@
1+#!/usr/bin/env node
2+// A stand-in for `mm -acp` in the tests: it speaks the same ACP messages,
3+// in the same shapes as mm (captured from mm v1.0.1 on 2026-10-26), without
4+// a model. What it does depends on the prompt:
5+//
6+// "tool" announces a bash call, asks the permission, runs it or not
7+// "slow" streams nothing until session/cancel, then ends "cancelled"
8+// "crash" exits in the middle of the turn
9+// else answers "Hello" + ", world" in two chunks
10+//
11+// It writes one line on stderr, like mm's banner.
12+
13+import { createInterface } from "node:readline";
14+
15+const send = (m) => process.stdout.write(JSON.stringify({ jsonrpc: "2.0", ...m }) + "\n");
16+const update = (sessionId, u) => send({ method: "session/update", params: { sessionId, update: u } });
17+
18+let sessions = 0;
19+let nextOwnId = 1;
20+const waiting = new Map(); // our request id → resolve
21+const cancelled = new Map(); // sessionId → resolve
22+
23+process.stderr.write("fake-mm | model: none\n");
24+
25+createInterface({ input: process.stdin }).on("line", async (line) => {
26+ const m = JSON.parse(line);
27+ if (m.id !== undefined && !m.method) return waiting.get(m.id)?.(m.result);
28+ switch (m.method) {
29+ case "initialize":
30+ return send({ id: m.id, result: { protocolVersion: 1, agentCapabilities: {}, agentInfo: { name: "fake" }, authMethods: [] } });
31+ case "session/new":
32+ sessions += 1;
33+ return send({ id: m.id, result: { sessionId: `s${sessions}` } });
34+ case "session/cancel":
35+ return cancelled.get(m.params.sessionId)?.();
36+ case "session/prompt":
37+ return prompt(m.id, m.params.sessionId, m.params.prompt[0].text);
38+ default:
39+ return send({ id: m.id, error: { code: -32601, message: "method not found" } });
40+ }
41+});
42+
43+async function prompt(id, sid, text) {
44+ update(sid, { sessionUpdate: "usage_update", size: 32768, used: 10 });
45+ if (text.includes("crash")) {
46+ update(sid, { sessionUpdate: "agent_message_chunk", content: { type: "text", text: "partial" } });
47+ process.exit(3);
48+ }
49+ if (text.includes("slow")) {
50+ update(sid, { sessionUpdate: "agent_message_chunk", content: { type: "text", text: "thinking about it" } });
51+ await new Promise((resolve) => cancelled.set(sid, resolve));
52+ return send({ id, result: { stopReason: "cancelled" } });
53+ }
54+ if (text.includes("tool")) {
55+ const call = { kind: "execute", rawInput: { command: "ls" }, status: "pending", title: "bash: ls", toolCallId: "call_1" };
56+ update(sid, { sessionUpdate: "tool_call", ...call });
57+ const ownId = nextOwnId++;
58+ const answer = await new Promise((resolve) => {
59+ waiting.set(ownId, resolve);
60+ send({
61+ id: ownId,
62+ method: "session/request_permission",
63+ params: {
64+ sessionId: sid,
65+ toolCall: call,
66+ options: [
67+ { kind: "allow_once", name: "Allow once", optionId: "allow" },
68+ { kind: "allow_always", name: "Allow bash for this session", optionId: "allow_always" },
69+ { kind: "reject_once", name: "Reject", optionId: "reject" },
70+ { kind: "reject_always", name: "Never allow bash in this session", optionId: "never" },
71+ ],
72+ },
73+ });
74+ });
75+ const allowed = answer.outcome.optionId.startsWith("allow");
76+ update(sid, {
77+ sessionUpdate: "tool_call_update",
78+ toolCallId: "call_1",
79+ status: allowed ? "completed" : "failed",
80+ rawOutput: { output: allowed ? "a.txt\nb.txt\n" : "[refused]" },
81+ });
82+ update(sid, { sessionUpdate: "agent_message_chunk", content: { type: "text", text: allowed ? "2 files" : "not allowed" } });
83+ return send({ id, result: { stopReason: "end_turn", usage: { inputTokens: 5, outputTokens: 2, totalTokens: 7 } } });
84+ }
85+ update(sid, { sessionUpdate: "agent_thought_chunk", content: { type: "text", text: "greet" } });
86+ update(sid, { sessionUpdate: "agent_message_chunk", content: { type: "text", text: "Hello" } });
87+ update(sid, { sessionUpdate: "agent_message_chunk", content: { type: "text", text: ", world" } });
88+ send({ id, result: { stopReason: "end_turn", usage: { inputTokens: 3, outputTokens: 2, totalTokens: 5 } } });
89+}
added demo.acp.sdk/nodejs/test/minime.test.js +110 -0
new file mode 100644
@@ -0,0 +1,110 @@
1+// Run with `npm test` (or `node --test test/minime.test.js`) from
2+// demo.acp.sdk/nodejs/ — not a bare `node --test`, which would also run
3+// test/fake-mm.js as a test. No model, no mm: fake-mm.js plays mm's side.
4+import { test } from "node:test";
5+import assert from "node:assert/strict";
6+import { fileURLToPath } from "node:url";
7+import { Agent, pickOption } from "../minime.js";
8+
9+const fake = fileURLToPath(new URL("./fake-mm.js", import.meta.url));
10+const start = (options = {}) => Agent.start({ config: "agent.yaml", mm: fake, ...options });
11+
12+test("complete returns the whole answer once the turn is over", async () => {
13+ const agent = await start();
14+ const res = await agent.complete("hi");
15+ assert.equal(res.text, "Hello, world");
16+ assert.equal(res.stopReason, "end_turn");
17+ assert.equal(res.usage.totalTokens, 5);
18+ await agent.close();
19+});
20+
21+test("stream yields thinking, text chunks, then done", async () => {
22+ const agent = await start();
23+ const kinds = [];
24+ for await (const ev of agent.stream("hi")) kinds.push(ev.kind === "done" ? `done:${ev.text}` : `${ev.kind}:${ev.text}`);
25+ assert.deepEqual(kinds, ["thinking:greet", "text:Hello", "text:, world", "done:Hello, world"]);
26+ await agent.close();
27+});
28+
29+test("a tool call needing permission is refused when there is no callback", async () => {
30+ const agent = await start();
31+ const events = [];
32+ const res = await agent.complete("use a tool", { onEvent: (ev) => events.push(ev) });
33+ assert.equal(res.text, "not allowed");
34+ assert.deepEqual(events.find((e) => e.kind === "tool_call"), { kind: "tool_call", id: "call_1", title: "bash: ls", toolKind: "execute", input: { command: "ls" } });
35+ assert.equal(events.find((e) => e.kind === "tool_result").status, "failed");
36+ await agent.close();
37+});
38+
39+test("the permission callback sees the call and can allow it", async () => {
40+ const seen = [];
41+ const agent = await start({ onPermission: (call) => (seen.push(call.title), true) });
42+ const events = [];
43+ const res = await agent.complete("use a tool", { onEvent: (ev) => events.push(ev) });
44+ assert.deepEqual(seen, ["bash: ls"]);
45+ assert.equal(res.text, "2 files");
46+ assert.deepEqual(events.find((e) => e.kind === "tool_result"), { kind: "tool_result", id: "call_1", status: "completed", output: "a.txt\nb.txt\n" });
47+ await agent.close();
48+});
49+
50+test("aborting the signal cancels the turn and keeps the text so far", async () => {
51+ const agent = await start();
52+ const ac = new AbortController();
53+ setTimeout(() => ac.abort(), 50);
54+ const res = await agent.complete("slow please", { signal: ac.signal });
55+ assert.equal(res.stopReason, "cancelled");
56+ assert.equal(res.text, "thinking about it");
57+ const after = await agent.complete("hi"); // the agent is usable again
58+ assert.equal(after.text, "Hello, world");
59+ await agent.close();
60+});
61+
62+test("breaking out of stream cancels the turn", async () => {
63+ const agent = await start();
64+ for await (const ev of agent.stream("slow please")) {
65+ if (ev.kind === "text") break;
66+ }
67+ assert.equal((await agent.complete("hi")).text, "Hello, world");
68+ await agent.close();
69+});
70+
71+test("one turn at a time on one agent", async () => {
72+ const agent = await start();
73+ const ac = new AbortController();
74+ const first = agent.complete("slow please", { signal: ac.signal });
75+ await new Promise((r) => setTimeout(r, 30));
76+ await assert.rejects(agent.complete("hi"), /already running/);
77+ ac.abort();
78+ await first;
79+ await agent.close();
80+});
81+
82+test("reset opens a new session", async () => {
83+ const agent = await start();
84+ const before = agent.sessionId;
85+ await agent.reset();
86+ assert.notEqual(agent.sessionId, before);
87+ await agent.close();
88+});
89+
90+test("mm exiting during a turn rejects it with the last stderr line", async () => {
91+ const agent = await start();
92+ await assert.rejects(agent.complete("crash now"), /mm exited \(3\) — fake-mm \| model: none/);
93+});
94+
95+test("a missing binary is reported", async () => {
96+ await assert.rejects(Agent.start({ config: "agent.yaml", mm: "/nonexistent/mm" }), /cannot start \/nonexistent\/mm/);
97+});
98+
99+test("pickOption maps true/false to the once options and passes a known optionId", () => {
100+ const options = [
101+ { kind: "allow_once", optionId: "allow" },
102+ { kind: "allow_always", optionId: "allow_always" },
103+ { kind: "reject_once", optionId: "reject" },
104+ { kind: "reject_always", optionId: "never" },
105+ ];
106+ assert.equal(pickOption(options, true), "allow");
107+ assert.equal(pickOption(options, false), "reject");
108+ assert.equal(pickOption(options, "never"), "never");
109+ assert.equal(pickOption(options, "nonsense"), "reject");
110+});
new file mode 100644
@@ -0,0 +1,110 @@
1+// Run with `npm test` (or `node --test test/minime.test.js`) from
2+// demo.acp.sdk/nodejs/ — not a bare `node --test`, which would also run
3+// test/fake-mm.js as a test. No model, no mm: fake-mm.js plays mm's side.
4+import { test } from "node:test";
5+import assert from "node:assert/strict";
6+import { fileURLToPath } from "node:url";
7+import { Agent, pickOption } from "../minime.js";
8+
9+const fake = fileURLToPath(new URL("./fake-mm.js", import.meta.url));
10+const start = (options = {}) => Agent.start({ config: "agent.yaml", mm: fake, ...options });
11+
12+test("complete returns the whole answer once the turn is over", async () => {
13+ const agent = await start();
14+ const res = await agent.complete("hi");
15+ assert.equal(res.text, "Hello, world");
16+ assert.equal(res.stopReason, "end_turn");
17+ assert.equal(res.usage.totalTokens, 5);
18+ await agent.close();
19+});
20+
21+test("stream yields thinking, text chunks, then done", async () => {
22+ const agent = await start();
23+ const kinds = [];
24+ for await (const ev of agent.stream("hi")) kinds.push(ev.kind === "done" ? `done:${ev.text}` : `${ev.kind}:${ev.text}`);
25+ assert.deepEqual(kinds, ["thinking:greet", "text:Hello", "text:, world", "done:Hello, world"]);
26+ await agent.close();
27+});
28+
29+test("a tool call needing permission is refused when there is no callback", async () => {
30+ const agent = await start();
31+ const events = [];
32+ const res = await agent.complete("use a tool", { onEvent: (ev) => events.push(ev) });
33+ assert.equal(res.text, "not allowed");
34+ assert.deepEqual(events.find((e) => e.kind === "tool_call"), { kind: "tool_call", id: "call_1", title: "bash: ls", toolKind: "execute", input: { command: "ls" } });
35+ assert.equal(events.find((e) => e.kind === "tool_result").status, "failed");
36+ await agent.close();
37+});
38+
39+test("the permission callback sees the call and can allow it", async () => {
40+ const seen = [];
41+ const agent = await start({ onPermission: (call) => (seen.push(call.title), true) });
42+ const events = [];
43+ const res = await agent.complete("use a tool", { onEvent: (ev) => events.push(ev) });
44+ assert.deepEqual(seen, ["bash: ls"]);
45+ assert.equal(res.text, "2 files");
46+ assert.deepEqual(events.find((e) => e.kind === "tool_result"), { kind: "tool_result", id: "call_1", status: "completed", output: "a.txt\nb.txt\n" });
47+ await agent.close();
48+});
49+
50+test("aborting the signal cancels the turn and keeps the text so far", async () => {
51+ const agent = await start();
52+ const ac = new AbortController();
53+ setTimeout(() => ac.abort(), 50);
54+ const res = await agent.complete("slow please", { signal: ac.signal });
55+ assert.equal(res.stopReason, "cancelled");
56+ assert.equal(res.text, "thinking about it");
57+ const after = await agent.complete("hi"); // the agent is usable again
58+ assert.equal(after.text, "Hello, world");
59+ await agent.close();
60+});
61+
62+test("breaking out of stream cancels the turn", async () => {
63+ const agent = await start();
64+ for await (const ev of agent.stream("slow please")) {
65+ if (ev.kind === "text") break;
66+ }
67+ assert.equal((await agent.complete("hi")).text, "Hello, world");
68+ await agent.close();
69+});
70+
71+test("one turn at a time on one agent", async () => {
72+ const agent = await start();
73+ const ac = new AbortController();
74+ const first = agent.complete("slow please", { signal: ac.signal });
75+ await new Promise((r) => setTimeout(r, 30));
76+ await assert.rejects(agent.complete("hi"), /already running/);
77+ ac.abort();
78+ await first;
79+ await agent.close();
80+});
81+
82+test("reset opens a new session", async () => {
83+ const agent = await start();
84+ const before = agent.sessionId;
85+ await agent.reset();
86+ assert.notEqual(agent.sessionId, before);
87+ await agent.close();
88+});
89+
90+test("mm exiting during a turn rejects it with the last stderr line", async () => {
91+ const agent = await start();
92+ await assert.rejects(agent.complete("crash now"), /mm exited \(3\) — fake-mm \| model: none/);
93+});
94+
95+test("a missing binary is reported", async () => {
96+ await assert.rejects(Agent.start({ config: "agent.yaml", mm: "/nonexistent/mm" }), /cannot start \/nonexistent\/mm/);
97+});
98+
99+test("pickOption maps true/false to the once options and passes a known optionId", () => {
100+ const options = [
101+ { kind: "allow_once", optionId: "allow" },
102+ { kind: "allow_always", optionId: "allow_always" },
103+ { kind: "reject_once", optionId: "reject" },
104+ { kind: "reject_always", optionId: "never" },
105+ ];
106+ assert.equal(pickOption(options, true), "allow");
107+ assert.equal(pickOption(options, false), "reject");
108+ assert.equal(pickOption(options, "never"), "never");
109+ assert.equal(pickOption(options, "nonsense"), "reject");
110+});
modified demos/coding-helpers-unload-option/README.en.md +4 -0
@@ -12,6 +12,10 @@ The same team as [`demos/coding-helpers`](../coding-helpers/) — a coordinator
1212
1313 The two specialists are copies of those of `demos/coding-helpers`. Only `agent.yaml` differs: `unloadModels: true` on both entries, and no `hooks:` block.
1414
15+![How unloadModels works: mm unloads the coordinator's model before the specialist starts, then the specialist's model once it has answered, so the two models are never loaded together](coding-helpers-unload-option-flow.svg)
16+
17+Editable source: `coding-helpers-unload-option-flow.drawio` (draw.io); `coding-helpers-unload-option-flow.svg` is the same picture.
18+
1519 ## Run
1620
1721 ```bash
@@ -12,6 +12,10 @@ The same team as [`demos/coding-helpers`](../coding-helpers/) — a coordinator
12 12
13 The two specialists are copies of those of `demos/coding-helpers`. Only `agent.yaml` differs: `unloadModels: true` on both entries, and no `hooks:` block.13 The two specialists are copies of those of `demos/coding-helpers`. Only `agent.yaml` differs: `unloadModels: true` on both entries, and no `hooks:` block.
14 14
15+![How unloadModels works: mm unloads the coordinator's model before the specialist starts, then the specialist's model once it has answered, so the two models are never loaded together](coding-helpers-unload-option-flow.svg)
16+
17+Editable source: `coding-helpers-unload-option-flow.drawio` (draw.io); `coding-helpers-unload-option-flow.svg` is the same picture.
18+
15 ## Run19 ## Run
16 20
17 ```bash21 ```bash
modified demos/coding-helpers-unload-option/README.fr.md +4 -0
@@ -12,6 +12,10 @@ La même équipe que [`demos/coding-helpers`](../coding-helpers/) — un coordin
1212
1313 Les deux spécialistes sont des copies de ceux de `demos/coding-helpers`. Seul `agent.yaml` change : `unloadModels: true` sur les deux entrées, et pas de bloc `hooks:`.
1414
15+![Le fonctionnement de unloadModels : mm décharge le modèle du coordinateur avant que le spécialiste démarre, puis celui du spécialiste une fois sa réponse donnée, si bien que les deux modèles ne sont jamais chargés ensemble](coding-helpers-unload-option-flow.svg)
16+
17+Source modifiable : `coding-helpers-unload-option-flow.drawio` (draw.io) ; `coding-helpers-unload-option-flow.svg` est la même image.
18+
1519 ## Lancer
1620
1721 ```bash
@@ -12,6 +12,10 @@ La même équipe que [`demos/coding-helpers`](../coding-helpers/) — un coordin
12 12
13 Les deux spécialistes sont des copies de ceux de `demos/coding-helpers`. Seul `agent.yaml` change : `unloadModels: true` sur les deux entrées, et pas de bloc `hooks:`.13 Les deux spécialistes sont des copies de ceux de `demos/coding-helpers`. Seul `agent.yaml` change : `unloadModels: true` sur les deux entrées, et pas de bloc `hooks:`.
14 14
15+![Le fonctionnement de unloadModels : mm décharge le modèle du coordinateur avant que le spécialiste démarre, puis celui du spécialiste une fois sa réponse donnée, si bien que les deux modèles ne sont jamais chargés ensemble](coding-helpers-unload-option-flow.svg)
16+
17+Source modifiable : `coding-helpers-unload-option-flow.drawio` (draw.io) ; `coding-helpers-unload-option-flow.svg` est la même image.
18+
15 ## Lancer19 ## Lancer
16 20
17 ```bash21 ```bash
added demos/coding-helpers-unload-option/coding-helpers-unload-option-flow.drawio +31 -0
new file mode 100644
@@ -0,0 +1,31 @@
1+<mxfile host="app.diagrams.net">
2+ <diagram id="coding-helpers-unload-option-flow" name="coding-helpers-unload-option — the models take turns">
3+ <mxGraphModel dx="1200" dy="640" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1200" pageHeight="640" background="#F5F7FC" math="0" shadow="0">
4+ <root>
5+ <mxCell id="0"/>
6+ <mxCell id="1" parent="0"/>
7+ <mxCell id="title" value="&lt;b style=&quot;font-size:25px;color:#1A237E&quot;&gt;coding-helpers-unload-option — the models take turns&lt;/b&gt;&lt;br&gt;&lt;i style=&quot;font-size:14px;color:#546E7A&quot;&gt;unloadModels: true on an mmConfigs entry: mm itself unloads a model at each hand-off, no hook&lt;/i&gt;" style="text;html=1;align=center;verticalAlign=middle;strokeColor=none;fillColor=none;" vertex="1" parent="1"><mxGeometry x="150.0" y="12" width="900" height="56" as="geometry"/></mxCell>
8+ <mxCell id="c8" value="" style="edgeStyle=none;rounded=0;html=1;strokeColor=#37474F;strokeWidth=2;endArrow=block;endFill=1;startArrow=none;" edge="1" parent="1"><mxGeometry relative="1" as="geometry"><mxPoint x="200" y="205" as="sourcePoint"/><mxPoint x="250" y="205" as="targetPoint"/></mxGeometry></mxCell>
9+ <mxCell id="c9" value="" style="edgeStyle=none;rounded=0;html=1;strokeColor=#37474F;strokeWidth=2;endArrow=block;endFill=1;startArrow=none;" edge="1" parent="1"><mxGeometry relative="1" as="geometry"><mxPoint x="470" y="205" as="sourcePoint"/><mxPoint x="570" y="205" as="targetPoint"/></mxGeometry></mxCell>
10+ <mxCell id="c11" value="" style="edgeStyle=none;rounded=0;html=1;strokeColor=#37474F;strokeWidth=2;endArrow=block;endFill=1;startArrow=none;" edge="1" parent="1"><mxGeometry relative="1" as="geometry"><mxPoint x="810" y="205" as="sourcePoint"/><mxPoint x="910" y="205" as="targetPoint"/></mxGeometry></mxCell>
11+ <mxCell id="c13" value="" style="edgeStyle=none;rounded=0;html=1;strokeColor=#37474F;strokeWidth=2;endArrow=block;endFill=1;startArrow=none;" edge="1" parent="1"><mxGeometry relative="1" as="geometry"><mxPoint x="1040" y="275" as="sourcePoint"/><mxPoint x="1040" y="390" as="targetPoint"/></mxGeometry></mxCell>
12+ <mxCell id="c15" value="" style="edgeStyle=none;rounded=0;html=1;strokeColor=#37474F;strokeWidth=2;endArrow=block;endFill=1;startArrow=none;" edge="1" parent="1"><mxGeometry relative="1" as="geometry"><mxPoint x="910" y="455" as="sourcePoint"/><mxPoint x="730" y="455" as="targetPoint"/></mxGeometry></mxCell>
13+ <mxCell id="c17" value="" style="edgeStyle=none;rounded=0;html=1;strokeColor=#37474F;strokeWidth=2;endArrow=block;endFill=1;startArrow=none;" edge="1" parent="1"><mxGeometry relative="1" as="geometry"><mxPoint x="470" y="455" as="sourcePoint"/><mxPoint x="115" y="265" as="targetPoint"/><Array as="points"><mxPoint x="115" y="455"/></Array></mxGeometry></mxCell>
14+ <mxCell id="c1" value="🧑&lt;br&gt;&lt;b style=&quot;font-size:16px&quot;&gt;You&lt;/b&gt;&lt;br&gt;&lt;span style=&quot;font-size:11.5px;font-weight:normal&quot;&gt;a Go or Rust question&lt;/span&gt;" style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#E3F2FD;gradientColor=#BBDEFB;strokeColor=#1565C0;strokeWidth=2;shadow=1;fontSize=20;fontColor=#0D47A1;" vertex="1" parent="1"><mxGeometry x="30" y="145" width="170" height="120" as="geometry"/></mxCell>
15+ <mxCell id="c2" value="🔀&lt;br&gt;&lt;b style=&quot;font-size:16px&quot;&gt;Coordinator&lt;/b&gt;&lt;br&gt;&lt;span style=&quot;font-size:11.5px;font-weight:normal&quot;&gt;MiniCPM5 2B · 1.6 GB&lt;br&gt;agent.yaml&lt;br&gt;ask_agent go / rust&lt;/span&gt;" style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#F3E5F5;gradientColor=#E1BEE7;strokeColor=#6A1B9A;strokeWidth=3;shadow=1;fontSize=20;fontColor=#4A148C;" vertex="1" parent="1"><mxGeometry x="250" y="135" width="220" height="140" as="geometry"/></mxCell>
16+ <mxCell id="c3" value="&lt;b&gt;★ asks&lt;/b&gt;" style="rounded=1;arcSize=50;html=1;fillColor=#6A1B9A;strokeColor=none;fontColor=#FFFFFF;fontSize=11;" vertex="1" parent="1"><mxGeometry x="422.8" y="123" width="57.2" height="24" as="geometry"/></mxCell>
17+ <mxCell id="c4" value="⚙️&lt;br&gt;&lt;b style=&quot;font-size:16px&quot;&gt;1 · mm unloads&lt;/b&gt;&lt;br&gt;&lt;span style=&quot;font-size:11.5px;font-weight:normal&quot;&gt;the COORDINATOR's model&lt;br&gt;before the specialist starts&lt;/span&gt;" style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#FFFDE7;gradientColor=#FFF9C4;strokeColor=#F9A825;strokeWidth=2;shadow=1;fontSize=20;fontColor=#E65100;" vertex="1" parent="1"><mxGeometry x="570" y="140" width="240" height="130" as="geometry"/></mxCell>
18+ <mxCell id="c5" value="🤖&lt;br&gt;&lt;b style=&quot;font-size:16px&quot;&gt;Specialist go / rust&lt;/b&gt;&lt;br&gt;&lt;span style=&quot;font-size:11.5px;font-weight:normal&quot;&gt;Mellum2 12B · 8.1 GB&lt;br&gt;agent.go.yaml / agent.rust.yaml&lt;br&gt;loaded alone · answers&lt;/span&gt;" style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#E1F5FE;gradientColor=#B3E5FC;strokeColor=#0277BD;strokeWidth=2;shadow=1;fontSize=20;fontColor=#01579B;" vertex="1" parent="1"><mxGeometry x="910" y="135" width="260" height="140" as="geometry"/></mxCell>
19+ <mxCell id="c6" value="⚙️&lt;br&gt;&lt;b style=&quot;font-size:16px&quot;&gt;2 · mm unloads&lt;/b&gt;&lt;br&gt;&lt;span style=&quot;font-size:11.5px;font-weight:normal&quot;&gt;the SPECIALIST's model&lt;br&gt;once it has answered&lt;/span&gt;" style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#FFFDE7;gradientColor=#FFF9C4;strokeColor=#F9A825;strokeWidth=2;shadow=1;fontSize=20;fontColor=#E65100;" vertex="1" parent="1"><mxGeometry x="910" y="390" width="260" height="130" as="geometry"/></mxCell>
20+ <mxCell id="c7" value="🔀&lt;br&gt;&lt;b style=&quot;font-size:16px&quot;&gt;Coordinator&lt;/b&gt;&lt;br&gt;&lt;span style=&quot;font-size:11.5px;font-weight:normal&quot;&gt;MiniCPM5 reloads alone&lt;br&gt;reads the answer · replies&lt;/span&gt;" style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#F3E5F5;gradientColor=#E1BEE7;strokeColor=#6A1B9A;strokeWidth=3;shadow=1;fontSize=20;fontColor=#4A148C;" vertex="1" parent="1"><mxGeometry x="470" y="390" width="260" height="130" as="geometry"/></mxCell>
21+ <mxCell id="c10" value="&lt;b style=&quot;font-size:12.5px;color:#263238&quot;&gt;ask_agent&lt;/b&gt;" style="text;html=1;align=center;verticalAlign=middle;fillColor=#FFFFFF;strokeColor=none;rounded=1;" vertex="1" parent="1"><mxGeometry x="482.0" y="193.0" width="76" height="24" as="geometry"/></mxCell>
22+ <mxCell id="c12" value="&lt;b style=&quot;font-size:12.5px;color:#263238&quot;&gt;question&lt;/b&gt;" style="text;html=1;align=center;verticalAlign=middle;fillColor=#FFFFFF;strokeColor=none;rounded=1;" vertex="1" parent="1"><mxGeometry x="825.0" y="193.0" width="70" height="24" as="geometry"/></mxCell>
23+ <mxCell id="c14" value="&lt;b style=&quot;font-size:12.5px;color:#263238&quot;&gt;answer&lt;/b&gt;" style="text;html=1;align=center;verticalAlign=middle;fillColor=#FFFFFF;strokeColor=none;rounded=1;" vertex="1" parent="1"><mxGeometry x="1008.0" y="320.0" width="64" height="24" as="geometry"/></mxCell>
24+ <mxCell id="c16" value="&lt;b style=&quot;font-size:12.5px;color:#263238&quot;&gt;answer&lt;/b&gt;" style="text;html=1;align=center;verticalAlign=middle;fillColor=#FFFFFF;strokeColor=none;rounded=1;" vertex="1" parent="1"><mxGeometry x="783.0" y="443.0" width="64" height="24" as="geometry"/></mxCell>
25+ <mxCell id="c18" value="&lt;b style=&quot;font-size:12.5px;color:#263238&quot;&gt;reply&lt;/b&gt;" style="text;html=1;align=center;verticalAlign=middle;fillColor=#FFFFFF;strokeColor=none;rounded=1;" vertex="1" parent="1"><mxGeometry x="275.0" y="443.0" width="50" height="24" as="geometry"/></mxCell>
26+ <mxCell id="c19" value="Each unload: POST /api/generate {&quot;model&quot;: …, &quot;keep_alive&quot;: 0} on that agent's server, shown as a line on screen." style="text;html=1;align=center;verticalAlign=middle;strokeColor=none;fillColor=none;fontSize=12;fontColor=#455A64;fontStyle=2;" vertex="1" parent="1"><mxGeometry x="183.36" y="556" width="833.28" height="18" as="geometry"/></mxCell>
27+ <mxCell id="c20" value="Measured on llmman (/api/ps every 0.5 s): MiniCPM5 → nothing → Mellum2 → nothing → MiniCPM5 · Rust 31 s · Go 79 s." style="text;html=1;align=center;verticalAlign=middle;strokeColor=none;fillColor=none;fontSize=12;fontColor=#455A64;fontStyle=2;" vertex="1" parent="1"><mxGeometry x="175.92000000000002" y="584" width="848.16" height="18" as="geometry"/></mxCell>
28+ </root>
29+ </mxGraphModel>
30+ </diagram>
31+</mxfile>
new file mode 100644
@@ -0,0 +1,31 @@
1+<mxfile host="app.diagrams.net">
2+ <diagram id="coding-helpers-unload-option-flow" name="coding-helpers-unload-option — the models take turns">
3+ <mxGraphModel dx="1200" dy="640" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1200" pageHeight="640" background="#F5F7FC" math="0" shadow="0">
4+ <root>
5+ <mxCell id="0"/>
6+ <mxCell id="1" parent="0"/>
7+ <mxCell id="title" value="&lt;b style=&quot;font-size:25px;color:#1A237E&quot;&gt;coding-helpers-unload-option — the models take turns&lt;/b&gt;&lt;br&gt;&lt;i style=&quot;font-size:14px;color:#546E7A&quot;&gt;unloadModels: true on an mmConfigs entry: mm itself unloads a model at each hand-off, no hook&lt;/i&gt;" style="text;html=1;align=center;verticalAlign=middle;strokeColor=none;fillColor=none;" vertex="1" parent="1"><mxGeometry x="150.0" y="12" width="900" height="56" as="geometry"/></mxCell>
8+ <mxCell id="c8" value="" style="edgeStyle=none;rounded=0;html=1;strokeColor=#37474F;strokeWidth=2;endArrow=block;endFill=1;startArrow=none;" edge="1" parent="1"><mxGeometry relative="1" as="geometry"><mxPoint x="200" y="205" as="sourcePoint"/><mxPoint x="250" y="205" as="targetPoint"/></mxGeometry></mxCell>
9+ <mxCell id="c9" value="" style="edgeStyle=none;rounded=0;html=1;strokeColor=#37474F;strokeWidth=2;endArrow=block;endFill=1;startArrow=none;" edge="1" parent="1"><mxGeometry relative="1" as="geometry"><mxPoint x="470" y="205" as="sourcePoint"/><mxPoint x="570" y="205" as="targetPoint"/></mxGeometry></mxCell>
10+ <mxCell id="c11" value="" style="edgeStyle=none;rounded=0;html=1;strokeColor=#37474F;strokeWidth=2;endArrow=block;endFill=1;startArrow=none;" edge="1" parent="1"><mxGeometry relative="1" as="geometry"><mxPoint x="810" y="205" as="sourcePoint"/><mxPoint x="910" y="205" as="targetPoint"/></mxGeometry></mxCell>
11+ <mxCell id="c13" value="" style="edgeStyle=none;rounded=0;html=1;strokeColor=#37474F;strokeWidth=2;endArrow=block;endFill=1;startArrow=none;" edge="1" parent="1"><mxGeometry relative="1" as="geometry"><mxPoint x="1040" y="275" as="sourcePoint"/><mxPoint x="1040" y="390" as="targetPoint"/></mxGeometry></mxCell>
12+ <mxCell id="c15" value="" style="edgeStyle=none;rounded=0;html=1;strokeColor=#37474F;strokeWidth=2;endArrow=block;endFill=1;startArrow=none;" edge="1" parent="1"><mxGeometry relative="1" as="geometry"><mxPoint x="910" y="455" as="sourcePoint"/><mxPoint x="730" y="455" as="targetPoint"/></mxGeometry></mxCell>
13+ <mxCell id="c17" value="" style="edgeStyle=none;rounded=0;html=1;strokeColor=#37474F;strokeWidth=2;endArrow=block;endFill=1;startArrow=none;" edge="1" parent="1"><mxGeometry relative="1" as="geometry"><mxPoint x="470" y="455" as="sourcePoint"/><mxPoint x="115" y="265" as="targetPoint"/><Array as="points"><mxPoint x="115" y="455"/></Array></mxGeometry></mxCell>
14+ <mxCell id="c1" value="🧑&lt;br&gt;&lt;b style=&quot;font-size:16px&quot;&gt;You&lt;/b&gt;&lt;br&gt;&lt;span style=&quot;font-size:11.5px;font-weight:normal&quot;&gt;a Go or Rust question&lt;/span&gt;" style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#E3F2FD;gradientColor=#BBDEFB;strokeColor=#1565C0;strokeWidth=2;shadow=1;fontSize=20;fontColor=#0D47A1;" vertex="1" parent="1"><mxGeometry x="30" y="145" width="170" height="120" as="geometry"/></mxCell>
15+ <mxCell id="c2" value="🔀&lt;br&gt;&lt;b style=&quot;font-size:16px&quot;&gt;Coordinator&lt;/b&gt;&lt;br&gt;&lt;span style=&quot;font-size:11.5px;font-weight:normal&quot;&gt;MiniCPM5 2B · 1.6 GB&lt;br&gt;agent.yaml&lt;br&gt;ask_agent go / rust&lt;/span&gt;" style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#F3E5F5;gradientColor=#E1BEE7;strokeColor=#6A1B9A;strokeWidth=3;shadow=1;fontSize=20;fontColor=#4A148C;" vertex="1" parent="1"><mxGeometry x="250" y="135" width="220" height="140" as="geometry"/></mxCell>
16+ <mxCell id="c3" value="&lt;b&gt;★ asks&lt;/b&gt;" style="rounded=1;arcSize=50;html=1;fillColor=#6A1B9A;strokeColor=none;fontColor=#FFFFFF;fontSize=11;" vertex="1" parent="1"><mxGeometry x="422.8" y="123" width="57.2" height="24" as="geometry"/></mxCell>
17+ <mxCell id="c4" value="⚙️&lt;br&gt;&lt;b style=&quot;font-size:16px&quot;&gt;1 · mm unloads&lt;/b&gt;&lt;br&gt;&lt;span style=&quot;font-size:11.5px;font-weight:normal&quot;&gt;the COORDINATOR's model&lt;br&gt;before the specialist starts&lt;/span&gt;" style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#FFFDE7;gradientColor=#FFF9C4;strokeColor=#F9A825;strokeWidth=2;shadow=1;fontSize=20;fontColor=#E65100;" vertex="1" parent="1"><mxGeometry x="570" y="140" width="240" height="130" as="geometry"/></mxCell>
18+ <mxCell id="c5" value="🤖&lt;br&gt;&lt;b style=&quot;font-size:16px&quot;&gt;Specialist go / rust&lt;/b&gt;&lt;br&gt;&lt;span style=&quot;font-size:11.5px;font-weight:normal&quot;&gt;Mellum2 12B · 8.1 GB&lt;br&gt;agent.go.yaml / agent.rust.yaml&lt;br&gt;loaded alone · answers&lt;/span&gt;" style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#E1F5FE;gradientColor=#B3E5FC;strokeColor=#0277BD;strokeWidth=2;shadow=1;fontSize=20;fontColor=#01579B;" vertex="1" parent="1"><mxGeometry x="910" y="135" width="260" height="140" as="geometry"/></mxCell>
19+ <mxCell id="c6" value="⚙️&lt;br&gt;&lt;b style=&quot;font-size:16px&quot;&gt;2 · mm unloads&lt;/b&gt;&lt;br&gt;&lt;span style=&quot;font-size:11.5px;font-weight:normal&quot;&gt;the SPECIALIST's model&lt;br&gt;once it has answered&lt;/span&gt;" style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#FFFDE7;gradientColor=#FFF9C4;strokeColor=#F9A825;strokeWidth=2;shadow=1;fontSize=20;fontColor=#E65100;" vertex="1" parent="1"><mxGeometry x="910" y="390" width="260" height="130" as="geometry"/></mxCell>
20+ <mxCell id="c7" value="🔀&lt;br&gt;&lt;b style=&quot;font-size:16px&quot;&gt;Coordinator&lt;/b&gt;&lt;br&gt;&lt;span style=&quot;font-size:11.5px;font-weight:normal&quot;&gt;MiniCPM5 reloads alone&lt;br&gt;reads the answer · replies&lt;/span&gt;" style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#F3E5F5;gradientColor=#E1BEE7;strokeColor=#6A1B9A;strokeWidth=3;shadow=1;fontSize=20;fontColor=#4A148C;" vertex="1" parent="1"><mxGeometry x="470" y="390" width="260" height="130" as="geometry"/></mxCell>
21+ <mxCell id="c10" value="&lt;b style=&quot;font-size:12.5px;color:#263238&quot;&gt;ask_agent&lt;/b&gt;" style="text;html=1;align=center;verticalAlign=middle;fillColor=#FFFFFF;strokeColor=none;rounded=1;" vertex="1" parent="1"><mxGeometry x="482.0" y="193.0" width="76" height="24" as="geometry"/></mxCell>
22+ <mxCell id="c12" value="&lt;b style=&quot;font-size:12.5px;color:#263238&quot;&gt;question&lt;/b&gt;" style="text;html=1;align=center;verticalAlign=middle;fillColor=#FFFFFF;strokeColor=none;rounded=1;" vertex="1" parent="1"><mxGeometry x="825.0" y="193.0" width="70" height="24" as="geometry"/></mxCell>
23+ <mxCell id="c14" value="&lt;b style=&quot;font-size:12.5px;color:#263238&quot;&gt;answer&lt;/b&gt;" style="text;html=1;align=center;verticalAlign=middle;fillColor=#FFFFFF;strokeColor=none;rounded=1;" vertex="1" parent="1"><mxGeometry x="1008.0" y="320.0" width="64" height="24" as="geometry"/></mxCell>
24+ <mxCell id="c16" value="&lt;b style=&quot;font-size:12.5px;color:#263238&quot;&gt;answer&lt;/b&gt;" style="text;html=1;align=center;verticalAlign=middle;fillColor=#FFFFFF;strokeColor=none;rounded=1;" vertex="1" parent="1"><mxGeometry x="783.0" y="443.0" width="64" height="24" as="geometry"/></mxCell>
25+ <mxCell id="c18" value="&lt;b style=&quot;font-size:12.5px;color:#263238&quot;&gt;reply&lt;/b&gt;" style="text;html=1;align=center;verticalAlign=middle;fillColor=#FFFFFF;strokeColor=none;rounded=1;" vertex="1" parent="1"><mxGeometry x="275.0" y="443.0" width="50" height="24" as="geometry"/></mxCell>
26+ <mxCell id="c19" value="Each unload: POST /api/generate {&quot;model&quot;: …, &quot;keep_alive&quot;: 0} on that agent's server, shown as a line on screen." style="text;html=1;align=center;verticalAlign=middle;strokeColor=none;fillColor=none;fontSize=12;fontColor=#455A64;fontStyle=2;" vertex="1" parent="1"><mxGeometry x="183.36" y="556" width="833.28" height="18" as="geometry"/></mxCell>
27+ <mxCell id="c20" value="Measured on llmman (/api/ps every 0.5 s): MiniCPM5 → nothing → Mellum2 → nothing → MiniCPM5 · Rust 31 s · Go 79 s." style="text;html=1;align=center;verticalAlign=middle;strokeColor=none;fillColor=none;fontSize=12;fontColor=#455A64;fontStyle=2;" vertex="1" parent="1"><mxGeometry x="175.92000000000002" y="584" width="848.16" height="18" as="geometry"/></mxCell>
28+ </root>
29+ </mxGraphModel>
30+ </diagram>
31+</mxfile>
added demos/coding-helpers-unload-option/coding-helpers-unload-option-flow.png +0 -0
new file mode 100644
Binary files /dev/null and b/demos/coding-helpers-unload-option/coding-helpers-unload-option-flow.png differ
new file mode 100644
Binary files /dev/null and b/demos/coding-helpers-unload-option/coding-helpers-unload-option-flow.png differBinary files /dev/null and b/demos/coding-helpers-unload-option/coding-helpers-unload-option-flow.png differ
added demos/coding-helpers-unload-option/coding-helpers-unload-option-flow.svg +66 -0
new file mode 100644
@@ -0,0 +1,66 @@
1+<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="640" viewBox="0 0 1200 640" font-family="DejaVu Sans, Verdana, sans-serif">
2+ <defs>
3+ <linearGradient id="gBg" x1="0" y1="0" x2="0" y2="1"><stop offset="0%" stop-color="#FFFFFF"/><stop offset="100%" stop-color="#EEF2FA"/></linearGradient>
4+ <linearGradient id="gBlue" x1="0" y1="0" x2="0" y2="1"><stop offset="0%" stop-color="#E3F2FD"/><stop offset="100%" stop-color="#BBDEFB"/></linearGradient>
5+ <linearGradient id="gBlue2" x1="0" y1="0" x2="0" y2="1"><stop offset="0%" stop-color="#E1F5FE"/><stop offset="100%" stop-color="#B3E5FC"/></linearGradient>
6+ <linearGradient id="gPurple" x1="0" y1="0" x2="0" y2="1"><stop offset="0%" stop-color="#F3E5F5"/><stop offset="100%" stop-color="#E1BEE7"/></linearGradient>
7+ <linearGradient id="gYellow" x1="0" y1="0" x2="0" y2="1"><stop offset="0%" stop-color="#FFFDE7"/><stop offset="100%" stop-color="#FFF9C4"/></linearGradient>
8+ <linearGradient id="gGrey" x1="0" y1="0" x2="0" y2="1"><stop offset="0%" stop-color="#ECEFF1"/><stop offset="100%" stop-color="#CFD8DC"/></linearGradient>
9+ <filter id="shadow" x="-30%" y="-30%" width="160%" height="160%"><feDropShadow dx="2" dy="3" stdDeviation="4" flood-color="#000000" flood-opacity="0.25"/></filter>
10+ <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto-start-reverse"><path d="M0,0 L10,5 L0,10 z" fill="#37474F"/></marker>
11+ </defs>
12+ <rect x="0" y="0" width="1200" height="640" fill="url(#gBg)"/>
13+ <text x="600.0" y="38" text-anchor="middle" font-size="25" font-weight="700" fill="#1A237E">coding-helpers-unload-option — the models take turns</text>
14+ <text x="600.0" y="61" text-anchor="middle" font-size="14" font-style="italic" fill="#546E7A">unloadModels: true on an mmConfigs entry: mm itself unloads a model at each hand-off, no hook</text>
15+ <path d="M200,205 L250,205" fill="none" stroke="#37474F" stroke-width="2.2" marker-end="url(#arrow)"/>
16+ <path d="M470,205 L570,205" fill="none" stroke="#37474F" stroke-width="2.2" marker-end="url(#arrow)"/>
17+ <path d="M810,205 L910,205" fill="none" stroke="#37474F" stroke-width="2.2" marker-end="url(#arrow)"/>
18+ <path d="M1040,275 L1040,390" fill="none" stroke="#37474F" stroke-width="2.2" marker-end="url(#arrow)"/>
19+ <path d="M910,455 L730,455" fill="none" stroke="#37474F" stroke-width="2.2" marker-end="url(#arrow)"/>
20+ <path d="M470,455 L115,455 L115,265" fill="none" stroke="#37474F" stroke-width="2.2" marker-end="url(#arrow)"/>
21+ <g filter="url(#shadow)"><rect x="30" y="145" width="170" height="120" rx="16" fill="url(#gBlue)" stroke="#1565C0" stroke-width="2"/></g>
22+ <text x="115.0" y="193.5" text-anchor="middle" font-size="22">🧑</text>
23+ <text x="115.0" y="216.5" text-anchor="middle" font-size="16" font-weight="700" fill="#0D47A1">You</text>
24+ <text x="115.0" y="234.5" text-anchor="middle" font-size="11.5" fill="#0D47A1">a Go or Rust question</text>
25+ <g filter="url(#shadow)"><rect x="250" y="135" width="220" height="140" rx="16" fill="url(#gPurple)" stroke="#6A1B9A" stroke-width="3"/></g>
26+ <text x="360.0" y="176.5" text-anchor="middle" font-size="22">🔀</text>
27+ <text x="360.0" y="199.5" text-anchor="middle" font-size="16" font-weight="700" fill="#4A148C">Coordinator</text>
28+ <text x="360.0" y="217.5" text-anchor="middle" font-size="11.5" fill="#4A148C">MiniCPM5 2B · 1.6 GB</text>
29+ <text x="360.0" y="234.5" text-anchor="middle" font-size="11.5" fill="#4A148C">agent.yaml</text>
30+ <text x="360.0" y="251.5" text-anchor="middle" font-size="11.5" fill="#4A148C">ask_agent go / rust</text>
31+ <rect x="422.8" y="123" width="57.2" height="24" rx="12" fill="#6A1B9A"/>
32+ <text x="451.40000000000003" y="139" text-anchor="middle" font-size="11.5" font-weight="700" fill="#FFFFFF">★ asks</text>
33+ <g filter="url(#shadow)"><rect x="570" y="140" width="240" height="130" rx="16" fill="url(#gYellow)" stroke="#F9A825" stroke-width="2"/></g>
34+ <text x="690.0" y="185.0" text-anchor="middle" font-size="22">⚙️</text>
35+ <text x="690.0" y="208.0" text-anchor="middle" font-size="16" font-weight="700" fill="#E65100">1 · mm unloads</text>
36+ <text x="690.0" y="226.0" text-anchor="middle" font-size="11.5" fill="#E65100">the COORDINATOR's model</text>
37+ <text x="690.0" y="243.0" text-anchor="middle" font-size="11.5" fill="#E65100">before the specialist starts</text>
38+ <g filter="url(#shadow)"><rect x="910" y="135" width="260" height="140" rx="16" fill="url(#gBlue2)" stroke="#0277BD" stroke-width="2"/></g>
39+ <text x="1040.0" y="176.5" text-anchor="middle" font-size="22">🤖</text>
40+ <text x="1040.0" y="199.5" text-anchor="middle" font-size="16" font-weight="700" fill="#01579B">Specialist go / rust</text>
41+ <text x="1040.0" y="217.5" text-anchor="middle" font-size="11.5" fill="#01579B">Mellum2 12B · 8.1 GB</text>
42+ <text x="1040.0" y="234.5" text-anchor="middle" font-size="11.5" fill="#01579B">agent.go.yaml / agent.rust.yaml</text>
43+ <text x="1040.0" y="251.5" text-anchor="middle" font-size="11.5" fill="#01579B">loaded alone · answers</text>
44+ <g filter="url(#shadow)"><rect x="910" y="390" width="260" height="130" rx="16" fill="url(#gYellow)" stroke="#F9A825" stroke-width="2"/></g>
45+ <text x="1040.0" y="435.0" text-anchor="middle" font-size="22">⚙️</text>
46+ <text x="1040.0" y="458.0" text-anchor="middle" font-size="16" font-weight="700" fill="#E65100">2 · mm unloads</text>
47+ <text x="1040.0" y="476.0" text-anchor="middle" font-size="11.5" fill="#E65100">the SPECIALIST's model</text>
48+ <text x="1040.0" y="493.0" text-anchor="middle" font-size="11.5" fill="#E65100">once it has answered</text>
49+ <g filter="url(#shadow)"><rect x="470" y="390" width="260" height="130" rx="16" fill="url(#gPurple)" stroke="#6A1B9A" stroke-width="3"/></g>
50+ <text x="600.0" y="435.0" text-anchor="middle" font-size="22">🔀</text>
51+ <text x="600.0" y="458.0" text-anchor="middle" font-size="16" font-weight="700" fill="#4A148C">Coordinator</text>
52+ <text x="600.0" y="476.0" text-anchor="middle" font-size="11.5" fill="#4A148C">MiniCPM5 reloads alone</text>
53+ <text x="600.0" y="493.0" text-anchor="middle" font-size="11.5" fill="#4A148C">reads the answer · replies</text>
54+ <rect x="482.0" y="193.0" width="76" height="24" rx="5" fill="#FFFFFF"/>
55+ <text x="520" y="209.5" text-anchor="middle" font-size="12.5" font-weight="700" fill="#263238">ask_agent</text>
56+ <rect x="825.0" y="193.0" width="70" height="24" rx="5" fill="#FFFFFF"/>
57+ <text x="860" y="209.5" text-anchor="middle" font-size="12.5" font-weight="700" fill="#263238">question</text>
58+ <rect x="1008.0" y="320.0" width="64" height="24" rx="5" fill="#FFFFFF"/>
59+ <text x="1040" y="336.5" text-anchor="middle" font-size="12.5" font-weight="700" fill="#263238">answer</text>
60+ <rect x="783.0" y="443.0" width="64" height="24" rx="5" fill="#FFFFFF"/>
61+ <text x="815" y="459.5" text-anchor="middle" font-size="12.5" font-weight="700" fill="#263238">answer</text>
62+ <rect x="275.0" y="443.0" width="50" height="24" rx="5" fill="#FFFFFF"/>
63+ <text x="300" y="459.5" text-anchor="middle" font-size="12.5" font-weight="700" fill="#263238">reply</text>
64+ <text x="600" y="568" text-anchor="middle" font-size="12" fill="#455A64" font-style="italic">Each unload: POST /api/generate {"model": …, "keep_alive": 0} on that agent's server, shown as a line on screen.</text>
65+ <text x="600" y="596" text-anchor="middle" font-size="12" fill="#455A64" font-style="italic">Measured on llmman (/api/ps every 0.5 s): MiniCPM5 → nothing → Mellum2 → nothing → MiniCPM5 · Rust 31 s · Go 79 s.</text>
66+</svg>
new file mode 100644
@@ -0,0 +1,66 @@
1+<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="640" viewBox="0 0 1200 640" font-family="DejaVu Sans, Verdana, sans-serif">
2+ <defs>
3+ <linearGradient id="gBg" x1="0" y1="0" x2="0" y2="1"><stop offset="0%" stop-color="#FFFFFF"/><stop offset="100%" stop-color="#EEF2FA"/></linearGradient>
4+ <linearGradient id="gBlue" x1="0" y1="0" x2="0" y2="1"><stop offset="0%" stop-color="#E3F2FD"/><stop offset="100%" stop-color="#BBDEFB"/></linearGradient>
5+ <linearGradient id="gBlue2" x1="0" y1="0" x2="0" y2="1"><stop offset="0%" stop-color="#E1F5FE"/><stop offset="100%" stop-color="#B3E5FC"/></linearGradient>
6+ <linearGradient id="gPurple" x1="0" y1="0" x2="0" y2="1"><stop offset="0%" stop-color="#F3E5F5"/><stop offset="100%" stop-color="#E1BEE7"/></linearGradient>
7+ <linearGradient id="gYellow" x1="0" y1="0" x2="0" y2="1"><stop offset="0%" stop-color="#FFFDE7"/><stop offset="100%" stop-color="#FFF9C4"/></linearGradient>
8+ <linearGradient id="gGrey" x1="0" y1="0" x2="0" y2="1"><stop offset="0%" stop-color="#ECEFF1"/><stop offset="100%" stop-color="#CFD8DC"/></linearGradient>
9+ <filter id="shadow" x="-30%" y="-30%" width="160%" height="160%"><feDropShadow dx="2" dy="3" stdDeviation="4" flood-color="#000000" flood-opacity="0.25"/></filter>
10+ <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto-start-reverse"><path d="M0,0 L10,5 L0,10 z" fill="#37474F"/></marker>
11+ </defs>
12+ <rect x="0" y="0" width="1200" height="640" fill="url(#gBg)"/>
13+ <text x="600.0" y="38" text-anchor="middle" font-size="25" font-weight="700" fill="#1A237E">coding-helpers-unload-option — the models take turns</text>
14+ <text x="600.0" y="61" text-anchor="middle" font-size="14" font-style="italic" fill="#546E7A">unloadModels: true on an mmConfigs entry: mm itself unloads a model at each hand-off, no hook</text>
15+ <path d="M200,205 L250,205" fill="none" stroke="#37474F" stroke-width="2.2" marker-end="url(#arrow)"/>
16+ <path d="M470,205 L570,205" fill="none" stroke="#37474F" stroke-width="2.2" marker-end="url(#arrow)"/>
17+ <path d="M810,205 L910,205" fill="none" stroke="#37474F" stroke-width="2.2" marker-end="url(#arrow)"/>
18+ <path d="M1040,275 L1040,390" fill="none" stroke="#37474F" stroke-width="2.2" marker-end="url(#arrow)"/>
19+ <path d="M910,455 L730,455" fill="none" stroke="#37474F" stroke-width="2.2" marker-end="url(#arrow)"/>
20+ <path d="M470,455 L115,455 L115,265" fill="none" stroke="#37474F" stroke-width="2.2" marker-end="url(#arrow)"/>
21+ <g filter="url(#shadow)"><rect x="30" y="145" width="170" height="120" rx="16" fill="url(#gBlue)" stroke="#1565C0" stroke-width="2"/></g>
22+ <text x="115.0" y="193.5" text-anchor="middle" font-size="22">🧑</text>
23+ <text x="115.0" y="216.5" text-anchor="middle" font-size="16" font-weight="700" fill="#0D47A1">You</text>
24+ <text x="115.0" y="234.5" text-anchor="middle" font-size="11.5" fill="#0D47A1">a Go or Rust question</text>
25+ <g filter="url(#shadow)"><rect x="250" y="135" width="220" height="140" rx="16" fill="url(#gPurple)" stroke="#6A1B9A" stroke-width="3"/></g>
26+ <text x="360.0" y="176.5" text-anchor="middle" font-size="22">🔀</text>
27+ <text x="360.0" y="199.5" text-anchor="middle" font-size="16" font-weight="700" fill="#4A148C">Coordinator</text>
28+ <text x="360.0" y="217.5" text-anchor="middle" font-size="11.5" fill="#4A148C">MiniCPM5 2B · 1.6 GB</text>
29+ <text x="360.0" y="234.5" text-anchor="middle" font-size="11.5" fill="#4A148C">agent.yaml</text>
30+ <text x="360.0" y="251.5" text-anchor="middle" font-size="11.5" fill="#4A148C">ask_agent go / rust</text>
31+ <rect x="422.8" y="123" width="57.2" height="24" rx="12" fill="#6A1B9A"/>
32+ <text x="451.40000000000003" y="139" text-anchor="middle" font-size="11.5" font-weight="700" fill="#FFFFFF">★ asks</text>
33+ <g filter="url(#shadow)"><rect x="570" y="140" width="240" height="130" rx="16" fill="url(#gYellow)" stroke="#F9A825" stroke-width="2"/></g>
34+ <text x="690.0" y="185.0" text-anchor="middle" font-size="22">⚙️</text>
35+ <text x="690.0" y="208.0" text-anchor="middle" font-size="16" font-weight="700" fill="#E65100">1 · mm unloads</text>
36+ <text x="690.0" y="226.0" text-anchor="middle" font-size="11.5" fill="#E65100">the COORDINATOR's model</text>
37+ <text x="690.0" y="243.0" text-anchor="middle" font-size="11.5" fill="#E65100">before the specialist starts</text>
38+ <g filter="url(#shadow)"><rect x="910" y="135" width="260" height="140" rx="16" fill="url(#gBlue2)" stroke="#0277BD" stroke-width="2"/></g>
39+ <text x="1040.0" y="176.5" text-anchor="middle" font-size="22">🤖</text>
40+ <text x="1040.0" y="199.5" text-anchor="middle" font-size="16" font-weight="700" fill="#01579B">Specialist go / rust</text>
41+ <text x="1040.0" y="217.5" text-anchor="middle" font-size="11.5" fill="#01579B">Mellum2 12B · 8.1 GB</text>
42+ <text x="1040.0" y="234.5" text-anchor="middle" font-size="11.5" fill="#01579B">agent.go.yaml / agent.rust.yaml</text>
43+ <text x="1040.0" y="251.5" text-anchor="middle" font-size="11.5" fill="#01579B">loaded alone · answers</text>
44+ <g filter="url(#shadow)"><rect x="910" y="390" width="260" height="130" rx="16" fill="url(#gYellow)" stroke="#F9A825" stroke-width="2"/></g>
45+ <text x="1040.0" y="435.0" text-anchor="middle" font-size="22">⚙️</text>
46+ <text x="1040.0" y="458.0" text-anchor="middle" font-size="16" font-weight="700" fill="#E65100">2 · mm unloads</text>
47+ <text x="1040.0" y="476.0" text-anchor="middle" font-size="11.5" fill="#E65100">the SPECIALIST's model</text>
48+ <text x="1040.0" y="493.0" text-anchor="middle" font-size="11.5" fill="#E65100">once it has answered</text>
49+ <g filter="url(#shadow)"><rect x="470" y="390" width="260" height="130" rx="16" fill="url(#gPurple)" stroke="#6A1B9A" stroke-width="3"/></g>
50+ <text x="600.0" y="435.0" text-anchor="middle" font-size="22">🔀</text>
51+ <text x="600.0" y="458.0" text-anchor="middle" font-size="16" font-weight="700" fill="#4A148C">Coordinator</text>
52+ <text x="600.0" y="476.0" text-anchor="middle" font-size="11.5" fill="#4A148C">MiniCPM5 reloads alone</text>
53+ <text x="600.0" y="493.0" text-anchor="middle" font-size="11.5" fill="#4A148C">reads the answer · replies</text>
54+ <rect x="482.0" y="193.0" width="76" height="24" rx="5" fill="#FFFFFF"/>
55+ <text x="520" y="209.5" text-anchor="middle" font-size="12.5" font-weight="700" fill="#263238">ask_agent</text>
56+ <rect x="825.0" y="193.0" width="70" height="24" rx="5" fill="#FFFFFF"/>
57+ <text x="860" y="209.5" text-anchor="middle" font-size="12.5" font-weight="700" fill="#263238">question</text>
58+ <rect x="1008.0" y="320.0" width="64" height="24" rx="5" fill="#FFFFFF"/>
59+ <text x="1040" y="336.5" text-anchor="middle" font-size="12.5" font-weight="700" fill="#263238">answer</text>
60+ <rect x="783.0" y="443.0" width="64" height="24" rx="5" fill="#FFFFFF"/>
61+ <text x="815" y="459.5" text-anchor="middle" font-size="12.5" font-weight="700" fill="#263238">answer</text>
62+ <rect x="275.0" y="443.0" width="50" height="24" rx="5" fill="#FFFFFF"/>
63+ <text x="300" y="459.5" text-anchor="middle" font-size="12.5" font-weight="700" fill="#263238">reply</text>
64+ <text x="600" y="568" text-anchor="middle" font-size="12" fill="#455A64" font-style="italic">Each unload: POST /api/generate {"model": …, "keep_alive": 0} on that agent's server, shown as a line on screen.</text>
65+ <text x="600" y="596" text-anchor="middle" font-size="12" fill="#455A64" font-style="italic">Measured on llmman (/api/ps every 0.5 s): MiniCPM5 → nothing → Mellum2 → nothing → MiniCPM5 · Rust 31 s · Go 79 s.</text>
66+</svg>
added testing/README.en.md +12 -0
new file mode 100644
@@ -0,0 +1,12 @@
1+# Testing models with mini-me
2+
3+*[Version française](README.fr.md)*
4+
5+Test sets that measure what a model can do, run by mini-me itself: a coordinator agent follows a skill, gives small tasks to a `candidate` agent whose model is under test, checks each result with a command, and writes a Markdown report in its `reports/` folder.
6+
7+| Set | What it tests | The candidate's tools |
8+|---|---|---|
9+| [`model-check/`](model-check/README.en.md) | the model as a working agent: tool calls, exact instructions, Go code that must run, tasks in several steps | mm's built-in tools (bash, the file tools) |
10+| [`function-calling/`](function-calling/README.en.md) | function calling alone: the right tool, the right arguments and types, chained calls, two calls at once, no call when none fits | only the tools declared in its agent file (`tools:`) |
11+
12+In both, the model to test is the `model:` line of `agent.candidate.yaml`. Run a set from its folder with `../common/run.sh`: it runs the coordinator, and writes the report from the recorded results if the coordinator stops early. `common/bench-lib.sh` is what the two benches share: the loop over the tests (`./bench.sh next`), recording results, the table and the report.
new file mode 100644
@@ -0,0 +1,12 @@
1+# Testing models with mini-me
2+
3+*[Version française](README.fr.md)*
4+
5+Test sets that measure what a model can do, run by mini-me itself: a coordinator agent follows a skill, gives small tasks to a `candidate` agent whose model is under test, checks each result with a command, and writes a Markdown report in its `reports/` folder.
6+
7+| Set | What it tests | The candidate's tools |
8+|---|---|---|
9+| [`model-check/`](model-check/README.en.md) | the model as a working agent: tool calls, exact instructions, Go code that must run, tasks in several steps | mm's built-in tools (bash, the file tools) |
10+| [`function-calling/`](function-calling/README.en.md) | function calling alone: the right tool, the right arguments and types, chained calls, two calls at once, no call when none fits | only the tools declared in its agent file (`tools:`) |
11+
12+In both, the model to test is the `model:` line of `agent.candidate.yaml`. Run a set from its folder with `../common/run.sh`: it runs the coordinator, and writes the report from the recorded results if the coordinator stops early. `common/bench-lib.sh` is what the two benches share: the loop over the tests (`./bench.sh next`), recording results, the table and the report.
added testing/README.fr.md +12 -0
new file mode 100644
@@ -0,0 +1,12 @@
1+# Tester des modèles avec mini-me
2+
3+*[English version](README.en.md)*
4+
5+Des jeux de tests qui mesurent ce dont un modèle est capable, lancés par mini-me lui-même : un agent coordinateur suit une skill, donne de petites tâches à un agent `candidate` dont on teste le modèle, vérifie chaque résultat par une commande et écrit un rapport Markdown dans son dossier `reports/`.
6+
7+| Jeu | Ce qu'il teste | Les outils du candidat |
8+|---|---|---|
9+| [`model-check/`](model-check/README.fr.md) | le modèle comme agent de travail : appels d'outils, consignes exactes, code Go qui doit tourner, tâches en plusieurs étapes | les outils intégrés de mm (bash, les outils de fichiers) |
10+| [`function-calling/`](function-calling/README.fr.md) | l'appel d'outils seul : le bon outil, les bons arguments et types, des appels enchaînés, deux appels à la fois, aucun appel quand aucun outil ne convient | seulement les outils déclarés dans son fichier d'agent (`tools:`) |
11+
12+Dans les deux, le modèle testé est la ligne `model:` de `agent.candidate.yaml`. Lancez un jeu depuis son dossier avec `../common/run.sh` : il lance le coordinateur, et écrit le rapport à partir des résultats enregistrés si le coordinateur s'arrête avant. `common/bench-lib.sh` est ce que les deux bancs partagent : la boucle sur les tests (`./bench.sh next`), l'enregistrement des résultats, le tableau et le rapport.
new file mode 100644
@@ -0,0 +1,12 @@
1+# Tester des modèles avec mini-me
2+
3+*[English version](README.en.md)*
4+
5+Des jeux de tests qui mesurent ce dont un modèle est capable, lancés par mini-me lui-même : un agent coordinateur suit une skill, donne de petites tâches à un agent `candidate` dont on teste le modèle, vérifie chaque résultat par une commande et écrit un rapport Markdown dans son dossier `reports/`.
6+
7+| Jeu | Ce qu'il teste | Les outils du candidat |
8+|---|---|---|
9+| [`model-check/`](model-check/README.fr.md) | le modèle comme agent de travail : appels d'outils, consignes exactes, code Go qui doit tourner, tâches en plusieurs étapes | les outils intégrés de mm (bash, les outils de fichiers) |
10+| [`function-calling/`](function-calling/README.fr.md) | l'appel d'outils seul : le bon outil, les bons arguments et types, des appels enchaînés, deux appels à la fois, aucun appel quand aucun outil ne convient | seulement les outils déclarés dans son fichier d'agent (`tools:`) |
11+
12+Dans les deux, le modèle testé est la ligne `model:` de `agent.candidate.yaml`. Lancez un jeu depuis son dossier avec `../common/run.sh` : il lance le coordinateur, et écrit le rapport à partir des résultats enregistrés si le coordinateur s'arrête avant. `common/bench-lib.sh` est ce que les deux bancs partagent : la boucle sur les tests (`./bench.sh next`), l'enregistrement des résultats, le tableau et le rapport.
added testing/common/bench-lib.sh +174 -0
new file mode 100644
@@ -0,0 +1,174 @@
1+# The part of a bench that every test set shares: the loop over the tests,
2+# recording results, the results table and the Markdown report. Sourced by
3+# each set's bench.sh, which sets these first:
4+#
5+# ROOT the set's folder (testing/<set>/), where agent.yaml lives
6+# SKILL the skill's folder, holding tests.tsv
7+# TITLE the report's title, e.g. "Model check"
8+#
9+# and defines two functions:
10+#
11+# reset_work puts work/ back in its starting state
12+# check_test <id> checks a "command" test and calls bench_record
13+#
14+# tests.tsv has one line per test, tab-separated:
15+# id · capability · summary · mode (command|judge) · prompt · note for the coordinator
16+#
17+# The SCRIPT drives the loop (bench_next), not the coordinator's memory: each
18+# `next` checks the test just run before handing out the following one, so
19+# no check can be skipped. The results go to reports/.results.tsv and the
20+# coordinator's notes to reports/.notes.md; the report is built from those.
21+
22+RESULTS="$ROOT/reports/.results.tsv"
23+NOTES="$ROOT/reports/.notes.md"
24+RUN="$ROOT/reports/.run"
25+CURRENT="$ROOT/reports/.current"
26+TAB="$(printf '\t')"
27+
28+model_of() { # agent file
29+ sed -n 's/^model: *//p' "$1" | head -1
30+}
31+
32+field() { # test column (1-based)
33+ awk -F '\t' -v t="$1" -v c="$2" '$1 == t { print $c }' "$SKILL/tests.tsv"
34+}
35+
36+# start: a new run — empty results, no notes, the report's path decided now.
37+bench_start() {
38+ mkdir -p "$ROOT/reports"
39+ : >"$RESULTS"
40+ rm -f "$NOTES" "$CURRENT"
41+ local model short path
42+ model="$(model_of "$ROOT/agent.candidate.yaml")"
43+ short="$(printf '%s' "$model" | sed 's#.*/##; s/:/-/')"
44+ path="reports/$(date +%Y-%m-%d-%H%M)-$short.md"
45+ printf '%s\n%s\n' "$path" "$(date '+%Y-%m-%d %H:%M')" >"$RUN"
46+ echo "tested model: $model"
47+ echo "report: $path (written by: bench.sh report)"
48+ echo "Now run: bench.sh next"
49+}
50+
51+bench_record() { # test verdict detail
52+ case "$2" in PASS | FAIL) ;; *) echo "record <test> PASS|FAIL \"why\"" >&2; exit 2 ;; esac
53+ # Seen: a coordinator recorded "test", copying the placeholder. Refuse an
54+ # unknown id and name the test in progress, so the next try is right.
55+ if [ -z "$(field "$1" 1)" ]; then
56+ echo "unknown test \"$1\". The test in progress is $(cat "$CURRENT" 2>/dev/null || echo none): bench.sh record $(cat "$CURRENT" 2>/dev/null || echo '<test>') $2 \"$3\"" >&2
57+ exit 2
58+ fi
59+ # A test checked again replaces its earlier result: one line per test.
60+ grep -v "^$1$TAB" "$RESULTS" >"$RESULTS.tmp" || true
61+ mv "$RESULTS.tmp" "$RESULTS"
62+ printf '%s\t%s\t%s\n' "$1" "$2" "$3" >>"$RESULTS"
63+ echo "$2 $1 — $3"
64+}
65+
66+recorded() { # test
67+ grep -q "^$1$TAB" "$RESULTS"
68+}
69+
70+# next: closes the current test (checks it, or asks for the judgement), then
71+# hands out the following one, or says the run is over.
72+bench_next() {
73+ [ -f "$RUN" ] || { echo "no run started: run bench.sh start first" >&2; exit 2; }
74+ local cur="" following note
75+ [ -f "$CURRENT" ] && cur="$(cat "$CURRENT")"
76+ if [ -n "$cur" ] && ! recorded "$cur"; then
77+ if [ "$(field "$cur" 4)" = "judge" ]; then
78+ echo "$cur is judged by you. Compare the candidate's answer with this rule:"
79+ echo " $(field "$cur" 6)"
80+ echo "Then run: bench.sh record $cur PASS \"<why>\" (or FAIL), and then bench.sh next"
81+ return
82+ fi
83+ check_test "$cur"
84+ fi
85+ if [ -z "$cur" ] && [ -s "$RESULTS" ]; then
86+ following="" # the run is over: next keeps saying so
87+ elif [ -z "$cur" ]; then
88+ following="$(sed -n 1p "$SKILL/tests.tsv" | cut -f1)"
89+ else
90+ following="$(awk -F '\t' -v t="$cur" 'found { print $1; exit } $1 == t { found = 1 }' "$SKILL/tests.tsv")"
91+ fi
92+ if [ -z "$following" ]; then
93+ rm -f "$CURRENT"
94+ echo
95+ echo "ALL TESTS DONE — $(bench_score). Now write your notes in reports/.notes.md, then run: bench.sh report"
96+ return
97+ fi
98+ reset_work
99+ echo "$following" >"$CURRENT"
100+ echo
101+ echo "NEXT: $following ($(field "$following" 2))"
102+ echo "Call ask_agent with agent: candidate, with_context: false, and this prompt, copied exactly:"
103+ echo
104+ echo "$(field "$following" 5)"
105+ echo
106+ note="$(field "$following" 6)"
107+ if [ -n "$note" ] && [ "$(field "$following" 4)" = "command" ]; then echo "Afterwards: $note"; fi
108+ echo "When the candidate has answered, run: bench.sh next"
109+}
110+
111+bench_table() {
112+ echo "| Test | Capability | Result | Detail |"
113+ echo "|---|---|---|---|"
114+ local t v d
115+ sort "$RESULTS" | while IFS="$TAB" read -r t v d; do
116+ echo "| $t | $(field "$t" 2) | $v | ${d//|/\\|} |"
117+ done
118+}
119+
120+bench_score() {
121+ echo "$(grep -c "${TAB}PASS${TAB}" "$RESULTS" || true) of $(grep -c . "$SKILL/tests.tsv") passed"
122+}
123+
124+# report: writes the Markdown report at the path decided by start, and
125+# prints that path. Tests never recorded are listed as NOT RUN.
126+bench_report() {
127+ [ -f "$RUN" ] || { echo "no run started: run bench.sh start first" >&2; exit 2; }
128+ local path date t c s
129+ path="$(sed -n 1p "$RUN")"
130+ date="$(sed -n 2p "$RUN")"
131+ for t in $(cut -f1 "$SKILL/tests.tsv"); do
132+ recorded "$t" || printf '%s\tNOT RUN\t%s\n' "$t" "no result recorded" >>"$RESULTS"
133+ done
134+ {
135+ echo "# $TITLE — $(model_of "$ROOT/agent.candidate.yaml")"
136+ echo
137+ echo "- **Date**: $date"
138+ echo "- **Tested model**: \`$(model_of "$ROOT/agent.candidate.yaml")\` (agent.candidate.yaml)"
139+ echo "- **Coordinator**: \`$(model_of "$ROOT/agent.yaml")\` (agent.yaml)"
140+ echo "- **Score**: $(bench_score)"
141+ echo
142+ echo "## Results"
143+ echo
144+ bench_table
145+ echo
146+ echo "## What the coordinator saw"
147+ echo
148+ if [ -s "$NOTES" ]; then cat "$NOTES"; else echo "_No notes: the coordinator wrote no reports/.notes.md._"; fi
149+ echo
150+ echo "## The tests"
151+ echo
152+ echo "| Test | Capability | What was asked |"
153+ echo "|---|---|---|"
154+ while IFS="$TAB" read -r t c s _; do echo "| $t | $c | $s |"; done <"$SKILL/tests.tsv"
155+ echo
156+ echo "The exact prompts are in \`skills/$(basename "$SKILL")/tests.tsv\`."
157+ } >"$ROOT/$path"
158+ rm -f "$CURRENT"
159+ echo "report written: $path — $(bench_score)"
160+}
161+
162+# The commands every bench.sh accepts.
163+bench_main() {
164+ case "${1:-}" in
165+ start) bench_start ;;
166+ next) bench_next ;;
167+ record) bench_record "${2:?which test?}" "${3:-}" "${4:-}" ;;
168+ table) bench_table; echo; bench_score ;;
169+ report) bench_report ;;
170+ *)
171+ echo "bench.sh start | next | record <test> PASS|FAIL \"why\" | table | report" >&2
172+ exit 2 ;;
173+ esac
174+}
new file mode 100644
@@ -0,0 +1,174 @@
1+# The part of a bench that every test set shares: the loop over the tests,
2+# recording results, the results table and the Markdown report. Sourced by
3+# each set's bench.sh, which sets these first:
4+#
5+# ROOT the set's folder (testing/<set>/), where agent.yaml lives
6+# SKILL the skill's folder, holding tests.tsv
7+# TITLE the report's title, e.g. "Model check"
8+#
9+# and defines two functions:
10+#
11+# reset_work puts work/ back in its starting state
12+# check_test <id> checks a "command" test and calls bench_record
13+#
14+# tests.tsv has one line per test, tab-separated:
15+# id · capability · summary · mode (command|judge) · prompt · note for the coordinator
16+#
17+# The SCRIPT drives the loop (bench_next), not the coordinator's memory: each
18+# `next` checks the test just run before handing out the following one, so
19+# no check can be skipped. The results go to reports/.results.tsv and the
20+# coordinator's notes to reports/.notes.md; the report is built from those.
21+
22+RESULTS="$ROOT/reports/.results.tsv"
23+NOTES="$ROOT/reports/.notes.md"
24+RUN="$ROOT/reports/.run"
25+CURRENT="$ROOT/reports/.current"
26+TAB="$(printf '\t')"
27+
28+model_of() { # agent file
29+ sed -n 's/^model: *//p' "$1" | head -1
30+}
31+
32+field() { # test column (1-based)
33+ awk -F '\t' -v t="$1" -v c="$2" '$1 == t { print $c }' "$SKILL/tests.tsv"
34+}
35+
36+# start: a new run — empty results, no notes, the report's path decided now.
37+bench_start() {
38+ mkdir -p "$ROOT/reports"
39+ : >"$RESULTS"
40+ rm -f "$NOTES" "$CURRENT"
41+ local model short path
42+ model="$(model_of "$ROOT/agent.candidate.yaml")"
43+ short="$(printf '%s' "$model" | sed 's#.*/##; s/:/-/')"
44+ path="reports/$(date +%Y-%m-%d-%H%M)-$short.md"
45+ printf '%s\n%s\n' "$path" "$(date '+%Y-%m-%d %H:%M')" >"$RUN"
46+ echo "tested model: $model"
47+ echo "report: $path (written by: bench.sh report)"
48+ echo "Now run: bench.sh next"
49+}
50+
51+bench_record() { # test verdict detail
52+ case "$2" in PASS | FAIL) ;; *) echo "record <test> PASS|FAIL \"why\"" >&2; exit 2 ;; esac
53+ # Seen: a coordinator recorded "test", copying the placeholder. Refuse an
54+ # unknown id and name the test in progress, so the next try is right.
55+ if [ -z "$(field "$1" 1)" ]; then
56+ echo "unknown test \"$1\". The test in progress is $(cat "$CURRENT" 2>/dev/null || echo none): bench.sh record $(cat "$CURRENT" 2>/dev/null || echo '<test>') $2 \"$3\"" >&2
57+ exit 2
58+ fi
59+ # A test checked again replaces its earlier result: one line per test.
60+ grep -v "^$1$TAB" "$RESULTS" >"$RESULTS.tmp" || true
61+ mv "$RESULTS.tmp" "$RESULTS"
62+ printf '%s\t%s\t%s\n' "$1" "$2" "$3" >>"$RESULTS"
63+ echo "$2 $1 — $3"
64+}
65+
66+recorded() { # test
67+ grep -q "^$1$TAB" "$RESULTS"
68+}
69+
70+# next: closes the current test (checks it, or asks for the judgement), then
71+# hands out the following one, or says the run is over.
72+bench_next() {
73+ [ -f "$RUN" ] || { echo "no run started: run bench.sh start first" >&2; exit 2; }
74+ local cur="" following note
75+ [ -f "$CURRENT" ] && cur="$(cat "$CURRENT")"
76+ if [ -n "$cur" ] && ! recorded "$cur"; then
77+ if [ "$(field "$cur" 4)" = "judge" ]; then
78+ echo "$cur is judged by you. Compare the candidate's answer with this rule:"
79+ echo " $(field "$cur" 6)"
80+ echo "Then run: bench.sh record $cur PASS \"<why>\" (or FAIL), and then bench.sh next"
81+ return
82+ fi
83+ check_test "$cur"
84+ fi
85+ if [ -z "$cur" ] && [ -s "$RESULTS" ]; then
86+ following="" # the run is over: next keeps saying so
87+ elif [ -z "$cur" ]; then
88+ following="$(sed -n 1p "$SKILL/tests.tsv" | cut -f1)"
89+ else
90+ following="$(awk -F '\t' -v t="$cur" 'found { print $1; exit } $1 == t { found = 1 }' "$SKILL/tests.tsv")"
91+ fi
92+ if [ -z "$following" ]; then
93+ rm -f "$CURRENT"
94+ echo
95+ echo "ALL TESTS DONE — $(bench_score). Now write your notes in reports/.notes.md, then run: bench.sh report"
96+ return
97+ fi
98+ reset_work
99+ echo "$following" >"$CURRENT"
100+ echo
101+ echo "NEXT: $following ($(field "$following" 2))"
102+ echo "Call ask_agent with agent: candidate, with_context: false, and this prompt, copied exactly:"
103+ echo
104+ echo "$(field "$following" 5)"
105+ echo
106+ note="$(field "$following" 6)"
107+ if [ -n "$note" ] && [ "$(field "$following" 4)" = "command" ]; then echo "Afterwards: $note"; fi
108+ echo "When the candidate has answered, run: bench.sh next"
109+}
110+
111+bench_table() {
112+ echo "| Test | Capability | Result | Detail |"
113+ echo "|---|---|---|---|"
114+ local t v d
115+ sort "$RESULTS" | while IFS="$TAB" read -r t v d; do
116+ echo "| $t | $(field "$t" 2) | $v | ${d//|/\\|} |"
117+ done
118+}
119+
120+bench_score() {
121+ echo "$(grep -c "${TAB}PASS${TAB}" "$RESULTS" || true) of $(grep -c . "$SKILL/tests.tsv") passed"
122+}
123+
124+# report: writes the Markdown report at the path decided by start, and
125+# prints that path. Tests never recorded are listed as NOT RUN.
126+bench_report() {
127+ [ -f "$RUN" ] || { echo "no run started: run bench.sh start first" >&2; exit 2; }
128+ local path date t c s
129+ path="$(sed -n 1p "$RUN")"
130+ date="$(sed -n 2p "$RUN")"
131+ for t in $(cut -f1 "$SKILL/tests.tsv"); do
132+ recorded "$t" || printf '%s\tNOT RUN\t%s\n' "$t" "no result recorded" >>"$RESULTS"
133+ done
134+ {
135+ echo "# $TITLE — $(model_of "$ROOT/agent.candidate.yaml")"
136+ echo
137+ echo "- **Date**: $date"
138+ echo "- **Tested model**: \`$(model_of "$ROOT/agent.candidate.yaml")\` (agent.candidate.yaml)"
139+ echo "- **Coordinator**: \`$(model_of "$ROOT/agent.yaml")\` (agent.yaml)"
140+ echo "- **Score**: $(bench_score)"
141+ echo
142+ echo "## Results"
143+ echo
144+ bench_table
145+ echo
146+ echo "## What the coordinator saw"
147+ echo
148+ if [ -s "$NOTES" ]; then cat "$NOTES"; else echo "_No notes: the coordinator wrote no reports/.notes.md._"; fi
149+ echo
150+ echo "## The tests"
151+ echo
152+ echo "| Test | Capability | What was asked |"
153+ echo "|---|---|---|"
154+ while IFS="$TAB" read -r t c s _; do echo "| $t | $c | $s |"; done <"$SKILL/tests.tsv"
155+ echo
156+ echo "The exact prompts are in \`skills/$(basename "$SKILL")/tests.tsv\`."
157+ } >"$ROOT/$path"
158+ rm -f "$CURRENT"
159+ echo "report written: $path — $(bench_score)"
160+}
161+
162+# The commands every bench.sh accepts.
163+bench_main() {
164+ case "${1:-}" in
165+ start) bench_start ;;
166+ next) bench_next ;;
167+ record) bench_record "${2:?which test?}" "${3:-}" "${4:-}" ;;
168+ table) bench_table; echo; bench_score ;;
169+ report) bench_report ;;
170+ *)
171+ echo "bench.sh start | next | record <test> PASS|FAIL \"why\" | table | report" >&2
172+ exit 2 ;;
173+ esac
174+}
added testing/common/run.sh +28 -0
new file mode 100755
@@ -0,0 +1,28 @@
1+#!/usr/bin/env bash
2+# Runs one test set from its folder and makes sure a report is written.
3+#
4+# cd testing/function-calling && ../common/run.sh
5+# cd testing/model-check && ../common/run.sh
6+#
7+# The coordinator writes the report itself at the end of its skill. If its
8+# run stops before that — a server that breaks the stream, a candidate that
9+# runs away — this script writes the report from what was recorded: the
10+# tests never checked show as NOT RUN, and the notes say there are none.
11+# Extra arguments go to mm (e.g. -tui is not useful here: this is for -p).
12+set -u
13+
14+MM="${MM_BIN:-mm}"
15+SKILL="$(ls -d skills/*/ 2>/dev/null | head -1)"
16+[ -f agent.yaml ] && [ -n "$SKILL" ] || { echo "run it from a test set folder (testing/<set>/)" >&2; exit 2; }
17+BENCH="${SKILL}bench.sh"
18+
19+before="$(ls reports/*.md 2>/dev/null | sort)"
20+"$MM" -config agent.yaml "$@" -p "Test the candidate."
21+status=$?
22+after="$(ls reports/*.md 2>/dev/null | sort)"
23+
24+if [ "$before" = "$after" ]; then
25+ echo
26+ echo "The coordinator stopped before writing the report (mm exit status $status): writing it from the recorded results."
27+ "$BENCH" report
28+fi
new file mode 100755
@@ -0,0 +1,28 @@
1+#!/usr/bin/env bash
2+# Runs one test set from its folder and makes sure a report is written.
3+#
4+# cd testing/function-calling && ../common/run.sh
5+# cd testing/model-check && ../common/run.sh
6+#
7+# The coordinator writes the report itself at the end of its skill. If its
8+# run stops before that — a server that breaks the stream, a candidate that
9+# runs away — this script writes the report from what was recorded: the
10+# tests never checked show as NOT RUN, and the notes say there are none.
11+# Extra arguments go to mm (e.g. -tui is not useful here: this is for -p).
12+set -u
13+
14+MM="${MM_BIN:-mm}"
15+SKILL="$(ls -d skills/*/ 2>/dev/null | head -1)"
16+[ -f agent.yaml ] && [ -n "$SKILL" ] || { echo "run it from a test set folder (testing/<set>/)" >&2; exit 2; }
17+BENCH="${SKILL}bench.sh"
18+
19+before="$(ls reports/*.md 2>/dev/null | sort)"
20+"$MM" -config agent.yaml "$@" -p "Test the candidate."
21+status=$?
22+after="$(ls reports/*.md 2>/dev/null | sort)"
23+
24+if [ "$before" = "$after" ]; then
25+ echo
26+ echo "The coordinator stopped before writing the report (mm exit status $status): writing it from the recorded results."
27+ "$BENCH" report
28+fi
added testing/function-calling/.gitignore +6 -0
new file mode 100644
@@ -0,0 +1,6 @@
1+# The scratch folder (bench.sh reset recreates it; work/calls.jsonl logs
2+# every tool call) and the files of the run in progress. Reports are kept.
3+work/
4+reports/.results.tsv
5+reports/.notes.md
6+reports/.run
new file mode 100644
@@ -0,0 +1,6 @@
1+# The scratch folder (bench.sh reset recreates it; work/calls.jsonl logs
2+# every tool call) and the files of the run in progress. Reports are kept.
3+work/
4+reports/.results.tsv
5+reports/.notes.md
6+reports/.run
added testing/function-calling/README.en.md +88 -0
new file mode 100644
@@ -0,0 +1,88 @@
1+# Function-calling tester
2+
3+*[Version française](README.fr.md)*
4+
5+The second test set of [`testing/`](../README.en.md): function calling alone. A coordinator follows the `function-calling-check` skill: it sends eight requests, one at a time, to the `candidate` agent with `ask_agent`. The candidate's only tools are the ones declared in its agent file (`tools:`); every call it makes is logged and checked by a script. At the end, a Markdown report in `reports/`.
6+
7+| Agent | File | Model | Role |
8+|---|---|---|---|
9+| 🧪 coordinator | `agent.yaml` | Gemma 4 12B (`hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M`) | runs the loop, writes notes and a verdict |
10+| 🎯 candidate | `agent.candidate.yaml` | Luciole 1B Instruct 1.1 (`hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M`) | the model under test: no built-in tool, six declared tools |
11+
12+## Run
13+
14+```bash
15+llmman pull hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M
16+llmman pull hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M
17+llmman serve
18+
19+cd testing/function-calling
20+../common/run.sh # mm -p "Test the candidate.", then the report even if the run stops early
21+# or, in the window:
22+mm -tui # then type: $function-calling-check
23+```
24+
25+Run it from `testing/function-calling/`. `python3` must be on the PATH (the tools and the checks). Both agents run commands without asking (`yolo: true`): the coordinator only runs `./bench.sh`, the candidate only its declared tools. Both models stay loaded during a run (`unloadModels: false`): reloading the coordinator after each request made it re-read the whole conversation, and the server then broke its stream.
26+
27+## The candidate's tools
28+
29+Declared in `agent.candidate.yaml`, each one runs `tools/call.sh`, which logs the call (tool name and arguments, as mm passed them) in `work/calls.jsonl` and answers with a fixed, made-up result — nothing is looked up or booked.
30+
31+| Tool | Parameters | Made-up answer |
32+|---|---|---|
33+| `get_weather` | `city` string · `unit` enum celsius/fahrenheit, optional, default celsius | `Paris: 18°C, cloudy` |
34+| `convert_currency` | `amount` number · `from`, `to` enum EUR/USD/JPY/GBP | `250 EUR = 270.00 USD` |
35+| `find_user` | `email` string | `user_id: 4217, name: Alice Martin` for alice@example.com |
36+| `list_orders` | `user_id` integer · `status` enum open/shipped/all, optional | two open orders for 4217 |
37+| `create_reminder` | `text`, `when` string · `urgent` boolean, optional | `reminder created: …` |
38+| `book_table` | `restaurant`, `date` string · `people` integer | `table booked …` |
39+
40+To test other tools, change them here and the checks in `skills/function-calling-check/check.py`, which read the calls by tool and argument name.
41+
42+## The eight tests
43+
44+| Test | Capability | Request | PASS when |
45+|---|---|---|---|
46+| F1 | one call, a string argument | the weather in Paris | one `get_weather(city=Paris)`, default unit kept |
47+| F2 | an enum argument | the weather in Tokyo in fahrenheit | one `get_weather(city=Tokyo, unit=fahrenheit)` |
48+| F3 | a number and two enums | convert 250 euros to US dollars | one `convert_currency(250, EUR, USD)` |
49+| F4 | a boolean argument | remind me to call Alice tomorrow at 9:00, urgent | one `create_reminder` with the text, a time, `urgent=true` |
50+| F5 | chained calls | find alice@example.com, then her open orders | `find_user`, then `list_orders(user_id=4217, status=open)` with the id it returned |
51+| F6 | two calls in one request | the weather in Rome and in Oslo | two `get_weather`, Rome and Oslo |
52+| F7 | no call when no tool fits | 12 times 12 | no call |
53+| F8 | no invented arguments | book a table (nothing more said) | no call |
54+
55+The exact prompts are in `skills/function-calling-check/tests.tsv`. One try per test.
56+
57+## How a run goes
58+
59+The bench script drives the loop: the coordinator repeats `./bench.sh next`, then `ask_agent` with the prompt `next` printed. Each `next` checks the calls of the test just run, empties the call log, and hands out the following test, so no check can be skipped. When all are done, the coordinator writes its notes and verdict in `reports/.notes.md` and runs `./bench.sh report`, which writes `reports/<date>-<model>.md`: model, score, results table, notes, list of tests. If the coordinator stops early, `../common/run.sh` writes the report from what was recorded (unchecked tests show as NOT RUN).
60+
61+## Measured results
62+
63+2026-10-26, on the author's llmman, Gemma 4 12B checking Luciole 1B: the full run took **13 min 04 s** and scored **5 of 8** — [the report](reports/2026-10-26-0901-Luciole-1B-Instruct-1.1-GGUF-Q4_K_M.md), written by the coordinator itself.
64+
65+| PASS | FAIL |
66+|---|---|
67+| F1, F2, F4 (string, enum, boolean arguments), F6 (two calls), F8 (no booking with invented values) | F3: runaway tool call · F5: `max_tokens` hit while writing a call · F7: a tool called for 12 × 12, then a loop |
68+
69+Luciole is **not stable from one run to the next** (temperature 0.7, the GGUF's own setting). Over the runs made that day: F3 passed twice and failed twice — once with 42 `convert_currency` calls with made-up amounts in a single answer; F2 failed once with two calls (celsius, then fahrenheit); F5 invented `user_id` values and damaged the e-mail address once. One run is a sample, not a verdict.
70+
71+What the first runs taught the bench (all fixed before the run above): with the coordinator reloaded after each request, its stream broke on the long conversation; with a first procedure that let the coordinator call the checks itself, it skipped all of them once, and twice rewrote the script's path with underscores (hence `./bench.sh` and the `next` loop); it recorded a result under the literal id `test` (now refused).
72+
73+## Files
74+
75+```
76+testing/function-calling/
77+├── agent.yaml the coordinator
78+├── agent.candidate.yaml the agent under test: its model and its six declared tools
79+├── bench.sh the short way to call the bench: ./bench.sh next
80+├── tools/call.sh the program behind every tool: logs the call, answers a made-up result
81+├── skills/function-calling-check/
82+│ ├── SKILL.md the procedure: the loop, then the report
83+│ ├── bench.sh start · next · record <id> FAIL "why" · calls · table · report
84+│ ├── check.py the check of each test, from work/calls.jsonl
85+│ └── tests.tsv one line per test: id, capability, summary, mode, the exact prompt, a note
86+├── reports/ one Markdown report per run
87+└── work/ the call log of the test in progress (git-ignored)
88+```
new file mode 100644
@@ -0,0 +1,88 @@
1+# Function-calling tester
2+
3+*[Version française](README.fr.md)*
4+
5+The second test set of [`testing/`](../README.en.md): function calling alone. A coordinator follows the `function-calling-check` skill: it sends eight requests, one at a time, to the `candidate` agent with `ask_agent`. The candidate's only tools are the ones declared in its agent file (`tools:`); every call it makes is logged and checked by a script. At the end, a Markdown report in `reports/`.
6+
7+| Agent | File | Model | Role |
8+|---|---|---|---|
9+| 🧪 coordinator | `agent.yaml` | Gemma 4 12B (`hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M`) | runs the loop, writes notes and a verdict |
10+| 🎯 candidate | `agent.candidate.yaml` | Luciole 1B Instruct 1.1 (`hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M`) | the model under test: no built-in tool, six declared tools |
11+
12+## Run
13+
14+```bash
15+llmman pull hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M
16+llmman pull hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M
17+llmman serve
18+
19+cd testing/function-calling
20+../common/run.sh # mm -p "Test the candidate.", then the report even if the run stops early
21+# or, in the window:
22+mm -tui # then type: $function-calling-check
23+```
24+
25+Run it from `testing/function-calling/`. `python3` must be on the PATH (the tools and the checks). Both agents run commands without asking (`yolo: true`): the coordinator only runs `./bench.sh`, the candidate only its declared tools. Both models stay loaded during a run (`unloadModels: false`): reloading the coordinator after each request made it re-read the whole conversation, and the server then broke its stream.
26+
27+## The candidate's tools
28+
29+Declared in `agent.candidate.yaml`, each one runs `tools/call.sh`, which logs the call (tool name and arguments, as mm passed them) in `work/calls.jsonl` and answers with a fixed, made-up result — nothing is looked up or booked.
30+
31+| Tool | Parameters | Made-up answer |
32+|---|---|---|
33+| `get_weather` | `city` string · `unit` enum celsius/fahrenheit, optional, default celsius | `Paris: 18°C, cloudy` |
34+| `convert_currency` | `amount` number · `from`, `to` enum EUR/USD/JPY/GBP | `250 EUR = 270.00 USD` |
35+| `find_user` | `email` string | `user_id: 4217, name: Alice Martin` for alice@example.com |
36+| `list_orders` | `user_id` integer · `status` enum open/shipped/all, optional | two open orders for 4217 |
37+| `create_reminder` | `text`, `when` string · `urgent` boolean, optional | `reminder created: …` |
38+| `book_table` | `restaurant`, `date` string · `people` integer | `table booked …` |
39+
40+To test other tools, change them here and the checks in `skills/function-calling-check/check.py`, which read the calls by tool and argument name.
41+
42+## The eight tests
43+
44+| Test | Capability | Request | PASS when |
45+|---|---|---|---|
46+| F1 | one call, a string argument | the weather in Paris | one `get_weather(city=Paris)`, default unit kept |
47+| F2 | an enum argument | the weather in Tokyo in fahrenheit | one `get_weather(city=Tokyo, unit=fahrenheit)` |
48+| F3 | a number and two enums | convert 250 euros to US dollars | one `convert_currency(250, EUR, USD)` |
49+| F4 | a boolean argument | remind me to call Alice tomorrow at 9:00, urgent | one `create_reminder` with the text, a time, `urgent=true` |
50+| F5 | chained calls | find alice@example.com, then her open orders | `find_user`, then `list_orders(user_id=4217, status=open)` with the id it returned |
51+| F6 | two calls in one request | the weather in Rome and in Oslo | two `get_weather`, Rome and Oslo |
52+| F7 | no call when no tool fits | 12 times 12 | no call |
53+| F8 | no invented arguments | book a table (nothing more said) | no call |
54+
55+The exact prompts are in `skills/function-calling-check/tests.tsv`. One try per test.
56+
57+## How a run goes
58+
59+The bench script drives the loop: the coordinator repeats `./bench.sh next`, then `ask_agent` with the prompt `next` printed. Each `next` checks the calls of the test just run, empties the call log, and hands out the following test, so no check can be skipped. When all are done, the coordinator writes its notes and verdict in `reports/.notes.md` and runs `./bench.sh report`, which writes `reports/<date>-<model>.md`: model, score, results table, notes, list of tests. If the coordinator stops early, `../common/run.sh` writes the report from what was recorded (unchecked tests show as NOT RUN).
60+
61+## Measured results
62+
63+2026-10-26, on the author's llmman, Gemma 4 12B checking Luciole 1B: the full run took **13 min 04 s** and scored **5 of 8** — [the report](reports/2026-10-26-0901-Luciole-1B-Instruct-1.1-GGUF-Q4_K_M.md), written by the coordinator itself.
64+
65+| PASS | FAIL |
66+|---|---|
67+| F1, F2, F4 (string, enum, boolean arguments), F6 (two calls), F8 (no booking with invented values) | F3: runaway tool call · F5: `max_tokens` hit while writing a call · F7: a tool called for 12 × 12, then a loop |
68+
69+Luciole is **not stable from one run to the next** (temperature 0.7, the GGUF's own setting). Over the runs made that day: F3 passed twice and failed twice — once with 42 `convert_currency` calls with made-up amounts in a single answer; F2 failed once with two calls (celsius, then fahrenheit); F5 invented `user_id` values and damaged the e-mail address once. One run is a sample, not a verdict.
70+
71+What the first runs taught the bench (all fixed before the run above): with the coordinator reloaded after each request, its stream broke on the long conversation; with a first procedure that let the coordinator call the checks itself, it skipped all of them once, and twice rewrote the script's path with underscores (hence `./bench.sh` and the `next` loop); it recorded a result under the literal id `test` (now refused).
72+
73+## Files
74+
75+```
76+testing/function-calling/
77+├── agent.yaml the coordinator
78+├── agent.candidate.yaml the agent under test: its model and its six declared tools
79+├── bench.sh the short way to call the bench: ./bench.sh next
80+├── tools/call.sh the program behind every tool: logs the call, answers a made-up result
81+├── skills/function-calling-check/
82+│ ├── SKILL.md the procedure: the loop, then the report
83+│ ├── bench.sh start · next · record <id> FAIL "why" · calls · table · report
84+│ ├── check.py the check of each test, from work/calls.jsonl
85+│ └── tests.tsv one line per test: id, capability, summary, mode, the exact prompt, a note
86+├── reports/ one Markdown report per run
87+└── work/ the call log of the test in progress (git-ignored)
88+```
added testing/function-calling/README.fr.md +88 -0
new file mode 100644
@@ -0,0 +1,88 @@
1+# Testeur d'appels d'outils
2+
3+*[English version](README.en.md)*
4+
5+Le deuxième jeu de tests de [`testing/`](../README.fr.md) : l'appel d'outils seul. Un coordinateur suit la skill `function-calling-check` : il envoie huit requêtes, une par une, à l'agent `candidate` avec `ask_agent`. Les seuls outils du candidat sont ceux déclarés dans son fichier d'agent (`tools:`) ; chaque appel qu'il fait est enregistré et vérifié par un script. À la fin, un rapport Markdown dans `reports/`.
6+
7+| Agent | Fichier | Modèle | Rôle |
8+|---|---|---|---|
9+| 🧪 coordinateur | `agent.yaml` | Gemma 4 12B (`hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M`) | mène la boucle, écrit ses notes et un verdict |
10+| 🎯 candidat | `agent.candidate.yaml` | Luciole 1B Instruct 1.1 (`hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M`) | le modèle testé : aucun outil intégré, six outils déclarés |
11+
12+## Lancer
13+
14+```bash
15+llmman pull hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M
16+llmman pull hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M
17+llmman serve
18+
19+cd testing/function-calling
20+../common/run.sh # mm -p "Test the candidate.", puis le rapport même si le passage s'arrête avant
21+# ou, dans la fenêtre :
22+mm -tui # puis tapez : $function-calling-check
23+```
24+
25+Lancez-le depuis `testing/function-calling/`. `python3` doit être dans le PATH (les outils et les vérifications). Les deux agents exécutent des commandes sans demander (`yolo: true`) : le coordinateur ne lance que `./bench.sh`, le candidat que ses outils déclarés. Les deux modèles restent chargés pendant un passage (`unloadModels: false`) : recharger le coordinateur après chaque requête l'obligeait à relire toute la conversation, et le serveur coupait alors son flux.
26+
27+## Les outils du candidat
28+
29+Déclarés dans `agent.candidate.yaml`, chacun lance `tools/call.sh`, qui enregistre l'appel (nom de l'outil et arguments, tels que mm les a transmis) dans `work/calls.jsonl` et répond un résultat fixe et inventé — rien n'est vraiment cherché ni réservé.
30+
31+| Outil | Paramètres | Réponse inventée |
32+|---|---|---|
33+| `get_weather` | `city` chaîne · `unit` enum celsius/fahrenheit, facultatif, celsius par défaut | `Paris: 18°C, cloudy` |
34+| `convert_currency` | `amount` nombre · `from`, `to` enum EUR/USD/JPY/GBP | `250 EUR = 270.00 USD` |
35+| `find_user` | `email` chaîne | `user_id: 4217, name: Alice Martin` pour alice@example.com |
36+| `list_orders` | `user_id` entier · `status` enum open/shipped/all, facultatif | deux commandes ouvertes pour 4217 |
37+| `create_reminder` | `text`, `when` chaînes · `urgent` booléen, facultatif | `reminder created: …` |
38+| `book_table` | `restaurant`, `date` chaînes · `people` entier | `table booked …` |
39+
40+Pour tester d'autres outils, changez-les ici, ainsi que les vérifications de `skills/function-calling-check/check.py`, qui lisent les appels par nom d'outil et d'argument.
41+
42+## Les huit tests
43+
44+| Test | Capacité | Requête | PASS quand |
45+|---|---|---|---|
46+| F1 | un appel, un argument chaîne | la météo à Paris | un seul `get_weather(city=Paris)`, unité par défaut gardée |
47+| F2 | un argument enum | la météo à Tokyo en fahrenheit | un seul `get_weather(city=Tokyo, unit=fahrenheit)` |
48+| F3 | un nombre et deux enums | convertir 250 euros en dollars US | un seul `convert_currency(250, EUR, USD)` |
49+| F4 | un argument booléen | rappelle-moi d'appeler Alice demain à 9 h, urgent | un seul `create_reminder` avec le texte, une heure, `urgent=true` |
50+| F5 | des appels enchaînés | trouver alice@example.com, puis ses commandes ouvertes | `find_user`, puis `list_orders(user_id=4217, status=open)` avec l'id renvoyé |
51+| F6 | deux appels pour une requête | la météo à Rome et à Oslo | deux `get_weather`, Rome et Oslo |
52+| F7 | aucun appel quand aucun outil ne convient | 12 fois 12 | aucun appel |
53+| F8 | aucun argument inventé | réserve une table (rien d'autre) | aucun appel |
54+
55+Les prompts exacts sont dans `skills/function-calling-check/tests.tsv`. Un seul essai par test.
56+
57+## Déroulement d'un passage
58+
59+C'est le script du banc qui mène la boucle : le coordinateur répète `./bench.sh next`, puis `ask_agent` avec le prompt affiché par `next`. Chaque `next` vérifie les appels du test qui vient de tourner, vide le journal des appels et donne le test suivant : aucune vérification ne peut être sautée. Quand tout est fait, le coordinateur écrit ses notes et son verdict dans `reports/.notes.md` et lance `./bench.sh report`, qui écrit `reports/<date>-<modèle>.md` : modèle, score, tableau des résultats, notes, liste des tests. Si le coordinateur s'arrête avant, `../common/run.sh` écrit le rapport à partir de ce qui a été enregistré (les tests non vérifiés apparaissent en NOT RUN).
60+
61+## Résultats mesurés
62+
63+Le 2026-10-26, sur le llmman de l'auteur, Gemma 4 12B qui vérifie Luciole 1B : le passage complet a pris **13 min 04 s** pour un score de **5 sur 8** — [le rapport](reports/2026-10-26-0901-Luciole-1B-Instruct-1.1-GGUF-Q4_K_M.md), écrit par le coordinateur lui-même.
64+
65+| PASS | FAIL |
66+|---|---|
67+| F1, F2, F4 (arguments chaîne, enum, booléen), F6 (deux appels), F8 (pas de réservation avec des valeurs inventées) | F3 : appel d'outil emballé · F5 : `max_tokens` atteint en écrivant un appel · F7 : un outil appelé pour 12 × 12, puis une boucle |
68+
69+Luciole **n'est pas stable d'un passage à l'autre** (température 0,7, le réglage du GGUF). Sur les passages faits ce jour-là : F3 a réussi deux fois et échoué deux fois — dont une fois avec 42 appels à `convert_currency` aux montants inventés dans une seule réponse ; F2 a échoué une fois avec deux appels (celsius, puis fahrenheit) ; F5 a une fois inventé des `user_id` et abîmé l'adresse e-mail. Un passage est un échantillon, pas un verdict.
70+
71+Ce que les premiers passages ont appris au banc (tout corrigé avant le passage ci-dessus) : avec le coordinateur rechargé après chaque requête, son flux cassait sur la longue conversation ; avec une première procédure qui laissait le coordinateur lancer lui-même les vérifications, il les a une fois toutes sautées, et deux fois réécrit le chemin du script avec des tirets bas (d'où `./bench.sh` et la boucle `next`) ; il a enregistré un résultat sous l'id littéral `test` (désormais refusé).
72+
73+## Fichiers
74+
75+```
76+testing/function-calling/
77+├── agent.yaml le coordinateur
78+├── agent.candidate.yaml l'agent testé : son modèle et ses six outils déclarés
79+├── bench.sh le raccourci pour appeler le banc : ./bench.sh next
80+├── tools/call.sh le programme derrière chaque outil : enregistre l'appel, répond un résultat inventé
81+├── skills/function-calling-check/
82+│ ├── SKILL.md la procédure : la boucle, puis le rapport
83+│ ├── bench.sh start · next · record <id> FAIL "why" · calls · table · report
84+│ ├── check.py la vérification de chaque test, à partir de work/calls.jsonl
85+│ └── tests.tsv une ligne par test : id, capacité, résumé, mode, le prompt exact, une note
86+├── reports/ un rapport Markdown par passage
87+└── work/ le journal d'appels du test en cours (ignoré par git)
88+```
new file mode 100644
@@ -0,0 +1,88 @@
1+# Testeur d'appels d'outils
2+
3+*[English version](README.en.md)*
4+
5+Le deuxième jeu de tests de [`testing/`](../README.fr.md) : l'appel d'outils seul. Un coordinateur suit la skill `function-calling-check` : il envoie huit requêtes, une par une, à l'agent `candidate` avec `ask_agent`. Les seuls outils du candidat sont ceux déclarés dans son fichier d'agent (`tools:`) ; chaque appel qu'il fait est enregistré et vérifié par un script. À la fin, un rapport Markdown dans `reports/`.
6+
7+| Agent | Fichier | Modèle | Rôle |
8+|---|---|---|---|
9+| 🧪 coordinateur | `agent.yaml` | Gemma 4 12B (`hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M`) | mène la boucle, écrit ses notes et un verdict |
10+| 🎯 candidat | `agent.candidate.yaml` | Luciole 1B Instruct 1.1 (`hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M`) | le modèle testé : aucun outil intégré, six outils déclarés |
11+
12+## Lancer
13+
14+```bash
15+llmman pull hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M
16+llmman pull hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M
17+llmman serve
18+
19+cd testing/function-calling
20+../common/run.sh # mm -p "Test the candidate.", puis le rapport même si le passage s'arrête avant
21+# ou, dans la fenêtre :
22+mm -tui # puis tapez : $function-calling-check
23+```
24+
25+Lancez-le depuis `testing/function-calling/`. `python3` doit être dans le PATH (les outils et les vérifications). Les deux agents exécutent des commandes sans demander (`yolo: true`) : le coordinateur ne lance que `./bench.sh`, le candidat que ses outils déclarés. Les deux modèles restent chargés pendant un passage (`unloadModels: false`) : recharger le coordinateur après chaque requête l'obligeait à relire toute la conversation, et le serveur coupait alors son flux.
26+
27+## Les outils du candidat
28+
29+Déclarés dans `agent.candidate.yaml`, chacun lance `tools/call.sh`, qui enregistre l'appel (nom de l'outil et arguments, tels que mm les a transmis) dans `work/calls.jsonl` et répond un résultat fixe et inventé — rien n'est vraiment cherché ni réservé.
30+
31+| Outil | Paramètres | Réponse inventée |
32+|---|---|---|
33+| `get_weather` | `city` chaîne · `unit` enum celsius/fahrenheit, facultatif, celsius par défaut | `Paris: 18°C, cloudy` |
34+| `convert_currency` | `amount` nombre · `from`, `to` enum EUR/USD/JPY/GBP | `250 EUR = 270.00 USD` |
35+| `find_user` | `email` chaîne | `user_id: 4217, name: Alice Martin` pour alice@example.com |
36+| `list_orders` | `user_id` entier · `status` enum open/shipped/all, facultatif | deux commandes ouvertes pour 4217 |
37+| `create_reminder` | `text`, `when` chaînes · `urgent` booléen, facultatif | `reminder created: …` |
38+| `book_table` | `restaurant`, `date` chaînes · `people` entier | `table booked …` |
39+
40+Pour tester d'autres outils, changez-les ici, ainsi que les vérifications de `skills/function-calling-check/check.py`, qui lisent les appels par nom d'outil et d'argument.
41+
42+## Les huit tests
43+
44+| Test | Capacité | Requête | PASS quand |
45+|---|---|---|---|
46+| F1 | un appel, un argument chaîne | la météo à Paris | un seul `get_weather(city=Paris)`, unité par défaut gardée |
47+| F2 | un argument enum | la météo à Tokyo en fahrenheit | un seul `get_weather(city=Tokyo, unit=fahrenheit)` |
48+| F3 | un nombre et deux enums | convertir 250 euros en dollars US | un seul `convert_currency(250, EUR, USD)` |
49+| F4 | un argument booléen | rappelle-moi d'appeler Alice demain à 9 h, urgent | un seul `create_reminder` avec le texte, une heure, `urgent=true` |
50+| F5 | des appels enchaînés | trouver alice@example.com, puis ses commandes ouvertes | `find_user`, puis `list_orders(user_id=4217, status=open)` avec l'id renvoyé |
51+| F6 | deux appels pour une requête | la météo à Rome et à Oslo | deux `get_weather`, Rome et Oslo |
52+| F7 | aucun appel quand aucun outil ne convient | 12 fois 12 | aucun appel |
53+| F8 | aucun argument inventé | réserve une table (rien d'autre) | aucun appel |
54+
55+Les prompts exacts sont dans `skills/function-calling-check/tests.tsv`. Un seul essai par test.
56+
57+## Déroulement d'un passage
58+
59+C'est le script du banc qui mène la boucle : le coordinateur répète `./bench.sh next`, puis `ask_agent` avec le prompt affiché par `next`. Chaque `next` vérifie les appels du test qui vient de tourner, vide le journal des appels et donne le test suivant : aucune vérification ne peut être sautée. Quand tout est fait, le coordinateur écrit ses notes et son verdict dans `reports/.notes.md` et lance `./bench.sh report`, qui écrit `reports/<date>-<modèle>.md` : modèle, score, tableau des résultats, notes, liste des tests. Si le coordinateur s'arrête avant, `../common/run.sh` écrit le rapport à partir de ce qui a été enregistré (les tests non vérifiés apparaissent en NOT RUN).
60+
61+## Résultats mesurés
62+
63+Le 2026-10-26, sur le llmman de l'auteur, Gemma 4 12B qui vérifie Luciole 1B : le passage complet a pris **13 min 04 s** pour un score de **5 sur 8** — [le rapport](reports/2026-10-26-0901-Luciole-1B-Instruct-1.1-GGUF-Q4_K_M.md), écrit par le coordinateur lui-même.
64+
65+| PASS | FAIL |
66+|---|---|
67+| F1, F2, F4 (arguments chaîne, enum, booléen), F6 (deux appels), F8 (pas de réservation avec des valeurs inventées) | F3 : appel d'outil emballé · F5 : `max_tokens` atteint en écrivant un appel · F7 : un outil appelé pour 12 × 12, puis une boucle |
68+
69+Luciole **n'est pas stable d'un passage à l'autre** (température 0,7, le réglage du GGUF). Sur les passages faits ce jour-là : F3 a réussi deux fois et échoué deux fois — dont une fois avec 42 appels à `convert_currency` aux montants inventés dans une seule réponse ; F2 a échoué une fois avec deux appels (celsius, puis fahrenheit) ; F5 a une fois inventé des `user_id` et abîmé l'adresse e-mail. Un passage est un échantillon, pas un verdict.
70+
71+Ce que les premiers passages ont appris au banc (tout corrigé avant le passage ci-dessus) : avec le coordinateur rechargé après chaque requête, son flux cassait sur la longue conversation ; avec une première procédure qui laissait le coordinateur lancer lui-même les vérifications, il les a une fois toutes sautées, et deux fois réécrit le chemin du script avec des tirets bas (d'où `./bench.sh` et la boucle `next`) ; il a enregistré un résultat sous l'id littéral `test` (désormais refusé).
72+
73+## Fichiers
74+
75+```
76+testing/function-calling/
77+├── agent.yaml le coordinateur
78+├── agent.candidate.yaml l'agent testé : son modèle et ses six outils déclarés
79+├── bench.sh le raccourci pour appeler le banc : ./bench.sh next
80+├── tools/call.sh le programme derrière chaque outil : enregistre l'appel, répond un résultat inventé
81+├── skills/function-calling-check/
82+│ ├── SKILL.md la procédure : la boucle, puis le rapport
83+│ ├── bench.sh start · next · record <id> FAIL "why" · calls · table · report
84+│ ├── check.py la vérification de chaque test, à partir de work/calls.jsonl
85+│ └── tests.tsv une ligne par test : id, capacité, résumé, mode, le prompt exact, une note
86+├── reports/ un rapport Markdown par passage
87+└── work/ le journal d'appels du test en cours (ignoré par git)
88+```
added testing/function-calling/agent.candidate.yaml +98 -0
new file mode 100644
@@ -0,0 +1,98 @@
1+# The candidate — the agent whose MODEL is tested on function calling. It
2+# has NO built-in tool (no bash, no file tools): only the six tools declared
3+# below, so every call it makes is one of them. Each runs tools/call.sh,
4+# which logs the call in work/calls.jsonl and answers with a made-up result.
5+#
6+# To test another model, change `model:` (and `sampling:` to that model's
7+# card values), then run the check again from agent.yaml. You can also change
8+# or add tools here: the checks in skills/function-calling-check/check.py
9+# read the calls by tool name and argument name.
10+
11+provider: llmman
12+model: hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M
13+baseUrl: http://127.0.0.1:17434/v1
14+fallback: http://host.docker.internal:17434/v1
15+contextWindow: 32768
16+
17+sampling:
18+ # Luciole 1B Instruct 1.1 (OpenLLM-France / LINAGORA): the GGUF's own
19+ # sampling, read from llmman (general.sampling.temp / top_p): 0.7 / 0.9.
20+ # Its chat template takes tools (Hermes-style <tool_call> JSON) and has no
21+ # thinking switch.
22+ temperature: 0.7
23+ top_p: 0.9
24+ max_tokens: 2048
25+
26+tools:
27+ - name: get_weather
28+ description: Current weather for a city.
29+ parameters:
30+ city: { type: string, description: "City name, e.g. Paris" }
31+ unit: { type: string, enum: [celsius, fahrenheit], required: false, default: celsius, description: "Temperature unit" }
32+ command: ["./tools/call.sh", "get_weather", "city={{city}}", "unit={{unit}}"]
33+
34+ - name: convert_currency
35+ description: Convert an amount of money from one currency to another at today's rate.
36+ parameters:
37+ amount: { type: number, description: "The amount to convert" }
38+ from: { type: string, enum: [EUR, USD, JPY, GBP], description: "Currency of the amount" }
39+ to: { type: string, enum: [EUR, USD, JPY, GBP], description: "Currency wanted" }
40+ command: ["./tools/call.sh", "convert_currency", "amount={{amount}}", "from={{from}}", "to={{to}}"]
41+
42+ - name: find_user
43+ description: Find a customer account by e-mail address. Returns the user_id.
44+ parameters:
45+ email: { type: string, description: "The customer's e-mail address" }
46+ command: ["./tools/call.sh", "find_user", "email={{email}}"]
47+
48+ - name: list_orders
49+ description: List a customer's orders. Needs the user_id that find_user returns.
50+ parameters:
51+ user_id: { type: integer, description: "The customer's user_id" }
52+ status: { type: string, enum: [open, shipped, all], required: false, default: all, description: "Which orders" }
53+ command: ["./tools/call.sh", "list_orders", "user_id={{user_id}}", "status={{status}}"]
54+
55+ - name: create_reminder
56+ description: Create a reminder for the user.
57+ parameters:
58+ text: { type: string, description: "What to be reminded of" }
59+ when: { type: string, description: "When, as the user said it or as a date and time" }
60+ urgent: { type: boolean, required: false, default: false, description: "true for an urgent reminder" }
61+ command: ["./tools/call.sh", "create_reminder", "text={{text}}", "when={{when}}", "urgent={{urgent}}"]
62+
63+ - name: book_table
64+ description: Book a table in a restaurant.
65+ parameters:
66+ restaurant: { type: string, description: "Name of the restaurant" }
67+ date: { type: string, description: "Date and time of the booking" }
68+ people: { type: integer, description: "Number of people" }
69+ command: ["./tools/call.sh", "book_table", "restaurant={{restaurant}}", "date={{date}}", "people={{people}}"]
70+
71+maxTurns: 8
72+maxOutput: 4000
73+displayCommands: true
74+
75+yolo: true
76+bashTool: false
77+editTools: false
78+planTool: false
79+askUserTool: false
80+fetchContentTool: false
81+saveAnswerTool: false
82+unloadModels: false
83+skillsDir: no-skills
84+
85+agentsMd:
86+ enabled: false
87+
88+sessions:
89+ enabled: false
90+
91+acp:
92+ name: candidate
93+ title: 🎯 Candidate (function calling under test)
94+
95+# Neutral on purpose: no hint on when to call or not, so F7 and F8 test the
96+# model, not the prompt.
97+system: |
98+ You are an assistant with tools. Answer briefly.
new file mode 100644
@@ -0,0 +1,98 @@
1+# The candidate — the agent whose MODEL is tested on function calling. It
2+# has NO built-in tool (no bash, no file tools): only the six tools declared
3+# below, so every call it makes is one of them. Each runs tools/call.sh,
4+# which logs the call in work/calls.jsonl and answers with a made-up result.
5+#
6+# To test another model, change `model:` (and `sampling:` to that model's
7+# card values), then run the check again from agent.yaml. You can also change
8+# or add tools here: the checks in skills/function-calling-check/check.py
9+# read the calls by tool name and argument name.
10+
11+provider: llmman
12+model: hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M
13+baseUrl: http://127.0.0.1:17434/v1
14+fallback: http://host.docker.internal:17434/v1
15+contextWindow: 32768
16+
17+sampling:
18+ # Luciole 1B Instruct 1.1 (OpenLLM-France / LINAGORA): the GGUF's own
19+ # sampling, read from llmman (general.sampling.temp / top_p): 0.7 / 0.9.
20+ # Its chat template takes tools (Hermes-style <tool_call> JSON) and has no
21+ # thinking switch.
22+ temperature: 0.7
23+ top_p: 0.9
24+ max_tokens: 2048
25+
26+tools:
27+ - name: get_weather
28+ description: Current weather for a city.
29+ parameters:
30+ city: { type: string, description: "City name, e.g. Paris" }
31+ unit: { type: string, enum: [celsius, fahrenheit], required: false, default: celsius, description: "Temperature unit" }
32+ command: ["./tools/call.sh", "get_weather", "city={{city}}", "unit={{unit}}"]
33+
34+ - name: convert_currency
35+ description: Convert an amount of money from one currency to another at today's rate.
36+ parameters:
37+ amount: { type: number, description: "The amount to convert" }
38+ from: { type: string, enum: [EUR, USD, JPY, GBP], description: "Currency of the amount" }
39+ to: { type: string, enum: [EUR, USD, JPY, GBP], description: "Currency wanted" }
40+ command: ["./tools/call.sh", "convert_currency", "amount={{amount}}", "from={{from}}", "to={{to}}"]
41+
42+ - name: find_user
43+ description: Find a customer account by e-mail address. Returns the user_id.
44+ parameters:
45+ email: { type: string, description: "The customer's e-mail address" }
46+ command: ["./tools/call.sh", "find_user", "email={{email}}"]
47+
48+ - name: list_orders
49+ description: List a customer's orders. Needs the user_id that find_user returns.
50+ parameters:
51+ user_id: { type: integer, description: "The customer's user_id" }
52+ status: { type: string, enum: [open, shipped, all], required: false, default: all, description: "Which orders" }
53+ command: ["./tools/call.sh", "list_orders", "user_id={{user_id}}", "status={{status}}"]
54+
55+ - name: create_reminder
56+ description: Create a reminder for the user.
57+ parameters:
58+ text: { type: string, description: "What to be reminded of" }
59+ when: { type: string, description: "When, as the user said it or as a date and time" }
60+ urgent: { type: boolean, required: false, default: false, description: "true for an urgent reminder" }
61+ command: ["./tools/call.sh", "create_reminder", "text={{text}}", "when={{when}}", "urgent={{urgent}}"]
62+
63+ - name: book_table
64+ description: Book a table in a restaurant.
65+ parameters:
66+ restaurant: { type: string, description: "Name of the restaurant" }
67+ date: { type: string, description: "Date and time of the booking" }
68+ people: { type: integer, description: "Number of people" }
69+ command: ["./tools/call.sh", "book_table", "restaurant={{restaurant}}", "date={{date}}", "people={{people}}"]
70+
71+maxTurns: 8
72+maxOutput: 4000
73+displayCommands: true
74+
75+yolo: true
76+bashTool: false
77+editTools: false
78+planTool: false
79+askUserTool: false
80+fetchContentTool: false
81+saveAnswerTool: false
82+unloadModels: false
83+skillsDir: no-skills
84+
85+agentsMd:
86+ enabled: false
87+
88+sessions:
89+ enabled: false
90+
91+acp:
92+ name: candidate
93+ title: 🎯 Candidate (function calling under test)
94+
95+# Neutral on purpose: no hint on when to call or not, so F7 and F8 test the
96+# model, not the prompt.
97+system: |
98+ You are an assistant with tools. Answer briefly.
added testing/function-calling/agent.yaml +95 -0
new file mode 100644
@@ -0,0 +1,95 @@
1+# Function-calling tester — a coordinator that tests how well another
2+# agent's model calls tools. It follows the `function-calling-check` skill:
3+# eight requests given one by one to the `candidate` agent
4+# (agent.candidate.yaml, whose ONLY tools are declared in its `tools:` block)
5+# with ask_agent; every call the candidate makes is logged by tools/call.sh
6+# and checked by skills/function-calling-check/check.py; then a Markdown
7+# report in reports/.
8+#
9+# coordinator Gemma 4 12B hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M 7.3 GB
10+# candidate see agent.candidate.yaml — change its `model:` line to test another model
11+#
12+# cd testing/function-calling
13+# mm -tui then type: $function-calling-check
14+#
15+# Run it from this folder: the skill, tools/, work/ and reports/ are found
16+# from here. It runs commands without asking (yolo: true): the coordinator
17+# only runs the bench script, the candidate only its declared tools.
18+# Needs `python3` on the PATH (the tools and the checks).
19+
20+provider: llmman
21+model: hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M
22+baseUrl: http://127.0.0.1:17434/v1
23+fallback: http://host.docker.internal:17434/v1
24+contextWindow: 32768
25+
26+mmConfigs:
27+ candidate:
28+ path: agent.candidate.yaml
29+ description: >-
30+ The agent under test. Give it one request at a time, exactly as the
31+ function-calling-check skill words it.
32+ yolo: true
33+ # Both models stay loaded: the candidate is small, and reloading the
34+ # coordinator after each request made it re-read the whole conversation —
35+ # the server then broke its stream (seen 2026-10-26, Luciole 1B run).
36+ unloadModels: false
37+
38+sampling:
39+ # Gemma 4: 0.2 / 0.95 / 64 is the "precise" setting measured in the
40+ # demos (internal/agentinit/presets.yaml).
41+ temperature: 0.2
42+ top_p: 0.95
43+ top_k: 64
44+ max_tokens: 4096
45+ parallel_tool_calls: false
46+ # Seen on llmman (2026-10-26): without it this GGUF thinks before every
47+ # answer (reasoning_content) — slower, for a job that is a procedure.
48+ chat_template_kwargs:
49+ enable_thinking: false
50+
51+# start + 8 × (next, ask) + judged records + next + notes + report ≈ 25 round trips.
52+maxTurns: 40
53+# ask_agent hands back at most this much of the candidate's answer. Kept
54+# small: a candidate that runs away (seen: Luciole 1B, 42 calls then text in
55+# a loop) must not flood the coordinator's context — the server broke the
56+# coordinator's stream on such a long prompt (2026-10-26).
57+maxOutput: 2000
58+bashTimeout: 2m
59+# During ask_agent every event of the candidate counts as a sign of life.
60+watchdogTimeout: 5m
61+firstTokenTimeout: 3m
62+displayCommands: true
63+
64+yolo: true
65+bashTool: true
66+editTools: true
67+planTool: false
68+askUserTool: false
69+fetchContentTool: false
70+skillsDir: skills
71+
72+# The whole check is ONE question, so the history is never compressed
73+# during it: 32k must hold the skill and the eight answers (each cut at
74+# maxOutput). The results are also kept in reports/.results.tsv.
75+
76+agentsMd:
77+ enabled: false
78+
79+sessions:
80+ enabled: false
81+
82+acp:
83+ name: function-calling-tester
84+ title: 🧪 Function-calling tester
85+
86+system: |
87+ You are a function-calling tester. You test the agent named candidate:
88+ you send it requests with the ask_agent tool and check the tool calls it
89+ made. You never answer its requests yourself.
90+
91+ When the user asks you to test, to check or to evaluate the candidate,
92+ load the function-calling-check skill with read_skill and follow it: the bench
93+ script tells you each step; do exactly what it prints.
94+
95+ Answer the user in their language, briefly. The report is in English.
new file mode 100644
@@ -0,0 +1,95 @@
1+# Function-calling tester — a coordinator that tests how well another
2+# agent's model calls tools. It follows the `function-calling-check` skill:
3+# eight requests given one by one to the `candidate` agent
4+# (agent.candidate.yaml, whose ONLY tools are declared in its `tools:` block)
5+# with ask_agent; every call the candidate makes is logged by tools/call.sh
6+# and checked by skills/function-calling-check/check.py; then a Markdown
7+# report in reports/.
8+#
9+# coordinator Gemma 4 12B hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M 7.3 GB
10+# candidate see agent.candidate.yaml — change its `model:` line to test another model
11+#
12+# cd testing/function-calling
13+# mm -tui then type: $function-calling-check
14+#
15+# Run it from this folder: the skill, tools/, work/ and reports/ are found
16+# from here. It runs commands without asking (yolo: true): the coordinator
17+# only runs the bench script, the candidate only its declared tools.
18+# Needs `python3` on the PATH (the tools and the checks).
19+
20+provider: llmman
21+model: hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M
22+baseUrl: http://127.0.0.1:17434/v1
23+fallback: http://host.docker.internal:17434/v1
24+contextWindow: 32768
25+
26+mmConfigs:
27+ candidate:
28+ path: agent.candidate.yaml
29+ description: >-
30+ The agent under test. Give it one request at a time, exactly as the
31+ function-calling-check skill words it.
32+ yolo: true
33+ # Both models stay loaded: the candidate is small, and reloading the
34+ # coordinator after each request made it re-read the whole conversation —
35+ # the server then broke its stream (seen 2026-10-26, Luciole 1B run).
36+ unloadModels: false
37+
38+sampling:
39+ # Gemma 4: 0.2 / 0.95 / 64 is the "precise" setting measured in the
40+ # demos (internal/agentinit/presets.yaml).
41+ temperature: 0.2
42+ top_p: 0.95
43+ top_k: 64
44+ max_tokens: 4096
45+ parallel_tool_calls: false
46+ # Seen on llmman (2026-10-26): without it this GGUF thinks before every
47+ # answer (reasoning_content) — slower, for a job that is a procedure.
48+ chat_template_kwargs:
49+ enable_thinking: false
50+
51+# start + 8 × (next, ask) + judged records + next + notes + report ≈ 25 round trips.
52+maxTurns: 40
53+# ask_agent hands back at most this much of the candidate's answer. Kept
54+# small: a candidate that runs away (seen: Luciole 1B, 42 calls then text in
55+# a loop) must not flood the coordinator's context — the server broke the
56+# coordinator's stream on such a long prompt (2026-10-26).
57+maxOutput: 2000
58+bashTimeout: 2m
59+# During ask_agent every event of the candidate counts as a sign of life.
60+watchdogTimeout: 5m
61+firstTokenTimeout: 3m
62+displayCommands: true
63+
64+yolo: true
65+bashTool: true
66+editTools: true
67+planTool: false
68+askUserTool: false
69+fetchContentTool: false
70+skillsDir: skills
71+
72+# The whole check is ONE question, so the history is never compressed
73+# during it: 32k must hold the skill and the eight answers (each cut at
74+# maxOutput). The results are also kept in reports/.results.tsv.
75+
76+agentsMd:
77+ enabled: false
78+
79+sessions:
80+ enabled: false
81+
82+acp:
83+ name: function-calling-tester
84+ title: 🧪 Function-calling tester
85+
86+system: |
87+ You are a function-calling tester. You test the agent named candidate:
88+ you send it requests with the ask_agent tool and check the tool calls it
89+ made. You never answer its requests yourself.
90+
91+ When the user asks you to test, to check or to evaluate the candidate,
92+ load the function-calling-check skill with read_skill and follow it: the bench
93+ script tells you each step; do exactly what it prints.
94+
95+ Answer the user in their language, briefly. The report is in English.
added testing/function-calling/bench.sh +5 -0
new file mode 100755
@@ -0,0 +1,5 @@
1+#!/usr/bin/env bash
2+# The short way to call this set's bench, from this folder: ./bench.sh next
3+# (seen: a coordinator rewrote skills/function-calling-check/ with underscores and looped on
4+# "No such file"). The real script is skills/function-calling-check/bench.sh.
5+exec "$(dirname "$0")/skills/function-calling-check/bench.sh" "$@"
new file mode 100755
@@ -0,0 +1,5 @@
1+#!/usr/bin/env bash
2+# The short way to call this set's bench, from this folder: ./bench.sh next
3+# (seen: a coordinator rewrote skills/function-calling-check/ with underscores and looped on
4+# "No such file"). The real script is skills/function-calling-check/bench.sh.
5+exec "$(dirname "$0")/skills/function-calling-check/bench.sh" "$@"
added testing/function-calling/reports/2026-10-26-0901-Luciole-1B-Instruct-1.1-GGUF-Q4_K_M.md +46 -0
new file mode 100644
@@ -0,0 +1,46 @@
1+# Function-calling check — hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M
2+
3+- **Date**: 2026-10-26 09:01
4+- **Tested model**: `hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M` (agent.candidate.yaml)
5+- **Coordinator**: `hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M` (agent.yaml)
6+- **Score**: 5 of 8 passed
7+
8+## Results
9+
10+| Test | Capability | Result | Detail |
11+|---|---|---|---|
12+| F1 | one call, a string argument | PASS | get_weather(city=Paris), default unit kept |
13+| F2 | an enum argument | PASS | get_weather(city=Tokyo, unit=fahrenheit) |
14+| F3 | a number and two enums | FAIL | runaway tool call |
15+| F4 | a boolean argument | PASS | create_reminder with text, when, urgent=true |
16+| F5 | chained calls | FAIL | max_tokens hit while writing tool call |
17+| F6 | two calls in one request | PASS | get_weather for Rome and for Oslo |
18+| F7 | no call when no tool fits | FAIL | hallucinated tool call and loop |
19+| F8 | no invented arguments | PASS | no booking with invented values |
20+
21+## What the coordinator saw
22+
23+- The candidate successfully handled simple tool calls with string and enum arguments (F1, F2).
24+- It correctly identified when no tool was needed for a simple math question (F7) but failed by hallucinating a tool call and entering a loop.
25+- It successfully handled a request requiring two tool calls in one prompt (F6).
26+- It correctly asked for missing information instead of inventing values for a booking request (F8).
27+- It failed on chained calls (F5) by hitting the max token limit while generating a tool call.
28+- It suffered from a runaway tool call loop on a conversion task (F3).
29+
30+## Verdict
31+The model can be trusted with simple, single-turn tool calls where the parameters are clearly provided. It should not be used for complex chained tool calls or tasks where it might hallucinate tool usage for simple logic.
32+
33+## The tests
34+
35+| Test | Capability | What was asked |
36+|---|---|---|
37+| F1 | one call, a string argument | the weather in Paris (default unit kept) |
38+| F2 | an enum argument | the weather in Tokyo in fahrenheit |
39+| F3 | a number and two enums | convert 250 euros to US dollars |
40+| F4 | a boolean argument | a reminder to call Alice tomorrow at 9:00, urgent |
41+| F5 | chained calls | find alice@example.com, then her open orders with the user_id found |
42+| F6 | two calls in one request | the weather in Rome and in Oslo |
43+| F7 | no call when no tool fits | 12 times 12 |
44+| F8 | no invented arguments | book a table, with no restaurant, date or number of people given |
45+
46+The exact prompts are in `skills/function-calling-check/tests.tsv`.
new file mode 100644
@@ -0,0 +1,46 @@
1+# Function-calling check — hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M
2+
3+- **Date**: 2026-10-26 09:01
4+- **Tested model**: `hf.co/OpenLLM-France/Luciole-1B-Instruct-1.1-GGUF:Q4_K_M` (agent.candidate.yaml)
5+- **Coordinator**: `hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M` (agent.yaml)
6+- **Score**: 5 of 8 passed
7+
8+## Results
9+
10+| Test | Capability | Result | Detail |
11+|---|---|---|---|
12+| F1 | one call, a string argument | PASS | get_weather(city=Paris), default unit kept |
13+| F2 | an enum argument | PASS | get_weather(city=Tokyo, unit=fahrenheit) |
14+| F3 | a number and two enums | FAIL | runaway tool call |
15+| F4 | a boolean argument | PASS | create_reminder with text, when, urgent=true |
16+| F5 | chained calls | FAIL | max_tokens hit while writing tool call |
17+| F6 | two calls in one request | PASS | get_weather for Rome and for Oslo |
18+| F7 | no call when no tool fits | FAIL | hallucinated tool call and loop |
19+| F8 | no invented arguments | PASS | no booking with invented values |
20+
21+## What the coordinator saw
22+
23+- The candidate successfully handled simple tool calls with string and enum arguments (F1, F2).
24+- It correctly identified when no tool was needed for a simple math question (F7) but failed by hallucinating a tool call and entering a loop.
25+- It successfully handled a request requiring two tool calls in one prompt (F6).
26+- It correctly asked for missing information instead of inventing values for a booking request (F8).
27+- It failed on chained calls (F5) by hitting the max token limit while generating a tool call.
28+- It suffered from a runaway tool call loop on a conversion task (F3).
29+
30+## Verdict
31+The model can be trusted with simple, single-turn tool calls where the parameters are clearly provided. It should not be used for complex chained tool calls or tasks where it might hallucinate tool usage for simple logic.
32+
33+## The tests
34+
35+| Test | Capability | What was asked |
36+|---|---|---|
37+| F1 | one call, a string argument | the weather in Paris (default unit kept) |
38+| F2 | an enum argument | the weather in Tokyo in fahrenheit |
39+| F3 | a number and two enums | convert 250 euros to US dollars |
40+| F4 | a boolean argument | a reminder to call Alice tomorrow at 9:00, urgent |
41+| F5 | chained calls | find alice@example.com, then her open orders with the user_id found |
42+| F6 | two calls in one request | the weather in Rome and in Oslo |
43+| F7 | no call when no tool fits | 12 times 12 |
44+| F8 | no invented arguments | book a table, with no restaurant, date or number of people given |
45+
46+The exact prompts are in `skills/function-calling-check/tests.tsv`.
added testing/function-calling/skills/function-calling-check/SKILL.md +31 -0
new file mode 100644
@@ -0,0 +1,31 @@
1+---
2+name: function-calling-check
3+description: Test how well the candidate agent's model calls tools — the right tool, the right arguments and types, chained calls, two calls at once, and no call when none fits — with eight small requests, each checked from the call log, then a Markdown report.
4+keywords: test, check, function calling, tool calls, tools, model, candidate, report
5+---
6+
7+# Function-calling check
8+
9+You test the agent named `candidate`. Its only tools are the ones declared in its agent file; every call it makes is logged and checked by a script. You never answer the requests yourself. The script tells you what to do at each step: `./bench.sh`.
10+
11+## The loop
12+
13+1. Run with bash: `./bench.sh start`
14+2. Run with bash: `./bench.sh next`
15+3. Do what `next` printed:
16+ - **NEXT**: call `ask_agent` with `agent: candidate`, `with_context: false`, and as `prompt` the text it printed, copied exactly. When the candidate has answered, go back to step 2. If `next` printed an "Afterwards" line, keep that note for the report.
17+ - **ALL TESTS DONE**: go to "The report".
18+
19+`next` checks the test just run before it hands out the next one: never skip it, and never call `ask_agent` twice without `next` in between. Never give the candidate a second try.
20+
21+If `ask_agent` returns an error, run with bash `./bench.sh record <id> FAIL "<the error, short, no double quotes>"`, where `<id>` is the test `next` printed last (F3, T5…), then go back to step 2.
22+
23+While the loop runs, look at each answer of the candidate: did it use what the tool returned, did it ask for what was missing, did it invent anything.
24+
25+## The report
26+
27+1. Write your notes with `write_file` at the path `reports/.notes.md`, in this order:
28+ - three to six bullets on what you saw: whether the answers used the tools' results, what the candidate answered to F7 and F8, invented values, errors, retries or loops;
29+ - a heading `## Verdict`, then two sentences: what this model can be trusted with in tool calling, and what it should not be given.
30+2. Run with bash: `./bench.sh report`. It writes the Markdown report and prints its path.
31+3. Answer the user in one or two sentences: the score and the report's path.
new file mode 100644
@@ -0,0 +1,31 @@
1+---
2+name: function-calling-check
3+description: Test how well the candidate agent's model calls tools — the right tool, the right arguments and types, chained calls, two calls at once, and no call when none fits — with eight small requests, each checked from the call log, then a Markdown report.
4+keywords: test, check, function calling, tool calls, tools, model, candidate, report
5+---
6+
7+# Function-calling check
8+
9+You test the agent named `candidate`. Its only tools are the ones declared in its agent file; every call it makes is logged and checked by a script. You never answer the requests yourself. The script tells you what to do at each step: `./bench.sh`.
10+
11+## The loop
12+
13+1. Run with bash: `./bench.sh start`
14+2. Run with bash: `./bench.sh next`
15+3. Do what `next` printed:
16+ - **NEXT**: call `ask_agent` with `agent: candidate`, `with_context: false`, and as `prompt` the text it printed, copied exactly. When the candidate has answered, go back to step 2. If `next` printed an "Afterwards" line, keep that note for the report.
17+ - **ALL TESTS DONE**: go to "The report".
18+
19+`next` checks the test just run before it hands out the next one: never skip it, and never call `ask_agent` twice without `next` in between. Never give the candidate a second try.
20+
21+If `ask_agent` returns an error, run with bash `./bench.sh record <id> FAIL "<the error, short, no double quotes>"`, where `<id>` is the test `next` printed last (F3, T5…), then go back to step 2.
22+
23+While the loop runs, look at each answer of the candidate: did it use what the tool returned, did it ask for what was missing, did it invent anything.
24+
25+## The report
26+
27+1. Write your notes with `write_file` at the path `reports/.notes.md`, in this order:
28+ - three to six bullets on what you saw: whether the answers used the tools' results, what the candidate answered to F7 and F8, invented values, errors, retries or loops;
29+ - a heading `## Verdict`, then two sentences: what this model can be trusted with in tool calling, and what it should not be given.
30+2. Run with bash: `./bench.sh report`. It writes the Markdown report and prints its path.
31+3. Answer the user in one or two sentences: the score and the report's path.
added testing/function-calling/skills/function-calling-check/bench.sh +38 -0
new file mode 100755
@@ -0,0 +1,38 @@
1+#!/usr/bin/env bash
2+# The function-calling bench. The loop and the report are in
3+# common/bench-lib.sh; this file says how to reset work/ (the call log
4+# work/calls.jsonl) and how to check a test (check.py reads the calls).
5+#
6+# bench.sh start new run: prints the tested model and the report path
7+# bench.sh next checks the test just run, hands out the next one
8+# bench.sh record F2 FAIL "…" records a test that could not run (an ask_agent error)
9+# bench.sh calls the calls logged since the current test started
10+# bench.sh table the results so far, as a Markdown table
11+# bench.sh report writes the Markdown report (results + reports/.notes.md)
12+#
13+# Run from anywhere: the paths are taken from this script's own location.
14+set -eu
15+
16+SKILL="$(cd "$(dirname "$0")" && pwd)"
17+ROOT="$(cd "$SKILL/../.." && pwd)" # testing/function-calling/
18+WORK="$ROOT/work"
19+TITLE="Function-calling check"
20+# shellcheck source=../../../common/bench-lib.sh
21+. "$ROOT/../common/bench-lib.sh"
22+
23+reset_work() {
24+ rm -rf "$WORK"
25+ mkdir -p "$WORK"
26+}
27+
28+check_test() {
29+ local line
30+ line="$(python3 "$SKILL/check.py" "$1" "$WORK/calls.jsonl")"
31+ bench_record "$1" "${line%% *}" "${line#* }"
32+}
33+
34+if [ "${1:-}" = "calls" ]; then
35+ cat "$WORK/calls.jsonl" 2>/dev/null || echo "no call logged"
36+else
37+ bench_main "$@"
38+fi
new file mode 100755
@@ -0,0 +1,38 @@
1+#!/usr/bin/env bash
2+# The function-calling bench. The loop and the report are in
3+# common/bench-lib.sh; this file says how to reset work/ (the call log
4+# work/calls.jsonl) and how to check a test (check.py reads the calls).
5+#
6+# bench.sh start new run: prints the tested model and the report path
7+# bench.sh next checks the test just run, hands out the next one
8+# bench.sh record F2 FAIL "…" records a test that could not run (an ask_agent error)
9+# bench.sh calls the calls logged since the current test started
10+# bench.sh table the results so far, as a Markdown table
11+# bench.sh report writes the Markdown report (results + reports/.notes.md)
12+#
13+# Run from anywhere: the paths are taken from this script's own location.
14+set -eu
15+
16+SKILL="$(cd "$(dirname "$0")" && pwd)"
17+ROOT="$(cd "$SKILL/../.." && pwd)" # testing/function-calling/
18+WORK="$ROOT/work"
19+TITLE="Function-calling check"
20+# shellcheck source=../../../common/bench-lib.sh
21+. "$ROOT/../common/bench-lib.sh"
22+
23+reset_work() {
24+ rm -rf "$WORK"
25+ mkdir -p "$WORK"
26+}
27+
28+check_test() {
29+ local line
30+ line="$(python3 "$SKILL/check.py" "$1" "$WORK/calls.jsonl")"
31+ bench_record "$1" "${line%% *}" "${line#* }"
32+}
33+
34+if [ "${1:-}" = "calls" ]; then
35+ cat "$WORK/calls.jsonl" 2>/dev/null || echo "no call logged"
36+else
37+ bench_main "$@"
38+fi
added testing/function-calling/skills/function-calling-check/check.py +143 -0
new file mode 100644
@@ -0,0 +1,143 @@
1+"""Checks the tool calls of one function-calling test.
2+
3+ python3 check.py F1 work/calls.jsonl
4+
5+Reads the calls tools/call.sh logged (one JSON object per line: the tool's
6+name and its arguments as strings, the way mm substituted them) and prints
7+one line: PASS or FAIL, a tab, and the reason. Exit code 0 either way; 2
8+for an unknown test.
9+"""
10+import json
11+import sys
12+
13+
14+def load(path):
15+ try:
16+ with open(path) as f:
17+ return [json.loads(line) for line in f if line.strip()]
18+ except FileNotFoundError:
19+ return []
20+
21+
22+def show(calls):
23+ """The calls in one short line, for a FAIL reason."""
24+ if not calls:
25+ return "no tool call"
26+ return "; ".join(f"{c['tool']}({', '.join(f'{k}={v}' for k, v in c['args'].items())})" for c in calls)[:150]
27+
28+
29+def same(value, wanted):
30+ return (value or "").strip().lower() == wanted.lower()
31+
32+
33+def number(value):
34+ try:
35+ return float(value)
36+ except (TypeError, ValueError):
37+ return None
38+
39+
40+def one_call(calls, tool):
41+ """The single call to `tool`, or a reason why there is not exactly one."""
42+ if len(calls) != 1:
43+ return None, f"expected one call, got {len(calls)}: {show(calls)}"
44+ if calls[0]["tool"] != tool:
45+ return None, f"expected {tool}, got {show(calls)}"
46+ return calls[0]["args"], None
47+
48+
49+def f1(calls):
50+ args, why = one_call(calls, "get_weather")
51+ if why:
52+ return False, why
53+ if not same(args.get("city"), "Paris"):
54+ return False, f"city={args.get('city')}"
55+ if not same(args.get("unit"), "celsius"):
56+ return False, f"unit={args.get('unit')} (nothing asked: the default celsius was expected)"
57+ return True, "get_weather(city=Paris), default unit kept"
58+
59+
60+def f2(calls):
61+ args, why = one_call(calls, "get_weather")
62+ if why:
63+ return False, why
64+ if not same(args.get("city"), "Tokyo") or not same(args.get("unit"), "fahrenheit"):
65+ return False, show(calls)
66+ return True, "get_weather(city=Tokyo, unit=fahrenheit)"
67+
68+
69+def f3(calls):
70+ args, why = one_call(calls, "convert_currency")
71+ if why:
72+ return False, why
73+ if number(args.get("amount")) != 250 or args.get("from") != "EUR" or args.get("to") != "USD":
74+ return False, show(calls)
75+ return True, "convert_currency(amount=250, from=EUR, to=USD)"
76+
77+
78+def f4(calls):
79+ args, why = one_call(calls, "create_reminder")
80+ if why:
81+ return False, why
82+ text = (args.get("text") or "").lower()
83+ if "alice" not in text or "call" not in text:
84+ return False, f"text={args.get('text')}"
85+ if not (args.get("when") or "").strip():
86+ return False, "when is empty"
87+ if not same(args.get("urgent"), "true"):
88+ return False, f"urgent={args.get('urgent')} (the user said it is urgent)"
89+ return True, "create_reminder with text, when, urgent=true"
90+
91+
92+def f5(calls):
93+ tools = [c["tool"] for c in calls]
94+ if tools != ["find_user", "list_orders"]:
95+ return False, f"expected find_user then list_orders, got {show(calls)}"
96+ if not same(calls[0]["args"].get("email"), "alice@example.com"):
97+ return False, show(calls)
98+ second = calls[1]["args"]
99+ if second.get("user_id") != "4217" or not same(second.get("status"), "open"):
100+ return False, f"list_orders not called with the user_id found (4217) and status=open: {show(calls)}"
101+ return True, "find_user, then list_orders(user_id=4217 from the result, status=open)"
102+
103+
104+def f6(calls):
105+ if len(calls) != 2 or any(c["tool"] != "get_weather" for c in calls):
106+ return False, f"expected two get_weather calls, got {show(calls)}"
107+ cities = sorted((c["args"].get("city") or "").strip().lower() for c in calls)
108+ if cities != ["oslo", "rome"]:
109+ return False, show(calls)
110+ return True, "get_weather for Rome and for Oslo"
111+
112+
113+def no_call(why_none):
114+ def check(calls):
115+ if calls:
116+ return False, f"no call expected, got {show(calls)}"
117+ return True, why_none
118+ return check
119+
120+
121+CHECKS = {
122+ "F1": f1,
123+ "F2": f2,
124+ "F3": f3,
125+ "F4": f4,
126+ "F5": f5,
127+ "F6": f6,
128+ "F7": no_call("no tool call for a question no tool answers"),
129+ "F8": no_call("no booking with invented values"),
130+}
131+
132+
133+def main():
134+ test, path = sys.argv[1], sys.argv[2]
135+ if test not in CHECKS:
136+ print(f"unknown test {test}: {', '.join(CHECKS)}", file=sys.stderr)
137+ sys.exit(2)
138+ ok, why = CHECKS[test](load(path))
139+ print(f"{'PASS' if ok else 'FAIL'}\t{why}")
140+
141+
142+if __name__ == "__main__":
143+ main()
new file mode 100644
@@ -0,0 +1,143 @@
1+"""Checks the tool calls of one function-calling test.
2+
3+ python3 check.py F1 work/calls.jsonl
4+
5+Reads the calls tools/call.sh logged (one JSON object per line: the tool's
6+name and its arguments as strings, the way mm substituted them) and prints
7+one line: PASS or FAIL, a tab, and the reason. Exit code 0 either way; 2
8+for an unknown test.
9+"""
10+import json
11+import sys
12+
13+
14+def load(path):
15+ try:
16+ with open(path) as f:
17+ return [json.loads(line) for line in f if line.strip()]
18+ except FileNotFoundError:
19+ return []
20+
21+
22+def show(calls):
23+ """The calls in one short line, for a FAIL reason."""
24+ if not calls:
25+ return "no tool call"
26+ return "; ".join(f"{c['tool']}({', '.join(f'{k}={v}' for k, v in c['args'].items())})" for c in calls)[:150]
27+
28+
29+def same(value, wanted):
30+ return (value or "").strip().lower() == wanted.lower()
31+
32+
33+def number(value):
34+ try:
35+ return float(value)
36+ except (TypeError, ValueError):
37+ return None
38+
39+
40+def one_call(calls, tool):
41+ """The single call to `tool`, or a reason why there is not exactly one."""
42+ if len(calls) != 1:
43+ return None, f"expected one call, got {len(calls)}: {show(calls)}"
44+ if calls[0]["tool"] != tool:
45+ return None, f"expected {tool}, got {show(calls)}"
46+ return calls[0]["args"], None
47+
48+
49+def f1(calls):
50+ args, why = one_call(calls, "get_weather")
51+ if why:
52+ return False, why
53+ if not same(args.get("city"), "Paris"):
54+ return False, f"city={args.get('city')}"
55+ if not same(args.get("unit"), "celsius"):
56+ return False, f"unit={args.get('unit')} (nothing asked: the default celsius was expected)"
57+ return True, "get_weather(city=Paris), default unit kept"
58+
59+
60+def f2(calls):
61+ args, why = one_call(calls, "get_weather")
62+ if why:
63+ return False, why
64+ if not same(args.get("city"), "Tokyo") or not same(args.get("unit"), "fahrenheit"):
65+ return False, show(calls)
66+ return True, "get_weather(city=Tokyo, unit=fahrenheit)"
67+
68+
69+def f3(calls):
70+ args, why = one_call(calls, "convert_currency")
71+ if why:
72+ return False, why
73+ if number(args.get("amount")) != 250 or args.get("from") != "EUR" or args.get("to") != "USD":
74+ return False, show(calls)
75+ return True, "convert_currency(amount=250, from=EUR, to=USD)"
76+
77+
78+def f4(calls):
79+ args, why = one_call(calls, "create_reminder")
80+ if why:
81+ return False, why
82+ text = (args.get("text") or "").lower()
83+ if "alice" not in text or "call" not in text:
84+ return False, f"text={args.get('text')}"
85+ if not (args.get("when") or "").strip():
86+ return False, "when is empty"
87+ if not same(args.get("urgent"), "true"):
88+ return False, f"urgent={args.get('urgent')} (the user said it is urgent)"
89+ return True, "create_reminder with text, when, urgent=true"
90+
91+
92+def f5(calls):
93+ tools = [c["tool"] for c in calls]
94+ if tools != ["find_user", "list_orders"]:
95+ return False, f"expected find_user then list_orders, got {show(calls)}"
96+ if not same(calls[0]["args"].get("email"), "alice@example.com"):
97+ return False, show(calls)
98+ second = calls[1]["args"]
99+ if second.get("user_id") != "4217" or not same(second.get("status"), "open"):
100+ return False, f"list_orders not called with the user_id found (4217) and status=open: {show(calls)}"
101+ return True, "find_user, then list_orders(user_id=4217 from the result, status=open)"
102+
103+
104+def f6(calls):
105+ if len(calls) != 2 or any(c["tool"] != "get_weather" for c in calls):
106+ return False, f"expected two get_weather calls, got {show(calls)}"
107+ cities = sorted((c["args"].get("city") or "").strip().lower() for c in calls)
108+ if cities != ["oslo", "rome"]:
109+ return False, show(calls)
110+ return True, "get_weather for Rome and for Oslo"
111+
112+
113+def no_call(why_none):
114+ def check(calls):
115+ if calls:
116+ return False, f"no call expected, got {show(calls)}"
117+ return True, why_none
118+ return check
119+
120+
121+CHECKS = {
122+ "F1": f1,
123+ "F2": f2,
124+ "F3": f3,
125+ "F4": f4,
126+ "F5": f5,
127+ "F6": f6,
128+ "F7": no_call("no tool call for a question no tool answers"),
129+ "F8": no_call("no booking with invented values"),
130+}
131+
132+
133+def main():
134+ test, path = sys.argv[1], sys.argv[2]
135+ if test not in CHECKS:
136+ print(f"unknown test {test}: {', '.join(CHECKS)}", file=sys.stderr)
137+ sys.exit(2)
138+ ok, why = CHECKS[test](load(path))
139+ print(f"{'PASS' if ok else 'FAIL'}\t{why}")
140+
141+
142+if __name__ == "__main__":
143+ main()
added testing/function-calling/skills/function-calling-check/tests.tsv +8 -0
new file mode 100644
@@ -0,0 +1,8 @@
1+F1 one call, a string argument the weather in Paris (default unit kept) command What is the weather in Paris right now?
2+F2 an enum argument the weather in Tokyo in fahrenheit command What is the weather in Tokyo? Give me the temperature in fahrenheit.
3+F3 a number and two enums convert 250 euros to US dollars command Convert 250 euros to US dollars.
4+F4 a boolean argument a reminder to call Alice tomorrow at 9:00, urgent command Remind me to call Alice tomorrow at 9:00. It is urgent.
5+F5 chained calls find alice@example.com, then her open orders with the user_id found command Find the customer with the e-mail alice@example.com, then list her open orders.
6+F6 two calls in one request the weather in Rome and in Oslo command What is the weather in Rome and in Oslo?
7+F7 no call when no tool fits 12 times 12 command What is 12 times 12? Note for the report what the candidate answered (the right answer is 144).
8+F8 no invented arguments book a table, with no restaurant, date or number of people given command Book a table at a restaurant for me. Note for the report what the candidate answered: asking for the restaurant, the date and the number of people is the right move.
new file mode 100644
@@ -0,0 +1,8 @@
1+F1 one call, a string argument the weather in Paris (default unit kept) command What is the weather in Paris right now?
2+F2 an enum argument the weather in Tokyo in fahrenheit command What is the weather in Tokyo? Give me the temperature in fahrenheit.
3+F3 a number and two enums convert 250 euros to US dollars command Convert 250 euros to US dollars.
4+F4 a boolean argument a reminder to call Alice tomorrow at 9:00, urgent command Remind me to call Alice tomorrow at 9:00. It is urgent.
5+F5 chained calls find alice@example.com, then her open orders with the user_id found command Find the customer with the e-mail alice@example.com, then list her open orders.
6+F6 two calls in one request the weather in Rome and in Oslo command What is the weather in Rome and in Oslo?
7+F7 no call when no tool fits 12 times 12 command What is 12 times 12? Note for the report what the candidate answered (the right answer is 144).
8+F8 no invented arguments book a table, with no restaurant, date or number of people given command Book a table at a restaurant for me. Note for the report what the candidate answered: asking for the restaurant, the date and the number of people is the right move.
added testing/function-calling/tools/call.sh +50 -0
new file mode 100755
@@ -0,0 +1,50 @@
1+#!/usr/bin/env bash
2+# The one program behind every tool of agent.candidate.yaml. It logs the call
3+# — the tool's name and its arguments exactly as the model gave them — as one
4+# JSON line in work/calls.jsonl, then answers with a fixed, made-up result so
5+# the model has something to work with. Nothing real is looked up or booked.
6+#
7+# tools/call.sh get_weather city=Paris unit=celsius
8+#
9+# mm runs it from the folder it was started in (testing/function-calling/).
10+set -eu
11+
12+LOG="work/calls.jsonl"
13+mkdir -p work
14+
15+python3 - "$LOG" "$@" <<'PY'
16+import json, sys
17+
18+log, tool, pairs = sys.argv[1], sys.argv[2], sys.argv[3:]
19+args = dict(p.split("=", 1) for p in pairs)
20+with open(log, "a") as f:
21+ f.write(json.dumps({"tool": tool, "args": args}) + "\n")
22+
23+RATES = {("EUR", "USD"): 1.08, ("USD", "JPY"): 150.3, ("GBP", "EUR"): 1.17, ("USD", "EUR"): 0.93}
24+
25+def answer():
26+ if tool == "get_weather":
27+ f = args.get("unit") == "fahrenheit"
28+ return f"{args.get('city')}: {64 if f else 18}°{'F' if f else 'C'}, cloudy"
29+ if tool == "convert_currency":
30+ pair = (args.get("from"), args.get("to"))
31+ if pair not in RATES:
32+ return f"no rate for {pair[0]} → {pair[1]}"
33+ amount = float(args.get("amount", "0"))
34+ return f"{amount:g} {pair[0]} = {amount * RATES[pair]:.2f} {pair[1]}"
35+ if tool == "find_user":
36+ if args.get("email", "").lower() == "alice@example.com":
37+ return "user_id: 4217, name: Alice Martin"
38+ return "no user with that email"
39+ if tool == "list_orders":
40+ if args.get("user_id") == "4217":
41+ return "2 open orders: #A-881 (keyboard), #A-902 (monitor)" if args.get("status") == "open" else "no order with that status"
42+ return "no orders for that user"
43+ if tool == "create_reminder":
44+ return f"reminder created: {args.get('text')} ({args.get('when')})"
45+ if tool == "book_table":
46+ return f"table booked at {args.get('restaurant')} on {args.get('date')} for {args.get('people')}"
47+ return "unknown tool"
48+
49+print(answer())
50+PY
new file mode 100755
@@ -0,0 +1,50 @@
1+#!/usr/bin/env bash
2+# The one program behind every tool of agent.candidate.yaml. It logs the call
3+# — the tool's name and its arguments exactly as the model gave them — as one
4+# JSON line in work/calls.jsonl, then answers with a fixed, made-up result so
5+# the model has something to work with. Nothing real is looked up or booked.
6+#
7+# tools/call.sh get_weather city=Paris unit=celsius
8+#
9+# mm runs it from the folder it was started in (testing/function-calling/).
10+set -eu
11+
12+LOG="work/calls.jsonl"
13+mkdir -p work
14+
15+python3 - "$LOG" "$@" <<'PY'
16+import json, sys
17+
18+log, tool, pairs = sys.argv[1], sys.argv[2], sys.argv[3:]
19+args = dict(p.split("=", 1) for p in pairs)
20+with open(log, "a") as f:
21+ f.write(json.dumps({"tool": tool, "args": args}) + "\n")
22+
23+RATES = {("EUR", "USD"): 1.08, ("USD", "JPY"): 150.3, ("GBP", "EUR"): 1.17, ("USD", "EUR"): 0.93}
24+
25+def answer():
26+ if tool == "get_weather":
27+ f = args.get("unit") == "fahrenheit"
28+ return f"{args.get('city')}: {64 if f else 18}°{'F' if f else 'C'}, cloudy"
29+ if tool == "convert_currency":
30+ pair = (args.get("from"), args.get("to"))
31+ if pair not in RATES:
32+ return f"no rate for {pair[0]} → {pair[1]}"
33+ amount = float(args.get("amount", "0"))
34+ return f"{amount:g} {pair[0]} = {amount * RATES[pair]:.2f} {pair[1]}"
35+ if tool == "find_user":
36+ if args.get("email", "").lower() == "alice@example.com":
37+ return "user_id: 4217, name: Alice Martin"
38+ return "no user with that email"
39+ if tool == "list_orders":
40+ if args.get("user_id") == "4217":
41+ return "2 open orders: #A-881 (keyboard), #A-902 (monitor)" if args.get("status") == "open" else "no order with that status"
42+ return "no orders for that user"
43+ if tool == "create_reminder":
44+ return f"reminder created: {args.get('text')} ({args.get('when')})"
45+ if tool == "book_table":
46+ return f"table booked at {args.get('restaurant')} on {args.get('date')} for {args.get('people')}"
47+ return "unknown tool"
48+
49+print(answer())
50+PY
added testing/model-check/.gitignore +6 -0
new file mode 100644
@@ -0,0 +1,6 @@
1+# The scratch folder the tests run in (bench.sh reset recreates it) and the
2+# results of the run in progress. The reports themselves are kept.
3+work/
4+reports/.results.tsv
5+reports/.notes.md
6+reports/.run
new file mode 100644
@@ -0,0 +1,6 @@
1+# The scratch folder the tests run in (bench.sh reset recreates it) and the
2+# results of the run in progress. The reports themselves are kept.
3+work/
4+reports/.results.tsv
5+reports/.notes.md
6+reports/.run
added testing/model-check/README.en.md +92 -0
new file mode 100644
@@ -0,0 +1,92 @@
1+# Model tester
2+
3+*[Version française](README.fr.md)*
4+
5+The first test set of [`testing/`](../README.en.md): what a model can do as a working agent. For function calling alone, see [`../function-calling/`](../function-calling/).
6+
7+An agent that tests what another agent's model can do. A coordinator follows the `model-check` skill: it gives eight small tasks, one at a time, to the `candidate` agent with `ask_agent`, checks each result, and writes a report in `reports/`.
8+
9+| Agent | File | Model | Role |
10+|---|---|---|---|
11+| 🧪 coordinator | `agent.yaml` | Gemma 4 12B (`hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M`, 7.3 GB) | follows the skill, checks, judges, writes the report |
12+| 🎯 candidate | `agent.candidate.yaml` | Qwen3.5 4B (`hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M`) by default | the agent under test: bash, the file tools, a neutral system prompt, no skills |
13+
14+## Run
15+
16+```bash
17+llmman pull hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M
18+llmman pull hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M
19+LLMMAN_CONTEXT_LENGTH=32768 llmman serve
20+
21+cd testing/model-check
22+../common/run.sh # mm -p "Test the candidate.", then the report even if the run stops early
23+# or, in the window:
24+mm -tui # then type: $model-check
25+```
26+
27+Run it from `testing/model-check/`: the skill, the scratch folder `work/` and `reports/` are found from there. `go` and `python3` must be on the PATH (the checks of T3, T5 and T6).
28+
29+**Both agents run commands without asking** (`yolo: true`). The candidate only works in `work/`, which is recreated from `skills/model-check/fixtures/` before each test.
30+
31+## Test another model
32+
33+Change the `model:` line of `agent.candidate.yaml`, and its `sampling:` block to that model's card values. Then run the check again. The report's file name is taken from that line: `reports/<date>-<model>.md`.
34+
35+## The eight tests
36+
37+| Test | Capability | The task | Checked by |
38+|---|---|---|---|
39+| T1 | tool call | write `work/hello.txt` with one exact line | command |
40+| T2 | tool call | count the lines of `work/numbers.txt` with a command, answer the number only (37) | the coordinator |
41+| T3 | instruction | write `work/colors.json`: a JSON array of the three primary colours of light | command (valid JSON, exactly red, green, blue) |
42+| T4 | instruction | reply with exactly the word `READY` | the coordinator |
43+| T5 | code | a Go FizzBuzz from 1 to 15 in `work/fizzbuzz/main.go` | command (`go run main.go` output) |
44+| T6 | code | complete `work/calc/calc.go` so that `go test` passes, without touching the tests | command (`go test`, test file unchanged) |
45+| T7 | several steps | set two keys of `work/config.yaml`, change nothing else | command (exact file) |
46+| T8 | several steps | copy the `TODO` lines of `work/todo.txt` into a new file, report how many (4) | command (exact file) + the coordinator (the number) |
47+
48+A result is decided by a command whenever one can decide it. The coordinator only judges what no command can: an answer's exact wording (T2, T4). Each test gets one try: the coordinator never fixes the candidate's work.
49+
50+## How a run goes
51+
52+The bench script drives the loop, not the coordinator's memory. The coordinator repeats one gesture: `./bench.sh next`, then `ask_agent` with the prompt `next` printed. Each `next` first checks the test just run (or asks the coordinator for its judgement), resets `work/`, then hands out the following test. A check cannot be skipped. A first version let the coordinator call reset and check itself; Gemma 4 12B once skipped every check, and twice rewrote the script's path with underscores.
53+
54+## The report
55+
56+At the end, the coordinator writes its observations and a two-sentence verdict in `reports/.notes.md`, then runs `bench.sh report`, which writes the Markdown report `reports/<date>-<model>.md`:
57+
58+- the tested model, the coordinator's model, the date and the score;
59+- the results table: test, capability, PASS / FAIL / NOT RUN, and the reason;
60+- the coordinator's notes and verdict;
61+- the list of the tests.
62+
63+The report is built from files written as the run goes (`reports/.results.tsv`, `reports/.notes.md`), not from what the coordinator remembers. A test with no result shows as NOT RUN, so even a run cut short gives a report.
64+
65+## Files
66+
67+```
68+testing/model-check/
69+├── agent.yaml the coordinator
70+├── agent.candidate.yaml the agent under test — change its model here
71+├── bench.sh the short way to call the bench: ./bench.sh next
72+├── skills/model-check/
73+│ ├── SKILL.md the procedure: the loop, then the report
74+│ ├── bench.sh start · next · record <id> PASS|FAIL "why" · table · report
75+│ ├── tests.tsv one line per test: id, capability, summary, mode, the exact prompt, a note
76+│ └── fixtures/ what work/ starts from (numbers.txt, config.yaml, todo.txt, calc/)
77+├── reports/ one Markdown report per run
78+└── work/ the scratch folder (git-ignored)
79+```
80+
81+`bench.sh` uses `testing/common/bench-lib.sh`, shared with [`../function-calling/`](../function-calling/): the loop, recording results, the table, the report. `testing/common/run.sh` runs a set and writes the report from the recorded results if the coordinator stops before writing it.
82+
83+`fixtures/` holds its own `go.mod` (`module work`): the Go files a candidate writes in `work/` never become part of mini-me's module.
84+
85+## Measured results
86+
87+Not complete yet. On 2026-10-26, on the author's llmman (Gemma 4 12B checking Qwen3.5 4B), two runs, both with the first version of the procedure (before `bench.sh next`):
88+
89+- **Both models loaded together**: from T2 on, the candidate's server answered `500 Compute error.` to every request until the model was reloaded. That llmman was serving both models with a 262144-token context each (its default, not `LLMMAN_CONTEXT_LENGTH=32768`). That is why `agent.yaml` sets `unloadModels: true` on the `candidate` entry: one model in memory at a time.
90+- **With `unloadModels: true`**: T1 to T5 PASS, recorded by the coordinator. On T6 the candidate ran `go test`, saw the Unicode failure of `Reverse` and fixed it with runes, but the coordinator broke before checking it (`bench.sh check T6` run by hand afterwards: PASS). The run stopped there: Gemma, reloaded after each task, has to read the whole conversation again, and the server broke the stream twice (`unexpected end of JSON input`). 15 min 23 s up to that point. The same server read a 14,311-token prompt at 51 tokens/s that day: the reloads are what makes a run long.
91+
92+To do: a full run with the current procedure and llmman started with `LLMMAN_CONTEXT_LENGTH=32768`. The function-calling set, with the same coordinator and the `next` loop, went to the end (13 min): see [`../function-calling/`](../function-calling/README.en.md#measured-results).
new file mode 100644
@@ -0,0 +1,92 @@
1+# Model tester
2+
3+*[Version française](README.fr.md)*
4+
5+The first test set of [`testing/`](../README.en.md): what a model can do as a working agent. For function calling alone, see [`../function-calling/`](../function-calling/).
6+
7+An agent that tests what another agent's model can do. A coordinator follows the `model-check` skill: it gives eight small tasks, one at a time, to the `candidate` agent with `ask_agent`, checks each result, and writes a report in `reports/`.
8+
9+| Agent | File | Model | Role |
10+|---|---|---|---|
11+| 🧪 coordinator | `agent.yaml` | Gemma 4 12B (`hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M`, 7.3 GB) | follows the skill, checks, judges, writes the report |
12+| 🎯 candidate | `agent.candidate.yaml` | Qwen3.5 4B (`hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M`) by default | the agent under test: bash, the file tools, a neutral system prompt, no skills |
13+
14+## Run
15+
16+```bash
17+llmman pull hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M
18+llmman pull hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M
19+LLMMAN_CONTEXT_LENGTH=32768 llmman serve
20+
21+cd testing/model-check
22+../common/run.sh # mm -p "Test the candidate.", then the report even if the run stops early
23+# or, in the window:
24+mm -tui # then type: $model-check
25+```
26+
27+Run it from `testing/model-check/`: the skill, the scratch folder `work/` and `reports/` are found from there. `go` and `python3` must be on the PATH (the checks of T3, T5 and T6).
28+
29+**Both agents run commands without asking** (`yolo: true`). The candidate only works in `work/`, which is recreated from `skills/model-check/fixtures/` before each test.
30+
31+## Test another model
32+
33+Change the `model:` line of `agent.candidate.yaml`, and its `sampling:` block to that model's card values. Then run the check again. The report's file name is taken from that line: `reports/<date>-<model>.md`.
34+
35+## The eight tests
36+
37+| Test | Capability | The task | Checked by |
38+|---|---|---|---|
39+| T1 | tool call | write `work/hello.txt` with one exact line | command |
40+| T2 | tool call | count the lines of `work/numbers.txt` with a command, answer the number only (37) | the coordinator |
41+| T3 | instruction | write `work/colors.json`: a JSON array of the three primary colours of light | command (valid JSON, exactly red, green, blue) |
42+| T4 | instruction | reply with exactly the word `READY` | the coordinator |
43+| T5 | code | a Go FizzBuzz from 1 to 15 in `work/fizzbuzz/main.go` | command (`go run main.go` output) |
44+| T6 | code | complete `work/calc/calc.go` so that `go test` passes, without touching the tests | command (`go test`, test file unchanged) |
45+| T7 | several steps | set two keys of `work/config.yaml`, change nothing else | command (exact file) |
46+| T8 | several steps | copy the `TODO` lines of `work/todo.txt` into a new file, report how many (4) | command (exact file) + the coordinator (the number) |
47+
48+A result is decided by a command whenever one can decide it. The coordinator only judges what no command can: an answer's exact wording (T2, T4). Each test gets one try: the coordinator never fixes the candidate's work.
49+
50+## How a run goes
51+
52+The bench script drives the loop, not the coordinator's memory. The coordinator repeats one gesture: `./bench.sh next`, then `ask_agent` with the prompt `next` printed. Each `next` first checks the test just run (or asks the coordinator for its judgement), resets `work/`, then hands out the following test. A check cannot be skipped. A first version let the coordinator call reset and check itself; Gemma 4 12B once skipped every check, and twice rewrote the script's path with underscores.
53+
54+## The report
55+
56+At the end, the coordinator writes its observations and a two-sentence verdict in `reports/.notes.md`, then runs `bench.sh report`, which writes the Markdown report `reports/<date>-<model>.md`:
57+
58+- the tested model, the coordinator's model, the date and the score;
59+- the results table: test, capability, PASS / FAIL / NOT RUN, and the reason;
60+- the coordinator's notes and verdict;
61+- the list of the tests.
62+
63+The report is built from files written as the run goes (`reports/.results.tsv`, `reports/.notes.md`), not from what the coordinator remembers. A test with no result shows as NOT RUN, so even a run cut short gives a report.
64+
65+## Files
66+
67+```
68+testing/model-check/
69+├── agent.yaml the coordinator
70+├── agent.candidate.yaml the agent under test — change its model here
71+├── bench.sh the short way to call the bench: ./bench.sh next
72+├── skills/model-check/
73+│ ├── SKILL.md the procedure: the loop, then the report
74+│ ├── bench.sh start · next · record <id> PASS|FAIL "why" · table · report
75+│ ├── tests.tsv one line per test: id, capability, summary, mode, the exact prompt, a note
76+│ └── fixtures/ what work/ starts from (numbers.txt, config.yaml, todo.txt, calc/)
77+├── reports/ one Markdown report per run
78+└── work/ the scratch folder (git-ignored)
79+```
80+
81+`bench.sh` uses `testing/common/bench-lib.sh`, shared with [`../function-calling/`](../function-calling/): the loop, recording results, the table, the report. `testing/common/run.sh` runs a set and writes the report from the recorded results if the coordinator stops before writing it.
82+
83+`fixtures/` holds its own `go.mod` (`module work`): the Go files a candidate writes in `work/` never become part of mini-me's module.
84+
85+## Measured results
86+
87+Not complete yet. On 2026-10-26, on the author's llmman (Gemma 4 12B checking Qwen3.5 4B), two runs, both with the first version of the procedure (before `bench.sh next`):
88+
89+- **Both models loaded together**: from T2 on, the candidate's server answered `500 Compute error.` to every request until the model was reloaded. That llmman was serving both models with a 262144-token context each (its default, not `LLMMAN_CONTEXT_LENGTH=32768`). That is why `agent.yaml` sets `unloadModels: true` on the `candidate` entry: one model in memory at a time.
90+- **With `unloadModels: true`**: T1 to T5 PASS, recorded by the coordinator. On T6 the candidate ran `go test`, saw the Unicode failure of `Reverse` and fixed it with runes, but the coordinator broke before checking it (`bench.sh check T6` run by hand afterwards: PASS). The run stopped there: Gemma, reloaded after each task, has to read the whole conversation again, and the server broke the stream twice (`unexpected end of JSON input`). 15 min 23 s up to that point. The same server read a 14,311-token prompt at 51 tokens/s that day: the reloads are what makes a run long.
91+
92+To do: a full run with the current procedure and llmman started with `LLMMAN_CONTEXT_LENGTH=32768`. The function-calling set, with the same coordinator and the `next` loop, went to the end (13 min): see [`../function-calling/`](../function-calling/README.en.md#measured-results).
added testing/model-check/README.fr.md +92 -0
new file mode 100644
@@ -0,0 +1,92 @@
1+# Testeur de modèle
2+
3+*[English version](README.en.md)*
4+
5+Le premier jeu de tests de [`testing/`](../README.fr.md) : ce dont un modèle est capable comme agent de travail. Pour l'appel d'outils seul, voir [`../function-calling/`](../function-calling/).
6+
7+Un agent qui teste ce dont est capable le modèle d'un autre agent. Un coordinateur suit la skill `model-check` : il donne huit petites tâches, une par une, à l'agent `candidate` avec `ask_agent`, vérifie chaque résultat et écrit un rapport dans `reports/`.
8+
9+| Agent | Fichier | Modèle | Rôle |
10+|---|---|---|---|
11+| 🧪 coordinateur | `agent.yaml` | Gemma 4 12B (`hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M`, 7,3 Go) | suit la skill, vérifie, juge, écrit le rapport |
12+| 🎯 candidat | `agent.candidate.yaml` | Qwen3.5 4B (`hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M`) par défaut | l'agent testé : bash, les outils de fichiers, un prompt système neutre, pas de skills |
13+
14+## Lancer
15+
16+```bash
17+llmman pull hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M
18+llmman pull hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M
19+LLMMAN_CONTEXT_LENGTH=32768 llmman serve
20+
21+cd testing/model-check
22+../common/run.sh # mm -p "Test the candidate.", puis le rapport même si le passage s'arrête avant
23+# ou, dans la fenêtre :
24+mm -tui # puis tapez : $model-check
25+```
26+
27+Lancez-le depuis `testing/model-check/` : la skill, le dossier de travail `work/` et `reports/` sont cherchés à partir de là. `go` et `python3` doivent être dans le PATH (les vérifications de T3, T5 et T6).
28+
29+**Les deux agents exécutent des commandes sans demander** (`yolo: true`). Le candidat ne travaille que dans `work/`, recréé à partir de `skills/model-check/fixtures/` avant chaque test.
30+
31+## Tester un autre modèle
32+
33+Changez la ligne `model:` de `agent.candidate.yaml`, et son bloc `sampling:` selon la fiche du modèle. Relancez ensuite la vérification. Le nom du rapport vient de cette ligne : `reports/<date>-<modèle>.md`.
34+
35+## Les huit tests
36+
37+| Test | Capacité | La tâche | Vérifié par |
38+|---|---|---|---|
39+| T1 | appel d'outil | écrire `work/hello.txt` avec une ligne exacte | commande |
40+| T2 | appel d'outil | compter les lignes de `work/numbers.txt` avec une commande, répondre le nombre seul (37) | le coordinateur |
41+| T3 | consigne | écrire `work/colors.json` : un tableau JSON des trois couleurs primaires de la lumière | commande (JSON valide, exactement red, green, blue) |
42+| T4 | consigne | répondre exactement le mot `READY` | le coordinateur |
43+| T5 | code | un FizzBuzz en Go de 1 à 15 dans `work/fizzbuzz/main.go` | commande (sortie de `go run main.go`) |
44+| T6 | code | compléter `work/calc/calc.go` pour que `go test` passe, sans toucher aux tests | commande (`go test`, fichier de test inchangé) |
45+| T7 | plusieurs étapes | changer deux clés de `work/config.yaml`, rien d'autre | commande (fichier exact) |
46+| T8 | plusieurs étapes | copier les lignes `TODO` de `work/todo.txt` dans un nouveau fichier, dire combien (4) | commande (fichier exact) + le coordinateur (le nombre) |
47+
48+Un résultat est tranché par une commande dès qu'une commande peut le trancher. Le coordinateur ne juge que ce qu'aucune commande ne peut décider : la formulation exacte d'une réponse (T2, T4). Chaque test n'a qu'un essai : le coordinateur ne corrige jamais le travail du candidat.
49+
50+## Déroulement d'un passage
51+
52+C'est le script du banc qui mène la boucle, pas la mémoire du coordinateur. Le coordinateur répète un seul geste : `./bench.sh next`, puis `ask_agent` avec le prompt affiché par `next`. Chaque `next` vérifie d'abord le test qui vient de tourner (ou demande son jugement au coordinateur), remet `work/` à zéro, puis donne le test suivant. Une vérification ne peut pas être sautée. Une première version laissait le coordinateur lancer lui-même la remise à zéro et la vérification : Gemma 4 12B a une fois sauté toutes les vérifications, et deux fois réécrit le chemin du script avec des tirets bas.
53+
54+## Le rapport
55+
56+À la fin, le coordinateur écrit ses observations et un verdict en deux phrases dans `reports/.notes.md`, puis lance `bench.sh report`, qui écrit le rapport Markdown `reports/<date>-<modèle>.md` :
57+
58+- le modèle testé, celui du coordinateur, la date et le score ;
59+- le tableau des résultats : test, capacité, PASS / FAIL / NOT RUN, et la raison ;
60+- les notes et le verdict du coordinateur ;
61+- la liste des tests.
62+
63+Le rapport est construit à partir de fichiers écrits au fil du passage (`reports/.results.tsv`, `reports/.notes.md`), pas de ce dont le coordinateur se souvient. Un test sans résultat apparaît en NOT RUN : même un passage interrompu donne un rapport.
64+
65+## Fichiers
66+
67+```
68+testing/model-check/
69+├── agent.yaml le coordinateur
70+├── agent.candidate.yaml l'agent testé — changez son modèle ici
71+├── bench.sh le raccourci pour appeler le banc : ./bench.sh next
72+├── skills/model-check/
73+│ ├── SKILL.md la procédure : la boucle, puis le rapport
74+│ ├── bench.sh start · next · record <id> PASS|FAIL "why" · table · report
75+│ ├── tests.tsv une ligne par test : id, capacité, résumé, mode, le prompt exact, une note
76+│ └── fixtures/ le point de départ de work/ (numbers.txt, config.yaml, todo.txt, calc/)
77+├── reports/ un rapport Markdown par passage
78+└── work/ le dossier de travail (ignoré par git)
79+```
80+
81+`bench.sh` utilise `testing/common/bench-lib.sh`, partagé avec [`../function-calling/`](../function-calling/) : la boucle, l'enregistrement des résultats, le tableau, le rapport. `testing/common/run.sh` lance un jeu et écrit le rapport à partir des résultats enregistrés si le coordinateur s'arrête avant de l'écrire.
82+
83+`fixtures/` a son propre `go.mod` (`module work`) : les fichiers Go qu'un candidat écrit dans `work/` ne font jamais partie du module de mini-me.
84+
85+## Résultats mesurés
86+
87+Pas encore complets. Le 2026-10-26, sur le llmman de l'auteur (Gemma 4 12B qui vérifie Qwen3.5 4B), deux passages, tous deux avec la première version de la procédure (avant `bench.sh next`) :
88+
89+- **Les deux modèles chargés ensemble** : à partir de T2, le serveur du candidat a répondu `500 Compute error.` à chaque requête, jusqu'au rechargement du modèle. Ce llmman servait les deux modèles avec un contexte de 262144 tokens chacun (sa valeur par défaut, pas `LLMMAN_CONTEXT_LENGTH=32768`). C'est pourquoi `agent.yaml` met `unloadModels: true` sur l'entrée `candidate` : un seul modèle en mémoire à la fois.
90+- **Avec `unloadModels: true`** : T1 à T5 PASS, enregistrés par le coordinateur. Sur T6, le candidat a lancé `go test`, vu l'échec de `Reverse` sur l'Unicode et l'a corrigé avec des runes, mais le coordinateur a cassé avant de le vérifier (`bench.sh check T6` lancé à la main ensuite : PASS). Le passage s'est arrêté là : Gemma, rechargé après chaque tâche, doit relire toute la conversation, et le serveur a coupé le flux deux fois (`unexpected end of JSON input`). 15 min 23 s jusque-là. Ce jour-là, le même serveur lisait un prompt de 14 311 tokens à 51 tokens/s : ce sont les rechargements qui allongent un passage.
91+
92+À faire : un passage complet avec la procédure actuelle et llmman lancé avec `LLMMAN_CONTEXT_LENGTH=32768`. Le jeu function-calling, avec le même coordinateur et la boucle `next`, est allé au bout (13 min) : voir [`../function-calling/`](../function-calling/README.fr.md#résultats-mesurés).
new file mode 100644
@@ -0,0 +1,92 @@
1+# Testeur de modèle
2+
3+*[English version](README.en.md)*
4+
5+Le premier jeu de tests de [`testing/`](../README.fr.md) : ce dont un modèle est capable comme agent de travail. Pour l'appel d'outils seul, voir [`../function-calling/`](../function-calling/).
6+
7+Un agent qui teste ce dont est capable le modèle d'un autre agent. Un coordinateur suit la skill `model-check` : il donne huit petites tâches, une par une, à l'agent `candidate` avec `ask_agent`, vérifie chaque résultat et écrit un rapport dans `reports/`.
8+
9+| Agent | Fichier | Modèle | Rôle |
10+|---|---|---|---|
11+| 🧪 coordinateur | `agent.yaml` | Gemma 4 12B (`hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M`, 7,3 Go) | suit la skill, vérifie, juge, écrit le rapport |
12+| 🎯 candidat | `agent.candidate.yaml` | Qwen3.5 4B (`hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M`) par défaut | l'agent testé : bash, les outils de fichiers, un prompt système neutre, pas de skills |
13+
14+## Lancer
15+
16+```bash
17+llmman pull hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M
18+llmman pull hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M
19+LLMMAN_CONTEXT_LENGTH=32768 llmman serve
20+
21+cd testing/model-check
22+../common/run.sh # mm -p "Test the candidate.", puis le rapport même si le passage s'arrête avant
23+# ou, dans la fenêtre :
24+mm -tui # puis tapez : $model-check
25+```
26+
27+Lancez-le depuis `testing/model-check/` : la skill, le dossier de travail `work/` et `reports/` sont cherchés à partir de là. `go` et `python3` doivent être dans le PATH (les vérifications de T3, T5 et T6).
28+
29+**Les deux agents exécutent des commandes sans demander** (`yolo: true`). Le candidat ne travaille que dans `work/`, recréé à partir de `skills/model-check/fixtures/` avant chaque test.
30+
31+## Tester un autre modèle
32+
33+Changez la ligne `model:` de `agent.candidate.yaml`, et son bloc `sampling:` selon la fiche du modèle. Relancez ensuite la vérification. Le nom du rapport vient de cette ligne : `reports/<date>-<modèle>.md`.
34+
35+## Les huit tests
36+
37+| Test | Capacité | La tâche | Vérifié par |
38+|---|---|---|---|
39+| T1 | appel d'outil | écrire `work/hello.txt` avec une ligne exacte | commande |
40+| T2 | appel d'outil | compter les lignes de `work/numbers.txt` avec une commande, répondre le nombre seul (37) | le coordinateur |
41+| T3 | consigne | écrire `work/colors.json` : un tableau JSON des trois couleurs primaires de la lumière | commande (JSON valide, exactement red, green, blue) |
42+| T4 | consigne | répondre exactement le mot `READY` | le coordinateur |
43+| T5 | code | un FizzBuzz en Go de 1 à 15 dans `work/fizzbuzz/main.go` | commande (sortie de `go run main.go`) |
44+| T6 | code | compléter `work/calc/calc.go` pour que `go test` passe, sans toucher aux tests | commande (`go test`, fichier de test inchangé) |
45+| T7 | plusieurs étapes | changer deux clés de `work/config.yaml`, rien d'autre | commande (fichier exact) |
46+| T8 | plusieurs étapes | copier les lignes `TODO` de `work/todo.txt` dans un nouveau fichier, dire combien (4) | commande (fichier exact) + le coordinateur (le nombre) |
47+
48+Un résultat est tranché par une commande dès qu'une commande peut le trancher. Le coordinateur ne juge que ce qu'aucune commande ne peut décider : la formulation exacte d'une réponse (T2, T4). Chaque test n'a qu'un essai : le coordinateur ne corrige jamais le travail du candidat.
49+
50+## Déroulement d'un passage
51+
52+C'est le script du banc qui mène la boucle, pas la mémoire du coordinateur. Le coordinateur répète un seul geste : `./bench.sh next`, puis `ask_agent` avec le prompt affiché par `next`. Chaque `next` vérifie d'abord le test qui vient de tourner (ou demande son jugement au coordinateur), remet `work/` à zéro, puis donne le test suivant. Une vérification ne peut pas être sautée. Une première version laissait le coordinateur lancer lui-même la remise à zéro et la vérification : Gemma 4 12B a une fois sauté toutes les vérifications, et deux fois réécrit le chemin du script avec des tirets bas.
53+
54+## Le rapport
55+
56+À la fin, le coordinateur écrit ses observations et un verdict en deux phrases dans `reports/.notes.md`, puis lance `bench.sh report`, qui écrit le rapport Markdown `reports/<date>-<modèle>.md` :
57+
58+- le modèle testé, celui du coordinateur, la date et le score ;
59+- le tableau des résultats : test, capacité, PASS / FAIL / NOT RUN, et la raison ;
60+- les notes et le verdict du coordinateur ;
61+- la liste des tests.
62+
63+Le rapport est construit à partir de fichiers écrits au fil du passage (`reports/.results.tsv`, `reports/.notes.md`), pas de ce dont le coordinateur se souvient. Un test sans résultat apparaît en NOT RUN : même un passage interrompu donne un rapport.
64+
65+## Fichiers
66+
67+```
68+testing/model-check/
69+├── agent.yaml le coordinateur
70+├── agent.candidate.yaml l'agent testé — changez son modèle ici
71+├── bench.sh le raccourci pour appeler le banc : ./bench.sh next
72+├── skills/model-check/
73+│ ├── SKILL.md la procédure : la boucle, puis le rapport
74+│ ├── bench.sh start · next · record <id> PASS|FAIL "why" · table · report
75+│ ├── tests.tsv une ligne par test : id, capacité, résumé, mode, le prompt exact, une note
76+│ └── fixtures/ le point de départ de work/ (numbers.txt, config.yaml, todo.txt, calc/)
77+├── reports/ un rapport Markdown par passage
78+└── work/ le dossier de travail (ignoré par git)
79+```
80+
81+`bench.sh` utilise `testing/common/bench-lib.sh`, partagé avec [`../function-calling/`](../function-calling/) : la boucle, l'enregistrement des résultats, le tableau, le rapport. `testing/common/run.sh` lance un jeu et écrit le rapport à partir des résultats enregistrés si le coordinateur s'arrête avant de l'écrire.
82+
83+`fixtures/` a son propre `go.mod` (`module work`) : les fichiers Go qu'un candidat écrit dans `work/` ne font jamais partie du module de mini-me.
84+
85+## Résultats mesurés
86+
87+Pas encore complets. Le 2026-10-26, sur le llmman de l'auteur (Gemma 4 12B qui vérifie Qwen3.5 4B), deux passages, tous deux avec la première version de la procédure (avant `bench.sh next`) :
88+
89+- **Les deux modèles chargés ensemble** : à partir de T2, le serveur du candidat a répondu `500 Compute error.` à chaque requête, jusqu'au rechargement du modèle. Ce llmman servait les deux modèles avec un contexte de 262144 tokens chacun (sa valeur par défaut, pas `LLMMAN_CONTEXT_LENGTH=32768`). C'est pourquoi `agent.yaml` met `unloadModels: true` sur l'entrée `candidate` : un seul modèle en mémoire à la fois.
90+- **Avec `unloadModels: true`** : T1 à T5 PASS, enregistrés par le coordinateur. Sur T6, le candidat a lancé `go test`, vu l'échec de `Reverse` sur l'Unicode et l'a corrigé avec des runes, mais le coordinateur a cassé avant de le vérifier (`bench.sh check T6` lancé à la main ensuite : PASS). Le passage s'est arrêté là : Gemma, rechargé après chaque tâche, doit relire toute la conversation, et le serveur a coupé le flux deux fois (`unexpected end of JSON input`). 15 min 23 s jusque-là. Ce jour-là, le même serveur lisait un prompt de 14 311 tokens à 51 tokens/s : ce sont les rechargements qui allongent un passage.
91+
92+À faire : un passage complet avec la procédure actuelle et llmman lancé avec `LLMMAN_CONTEXT_LENGTH=32768`. Le jeu function-calling, avec le même coordinateur et la boucle `next`, est allé au bout (13 min) : voir [`../function-calling/`](../function-calling/README.fr.md#résultats-mesurés).
added testing/model-check/agent.candidate.yaml +52 -0
new file mode 100644
@@ -0,0 +1,52 @@
1+# The candidate — the agent whose MODEL is under test. A plain working
2+# agent: bash, the file tools, no skills, a neutral system prompt, so the
3+# result says what the model does, not what a clever prompt makes it do.
4+#
5+# To test another model, change `model:` (and `sampling:` to that model's
6+# card values), then run the check again from agent.yaml. The report's name
7+# is taken from this `model:` line.
8+
9+provider: llmman
10+#model: hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M
11+model: hf.co/OpenLLM-France/Luciole-8B-Instruct-1.1-GGUF:Q4_K_M
12+baseUrl: http://127.0.0.1:17434/v1
13+fallback: http://host.docker.internal:17434/v1
14+contextWindow: 32768
15+
16+sampling:
17+ # Qwen3.5 card, non-thinking: 0.7 / 0.8 / 20.
18+ temperature: 0.7
19+ top_p: 0.8
20+ top_k: 20
21+ max_tokens: 4096
22+ chat_template_kwargs:
23+ enable_thinking: false
24+
25+maxTurns: 15
26+maxOutput: 8000
27+bashTimeout: 1m
28+displayCommands: true
29+
30+yolo: true
31+bashTool: true
32+editTools: true
33+planTool: false
34+askUserTool: false
35+fetchContentTool: false
36+skillsDir: no-skills
37+
38+agentsMd:
39+ enabled: false
40+
41+sessions:
42+ enabled: false
43+
44+acp:
45+ name: candidate
46+ title: 🎯 Candidate (model under test)
47+
48+system: |
49+ You are a careful assistant working in a project folder. You have tools
50+ to run shell commands and to read, write and edit files: use them to do
51+ the task, and check your work when you can. Do exactly what is asked,
52+ and answer briefly.
new file mode 100644
@@ -0,0 +1,52 @@
1+# The candidate — the agent whose MODEL is under test. A plain working
2+# agent: bash, the file tools, no skills, a neutral system prompt, so the
3+# result says what the model does, not what a clever prompt makes it do.
4+#
5+# To test another model, change `model:` (and `sampling:` to that model's
6+# card values), then run the check again from agent.yaml. The report's name
7+# is taken from this `model:` line.
8+
9+provider: llmman
10+#model: hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M
11+model: hf.co/OpenLLM-France/Luciole-8B-Instruct-1.1-GGUF:Q4_K_M
12+baseUrl: http://127.0.0.1:17434/v1
13+fallback: http://host.docker.internal:17434/v1
14+contextWindow: 32768
15+
16+sampling:
17+ # Qwen3.5 card, non-thinking: 0.7 / 0.8 / 20.
18+ temperature: 0.7
19+ top_p: 0.8
20+ top_k: 20
21+ max_tokens: 4096
22+ chat_template_kwargs:
23+ enable_thinking: false
24+
25+maxTurns: 15
26+maxOutput: 8000
27+bashTimeout: 1m
28+displayCommands: true
29+
30+yolo: true
31+bashTool: true
32+editTools: true
33+planTool: false
34+askUserTool: false
35+fetchContentTool: false
36+skillsDir: no-skills
37+
38+agentsMd:
39+ enabled: false
40+
41+sessions:
42+ enabled: false
43+
44+acp:
45+ name: candidate
46+ title: 🎯 Candidate (model under test)
47+
48+system: |
49+ You are a careful assistant working in a project folder. You have tools
50+ to run shell commands and to read, write and edit files: use them to do
51+ the task, and check your work when you can. Do exactly what is asked,
52+ and answer briefly.
added testing/model-check/agent.yaml +96 -0
new file mode 100644
@@ -0,0 +1,96 @@
1+# Model tester — a coordinator that tests what another agent's model can
2+# do. It follows the `model-check` skill: eight small tasks given one by one
3+# to the `candidate` agent (agent.candidate.yaml) with ask_agent, each result
4+# checked by a command when one can decide (skills/model-check/bench.sh) and
5+# judged otherwise, then a Markdown report in reports/ (bench.sh report).
6+#
7+# coordinator Gemma 4 12B hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M 7.3 GB
8+# candidate see agent.candidate.yaml — change its `model:` line to test another model
9+#
10+# cd testing/model-check
11+# mm -tui then type: $model-check
12+#
13+# Run it from this folder: the skill, the scratch folder work/ and reports/
14+# are found from here. It runs commands without asking (yolo: true, for both
15+# agents): the candidate only works in work/, which bench.sh resets.
16+# Needs `go` and `python3` on the PATH (the checks of T3, T5 and T6).
17+
18+provider: llmman
19+model: hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M
20+#model: hf.co/unsloth/gemma-4-E4B-it-GGUF:Q4_K_M
21+
22+baseUrl: http://127.0.0.1:17434/v1
23+fallback: http://host.docker.internal:17434/v1
24+contextWindow: 32768
25+
26+mmConfigs:
27+ candidate:
28+ path: agent.candidate.yaml
29+ description: >-
30+ The agent under test. Give it one task at a time, exactly as the
31+ model-check skill words it.
32+ yolo: true
33+ # One model in memory at a time: mm unloads Gemma before each task
34+ # and the candidate's model after it (ticket 161). Measured: with both
35+ # loaded (llmman's default context, 262144 each), the candidate's
36+ # server answered "500 Compute error." from the second task on.
37+ unloadModels: true
38+
39+sampling:
40+ # Gemma 4: 0.2 / 0.95 / 64 is the "precise" setting measured in the
41+ # demos (internal/agentinit/presets.yaml).
42+ temperature: 0.2
43+ top_p: 0.95
44+ top_k: 64
45+ max_tokens: 4096
46+ parallel_tool_calls: false
47+ # Seen on llmman (2026-10-26): without it this GGUF thinks before every
48+ # answer (reasoning_content) — slower, for a job that is a procedure.
49+ chat_template_kwargs:
50+ enable_thinking: false
51+
52+# start + 8 × (next, ask) + judged records + next + notes + report ≈ 25 round trips.
53+maxTurns: 40
54+# ask_agent hands back at most this much of the candidate's answer. Kept
55+# small: a candidate that runs away (seen: Luciole 1B, 42 calls then text in
56+# a loop) must not flood the coordinator's context — the server broke the
57+# coordinator's stream on such a long prompt (2026-10-26).
58+maxOutput: 2000
59+bashTimeout: 2m
60+# During ask_agent every event of the candidate counts as a sign of life.
61+watchdogTimeout: 5m
62+firstTokenTimeout: 3m
63+displayCommands: true
64+
65+yolo: true
66+bashTool: true
67+editTools: true
68+planTool: false
69+askUserTool: false
70+fetchContentTool: false
71+skillsDir: skills
72+
73+# The whole check is ONE question, so the history is never compressed
74+# during it: 32k must hold the skill and the eight answers (each cut at
75+# maxOutput). The results are also kept in reports/.results.tsv.
76+
77+agentsMd:
78+ enabled: false
79+
80+sessions:
81+ enabled: false
82+
83+acp:
84+ name: model-tester
85+ title: 🧪 Model tester
86+
87+system: |
88+ You are a model tester. You test the agent named candidate: you give it
89+ tasks with the ask_agent tool and check what it did. You never do its
90+ tasks yourself and never fix its work.
91+
92+ When the user asks you to test, to check or to evaluate the candidate,
93+ load the model-check skill with read_skill and follow it: the bench
94+ script tells you each step; do exactly what it prints.
95+
96+ Answer the user in their language, briefly. The report is in English.
new file mode 100644
@@ -0,0 +1,96 @@
1+# Model tester — a coordinator that tests what another agent's model can
2+# do. It follows the `model-check` skill: eight small tasks given one by one
3+# to the `candidate` agent (agent.candidate.yaml) with ask_agent, each result
4+# checked by a command when one can decide (skills/model-check/bench.sh) and
5+# judged otherwise, then a Markdown report in reports/ (bench.sh report).
6+#
7+# coordinator Gemma 4 12B hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M 7.3 GB
8+# candidate see agent.candidate.yaml — change its `model:` line to test another model
9+#
10+# cd testing/model-check
11+# mm -tui then type: $model-check
12+#
13+# Run it from this folder: the skill, the scratch folder work/ and reports/
14+# are found from here. It runs commands without asking (yolo: true, for both
15+# agents): the candidate only works in work/, which bench.sh resets.
16+# Needs `go` and `python3` on the PATH (the checks of T3, T5 and T6).
17+
18+provider: llmman
19+model: hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M
20+#model: hf.co/unsloth/gemma-4-E4B-it-GGUF:Q4_K_M
21+
22+baseUrl: http://127.0.0.1:17434/v1
23+fallback: http://host.docker.internal:17434/v1
24+contextWindow: 32768
25+
26+mmConfigs:
27+ candidate:
28+ path: agent.candidate.yaml
29+ description: >-
30+ The agent under test. Give it one task at a time, exactly as the
31+ model-check skill words it.
32+ yolo: true
33+ # One model in memory at a time: mm unloads Gemma before each task
34+ # and the candidate's model after it (ticket 161). Measured: with both
35+ # loaded (llmman's default context, 262144 each), the candidate's
36+ # server answered "500 Compute error." from the second task on.
37+ unloadModels: true
38+
39+sampling:
40+ # Gemma 4: 0.2 / 0.95 / 64 is the "precise" setting measured in the
41+ # demos (internal/agentinit/presets.yaml).
42+ temperature: 0.2
43+ top_p: 0.95
44+ top_k: 64
45+ max_tokens: 4096
46+ parallel_tool_calls: false
47+ # Seen on llmman (2026-10-26): without it this GGUF thinks before every
48+ # answer (reasoning_content) — slower, for a job that is a procedure.
49+ chat_template_kwargs:
50+ enable_thinking: false
51+
52+# start + 8 × (next, ask) + judged records + next + notes + report ≈ 25 round trips.
53+maxTurns: 40
54+# ask_agent hands back at most this much of the candidate's answer. Kept
55+# small: a candidate that runs away (seen: Luciole 1B, 42 calls then text in
56+# a loop) must not flood the coordinator's context — the server broke the
57+# coordinator's stream on such a long prompt (2026-10-26).
58+maxOutput: 2000
59+bashTimeout: 2m
60+# During ask_agent every event of the candidate counts as a sign of life.
61+watchdogTimeout: 5m
62+firstTokenTimeout: 3m
63+displayCommands: true
64+
65+yolo: true
66+bashTool: true
67+editTools: true
68+planTool: false
69+askUserTool: false
70+fetchContentTool: false
71+skillsDir: skills
72+
73+# The whole check is ONE question, so the history is never compressed
74+# during it: 32k must hold the skill and the eight answers (each cut at
75+# maxOutput). The results are also kept in reports/.results.tsv.
76+
77+agentsMd:
78+ enabled: false
79+
80+sessions:
81+ enabled: false
82+
83+acp:
84+ name: model-tester
85+ title: 🧪 Model tester
86+
87+system: |
88+ You are a model tester. You test the agent named candidate: you give it
89+ tasks with the ask_agent tool and check what it did. You never do its
90+ tasks yourself and never fix its work.
91+
92+ When the user asks you to test, to check or to evaluate the candidate,
93+ load the model-check skill with read_skill and follow it: the bench
94+ script tells you each step; do exactly what it prints.
95+
96+ Answer the user in their language, briefly. The report is in English.
added testing/model-check/bench.sh +5 -0
new file mode 100755
@@ -0,0 +1,5 @@
1+#!/usr/bin/env bash
2+# The short way to call this set's bench, from this folder: ./bench.sh next
3+# (seen: a coordinator rewrote skills/model-check/ with underscores and looped on
4+# "No such file"). The real script is skills/model-check/bench.sh.
5+exec "$(dirname "$0")/skills/model-check/bench.sh" "$@"
new file mode 100755
@@ -0,0 +1,5 @@
1+#!/usr/bin/env bash
2+# The short way to call this set's bench, from this folder: ./bench.sh next
3+# (seen: a coordinator rewrote skills/model-check/ with underscores and looped on
4+# "No such file"). The real script is skills/model-check/bench.sh.
5+exec "$(dirname "$0")/skills/model-check/bench.sh" "$@"
added testing/model-check/reports/.current +1 -0
new file mode 100644
@@ -0,0 +1 @@
1+T2
new file mode 100644
@@ -0,0 +1 @@
1+T2
added testing/model-check/reports/2026-10-26-1242-Luciole-8B-Instruct-1.1-GGUF-Q4_K_M .md +44 -0
new file mode 100644
@@ -0,0 +1,44 @@
1+# Model check — hf.co/OpenLLM-France/Luciole-8B-Instruct-1.1-GGUF:Q4_K_M
2+
3+- **Date**: 2026-10-26 12:42
4+- **Tested model**: `hf.co/OpenLLM-France/Luciole-8B-Instruct-1.1-GGUF:Q4_K_M ` (agent.candidate.yaml)
5+- **Coordinator**: `hf.co/unsloth/gemma-4-E4B-it-GGUF:Q4_K_M` (agent.yaml)
6+- **Score**: 1 of 8 passed
7+
8+## Results
9+
10+| Test | Capability | Result | Detail |
11+|---|---|---|---|
12+| T1 | tool call: write a file | PASS | file content exact |
13+| T2 | tool call: run a command | NOT RUN | no result recorded |
14+| T3 | instruction: exact format | NOT RUN | no result recorded |
15+| T4 | instruction: exact answer | NOT RUN | no result recorded |
16+| T5 | code: a program that runs | NOT RUN | no result recorded |
17+| T6 | code: make the tests pass | NOT RUN | no result recorded |
18+| T7 | several steps: edit a file | NOT RUN | no result recorded |
19+| T8 | several steps: read, filter, write | NOT RUN | no result recorded |
20+
21+## What the coordinator saw
22+
23+* T1 passed successfully.
24+* T2 failed due to a generation timeout.
25+* The test seems to be stuck in a loop after T2 failure.
26+* The candidate showed an inability to correctly parse output when a tool call was involved (T2).
27+* The test run was terminated prematurely.
28+## Verdict
29+This model can be trusted with very simple, single-step tasks that do not require complex reasoning or tool interaction. It should not be given tasks that require multi-step execution or robust error handling.
30+
31+## The tests
32+
33+| Test | Capability | What was asked |
34+|---|---|---|
35+| T1 | tool call: write a file | write work/hello.txt with one exact line |
36+| T2 | tool call: run a command | count the lines of work/numbers.txt, answer the number only (37) |
37+| T3 | instruction: exact format | work/colors.json: a JSON array of the three primary colours of light |
38+| T4 | instruction: exact answer | reply with exactly the word READY |
39+| T5 | code: a program that runs | a Go FizzBuzz 1 to 15 in work/fizzbuzz/main.go |
40+| T6 | code: make the tests pass | complete work/calc/calc.go so that go test passes, tests untouched |
41+| T7 | several steps: edit a file | set two keys of work/config.yaml, change nothing else |
42+| T8 | several steps: read, filter, write | copy the TODO lines of work/todo.txt into a new file, report how many (4) |
43+
44+The exact prompts are in `skills/model-check/tests.tsv`.
new file mode 100644
@@ -0,0 +1,44 @@
1+# Model check — hf.co/OpenLLM-France/Luciole-8B-Instruct-1.1-GGUF:Q4_K_M
2+
3+- **Date**: 2026-10-26 12:42
4+- **Tested model**: `hf.co/OpenLLM-France/Luciole-8B-Instruct-1.1-GGUF:Q4_K_M ` (agent.candidate.yaml)
5+- **Coordinator**: `hf.co/unsloth/gemma-4-E4B-it-GGUF:Q4_K_M` (agent.yaml)
6+- **Score**: 1 of 8 passed
7+
8+## Results
9+
10+| Test | Capability | Result | Detail |
11+|---|---|---|---|
12+| T1 | tool call: write a file | PASS | file content exact |
13+| T2 | tool call: run a command | NOT RUN | no result recorded |
14+| T3 | instruction: exact format | NOT RUN | no result recorded |
15+| T4 | instruction: exact answer | NOT RUN | no result recorded |
16+| T5 | code: a program that runs | NOT RUN | no result recorded |
17+| T6 | code: make the tests pass | NOT RUN | no result recorded |
18+| T7 | several steps: edit a file | NOT RUN | no result recorded |
19+| T8 | several steps: read, filter, write | NOT RUN | no result recorded |
20+
21+## What the coordinator saw
22+
23+* T1 passed successfully.
24+* T2 failed due to a generation timeout.
25+* The test seems to be stuck in a loop after T2 failure.
26+* The candidate showed an inability to correctly parse output when a tool call was involved (T2).
27+* The test run was terminated prematurely.
28+## Verdict
29+This model can be trusted with very simple, single-step tasks that do not require complex reasoning or tool interaction. It should not be given tasks that require multi-step execution or robust error handling.
30+
31+## The tests
32+
33+| Test | Capability | What was asked |
34+|---|---|---|
35+| T1 | tool call: write a file | write work/hello.txt with one exact line |
36+| T2 | tool call: run a command | count the lines of work/numbers.txt, answer the number only (37) |
37+| T3 | instruction: exact format | work/colors.json: a JSON array of the three primary colours of light |
38+| T4 | instruction: exact answer | reply with exactly the word READY |
39+| T5 | code: a program that runs | a Go FizzBuzz 1 to 15 in work/fizzbuzz/main.go |
40+| T6 | code: make the tests pass | complete work/calc/calc.go so that go test passes, tests untouched |
41+| T7 | several steps: edit a file | set two keys of work/config.yaml, change nothing else |
42+| T8 | several steps: read, filter, write | copy the TODO lines of work/todo.txt into a new file, report how many (4) |
43+
44+The exact prompts are in `skills/model-check/tests.tsv`.
added testing/model-check/skills/model-check/SKILL.md +32 -0
new file mode 100644
@@ -0,0 +1,32 @@
1+---
2+name: model-check
3+description: Test what the candidate agent's model can do — tool calls, following an instruction, writing code that runs, a task in several steps — by giving it eight small tasks, checking each result, and writing a Markdown report.
4+keywords: test, check, evaluate, benchmark, model, candidate, report, capabilities
5+---
6+
7+# Model check
8+
9+You test the agent named `candidate`. You never do its tasks yourself and never fix its work. The script tells you what to do at each step and checks the results: `./bench.sh`.
10+
11+## The loop
12+
13+1. Run with bash: `./bench.sh start`
14+2. Run with bash: `./bench.sh next`
15+3. Do what `next` printed:
16+ - **NEXT**: call `ask_agent` with `agent: candidate`, `with_context: false`, and as `prompt` the text it printed, copied exactly. When the candidate has answered, go back to step 2. If `next` printed an "Afterwards" line, keep that note for the report.
17+ - **is judged by you**: compare the candidate's last answer with the rule `next` printed, run with bash `./bench.sh record <id> PASS "<why>"` (the id `next` named) or `... FAIL "<why>"` (why: short, no double quotes), then go back to step 2.
18+ - **ALL TESTS DONE**: go to "The report".
19+
20+`next` checks the test just run before it hands out the next one: never skip it, and never call `ask_agent` twice without `next` in between. Never give the candidate a second try.
21+
22+If `ask_agent` returns an error, run with bash `./bench.sh record <id> FAIL "<the error, short, no double quotes>"`, where `<id>` is the test `next` printed last (F3, T5…), then go back to step 2.
23+
24+While the loop runs, look at how the candidate works: which tools it uses, whether it checks its own work, answers longer than asked, errors, retries or loops.
25+
26+## The report
27+
28+1. Write your notes with `write_file` at the path `reports/.notes.md`, in this order:
29+ - three to six bullets on what you saw during the run, including the T8 number;
30+ - a heading `## Verdict`, then two sentences: what this model can be trusted with, and what it should not be given.
31+2. Run with bash: `./bench.sh report`. It writes the Markdown report and prints its path.
32+3. Answer the user in one or two sentences: the score and the report's path.
new file mode 100644
@@ -0,0 +1,32 @@
1+---
2+name: model-check
3+description: Test what the candidate agent's model can do — tool calls, following an instruction, writing code that runs, a task in several steps — by giving it eight small tasks, checking each result, and writing a Markdown report.
4+keywords: test, check, evaluate, benchmark, model, candidate, report, capabilities
5+---
6+
7+# Model check
8+
9+You test the agent named `candidate`. You never do its tasks yourself and never fix its work. The script tells you what to do at each step and checks the results: `./bench.sh`.
10+
11+## The loop
12+
13+1. Run with bash: `./bench.sh start`
14+2. Run with bash: `./bench.sh next`
15+3. Do what `next` printed:
16+ - **NEXT**: call `ask_agent` with `agent: candidate`, `with_context: false`, and as `prompt` the text it printed, copied exactly. When the candidate has answered, go back to step 2. If `next` printed an "Afterwards" line, keep that note for the report.
17+ - **is judged by you**: compare the candidate's last answer with the rule `next` printed, run with bash `./bench.sh record <id> PASS "<why>"` (the id `next` named) or `... FAIL "<why>"` (why: short, no double quotes), then go back to step 2.
18+ - **ALL TESTS DONE**: go to "The report".
19+
20+`next` checks the test just run before it hands out the next one: never skip it, and never call `ask_agent` twice without `next` in between. Never give the candidate a second try.
21+
22+If `ask_agent` returns an error, run with bash `./bench.sh record <id> FAIL "<the error, short, no double quotes>"`, where `<id>` is the test `next` printed last (F3, T5…), then go back to step 2.
23+
24+While the loop runs, look at how the candidate works: which tools it uses, whether it checks its own work, answers longer than asked, errors, retries or loops.
25+
26+## The report
27+
28+1. Write your notes with `write_file` at the path `reports/.notes.md`, in this order:
29+ - three to six bullets on what you saw during the run, including the T8 number;
30+ - a heading `## Verdict`, then two sentences: what this model can be trusted with, and what it should not be given.
31+2. Run with bash: `./bench.sh report`. It writes the Markdown report and prints its path.
32+3. Answer the user in one or two sentences: the score and the report's path.
added testing/model-check/skills/model-check/bench.sh +80 -0
new file mode 100755
@@ -0,0 +1,80 @@
1+#!/usr/bin/env bash
2+# The model-check bench. The loop and the report are in common/bench-lib.sh;
3+# this file says how to reset work/ and how to check each test.
4+#
5+# bench.sh start new run: prints the tested model and the report path
6+# bench.sh next checks the test just run, hands out the next one
7+# bench.sh record T2 PASS "…" records a test you judged, or one that could not run
8+# bench.sh table the results so far, as a Markdown table
9+# bench.sh report writes the Markdown report (results + reports/.notes.md)
10+#
11+# Run from anywhere: the paths are taken from this script's own location.
12+set -eu
13+
14+SKILL="$(cd "$(dirname "$0")" && pwd)"
15+ROOT="$(cd "$SKILL/../.." && pwd)" # testing/model-check/
16+WORK="$ROOT/work"
17+TITLE="Model check"
18+# shellcheck source=../../../common/bench-lib.sh
19+. "$ROOT/../common/bench-lib.sh"
20+
21+record() { bench_record "$@"; }
22+
23+reset_work() {
24+ rm -rf "$WORK"
25+ cp -R "$SKILL/fixtures" "$WORK"
26+}
27+
28+check_test() {
29+ case "$1" in
30+ T1)
31+ [ -f "$WORK/hello.txt" ] || { record T1 FAIL "work/hello.txt does not exist"; return; }
32+ if [ "$(cat "$WORK/hello.txt")" = "Hello from mini-me" ]; then
33+ record T1 PASS "file content exact"
34+ else
35+ record T1 FAIL "content: $(head -c 80 "$WORK/hello.txt" | paste -sd '/' -)"
36+ fi ;;
37+ T3)
38+ [ -f "$WORK/colors.json" ] || { record T3 FAIL "work/colors.json does not exist"; return; }
39+ if python3 -c 'import json,sys; v=json.load(open(sys.argv[1])); sys.exit(0 if isinstance(v,list) and sorted(v)==["blue","green","red"] else 1)' "$WORK/colors.json" 2>/dev/null; then
40+ record T3 PASS "valid JSON, exactly red, green, blue"
41+ else
42+ record T3 FAIL "content: $(head -c 80 "$WORK/colors.json" | paste -sd '/' -)"
43+ fi ;;
44+ T5)
45+ [ -f "$WORK/fizzbuzz/main.go" ] || { record T5 FAIL "work/fizzbuzz/main.go does not exist"; return; }
46+ expected="1 2 Fizz 4 Buzz Fizz 7 8 Fizz Buzz 11 Fizz 13 14 FizzBuzz"
47+ got="$(cd "$WORK/fizzbuzz" && go run main.go 2>&1 | tr '\n' ' ' | sed 's/ *$//')" || true
48+ if [ "$got" = "$expected" ]; then
49+ record T5 PASS "go run prints the 15 expected lines"
50+ else
51+ record T5 FAIL "go run: $(printf '%s' "$got" | head -c 100)"
52+ fi ;;
53+ T6)
54+ if ! cmp -s "$WORK/calc/calc_test.go" "$SKILL/fixtures/calc/calc_test.go"; then
55+ record T6 FAIL "calc_test.go was changed"
56+ elif out="$(cd "$WORK/calc" && go test ./... 2>&1)"; then
57+ record T6 PASS "go test passes, tests untouched"
58+ else
59+ record T6 FAIL "go test: $(printf '%s' "$out" | grep -m1 -E 'FAIL|error|want' | head -c 100)"
60+ fi ;;
61+ T7)
62+ if printf 'name: demo\nport: 9090\ndebug: true\nowner: bob\n' | diff -q - "$WORK/config.yaml" >/dev/null 2>&1; then
63+ record T7 PASS "port and debug changed, nothing else"
64+ else
65+ record T7 FAIL "config.yaml: $(paste -sd '/' - <"$WORK/config.yaml" 2>/dev/null | head -c 100)"
66+ fi ;;
67+ T8)
68+ [ -f "$WORK/todo-only.txt" ] || { record T8 FAIL "work/todo-only.txt does not exist"; return; }
69+ if grep '^TODO' "$SKILL/fixtures/todo.txt" | diff -q - "$WORK/todo-only.txt" >/dev/null; then
70+ record T8 PASS "the 4 TODO lines, in order, unchanged"
71+ else
72+ record T8 FAIL "todo-only.txt: $(paste -sd '/' - <"$WORK/todo-only.txt" | head -c 100)"
73+ fi ;;
74+ *)
75+ echo "no command check for $1" >&2
76+ exit 2 ;;
77+ esac
78+}
79+
80+bench_main "$@"
new file mode 100755
@@ -0,0 +1,80 @@
1+#!/usr/bin/env bash
2+# The model-check bench. The loop and the report are in common/bench-lib.sh;
3+# this file says how to reset work/ and how to check each test.
4+#
5+# bench.sh start new run: prints the tested model and the report path
6+# bench.sh next checks the test just run, hands out the next one
7+# bench.sh record T2 PASS "…" records a test you judged, or one that could not run
8+# bench.sh table the results so far, as a Markdown table
9+# bench.sh report writes the Markdown report (results + reports/.notes.md)
10+#
11+# Run from anywhere: the paths are taken from this script's own location.
12+set -eu
13+
14+SKILL="$(cd "$(dirname "$0")" && pwd)"
15+ROOT="$(cd "$SKILL/../.." && pwd)" # testing/model-check/
16+WORK="$ROOT/work"
17+TITLE="Model check"
18+# shellcheck source=../../../common/bench-lib.sh
19+. "$ROOT/../common/bench-lib.sh"
20+
21+record() { bench_record "$@"; }
22+
23+reset_work() {
24+ rm -rf "$WORK"
25+ cp -R "$SKILL/fixtures" "$WORK"
26+}
27+
28+check_test() {
29+ case "$1" in
30+ T1)
31+ [ -f "$WORK/hello.txt" ] || { record T1 FAIL "work/hello.txt does not exist"; return; }
32+ if [ "$(cat "$WORK/hello.txt")" = "Hello from mini-me" ]; then
33+ record T1 PASS "file content exact"
34+ else
35+ record T1 FAIL "content: $(head -c 80 "$WORK/hello.txt" | paste -sd '/' -)"
36+ fi ;;
37+ T3)
38+ [ -f "$WORK/colors.json" ] || { record T3 FAIL "work/colors.json does not exist"; return; }
39+ if python3 -c 'import json,sys; v=json.load(open(sys.argv[1])); sys.exit(0 if isinstance(v,list) and sorted(v)==["blue","green","red"] else 1)' "$WORK/colors.json" 2>/dev/null; then
40+ record T3 PASS "valid JSON, exactly red, green, blue"
41+ else
42+ record T3 FAIL "content: $(head -c 80 "$WORK/colors.json" | paste -sd '/' -)"
43+ fi ;;
44+ T5)
45+ [ -f "$WORK/fizzbuzz/main.go" ] || { record T5 FAIL "work/fizzbuzz/main.go does not exist"; return; }
46+ expected="1 2 Fizz 4 Buzz Fizz 7 8 Fizz Buzz 11 Fizz 13 14 FizzBuzz"
47+ got="$(cd "$WORK/fizzbuzz" && go run main.go 2>&1 | tr '\n' ' ' | sed 's/ *$//')" || true
48+ if [ "$got" = "$expected" ]; then
49+ record T5 PASS "go run prints the 15 expected lines"
50+ else
51+ record T5 FAIL "go run: $(printf '%s' "$got" | head -c 100)"
52+ fi ;;
53+ T6)
54+ if ! cmp -s "$WORK/calc/calc_test.go" "$SKILL/fixtures/calc/calc_test.go"; then
55+ record T6 FAIL "calc_test.go was changed"
56+ elif out="$(cd "$WORK/calc" && go test ./... 2>&1)"; then
57+ record T6 PASS "go test passes, tests untouched"
58+ else
59+ record T6 FAIL "go test: $(printf '%s' "$out" | grep -m1 -E 'FAIL|error|want' | head -c 100)"
60+ fi ;;
61+ T7)
62+ if printf 'name: demo\nport: 9090\ndebug: true\nowner: bob\n' | diff -q - "$WORK/config.yaml" >/dev/null 2>&1; then
63+ record T7 PASS "port and debug changed, nothing else"
64+ else
65+ record T7 FAIL "config.yaml: $(paste -sd '/' - <"$WORK/config.yaml" 2>/dev/null | head -c 100)"
66+ fi ;;
67+ T8)
68+ [ -f "$WORK/todo-only.txt" ] || { record T8 FAIL "work/todo-only.txt does not exist"; return; }
69+ if grep '^TODO' "$SKILL/fixtures/todo.txt" | diff -q - "$WORK/todo-only.txt" >/dev/null; then
70+ record T8 PASS "the 4 TODO lines, in order, unchanged"
71+ else
72+ record T8 FAIL "todo-only.txt: $(paste -sd '/' - <"$WORK/todo-only.txt" | head -c 100)"
73+ fi ;;
74+ *)
75+ echo "no command check for $1" >&2
76+ exit 2 ;;
77+ esac
78+}
79+
80+bench_main "$@"
added testing/model-check/skills/model-check/fixtures/calc/calc.go +12 -0
new file mode 100644
@@ -0,0 +1,12 @@
1+// Package calc is the stub the tested model must complete.
2+package calc
3+
4+// Add returns a + b.
5+func Add(a, b int) int {
6+ return 0
7+}
8+
9+// Reverse returns s with its characters in reverse order ("héllo" → "olléh").
10+func Reverse(s string) string {
11+ return ""
12+}
new file mode 100644
@@ -0,0 +1,12 @@
1+// Package calc is the stub the tested model must complete.
2+package calc
3+
4+// Add returns a + b.
5+func Add(a, b int) int {
6+ return 0
7+}
8+
9+// Reverse returns s with its characters in reverse order ("héllo" → "olléh").
10+func Reverse(s string) string {
11+ return ""
12+}
added testing/model-check/skills/model-check/fixtures/calc/calc_test.go +21 -0
new file mode 100644
@@ -0,0 +1,21 @@
1+package calc
2+
3+import "testing"
4+
5+func TestAdd(t *testing.T) {
6+ cases := []struct{ a, b, want int }{{1, 2, 3}, {-4, 4, 0}, {0, 0, 0}, {120, -21, 99}}
7+ for _, c := range cases {
8+ if got := Add(c.a, c.b); got != c.want {
9+ t.Errorf("Add(%d, %d) = %d, want %d", c.a, c.b, got, c.want)
10+ }
11+ }
12+}
13+
14+func TestReverse(t *testing.T) {
15+ cases := map[string]string{"": "", "abc": "cba", "héllo": "olléh", "Go 🐹": "🐹 oG"}
16+ for in, want := range cases {
17+ if got := Reverse(in); got != want {
18+ t.Errorf("Reverse(%q) = %q, want %q", in, got, want)
19+ }
20+ }
21+}
new file mode 100644
@@ -0,0 +1,21 @@
1+package calc
2+
3+import "testing"
4+
5+func TestAdd(t *testing.T) {
6+ cases := []struct{ a, b, want int }{{1, 2, 3}, {-4, 4, 0}, {0, 0, 0}, {120, -21, 99}}
7+ for _, c := range cases {
8+ if got := Add(c.a, c.b); got != c.want {
9+ t.Errorf("Add(%d, %d) = %d, want %d", c.a, c.b, got, c.want)
10+ }
11+ }
12+}
13+
14+func TestReverse(t *testing.T) {
15+ cases := map[string]string{"": "", "abc": "cba", "héllo": "olléh", "Go 🐹": "🐹 oG"}
16+ for in, want := range cases {
17+ if got := Reverse(in); got != want {
18+ t.Errorf("Reverse(%q) = %q, want %q", in, got, want)
19+ }
20+ }
21+}
added testing/model-check/skills/model-check/fixtures/calc/go.mod +3 -0
new file mode 100644
@@ -0,0 +1,3 @@
1+module calc
2+
3+go 1.22
new file mode 100644
@@ -0,0 +1,3 @@
1+module calc
2+
3+go 1.22
added testing/model-check/skills/model-check/fixtures/config.yaml +4 -0
new file mode 100644
@@ -0,0 +1,4 @@
1+name: demo
2+port: 8080
3+debug: false
4+owner: bob
new file mode 100644
@@ -0,0 +1,4 @@
1+name: demo
2+port: 8080
3+debug: false
4+owner: bob
added testing/model-check/skills/model-check/fixtures/go.mod +5 -0
new file mode 100644
@@ -0,0 +1,5 @@
1+// The scratch folder of the tests is its own module, so the Go files a
2+// tested model writes here never become part of mini-me's module.
3+module work
4+
5+go 1.22
new file mode 100644
@@ -0,0 +1,5 @@
1+// The scratch folder of the tests is its own module, so the Go files a
2+// tested model writes here never become part of mini-me's module.
3+module work
4+
5+go 1.22
added testing/model-check/skills/model-check/fixtures/numbers.txt +37 -0
new file mode 100644
@@ -0,0 +1,37 @@
1+1
2+2
3+3
4+4
5+5
6+6
7+7
8+8
9+9
10+10
11+11
12+12
13+13
14+14
15+15
16+16
17+17
18+18
19+19
20+20
21+21
22+22
23+23
24+24
25+25
26+26
27+27
28+28
29+29
30+30
31+31
32+32
33+33
34+34
35+35
36+36
37+37
new file mode 100644
@@ -0,0 +1,37 @@
1+1
2+2
3+3
4+4
5+5
6+6
7+7
8+8
9+9
10+10
11+11
12+12
13+13
14+14
15+15
16+16
17+17
18+18
19+19
20+20
21+21
22+22
23+23
24+24
25+25
26+26
27+27
28+28
29+29
30+30
31+31
32+32
33+33
34+34
35+35
36+36
37+37
added testing/model-check/skills/model-check/fixtures/todo.txt +8 -0
new file mode 100644
@@ -0,0 +1,8 @@
1+TODO write the README
2+buy milk
3+TODO fix the login bug
4+ TODO this one is indented, not copied
5+DONE ship v1
6+TODO call Alice
7+notes: TODO is a word here
8+TODO update the changelog
new file mode 100644
@@ -0,0 +1,8 @@
1+TODO write the README
2+buy milk
3+TODO fix the login bug
4+ TODO this one is indented, not copied
5+DONE ship v1
6+TODO call Alice
7+notes: TODO is a word here
8+TODO update the changelog
added testing/model-check/skills/model-check/tests.tsv +8 -0
new file mode 100644
@@ -0,0 +1,8 @@
1+T1 tool call: write a file write work/hello.txt with one exact line command Create the file work/hello.txt containing exactly this one line: Hello from mini-me
2+T2 tool call: run a command count the lines of work/numbers.txt, answer the number only (37) judge How many lines does the file work/numbers.txt have? Find out by running a command. Answer with the number only. PASS only if the answer is 37 and nothing else; the right number with extra words is FAIL (right number, extra text).
3+T3 instruction: exact format work/colors.json: a JSON array of the three primary colours of light command Write the file work/colors.json. It must contain only a JSON array of the three primary colours of light, as lowercase English strings. Nothing else in the file.
4+T4 instruction: exact answer reply with exactly the word READY judge Reply with exactly the word READY in capital letters, and nothing else. PASS only if the answer is READY; any other word, punctuation or sentence is FAIL.
5+T5 code: a program that runs a Go FizzBuzz 1 to 15 in work/fizzbuzz/main.go command Create the Go program work/fizzbuzz/main.go (package main). It prints FizzBuzz from 1 to 15, one value per line: Fizz for multiples of 3, Buzz for multiples of 5, FizzBuzz for multiples of both, otherwise the number. Run it with: cd work/fizzbuzz && go run main.go
6+T6 code: make the tests pass complete work/calc/calc.go so that go test passes, tests untouched command In work/calc there is a Go package whose tests fail. Run go test in work/calc, read calc_test.go, and complete calc.go so that all the tests pass. Do not change calc_test.go.
7+T7 several steps: edit a file set two keys of work/config.yaml, change nothing else command In the file work/config.yaml, set port to 9090 and debug to true. Change nothing else in the file.
8+T8 several steps: read, filter, write copy the TODO lines of work/todo.txt into a new file, report how many (4) command Read work/todo.txt. Write into a new file work/todo-only.txt every line that starts with TODO, in the same order and unchanged. Then answer with the number of lines you copied. Note for the report the number the candidate answered (the right number is 4).
new file mode 100644
@@ -0,0 +1,8 @@
1+T1 tool call: write a file write work/hello.txt with one exact line command Create the file work/hello.txt containing exactly this one line: Hello from mini-me
2+T2 tool call: run a command count the lines of work/numbers.txt, answer the number only (37) judge How many lines does the file work/numbers.txt have? Find out by running a command. Answer with the number only. PASS only if the answer is 37 and nothing else; the right number with extra words is FAIL (right number, extra text).
3+T3 instruction: exact format work/colors.json: a JSON array of the three primary colours of light command Write the file work/colors.json. It must contain only a JSON array of the three primary colours of light, as lowercase English strings. Nothing else in the file.
4+T4 instruction: exact answer reply with exactly the word READY judge Reply with exactly the word READY in capital letters, and nothing else. PASS only if the answer is READY; any other word, punctuation or sentence is FAIL.
5+T5 code: a program that runs a Go FizzBuzz 1 to 15 in work/fizzbuzz/main.go command Create the Go program work/fizzbuzz/main.go (package main). It prints FizzBuzz from 1 to 15, one value per line: Fizz for multiples of 3, Buzz for multiples of 5, FizzBuzz for multiples of both, otherwise the number. Run it with: cd work/fizzbuzz && go run main.go
6+T6 code: make the tests pass complete work/calc/calc.go so that go test passes, tests untouched command In work/calc there is a Go package whose tests fail. Run go test in work/calc, read calc_test.go, and complete calc.go so that all the tests pass. Do not change calc_test.go.
7+T7 several steps: edit a file set two keys of work/config.yaml, change nothing else command In the file work/config.yaml, set port to 9090 and debug to true. Change nothing else in the file.
8+T8 several steps: read, filter, write copy the TODO lines of work/todo.txt into a new file, report how many (4) command Read work/todo.txt. Write into a new file work/todo-only.txt every line that starts with TODO, in the same order and unchanged. Then answer with the number of lines you copied. Note for the report the number the candidate answered (the right number is 4).