rickub
bots-garden/mini-mepublic Fork
⑂ fc4190d
Branches
feature/agents-file feature/ask-agent-hooks feature/ask-cli feature/declared-tools feature/display-called-hooks feature/fetch-build-in feature/genkit-v1.13.1 feature/github-mcp-in-sbx feature/go-1.27.3 feature/hooks feature/improve-init feature/interactive-prompt feature/last-question feature/llmman-provider feature/log-errors feature/lsp-builtin feature/mcp-gateway-session feature/mcp-support feature/memory-search feature/mini-mes feature/mm-init feature/open-terminal feature/prompt-flag feature/response-request feature/resume-acp feature/sbx-template-kit feature/search-in-doc feature/selectors feature/skill-discovery feature/structured-output feature/system-one feature/talk-to-another-mm feature/tasks-builtin feature/try-again feature/unload-model feature/wasm-custom-tools feature/yolo feature/zed-skill-selector fix/generation-timeout fix/plan-mode fix/too-long-completion fix/tool-order fix/word-wrap improve-mem-usage main thinking
Tags
v0.0.0 v0.0.1 v0.0.2 v0.0.3 v0.0.4 v0.1.0 v0.1.1 v0.1.2 v0.1.3 v0.1.4 v0.1.5 v0.1.6 v0.1.7 v0.1.8 v0.1.9 v0.2.0 v0.2.1 v0.2.2 v0.2.3 v0.2.4 v0.2.5 v0.2.6 v0.3.0 v0.4.0 v0.5.0 v0.6.0 v0.7.0 v0.8.0 v0.9.0 v1.0.0 View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/bots-garden/mini-me.git
git clone ssh://git@rickub.com/bots-garden/mini-me.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Download ZIP

🛟 Updated.

k33g committed 2026-10-20T10:11:30+02:00 Browse files
fc4190d parent: 78672b6
modified .tickets/issues/0104-how-to-search-in-a-documentation-markdown-files.yaml +73 -3
@@ -6,18 +6,88 @@ author:
66 name: k33g
77 email: ph.charriere@gmail.com
88 createdAt: 2026-10-20T07:40:25.739Z
9-updatedAt: 2026-10-20T07:43:51.546Z
9+updatedAt: 2026-10-20T19:00:00.000Z
1010 labels: []
1111 milestone: v0.8.0
1212 body: ""
1313 tasks:
1414 - id: 1
1515 title: Search if a specific CLI already exists
16- state: open
16+ state: closed
1717 priority: urgent
1818 author:
1919 name: k33g
2020 email: ph.charriere@gmail.com
2121 createdAt: 2026-10-20T07:43:45.685Z
2222 updatedAt: 2026-10-20T07:43:51.543Z
23-comments: []
23+comments:
24+ - id: 1
25+ author:
26+ name: k33g
27+ email: ph.charriere@gmail.com
28+ createdAt: 2026-10-20T19:00:00.000Z
29+ body: |
30+ Task 1 — web search (2026-10-20). Yes: dedicated CLIs exist. "Read" =
31+ checked on the project page; others are from search results only.
32+
33+ ## The one that fits: qmd (github.com/tobi/qmd) — read
34+ - "An on-device search engine for everything you need to remember":
35+ markdown notes, transcripts, documentation. MIT. Node 22+ or Bun 1+;
36+ `npm install -g @tobilu/qmd` (or `npx @tobilu/qmd …`). macOS needs
37+ Homebrew SQLite.
38+ - Index: SQLite (FTS5 + sqlite-vec) in ~/.cache/qmd/index.sqlite;
39+ folders added as collections (`qmd collection add`), `qmd embed`
40+ for vectors. Chunks ~900 tokens, 15 % overlap, cut at headings /
41+ code fences / paragraphs.
42+ - Three modes: `qmd search` = BM25 keywords, NO model;
43+ `qmd vsearch` = vectors (embeddinggemma-300M Q8_0, ~300 MB — the same
44+ model our llmman already has); `qmd query` = hybrid + query
45+ expansion (qmd-query-expansion-1.7B, ~1.1 GB) + reranking
46+ (qwen3-reranker-0.6b, ~640 MB). Models via node-llama-cpp, in
47+ ~/.cache/qmd/models/.
48+ - For agents: `--format json|csv|md|xml|files`, `--explain`,
49+ `qmd get` / `multi-get`; `qmd mcp` (stdio, tools query, get,
50+ multi_get, status, metadata) and `qmd mcp --http` (localhost:8181,
51+ models loaded once). Last release seen: v2.5.3 (2026-05-28).
52+ - Search results also list a Rust port (rqmd) — not checked.
53+
54+ ## Other tools
55+ - mdq (github.com/yshavit/mdq, Rust, MIT/Apache) — read: "jq for
56+ Markdown", selects sections (`mdq '# usage'`), lists, links, tables
57+ in ONE document (stdin); `--output json`. A complement (extract the
58+ section a search found), not a search engine.
59+ - NoteBrain (Go, MIT) — read: semantic search, local ONNX
60+ all-MiniLM-L6-v2, embedded ChromaDB, JSON/TSV output; built around
61+ Obsidian vaults (wikilinks), needs CGO; no MCP mentioned.
62+ - From search results only: docbert (Rust, BM25 then ColBERT
63+ rerank), mdsearch-cli (Python, sentence-transformers + FAISS, CLI +
64+ MCP; needs a Hugging Face token), obsidian-notes-rag (embeddings via
65+ Ollama/LM Studio, sqlite-vec), srcsearch (Tantivy BM25 over Rust +
66+ Markdown), grepdown / fastkb / mdquery (SQLite FTS5), zk (Go, notes
67+ with a full-text index), mq (harehare, jq-like for Markdown),
68+ puya/mdparser (Python, section-limited search). ripgrep has no
69+ section-aware mode (`rg -t md` only filters files).
70+
71+ ## What it means for mini-me (proposal, to decide)
72+ mm needs no new code to use qmd: it already has MCP over stdio and
73+ declared tools (ticket 88).
74+ 1. MCP: `qmd mcp` as an MCP server → the model gets query / get.
75+ 2. Declared tool: `qmd search {{query}} --format json -c docs` (BM25,
76+ no model loaded beside the chat model — fits a small machine), and
77+ optionally `qmd query` for the hybrid mode when RAM allows.
78+ 3. A demo (e.g. demos/docs-search/) on mm's own docs/ (EN+FR): measure
79+ BM25 vs hybrid on a small model, and whether hybrid fixes what
80+ search_memory cannot (FR question / EN text, synonyms — ticket 48
81+ comment 3).
82+ Trade-off: qmd brings Node/Bun and its own model downloads (up to
83+ ~2 GB for `qmd query`); `qmd search` alone is model-free.
84+ - id: 2
85+ author:
86+ name: k33g
87+ email: ph.charriere@gmail.com
88+ createdAt: 2026-10-20T20:00:00.000Z
89+ body: |
90+ Decided by the user: build it into mm — a new built-in tool search_doc,
91+ BM25, passages with the link to their file; search_memory is kept as
92+ it is. Ticket 106.
93+
@@ -6,18 +6,88 @@ author:
6 name: k33g6 name: k33g
7 email: ph.charriere@gmail.com7 email: ph.charriere@gmail.com
8 createdAt: 2026-10-20T07:40:25.739Z8 createdAt: 2026-10-20T07:40:25.739Z
9-updatedAt: 2026-10-20T07:43:51.546Z9+updatedAt: 2026-10-20T19:00:00.000Z
10 labels: []10 labels: []
11 milestone: v0.8.011 milestone: v0.8.0
12 body: ""12 body: ""
13 tasks:13 tasks:
14 - id: 114 - id: 1
15 title: Search if a specific CLI already exists15 title: Search if a specific CLI already exists
16- state: open16+ state: closed
17 priority: urgent17 priority: urgent
18 author:18 author:
19 name: k33g19 name: k33g
20 email: ph.charriere@gmail.com20 email: ph.charriere@gmail.com
21 createdAt: 2026-10-20T07:43:45.685Z21 createdAt: 2026-10-20T07:43:45.685Z
22 updatedAt: 2026-10-20T07:43:51.543Z22 updatedAt: 2026-10-20T07:43:51.543Z
23-comments: []23+comments:
24+ - id: 1
25+ author:
26+ name: k33g
27+ email: ph.charriere@gmail.com
28+ createdAt: 2026-10-20T19:00:00.000Z
29+ body: |
30+ Task 1 — web search (2026-10-20). Yes: dedicated CLIs exist. "Read" =
31+ checked on the project page; others are from search results only.
32+
33+ ## The one that fits: qmd (github.com/tobi/qmd) — read
34+ - "An on-device search engine for everything you need to remember":
35+ markdown notes, transcripts, documentation. MIT. Node 22+ or Bun 1+;
36+ `npm install -g @tobilu/qmd` (or `npx @tobilu/qmd …`). macOS needs
37+ Homebrew SQLite.
38+ - Index: SQLite (FTS5 + sqlite-vec) in ~/.cache/qmd/index.sqlite;
39+ folders added as collections (`qmd collection add`), `qmd embed`
40+ for vectors. Chunks ~900 tokens, 15 % overlap, cut at headings /
41+ code fences / paragraphs.
42+ - Three modes: `qmd search` = BM25 keywords, NO model;
43+ `qmd vsearch` = vectors (embeddinggemma-300M Q8_0, ~300 MB — the same
44+ model our llmman already has); `qmd query` = hybrid + query
45+ expansion (qmd-query-expansion-1.7B, ~1.1 GB) + reranking
46+ (qwen3-reranker-0.6b, ~640 MB). Models via node-llama-cpp, in
47+ ~/.cache/qmd/models/.
48+ - For agents: `--format json|csv|md|xml|files`, `--explain`,
49+ `qmd get` / `multi-get`; `qmd mcp` (stdio, tools query, get,
50+ multi_get, status, metadata) and `qmd mcp --http` (localhost:8181,
51+ models loaded once). Last release seen: v2.5.3 (2026-05-28).
52+ - Search results also list a Rust port (rqmd) — not checked.
53+
54+ ## Other tools
55+ - mdq (github.com/yshavit/mdq, Rust, MIT/Apache) — read: "jq for
56+ Markdown", selects sections (`mdq '# usage'`), lists, links, tables
57+ in ONE document (stdin); `--output json`. A complement (extract the
58+ section a search found), not a search engine.
59+ - NoteBrain (Go, MIT) — read: semantic search, local ONNX
60+ all-MiniLM-L6-v2, embedded ChromaDB, JSON/TSV output; built around
61+ Obsidian vaults (wikilinks), needs CGO; no MCP mentioned.
62+ - From search results only: docbert (Rust, BM25 then ColBERT
63+ rerank), mdsearch-cli (Python, sentence-transformers + FAISS, CLI +
64+ MCP; needs a Hugging Face token), obsidian-notes-rag (embeddings via
65+ Ollama/LM Studio, sqlite-vec), srcsearch (Tantivy BM25 over Rust +
66+ Markdown), grepdown / fastkb / mdquery (SQLite FTS5), zk (Go, notes
67+ with a full-text index), mq (harehare, jq-like for Markdown),
68+ puya/mdparser (Python, section-limited search). ripgrep has no
69+ section-aware mode (`rg -t md` only filters files).
70+
71+ ## What it means for mini-me (proposal, to decide)
72+ mm needs no new code to use qmd: it already has MCP over stdio and
73+ declared tools (ticket 88).
74+ 1. MCP: `qmd mcp` as an MCP server → the model gets query / get.
75+ 2. Declared tool: `qmd search {{query}} --format json -c docs` (BM25,
76+ no model loaded beside the chat model — fits a small machine), and
77+ optionally `qmd query` for the hybrid mode when RAM allows.
78+ 3. A demo (e.g. demos/docs-search/) on mm's own docs/ (EN+FR): measure
79+ BM25 vs hybrid on a small model, and whether hybrid fixes what
80+ search_memory cannot (FR question / EN text, synonyms — ticket 48
81+ comment 3).
82+ Trade-off: qmd brings Node/Bun and its own model downloads (up to
83+ ~2 GB for `qmd query`); `qmd search` alone is model-free.
84+ - id: 2
85+ author:
86+ name: k33g
87+ email: ph.charriere@gmail.com
88+ createdAt: 2026-10-20T20:00:00.000Z
89+ body: |
90+ Decided by the user: build it into mm — a new built-in tool search_doc,
91+ BM25, passages with the link to their file; search_memory is kept as
92+ it is. Ticket 106.
93+
added .tickets/issues/0106-search-doc-bm25-search-in-text-files.yaml +131 -0
new file mode 100644
@@ -0,0 +1,131 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 106
3+title: search_doc — BM25 search in the text files of a folder (built-in tool)
4+state: open
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-20T20:00:00.000Z
9+updatedAt: 2026-10-20T20:00:00.000Z
10+labels:
11+ - kind::feature
12+milestone: v0.8.0
13+epic: Builtin tools
14+related:
15+ - 104
16+ - 48
17+body: |
18+ A built-in tool, search_doc, that lets the agent search the text files of
19+ one or more folders — Markdown (.md), plain text (.txt), AsciiDoc (.adoc,
20+ .asciidoc)… — and get back the relevant passages, each with the link to
21+ its file. Ranking: BM25, no model (like `qmd search`, ticket 104).
22+
23+ User's decisions (2026-10-20):
24+ - a NEW tool, search_doc; search_memory (ticket 48) is NOT replaced and
25+ keeps its own engine;
26+ - BM25;
27+ - each passage comes with the link to its file.
28+
29+ Proposal (to confirm before coding):
30+ - Config block, off by default so no existing agent changes:
31+ docSearch:
32+ enabled: true
33+ dirs: [docs] # relative to the working directory
34+ # (the session cwd over ACP); absolute ok
35+ extensions: [.md, .markdown, .txt, .adoc, .asciidoc, .rst]
36+ maxFileSize: 1MB # bigger files skipped, banner-free
37+ - Tool search_doc {query, limit?} (limit default 5, cap 20). One hit per
38+ passage:
39+ docs/en/how-to/keep-sessions.md:46 · Keep sessions › Find out why a run failed
40+ …the text of the passage around the words found…
41+ The link is the path relative to the working directory plus the
42+ passage's first line (`path:line`), what an editor and read_file both
43+ take; the model can then read_file the whole page.
44+ - Passages = sections: a Markdown / AsciiDoc heading starts one (headings
45+ inside code fences ignored), keeping the heading path; plain text is cut
46+ at blank lines; a long section is cut by paragraphs (~1,500 chars).
47+ - Words: the same folding as search_memory (case and accents do not
48+ count). BM25 k1 1.2, b 0.75, IDF over the passages of the folders; words
49+ found in a heading weigh more.
50+ - No index: the folders are read at each call. Measured for scale: mm's
51+ docs/ = 133 .md files, 1.3 MB, ~152k words, 1,208 headings.
52+ - Never asks for permission (it only reads, like read_file); ACP tool
53+ kind `search`; output capped by maxOutput.
54+
55+ Known limits (stated up front): no stemming ("migration" ≠
56+ "migrations" unless prefix matching is added); no match across languages
57+ or synonyms — the same class of limit as search_memory (ticket 48
58+ comment 3) and as qmd's own BM25 mode.
59+
60+ Open questions:
61+ - a /docs <words> command for the user too, as /memory is for
62+ search_memory?
63+ - prefix matching for words of 4+ letters (a cheap answer to plurals), or
64+ exact words only?
65+ - other extensions by default (.rst, .org, .html)? follow symlinks?
66+ - the key name: `docSearch` (proposed) or under an existing block?
67+tasks:
68+ - id: 1
69+ title: "Agree on the config key, defaults and the open questions below"
70+ state: open
71+ priority: none
72+ author:
73+ name: k33g
74+ email: ph.charriere@gmail.com
75+ createdAt: 2026-10-20T20:00:00.000Z
76+ updatedAt: 2026-10-20T20:00:00.000Z
77+ - id: 2
78+ title: "docsearch: chunker — Markdown (# headings, not inside code fences), AsciiDoc (= headings), plain text (blank-line paragraphs); heading path and line kept; long sections split by paragraphs — with tests"
79+ state: open
80+ priority: none
81+ author:
82+ name: k33g
83+ email: ph.charriere@gmail.com
84+ createdAt: 2026-10-20T20:00:00.000Z
85+ updatedAt: 2026-10-20T20:00:00.000Z
86+ - id: 3
87+ title: "docsearch: BM25 (k1 1.2, b 0.75) over the chunks, words folded like store.fold (case, accents), heading boost — with tests"
88+ state: open
89+ priority: none
90+ author:
91+ name: k33g
92+ email: ph.charriere@gmail.com
93+ createdAt: 2026-10-20T20:00:00.000Z
94+ updatedAt: 2026-10-20T20:00:00.000Z
95+ - id: 4
96+ title: "docsearch: walk the folders (extensions, size cap, skip hidden dirs, .git, node_modules), scanned at each call, no index — with tests"
97+ state: open
98+ priority: none
99+ author:
100+ name: k33g
101+ email: ph.charriere@gmail.com
102+ createdAt: 2026-10-20T20:00:00.000Z
103+ updatedAt: 2026-10-20T20:00:00.000Z
104+ - id: 5
105+ title: "built-in tool search_doc {query, limit?}: hits = path:line, heading path, snippet; declared only when the config enables it; never asks; ACP kind search — with tests"
106+ state: open
107+ priority: none
108+ author:
109+ name: k33g
110+ email: ph.charriere@gmail.com
111+ createdAt: 2026-10-20T20:00:00.000Z
112+ updatedAt: 2026-10-20T20:00:00.000Z
113+ - id: 6
114+ title: "Measure on a small model (llmman) over mm's own docs/ (EN+FR): found when the words are there, not called otherwise; time per call on docs/"
115+ state: open
116+ priority: none
117+ author:
118+ name: k33g
119+ email: ph.charriere@gmail.com
120+ createdAt: 2026-10-20T20:00:00.000Z
121+ updatedAt: 2026-10-20T20:00:00.000Z
122+ - id: 7
123+ title: "Docs EN+FR (reference configuration and tools, how-to, explanation: why BM25 and no index), demo if useful, quality gate, .memory"
124+ state: open
125+ priority: none
126+ author:
127+ name: k33g
128+ email: ph.charriere@gmail.com
129+ createdAt: 2026-10-20T20:00:00.000Z
130+ updatedAt: 2026-10-20T20:00:00.000Z
131+comments: []
new file mode 100644
@@ -0,0 +1,131 @@
1+# Managed by IssueSpec. Hand edits are welcome; keep the schema valid.
2+id: 106
3+title: search_doc — BM25 search in the text files of a folder (built-in tool)
4+state: open
5+author:
6+ name: k33g
7+ email: ph.charriere@gmail.com
8+createdAt: 2026-10-20T20:00:00.000Z
9+updatedAt: 2026-10-20T20:00:00.000Z
10+labels:
11+ - kind::feature
12+milestone: v0.8.0
13+epic: Builtin tools
14+related:
15+ - 104
16+ - 48
17+body: |
18+ A built-in tool, search_doc, that lets the agent search the text files of
19+ one or more folders — Markdown (.md), plain text (.txt), AsciiDoc (.adoc,
20+ .asciidoc)… — and get back the relevant passages, each with the link to
21+ its file. Ranking: BM25, no model (like `qmd search`, ticket 104).
22+
23+ User's decisions (2026-10-20):
24+ - a NEW tool, search_doc; search_memory (ticket 48) is NOT replaced and
25+ keeps its own engine;
26+ - BM25;
27+ - each passage comes with the link to its file.
28+
29+ Proposal (to confirm before coding):
30+ - Config block, off by default so no existing agent changes:
31+ docSearch:
32+ enabled: true
33+ dirs: [docs] # relative to the working directory
34+ # (the session cwd over ACP); absolute ok
35+ extensions: [.md, .markdown, .txt, .adoc, .asciidoc, .rst]
36+ maxFileSize: 1MB # bigger files skipped, banner-free
37+ - Tool search_doc {query, limit?} (limit default 5, cap 20). One hit per
38+ passage:
39+ docs/en/how-to/keep-sessions.md:46 · Keep sessions › Find out why a run failed
40+ …the text of the passage around the words found…
41+ The link is the path relative to the working directory plus the
42+ passage's first line (`path:line`), what an editor and read_file both
43+ take; the model can then read_file the whole page.
44+ - Passages = sections: a Markdown / AsciiDoc heading starts one (headings
45+ inside code fences ignored), keeping the heading path; plain text is cut
46+ at blank lines; a long section is cut by paragraphs (~1,500 chars).
47+ - Words: the same folding as search_memory (case and accents do not
48+ count). BM25 k1 1.2, b 0.75, IDF over the passages of the folders; words
49+ found in a heading weigh more.
50+ - No index: the folders are read at each call. Measured for scale: mm's
51+ docs/ = 133 .md files, 1.3 MB, ~152k words, 1,208 headings.
52+ - Never asks for permission (it only reads, like read_file); ACP tool
53+ kind `search`; output capped by maxOutput.
54+
55+ Known limits (stated up front): no stemming ("migration" ≠
56+ "migrations" unless prefix matching is added); no match across languages
57+ or synonyms — the same class of limit as search_memory (ticket 48
58+ comment 3) and as qmd's own BM25 mode.
59+
60+ Open questions:
61+ - a /docs <words> command for the user too, as /memory is for
62+ search_memory?
63+ - prefix matching for words of 4+ letters (a cheap answer to plurals), or
64+ exact words only?
65+ - other extensions by default (.rst, .org, .html)? follow symlinks?
66+ - the key name: `docSearch` (proposed) or under an existing block?
67+tasks:
68+ - id: 1
69+ title: "Agree on the config key, defaults and the open questions below"
70+ state: open
71+ priority: none
72+ author:
73+ name: k33g
74+ email: ph.charriere@gmail.com
75+ createdAt: 2026-10-20T20:00:00.000Z
76+ updatedAt: 2026-10-20T20:00:00.000Z
77+ - id: 2
78+ title: "docsearch: chunker — Markdown (# headings, not inside code fences), AsciiDoc (= headings), plain text (blank-line paragraphs); heading path and line kept; long sections split by paragraphs — with tests"
79+ state: open
80+ priority: none
81+ author:
82+ name: k33g
83+ email: ph.charriere@gmail.com
84+ createdAt: 2026-10-20T20:00:00.000Z
85+ updatedAt: 2026-10-20T20:00:00.000Z
86+ - id: 3
87+ title: "docsearch: BM25 (k1 1.2, b 0.75) over the chunks, words folded like store.fold (case, accents), heading boost — with tests"
88+ state: open
89+ priority: none
90+ author:
91+ name: k33g
92+ email: ph.charriere@gmail.com
93+ createdAt: 2026-10-20T20:00:00.000Z
94+ updatedAt: 2026-10-20T20:00:00.000Z
95+ - id: 4
96+ title: "docsearch: walk the folders (extensions, size cap, skip hidden dirs, .git, node_modules), scanned at each call, no index — with tests"
97+ state: open
98+ priority: none
99+ author:
100+ name: k33g
101+ email: ph.charriere@gmail.com
102+ createdAt: 2026-10-20T20:00:00.000Z
103+ updatedAt: 2026-10-20T20:00:00.000Z
104+ - id: 5
105+ title: "built-in tool search_doc {query, limit?}: hits = path:line, heading path, snippet; declared only when the config enables it; never asks; ACP kind search — with tests"
106+ state: open
107+ priority: none
108+ author:
109+ name: k33g
110+ email: ph.charriere@gmail.com
111+ createdAt: 2026-10-20T20:00:00.000Z
112+ updatedAt: 2026-10-20T20:00:00.000Z
113+ - id: 6
114+ title: "Measure on a small model (llmman) over mm's own docs/ (EN+FR): found when the words are there, not called otherwise; time per call on docs/"
115+ state: open
116+ priority: none
117+ author:
118+ name: k33g
119+ email: ph.charriere@gmail.com
120+ createdAt: 2026-10-20T20:00:00.000Z
121+ updatedAt: 2026-10-20T20:00:00.000Z
122+ - id: 7
123+ title: "Docs EN+FR (reference configuration and tools, how-to, explanation: why BM25 and no index), demo if useful, quality gate, .memory"
124+ state: open
125+ priority: none
126+ author:
127+ name: k33g
128+ email: ph.charriere@gmail.com
129+ createdAt: 2026-10-20T20:00:00.000Z
130+ updatedAt: 2026-10-20T20:00:00.000Z
131+comments: []