Documentation search — explanation

What is this about?

/docs and the search_docs tool (ticket 106) search the text files of some folders and return passages — a section of a page, with its heading path and its path:line link. They answer a different need from search_memory: the stored sessions hold what was said and decided; a documentation folder holds what is written and maintained. The two searches are separate tools on purpose, with separate engines.

Why BM25, and no model

The search ranks passages with BM25, the scoring of most full-text engines: a word counts more when it is rare in the folder and when it appears often in the passage, and long passages are not favoured for being long. It needs no model. The alternatives looked at (ticket 104) — qmd's vector and hybrid modes, embedding-based tools — find passages by meaning, across languages and synonyms, but load one to three extra models (300 MB to 2 GB for qmd query) next to the chat model, on machines where memory is what runs out first. qmd search, its BM25 mode, is the closest equivalent of what mm does, and it needs Node or Bun, an index to build and keep up to date.

BM25 also fixes what made the stored-session search noisy at first: there, every word counted the same, so a short common word matched nearly every message. With BM25 a word found everywhere weighs almost nothing by itself.

Why no index

The folders are read at every call. mm's own docs/ — 137 Markdown files, 1.3 MB, 1,320 passages — is read, cut and scored in about 45 ms; the ten files of the demo (44 passages) in about 1 ms (measured 2026-10-20, best of ten runs). An index would make that faster and add what an index brings: a place to store it, a step to rebuild it, and answers that are wrong between a change and the rebuild. For a documentation folder, read when a question needs it, a scan is fast enough and never stale.

Why passages are sections

A hit is meant to be read by the model, so it must be a unit of meaning that fits next to four others: a section — a heading and what follows it — is what the author wrote as one. Its heading path ("Lists and selection › Keep the selection with ListState") says what it is about even when its text does not repeat it, which is why heading words weigh more. A long section is cut at blank lines, never inside a code block: half a program helps no one. The link points at the passage's first line, so the model can read_file the whole page when a passage is not enough.

Why prefixes, and no stemming

Words of four letters or more also match longer words (list → ListState) and plurals (widgets → widget). That is a cheap answer to the commonest misses, for any language. A real stemmer would do better for one language and worse for the others, and would be one more dependency.

What it does not do

It matches words, not meanings. "Comment découper l'écran ?" finds nothing in English tutorials. In the demo, a 4B model told to search with the documentation's words translated the question itself — "header footer body layout" — every time (6 of 6 French questions); the search did not have to.

How it relates to the rest

internal/docsearch holds the engine (passages, BM25, the walk through the folders) and depends only on internal/config and internal/textfold, the case-and-accent folding it shares with the stored-session search. /docs is in the shared command table; the REPL, the window and the ACP façade each answer it from their working directory. search_docs is declared in main.go beside the other built-in tools.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# Documentation search — explanation

## What is this about?

`/docs` and the `search_docs` tool (ticket 106) search the text files of some folders and return passages — a section of a page, with its heading path and its `path:line` link. They answer a different need from [`search_memory`](persistent-sessions.md): the stored sessions hold what was said and decided; a documentation folder holds what is written and maintained. The two searches are separate tools on purpose, with separate engines.

## Why BM25, and no model

The search ranks passages with BM25, the scoring of most full-text engines: a word counts more when it is rare in the folder and when it appears often in the passage, and long passages are not favoured for being long. It needs no model. The alternatives looked at (ticket 104) — `qmd`'s vector and hybrid modes, embedding-based tools — find passages by meaning, across languages and synonyms, but load one to three extra models (300 MB to 2 GB for `qmd query`) next to the chat model, on machines where memory is what runs out first. `qmd search`, its BM25 mode, is the closest equivalent of what mm does, and it needs Node or Bun, an index to build and keep up to date.

BM25 also fixes what made the stored-session search noisy at first: there, every word counted the same, so a short common word matched nearly every message. With BM25 a word found everywhere weighs almost nothing by itself.

## Why no index

The folders are read at every call. mm's own `docs/` — 137 Markdown files, 1.3 MB, 1,320 passages — is read, cut and scored in about 45 ms; the ten files of the demo (44 passages) in about 1 ms (measured 2026-10-20, best of ten runs). An index would make that faster and add what an index brings: a place to store it, a step to rebuild it, and answers that are wrong between a change and the rebuild. For a documentation folder, read when a question needs it, a scan is fast enough and never stale.

## Why passages are sections

A hit is meant to be read by the model, so it must be a unit of meaning that fits next to four others: a section — a heading and what follows it — is what the author wrote as one. Its heading path ("Lists and selection › Keep the selection with ListState") says what it is about even when its text does not repeat it, which is why heading words weigh more. A long section is cut at blank lines, never inside a code block: half a program helps no one. The link points at the passage's first line, so the model can `read_file` the whole page when a passage is not enough.

## Why prefixes, and no stemming

Words of four letters or more also match longer words (`list` → `ListState`) and plurals (`widgets` → `widget`). That is a cheap answer to the commonest misses, for any language. A real stemmer would do better for one language and worse for the others, and would be one more dependency.

## What it does not do

It matches words, not meanings. "Comment découper l'écran ?" finds nothing in English tutorials. In the demo, a 4B model told to search with the documentation's words translated the question itself — "header footer body layout" — every time (6 of 6 French questions); the search did not have to.

## How it relates to the rest

`internal/docsearch` holds the engine (passages, BM25, the walk through the folders) and depends only on `internal/config` and `internal/textfold`, the case-and-accent folding it shares with the stored-session search. `/docs` is in the shared command table; the REPL, the window and the ACP façade each answer it from their working directory. `search_docs` is declared in `main.go` beside the other built-in tools.