The `fetch_content` tool — explanation

What is this about?

fetch_content gives the model one thing it could only do clumsily before: read a web page. With bash alone, a model runs curl, gets hundreds of kilobytes of HTML — scripts, styles, menus, tracking code — and either drowns its context window in it or summarises a page it never really saw. fetch_content downloads the page, throws away what is not content, and hands back Markdown: the format a language model reads best, and the one a user can keep.

The work is split in two. internal/webmd is a plain Go package with three stages — Fetcher.Fetch, Clean, Convert — that knows nothing about models. internal/tools/fetchcontent.go is the thin tool around it: arguments, display, permission, and the text the model gets back.

Why a built-in tool, and not the MCP `fetch` server

An MCP fetch server can be wired into mm already. The built-in tool exists for what such a server does not guarantee:

It works with nothing else installed. No Node or Python runtime, no gateway: the conversion is in the binary.
It can save without returning. "Save it to hello.txt, don't show it" is common, and a tool that always returns the page makes the model pay for the whole page in its context just to copy it into a file — which a small local model does badly anyway. output writes the file directly and returns one line.
Its conversion rules are ours. The specification it follows (see the reference) decides headings, lists, tables, escaping; the tests pin them.

The name is fetch_content, not fetch, so it never collides with an MCP tool of that name.

Why the conversion is written here

A library such as html-to-markdown was considered. The specification asks for behaviour that such libraries either do not have or make awkward: complex tables kept as raw HTML or flattened on demand, a clean-up pass driven by roles and class words with a guard for layout wrappers, links resolved against the final URL, a strict "one trailing newline, no preamble" output. Bending a general-purpose converter to all of it would have cost about as much as writing the few hundred lines, and would have added a dependency whose defaults change under us. The parsing itself is not ours: golang.org/x/net/html is the HTML5 parser, so broken HTML is repaired exactly the way a browser repairs it.

Why robots.txt is honoured

mm fetches on behalf of a person, one page at a time — closer to a browser than to a crawler. It still honours robots.txt, because the request said so and because an agent that ignores a site's stated wishes is a poor citizen however few requests it sends. It identifies itself as mm/<version>, so a site can address it by name.

The rules follow RFC 9309, including its cautious corners: a site whose robots.txt answers with a server error, or cannot be reached, is treated as closed — the RFC's choice, and the only safe reading of "we could not ask". Every redirection is checked too, so a redirect cannot lead into a forbidden path.

Why failures are text with a number

Every mm tool returns failures as text, so the model can read them and react. fetch_content adds the error code of its specification (1 conversion, 2 input, 3 usage) because the three call for different reactions: fix the call (3), give up or try another page (2), or tell the user the thing is not a web page (1). A model that sees error 2 (input) — … robots.txt does not allow "mm" has no reason to try curl next.

Choices that trade exactness for safety

Off by default. It is the only built-in tool that reaches the network; an operator turns it on knowingly.
Limits. 30 seconds, 5 MiB, 10 redirections: a tool call must not hang a turn or fill memory.
Encoding. The declared encoding wins (BOM, then HTTP header, then <meta>); when nothing is declared, valid UTF-8 is taken as UTF-8 and anything else as windows-1252 — the web's historical default.
Images as data: URIs are reduced to their alt text: kilobytes of base64 are no use to a reader, human or model.

Known limits

The clean-up is heuristic. It removes navigation, ads and banners that are marked as such; an ad without a telling class or role stays, and an unusual class name can, rarely, remove content. Anything wrapping a main, an article or an h1 is protected.
Inline elements that a page separates only with CSS come out glued together: tags rendered as <a>go</a><a>tui</a> read gotui.
Pages built by JavaScript in the browser have little content in their HTML; fetch_content does not run scripts.
Only HTML is converted: a PDF or a JSON document is refused with error 1.

How it relates to the rest

fetch_content follows the same contract as the other tools (see architecture): one display line in the terminal, an event with a kind (fetch) for ACP and the window, a permission request only when it writes a file, and a record in the loop detector. Its output goes through the same maxOutput truncation as bash.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


# The `fetch_content` tool — explanation

## What is this about?

`fetch_content` gives the model one thing it could only do clumsily before: read a web page. With `bash` alone, a model runs `curl`, gets hundreds of kilobytes of HTML — scripts, styles, menus, tracking code — and either drowns its context window in it or summarises a page it never really saw. `fetch_content` downloads the page, throws away what is not content, and hands back Markdown: the format a language model reads best, and the one a user can keep.

The work is split in two. `internal/webmd` is a plain Go package with three stages — `Fetcher.Fetch`, `Clean`, `Convert` — that knows nothing about models. `internal/tools/fetchcontent.go` is the thin tool around it: arguments, display, permission, and the text the model gets back.

## Why a built-in tool, and not the MCP `fetch` server

An MCP `fetch` server can be wired into `mm` already. The built-in tool exists for what such a server does not guarantee:

- **It works with nothing else installed.** No Node or Python runtime, no gateway: the conversion is in the binary.
- **It can save without returning.** "Save it to hello.txt, don't show it" is common, and a tool that always returns the page makes the model pay for the whole page in its context just to copy it into a file — which a small local model does badly anyway. `output` writes the file directly and returns one line.
- **Its conversion rules are ours.** The specification it follows (see the reference) decides headings, lists, tables, escaping; the tests pin them.

The name is `fetch_content`, not `fetch`, so it never collides with an MCP tool of that name.

## Why the conversion is written here

A library such as html-to-markdown was considered. The specification asks for behaviour that such libraries either do not have or make awkward: complex tables kept as raw HTML *or* flattened on demand, a clean-up pass driven by roles and class words with a guard for layout wrappers, links resolved against the final URL, a strict "one trailing newline, no preamble" output. Bending a general-purpose converter to all of it would have cost about as much as writing the few hundred lines, and would have added a dependency whose defaults change under us. The parsing itself is not ours: `golang.org/x/net/html` is the HTML5 parser, so broken HTML is repaired exactly the way a browser repairs it.

## Why robots.txt is honoured

`mm` fetches on behalf of a person, one page at a time — closer to a browser than to a crawler. It still honours `robots.txt`, because the request said so and because an agent that ignores a site's stated wishes is a poor citizen however few requests it sends. It identifies itself as `mm/<version>`, so a site can address it by name.

The rules follow RFC 9309, including its cautious corners: a site whose `robots.txt` answers with a server error, or cannot be reached, is treated as closed — the RFC's choice, and the only safe reading of "we could not ask". Every redirection is checked too, so a redirect cannot lead into a forbidden path.

## Why failures are text with a number

Every `mm` tool returns failures as text, so the model can read them and react. `fetch_content` adds the error code of its specification (1 conversion, 2 input, 3 usage) because the three call for different reactions: fix the call (3), give up or try another page (2), or tell the user the thing is not a web page (1). A model that sees `error 2 (input) — … robots.txt does not allow "mm"` has no reason to try `curl` next.

## Choices that trade exactness for safety

- **Off by default.** It is the only built-in tool that reaches the network; an operator turns it on knowingly.
- **Limits.** 30 seconds, 5 MiB, 10 redirections: a tool call must not hang a turn or fill memory.
- **Encoding.** The declared encoding wins (BOM, then HTTP header, then `<meta>`); when nothing is declared, valid UTF-8 is taken as UTF-8 and anything else as windows-1252 — the web's historical default.
- **Images as `data:` URIs** are reduced to their alt text: kilobytes of base64 are no use to a reader, human or model.

## Known limits

- The clean-up is heuristic. It removes navigation, ads and banners that are marked as such; an ad without a telling class or role stays, and an unusual class name can, rarely, remove content. Anything wrapping a `main`, an `article` or an `h1` is protected.
- Inline elements that a page separates only with CSS come out glued together: tags rendered as `<a>go</a><a>tui</a>` read `gotui`.
- Pages built by JavaScript in the browser have little content in their HTML; `fetch_content` does not run scripts.
- Only HTML is converted: a PDF or a JSON document is refused with `error 1`.

## How it relates to the rest

`fetch_content` follows the same contract as the other tools (see [architecture](architecture.md)): one display line in the terminal, an event with a kind (`fetch`) for ACP and the window, a permission request only when it writes a file, and a record in the loop detector. Its output goes through the same `maxOutput` truncation as `bash`.

The fetch_content tool — explanation