bots-garden/mini-me · context-compression.md

⑂ main

context-compression.md · 39 lines · 3.8 KBmarkdown    Blame HistoryRaw
   1# Context compression — explanation
3## What is this about?
5Nothing in an agent loop ever shortens the history. Measured against a fake engine, the messages sent to the model grew 2 → 5 → 7 → 9 over four requests, and every `bash` output, up to `maxOutput` characters, and every loaded skill stay until `/quit`. On a local model the context window is fixed when the server loads, and the whole history is re-read at every turn on the laptop's hardware. A long session therefore first shows up as a false watchdog stall, then as an error from the server.
7The `compact` package replaces the old question turns by one summary written by the model itself, and keeps the last few turns raw.
9## Why it is designed this way
11**A hybrid: summary plus raw recent turns.** The recent turns are where the model is working; summarising them makes it re-run what it just did. The default keeps three question turns, enough for "do X", "now fix it", "and test it". Everything older becomes a pair of messages inserted right after the system prompt: a user message carrying the notes, and a short model acknowledgement.
13**The cut falls on a turn boundary.** A tool call and its result must never be split; reusing whole turns guarantees it.
15**Two triggers, because the window is not always known.** The primary trigger is a share of the context window, 75 % by default, leaving a quarter for the next question, its tool outputs and the answer. The server's own input-token count from the last call is preferred to the local estimate when it is larger, because it sees the tools' JSON and the chat template. When the window is unknown, Docker Model Runner tells nothing and `contextWindow` may be `0`, only the message-count trigger can fire, and the agent says so at start-up.
17**One key for the window.** The banner and the trigger read the same `contextWindow`; two keys for the same number would drift apart.
19**Earlier summaries are merged, not re-summarised.** The two inserted messages carry a marker; a later compression recognises them and folds the new old turns into the existing summary, because summarising a summary loses detail exponentially.
21**A fixed-shape summary.** The built-in prompt asks for seven sections with exact paths, command lines and error messages, and forbids quoting file contents: a file may have changed since it was read, and the system prompt already forbids stating what a command in this answer did not return. The summary request replaces the agent's own system prompt with a note-taker's, because "you are a coding agent, run commands" sent with a history full of tool calls invites the model to keep acting.
23**Failure costs nothing.** If the summary request fails, server down, watchdog, empty result, the history is left untouched and the next question goes out as before. The compression exists to keep the next question possible; it must never cost one.
25**The watchdog is twice as patient here.** The summary is the longest prefill of the session and nothing streams while it is written.
27**`/compact` skips the threshold, not the boundary.** It still keeps the last turns and says so when nothing is older. Compression only ever runs between two generations, because Genkit holds its own copy of the conversation while a turn runs.
29## Rejected alternatives
31- **Truncating the oldest messages.** Rejected: it would drop tool results whose calls remain, and lose the goal of the session.
32- **Summarising everything, recent turns included.** Rejected: the model then redoes its last steps.
33- **A separate `context.contextWindow` key.** Rejected in favour of the single top-level key.
35## How it relates to the rest
37- Turning it on and reading the report: [how to keep a long session inside the context window](../how-to/manage-the-context-window.md).
38- The `context.*` keys: [configuration reference](../reference/configuration.md).
39- Where the window comes from: [providers](providers.md).

💾 Saved. d722711 k33g 6h ago	1	# Context compression — explanation
	2
	3	## What is this about?
	4
	5	Nothing in an agent loop ever shortens the history. Measured against a fake engine, the messages sent to the model grew 2 → 5 → 7 → 9 over four requests, and every `bash` output, up to `maxOutput` characters, and every loaded skill stay until `/quit`. On a local model the context window is fixed when the server loads, and the whole history is re-read at every turn on the laptop's hardware. A long session therefore first shows up as a false watchdog stall, then as an error from the server.
	6
	7	The `compact` package replaces the old question turns by one summary written by the model itself, and keeps the last few turns raw.
	8
	9	## Why it is designed this way
	10
	11	A hybrid: summary plus raw recent turns. The recent turns are where the model is working; summarising them makes it re-run what it just did. The default keeps three question turns, enough for "do X", "now fix it", "and test it". Everything older becomes a pair of messages inserted right after the system prompt: a user message carrying the notes, and a short model acknowledgement.
	12
	13	The cut falls on a turn boundary. A tool call and its result must never be split; reusing whole turns guarantees it.
	14
	15	Two triggers, because the window is not always known. The primary trigger is a share of the context window, 75 % by default, leaving a quarter for the next question, its tool outputs and the answer. The server's own input-token count from the last call is preferred to the local estimate when it is larger, because it sees the tools' JSON and the chat template. When the window is unknown, Docker Model Runner tells nothing and `contextWindow` may be `0`, only the message-count trigger can fire, and the agent says so at start-up.
	16
	17	One key for the window. The banner and the trigger read the same `contextWindow`; two keys for the same number would drift apart.
	18
	19	Earlier summaries are merged, not re-summarised. The two inserted messages carry a marker; a later compression recognises them and folds the new old turns into the existing summary, because summarising a summary loses detail exponentially.
	20
	21	A fixed-shape summary. The built-in prompt asks for seven sections with exact paths, command lines and error messages, and forbids quoting file contents: a file may have changed since it was read, and the system prompt already forbids stating what a command in this answer did not return. The summary request replaces the agent's own system prompt with a note-taker's, because "you are a coding agent, run commands" sent with a history full of tool calls invites the model to keep acting.
	22
	23	Failure costs nothing. If the summary request fails, server down, watchdog, empty result, the history is left untouched and the next question goes out as before. The compression exists to keep the next question possible; it must never cost one.
	24
	25	The watchdog is twice as patient here. The summary is the longest prefill of the session and nothing streams while it is written.
	26
	27	`/compact` skips the threshold, not the boundary. It still keeps the last turns and says so when nothing is older. Compression only ever runs between two generations, because Genkit holds its own copy of the conversation while a turn runs.
	28
	29	## Rejected alternatives
	30
	31	- Truncating the oldest messages. Rejected: it would drop tool results whose calls remain, and lose the goal of the session.
	32	- Summarising everything, recent turns included. Rejected: the model then redoes its last steps.
	33	- A separate `context.contextWindow` key. Rejected in favour of the single top-level key.
	34
	35	## How it relates to the rest
	36
	37	- Turning it on and reading the report: [how to keep a long session inside the context window](../how-to/manage-the-context-window.md).
	38	- The `context.*` keys: [configuration reference](../reference/configuration.md).
	39	- Where the window comes from: [providers](providers.md).