Memory and local models — explanation

What is this about?

Sidekicks' agents usually run local language models: the model is loaded into the computer's memory by a server (llmman) and stays there until it is unloaded. The development machine is a MacBook Air with 32 GB: with more than two small models loaded, memory pressure stops it. This constraint shapes both the test tooling and the product.

In the tooling

The constraint used to rely on the developer's memory ("never more than two agents"). It now relies on the tool: scripts/agents.sh is the only way the project starts a test agent, and it refuses a third one. It also unloads an agent's model when the agent stops, but only if the model is actually loaded — asking llmman to unload a model that is not in memory would load it first. Bob and Riker use the same model, so it is kept while one of them still runs.

Because the guard lives in a script that passes stdio through, it also protects the agents started by an ACP client such as Zed, by the spike client, and later by Sidekicks' own tests.

In the product

The same reasoning applies to the people who will use Sidekicks on their laptops, so the application applies it (internal/agents, milestone M1; ADR 0003):

agents start when first called, not when the application opens;
an agent unused for a while (ten minutes by default) is stopped to free its model;
a limit on simultaneous agents (limits.maxRunningAgents, two by default) is part of the configuration. When it is reached, the agent nobody has used for the longest time makes room — provided it is idle: an agent that is answering is never interrupted, and the start is refused instead;
in a channel with several agents, only the agents mentioned answer, one after the other by default, rather than every agent at once.

The price of stopping idle agents is memory of another kind: mm cannot reload a session, so an agent restarted after a pause no longer remembers the beginning of the conversation. Sidekicks says so in the conversation (milestone M2) and will give the agent the context back with channels (issue #25).

Rejected alternatives

Starting every configured agent at launch: simple, and unusable with four local models.
Letting each agent unload its own model: mm can (/unload), but only Sidekicks knows whether another agent still needs the same model.

How it relates to the rest

The agent manager that will enforce this in the application is internal/agents (see Architecture); the test agents themselves are presented in How to run a test agent.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# Memory and local models — explanation

## What is this about?

Sidekicks' agents usually run **local** language models: the model is loaded into the computer's memory by a server (llmman) and stays there until it is unloaded. The development machine is a MacBook Air with 32 GB: with more than two small models loaded, memory pressure stops it. This constraint shapes both the test tooling and the product.

## In the tooling

The constraint used to rely on the developer's memory ("never more than two agents"). It now relies on the tool: [`scripts/agents.sh`](../reference/agents-script.md) is the only way the project starts a test agent, and it refuses a third one. It also unloads an agent's model when the agent stops, but only if the model is actually loaded — asking llmman to unload a model that is not in memory would load it first. Bob and Riker use the same model, so it is kept while one of them still runs.

Because the guard lives in a script that passes stdio through, it also protects the agents started by an ACP client such as Zed, by the spike client, and later by Sidekicks' own tests.

## In the product

The same reasoning applies to the people who will use Sidekicks on their laptops, so the application applies it (`internal/agents`, milestone M1; [ADR 0003](../../adr/0003-architecture.md)):

- agents start **when first called**, not when the application opens;
- an agent unused for a while (ten minutes by default) is stopped to free its model;
- a limit on simultaneous agents (`limits.maxRunningAgents`, two by default) is part of the [configuration](../reference/configuration.md). When it is reached, the agent nobody has used for the longest time makes room — provided it is idle: an agent that is answering is never interrupted, and the start is refused instead;
- in a channel with several agents, only the agents **mentioned** answer, one after the other by default, rather than every agent at once.

The price of stopping idle agents is memory of another kind: mm cannot reload a session, so an agent restarted after a pause no longer remembers the beginning of the conversation. Sidekicks says so in the conversation (milestone M2) and will give the agent the context back with channels (issue #25).

## Rejected alternatives

- **Starting every configured agent at launch**: simple, and unusable with four local models.
- **Letting each agent unload its own model**: mm can (`/unload`), but only Sidekicks knows whether another agent still needs the same model.

## How it relates to the rest

The agent manager that will enforce this in the application is `internal/agents` (see [Architecture](architecture.md)); the test agents themselves are presented in [How to run a test agent](../how-to/run-a-test-agent.md).