Memory and local models — explanation

What is this about?

Sidekicks' agents usually run local language models: the model is loaded into the computer's memory by a server (llmman) and stays there until it is unloaded. The development machine is a MacBook Air with 32 GB: with more than two small models loaded, memory pressure stops it. This constraint shapes both the test tooling and the product.

In the tooling

The constraint used to rely on the developer's memory ("never more than two agents"). It now relies on the tool: scripts/agents.sh is the only way the project starts a test agent, and it refuses a third one. It also unloads an agent's model when the agent stops, but only if the model is actually loaded — asking llmman to unload a model that is not in memory would load it first. Bob and Riker use the same model, so it is kept while one of them still runs.

Because the guard lives in a script that passes stdio through, it also protects the agents started by an ACP client such as Zed, by the spike client, and later by Sidekicks' own tests.

In the product

The same reasoning applies to the people who will use Sidekicks on their laptops, so the architecture carries it (ADR 0003):

agents start when first called, not when the application opens;
an agent unused for a while (ten minutes by default) is stopped to free its model;
a limit on simultaneous agents (limits.maxRunningAgents, two by default) is part of the configuration;
in a channel with several agents, only the agents mentioned answer, one after the other by default, rather than every agent at once.

Rejected alternatives

Starting every configured agent at launch: simple, and unusable with four local models.
Letting each agent unload its own model: mm can (/unload), but only Sidekicks knows whether another agent still needs the same model.

How it relates to the rest

The agent manager that will enforce this in the application is internal/agents (see Architecture); the test agents themselves are presented in How to run a test agent.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


# Memory and local models — explanation

## What is this about?

Sidekicks' agents usually run **local** language models: the model is loaded into the computer's memory by a server (llmman) and stays there until it is unloaded. The development machine is a MacBook Air with 32 GB: with more than two small models loaded, memory pressure stops it. This constraint shapes both the test tooling and the product.

## In the tooling

The constraint used to rely on the developer's memory ("never more than two agents"). It now relies on the tool: [`scripts/agents.sh`](../reference/agents-script.md) is the only way the project starts a test agent, and it refuses a third one. It also unloads an agent's model when the agent stops, but only if the model is actually loaded — asking llmman to unload a model that is not in memory would load it first. Bob and Riker use the same model, so it is kept while one of them still runs.

Because the guard lives in a script that passes stdio through, it also protects the agents started by an ACP client such as Zed, by the spike client, and later by Sidekicks' own tests.

## In the product

The same reasoning applies to the people who will use Sidekicks on their laptops, so the architecture carries it ([ADR 0003](../../adr/0003-architecture.md)):

- agents start **when first called**, not when the application opens;
- an agent unused for a while (ten minutes by default) is stopped to free its model;
- a limit on simultaneous agents (`limits.maxRunningAgents`, two by default) is part of the [configuration](../reference/configuration.md);
- in a channel with several agents, only the agents **mentioned** answer, one after the other by default, rather than every agent at once.

## Rejected alternatives

- **Starting every configured agent at launch**: simple, and unusable with four local models.
- **Letting each agent unload its own model**: mm can (`/unload`), but only Sidekicks knows whether another agent still needs the same model.

## How it relates to the rest

The agent manager that will enforce this in the application is `internal/agents` (see [Architecture](architecture.md)); the test agents themselves are presented in [How to run a test agent](../how-to/run-a-test-agent.md).