Providers: one implementation per protocol — explanation

What is this about?

mm talks to a local model server through the OpenAI chat-completions API. Two servers are supported out of the box, Docker Model Runner and llama.cpp's llama-server, under the names dmr and llamacpp. The engine package used to be tied to Docker Model Runner; the provider abstraction is what generalised it.

Why it is designed this way

One provider type per wire protocol, not per vendor. When the engine was split out, exactly one function and two string constants were specific to Docker Model Runner. Everything else, the streaming, the watchdog, the retry, the command counter, behaved the same against a fake engine that is not DMR either. So dmr and llamacpp are two data entries of a single openaiCompat type: a default base URL, an optional fallback URL, a key policy, and the words used in error messages. Adding another OpenAI-compatible server is a registry entry; adding a different protocol would be a new type.

A provider owns four things. It resolves the configuration into a concrete backend (URL after environment overrides and fallback probe, key read from the environment, model name); it opens Genkit with its plugin; it probes the server before the first question; and it explains a transport error in one line.

The fallback exists for containers. On the host, Docker Model Runner listens on localhost:12434; from inside a container or a sandbox the same server is reachable as host.docker.internal. If the primary URL does not answer /models within two seconds, the fallback is used. An explicit environment override skips the probe: whoever set it knows where the server is. An explicit empty fallback: "" disables it, which is what a test configuration needs to stay pinned to one URL.

The key never goes in the YAML. The file names the environment variable that holds the key, so it can be committed and shown on screen. Servers that ignore the header get a placeholder, because the OpenAI client wants something there. A 401 is then explained by naming the variable the provider actually read, or, when none is configured, by pointing at the URL.

The probe is advice, never a failure. On a demo machine the server is often started after the agent. The probe checks reachability, warns when the configured model is not in DMR's list, and on llama.cpp reads the served context size from /props, one level above /v1, because that endpoint reports what the server serves, not what the model was trained with. When the window is still unknown and compression is on, the REPL probes again before the first question.

Errors are rewritten in the server's own words. A stack of wrapped messages is noise; "tool calls need llama-server started with --jinja" is the whole diagnosis. The provider recognises connection refusals, the missing --jinja flag, 401/403, 404 with a pull or alias hint, 429 and 5xx. Unknown errors come back unchanged: a raw message beats a wrong hint.

Rejected alternatives

A provider per vendor with duplicated engine code. Rejected: measured, the vendor-specific part was three identifiers.
Putting the API key in the configuration file. Rejected because the file is meant to be committed and shown.
Failing at start-up when the server is down. Rejected because starting the server second is the normal demo situation.

How it relates to the rest

The two configuration files that parametrise each provider are listed in the configuration reference.
The window the probe learns is what context compression measures against.
Switching servers for one run is covered in how to run on llama.cpp.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# Providers: one implementation per protocol — explanation

## What is this about?

`mm` talks to a local model server through the OpenAI chat-completions API. Two servers are supported out of the box, Docker Model Runner and llama.cpp's `llama-server`, under the names `dmr` and `llamacpp`. The engine package used to be tied to Docker Model Runner; the provider abstraction is what generalised it.

## Why it is designed this way

**One provider type per wire protocol, not per vendor.** When the engine was split out, exactly one function and two string constants were specific to Docker Model Runner. Everything else, the streaming, the watchdog, the retry, the command counter, behaved the same against a fake engine that is not DMR either. So `dmr` and `llamacpp` are two data entries of a single `openaiCompat` type: a default base URL, an optional fallback URL, a key policy, and the words used in error messages. Adding another OpenAI-compatible server is a registry entry; adding a different protocol would be a new type.

**A provider owns four things.** It resolves the configuration into a concrete backend (URL after environment overrides and fallback probe, key read from the environment, model name); it opens Genkit with its plugin; it probes the server before the first question; and it explains a transport error in one line.

**The fallback exists for containers.** On the host, Docker Model Runner listens on `localhost:12434`; from inside a container or a sandbox the same server is reachable as `host.docker.internal`. If the primary URL does not answer `/models` within two seconds, the fallback is used. An explicit environment override skips the probe: whoever set it knows where the server is. An explicit empty `fallback: ""` disables it, which is what a test configuration needs to stay pinned to one URL.

**The key never goes in the YAML.** The file names the environment variable that holds the key, so it can be committed and shown on screen. Servers that ignore the header get a placeholder, because the OpenAI client wants something there. A 401 is then explained by naming the variable the provider actually read, or, when none is configured, by pointing at the URL.

**The probe is advice, never a failure.** On a demo machine the server is often started after the agent. The probe checks reachability, warns when the configured model is not in DMR's list, and on llama.cpp reads the served context size from `/props`, one level above `/v1`, because that endpoint reports what the server serves, not what the model was trained with. When the window is still unknown and compression is on, the REPL probes again before the first question.

**Errors are rewritten in the server's own words.** A stack of wrapped messages is noise; "tool calls need llama-server started with --jinja" is the whole diagnosis. The provider recognises connection refusals, the missing `--jinja` flag, 401/403, 404 with a pull or alias hint, 429 and 5xx. Unknown errors come back unchanged: a raw message beats a wrong hint.

## Rejected alternatives

- **A provider per vendor with duplicated engine code.** Rejected: measured, the vendor-specific part was three identifiers.
- **Putting the API key in the configuration file.** Rejected because the file is meant to be committed and shown.
- **Failing at start-up when the server is down.** Rejected because starting the server second is the normal demo situation.

## How it relates to the rest

- The two configuration files that parametrise each provider are listed in the [configuration reference](../reference/configuration.md).
- The window the probe learns is what [context compression](context-compression.md) measures against.
- Switching servers for one run is covered in [how to run on llama.cpp](../how-to/run-with-llama-cpp.md).