Providers: one implementation per protocol — explanation

What is this about?

mm talks to a local model server through the OpenAI chat-completions API. Two servers are supported out of the box, Docker Model Runner and llama.cpp's llama-server, under the names dmr and llamacpp. The engine package used to be tied to Docker Model Runner; the provider abstraction is what generalised it.

Why it is designed this way

One provider type per wire protocol, not per vendor. When the engine was split out, exactly one function and two string constants were specific to Docker Model Runner. Everything else, the streaming, the watchdog, the retry, the command counter, behaved the same against a fake engine that is not DMR either. So dmr and llamacpp are two data entries of a single openaiCompat type: a default base URL, an optional fallback URL, a key policy, and the words used in error messages. Adding another OpenAI-compatible server is a registry entry; adding a different protocol would be a new type.

A provider owns four things. It resolves the configuration into a concrete backend (URL after environment overrides and fallback probe, key read from the environment, model name); it opens Genkit with its plugin; it probes the server before the first question; and it explains a transport error in one line.

The fallback exists for containers. On the host, Docker Model Runner listens on localhost:12434; from inside a container or a sandbox the same server is reachable as host.docker.internal. If the primary URL does not answer /models within two seconds, the fallback is used. An explicit environment override skips the probe: whoever set it knows where the server is. An explicit empty fallback: "" disables it, which is what a test configuration needs to stay pinned to one URL.

The key never goes in the YAML. The file names the environment variable that holds the key, so it can be committed and shown on screen. Servers that ignore the header get a placeholder, because the OpenAI client wants something there. A 401 is then explained by naming the variable the provider actually read, or, when none is configured, by pointing at the URL.

The probe is advice, never a failure. On a demo machine the server is often started after the agent. The probe checks reachability, warns when the configured model is not in DMR's list, and on llama.cpp reads the served context size from /props, one level above /v1, because that endpoint reports what the server serves, not what the model was trained with. When the window is still unknown and compression is on, the REPL probes again before the first question.

Errors are rewritten in the server's own words. A stack of wrapped messages is noise; "tool calls need llama-server started with --jinja" is the whole diagnosis. The provider recognises connection refusals, the missing --jinja flag, 401/403, 404 with a pull or alias hint, 429 and 5xx. Unknown errors come back unchanged: a raw message beats a wrong hint.

A stream the server breaks is retried, and its words are kept. llama.cpp reports a failure in the middle of a streamed answer as an SSE error: event (the 200 status is already sent). openai-go only reads data: lines, so the message used to vanish behind unexpected end of JSON input. The tap that reads the stream for reasoning also picks up those events, and the model call that broke is retried once — that call only: the tools that ran earlier in the turn are not replayed, and text the broken attempt already streamed is not taken back. If it breaks again, the error shows the server's own message (the model server reported an error while generating: Failed to parse input at pos …), or says the connection broke. Either way the history is kept: asking to continue picks the work up.

Rejected alternatives

A provider per vendor with duplicated engine code. Rejected: measured, the vendor-specific part was three identifiers.
Putting the API key in the configuration file. Rejected because the file is meant to be committed and shown.
Failing at start-up when the server is down. Rejected because starting the server second is the normal demo situation.

How it relates to the rest

The two configuration files that parametrise each provider are listed in the configuration reference.
The window the probe learns is what context compression measures against.
Switching servers for one run is covered in how to run on llama.cpp.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


# Providers: one implementation per protocol — explanation

## What is this about?

`mm` talks to a local model server through the OpenAI chat-completions API. Two servers are supported out of the box, Docker Model Runner and llama.cpp's `llama-server`, under the names `dmr` and `llamacpp`. The engine package used to be tied to Docker Model Runner; the provider abstraction is what generalised it.

## Why it is designed this way

**One provider type per wire protocol, not per vendor.** When the engine was split out, exactly one function and two string constants were specific to Docker Model Runner. Everything else, the streaming, the watchdog, the retry, the command counter, behaved the same against a fake engine that is not DMR either. So `dmr` and `llamacpp` are two data entries of a single `openaiCompat` type: a default base URL, an optional fallback URL, a key policy, and the words used in error messages. Adding another OpenAI-compatible server is a registry entry; adding a different protocol would be a new type.

**A provider owns four things.** It resolves the configuration into a concrete backend (URL after environment overrides and fallback probe, key read from the environment, model name); it opens Genkit with its plugin; it probes the server before the first question; and it explains a transport error in one line.

**The fallback exists for containers.** On the host, Docker Model Runner listens on `localhost:12434`; from inside a container or a sandbox the same server is reachable as `host.docker.internal`. If the primary URL does not answer `/models` within two seconds, the fallback is used. An explicit environment override skips the probe: whoever set it knows where the server is. An explicit empty `fallback: ""` disables it, which is what a test configuration needs to stay pinned to one URL.

**The key never goes in the YAML.** The file names the environment variable that holds the key, so it can be committed and shown on screen. Servers that ignore the header get a placeholder, because the OpenAI client wants something there. A 401 is then explained by naming the variable the provider actually read, or, when none is configured, by pointing at the URL.

**The probe is advice, never a failure.** On a demo machine the server is often started after the agent. The probe checks reachability, warns when the configured model is not in DMR's list, and on llama.cpp reads the served context size from `/props`, one level above `/v1`, because that endpoint reports what the server serves, not what the model was trained with. When the window is still unknown and compression is on, the REPL probes again before the first question.

**Errors are rewritten in the server's own words.** A stack of wrapped messages is noise; "tool calls need llama-server started with --jinja" is the whole diagnosis. The provider recognises connection refusals, the missing `--jinja` flag, 401/403, 404 with a pull or alias hint, 429 and 5xx. Unknown errors come back unchanged: a raw message beats a wrong hint.

**A stream the server breaks is retried, and its words are kept.** llama.cpp reports a failure in the middle of a streamed answer as an SSE `error:` event (the 200 status is already sent). openai-go only reads `data:` lines, so the message used to vanish behind `unexpected end of JSON input`. The tap that reads the stream for reasoning also picks up those events, and the model call that broke is retried once — that call only: the tools that ran earlier in the turn are not replayed, and text the broken attempt already streamed is not taken back. If it breaks again, the error shows the server's own message (`the model server reported an error while generating: Failed to parse input at pos …`), or says the connection broke. Either way the history is kept: asking to continue picks the work up.

## Rejected alternatives

- **A provider per vendor with duplicated engine code.** Rejected: measured, the vendor-specific part was three identifiers.
- **Putting the API key in the configuration file.** Rejected because the file is meant to be committed and shown.
- **Failing at start-up when the server is down.** Rejected because starting the server second is the normal demo situation.

## How it relates to the rest

- The two configuration files that parametrise each provider are listed in the [configuration reference](../reference/configuration.md).
- The window the probe learns is what [context compression](context-compression.md) measures against.
- Switching servers for one run is covered in [how to run on llama.cpp](../how-to/run-with-llama-cpp.md).