Providers: one implementation per protocol — explanation

What is this about?

mm talks to a local model server through the OpenAI chat-completions API. Four servers are supported out of the box: Docker Model Runner, llama.cpp's llama-server, llmman and Ollama, under the names dmr, llamacpp, llmman and ollama. The engine package used to be tied to Docker Model Runner; the provider abstraction is what generalised it.

Why it is designed this way

One provider type per wire protocol, not per vendor. When the engine was split out, exactly one function and two string constants were specific to Docker Model Runner. Everything else, the streaming, the watchdog, the retry, the command counter, behaved the same against a fake engine that is not DMR either. So dmr and llamacpp are two data entries of a single openaiCompat type: a default base URL, an optional fallback URL, a key policy, and the words used in error messages. Adding another OpenAI-compatible server is a registry entry; adding a different protocol would be a new type.

A provider owns four things. It resolves the configuration into a concrete backend (URL after environment overrides and fallback probe, key read from the environment, model name); it opens Genkit with its plugin; it probes the server before the first question; and it explains a transport error in one line.

The fallback exists for containers. On the host, Docker Model Runner listens on localhost:12434; from inside a container or a sandbox the same server is reachable as host.docker.internal. If the primary URL does not answer /models within two seconds, the fallback is used. An explicit environment override skips the probe: whoever set it knows where the server is. An explicit empty fallback: "" disables it, which is what a test configuration needs to stay pinned to one URL.

The key never goes in the YAML. The file names the environment variable that holds the key, so it can be committed and shown on screen. Servers that ignore the header get a placeholder, because the OpenAI client wants something there. A 401 is then explained by naming the variable the provider actually read, or, when none is configured, by pointing at the URL.

The probe is advice, never a failure. On a demo machine the server is often started after the agent. The probe checks reachability, warns when the configured model is not in DMR's list, and on llama.cpp reads the served context size from /props, one level above /v1, because that endpoint reports what the server serves, not what the model was trained with. When the window is still unknown and compression is on, the REPL probes again before the first question.

llmman is a third registry entry, with its own probe. llmman (llmman serve, port 17434) is a daemon in front of one llama-server per loaded model, and speaks several APIs; mm uses only its OpenAI one. Everything llama.cpp-specific passes through it — prompt progress, reasoning, enable_thinking: false for summaries — so it gets the same treatment as llamacpp. Its /props, though, is a stub (n_ctx 4096, no chat template, measured on llmman 0.1.498), so the probe reads the served window from /api/ps instead, for the configured model, once it is loaded. The trained context from /api/show is not used: LLMMAN_CONTEXT_LENGTH can serve less, and a window too large means compression that never triggers before the server refuses the prompt. A model missing from /v1/models is only a warning: llmman pulls it on the first request.

Ollama is a fourth entry, and shows why each server keeps its own. Ollama's /v1 looks like the others, but measured on Ollama 0.35.0 it differs in three ways. Reasoning streams as delta.reasoning, which the tap already reads. No prompt progress comes back, so return_progress is not sent. And chat_template_kwargs is ignored: the summary of a compaction reasoned until its token limit and came back empty; reasoning_effort: "none" is what Ollama obeys, and with it the same compaction succeeded. That is why how to turn reasoning off is a per-provider field, not something implied by "is llama.cpp". mm uses it for summaries only; for every turn, the user writes the same server-specific key in sampling, which forwards keys the OpenAI API does not know since Genkit 1.12 (see How to turn a model's reasoning off). Like llmman, Ollama reports the served window on /api/ps once the model is loaded (the two share that reading); unlike llmman, it does not pull a missing model, so the probe's warning says ollama pull.

Errors are rewritten in the server's own words. A stack of wrapped messages is noise; "tool calls need llama-server started with --jinja" is the whole diagnosis. The provider recognises connection refusals, the missing --jinja flag, 401/403, 404 with a pull or alias hint, 429 and 5xx. Unknown errors come back unchanged: a raw message beats a wrong hint.

A stream the server breaks is retried, and its words are kept. llama.cpp reports a failure in the middle of a streamed answer as an SSE error: event (the 200 status is already sent). openai-go only reads data: lines, so the message used to vanish behind unexpected end of JSON input. The tap that reads the stream for reasoning also picks up those events, and the model call that broke is retried once — that call only: the tools that ran earlier in the turn are not replayed, and text the broken attempt already streamed is not taken back. If it breaks again, the error shows the server's own message (the model server reported an error while generating: Failed to parse input at pos …), or says the connection broke. Either way the history is kept: asking to continue picks the work up.

The tools go out in the same order on every request. A llama.cpp server keeps the prompt it has just read and, on the next request, reads again only what changed after the longest common beginning. The chat template puts the tools near the top — in the system block for Qwen-style templates — so their order decides how much of the history survives. Genkit builds the request's tool list from a Go map, whose order is random: each question of a session sent the same tools in another order, and the server read the whole prompt again (ticket 53; still the case in Genkit v1.13.1). mm now sorts the tools by name just before each model call. Measured with Mellum 12B on Docker Model Runner: from the second question on, a question reads about 20 new tokens in a quarter of a second, where it re-read about 3,400 tokens in seven seconds whenever the order changed.

Rejected alternatives

A provider per vendor with duplicated engine code. Rejected: measured, the vendor-specific part was three identifiers.
Putting the API key in the configuration file. Rejected because the file is meant to be committed and shown.
Failing at start-up when the server is down. Rejected because starting the server second is the normal demo situation.
Keeping the order in which main.go declares the tools. Rejected: Genkit loses that order before the request is built, and an order that depends on declaration would change with the configuration and with MCP servers anyway. The name is the one key that depends on the tools alone.
Waiting for a Genkit release that fixes it. Rejected for now: v1.13.1 and the main branch still build the list from a map.

How it relates to the rest

The two configuration files that parametrise each provider are listed in the configuration reference.
The window the probe learns is what context compression measures against.
Switching servers for one run is covered in how to run on llama.cpp how to run on llmman and how to run on Ollama.
Checking the tool order and measuring the cache gain on your own server: how to check that the tools go out in a stable order.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


# Providers: one implementation per protocol — explanation

## What is this about?

`mm` talks to a local model server through the OpenAI chat-completions API. Four servers are supported out of the box: Docker Model Runner, llama.cpp's `llama-server`, llmman and Ollama, under the names `dmr`, `llamacpp`, `llmman` and `ollama`. The engine package used to be tied to Docker Model Runner; the provider abstraction is what generalised it.

## Why it is designed this way

**One provider type per wire protocol, not per vendor.** When the engine was split out, exactly one function and two string constants were specific to Docker Model Runner. Everything else, the streaming, the watchdog, the retry, the command counter, behaved the same against a fake engine that is not DMR either. So `dmr` and `llamacpp` are two data entries of a single `openaiCompat` type: a default base URL, an optional fallback URL, a key policy, and the words used in error messages. Adding another OpenAI-compatible server is a registry entry; adding a different protocol would be a new type.

**A provider owns four things.** It resolves the configuration into a concrete backend (URL after environment overrides and fallback probe, key read from the environment, model name); it opens Genkit with its plugin; it probes the server before the first question; and it explains a transport error in one line.

**The fallback exists for containers.** On the host, Docker Model Runner listens on `localhost:12434`; from inside a container or a sandbox the same server is reachable as `host.docker.internal`. If the primary URL does not answer `/models` within two seconds, the fallback is used. An explicit environment override skips the probe: whoever set it knows where the server is. An explicit empty `fallback: ""` disables it, which is what a test configuration needs to stay pinned to one URL.

**The key never goes in the YAML.** The file names the environment variable that holds the key, so it can be committed and shown on screen. Servers that ignore the header get a placeholder, because the OpenAI client wants something there. A 401 is then explained by naming the variable the provider actually read, or, when none is configured, by pointing at the URL.

**The probe is advice, never a failure.** On a demo machine the server is often started after the agent. The probe checks reachability, warns when the configured model is not in DMR's list, and on llama.cpp reads the served context size from `/props`, one level above `/v1`, because that endpoint reports what the server serves, not what the model was trained with. When the window is still unknown and compression is on, the REPL probes again before the first question.

**llmman is a third registry entry, with its own probe.** [llmman](https://github.com/llmmanorg/llmman) (`llmman serve`, port 17434) is a daemon in front of one `llama-server` per loaded model, and speaks several APIs; `mm` uses only its OpenAI one. Everything llama.cpp-specific passes through it — prompt progress, reasoning, `enable_thinking: false` for summaries — so it gets the same treatment as `llamacpp`. Its `/props`, though, is a stub (`n_ctx` 4096, no chat template, measured on llmman 0.1.498), so the probe reads the served window from `/api/ps` instead, for the configured model, once it is loaded. The trained context from `/api/show` is not used: `LLMMAN_CONTEXT_LENGTH` can serve less, and a window too large means compression that never triggers before the server refuses the prompt. A model missing from `/v1/models` is only a warning: llmman pulls it on the first request.

**Ollama is a fourth entry, and shows why each server keeps its own.** Ollama's `/v1` looks like the others, but measured on Ollama 0.35.0 it differs in three ways. Reasoning streams as `delta.reasoning`, which the tap already reads. No prompt progress comes back, so `return_progress` is not sent. And `chat_template_kwargs` is ignored: the summary of a compaction reasoned until its token limit and came back empty; `reasoning_effort: "none"` is what Ollama obeys, and with it the same compaction succeeded. That is why how to turn reasoning off is a per-provider field, not something implied by "is llama.cpp". `mm` uses it for summaries only; for every turn, the user writes the same server-specific key in `sampling`, which forwards keys the OpenAI API does not know since Genkit 1.12 (see [How to turn a model's reasoning off](../how-to/turn-reasoning-off.md)). Like llmman, Ollama reports the served window on `/api/ps` once the model is loaded (the two share that reading); unlike llmman, it does not pull a missing model, so the probe's warning says `ollama pull`.

**Errors are rewritten in the server's own words.** A stack of wrapped messages is noise; "tool calls need llama-server started with --jinja" is the whole diagnosis. The provider recognises connection refusals, the missing `--jinja` flag, 401/403, 404 with a pull or alias hint, 429 and 5xx. Unknown errors come back unchanged: a raw message beats a wrong hint.

**A stream the server breaks is retried, and its words are kept.** llama.cpp reports a failure in the middle of a streamed answer as an SSE `error:` event (the 200 status is already sent). openai-go only reads `data:` lines, so the message used to vanish behind `unexpected end of JSON input`. The tap that reads the stream for reasoning also picks up those events, and the model call that broke is retried once — that call only: the tools that ran earlier in the turn are not replayed, and text the broken attempt already streamed is not taken back. If it breaks again, the error shows the server's own message (`the model server reported an error while generating: Failed to parse input at pos …`), or says the connection broke. Either way the history is kept: asking to continue picks the work up.

**The tools go out in the same order on every request.** A llama.cpp server keeps the prompt it has just read and, on the next request, reads again only what changed after the longest common beginning. The chat template puts the tools near the top — in the system block for Qwen-style templates — so their order decides how much of the history survives. Genkit builds the request's tool list from a Go map, whose order is random: each question of a session sent the same tools in another order, and the server read the whole prompt again (ticket 53; still the case in Genkit v1.13.1). `mm` now sorts the tools by name just before each model call. Measured with Mellum 12B on Docker Model Runner: from the second question on, a question reads about 20 new tokens in a quarter of a second, where it re-read about 3,400 tokens in seven seconds whenever the order changed.

## Rejected alternatives

- **A provider per vendor with duplicated engine code.** Rejected: measured, the vendor-specific part was three identifiers.
- **Putting the API key in the configuration file.** Rejected because the file is meant to be committed and shown.
- **Failing at start-up when the server is down.** Rejected because starting the server second is the normal demo situation.
- **Keeping the order in which `main.go` declares the tools.** Rejected: Genkit loses that order before the request is built, and an order that depends on declaration would change with the configuration and with MCP servers anyway. The name is the one key that depends on the tools alone.
- **Waiting for a Genkit release that fixes it.** Rejected for now: v1.13.1 and the main branch still build the list from a map.

## How it relates to the rest

- The two configuration files that parametrise each provider are listed in the [configuration reference](../reference/configuration.md).
- The window the probe learns is what [context compression](context-compression.md) measures against.
- Switching servers for one run is covered in [how to run on llama.cpp](../how-to/run-with-llama-cpp.md) [how to run on llmman](../how-to/run-with-llmman.md) and [how to run on Ollama](../how-to/run-with-ollama.md).
- Checking the tool order and measuring the cache gain on your own server: [how to check that the tools go out in a stable order](../how-to/verify-tool-order.md).