bots-garden/mini-me · f1179ac

modified docs/en/explanation/providers.md +2 -0 Viewed

@@ -18,6 +18,8 @@
18	18
19	19	Errors are rewritten in the server's own words. A stack of wrapped messages is noise; "tool calls need llama-server started with --jinja" is the whole diagnosis. The provider recognises connection refusals, the missing `--jinja` flag, 401/403, 404 with a pull or alias hint, 429 and 5xx. Unknown errors come back unchanged: a raw message beats a wrong hint.
20	20
	21	+A stream the server breaks is retried, and its words are kept. llama.cpp reports a failure in the middle of a streamed answer as an SSE `error:` event (the 200 status is already sent). openai-go only reads `data:` lines, so the message used to vanish behind `unexpected end of JSON input`. The tap that reads the stream for reasoning also picks up those events, and the model call that broke is retried once — that call only: the tools that ran earlier in the turn are not replayed, and text the broken attempt already streamed is not taken back. If it breaks again, the error shows the server's own message (`the model server reported an error while generating: Failed to parse input at pos …`), or says the connection broke. Either way the history is kept: asking to continue picks the work up.
	22	+
21	23	## Rejected alternatives
22	24
23	25	- A provider per vendor with duplicated engine code. Rejected: measured, the vendor-specific part was three identifiers.

  **Errors are rewritten in the server's own words.** A stack of wrapped messages is noise; "tool calls need llama-server started with --jinja" is the whole diagnosis. The provider recognises connection refusals, the missing `--jinja` flag, 401/403, 404 with a pull or alias hint, 429 and 5xx. Unknown errors come back unchanged: a raw message beats a wrong hint.
++**A stream the server breaks is retried, and its words are kept.** llama.cpp reports a failure in the middle of a streamed answer as an SSE `error:` event (the 200 status is already sent). openai-go only reads `data:` lines, so the message used to vanish behind `unexpected end of JSON input`. The tap that reads the stream for reasoning also picks up those events, and the model call that broke is retried once — that call only: the tools that ran earlier in the turn are not replayed, and text the broken attempt already streamed is not taken back. If it breaks again, the error shows the server's own message (`the model server reported an error while generating: Failed to parse input at pos …`), or says the connection broke. Either way the history is kept: asking to continue picks the work up.
++
  ## Rejected alternatives
  - **A provider per vendor with duplicated engine code.** Rejected: measured, the vendor-specific part was three identifiers.

modified docs/fr/explanation/petits-llms.adoc +2 -0 Viewed

@@ -236,6 +236,7 @@ Un petit modèle choisit mal une procédure sur son seul nom. `internal/skills`
236	236	* Fournisseurs : Docker Model Runner et llama-server sont deux paramétrages d'un même client OpenAI-compatible (`internal/engine/openai_compat.go`), avec une URL de repli (`host.docker.internal`) pour un agent en conteneur.
237	237	* Erreurs expliquées en une ligne : `Explain` traduit « connection refused » en « start it: llama-server … --jinja », signale l'absence de `--jinja` (nécessaire aux appels d'outils), et le 401 sur un serveur sans clé.
238	238	* Sondage de la fenêtre : `/props` de llama-server donne la taille servie ; la bannière en affiche l'origine.
	239	+* Flux interrompu par le serveur : llama.cpp signale une erreur en cours de génération par un événement SSE `error:`, qu'openai-go ignore (il ne restait que `unexpected end of JSON input`). `mm` récupère ce message (`internal/engine/servererr.go`) et relance une fois l'appel au modèle qui a échoué, sans rejouer les outils déjà exécutés dans le tour. Observé le 26/09/2026 : un tour perdu après vingt appels d'outils et un fichier écrit.
239	240
240	241	== Une méthode : mesurer
241	242
@@ -440,6 +441,7 @@ Sur une machine locale, un préfixe en cache fait la différence entre une répo
440	441	\| Raisonnement \| `internal/engine/reasoning.go`, `internal/engine/thoughts.go`
441	442	\| Usage en streaming \| `internal/engine/usage.go`
442	443	\| Appels d'outils coupés ou mal formés \| `internal/engine/toolargs.go`
	444	+\| Erreurs serveur en cours de flux \| `internal/engine/servererr.go`
443	445	\| Compression \| `internal/compact/`
444	446	\| Skills et catalogue \| `internal/skills/`, `skills/`
445	447	\| Évaluation du routage \| `internal/eval/`

  * *Fournisseurs* : Docker Model Runner et llama-server sont deux paramétrages d'un même client OpenAI-compatible (`internal/engine/openai_compat.go`), avec une URL de repli (`host.docker.internal`) pour un agent en conteneur.
  * *Erreurs expliquées en une ligne* : `Explain` traduit « connection refused » en « start it: llama-server … --jinja », signale l'absence de `--jinja` (nécessaire aux appels d'outils), et le 401 sur un serveur sans clé.
  * *Sondage de la fenêtre* : `/props` de llama-server donne la taille servie ; la bannière en affiche l'origine.
++* *Flux interrompu par le serveur* : llama.cpp signale une erreur en cours de génération par un événement SSE `error:`, qu'openai-go ignore (il ne restait que `unexpected end of JSON input`). `mm` récupère ce message (`internal/engine/servererr.go`) et relance *une fois* l'appel au modèle qui a échoué, sans rejouer les outils déjà exécutés dans le tour. Observé le 26/09/2026 : un tour perdu après vingt appels d'outils et un fichier écrit.
  == Une méthode : mesurer
  | Raisonnement | `internal/engine/reasoning.go`, `internal/engine/thoughts.go`
  | Usage en streaming | `internal/engine/usage.go`
  | Appels d'outils coupés ou mal formés | `internal/engine/toolargs.go`
++| Erreurs serveur en cours de flux | `internal/engine/servererr.go`
  | Compression | `internal/compact/`
  | Skills et catalogue | `internal/skills/`, `skills/`
  | Évaluation du routage | `internal/eval/`

modified docs/fr/explanation/providers.md +2 -0 Viewed

@@ -18,6 +18,8 @@
18	18
19	19	Les erreurs sont réécrites dans les mots du serveur. Une pile de messages imbriqués est du bruit ; « tool calls need llama-server started with --jinja » est tout le diagnostic. Le fournisseur reconnaît les refus de connexion, l'option `--jinja` manquante, les 401/403, les 404 avec une indication de pull ou d'alias, les 429 et les 5xx. Les erreurs inconnues reviennent inchangées : un message brut vaut mieux qu'une mauvaise piste.
20	20
	21	+Un flux que le serveur interrompt est relancé, et ses mots sont gardés. llama.cpp signale un échec au milieu d'une réponse diffusée par un événement SSE `error:` (le statut 200 est déjà parti). openai-go ne lit que les lignes `data:` : le message disparaissait derrière `unexpected end of JSON input`. Le tap qui lit déjà le flux pour le raisonnement récupère aussi ces événements, et l'appel au modèle qui a échoué est relancé une fois — cet appel seulement : les outils déjà exécutés dans le tour ne sont pas rejoués, et le texte déjà diffusé par la tentative interrompue n'est pas retiré. S'il échoue encore, l'erreur donne le message du serveur (`the model server reported an error while generating: Failed to parse input at pos …`), ou indique que la connexion a été coupée. Dans les deux cas l'historique est conservé : demander de continuer reprend le travail.
	22	+
21	23	## Alternatives écartées
22	24
23	25	- Un fournisseur par éditeur avec du code moteur dupliqué. Écarté : mesurée, la partie spécifique à l'éditeur tenait en trois identifiants.

  **Les erreurs sont réécrites dans les mots du serveur.** Une pile de messages imbriqués est du bruit ; « tool calls need llama-server started with --jinja » est tout le diagnostic. Le fournisseur reconnaît les refus de connexion, l'option `--jinja` manquante, les 401/403, les 404 avec une indication de pull ou d'alias, les 429 et les 5xx. Les erreurs inconnues reviennent inchangées : un message brut vaut mieux qu'une mauvaise piste.
++**Un flux que le serveur interrompt est relancé, et ses mots sont gardés.** llama.cpp signale un échec au milieu d'une réponse diffusée par un événement SSE `error:` (le statut 200 est déjà parti). openai-go ne lit que les lignes `data:` : le message disparaissait derrière `unexpected end of JSON input`. Le tap qui lit déjà le flux pour le raisonnement récupère aussi ces événements, et l'appel au modèle qui a échoué est relancé une fois — cet appel seulement : les outils déjà exécutés dans le tour ne sont pas rejoués, et le texte déjà diffusé par la tentative interrompue n'est pas retiré. S'il échoue encore, l'erreur donne le message du serveur (`the model server reported an error while generating: Failed to parse input at pos …`), ou indique que la connexion a été coupée. Dans les deux cas l'historique est conservé : demander de continuer reprend le travail.
++
  ## Alternatives écartées
  - **Un fournisseur par éditeur avec du code moteur dupliqué.** Écarté : mesurée, la partie spécifique à l'éditeur tenait en trois identifiants.

modified internal/engine/engine.go +28 -0 Viewed

@@ -137,6 +137,12 @@ func (e *Engine) Explain(err error) string {
137	137	if line, ok := badToolArgs(err); ok {
138	138	return line
139	139	}
	140	+ // A stream the server broke, even after the retry: the server's own
	141	+ // words when it streamed some (servererr.go), never "unexpected end of
	142	+ // JSON input" alone.
	143	+ if line, ok := badStream(err); ok {
	144	+ return line
	145	+ }
140	146	return e.Provider.Explain(err, e.Backend)
141	147	}
142	148
@@ -259,6 +265,20 @@ func (e Engine) Generate(ctx context.Context, messages []ai.Message, tools []a
259	265	WrapModel: func(ctx context.Context, params ai.ModelParams, next ai.ModelNext) (ai.ModelResponse, error) {
260	266	full = slices.Clone(params.Request.Messages)
261	267	resp, err := next(ctx, params)
	268	+ err = withServerMessage(ctx, err)
	269	+ // A stream the server broke — an error event, a cut
	270	+ // connection (servererr.go) — is retried ONCE, this model call
	271	+ // only. The turn-level retry below cannot help once a tool has
	272	+ // run (it would replay the tools); this one replays nothing but
	273	+ // the call that failed. Seen: a turn ended on "unexpected end
	274	+ // of JSON input" after twenty tool calls and a written file.
	275	+ // Text the broken attempt already streamed is not taken back.
	276	+ if streamBroken(err) && ctx.Err() == nil {
	277	+ spinner.Stop()
	278	+ fmt.Fprintf(ui.Out, "\n[%s — retrying this model call once]\n", streamNotice(err))
	279	+ resp, err = next(ctx, params)
	280	+ err = withServerMessage(ctx, err)
	281	+ }
262	282	// A tool call whose arguments did not parse — cut off by
263	283	// max_tokens, most of the time — is given back to the model
264	284	// with a note instead of ending the turn. The note goes into
@@ -283,6 +303,7 @@ func (e Engine) Generate(ctx context.Context, messages []ai.Message, tools []a
283	303	params.Request.Messages = append(slices.Clone(params.Request.Messages), argsRetryNote(truncated))
284	304	full = slices.Clone(params.Request.Messages)
285	305	resp, err = next(ctx, params)
	306	+ err = withServerMessage(ctx, err)
286	307	}
287	308	// The server's own count of what it just read: the one exact
288	309	// measure of the context size, kept for the compression
@@ -387,6 +408,9 @@ func (e Engine) Generate(ctx context.Context, messages []ai.Message, tools []a
387	408	think.emit(text)
388	409	}
389	410	})
	411	+ // The error events a server streams, for the model calls to report
	412	+ // (servererr.go; read back in WrapModel through the context).
	413	+ genCtx, _ = withServerErrors(genCtx)
390	414
391	415	spinner.Start("Thinking")
392	416	defer spinner.Stop()
@@ -527,6 +551,7 @@ func (e Engine) Summarize(ctx context.Context, messages []ai.Message, maxToken
527	551	// behind the user's back, and a model that thinks about it must not spill
528	552	// that onto the screen in the middle of their conversation.
529	553	genCtx = WithReasoning(genCtx, func(string) { beat() })
	554	+ genCtx, srvErrs := withServerErrors(genCtx)
530	555
531	556	resp, err := genkit.Generate(genCtx, e.G,
532	557	ai.WithModelName(e.Model),
@@ -542,6 +567,9 @@ func (e Engine) Summarize(ctx context.Context, messages []ai.Message, maxToken
542	567	if genCtx.Err() != nil && ctx.Err() == nil {
543	568	return "", fmt.Errorf("summary timed out (stalled)")
544	569	}
	570	+ if msg := srvErrs.take(); msg != "" {
	571	+ return "", &serverError{msg: msg, err: err}
	572	+ }
545	573	return "", err
546	574	}
547	575	text := strings.TrimSpace(resp.Text())

  	if line, ok := badToolArgs(err); ok {
  		return line
  	}
++	// A stream the server broke, even after the retry: the server's own
++	// words when it streamed some (servererr.go), never "unexpected end of
++	// JSON input" alone.
++	if line, ok := badStream(err); ok {
++		return line
++	}
  	return e.Provider.Explain(err, e.Backend)
  }
  			WrapModel: func(ctx context.Context, params *ai.ModelParams, next ai.ModelNext) (*ai.ModelResponse, error) {
  				full = slices.Clone(params.Request.Messages)
  				resp, err := next(ctx, params)
++				err = withServerMessage(ctx, err)
++				// A stream the server broke — an error event, a cut
++				// connection (servererr.go) — is retried ONCE, this model call
++				// only. The turn-level retry below cannot help once a tool has
++				// run (it would replay the tools); this one replays nothing but
++				// the call that failed. Seen: a turn ended on "unexpected end
++				// of JSON input" after twenty tool calls and a written file.
++				// Text the broken attempt already streamed is not taken back.
++				if streamBroken(err) && ctx.Err() == nil {
++					spinner.Stop()
++					fmt.Fprintf(ui.Out, "\n[%s — retrying this model call once]\n", streamNotice(err))
++					resp, err = next(ctx, params)
++					err = withServerMessage(ctx, err)
++				}
  				// A tool call whose arguments did not parse — cut off by
  				// max_tokens, most of the time — is given back to the model
  				// with a note instead of ending the turn. The note goes into
  					params.Request.Messages = append(slices.Clone(params.Request.Messages), argsRetryNote(truncated))
  					full = slices.Clone(params.Request.Messages)
  					resp, err = next(ctx, params)
++					err = withServerMessage(ctx, err)
  				}
  				// The server's own count of what it just read: the one exact
  				// measure of the context size, kept for the compression
  			think.emit(text)
  		}
  	})
++	// The error events a server streams, for the model calls to report
++	// (servererr.go; read back in WrapModel through the context).
++	genCtx, _ = withServerErrors(genCtx)
  	spinner.Start("Thinking")
  	defer spinner.Stop()
  	// behind the user's back, and a model that thinks about it must not spill
  	// that onto the screen in the middle of their conversation.
  	genCtx = WithReasoning(genCtx, func(string) { beat() })
++	genCtx, srvErrs := withServerErrors(genCtx)
  	resp, err := genkit.Generate(genCtx, e.G,
  		ai.WithModelName(e.Model),
  		if genCtx.Err() != nil && ctx.Err() == nil {
  			return "", fmt.Errorf("summary timed out (stalled)")
  		}
++		if msg := srvErrs.take(); msg != "" {
++			return "", &serverError{msg: msg, err: err}
++		}
  		return "", err
  	}
  	text := strings.TrimSpace(resp.Text())

modified internal/engine/reasoning.go +22 -6 Viewed

@@ -53,18 +53,24 @@ func reasoningFrom(ctx context.Context) func(string) {
53	53	}
54	54
55	55	// reasoningTap is the openai-go middleware. It only ever wraps a streamed
56		-// response, and only when somebody is listening; everything else is handed
	56	+// response, and only when somebody is listening — for the reasoning, or for
	57	+// the error events a server streams (servererr.go); everything else is handed
57	58	// back exactly as it came.
58	59	func reasoningTap(req http.Request, next option.MiddlewareNext) (http.Response, error) {
59	60	on := reasoningFrom(req.Context())
	61	+ errs := serverErrorsFrom(req.Context())
60	62	resp, err := next(req)
61		- if on == nil \|\| err != nil \|\| resp == nil \|\| resp.Body == nil {
	63	+ if (on == nil && errs == nil) \|\| err != nil \|\| resp == nil \|\| resp.Body == nil {
62	64	return resp, err
63	65	}
64	66	if !strings.Contains(resp.Header.Get("Content-Type"), "text/event-stream") {
65	67	return resp, nil
66	68	}
67		- resp.Body = &sseTap{rc: resp.Body, on: on}
	69	+ tap := &sseTap{rc: resp.Body, on: on}
	70	+ if errs != nil {
	71	+ tap.onErr = errs.record
	72	+ }
	73	+ resp.Body = tap
68	74	return resp, nil
69	75	}
70	76
@@ -77,9 +83,10 @@ const maxSSELine = 1 << 20
77	83	// while that chunk is being handed to the plugin, so it cannot arrive after
78	84	// the answer it preceded.
79	85	type sseTap struct {
80		- rc io.ReadCloser
81		- on func(string)
82		- buf []byte
	86	+ rc io.ReadCloser
	87	+ on func(string) // reasoning; nil = not listened to
	88	+ onErr func(string) // a server error event; nil = not listened to
	89	+ buf []byte
83	90	}
84	91
85	92	func (t *sseTap) Read(p []byte) (int, error) {
@@ -105,6 +112,15 @@ func (t *sseTap) scan(b []byte) {
105	112	}
106	113	line := string(t.buf[:i])
107	114	t.buf = t.buf[i+1:]
	115	+ if msg := serverErrorOf(line); msg != "" {
	116	+ if t.onErr != nil {
	117	+ t.onErr(msg)
	118	+ }
	119	+ continue
	120	+ }
	121	+ if t.on == nil {
	122	+ continue
	123	+ }
108	124	if text := reasoningOf(line); text != "" {
109	125	t.on(text)
110	126	}

  }
  // reasoningTap is the openai-go middleware. It only ever wraps a streamed
--// response, and only when somebody is listening; everything else is handed
++// response, and only when somebody is listening — for the reasoning, or for
++// the error events a server streams (servererr.go); everything else is handed
  // back exactly as it came.
  func reasoningTap(req *http.Request, next option.MiddlewareNext) (*http.Response, error) {
  	on := reasoningFrom(req.Context())
++	errs := serverErrorsFrom(req.Context())
  	resp, err := next(req)
--	if on == nil || err != nil || resp == nil || resp.Body == nil {
++	if (on == nil && errs == nil) || err != nil || resp == nil || resp.Body == nil {
  		return resp, err
  	}
  	if !strings.Contains(resp.Header.Get("Content-Type"), "text/event-stream") {
  		return resp, nil
  	}
--	resp.Body = &sseTap{rc: resp.Body, on: on}
++	tap := &sseTap{rc: resp.Body, on: on}
++	if errs != nil {
++		tap.onErr = errs.record
++	}
++	resp.Body = tap
  	return resp, nil
  }
  // while that chunk is being handed to the plugin, so it cannot arrive after
  // the answer it preceded.
  type sseTap struct {
--	rc  io.ReadCloser
++	rc    io.ReadCloser
--	on  func(string)
++	on    func(string) // reasoning; nil = not listened to
--	buf []byte
++	onErr func(string) // a server error event; nil = not listened to
++	buf   []byte
  }
  func (t *sseTap) Read(p []byte) (int, error) {
  		}
  		line := string(t.buf[:i])
  		t.buf = t.buf[i+1:]
++		if msg := serverErrorOf(line); msg != "" {
++			if t.onErr != nil {
++				t.onErr(msg)
++			}
++			continue
++		}
++		if t.on == nil {
++			continue
++		}
  		if text := reasoningOf(line); text != "" {
  			t.on(text)
  		}

added internal/engine/servererr.go +179 -0 Viewed

new file mode 100644
@@ -0,0 +1,179 @@
	1	+package engine
	2	+
	3	+import (
	4	+ "context"
	5	+ "encoding/json"
	6	+ "errors"
	7	+ "fmt"
	8	+ "strings"
	9	+ "sync"
	10	+)
	11	+
	12	+// A server that fails in the middle of a streamed answer cannot change the
	13	+// HTTP status any more — it already sent 200 — so llama.cpp (llama-server,
	14	+// and Docker Model Runner's llama.cpp backend) writes the failure as an SSE
	15	+// event of its own:
	16	+//
	17	+// error: {"code":500,"message":"Failed to parse input at pos 1234","type":"server_error"}
	18	+//
	19	+// openai-go only reads `data:` lines. It sees an event with no data, decodes
	20	+// the empty string, and reports `unexpected end of JSON input` — the server's
	21	+// message is gone. Measured on 2026-09-26 against a fake SSE server: an
	22	+// `error:` event gives exactly that text, a cut connection gives `unexpected
	23	+// EOF`. A user's session ended on it after twenty tool calls, with nothing
	24	+// to say why.
	25	+//
	26	+// So the tap that already reads the stream for reasoning (reasoning.go) also
	27	+// picks up `error:` lines, and the model call that failed reports what the
	28	+// server said (serverError).
	29	+
	30	+// serverErrKey is the context key carrying a generation's recorder.
	31	+type serverErrKey struct{}
	32	+
	33	+// serverErrors keeps the last error event the server streamed. The tap writes
	34	+// from the HTTP reader's goroutine, the model call reads after it returned.
	35	+type serverErrors struct {
	36	+ mu sync.Mutex
	37	+ last string
	38	+}
	39	+
	40	+// withServerErrors returns a context whose streamed requests record the
	41	+// server's error events, and the recorder to read them from.
	42	+func withServerErrors(ctx context.Context) (context.Context, *serverErrors) {
	43	+ s := &serverErrors{}
	44	+ return context.WithValue(ctx, serverErrKey{}, s), s
	45	+}
	46	+
	47	+func serverErrorsFrom(ctx context.Context) *serverErrors {
	48	+ s, _ := ctx.Value(serverErrKey{}).(*serverErrors)
	49	+ return s
	50	+}
	51	+
	52	+func (s *serverErrors) record(msg string) {
	53	+ s.mu.Lock()
	54	+ s.last = msg
	55	+ s.mu.Unlock()
	56	+}
	57	+
	58	+// take returns the recorded message and forgets it, so a retried call does
	59	+// not inherit the error of the attempt before it.
	60	+func (s *serverErrors) take() string {
	61	+ if s == nil {
	62	+ return ""
	63	+ }
	64	+ s.mu.Lock()
	65	+ defer s.mu.Unlock()
	66	+ msg := s.last
	67	+ s.last = ""
	68	+ return msg
	69	+}
	70	+
	71	+// serverErrorOf returns the message of an `error:` SSE line, "" for any other
	72	+// line. The payload is llama.cpp's {"code","message","type"}; an OpenAI-style
	73	+// {"error":{"message"}} is read too, and anything else is kept as raw text —
	74	+// the point is to show what the server said, whatever its shape.
	75	+func serverErrorOf(line string) string {
	76	+ payload, ok := strings.CutPrefix(strings.TrimSpace(line), "error:")
	77	+ if !ok {
	78	+ return ""
	79	+ }
	80	+ payload = strings.TrimSpace(payload)
	81	+ if payload == "" {
	82	+ return "the server sent an empty error event"
	83	+ }
	84	+ var e struct {
	85	+ Message string `json:"message"`
	86	+ Error *struct {
	87	+ Message string `json:"message"`
	88	+ } `json:"error"`
	89	+ }
	90	+ if json.Unmarshal([]byte(payload), &e) == nil {
	91	+ switch {
	92	+ case e.Message != "":
	93	+ return e.Message
	94	+ case e.Error != nil && e.Error.Message != "":
	95	+ return e.Error.Message
	96	+ }
	97	+ }
	98	+ return lastRunes(payload, 300)
	99	+}
	100	+
	101	+// serverError is a model call that failed on an error the server streamed.
	102	+type serverError struct {
	103	+ msg string
	104	+ err error
	105	+}
	106	+
	107	+func (e *serverError) Error() string {
	108	+ return serverErrorPrefix + e.msg + " (" + e.err.Error() + ")"
	109	+}
	110	+
	111	+func (e *serverError) Unwrap() error { return e.err }
	112	+
	113	+const serverErrorPrefix = "the model server reported an error while generating: "
	114	+
	115	+// withServerMessage attaches what the server streamed as an error to the
	116	+// failure of the model call it belongs to, and forgets it either way.
	117	+func withServerMessage(ctx context.Context, err error) error {
	118	+ msg := serverErrorsFrom(ctx).take()
	119	+ if err == nil \|\| msg == "" {
	120	+ return err
	121	+ }
	122	+ var se *serverError
	123	+ if errors.As(err, &se) {
	124	+ return err
	125	+ }
	126	+ return &serverError{msg: msg, err: err}
	127	+}
	128	+
	129	+// streamBroken reports whether a model call failed because its stream broke —
	130	+// a server error event or a cut connection — rather than because of what the
	131	+// model wrote (argsError), our own stop (the runaway guard fails the
	132	+// streaming callback: "callback error"), or a cancellation.
	133	+func streamBroken(err error) bool {
	134	+ if err == nil {
	135	+ return false
	136	+ }
	137	+ var se *serverError
	138	+ if errors.As(err, &se) {
	139	+ return true
	140	+ }
	141	+ msg := err.Error()
	142	+ return strings.Contains(msg, "stream error:") && !strings.Contains(msg, "context canceled")
	143	+}
	144	+
	145	+// streamNotice is the one line printed before the retry: the server's words,
	146	+// or what broke.
	147	+func streamNotice(err error) string {
	148	+ var se *serverError
	149	+ if errors.As(err, &se) {
	150	+ return "the model server reported an error: " + se.msg
	151	+ }
	152	+ msg := err.Error()
	153	+ if i := strings.Index(msg, "stream error: "); i >= 0 {
	154	+ msg = msg[i+len("stream error: "):]
	155	+ }
	156	+ return "the stream from the model server broke (" + lastRunes(msg, 80) + ")"
	157	+}
	158	+
	159	+// badStream is Explain's line for a broken stream once the retry is spent:
	160	+// the server's own words when it gave some, what a cut connection means
	161	+// otherwise. Found in the text as well as by type: Genkit may rebuild the
	162	+// error as a string.
	163	+func badStream(err error) (string, bool) {
	164	+ if err == nil {
	165	+ return "", false
	166	+ }
	167	+ msg := err.Error()
	168	+ if i := strings.Index(msg, serverErrorPrefix); i >= 0 {
	169	+ said := msg[i+len(serverErrorPrefix):]
	170	+ if j := strings.LastIndex(said, " ("); j >= 0 {
	171	+ said = said[:j]
	172	+ }
	173	+ return fmt.Sprintf("%s%s — the server's logs say more (docker model logs, or llama-server's terminal); the work done before is kept: ask to continue", serverErrorPrefix, said), true
	174	+ }
	175	+ if streamBroken(err) {
	176	+ return "the connection to the model server broke during generation (" + lastRunes(msg, 80) + ") — check that the server is still running (docker model logs, or llama-server's terminal); the work done before is kept: ask to continue", true
	177	+ }
	178	+ return "", false
	179	+}

++package engine
++
++import (
++	"context"
++	"encoding/json"
++	"errors"
++	"fmt"
++	"strings"
++	"sync"
++)
++
++// A server that fails in the middle of a streamed answer cannot change the
++// HTTP status any more — it already sent 200 — so llama.cpp (llama-server,
++// and Docker Model Runner's llama.cpp backend) writes the failure as an SSE
++// event of its own:
++//
++//	error: {"code":500,"message":"Failed to parse input at pos 1234","type":"server_error"}
++//
++// openai-go only reads `data:` lines. It sees an event with no data, decodes
++// the empty string, and reports `unexpected end of JSON input` — the server's
++// message is gone. Measured on 2026-09-26 against a fake SSE server: an
++// `error:` event gives exactly that text, a cut connection gives `unexpected
++// EOF`. A user's session ended on it after twenty tool calls, with nothing
++// to say why.
++//
++// So the tap that already reads the stream for reasoning (reasoning.go) also
++// picks up `error:` lines, and the model call that failed reports what the
++// server said (serverError).
++
++// serverErrKey is the context key carrying a generation's recorder.
++type serverErrKey struct{}
++
++// serverErrors keeps the last error event the server streamed. The tap writes
++// from the HTTP reader's goroutine, the model call reads after it returned.
++type serverErrors struct {
++	mu   sync.Mutex
++	last string
++}
++
++// withServerErrors returns a context whose streamed requests record the
++// server's error events, and the recorder to read them from.
++func withServerErrors(ctx context.Context) (context.Context, *serverErrors) {
++	s := &serverErrors{}
++	return context.WithValue(ctx, serverErrKey{}, s), s
++}
++
++func serverErrorsFrom(ctx context.Context) *serverErrors {
++	s, _ := ctx.Value(serverErrKey{}).(*serverErrors)
++	return s
++}
++
++func (s *serverErrors) record(msg string) {
++	s.mu.Lock()
++	s.last = msg
++	s.mu.Unlock()
++}
++
++// take returns the recorded message and forgets it, so a retried call does
++// not inherit the error of the attempt before it.
++func (s *serverErrors) take() string {
++	if s == nil {
++		return ""
++	}
++	s.mu.Lock()
++	defer s.mu.Unlock()
++	msg := s.last
++	s.last = ""
++	return msg
++}
++
++// serverErrorOf returns the message of an `error:` SSE line, "" for any other
++// line. The payload is llama.cpp's {"code","message","type"}; an OpenAI-style
++// {"error":{"message"}} is read too, and anything else is kept as raw text —
++// the point is to show what the server said, whatever its shape.
++func serverErrorOf(line string) string {
++	payload, ok := strings.CutPrefix(strings.TrimSpace(line), "error:")
++	if !ok {
++		return ""
++	}
++	payload = strings.TrimSpace(payload)
++	if payload == "" {
++		return "the server sent an empty error event"
++	}
++	var e struct {
++		Message string `json:"message"`
++		Error   *struct {
++			Message string `json:"message"`
++		} `json:"error"`
++	}
++	if json.Unmarshal([]byte(payload), &e) == nil {
++		switch {
++		case e.Message != "":
++			return e.Message
++		case e.Error != nil && e.Error.Message != "":
++			return e.Error.Message
++		}
++	}
++	return lastRunes(payload, 300)
++}
++
++// serverError is a model call that failed on an error the server streamed.
++type serverError struct {
++	msg string
++	err error
++}
++
++func (e *serverError) Error() string {
++	return serverErrorPrefix + e.msg + " (" + e.err.Error() + ")"
++}
++
++func (e *serverError) Unwrap() error { return e.err }
++
++const serverErrorPrefix = "the model server reported an error while generating: "
++
++// withServerMessage attaches what the server streamed as an error to the
++// failure of the model call it belongs to, and forgets it either way.
++func withServerMessage(ctx context.Context, err error) error {
++	msg := serverErrorsFrom(ctx).take()
++	if err == nil || msg == "" {
++		return err
++	}
++	var se *serverError
++	if errors.As(err, &se) {
++		return err
++	}
++	return &serverError{msg: msg, err: err}
++}
++
++// streamBroken reports whether a model call failed because its stream broke —
++// a server error event or a cut connection — rather than because of what the
++// model wrote (argsError), our own stop (the runaway guard fails the
++// streaming callback: "callback error"), or a cancellation.
++func streamBroken(err error) bool {
++	if err == nil {
++		return false
++	}
++	var se *serverError
++	if errors.As(err, &se) {
++		return true
++	}
++	msg := err.Error()
++	return strings.Contains(msg, "stream error:") && !strings.Contains(msg, "context canceled")
++}
++
++// streamNotice is the one line printed before the retry: the server's words,
++// or what broke.
++func streamNotice(err error) string {
++	var se *serverError
++	if errors.As(err, &se) {
++		return "the model server reported an error: " + se.msg
++	}
++	msg := err.Error()
++	if i := strings.Index(msg, "stream error: "); i >= 0 {
++		msg = msg[i+len("stream error: "):]
++	}
++	return "the stream from the model server broke (" + lastRunes(msg, 80) + ")"
++}
++
++// badStream is Explain's line for a broken stream once the retry is spent:
++// the server's own words when it gave some, what a cut connection means
++// otherwise. Found in the text as well as by type: Genkit may rebuild the
++// error as a string.
++func badStream(err error) (string, bool) {
++	if err == nil {
++		return "", false
++	}
++	msg := err.Error()
++	if i := strings.Index(msg, serverErrorPrefix); i >= 0 {
++		said := msg[i+len(serverErrorPrefix):]
++		if j := strings.LastIndex(said, " ("); j >= 0 {
++			said = said[:j]
++		}
++		return fmt.Sprintf("%s%s — the server's logs say more (docker model logs, or llama-server's terminal); the work done before is kept: ask to continue", serverErrorPrefix, said), true
++	}
++	if streamBroken(err) {
++		return "the connection to the model server broke during generation (" + lastRunes(msg, 80) + ") — check that the server is still running (docker model logs, or llama-server's terminal); the work done before is kept: ask to continue", true
++	}
++	return "", false
++}

added internal/engine/servererr_test.go +134 -0 Viewed

new file mode 100644
@@ -0,0 +1,134 @@
	1	+package engine
	2	+
	3	+import (
	4	+ "context"
	5	+ "fmt"
	6	+ "net/http"
	7	+ "net/http/httptest"
	8	+ "strings"
	9	+ "sync/atomic"
	10	+ "testing"
	11	+
	12	+ "github.com/firebase/genkit/go/ai"
	13	+)
	14	+
	15	+func sseChunkLine(delta, finish string) string {
	16	+ f := "null"
	17	+ if finish != "" {
	18	+ f = `"` + finish + `"`
	19	+ }
	20	+ return fmt.Sprintf(`data: {"id":"x","object":"chat.completion.chunk","created":1,"model":"fake-model","choices":[{"index":0,"delta":%s,"finish_reason":%s}]}`+"\n\n", delta, f)
	21	+}
	22	+
	23	+// fakeLlama is an OpenAI-compatible server whose streamed answers are
	24	+// written by `answer`, given the 1-based number of the call.
	25	+func fakeLlama(t testing.T, answer func(call int, w http.ResponseWriter)) (httptest.Server, *atomic.Int32) {
	26	+ t.Helper()
	27	+ var calls atomic.Int32
	28	+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	29	+ if !strings.HasSuffix(r.URL.Path, "/chat/completions") {
	30	+ http.NotFound(w, r)
	31	+ return
	32	+ }
	33	+ w.Header().Set("Content-Type", "text/event-stream")
	34	+ answer(int(calls.Add(1)), w)
	35	+ }))
	36	+ t.Cleanup(srv.Close)
	37	+ return srv, &calls
	38	+}
	39	+
	40	+func goodAnswer(w http.ResponseWriter) {
	41	+ fmt.Fprint(w, sseChunkLine(`{"role":"assistant","content":"all good"}`, ""))
	42	+ fmt.Fprint(w, sseChunkLine(`{}`, "stop"))
	43	+ fmt.Fprint(w, "data: [DONE]\n\n")
	44	+}
	45	+
	46	+// llama.cpp's way of failing mid-stream: an `error:` event, after 200 and
	47	+// some content.
	48	+func errorEvent(w http.ResponseWriter) {
	49	+ fmt.Fprint(w, sseChunkLine(`{"role":"assistant","content":"Hel"}`, ""))
	50	+ fmt.Fprint(w, `error: {"code":500,"message":"Failed to parse input at pos 12","type":"server_error"}`+"\n\n")
	51	+}
	52	+
	53	+func engineOn(t testing.T, baseURL string) Engine {
	54	+ t.Helper()
	55	+ p, _ := Lookup("llamacpp")
	56	+ b := Backend{Provider: "llamacpp", BaseURL: baseURL, Model: "fake-model", APIKey: "not-needed"}
	57	+ g, model, err := p.Open(context.Background(), b)
	58	+ if err != nil {
	59	+ t.Fatal(err)
	60	+ }
	61	+ return &Engine{G: g, Model: model, Provider: p, Backend: b}
	62	+}
	63	+
	64	+// One broken stream: the model call is retried once and the turn succeeds.
	65	+func TestBrokenStreamIsRetriedOnce(t *testing.T) {
	66	+ srv, calls := fakeLlama(t, func(call int, w http.ResponseWriter) {
	67	+ if call == 1 {
	68	+ errorEvent(w)
	69	+ return
	70	+ }
	71	+ goodAnswer(w)
	72	+ })
	73	+ resp, _, err := engineOn(t, srv.URL+"/v1").Generate(context.Background(), []*ai.Message{ai.NewUserTextMessage("hi")}, nil)
	74	+ if err != nil {
	75	+ t.Fatalf("Generate: %v — the broken stream should have been retried", err)
	76	+ }
	77	+ if resp.Text() != "all good" \|\| calls.Load() != 2 {
	78	+ t.Errorf("answer %q after %d call(s), want \"all good\" after 2", resp.Text(), calls.Load())
	79	+ }
	80	+}
	81	+
	82	+// A server that keeps failing: the error names what the server said, not
	83	+// "unexpected end of JSON input".
	84	+func TestServerErrorEventIsExplained(t *testing.T) {
	85	+ srv, calls := fakeLlama(t, func(_ int, w http.ResponseWriter) { errorEvent(w) })
	86	+ e := engineOn(t, srv.URL+"/v1")
	87	+ _, _, err := e.Generate(context.Background(), []*ai.Message{ai.NewUserTextMessage("hi")}, nil)
	88	+ if err == nil {
	89	+ t.Fatal("Generate succeeded against a failing server")
	90	+ }
	91	+ if calls.Load() < 2 {
	92	+ t.Errorf("%d call(s): the model call was not retried", calls.Load())
	93	+ }
	94	+ got := e.Explain(err)
	95	+ if !strings.Contains(got, "Failed to parse input at pos 12") \|\| strings.Contains(got, "unexpected end of JSON input") {
	96	+ t.Errorf("Explain = %q, want the server's message", got)
	97	+ }
	98	+}
	99	+
	100	+// A connection cut mid-stream (a crash): no server message to report, but a
	101	+// line that says what happened.
	102	+func TestCutStreamIsExplained(t *testing.T) {
	103	+ srv, _ := fakeLlama(t, func(_ int, w http.ResponseWriter) {
	104	+ fmt.Fprint(w, sseChunkLine(`{"role":"assistant","content":"Hel"}`, ""))
	105	+ w.(http.Flusher).Flush()
	106	+ conn, _, _ := w.(http.Hijacker).Hijack()
	107	+ conn.Close()
	108	+ })
	109	+ e := engineOn(t, srv.URL+"/v1")
	110	+ _, _, err := e.Generate(context.Background(), []*ai.Message{ai.NewUserTextMessage("hi")}, nil)
	111	+ if err == nil {
	112	+ t.Fatal("Generate succeeded against a server that hangs up")
	113	+ }
	114	+ if got := e.Explain(err); !strings.Contains(got, "connection to the model server broke") {
	115	+ t.Errorf("Explain = %q", got)
	116	+ }
	117	+}
	118	+
	119	+func TestServerErrorOf(t *testing.T) {
	120	+ cases := map[string]string{
	121	+ `error: {"code":500,"message":"Failed to parse input","type":"server_error"}`: "Failed to parse input",
	122	+ `error: {"error":{"message":"out of memory"}}`: "out of memory",
	123	+ `error: boom`: "boom",
	124	+ `error:`: "the server sent an empty error event",
	125	+ `data: {"choices":[]}`: "",
	126	+ `: keep-alive`: "",
	127	+ `data: {"error":"not an event"}`: "",
	128	+ }
	129	+ for line, want := range cases {
	130	+ if got := serverErrorOf(line); got != want {
	131	+ t.Errorf("serverErrorOf(%q) = %q, want %q", line, got, want)
	132	+ }
	133	+ }
	134	+}

++package engine
++
++import (
++	"context"
++	"fmt"
++	"net/http"
++	"net/http/httptest"
++	"strings"
++	"sync/atomic"
++	"testing"
++
++	"github.com/firebase/genkit/go/ai"
++)
++
++func sseChunkLine(delta, finish string) string {
++	f := "null"
++	if finish != "" {
++		f = `"` + finish + `"`
++	}
++	return fmt.Sprintf(`data: {"id":"x","object":"chat.completion.chunk","created":1,"model":"fake-model","choices":[{"index":0,"delta":%s,"finish_reason":%s}]}`+"\n\n", delta, f)
++}
++
++// fakeLlama is an OpenAI-compatible server whose streamed answers are
++// written by `answer`, given the 1-based number of the call.
++func fakeLlama(t *testing.T, answer func(call int, w http.ResponseWriter)) (*httptest.Server, *atomic.Int32) {
++	t.Helper()
++	var calls atomic.Int32
++	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
++		if !strings.HasSuffix(r.URL.Path, "/chat/completions") {
++			http.NotFound(w, r)
++			return
++		}
++		w.Header().Set("Content-Type", "text/event-stream")
++		answer(int(calls.Add(1)), w)
++	}))
++	t.Cleanup(srv.Close)
++	return srv, &calls
++}
++
++func goodAnswer(w http.ResponseWriter) {
++	fmt.Fprint(w, sseChunkLine(`{"role":"assistant","content":"all good"}`, ""))
++	fmt.Fprint(w, sseChunkLine(`{}`, "stop"))
++	fmt.Fprint(w, "data: [DONE]\n\n")
++}
++
++// llama.cpp's way of failing mid-stream: an `error:` event, after 200 and
++// some content.
++func errorEvent(w http.ResponseWriter) {
++	fmt.Fprint(w, sseChunkLine(`{"role":"assistant","content":"Hel"}`, ""))
++	fmt.Fprint(w, `error: {"code":500,"message":"Failed to parse input at pos 12","type":"server_error"}`+"\n\n")
++}
++
++func engineOn(t *testing.T, baseURL string) *Engine {
++	t.Helper()
++	p, _ := Lookup("llamacpp")
++	b := Backend{Provider: "llamacpp", BaseURL: baseURL, Model: "fake-model", APIKey: "not-needed"}
++	g, model, err := p.Open(context.Background(), b)
++	if err != nil {
++		t.Fatal(err)
++	}
++	return &Engine{G: g, Model: model, Provider: p, Backend: b}
++}
++
++// One broken stream: the model call is retried once and the turn succeeds.
++func TestBrokenStreamIsRetriedOnce(t *testing.T) {
++	srv, calls := fakeLlama(t, func(call int, w http.ResponseWriter) {
++		if call == 1 {
++			errorEvent(w)
++			return
++		}
++		goodAnswer(w)
++	})
++	resp, _, err := engineOn(t, srv.URL+"/v1").Generate(context.Background(), []*ai.Message{ai.NewUserTextMessage("hi")}, nil)
++	if err != nil {
++		t.Fatalf("Generate: %v — the broken stream should have been retried", err)
++	}
++	if resp.Text() != "all good" || calls.Load() != 2 {
++		t.Errorf("answer %q after %d call(s), want \"all good\" after 2", resp.Text(), calls.Load())
++	}
++}
++
++// A server that keeps failing: the error names what the server said, not
++// "unexpected end of JSON input".
++func TestServerErrorEventIsExplained(t *testing.T) {
++	srv, calls := fakeLlama(t, func(_ int, w http.ResponseWriter) { errorEvent(w) })
++	e := engineOn(t, srv.URL+"/v1")
++	_, _, err := e.Generate(context.Background(), []*ai.Message{ai.NewUserTextMessage("hi")}, nil)
++	if err == nil {
++		t.Fatal("Generate succeeded against a failing server")
++	}
++	if calls.Load() < 2 {
++		t.Errorf("%d call(s): the model call was not retried", calls.Load())
++	}
++	got := e.Explain(err)
++	if !strings.Contains(got, "Failed to parse input at pos 12") || strings.Contains(got, "unexpected end of JSON input") {
++		t.Errorf("Explain = %q, want the server's message", got)
++	}
++}
++
++// A connection cut mid-stream (a crash): no server message to report, but a
++// line that says what happened.
++func TestCutStreamIsExplained(t *testing.T) {
++	srv, _ := fakeLlama(t, func(_ int, w http.ResponseWriter) {
++		fmt.Fprint(w, sseChunkLine(`{"role":"assistant","content":"Hel"}`, ""))
++		w.(http.Flusher).Flush()
++		conn, _, _ := w.(http.Hijacker).Hijack()
++		conn.Close()
++	})
++	e := engineOn(t, srv.URL+"/v1")
++	_, _, err := e.Generate(context.Background(), []*ai.Message{ai.NewUserTextMessage("hi")}, nil)
++	if err == nil {
++		t.Fatal("Generate succeeded against a server that hangs up")
++	}
++	if got := e.Explain(err); !strings.Contains(got, "connection to the model server broke") {
++		t.Errorf("Explain = %q", got)
++	}
++}
++
++func TestServerErrorOf(t *testing.T) {
++	cases := map[string]string{
++		`error: {"code":500,"message":"Failed to parse input","type":"server_error"}`: "Failed to parse input",
++		`error: {"error":{"message":"out of memory"}}`:                                "out of memory",
++		`error: boom`:                    "boom",
++		`error:`:                         "the server sent an empty error event",
++		`data: {"choices":[]}`:           "",
++		`: keep-alive`:                   "",
++		`data: {"error":"not an event"}`: "",
++	}
++	for line, want := range cases {
++		if got := serverErrorOf(line); got != want {
++			t.Errorf("serverErrorOf(%q) = %q, want %q", line, got, want)
++		}
++	}
++}

🛟 Updated.