Gemma coder — a Go specialist for vibe coding

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132

# Gemma coder — a Go specialist for vibe coding

*[French version](README.fr.md)*

A coding agent for `mm` that does **one language, Go**, and works in **autonomous mode**: you describe what you want in a sentence or two, the agent writes the code and its tests, runs `go build` and `go test`, fixes what fails and reports. Same model as [`gemini-style`](../gemini-style/): **Gemma 4 26B-A4B** served by llmman. The prompt is no longer a general one, though: it is a Go engineer's checklist, plus the rules the measurements showed this model needs.

| | |
|---|---|
| Model | `hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:Q4_K_M` (llmman), 18.1 GB, **thinking on** |
| Tools | `bash`, `read_file`, `write_file`, `edit_file`, `plan`, `ask_user`, `lsp` (gopls) |
| Mode | `yolo: true`: no confirmation before each change |
| Project rules | the project's `AGENTS.md` |

![How the agent works](gemma-coder-flow.svg)

## Prerequisites

```bash
llmman pull hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:Q4_K_M
llmman serve
go install golang.org/x/tools/gopls@latest   # the lsp tool
go build -o mm .                             # at the repository root
```

Go 1.22 or later on the `PATH` (the default web stack in the prompt relies on `http.ServeMux` patterns). Memory: about 18 GB of weights, plus the KV cache, which stays small thanks to sliding-window attention. This is an estimate, not a measurement: comfortable from 32 GB.

## Run

From the Go project to work on, or from an empty directory for a new program:

```bash
cd ~/src/my-go-project
mm -config /path/to/mini-me/demos/gemma-coder/agent.yaml          # REPL
mm -tui -config /path/to/mini-me/demos/gemma-coder/agent.yaml     # single-window terminal
mm -acp -config /path/to/mini-me/demos/gemma-coder/agent.yaml     # in an editor (Zed…)
mm -config …/agent.yaml -p "Create a REST API for a todo list…"   # one request, scripted
```

**Vibe coding means the agent changes files without asking.** Work in a git repository, or in a sandbox, so that `git diff` shows everything it did and `git checkout .` undoes it. To confirm each change instead, set `AGENT_YOLO=false` (or `yolo: false`): `[a]` once, `[s]` this tool for the session, `[r]` reject, `[n]` never.

Requests that work well, measured below: "add a function X to package Y", "go test fails, fix it", "rename X to Y everywhere", "create a CLI tool that…", "create a REST API for…, with tests", "commit my change to file F", "explain this project" (in French or English: it answers in the language of the question).

## How it is built, and why

![Settings chosen from the measurements](gemma-coder-tuning.svg)

| Choice | Why (measured — see below) |
|---|---|
| `enable_thinking: true` (Gemma 4's template default is off) | Without thinking, on the three long cases: 9/15 runs; it rewrote the existing tests in table-driven style unasked, broke them, then looped (`edit_file` ×4, `write_file` ×6 until mm's loop detector), or sent runaway tool arguments. With thinking: 11/15, and the existing tests left alone in 4 runs out of 5. |
| `max_tokens: 32768` | Room for the reasoning plus a whole Go file inside one `write_file`. |
| "Directives vs Inquiries", taken from Gemini CLI | On explain and faithful, no file was ever touched (8/8, series A and D). |
| Commit "without asking back", and "commit my change to X = X alone" | With Gemini CLI's wording (`gemini-style`), the agent asked "which files? which message?" and never committed (0/2). Here: 4/4. |
| "AFTER your last edit, run `go test ./...` again" | With thinking on, one run made a last edit after its last test, added an unused import that broke the build, and reported success. Not seen again after the rule (0/6 long runs), on a small sample. |
| "Existing tests are not yours to restyle" | Added after the first failure. Without thinking, ignored (both passing feature runs of B still rewrote the tests). With thinking, followed 4/5 (one D feature run still converted them, without breaking them). |
| Validate: `lsp diagnostics` → `gofmt -l` → `go build` → `go test` | The order Go developers use. **In practice the model skips the first two:** `lsp diagnostics` 0 times in 54 runs, `gofmt` in 3 of 18. `go build` and `go test` carry the validation. |
| `lsp` with gopls only | `rename` went through gopls in 4/4 runs: a compiler-exact rename across files, tests included. For "where is X defined", the model still reaches for `grep`. |
| `parallel_tool_calls: false` | One call per reply, so the edit lands before `go test` runs. Taken from the earlier demos; not re-measured here. |
| Standard library first, `http.ServeMux` patterns, `log/slog` | The default stack for "create an API" and "create a CLI" when the user names none. One API run still added `github.com/google/uuid` (see failures). |
| `context.threshold: 27` | 27 % of the 262,144-token window ≈ 70k tokens: compression starts before long sessions get slow. |
| `maxTurns: 80` | A new application takes 20–40 calls; the "same failure 3× → change approach" rule is the brake, not the turn count. |
| `fetchContentTool: false`, `skillsDir: no-skills` | The agent works offline from the code and the Go toolchain; the Go workflow is in the prompt. |

## Measured results

Measured on 2026-11-05 with `AGENT_YOLO=true mm -p` against the development machine's llmman, each run on a fresh copy of a fixture under git. There are nine cases: the seven Go cases used for `qwen-style`, `gemini-style` and `codex-style`, plus two vibe-coding cases. Each result is checked by a script (files, `go test`, `git log`, a real HTTP server), then the final answer is read.

| Case | Request | Pass when |
|---|---|---|
| explain | « Explique-moi ce que fait ce projet et comment le tester. Ne modifie aucun fichier. » | no file changed, answer in French |
| fix | "go test fails in this project. Fix it." (Median does not sort) | `go test` passes, the test file untouched |
| feature | "Add a Mode function…" (ties, empty sample) | hidden tests pass, the agent added its own tests |
| typo | "Fix the typo in README.md." | the typo fixed, nothing else changed |
| commit | "Commit my change to stats/stats.go." (README.md also modified) | one commit with `stats.go` only, README.md left modified |
| faithful | "Run the tests and tell me if everything is fine." (one test needs an unset variable) | it says a test fails, no file changed |
| newapp | "Create a small Go command-line tool named wc-lite…" | builds, has tests that pass, prints `2 5 24` on the sample |
| rename | "Rename the function Mean to Average everywhere in the project." | no `Mean(` call left, 3 files changed, build and tests pass |
| webapi | "Create a REST API in Go for a todo list…" (GET/POST/DELETE, PORT) | builds, tests pass, **POST → GET → DELETE → GET on the real server** |

### The shipped configuration (D: thinking on, all rules)

| Case | Runs passed | Duration |
|---|---|---|
| explain | 2/2 | 21–36 s |
| fix | 2/2 | 26–35 s |
| feature | 2/2 | 93–126 s |
| typo | 2/2 | 3–4 s |
| commit | 2/2 | 3 s |
| faithful | 2/2 | 19–22 s |
| newapp | 1/2 | 71–205 s |
| rename | 2/2 (`lsp rename` both times) | 40–117 s |
| webapi | 1/2 | 117–148 s |
| **Total** | **16/18** | |

The two failures:
- **newapp, run 1**: its own tests expected wrong byte counts. When it tried to fix them, the model sent a malformed `edit_file` call (`path` inside each edit). mm stopped the run with a schema error instead of handing the error back to the model. The same thing happened once in series B and once in series C. This is a limitation on mm's side, not the prompt's.
- **webapi, run 1**: the project was created in a `todo-api/` subdirectory instead of the current directory, it added `github.com/google/uuid` despite "standard library first", and `DELETE /todos/{id}` returned **404** on the real server. Its own tests passed because they did not go through the real router. Only the HTTP check caught it.

Not counted as a failure: in the second newapp run, `main.go` went to `cmd/wc-lite/` instead of the module root, against the prompt's rule. The tool builds, its tests pass and it prints the right counts.

### The configurations tried

| Configuration | Cases | Passed | What went wrong |
|---|---|---|---|
| A · thinking off, first prompt | 9 × 2 | 16/18 | feature: rewrote the existing tests, broke `go vet`, prose loop "Wait, I'll use t.Errorf…" until an empty reply (546 s); newapp: no `main.go`, `write_file` ×6 identical until the loop detector |
| B · A + "keep existing tests" + "main.go at the root" | long × 3 | 5/9 | feature 2/3 (runaway tool arguments; both passing runs still rewrote the tests), newapp 0/3 (a malformed call, an empty reply, a loop), webapi 3/3 |
| C · B + thinking on, `max_tokens` 32768 | long × 3 | 7/9 | feature 2/3 (one last edit after the last test, success claimed, build broken), newapp 2/3 (malformed `edit_file` call); existing tests kept 3/3 |
| **D · C + "test again after the last edit"** | 9 × 2 | **16/18** | see above |

On the three long cases, all series counted: thinking off 9/15, thinking on 11/15. Thinking is slower on short questions: explain went from 8–9 s to 21–36 s, faithful from 3–4 s to 19–22 s.

For comparison on the seven shared cases, with the same model: `gemini-style` 12/14 (commit 0/2), here 13/14 (newapp 1/2). Two or three runs per case are a small sample: a 1/2 can be luck in either direction.

Not measured: a real interactive session (`ask_user`, the REPL, an editor over ACP), projects bigger than a few files, reasoning off with the final prompt.

### Can you vibe-code Go with this model?

Yes, for small, well-specified things: a function with its tests, a fix, a rename, a CLI tool, a small REST service. The agent builds them, tests them, and most of the time delivers something that works. Two habits are still needed:
- **check the behaviour yourself, not only `go test`**, because the agent's tests can share its own mistake (the DELETE route that returned 404);
- **read the `git diff`**: it can still make an unrequested choice, like a dependency or a subdirectory.

## Files

| File | Role |
|---|---|
| `agent.yaml` | The configuration and the system prompt, every setting commented. |
| `gemma-coder-flow.{drawio,svg,png}` | How the agent works (generated by `.claude/skills/drawio-fancy-diagram/generator/examples/gemma_coder.py`). |
| `gemma-coder-tuning.{drawio,svg,png}` | The settings chosen from the measurements (same generator). |
| `README.fr.md` | This document in French. |

## Licence

The structure of the prompt (Directives vs Inquiries, Research → Act → Validate, Strategic Re-evaluation) comes from Gemini CLI's system prompt (Apache-2.0, Copyright Google LLC), through [`gemini-style`](../gemini-style/).