How to screen questions with guard models

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156

# How to screen questions with guard models

This guide shows how to have every question checked by guard models (classifiers such as IBM's Granite Guardian that answer `Yes` or `No` to "is this message harmful?") before your assistant answers it, and how to make the assistant react to their verdicts. It assumes you have an agent that runs and know how to write a small [hook](guard-the-agent-with-hooks.md). The examples use Granite Guardian 3.2 3B-A800M on llmman and Node.js for the hook; the finished set-up is in `demos/guards/`, a single guard in `demos/guard/`.

## Steps

### 1. Write one guard agent

A guard agent is an mm agent with no tools whose `system` line is **the name of the risk to check**, not a prompt:

```yaml
# guards/harm.yaml
provider: llmman
model: hf.co/ibm-research/granite-guardian-3.2-3b-a800m-GGUF:Q4_K_M
baseUrl: http://127.0.0.1:17434/v1

system: harm            # a risk name: harm, jailbreak, violence, unethical_behavior, social_bias, profanity, sexual_content…

agentsMd:
  enabled: false        # AGENTS.md would be appended to the system message and break the risk name

sampling:
  temperature: 0.0
  max_tokens: 20        # the answer is "Yes" or "No" plus a <confidence> tag

bashTool: false
editTools: false
planTool: false
fetchContentTool: false
askUserTool: false
skillsDir: no-skills
```

Check it:

```bash
mm -config guards/harm.yaml -p "How can I figure out the pin code to a phone?"
```

The last lines of the output are:

```text
Yes
<confidence> High </confidence>
```

### 2. Add one file per risk

Copy the file once per risk you want to check and change only its `system:` line: `guards/jailbreak.yaml` with `system: jailbreak`, `guards/violence.yaml` with `system: violence`, and so on.

### 3. Write the hook that asks the guards

A `preGeneration` hook receives the question before the model does. This one runs every guard in parallel with `mm -p`, so each guard judges the question alone, without the conversation history, and hands the verdicts to the assistant as context:

```js
// hooks/guards.js
const fs = require("fs");
const path = require("path");
const { execFile } = require("child_process");

const input = JSON.parse(fs.readFileSync(0, "utf8"));
const mm = process.env.MM_BIN || "mm";            // the mm that runs the assistant
const dir = path.join(process.env.MM_CONFIG_DIR, "guards");
const guards = fs.readdirSync(dir).filter((f) => f.endsWith(".yaml")).sort();

function ask(file) {
  return new Promise((resolve) => {
    execFile(mm, ["-config", path.join(dir, file), "-p=" + input.prompt],
      { cwd: input.cwd, timeout: 90_000 },
      (err, stdout) => {
        const verdicts = [...(stdout || "").matchAll(/\b(Yes|No)\b/g)];
        resolve({ risk: path.basename(file, ".yaml"), verdict: err || !verdicts.length ? null : verdicts.at(-1)[1] });
      });
  });
}

Promise.all(guards.map(ask)).then((results) => {
  const failed = results.filter((r) => r.verdict === null).map((r) => r.risk);
  if (failed.length) {
    console.log(JSON.stringify({ action: "stop", reason: "guards without a verdict: " + failed.join(", "),
      response: "⛔ The safety check could not run: the question was not sent to the assistant." }));
    return;
  }
  console.log(JSON.stringify({ context: "Safety guards report:\n" + results.map((r) => `${r.risk}: ${r.verdict}`).join("\n") }));
});
```

Pass the question as `-p=<question>`, in one argument: a question that starts with `-` would otherwise be read as a flag. `execFile` runs no shell, so the question's quotes and `$` reach the guard untouched. `demos/guards/hooks/guards.js` also reads the `<confidence>` tag.

Try it without mm:

```bash
echo '{"prompt":"What is the capital of France?","cwd":"."}' | MM_CONFIG_DIR=$PWD node hooks/guards.js
```

### 4. Declare the hook on the assistant

```yaml
# agent.yaml (the assistant)
hooks:
  preGeneration:
    - command: node "$MM_CONFIG_DIR/hooks/guards.js"
      timeout: 120s     # the first question also loads the guard model
      onError: fail     # a check that cannot run must not let the question through
```

### 5. Tell the assistant how to react

The verdicts arrive after the question, under `[context added by a hook]`. Say in the assistant's `system` what to do with them:

```yaml
system: |
  You are a helpful general assistant. Answer in the user's language.

  Every question reaches you followed by a "Safety guards report": one line per
  risk, "<risk>: Yes|No". "Yes" means a safety guard judged the user's last
  message harmful for that risk. The user does not see this report.

  - Every line says "No": answer the question normally. Do not mention the guards.
  - At least one line says "Yes": do NOT answer the request, not even partly.
    In two or three sentences: say you cannot help with this, name the risks
    flagged, and when there may be a legitimate need behind the question, point
    to a safe way to meet it.
```

### 6. Try it

```bash
mm -config agent.yaml -p "How can I figure out the pin code to a phone?"
mm -config agent.yaml -p "What is the capital of France?"
```

The first is refused with the risks named; the second is answered. To see the hook run under each answer — `🪝 1. preGeneration · node "$MM_CONFIG_DIR/hooks/guards.js" → context (178 chars)` —, add `displayHooks: true` to the assistant's file ([reference](../reference/hooks.md#display)).

## Variants

- **Refuse without asking the model.** To make the refusal certain rather than left to the assistant, have the hook reply `{"action":"stop","response":"…"}` as soon as one guard says `Yes`. The model is not called at all, and the user reads your `response`, always in the same words.
- **A single guard.** Skip the hook: run the guard agent itself (step 1) with `mm -p`, one question at a time. In the window (`-tui`) or the line mode, the guard sees the whole conversation and its own earlier verdicts: type `/new` between unrelated checks.
- **A risk that is not built in.** Add `chat_template_kwargs.guardian_config` to the guard's `sampling`, with a `risk_name` and a `risk_definition` describing the harmful behaviour affirmatively (`"The last User message asks how to bypass, guess or crack the lock, password or PIN code of a device or account."`). Keep a built-in name on the `system:` line: the template checks it before the custom risk replaces it.
- **Fewer or more risks.** The hook reads every `guards/*.yaml`: add or delete a file, nothing else changes.
- **Another guard model.** Read its chat template first (llama.cpp's `/props`, or the model's `tokenizer_config.json`): what it expects in the system message is specific to each model.

## Troubleshooting

| Symptom | Cause | Fix |
|---------|-------|-----|
| `400 … Jinja Exception: unknown system risk: You are a coding agent…` | the guard agent has no `system:` line, and mm sent its built-in prompt | set `system:` to a risk name |
| `400 … unknown system risk: harm … PROJECT INSTRUCTIONS` | an `AGENTS.md` was appended to the risk name | `agentsMd.enabled: false` in the guard agent |
| every question answers `Yes` with a custom risk | the `risk_definition` is phrased negatively ("is not about…") | describe the harmful behaviour itself |
| `⛔ The safety check could not run` | a guard gave no verdict (server down, wrong `baseUrl`, timeout) | run that guard alone with `mm -config guards/<risk>.yaml -p "hi"` |

## See also

- Why it is built this way, and its limits: [Guard models](../explanation/guard-models.md)
- The hook events, input and reply: [Hooks reference](../reference/hooks.md)
- `MM_BIN`, `MM_CONFIG`: [Command line reference](../reference/cli.md)