atop-analyze

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295

# atop-analyze

Ask Claude what used your CPU last night.

Linux keeps no CPU history of its own — `/proc/stat` holds only cumulative
counters since boot, so `top` and `vmstat` derive their percentages by sampling
those twice. History exists only if something writes it down. [atop][atop] does,
including per-process CPU, which means you can go back and ask *what* was eating
the CPU at 03:00 rather than only *that* something was.

This script reads those logs, builds a compact digest, and hands it to Claude
for interpretation.

## Requirements

- `atop`, with its logging service running (see [Setup](#setup))
- Python 3.10+ (standard library only)
- [Claude Code][cc] on `PATH`, logged in

The model runs through `claude -p`, so this uses whatever credential Claude Code
is logged in with — a Pro/Max subscription included. No `ANTHROPIC_API_KEY` and
no `anthropic` package. `--digest-only` needs neither Claude Code nor a network.

## Usage

```sh
./atop_analyze.py                      # today
./atop_analyze.py --date 20260920      # a past day
./atop_analyze.py -b 03:00 -e 05:00    # a window
./atop_analyze.py -q "why the iowait spike at 04:10?"
./atop_analyze.py --digest-only        # just the numbers, no model
./atop_analyze.py --clickhouse-url http://localhost:8123 --clickhouse-init
```

Useful flags: `--top N` (processes to rank), `--max-rows N` (timeline rows),
`--effort low|medium|high|xhigh|max`, `--model`, `--claude-bin`.

## ClickHouse export

Pass `--clickhouse-url` (or set `CLICKHOUSE_URL`) to persist every parsed
interval to ClickHouse. The export happens before the optional Claude call, so
it also works with `--digest-only`:

```sh
# One-time schema setup, then import today's atop log.
./atop_analyze.py --digest-only \
  --clickhouse-url http://localhost:8123 \
  --clickhouse-init

# Credentials can stay out of shell history.
export CLICKHOUSE_URL=https://clickhouse.example:8443
export CLICKHOUSE_USER=atop_writer
export CLICKHOUSE_PASSWORD='…'
./atop_analyze.py --date 20260920 --digest-only
```

`--clickhouse-init` creates `atop_samples` (one host interval per row) and
`atop_processes` (one observed process per interval) in the selected database.
Use `--clickhouse-database`, `--clickhouse-sample-table`, and
`--clickhouse-process-table` to change those names. Existing tables are never
modified; omit `--clickhouse-init` once the schema exists.

The sample table records CPU mode percentages, load averages, context switches,
interrupts, host, epoch, core count, and interval length. The process table
records host, epoch, PID, process name, whole-machine CPU percentage, and
interval length. Tables use `ReplacingMergeTree`, ordered by their natural
sample keys, so reimporting the same log converges to one row per key after
ClickHouse merges parts.

### Scale-to-zero Modal endpoint

For low-volume ingestion, [modal_clickhouse.py](modal_clickhouse.py) runs the
ClickHouse HTTP API as a protected Modal Web Function. It starts only when an
HTTP request arrives and is eligible to scale to zero after five minutes idle.
Its data directory is retained in the named `atop-clickhouse-data` Modal
Volume. It intentionally limits Modal to one container because the Volume must
not have concurrent ClickHouse writers.

```sh
# Creates the Volume automatically during deployment.
modal deploy modal_clickhouse.py

# Create a Modal Proxy Token, then use the deployed URL printed above.
modal workspace proxy-tokens --help
export CLICKHOUSE_URL='https://<workspace>--atop-clickhouse-clickhouse.modal.run'
export MODAL_KEY='wk-…'
export MODAL_SECRET='ws-…'
./atop_analyze.py --digest-only --clickhouse-init
```

Modal Proxy Token credentials protect the endpoint at the edge; the collector
passes them as `Modal-Key` and `Modal-Secret`. They may also be supplied with
`--clickhouse-modal-key` and `--clickhouse-modal-secret` instead of environment
variables. Expect a cold-start delay on the first request after the endpoint
has scaled down. Grafana's ClickHouse data source allows for that startup delay
with a 120-second connection timeout and a 180-second query timeout; a failed
startup still surfaces as an error rather than an indefinite loading state.

### Modal ingestion app

`modal_atop_analyze.py` makes the collector a Modal app. Its local entrypoint
reads the atop binary log from the host where it exists, while a Modal Function
parses it and writes it to private ClickHouse. The ClickHouse URL and Proxy
Token are stored in Modal's `atop-clickhouse-proxy` Secret, so the local command
does not need `CLICKHOUSE_URL`, `MODAL_KEY`, or `MODAL_SECRET`.

```sh
modal deploy modal_atop_analyze.py
modal run modal_atop_analyze.py --date 20260921 --initialize
modal run modal_atop_analyze.py --begin 03:00 --end 05:00
```

Use `--file /path/to/atop_log` or `--logdir /path/to/logdir` when the source
log is elsewhere. The command still needs a normal Modal CLI login to invoke
the Function, but no longer has direct database credentials.

#### Automatic daily export

The included systemd units export the completed prior-day log at 00:15. This
avoids repeatedly importing a still-growing log and works even if the machine
was off at the scheduled time (`Persistent=true`). They are configured for this
host's `nandi` user and Modal CLI location:

```sh
run0 python3 systemd/install.py --enable
systemctl list-timers atop-modal-export.timer
```

To test it without waiting until midnight, run:

```sh
just fire
just logs
```

If an export fails, `atop-modal-export-failure.service` writes a high-priority
journal record and displays a short `wall` message to active local terminals.
Inspect failures with:

```sh
journalctl -u atop-modal-export.service --no-pager -n 100
```

#### Task shortcuts

The repository `justfile` keeps the routine commands in one place:

```sh
just install      # install/update units and enable the timer
just status       # timer state and next run
just fire         # run yesterday's scheduled export now
just test-today   # manually export today's available samples
just logs         # recent exporter output
```

### Grafana dashboard

`modal_grafana.py` adds a browser-facing Grafana service with a provisioned
ClickHouse data source and an **Atop overview** dashboard (CPU/iowait, load,
and top processes). Grafana is public at the HTTP layer but authenticates every
visitor through Pocket ID; its Modal Proxy Token stays in a Modal Secret and is
never sent to the browser.

Before deploying, create a ClickHouse user named `grafana_reader` with a
nonempty password. Grant it `SELECT` on `default.atop_samples` and
`default.atop_processes`, plus `SHOW TABLES` and `SHOW DATABASES` for the
`default` database. Store the matching password in a separate Modal Secret
named `atop-grafana-clickhouse-auth` under the key
`CLICKHOUSE_GRAFANA_PASSWORD`. Grafana requires that Secret at deployment;
its `default` ClickHouse user remains passwordless for ingestion but is not
used by Grafana. Do not put the password in the repository.

```sh
# In Pocket ID, create a confidential OIDC client and restrict it to the users
# or groups that may view Grafana. Its redirect URI must exactly be:
# ${GRAFANA_URL}/login/generic_oauth
#
# Find the four endpoint values in Pocket ID’s OIDC Discovery URL:
# https://<pocket-id-host>/.well-known/openid-configuration
# Use its authorization_endpoint, token_endpoint, userinfo_endpoint, and
# jwks_uri values below. GRAFANA_URL is the public Modal URL, with no trailing
# slash (for example, https://<workspace>--atop-grafana-grafana.modal.run).
# The ClickHouse URL is hostname only: no https:// prefix and no path.
modal secret create atop-grafana \
  CLICKHOUSE_HOST='codegod100--atop-clickhouse-clickhouse.modal.run' \
  MODAL_KEY='wk-…' \
  MODAL_SECRET='ws-…' \
  GF_SERVER_ROOT_URL='https://<workspace>--atop-grafana-grafana.modal.run' \
  GF_AUTH_GENERIC_OAUTH_CLIENT_ID='…' \
  GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET='…' \
  GF_AUTH_GENERIC_OAUTH_AUTH_URL='<authorization-url-from-pocket-id>' \
  GF_AUTH_GENERIC_OAUTH_TOKEN_URL='<token-url-from-pocket-id>' \
  GF_AUTH_GENERIC_OAUTH_API_URL='<userinfo-url-from-pocket-id>' \
  GF_AUTH_GENERIC_OAUTH_JWK_SET_URL='<jwks-url-from-pocket-id>'

modal deploy modal_grafana.py
```

The deployment creates a persistent Modal Volume named `atop-grafana-data`
and mounts it at `/var/lib/grafana`, where Grafana stores its SQLite database
and saved settings. The ClickHouse plugin stays in the image at
`/opt/grafana/plugins`, so the data mount does not hide it. Keep Grafana at one
container while using SQLite; back up the Volume before replacing or deleting
it. For example, run
`modal volume get atop-grafana-data grafana.db ./grafana.db.backup` and store
that backup securely because the database may contain credentials. To recover,
stop Grafana, upload the backup with
`modal volume put --force atop-grafana-data ./grafana.db.backup grafana.db`,
then redeploy. If an older deployment already has dashboards or other changes
not in provisioning, migrate its database before deploying this new mount.

Or run `python create_grafana_secret.py` to fetch those four public endpoint
values from Pocket ID's discovery document and securely prompt for the client
credentials and Modal token. It creates the same `atop-grafana` secret without
printing the private values. It does not create the separate ClickHouse user or
password Secret.

Grafana redirects visitors to Pocket ID and does not expose a local password
form. It uses PKCE, validates the signed ID token with Pocket ID’s JWKS, and
requests a refresh token so an active Grafana session does not end at access
token expiry. Grafana scales to zero after five idle minutes; the first visit
after that can take a short time while Grafana and ClickHouse start. The source
and starter dashboard are under `grafana/provisioning/`.

## What it sends

Not the raw log — a day of samples would be millions of tokens. It sends a
bounded digest with three parts:

- **Timeline** — per-interval sys/user/wait/irq/steal, load average, top 3
  consumers. Samples beyond `--max-rows` (default 60) are averaged into buckets,
  so a full day stays compact.
- **Top processes** — ranked by CPU-seconds over the window, with average cores
  and distinct pid count.
- **Peak sample** — the busiest interval broken down by process.

Percentages are of the **whole machine**: one saturated core on an 8-core box is
~12.5%, not 100%. The system prompt tells the model as much, along with the other
traps that make this analysis go wrong — iowait is I/O blocking rather than CPU
pressure, load average counts uninterruptible sleep too, and coarse intervals
average short spikes away.

## Setup

```sh
# Arch
run0 pacman -S atop
run0 systemctl enable --now atop.service atopacct.service atop-rotate.timer
```

`atopacct.service` adds process accounting, so processes that exited during an
interval are still attributed. `atop-rotate.timer` rolls the log at midnight and
prunes old ones — without it atop appends to one file forever.

The default sample interval is 600s, which averages a two-minute spike into
near-invisibility. For chasing spikes, drop it to 60s:

```sh
run0 sed -i 's/^LOGINTERVAL=.*/LOGINTERVAL=60/' /etc/default/atop
run0 systemctl restart atop.service
```

That is ~10x the samples; `LOGGENERATIONS` in the same file controls retention
(28 days by default).

## Reading atop directly

The digest is a summary. For the full picture:

```sh
atop -r /var/log/atop/atop_20260921            # t / T to step samples, c for command lines
atop -r /var/log/atop/atop_20260921 -b 03:00   # jump to a time
```

## Notes

Field offsets come from `atop(1)`'s PARSABLE OUTPUT section. Two things that bite
when parsing it yourself:

- `PRC` emits a line per **thread** as well as per process, and a
  single-threaded process's thread line duplicates it exactly. Filter on the
  `is_process` field or you will double-count.
- With `-b`/`-e`, atop interleaves its *interactive* report into stdout, and
  those lines can begin with a valid label (lowercase `cpu`). A label whitelist
  alone is not enough.

`-Z` keeps the field count constant at 27; without it, spaces in process names
make it vary.

atop's first sample after a `RESET` marker holds totals since boot rather than
interval deltas, so it is skipped. A freshly started atop therefore has nothing
to report until one more interval elapses.

[atop]: https://www.atoptool.nl/
[cc]: https://claude.com/claude-code