atop-analyze

Ask Claude what used your CPU last night.

Linux keeps no CPU history of its own — /proc/stat holds only cumulative
counters since boot, so top and vmstat derive their percentages by sampling
those twice. History exists only if something writes it down. atop does,
including per-process CPU, which means you can go back and ask what was eating
the CPU at 03:00 rather than only that something was.

This script reads those logs, builds a compact digest, and hands it to Claude
for interpretation.

Requirements

atop, with its logging service running (see Setup)
Python 3.10+ (standard library only)
Claude Code on PATH, logged in

The model runs through claude -p, so this uses whatever credential Claude Code
is logged in with — a Pro/Max subscription included. No ANTHROPIC_API_KEY and
no anthropic package. --digest-only needs neither Claude Code nor a network.

Usage

./atop_analyze.py                      # today
./atop_analyze.py --date 20260920      # a past day
./atop_analyze.py -b 03:00 -e 05:00    # a window
./atop_analyze.py -q "why the iowait spike at 04:10?"
./atop_analyze.py --digest-only        # just the numbers, no model
./atop_analyze.py --clickhouse-url http://localhost:8123 --clickhouse-init

Useful flags: --top N (processes to rank), --max-rows N (timeline rows),
--effort low|medium|high|xhigh|max, --model, --claude-bin.

ClickHouse export

Pass --clickhouse-url (or set CLICKHOUSE_URL) to persist every parsed
interval to ClickHouse. The export happens before the optional Claude call, so
it also works with --digest-only:

# One-time schema setup, then import today's atop log.
./atop_analyze.py --digest-only \
  --clickhouse-url http://localhost:8123 \
  --clickhouse-init

# Credentials can stay out of shell history.
export CLICKHOUSE_URL=https://clickhouse.example:8443
export CLICKHOUSE_USER=atop_writer
export CLICKHOUSE_PASSWORD='…'
./atop_analyze.py --date 20260920 --digest-only

--clickhouse-init creates atop_samples (one host interval per row) and
atop_processes (one observed process per interval) in the selected database.
Use --clickhouse-database, --clickhouse-sample-table, and
--clickhouse-process-table to change those names. Existing tables are never
modified; omit --clickhouse-init once the schema exists.

The sample table records CPU mode percentages, load averages, context switches,
interrupts, host, epoch, core count, and interval length. The process table
records host, epoch, PID, process name, whole-machine CPU percentage, and
interval length. Tables use ReplacingMergeTree, ordered by their natural
sample keys, so reimporting the same log converges to one row per key after
ClickHouse merges parts.

For low-volume ingestion, modal_clickhouse.py runs the
ClickHouse HTTP API as a protected Modal Web Function. It starts only when an
HTTP request arrives and is eligible to scale to zero after 60 seconds idle.
Its data directory is retained in the named atop-clickhouse-data Modal
Volume. It intentionally limits Modal to one container because the Volume must
not have concurrent ClickHouse writers.

# Creates the Volume automatically during deployment.
modal deploy modal_clickhouse.py

# Create a Modal Proxy Token, then use the deployed URL printed above.
modal workspace proxy-tokens --help
export CLICKHOUSE_URL='https://<workspace>--atop-clickhouse-clickhouse.modal.run'
export MODAL_KEY='wk-…'
export MODAL_SECRET='ws-…'
./atop_analyze.py --digest-only --clickhouse-init

Modal Proxy Token credentials protect the endpoint at the edge; the collector
passes them as Modal-Key and Modal-Secret. They may also be supplied with
--clickhouse-modal-key and --clickhouse-modal-secret instead of environment
variables. Expect a cold-start delay on the first request after the endpoint
has scaled down.

modal_atop_analyze.py makes the collector a Modal app. Its local entrypoint
reads the atop binary log from the host where it exists, while a Modal Function
parses it and writes it to private ClickHouse. The ClickHouse URL and Proxy
Token are stored in Modal's atop-clickhouse-proxy Secret, so the local command
does not need CLICKHOUSE_URL, MODAL_KEY, or MODAL_SECRET.

modal deploy modal_atop_analyze.py
modal run modal_atop_analyze.py --date 20260921 --initialize
modal run modal_atop_analyze.py --begin 03:00 --end 05:00

Use --file /path/to/atop_log or --logdir /path/to/logdir when the source
log is elsewhere. The command still needs a normal Modal CLI login to invoke
the Function, but no longer has direct database credentials.

Automatic daily export

The included systemd units export the completed prior-day log at 00:15. This
avoids repeatedly importing a still-growing log and works even if the machine
was off at the scheduled time (Persistent=true). They are configured for this
host's nandi user and Modal CLI location:

run0 python3 systemd/install.py --enable
systemctl list-timers atop-modal-export.timer

To test it without waiting until midnight, run:

just fire
just logs

If an export fails, atop-modal-export-failure.service writes a high-priority
journal record and displays a short wall message to active local terminals.
Inspect failures with:

journalctl -u atop-modal-export.service --no-pager -n 100

Task shortcuts

The repository justfile keeps the routine commands in one place:

just install      # install/update units and enable the timer
just status       # timer state and next run
just fire         # run yesterday's scheduled export now
just test-today   # manually export today's available samples
just logs         # recent exporter output

Grafana dashboard

modal_grafana.py adds a browser-facing Grafana service with a provisioned
ClickHouse data source and an Atop overview dashboard (CPU/iowait, load,
and top processes). Grafana is public at the HTTP layer but authenticates every
visitor through Pocket ID; its Modal Proxy Token stays in a Modal Secret and is
never sent to the browser.

# In Pocket ID, create a confidential OIDC client and restrict it to the users
# or groups that may view Grafana. Its redirect URI must exactly be:
# ${GRAFANA_URL}/login/generic_oauth
#
# Find the four endpoint values in Pocket ID’s OIDC Discovery URL:
# https://<pocket-id-host>/.well-known/openid-configuration
# Use its authorization_endpoint, token_endpoint, userinfo_endpoint, and
# jwks_uri values below. GRAFANA_URL is the public Modal URL, with no trailing
# slash (for example, https://<workspace>--atop-grafana-grafana.modal.run).
# The ClickHouse URL is hostname only: no https:// prefix and no path.
modal secret create atop-grafana \
  CLICKHOUSE_HOST='codegod100--atop-clickhouse-clickhouse.modal.run' \
  MODAL_KEY='wk-…' \
  MODAL_SECRET='ws-…' \
  GF_SERVER_ROOT_URL='https://<workspace>--atop-grafana-grafana.modal.run' \
  GF_AUTH_GENERIC_OAUTH_CLIENT_ID='…' \
  GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET='…' \
  GF_AUTH_GENERIC_OAUTH_AUTH_URL='<authorization-url-from-pocket-id>' \
  GF_AUTH_GENERIC_OAUTH_TOKEN_URL='<token-url-from-pocket-id>' \
  GF_AUTH_GENERIC_OAUTH_API_URL='<userinfo-url-from-pocket-id>' \
  GF_AUTH_GENERIC_OAUTH_JWK_SET_URL='<jwks-url-from-pocket-id>'

modal deploy modal_grafana.py

Or run python create_grafana_secret.py to fetch those four public endpoint
values from Pocket ID's discovery document and securely prompt for the client
credentials and Modal token. It creates the same atop-grafana secret without
printing the private values.

Grafana redirects visitors to Pocket ID and does not expose a local password
form. It uses PKCE, validates the signed ID token with Pocket ID’s JWKS, and
requests a refresh token so an active Grafana session does not end at access
token expiry. Grafana scales to zero after five idle minutes; the first visit
after that can take a short time while Grafana and ClickHouse start. The source
and starter dashboard are under grafana/provisioning/.

What it sends

Not the raw log — a day of samples would be millions of tokens. It sends a
bounded digest with three parts:

Timeline — per-interval sys/user/wait/irq/steal, load average, top 3
consumers. Samples beyond --max-rows (default 60) are averaged into buckets,
so a full day stays compact.
Top processes — ranked by CPU-seconds over the window, with average cores
and distinct pid count.
Peak sample — the busiest interval broken down by process.

Percentages are of the whole machine: one saturated core on an 8-core box is
~12.5%, not 100%. The system prompt tells the model as much, along with the other
traps that make this analysis go wrong — iowait is I/O blocking rather than CPU
pressure, load average counts uninterruptible sleep too, and coarse intervals
average short spikes away.

Setup

# Arch
run0 pacman -S atop
run0 systemctl enable --now atop.service atopacct.service atop-rotate.timer

atopacct.service adds process accounting, so processes that exited during an
interval are still attributed. atop-rotate.timer rolls the log at midnight and
prunes old ones — without it atop appends to one file forever.

The default sample interval is 600s, which averages a two-minute spike into
near-invisibility. For chasing spikes, drop it to 60s:

run0 sed -i 's/^LOGINTERVAL=.*/LOGINTERVAL=60/' /etc/default/atop
run0 systemctl restart atop.service

That is ~10x the samples; LOGGENERATIONS in the same file controls retention
(28 days by default).

Reading atop directly

The digest is a summary. For the full picture:

atop -r /var/log/atop/atop_20260921            # t / T to step samples, c for command lines
atop -r /var/log/atop/atop_20260921 -b 03:00   # jump to a time

Notes

Field offsets come from atop(1)'s PARSABLE OUTPUT section. Two things that bite
when parsing it yourself:

PRC emits a line per thread as well as per process, and a
single-threaded process's thread line duplicates it exactly. Filter on the
is_process field or you will double-count.
With -b/-e, atop interleaves its interactive report into stdout, and
those lines can begin with a valid label (lowercase cpu). A label whitelist
alone is not enough.

-Z keeps the field count constant at 27; without it, spaces in process names
make it vary.

atop's first sample after a RESET marker holds totals since boot rather than
interval deltas, so it is skipped. A freshly started atop therefore has nothing
to report until one more interval elapses.