rickub
nandi/atop-analyzepublic⑂ Fork 0
⑂ bf0b684
Branches
main View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/nandi/atop-analyze.git
git clone ssh://git@rickub.com/nandi/atop-analyze.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Add ClickHouse export support and Modal deployment documentation

nandithebull committed 2026-09-21T19:29:22-07:00 Browse files
bf0b684 parent: 167e823
modified README.md +111 -0
@@ -29,11 +29,122 @@ no `anthropic` package. `--digest-only` needs neither Claude Code nor a network.
2929 ./atop_analyze.py -b 03:00 -e 05:00 # a window
3030 ./atop_analyze.py -q "why the iowait spike at 04:10?"
3131 ./atop_analyze.py --digest-only # just the numbers, no model
32+./atop_analyze.py --clickhouse-url http://localhost:8123 --clickhouse-init
3233 ```
3334
3435 Useful flags: `--top N` (processes to rank), `--max-rows N` (timeline rows),
3536 `--effort low|medium|high|xhigh|max`, `--model`, `--claude-bin`.
3637
38+## ClickHouse export
39+
40+Pass `--clickhouse-url` (or set `CLICKHOUSE_URL`) to persist every parsed
41+interval to ClickHouse. The export happens before the optional Claude call, so
42+it also works with `--digest-only`:
43+
44+```sh
45+# One-time schema setup, then import today's atop log.
46+./atop_analyze.py --digest-only \
47+ --clickhouse-url http://localhost:8123 \
48+ --clickhouse-init
49+
50+# Credentials can stay out of shell history.
51+export CLICKHOUSE_URL=https://clickhouse.example:8443
52+export CLICKHOUSE_USER=atop_writer
53+export CLICKHOUSE_PASSWORD='…'
54+./atop_analyze.py --date 20260920 --digest-only
55+```
56+
57+`--clickhouse-init` creates `atop_samples` (one host interval per row) and
58+`atop_processes` (one observed process per interval) in the selected database.
59+Use `--clickhouse-database`, `--clickhouse-sample-table`, and
60+`--clickhouse-process-table` to change those names. Existing tables are never
61+modified; omit `--clickhouse-init` once the schema exists.
62+
63+The sample table records CPU mode percentages, load averages, context switches,
64+interrupts, host, epoch, core count, and interval length. The process table
65+records host, epoch, PID, process name, whole-machine CPU percentage, and
66+interval length. Tables use `ReplacingMergeTree`, ordered by their natural
67+sample keys, so reimporting the same log converges to one row per key after
68+ClickHouse merges parts.
69+
70+### Scale-to-zero Modal endpoint
71+
72+For low-volume ingestion, [modal_clickhouse.py](modal_clickhouse.py) runs the
73+ClickHouse HTTP API as a protected Modal Web Function. It starts only when an
74+HTTP request arrives and is eligible to scale to zero after 60 seconds idle.
75+Its data directory is retained in the named `atop-clickhouse-data` Modal
76+Volume. It intentionally limits Modal to one container because the Volume must
77+not have concurrent ClickHouse writers.
78+
79+```sh
80+# Creates the Volume automatically during deployment.
81+modal deploy modal_clickhouse.py
82+
83+# Create a Modal Proxy Token, then use the deployed URL printed above.
84+modal workspace proxy-tokens --help
85+export CLICKHOUSE_URL='https://<workspace>--atop-clickhouse-clickhouse.modal.run'
86+export MODAL_KEY='wk-…'
87+export MODAL_SECRET='ws-…'
88+./atop_analyze.py --digest-only --clickhouse-init
89+```
90+
91+Modal Proxy Token credentials protect the endpoint at the edge; the collector
92+passes them as `Modal-Key` and `Modal-Secret`. They may also be supplied with
93+`--clickhouse-modal-key` and `--clickhouse-modal-secret` instead of environment
94+variables. Expect a cold-start delay on the first request after the endpoint
95+has scaled down.
96+
97+### Modal ingestion app
98+
99+`modal_atop_analyze.py` makes the collector a Modal app. Its local entrypoint
100+reads the atop binary log from the host where it exists, while a Modal Function
101+parses it and writes it to private ClickHouse. The ClickHouse URL and Proxy
102+Token are stored in Modal's `atop-clickhouse-proxy` Secret, so the local command
103+does not need `CLICKHOUSE_URL`, `MODAL_KEY`, or `MODAL_SECRET`.
104+
105+```sh
106+modal deploy modal_atop_analyze.py
107+modal run modal_atop_analyze.py --date 20260921 --initialize
108+modal run modal_atop_analyze.py --begin 03:00 --end 05:00
109+```
110+
111+Use `--file /path/to/atop_log` or `--logdir /path/to/logdir` when the source
112+log is elsewhere. The command still needs a normal Modal CLI login to invoke
113+the Function, but no longer has direct database credentials.
114+
115+### Grafana dashboard
116+
117+`modal_grafana.py` adds a browser-facing Grafana service with a provisioned
118+ClickHouse data source and an **Atop overview** dashboard (CPU/iowait, load,
119+and top processes). Grafana is public at the HTTP layer but requires its own
120+admin login; its Modal Proxy Token stays in a Modal Secret and is never sent to
121+the browser.
122+
123+```sh
124+# In Pocket ID, create an OIDC client whose redirect URI is:
125+# https://codegod100--atop-grafana-grafana.modal.run/login/generic_oauth
126+# Copy the client credentials and OIDC endpoints Pocket ID displays.
127+# The ClickHouse URL is hostname only: no https:// prefix and no path.
128+modal secret create atop-grafana \
129+ CLICKHOUSE_HOST='codegod100--atop-clickhouse-clickhouse.modal.run' \
130+ MODAL_KEY='wk-…' \
131+ MODAL_SECRET='ws-…' \
132+ GF_SECURITY_ADMIN_PASSWORD='emergency-local-admin-password' \
133+ GF_AUTH_GENERIC_OAUTH_CLIENT_ID='…' \
134+ GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET='…' \
135+ GF_AUTH_GENERIC_OAUTH_AUTH_URL='<authorization-url-from-pocket-id>' \
136+ GF_AUTH_GENERIC_OAUTH_TOKEN_URL='<token-url-from-pocket-id>' \
137+ GF_AUTH_GENERIC_OAUTH_API_URL='<userinfo-url-from-pocket-id>' \
138+ GF_AUTH_GENERIC_OAUTH_JWK_SET_URL='<jwks-url-from-pocket-id>'
139+
140+modal deploy modal_grafana.py
141+```
142+
143+Grafana redirects visitors to the configured Generic OIDC provider; it does not
144+expose a local password form. Grafana scales to zero after five idle minutes;
145+the first visit after that can take a short time while Grafana and ClickHouse
146+start. The source and starter dashboard are under `grafana/provisioning/`.
147+
37148 ## What it sends
38149
39150 Not the raw log — a day of samples would be millions of tokens. It sends a
@@ -29,11 +29,122 @@ no `anthropic` package. `--digest-only` needs neither Claude Code nor a network.
29 ./atop_analyze.py -b 03:00 -e 05:00 # a window29 ./atop_analyze.py -b 03:00 -e 05:00 # a window
30 ./atop_analyze.py -q "why the iowait spike at 04:10?"30 ./atop_analyze.py -q "why the iowait spike at 04:10?"
31 ./atop_analyze.py --digest-only # just the numbers, no model31 ./atop_analyze.py --digest-only # just the numbers, no model
32+./atop_analyze.py --clickhouse-url http://localhost:8123 --clickhouse-init
32 ```33 ```
33 34
34 Useful flags: `--top N` (processes to rank), `--max-rows N` (timeline rows),35 Useful flags: `--top N` (processes to rank), `--max-rows N` (timeline rows),
35 `--effort low|medium|high|xhigh|max`, `--model`, `--claude-bin`.36 `--effort low|medium|high|xhigh|max`, `--model`, `--claude-bin`.
36 37
38+## ClickHouse export
39+
40+Pass `--clickhouse-url` (or set `CLICKHOUSE_URL`) to persist every parsed
41+interval to ClickHouse. The export happens before the optional Claude call, so
42+it also works with `--digest-only`:
43+
44+```sh
45+# One-time schema setup, then import today's atop log.
46+./atop_analyze.py --digest-only \
47+ --clickhouse-url http://localhost:8123 \
48+ --clickhouse-init
49+
50+# Credentials can stay out of shell history.
51+export CLICKHOUSE_URL=https://clickhouse.example:8443
52+export CLICKHOUSE_USER=atop_writer
53+export CLICKHOUSE_PASSWORD='…'
54+./atop_analyze.py --date 20260920 --digest-only
55+```
56+
57+`--clickhouse-init` creates `atop_samples` (one host interval per row) and
58+`atop_processes` (one observed process per interval) in the selected database.
59+Use `--clickhouse-database`, `--clickhouse-sample-table`, and
60+`--clickhouse-process-table` to change those names. Existing tables are never
61+modified; omit `--clickhouse-init` once the schema exists.
62+
63+The sample table records CPU mode percentages, load averages, context switches,
64+interrupts, host, epoch, core count, and interval length. The process table
65+records host, epoch, PID, process name, whole-machine CPU percentage, and
66+interval length. Tables use `ReplacingMergeTree`, ordered by their natural
67+sample keys, so reimporting the same log converges to one row per key after
68+ClickHouse merges parts.
69+
70+### Scale-to-zero Modal endpoint
71+
72+For low-volume ingestion, [modal_clickhouse.py](modal_clickhouse.py) runs the
73+ClickHouse HTTP API as a protected Modal Web Function. It starts only when an
74+HTTP request arrives and is eligible to scale to zero after 60 seconds idle.
75+Its data directory is retained in the named `atop-clickhouse-data` Modal
76+Volume. It intentionally limits Modal to one container because the Volume must
77+not have concurrent ClickHouse writers.
78+
79+```sh
80+# Creates the Volume automatically during deployment.
81+modal deploy modal_clickhouse.py
82+
83+# Create a Modal Proxy Token, then use the deployed URL printed above.
84+modal workspace proxy-tokens --help
85+export CLICKHOUSE_URL='https://<workspace>--atop-clickhouse-clickhouse.modal.run'
86+export MODAL_KEY='wk-…'
87+export MODAL_SECRET='ws-…'
88+./atop_analyze.py --digest-only --clickhouse-init
89+```
90+
91+Modal Proxy Token credentials protect the endpoint at the edge; the collector
92+passes them as `Modal-Key` and `Modal-Secret`. They may also be supplied with
93+`--clickhouse-modal-key` and `--clickhouse-modal-secret` instead of environment
94+variables. Expect a cold-start delay on the first request after the endpoint
95+has scaled down.
96+
97+### Modal ingestion app
98+
99+`modal_atop_analyze.py` makes the collector a Modal app. Its local entrypoint
100+reads the atop binary log from the host where it exists, while a Modal Function
101+parses it and writes it to private ClickHouse. The ClickHouse URL and Proxy
102+Token are stored in Modal's `atop-clickhouse-proxy` Secret, so the local command
103+does not need `CLICKHOUSE_URL`, `MODAL_KEY`, or `MODAL_SECRET`.
104+
105+```sh
106+modal deploy modal_atop_analyze.py
107+modal run modal_atop_analyze.py --date 20260921 --initialize
108+modal run modal_atop_analyze.py --begin 03:00 --end 05:00
109+```
110+
111+Use `--file /path/to/atop_log` or `--logdir /path/to/logdir` when the source
112+log is elsewhere. The command still needs a normal Modal CLI login to invoke
113+the Function, but no longer has direct database credentials.
114+
115+### Grafana dashboard
116+
117+`modal_grafana.py` adds a browser-facing Grafana service with a provisioned
118+ClickHouse data source and an **Atop overview** dashboard (CPU/iowait, load,
119+and top processes). Grafana is public at the HTTP layer but requires its own
120+admin login; its Modal Proxy Token stays in a Modal Secret and is never sent to
121+the browser.
122+
123+```sh
124+# In Pocket ID, create an OIDC client whose redirect URI is:
125+# https://codegod100--atop-grafana-grafana.modal.run/login/generic_oauth
126+# Copy the client credentials and OIDC endpoints Pocket ID displays.
127+# The ClickHouse URL is hostname only: no https:// prefix and no path.
128+modal secret create atop-grafana \
129+ CLICKHOUSE_HOST='codegod100--atop-clickhouse-clickhouse.modal.run' \
130+ MODAL_KEY='wk-…' \
131+ MODAL_SECRET='ws-…' \
132+ GF_SECURITY_ADMIN_PASSWORD='emergency-local-admin-password' \
133+ GF_AUTH_GENERIC_OAUTH_CLIENT_ID='…' \
134+ GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET='…' \
135+ GF_AUTH_GENERIC_OAUTH_AUTH_URL='<authorization-url-from-pocket-id>' \
136+ GF_AUTH_GENERIC_OAUTH_TOKEN_URL='<token-url-from-pocket-id>' \
137+ GF_AUTH_GENERIC_OAUTH_API_URL='<userinfo-url-from-pocket-id>' \
138+ GF_AUTH_GENERIC_OAUTH_JWK_SET_URL='<jwks-url-from-pocket-id>'
139+
140+modal deploy modal_grafana.py
141+```
142+
143+Grafana redirects visitors to the configured Generic OIDC provider; it does not
144+expose a local password form. Grafana scales to zero after five idle minutes;
145+the first visit after that can take a short time while Grafana and ClickHouse
146+start. The source and starter dashboard are under `grafana/provisioning/`.
147+
37 ## What it sends148 ## What it sends
38 149
39 Not the raw log — a day of samples would be millions of tokens. It sends a150 Not the raw log — a day of samples would be millions of tokens. It sends a
modified atop_analyze.py +140 -2
@@ -21,11 +21,16 @@ fields -- label host epoch date time interval -- and the rest is label-specific.
2121 from __future__ import annotations
2222
2323 import argparse
24+import base64
25+import json
2426 import os
2527 import re
2628 import shutil
2729 import subprocess
2830 import sys
31+import urllib.error
32+import urllib.parse
33+import urllib.request
2934 from collections import defaultdict
3035 from dataclasses import dataclass, field
3136 from datetime import datetime
@@ -33,6 +38,8 @@ from datetime import datetime
3338 DEFAULT_LOGDIR = "/var/log/atop"
3439 CLAUDE_BIN = "claude"
3540 CLAUDE_MODEL = "opus"
41+CLICKHOUSE_URL_ENV = "CLICKHOUSE_URL"
42+IDENTIFIER_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
3643
3744 WANTED_LABELS = {"CPU", "CPL", "PRC"}
3845 DATE_RE = re.compile(r"^\d{4}/\d{2}/\d{2}$")
@@ -65,6 +72,7 @@ class Sample:
6572 epoch: int
6673 when: str
6774 interval: int
75+ host: str = ""
6876 ncpu: int = 0
6977 hertz: int = 100
7078 modes: dict[str, int] = field(default_factory=dict)
@@ -147,7 +155,7 @@ def parse(raw: str) -> list[Sample]:
147155 parts = line.split()
148156 if len(parts) < 7:
149157 continue
150- label, _host, epoch, date, tm = parts[:5]
158+ label, host, epoch, date, tm = parts[:5]
151159 # With -b/-e, atop interleaves its interactive-format report into
152160 # stdout. Those lines can start with a real label (lowercase "cpu"),
153161 # so also require the date field to actually be a date.
@@ -160,7 +168,7 @@ def parse(raw: str) -> list[Sample]:
160168 continue
161169
162170 if current is None:
163- current = Sample(epoch=int(epoch), when=tm, interval=interval)
171+ current = Sample(epoch=int(epoch), when=tm, interval=interval, host=host)
164172
165173 try:
166174 if label == "CPU":
@@ -192,6 +200,98 @@ def parse(raw: str) -> list[Sample]:
192200 return samples
193201
194202
203+def clickhouse_identifier(value: str, option: str) -> str:
204+ """Validate a database or table name before embedding it in SQL."""
205+ if not IDENTIFIER_RE.match(value):
206+ raise AtopError(f"{option} must contain only letters, digits, and underscores")
207+ return value
208+
209+
210+def clickhouse_request(
211+ url: str, query: str, body: str, user: str | None, password: str | None,
212+ modal_key: str | None, modal_secret: str | None,
213+) -> None:
214+ """Send one query to ClickHouse's HTTP interface."""
215+ separator = "&" if "?" in url else "?"
216+ endpoint = f"{url.rstrip('/')}{separator}{urllib.parse.urlencode({'query': query})}"
217+ request = urllib.request.Request(
218+ endpoint,
219+ data=body.encode(),
220+ headers={"Content-Type": "application/json; charset=utf-8"},
221+ method="POST",
222+ )
223+ if user is not None:
224+ token = base64.b64encode(f"{user}:{password or ''}".encode()).decode()
225+ request.add_header("Authorization", f"Basic {token}")
226+ if modal_key is not None:
227+ request.add_header("Modal-Key", modal_key)
228+ request.add_header("Modal-Secret", modal_secret or "")
229+ try:
230+ with urllib.request.urlopen(request, timeout=30) as response:
231+ if response.status >= 300:
232+ raise AtopError(f"ClickHouse returned HTTP {response.status}")
233+ except urllib.error.HTTPError as e:
234+ detail = e.read().decode(errors="replace").strip()
235+ raise AtopError(f"ClickHouse returned HTTP {e.code}: {detail}") from e
236+ except urllib.error.URLError as e:
237+ raise AtopError(f"could not reach ClickHouse at {url}: {e.reason}") from e
238+
239+
240+def write_clickhouse(
241+ samples: list[Sample], url: str, database: str, sample_table: str,
242+ process_table: str, user: str | None, password: str | None,
243+ modal_key: str | None, modal_secret: str | None, initialize: bool,
244+) -> None:
245+ """Store parsed atop samples and per-process CPU attribution in ClickHouse."""
246+ database = clickhouse_identifier(database, "--clickhouse-database")
247+ sample_table = clickhouse_identifier(sample_table, "--clickhouse-sample-table")
248+ process_table = clickhouse_identifier(process_table, "--clickhouse-process-table")
249+ samples_target = f"`{database}`.`{sample_table}`"
250+ processes_target = f"`{database}`.`{process_table}`"
251+
252+ if initialize:
253+ clickhouse_request(url, f"CREATE DATABASE IF NOT EXISTS `{database}`", "", user, password,
254+ modal_key, modal_secret)
255+ clickhouse_request(url, f"""CREATE TABLE IF NOT EXISTS {samples_target} (
256+ host LowCardinality(String), epoch DateTime, interval_seconds UInt32,
257+ ncpu UInt16, user_pct Float32, sys_pct Float32, nice_pct Float32,
258+ idle_pct Float32, wait_pct Float32, irq_pct Float32, softirq_pct Float32,
259+ steal_pct Float32, busy_pct Float32, load1 Float32, load5 Float32,
260+ load15 Float32, ctxsw UInt64, intr UInt64
261+ ) ENGINE = ReplacingMergeTree ORDER BY (host, epoch)""", "", user, password,
262+ modal_key, modal_secret)
263+ clickhouse_request(url, f"""CREATE TABLE IF NOT EXISTS {processes_target} (
264+ host LowCardinality(String), epoch DateTime, pid UInt32,
265+ name LowCardinality(String), cpu_pct Float32, interval_seconds UInt32
266+ ) ENGINE = ReplacingMergeTree ORDER BY (host, epoch, pid, name)""", "", user, password,
267+ modal_key, modal_secret)
268+
269+ sample_rows = [
270+ {
271+ "host": s.host, "epoch": s.epoch, "interval_seconds": s.interval,
272+ "ncpu": s.ncpu, "user_pct": s.pct("user"), "sys_pct": s.pct("sys"),
273+ "nice_pct": s.pct("nice"), "idle_pct": s.pct("idle"),
274+ "wait_pct": s.pct("wait"), "irq_pct": s.pct("irq"),
275+ "softirq_pct": s.pct("softirq"), "steal_pct": s.pct("steal"),
276+ "busy_pct": s.busy_pct, "load1": s.load1, "load5": s.load5,
277+ "load15": s.load15, "ctxsw": s.ctxsw, "intr": s.intr,
278+ }
279+ for s in samples
280+ ]
281+ process_rows = [
282+ {"host": s.host, "epoch": s.epoch, "pid": pid, "name": name,
283+ "cpu_pct": pct, "interval_seconds": s.interval}
284+ for s in samples for name, pid, pct in s.procs
285+ ]
286+ clickhouse_request(url, f"INSERT INTO {samples_target} FORMAT JSONEachRow",
287+ "\n".join(json.dumps(row) for row in sample_rows), user, password,
288+ modal_key, modal_secret)
289+ if process_rows:
290+ clickhouse_request(url, f"INSERT INTO {processes_target} FORMAT JSONEachRow",
291+ "\n".join(json.dumps(row) for row in process_rows), user, password,
292+ modal_key, modal_secret)
293+
294+
195295 def bucket(samples: list[Sample], max_rows: int) -> list[list[Sample]]:
196296 """Group samples into at most max_rows contiguous buckets."""
197297 if len(samples) <= max_rows:
@@ -337,6 +437,24 @@ def main() -> int:
337437 help="path to the claude CLI")
338438 p.add_argument("--digest-only", action="store_true",
339439 help="print the digest and exit; runs no model")
440+ p.add_argument("--clickhouse-url", default=os.environ.get(CLICKHOUSE_URL_ENV),
441+ help=f"ClickHouse HTTP endpoint (default: ${CLICKHOUSE_URL_ENV})")
442+ p.add_argument("--clickhouse-database", default="default",
443+ help="ClickHouse database (default: default)")
444+ p.add_argument("--clickhouse-sample-table", default="atop_samples",
445+ help="ClickHouse interval table (default: atop_samples)")
446+ p.add_argument("--clickhouse-process-table", default="atop_processes",
447+ help="ClickHouse process-attribution table (default: atop_processes)")
448+ p.add_argument("--clickhouse-user", default=os.environ.get("CLICKHOUSE_USER"),
449+ help="ClickHouse basic-auth user (default: $CLICKHOUSE_USER)")
450+ p.add_argument("--clickhouse-password", default=os.environ.get("CLICKHOUSE_PASSWORD"),
451+ help="ClickHouse basic-auth password (default: $CLICKHOUSE_PASSWORD)")
452+ p.add_argument("--clickhouse-modal-key", default=os.environ.get("MODAL_KEY"),
453+ help="Modal Proxy Token ID for a protected Modal endpoint (default: $MODAL_KEY)")
454+ p.add_argument("--clickhouse-modal-secret", default=os.environ.get("MODAL_SECRET"),
455+ help="Modal Proxy Token secret for a protected Modal endpoint (default: $MODAL_SECRET)")
456+ p.add_argument("--clickhouse-init", action="store_true",
457+ help="create the ClickHouse database and tables if missing")
340458 args = p.parse_args()
341459
342460 try:
@@ -356,6 +474,26 @@ def main() -> int:
356474 file=sys.stderr)
357475 return 1
358476
477+ if args.clickhouse_init and not args.clickhouse_url:
478+ p.error("--clickhouse-init requires --clickhouse-url or $CLICKHOUSE_URL")
479+ if bool(args.clickhouse_modal_key) != bool(args.clickhouse_modal_secret):
480+ p.error("--clickhouse-modal-key and --clickhouse-modal-secret must be supplied together")
481+ if args.clickhouse_url:
482+ try:
483+ write_clickhouse(
484+ samples, args.clickhouse_url, args.clickhouse_database,
485+ args.clickhouse_sample_table, args.clickhouse_process_table,
486+ args.clickhouse_user, args.clickhouse_password,
487+ args.clickhouse_modal_key, args.clickhouse_modal_secret,
488+ args.clickhouse_init,
489+ )
490+ except AtopError as e:
491+ print(f"error: ClickHouse export failed: {e}", file=sys.stderr)
492+ return 1
493+ process_count = sum(len(s.procs) for s in samples)
494+ print(f"# exported {len(samples)} samples and {process_count} process rows to ClickHouse",
495+ file=sys.stderr)
496+
359497 text = digest(samples, args.top, args.max_rows)
360498 if args.digest_only:
361499 print(text)
@@ -21,11 +21,16 @@ fields -- label host epoch date time interval -- and the rest is label-specific.
21 from __future__ import annotations21 from __future__ import annotations
22 22
23 import argparse23 import argparse
24+import base64
25+import json
24 import os26 import os
25 import re27 import re
26 import shutil28 import shutil
27 import subprocess29 import subprocess
28 import sys30 import sys
31+import urllib.error
32+import urllib.parse
33+import urllib.request
29 from collections import defaultdict34 from collections import defaultdict
30 from dataclasses import dataclass, field35 from dataclasses import dataclass, field
31 from datetime import datetime36 from datetime import datetime
@@ -33,6 +38,8 @@ from datetime import datetime
33 DEFAULT_LOGDIR = "/var/log/atop"38 DEFAULT_LOGDIR = "/var/log/atop"
34 CLAUDE_BIN = "claude"39 CLAUDE_BIN = "claude"
35 CLAUDE_MODEL = "opus"40 CLAUDE_MODEL = "opus"
41+CLICKHOUSE_URL_ENV = "CLICKHOUSE_URL"
42+IDENTIFIER_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
36 43
37 WANTED_LABELS = {"CPU", "CPL", "PRC"}44 WANTED_LABELS = {"CPU", "CPL", "PRC"}
38 DATE_RE = re.compile(r"^\d{4}/\d{2}/\d{2}$")45 DATE_RE = re.compile(r"^\d{4}/\d{2}/\d{2}$")
@@ -65,6 +72,7 @@ class Sample:
65 epoch: int72 epoch: int
66 when: str73 when: str
67 interval: int74 interval: int
75+ host: str = ""
68 ncpu: int = 076 ncpu: int = 0
69 hertz: int = 10077 hertz: int = 100
70 modes: dict[str, int] = field(default_factory=dict)78 modes: dict[str, int] = field(default_factory=dict)
@@ -147,7 +155,7 @@ def parse(raw: str) -> list[Sample]:
147 parts = line.split()155 parts = line.split()
148 if len(parts) < 7:156 if len(parts) < 7:
149 continue157 continue
150- label, _host, epoch, date, tm = parts[:5]158+ label, host, epoch, date, tm = parts[:5]
151 # With -b/-e, atop interleaves its interactive-format report into159 # With -b/-e, atop interleaves its interactive-format report into
152 # stdout. Those lines can start with a real label (lowercase "cpu"),160 # stdout. Those lines can start with a real label (lowercase "cpu"),
153 # so also require the date field to actually be a date.161 # so also require the date field to actually be a date.
@@ -160,7 +168,7 @@ def parse(raw: str) -> list[Sample]:
160 continue168 continue
161 169
162 if current is None:170 if current is None:
163- current = Sample(epoch=int(epoch), when=tm, interval=interval)171+ current = Sample(epoch=int(epoch), when=tm, interval=interval, host=host)
164 172
165 try:173 try:
166 if label == "CPU":174 if label == "CPU":
@@ -192,6 +200,98 @@ def parse(raw: str) -> list[Sample]:
192 return samples200 return samples
193 201
194 202
203+def clickhouse_identifier(value: str, option: str) -> str:
204+ """Validate a database or table name before embedding it in SQL."""
205+ if not IDENTIFIER_RE.match(value):
206+ raise AtopError(f"{option} must contain only letters, digits, and underscores")
207+ return value
208+
209+
210+def clickhouse_request(
211+ url: str, query: str, body: str, user: str | None, password: str | None,
212+ modal_key: str | None, modal_secret: str | None,
213+) -> None:
214+ """Send one query to ClickHouse's HTTP interface."""
215+ separator = "&" if "?" in url else "?"
216+ endpoint = f"{url.rstrip('/')}{separator}{urllib.parse.urlencode({'query': query})}"
217+ request = urllib.request.Request(
218+ endpoint,
219+ data=body.encode(),
220+ headers={"Content-Type": "application/json; charset=utf-8"},
221+ method="POST",
222+ )
223+ if user is not None:
224+ token = base64.b64encode(f"{user}:{password or ''}".encode()).decode()
225+ request.add_header("Authorization", f"Basic {token}")
226+ if modal_key is not None:
227+ request.add_header("Modal-Key", modal_key)
228+ request.add_header("Modal-Secret", modal_secret or "")
229+ try:
230+ with urllib.request.urlopen(request, timeout=30) as response:
231+ if response.status >= 300:
232+ raise AtopError(f"ClickHouse returned HTTP {response.status}")
233+ except urllib.error.HTTPError as e:
234+ detail = e.read().decode(errors="replace").strip()
235+ raise AtopError(f"ClickHouse returned HTTP {e.code}: {detail}") from e
236+ except urllib.error.URLError as e:
237+ raise AtopError(f"could not reach ClickHouse at {url}: {e.reason}") from e
238+
239+
240+def write_clickhouse(
241+ samples: list[Sample], url: str, database: str, sample_table: str,
242+ process_table: str, user: str | None, password: str | None,
243+ modal_key: str | None, modal_secret: str | None, initialize: bool,
244+) -> None:
245+ """Store parsed atop samples and per-process CPU attribution in ClickHouse."""
246+ database = clickhouse_identifier(database, "--clickhouse-database")
247+ sample_table = clickhouse_identifier(sample_table, "--clickhouse-sample-table")
248+ process_table = clickhouse_identifier(process_table, "--clickhouse-process-table")
249+ samples_target = f"`{database}`.`{sample_table}`"
250+ processes_target = f"`{database}`.`{process_table}`"
251+
252+ if initialize:
253+ clickhouse_request(url, f"CREATE DATABASE IF NOT EXISTS `{database}`", "", user, password,
254+ modal_key, modal_secret)
255+ clickhouse_request(url, f"""CREATE TABLE IF NOT EXISTS {samples_target} (
256+ host LowCardinality(String), epoch DateTime, interval_seconds UInt32,
257+ ncpu UInt16, user_pct Float32, sys_pct Float32, nice_pct Float32,
258+ idle_pct Float32, wait_pct Float32, irq_pct Float32, softirq_pct Float32,
259+ steal_pct Float32, busy_pct Float32, load1 Float32, load5 Float32,
260+ load15 Float32, ctxsw UInt64, intr UInt64
261+ ) ENGINE = ReplacingMergeTree ORDER BY (host, epoch)""", "", user, password,
262+ modal_key, modal_secret)
263+ clickhouse_request(url, f"""CREATE TABLE IF NOT EXISTS {processes_target} (
264+ host LowCardinality(String), epoch DateTime, pid UInt32,
265+ name LowCardinality(String), cpu_pct Float32, interval_seconds UInt32
266+ ) ENGINE = ReplacingMergeTree ORDER BY (host, epoch, pid, name)""", "", user, password,
267+ modal_key, modal_secret)
268+
269+ sample_rows = [
270+ {
271+ "host": s.host, "epoch": s.epoch, "interval_seconds": s.interval,
272+ "ncpu": s.ncpu, "user_pct": s.pct("user"), "sys_pct": s.pct("sys"),
273+ "nice_pct": s.pct("nice"), "idle_pct": s.pct("idle"),
274+ "wait_pct": s.pct("wait"), "irq_pct": s.pct("irq"),
275+ "softirq_pct": s.pct("softirq"), "steal_pct": s.pct("steal"),
276+ "busy_pct": s.busy_pct, "load1": s.load1, "load5": s.load5,
277+ "load15": s.load15, "ctxsw": s.ctxsw, "intr": s.intr,
278+ }
279+ for s in samples
280+ ]
281+ process_rows = [
282+ {"host": s.host, "epoch": s.epoch, "pid": pid, "name": name,
283+ "cpu_pct": pct, "interval_seconds": s.interval}
284+ for s in samples for name, pid, pct in s.procs
285+ ]
286+ clickhouse_request(url, f"INSERT INTO {samples_target} FORMAT JSONEachRow",
287+ "\n".join(json.dumps(row) for row in sample_rows), user, password,
288+ modal_key, modal_secret)
289+ if process_rows:
290+ clickhouse_request(url, f"INSERT INTO {processes_target} FORMAT JSONEachRow",
291+ "\n".join(json.dumps(row) for row in process_rows), user, password,
292+ modal_key, modal_secret)
293+
294+
195 def bucket(samples: list[Sample], max_rows: int) -> list[list[Sample]]:295 def bucket(samples: list[Sample], max_rows: int) -> list[list[Sample]]:
196 """Group samples into at most max_rows contiguous buckets."""296 """Group samples into at most max_rows contiguous buckets."""
197 if len(samples) <= max_rows:297 if len(samples) <= max_rows:
@@ -337,6 +437,24 @@ def main() -> int:
337 help="path to the claude CLI")437 help="path to the claude CLI")
338 p.add_argument("--digest-only", action="store_true",438 p.add_argument("--digest-only", action="store_true",
339 help="print the digest and exit; runs no model")439 help="print the digest and exit; runs no model")
440+ p.add_argument("--clickhouse-url", default=os.environ.get(CLICKHOUSE_URL_ENV),
441+ help=f"ClickHouse HTTP endpoint (default: ${CLICKHOUSE_URL_ENV})")
442+ p.add_argument("--clickhouse-database", default="default",
443+ help="ClickHouse database (default: default)")
444+ p.add_argument("--clickhouse-sample-table", default="atop_samples",
445+ help="ClickHouse interval table (default: atop_samples)")
446+ p.add_argument("--clickhouse-process-table", default="atop_processes",
447+ help="ClickHouse process-attribution table (default: atop_processes)")
448+ p.add_argument("--clickhouse-user", default=os.environ.get("CLICKHOUSE_USER"),
449+ help="ClickHouse basic-auth user (default: $CLICKHOUSE_USER)")
450+ p.add_argument("--clickhouse-password", default=os.environ.get("CLICKHOUSE_PASSWORD"),
451+ help="ClickHouse basic-auth password (default: $CLICKHOUSE_PASSWORD)")
452+ p.add_argument("--clickhouse-modal-key", default=os.environ.get("MODAL_KEY"),
453+ help="Modal Proxy Token ID for a protected Modal endpoint (default: $MODAL_KEY)")
454+ p.add_argument("--clickhouse-modal-secret", default=os.environ.get("MODAL_SECRET"),
455+ help="Modal Proxy Token secret for a protected Modal endpoint (default: $MODAL_SECRET)")
456+ p.add_argument("--clickhouse-init", action="store_true",
457+ help="create the ClickHouse database and tables if missing")
340 args = p.parse_args()458 args = p.parse_args()
341 459
342 try:460 try:
@@ -356,6 +474,26 @@ def main() -> int:
356 file=sys.stderr)474 file=sys.stderr)
357 return 1475 return 1
358 476
477+ if args.clickhouse_init and not args.clickhouse_url:
478+ p.error("--clickhouse-init requires --clickhouse-url or $CLICKHOUSE_URL")
479+ if bool(args.clickhouse_modal_key) != bool(args.clickhouse_modal_secret):
480+ p.error("--clickhouse-modal-key and --clickhouse-modal-secret must be supplied together")
481+ if args.clickhouse_url:
482+ try:
483+ write_clickhouse(
484+ samples, args.clickhouse_url, args.clickhouse_database,
485+ args.clickhouse_sample_table, args.clickhouse_process_table,
486+ args.clickhouse_user, args.clickhouse_password,
487+ args.clickhouse_modal_key, args.clickhouse_modal_secret,
488+ args.clickhouse_init,
489+ )
490+ except AtopError as e:
491+ print(f"error: ClickHouse export failed: {e}", file=sys.stderr)
492+ return 1
493+ process_count = sum(len(s.procs) for s in samples)
494+ print(f"# exported {len(samples)} samples and {process_count} process rows to ClickHouse",
495+ file=sys.stderr)
496+
359 text = digest(samples, args.top, args.max_rows)497 text = digest(samples, args.top, args.max_rows)
360 if args.digest_only:498 if args.digest_only:
361 print(text)499 print(text)
added grafana/provisioning/dashboards/dashboards.yaml +11 -0
new file mode 100644
@@ -0,0 +1,11 @@
1+apiVersion: 1
2+
3+providers:
4+ - name: Atop
5+ orgId: 1
6+ folder: Atop
7+ type: file
8+ disableDeletion: true
9+ editable: true
10+ options:
11+ path: /etc/grafana/provisioning/dashboards/json
new file mode 100644
@@ -0,0 +1,11 @@
1+apiVersion: 1
2+
3+providers:
4+ - name: Atop
5+ orgId: 1
6+ folder: Atop
7+ type: file
8+ disableDeletion: true
9+ editable: true
10+ options:
11+ path: /etc/grafana/provisioning/dashboards/json
added grafana/provisioning/dashboards/json/atop-overview.json +41 -0
new file mode 100644
@@ -0,0 +1,41 @@
1+{
2+ "annotations": {"list": []},
3+ "editable": true,
4+ "panels": [
5+ {
6+ "datasource": {"type": "grafana-clickhouse-datasource", "uid": "atop-clickhouse"},
7+ "fieldConfig": {"defaults": {"unit": "percent"}, "overrides": []},
8+ "gridPos": {"h": 9, "w": 12, "x": 0, "y": 0},
9+ "id": 1,
10+ "targets": [{"format": 1, "query": "SELECT epoch AS time, busy_pct AS busy, wait_pct AS iowait FROM atop_samples WHERE $__timeFilter(epoch) ORDER BY epoch", "refId": "A"}],
11+ "title": "CPU busy and I/O wait",
12+ "type": "timeseries"
13+ },
14+ {
15+ "datasource": {"type": "grafana-clickhouse-datasource", "uid": "atop-clickhouse"},
16+ "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []},
17+ "gridPos": {"h": 9, "w": 12, "x": 12, "y": 0},
18+ "id": 2,
19+ "targets": [{"format": 1, "query": "SELECT epoch AS time, load1 AS load_1m, load5 AS load_5m, load15 AS load_15m FROM atop_samples WHERE $__timeFilter(epoch) ORDER BY epoch", "refId": "A"}],
20+ "title": "Load average",
21+ "type": "timeseries"
22+ },
23+ {
24+ "datasource": {"type": "grafana-clickhouse-datasource", "uid": "atop-clickhouse"},
25+ "fieldConfig": {"defaults": {}, "overrides": []},
26+ "gridPos": {"h": 9, "w": 24, "x": 0, "y": 9},
27+ "id": 3,
28+ "options": {"showHeader": true},
29+ "targets": [{"format": 1, "query": "SELECT name AS process, round(avg(cpu_pct), 2) AS avg_cpu_pct, round(max(cpu_pct), 2) AS peak_cpu_pct FROM atop_processes WHERE $__timeFilter(epoch) GROUP BY name ORDER BY avg_cpu_pct DESC LIMIT 15", "refId": "A"}],
30+ "title": "Top processes",
31+ "type": "table"
32+ }
33+ ],
34+ "schemaVersion": 39,
35+ "tags": ["atop", "clickhouse"],
36+ "templating": {"list": []},
37+ "time": {"from": "now-24h", "to": "now"},
38+ "title": "Atop overview",
39+ "uid": "atop-overview",
40+ "version": 1
41+}
new file mode 100644
@@ -0,0 +1,41 @@
1+{
2+ "annotations": {"list": []},
3+ "editable": true,
4+ "panels": [
5+ {
6+ "datasource": {"type": "grafana-clickhouse-datasource", "uid": "atop-clickhouse"},
7+ "fieldConfig": {"defaults": {"unit": "percent"}, "overrides": []},
8+ "gridPos": {"h": 9, "w": 12, "x": 0, "y": 0},
9+ "id": 1,
10+ "targets": [{"format": 1, "query": "SELECT epoch AS time, busy_pct AS busy, wait_pct AS iowait FROM atop_samples WHERE $__timeFilter(epoch) ORDER BY epoch", "refId": "A"}],
11+ "title": "CPU busy and I/O wait",
12+ "type": "timeseries"
13+ },
14+ {
15+ "datasource": {"type": "grafana-clickhouse-datasource", "uid": "atop-clickhouse"},
16+ "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []},
17+ "gridPos": {"h": 9, "w": 12, "x": 12, "y": 0},
18+ "id": 2,
19+ "targets": [{"format": 1, "query": "SELECT epoch AS time, load1 AS load_1m, load5 AS load_5m, load15 AS load_15m FROM atop_samples WHERE $__timeFilter(epoch) ORDER BY epoch", "refId": "A"}],
20+ "title": "Load average",
21+ "type": "timeseries"
22+ },
23+ {
24+ "datasource": {"type": "grafana-clickhouse-datasource", "uid": "atop-clickhouse"},
25+ "fieldConfig": {"defaults": {}, "overrides": []},
26+ "gridPos": {"h": 9, "w": 24, "x": 0, "y": 9},
27+ "id": 3,
28+ "options": {"showHeader": true},
29+ "targets": [{"format": 1, "query": "SELECT name AS process, round(avg(cpu_pct), 2) AS avg_cpu_pct, round(max(cpu_pct), 2) AS peak_cpu_pct FROM atop_processes WHERE $__timeFilter(epoch) GROUP BY name ORDER BY avg_cpu_pct DESC LIMIT 15", "refId": "A"}],
30+ "title": "Top processes",
31+ "type": "table"
32+ }
33+ ],
34+ "schemaVersion": 39,
35+ "tags": ["atop", "clickhouse"],
36+ "templating": {"list": []},
37+ "time": {"from": "now-24h", "to": "now"},
38+ "title": "Atop overview",
39+ "uid": "atop-overview",
40+ "version": 1
41+}
added grafana/provisioning/datasources/clickhouse.yaml +23 -0
new file mode 100644
@@ -0,0 +1,23 @@
1+apiVersion: 1
2+
3+datasources:
4+ - name: Atop ClickHouse
5+ uid: atop-clickhouse
6+ type: grafana-clickhouse-datasource
7+ access: proxy
8+ isDefault: true
9+ jsonData:
10+ host: $__env{CLICKHOUSE_HOST}
11+ port: 443
12+ protocol: http
13+ secure: true
14+ username: default
15+ defaultDatabase: default
16+ httpHeaders:
17+ - name: Modal-Key
18+ secure: true
19+ - name: Modal-Secret
20+ secure: true
21+ secureJsonData:
22+ secureHttpHeaders.Modal-Key: $__env{MODAL_KEY}
23+ secureHttpHeaders.Modal-Secret: $__env{MODAL_SECRET}
new file mode 100644
@@ -0,0 +1,23 @@
1+apiVersion: 1
2+
3+datasources:
4+ - name: Atop ClickHouse
5+ uid: atop-clickhouse
6+ type: grafana-clickhouse-datasource
7+ access: proxy
8+ isDefault: true
9+ jsonData:
10+ host: $__env{CLICKHOUSE_HOST}
11+ port: 443
12+ protocol: http
13+ secure: true
14+ username: default
15+ defaultDatabase: default
16+ httpHeaders:
17+ - name: Modal-Key
18+ secure: true
19+ - name: Modal-Secret
20+ secure: true
21+ secureJsonData:
22+ secureHttpHeaders.Modal-Key: $__env{MODAL_KEY}
23+ secureHttpHeaders.Modal-Secret: $__env{MODAL_SECRET}
added modal_atop_analyze.py +72 -0
new file mode 100644
@@ -0,0 +1,72 @@
1+"""Modal ingestion app for atop history stored on the local machine.
2+
3+Run from the machine that has ``/var/log/atop``:
4+
5+ modal run modal_atop_analyze.py --date 20260921 --initialize
6+
7+The local entrypoint only reads the binary atop log. Parsing and the ClickHouse
8+write run in Modal; the ClickHouse endpoint and Proxy Token live in the
9+``atop-clickhouse-proxy`` Modal Secret, never in local shell environment.
10+"""
11+
12+from __future__ import annotations
13+
14+import os
15+
16+import modal
17+
18+from atop_analyze import AtopError, log_path, parse, run_atop, write_clickhouse
19+
20+app = modal.App("atop-analyze")
21+clickhouse_secret = modal.Secret.from_name(
22+ "atop-clickhouse-proxy",
23+ required_keys=["CLICKHOUSE_URL", "MODAL_KEY", "MODAL_SECRET"],
24+)
25+
26+
27+@app.function(secrets=[clickhouse_secret], timeout=15 * 60)
28+def ingest(raw_atop: str, initialize: bool = False) -> dict[str, int]:
29+ """Parse an atop export and persist its samples through private ClickHouse."""
30+ samples = parse(raw_atop)
31+ if not samples:
32+ raise RuntimeError("no usable atop samples in the supplied log range")
33+
34+ write_clickhouse(
35+ samples,
36+ os.environ["CLICKHOUSE_URL"],
37+ os.environ.get("CLICKHOUSE_DATABASE", "default"),
38+ os.environ.get("CLICKHOUSE_SAMPLE_TABLE", "atop_samples"),
39+ os.environ.get("CLICKHOUSE_PROCESS_TABLE", "atop_processes"),
40+ None,
41+ None,
42+ os.environ["MODAL_KEY"],
43+ os.environ["MODAL_SECRET"],
44+ initialize,
45+ )
46+ return {
47+ "samples": len(samples),
48+ "process_rows": sum(len(sample.procs) for sample in samples),
49+ }
50+
51+
52+@app.local_entrypoint()
53+def export(
54+ date: str | None = None,
55+ file: str | None = None,
56+ logdir: str = "/var/log/atop",
57+ begin: str | None = None,
58+ end: str | None = None,
59+ initialize: bool = False,
60+) -> None:
61+ """Read the local host's atop log, then delegate ingestion to Modal."""
62+ try:
63+ path = file or log_path(date, logdir)
64+ raw_atop = run_atop(path, begin, end)
65+ except AtopError as error:
66+ raise RuntimeError(f"could not read local atop history: {error}") from error
67+
68+ result = ingest.remote(raw_atop, initialize)
69+ print(
70+ f"exported {result['samples']} samples and {result['process_rows']} "
71+ f"process rows from {path}"
72+ )
new file mode 100644
@@ -0,0 +1,72 @@
1+"""Modal ingestion app for atop history stored on the local machine.
2+
3+Run from the machine that has ``/var/log/atop``:
4+
5+ modal run modal_atop_analyze.py --date 20260921 --initialize
6+
7+The local entrypoint only reads the binary atop log. Parsing and the ClickHouse
8+write run in Modal; the ClickHouse endpoint and Proxy Token live in the
9+``atop-clickhouse-proxy`` Modal Secret, never in local shell environment.
10+"""
11+
12+from __future__ import annotations
13+
14+import os
15+
16+import modal
17+
18+from atop_analyze import AtopError, log_path, parse, run_atop, write_clickhouse
19+
20+app = modal.App("atop-analyze")
21+clickhouse_secret = modal.Secret.from_name(
22+ "atop-clickhouse-proxy",
23+ required_keys=["CLICKHOUSE_URL", "MODAL_KEY", "MODAL_SECRET"],
24+)
25+
26+
27+@app.function(secrets=[clickhouse_secret], timeout=15 * 60)
28+def ingest(raw_atop: str, initialize: bool = False) -> dict[str, int]:
29+ """Parse an atop export and persist its samples through private ClickHouse."""
30+ samples = parse(raw_atop)
31+ if not samples:
32+ raise RuntimeError("no usable atop samples in the supplied log range")
33+
34+ write_clickhouse(
35+ samples,
36+ os.environ["CLICKHOUSE_URL"],
37+ os.environ.get("CLICKHOUSE_DATABASE", "default"),
38+ os.environ.get("CLICKHOUSE_SAMPLE_TABLE", "atop_samples"),
39+ os.environ.get("CLICKHOUSE_PROCESS_TABLE", "atop_processes"),
40+ None,
41+ None,
42+ os.environ["MODAL_KEY"],
43+ os.environ["MODAL_SECRET"],
44+ initialize,
45+ )
46+ return {
47+ "samples": len(samples),
48+ "process_rows": sum(len(sample.procs) for sample in samples),
49+ }
50+
51+
52+@app.local_entrypoint()
53+def export(
54+ date: str | None = None,
55+ file: str | None = None,
56+ logdir: str = "/var/log/atop",
57+ begin: str | None = None,
58+ end: str | None = None,
59+ initialize: bool = False,
60+) -> None:
61+ """Read the local host's atop log, then delegate ingestion to Modal."""
62+ try:
63+ path = file or log_path(date, logdir)
64+ raw_atop = run_atop(path, begin, end)
65+ except AtopError as error:
66+ raise RuntimeError(f"could not read local atop history: {error}") from error
67+
68+ result = ingest.remote(raw_atop, initialize)
69+ print(
70+ f"exported {result['samples']} samples and {result['process_rows']} "
71+ f"process rows from {path}"
72+ )
added modal_clickhouse.py +54 -0
new file mode 100644
@@ -0,0 +1,54 @@
1+"""A scale-to-zero ClickHouse HTTP endpoint for low-volume atop data.
2+
3+Deploy with:
4+
5+ modal deploy modal_clickhouse.py
6+
7+The endpoint requires a Modal Proxy Token. It starts ClickHouse when a request
8+arrives, retains data in the named Modal Volume, and can scale to zero after it
9+has been idle. Keep max_containers at one: a Modal Volume is not a suitable
10+shared filesystem for multiple ClickHouse writers.
11+"""
12+
13+import socket
14+import subprocess
15+import time
16+
17+import modal
18+
19+APP_NAME = "atop-clickhouse"
20+DATA_VOLUME_NAME = "atop-clickhouse-data"
21+IDLE_SECONDS = 60
22+
23+app = modal.App(APP_NAME)
24+data_volume = modal.Volume.from_name(DATA_VOLUME_NAME, create_if_missing=True)
25+image = modal.Image.from_registry(
26+ "clickhouse/clickhouse-server:25.8", add_python="3.12"
27+)
28+
29+
30+@app.function(
31+ image=image,
32+ volumes={"/var/lib/clickhouse": data_volume},
33+ min_containers=0,
34+ max_containers=1,
35+ scaledown_window=IDLE_SECONDS,
36+ timeout=10 * 60,
37+)
38+@modal.web_server(8123, startup_timeout=120, requires_proxy_auth=True)
39+def clickhouse() -> None:
40+ """Start ClickHouse's native HTTP API; Modal proxies requests to port 8123."""
41+ process = subprocess.Popen(
42+ ["clickhouse-server", "--config-file=/etc/clickhouse-server/config.xml"]
43+ )
44+ deadline = time.monotonic() + 90
45+ while time.monotonic() < deadline:
46+ if process.poll() is not None:
47+ raise RuntimeError(f"ClickHouse exited during startup ({process.returncode})")
48+ with socket.socket() as sock:
49+ sock.settimeout(0.2)
50+ if sock.connect_ex(("127.0.0.1", 8123)) == 0:
51+ return
52+ time.sleep(0.2)
53+ process.terminate()
54+ raise RuntimeError("ClickHouse did not open port 8123 within 90 seconds")
new file mode 100644
@@ -0,0 +1,54 @@
1+"""A scale-to-zero ClickHouse HTTP endpoint for low-volume atop data.
2+
3+Deploy with:
4+
5+ modal deploy modal_clickhouse.py
6+
7+The endpoint requires a Modal Proxy Token. It starts ClickHouse when a request
8+arrives, retains data in the named Modal Volume, and can scale to zero after it
9+has been idle. Keep max_containers at one: a Modal Volume is not a suitable
10+shared filesystem for multiple ClickHouse writers.
11+"""
12+
13+import socket
14+import subprocess
15+import time
16+
17+import modal
18+
19+APP_NAME = "atop-clickhouse"
20+DATA_VOLUME_NAME = "atop-clickhouse-data"
21+IDLE_SECONDS = 60
22+
23+app = modal.App(APP_NAME)
24+data_volume = modal.Volume.from_name(DATA_VOLUME_NAME, create_if_missing=True)
25+image = modal.Image.from_registry(
26+ "clickhouse/clickhouse-server:25.8", add_python="3.12"
27+)
28+
29+
30+@app.function(
31+ image=image,
32+ volumes={"/var/lib/clickhouse": data_volume},
33+ min_containers=0,
34+ max_containers=1,
35+ scaledown_window=IDLE_SECONDS,
36+ timeout=10 * 60,
37+)
38+@modal.web_server(8123, startup_timeout=120, requires_proxy_auth=True)
39+def clickhouse() -> None:
40+ """Start ClickHouse's native HTTP API; Modal proxies requests to port 8123."""
41+ process = subprocess.Popen(
42+ ["clickhouse-server", "--config-file=/etc/clickhouse-server/config.xml"]
43+ )
44+ deadline = time.monotonic() + 90
45+ while time.monotonic() < deadline:
46+ if process.poll() is not None:
47+ raise RuntimeError(f"ClickHouse exited during startup ({process.returncode})")
48+ with socket.socket() as sock:
49+ sock.settimeout(0.2)
50+ if sock.connect_ex(("127.0.0.1", 8123)) == 0:
51+ return
52+ time.sleep(0.2)
53+ process.terminate()
54+ raise RuntimeError("ClickHouse did not open port 8123 within 90 seconds")
added modal_grafana.py +63 -0
new file mode 100644
@@ -0,0 +1,63 @@
1+"""Browser-facing Grafana for the private scale-to-zero ClickHouse endpoint.
2+
3+Create a Modal Secret named ``atop-grafana`` before deploying. It needs:
4+``CLICKHOUSE_HOST`` (hostname only), ``MODAL_KEY``, ``MODAL_SECRET``, and
5+``GF_SECURITY_ADMIN_PASSWORD``. Deploy with ``modal deploy modal_grafana.py``.
6+"""
7+
8+import subprocess
9+import time
10+
11+import modal
12+
13+APP_NAME = "atop-grafana"
14+IDLE_SECONDS = 300
15+
16+app = modal.App(APP_NAME)
17+grafana_secret = modal.Secret.from_name(
18+ "atop-grafana",
19+ required_keys=[
20+ "CLICKHOUSE_HOST", "MODAL_KEY", "MODAL_SECRET", "GF_SECURITY_ADMIN_PASSWORD",
21+ "GF_AUTH_GENERIC_OAUTH_CLIENT_ID", "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET",
22+ "GF_AUTH_GENERIC_OAUTH_AUTH_URL", "GF_AUTH_GENERIC_OAUTH_TOKEN_URL",
23+ "GF_AUTH_GENERIC_OAUTH_API_URL", "GF_AUTH_GENERIC_OAUTH_JWK_SET_URL",
24+ ],
25+)
26+image = (
27+ modal.Image.from_registry("grafana/grafana:11.6.0", add_python="3.12")
28+ .run_commands("grafana cli plugins install grafana-clickhouse-datasource")
29+ .add_local_dir("grafana/provisioning", "/etc/grafana/provisioning", copy=True)
30+ .env(
31+ {
32+ "GF_AUTH_DISABLE_LOGIN_FORM": "true",
33+ "GF_AUTH_GENERIC_OAUTH_ENABLED": "true",
34+ "GF_AUTH_GENERIC_OAUTH_NAME": "Pocket ID",
35+ "GF_AUTH_GENERIC_OAUTH_SCOPES": "openid profile email",
36+ "GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP": "true",
37+ "GF_AUTH_GENERIC_OAUTH_AUTO_LOGIN": "true",
38+ "GF_AUTH_GENERIC_OAUTH_USE_PKCE": "true",
39+ "GF_AUTH_GENERIC_OAUTH_VALIDATE_ID_TOKEN": "true",
40+ }
41+ )
42+)
43+
44+
45+@app.function(
46+ image=image,
47+ secrets=[grafana_secret],
48+ min_containers=0,
49+ max_containers=1,
50+ scaledown_window=IDLE_SECONDS,
51+ timeout=15 * 60,
52+)
53+@modal.web_server(3000, startup_timeout=120)
54+def grafana() -> None:
55+ """Run a public Grafana login page; its data-source credentials stay server-side."""
56+ process = subprocess.Popen(
57+ ["grafana", "server", "--homepath=/usr/share/grafana", "--config=/etc/grafana/grafana.ini"]
58+ )
59+ # modal.web_server waits for port 3000. Catch an immediate startup failure
60+ # early so its cause appears in Modal logs rather than as a proxy timeout.
61+ time.sleep(1)
62+ if process.poll() is not None:
63+ raise RuntimeError(f"Grafana exited during startup ({process.returncode})")
new file mode 100644
@@ -0,0 +1,63 @@
1+"""Browser-facing Grafana for the private scale-to-zero ClickHouse endpoint.
2+
3+Create a Modal Secret named ``atop-grafana`` before deploying. It needs:
4+``CLICKHOUSE_HOST`` (hostname only), ``MODAL_KEY``, ``MODAL_SECRET``, and
5+``GF_SECURITY_ADMIN_PASSWORD``. Deploy with ``modal deploy modal_grafana.py``.
6+"""
7+
8+import subprocess
9+import time
10+
11+import modal
12+
13+APP_NAME = "atop-grafana"
14+IDLE_SECONDS = 300
15+
16+app = modal.App(APP_NAME)
17+grafana_secret = modal.Secret.from_name(
18+ "atop-grafana",
19+ required_keys=[
20+ "CLICKHOUSE_HOST", "MODAL_KEY", "MODAL_SECRET", "GF_SECURITY_ADMIN_PASSWORD",
21+ "GF_AUTH_GENERIC_OAUTH_CLIENT_ID", "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET",
22+ "GF_AUTH_GENERIC_OAUTH_AUTH_URL", "GF_AUTH_GENERIC_OAUTH_TOKEN_URL",
23+ "GF_AUTH_GENERIC_OAUTH_API_URL", "GF_AUTH_GENERIC_OAUTH_JWK_SET_URL",
24+ ],
25+)
26+image = (
27+ modal.Image.from_registry("grafana/grafana:11.6.0", add_python="3.12")
28+ .run_commands("grafana cli plugins install grafana-clickhouse-datasource")
29+ .add_local_dir("grafana/provisioning", "/etc/grafana/provisioning", copy=True)
30+ .env(
31+ {
32+ "GF_AUTH_DISABLE_LOGIN_FORM": "true",
33+ "GF_AUTH_GENERIC_OAUTH_ENABLED": "true",
34+ "GF_AUTH_GENERIC_OAUTH_NAME": "Pocket ID",
35+ "GF_AUTH_GENERIC_OAUTH_SCOPES": "openid profile email",
36+ "GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP": "true",
37+ "GF_AUTH_GENERIC_OAUTH_AUTO_LOGIN": "true",
38+ "GF_AUTH_GENERIC_OAUTH_USE_PKCE": "true",
39+ "GF_AUTH_GENERIC_OAUTH_VALIDATE_ID_TOKEN": "true",
40+ }
41+ )
42+)
43+
44+
45+@app.function(
46+ image=image,
47+ secrets=[grafana_secret],
48+ min_containers=0,
49+ max_containers=1,
50+ scaledown_window=IDLE_SECONDS,
51+ timeout=15 * 60,
52+)
53+@modal.web_server(3000, startup_timeout=120)
54+def grafana() -> None:
55+ """Run a public Grafana login page; its data-source credentials stay server-side."""
56+ process = subprocess.Popen(
57+ ["grafana", "server", "--homepath=/usr/share/grafana", "--config=/etc/grafana/grafana.ini"]
58+ )
59+ # modal.web_server waits for port 3000. Catch an immediate startup failure
60+ # early so its cause appears in Modal logs rather than as a proxy timeout.
61+ time.sleep(1)
62+ if process.poll() is not None:
63+ raise RuntimeError(f"Grafana exited during startup ({process.returncode})")