rickub
nandi/atop-analyzepublic⑂ Fork 0
⑂ a275b50
Branches
main View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/nandi/atop-analyze.git
git clone ssh://git@rickub.com/nandi/atop-analyze.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Configure Pocket ID OAuth and daily Modal exports

nandithebull committed 2026-09-21T19:36:03-07:00 Browse files
a275b50 parent: bf0b684
modified README.md +41 -11
@@ -112,24 +112,52 @@ Use `--file /path/to/atop_log` or `--logdir /path/to/logdir` when the source
112112 log is elsewhere. The command still needs a normal Modal CLI login to invoke
113113 the Function, but no longer has direct database credentials.
114114
115+#### Automatic daily export
116+
117+The included systemd units export the completed prior-day log at 00:15. This
118+avoids repeatedly importing a still-growing log and works even if the machine
119+was off at the scheduled time (`Persistent=true`). They are configured for this
120+host's `nandi` user and Modal CLI location:
121+
122+```sh
123+run0 install -m 0644 systemd/atop-modal-export.service /etc/systemd/system/
124+run0 install -m 0644 systemd/atop-modal-export.timer /etc/systemd/system/
125+run0 systemctl daemon-reload
126+run0 systemctl enable --now atop-modal-export.timer
127+systemctl list-timers atop-modal-export.timer
128+```
129+
130+To test it without waiting until midnight, run:
131+
132+```sh
133+run0 systemctl start atop-modal-export.service
134+journalctl -u atop-modal-export.service --no-pager -n 50
135+```
136+
115137 ### Grafana dashboard
116138
117139 `modal_grafana.py` adds a browser-facing Grafana service with a provisioned
118140 ClickHouse data source and an **Atop overview** dashboard (CPU/iowait, load,
119-and top processes). Grafana is public at the HTTP layer but requires its own
120-admin login; its Modal Proxy Token stays in a Modal Secret and is never sent to
121-the browser.
141+and top processes). Grafana is public at the HTTP layer but authenticates every
142+visitor through Pocket ID; its Modal Proxy Token stays in a Modal Secret and is
143+never sent to the browser.
122144
123145 ```sh
124-# In Pocket ID, create an OIDC client whose redirect URI is:
125-# https://codegod100--atop-grafana-grafana.modal.run/login/generic_oauth
126-# Copy the client credentials and OIDC endpoints Pocket ID displays.
146+# In Pocket ID, create a confidential OIDC client and restrict it to the users
147+# or groups that may view Grafana. Its redirect URI must exactly be:
148+# ${GRAFANA_URL}/login/generic_oauth
149+#
150+# Find the four endpoint values in Pocket ID’s OIDC Discovery URL:
151+# https://<pocket-id-host>/.well-known/openid-configuration
152+# Use its authorization_endpoint, token_endpoint, userinfo_endpoint, and
153+# jwks_uri values below. GRAFANA_URL is the public Modal URL, with no trailing
154+# slash (for example, https://<workspace>--atop-grafana-grafana.modal.run).
127155 # The ClickHouse URL is hostname only: no https:// prefix and no path.
128156 modal secret create atop-grafana \
129157 CLICKHOUSE_HOST='codegod100--atop-clickhouse-clickhouse.modal.run' \
130158 MODAL_KEY='wk-…' \
131159 MODAL_SECRET='ws-…' \
132- GF_SECURITY_ADMIN_PASSWORD='emergency-local-admin-password' \
160+ GF_SERVER_ROOT_URL='https://<workspace>--atop-grafana-grafana.modal.run' \
133161 GF_AUTH_GENERIC_OAUTH_CLIENT_ID='…' \
134162 GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET='…' \
135163 GF_AUTH_GENERIC_OAUTH_AUTH_URL='<authorization-url-from-pocket-id>' \
@@ -140,10 +168,12 @@ modal secret create atop-grafana \
140168 modal deploy modal_grafana.py
141169 ```
142170
143-Grafana redirects visitors to the configured Generic OIDC provider; it does not
144-expose a local password form. Grafana scales to zero after five idle minutes;
145-the first visit after that can take a short time while Grafana and ClickHouse
146-start. The source and starter dashboard are under `grafana/provisioning/`.
171+Grafana redirects visitors to Pocket ID and does not expose a local password
172+form. It uses PKCE, validates the signed ID token with Pocket ID’s JWKS, and
173+requests a refresh token so an active Grafana session does not end at access
174+token expiry. Grafana scales to zero after five idle minutes; the first visit
175+after that can take a short time while Grafana and ClickHouse start. The source
176+and starter dashboard are under `grafana/provisioning/`.
147177
148178 ## What it sends
149179
@@ -112,24 +112,52 @@ Use `--file /path/to/atop_log` or `--logdir /path/to/logdir` when the source
112 log is elsewhere. The command still needs a normal Modal CLI login to invoke112 log is elsewhere. The command still needs a normal Modal CLI login to invoke
113 the Function, but no longer has direct database credentials.113 the Function, but no longer has direct database credentials.
114 114
115+#### Automatic daily export
116+
117+The included systemd units export the completed prior-day log at 00:15. This
118+avoids repeatedly importing a still-growing log and works even if the machine
119+was off at the scheduled time (`Persistent=true`). They are configured for this
120+host's `nandi` user and Modal CLI location:
121+
122+```sh
123+run0 install -m 0644 systemd/atop-modal-export.service /etc/systemd/system/
124+run0 install -m 0644 systemd/atop-modal-export.timer /etc/systemd/system/
125+run0 systemctl daemon-reload
126+run0 systemctl enable --now atop-modal-export.timer
127+systemctl list-timers atop-modal-export.timer
128+```
129+
130+To test it without waiting until midnight, run:
131+
132+```sh
133+run0 systemctl start atop-modal-export.service
134+journalctl -u atop-modal-export.service --no-pager -n 50
135+```
136+
115 ### Grafana dashboard137 ### Grafana dashboard
116 138
117 `modal_grafana.py` adds a browser-facing Grafana service with a provisioned139 `modal_grafana.py` adds a browser-facing Grafana service with a provisioned
118 ClickHouse data source and an **Atop overview** dashboard (CPU/iowait, load,140 ClickHouse data source and an **Atop overview** dashboard (CPU/iowait, load,
119-and top processes). Grafana is public at the HTTP layer but requires its own141+and top processes). Grafana is public at the HTTP layer but authenticates every
120-admin login; its Modal Proxy Token stays in a Modal Secret and is never sent to142+visitor through Pocket ID; its Modal Proxy Token stays in a Modal Secret and is
121-the browser.143+never sent to the browser.
122 144
123 ```sh145 ```sh
124-# In Pocket ID, create an OIDC client whose redirect URI is:146+# In Pocket ID, create a confidential OIDC client and restrict it to the users
125-# https://codegod100--atop-grafana-grafana.modal.run/login/generic_oauth147+# or groups that may view Grafana. Its redirect URI must exactly be:
126-# Copy the client credentials and OIDC endpoints Pocket ID displays.148+# ${GRAFANA_URL}/login/generic_oauth
149+#
150+# Find the four endpoint values in Pocket ID’s OIDC Discovery URL:
151+# https://<pocket-id-host>/.well-known/openid-configuration
152+# Use its authorization_endpoint, token_endpoint, userinfo_endpoint, and
153+# jwks_uri values below. GRAFANA_URL is the public Modal URL, with no trailing
154+# slash (for example, https://<workspace>--atop-grafana-grafana.modal.run).
127 # The ClickHouse URL is hostname only: no https:// prefix and no path.155 # The ClickHouse URL is hostname only: no https:// prefix and no path.
128 modal secret create atop-grafana \156 modal secret create atop-grafana \
129 CLICKHOUSE_HOST='codegod100--atop-clickhouse-clickhouse.modal.run' \157 CLICKHOUSE_HOST='codegod100--atop-clickhouse-clickhouse.modal.run' \
130 MODAL_KEY='wk-…' \158 MODAL_KEY='wk-…' \
131 MODAL_SECRET='ws-…' \159 MODAL_SECRET='ws-…' \
132- GF_SECURITY_ADMIN_PASSWORD='emergency-local-admin-password' \160+ GF_SERVER_ROOT_URL='https://<workspace>--atop-grafana-grafana.modal.run' \
133 GF_AUTH_GENERIC_OAUTH_CLIENT_ID='…' \161 GF_AUTH_GENERIC_OAUTH_CLIENT_ID='…' \
134 GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET='…' \162 GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET='…' \
135 GF_AUTH_GENERIC_OAUTH_AUTH_URL='<authorization-url-from-pocket-id>' \163 GF_AUTH_GENERIC_OAUTH_AUTH_URL='<authorization-url-from-pocket-id>' \
@@ -140,10 +168,12 @@ modal secret create atop-grafana \
140 modal deploy modal_grafana.py168 modal deploy modal_grafana.py
141 ```169 ```
142 170
143-Grafana redirects visitors to the configured Generic OIDC provider; it does not171+Grafana redirects visitors to Pocket ID and does not expose a local password
144-expose a local password form. Grafana scales to zero after five idle minutes;172+form. It uses PKCE, validates the signed ID token with Pocket ID’s JWKS, and
145-the first visit after that can take a short time while Grafana and ClickHouse173+requests a refresh token so an active Grafana session does not end at access
146-start. The source and starter dashboard are under `grafana/provisioning/`.174+token expiry. Grafana scales to zero after five idle minutes; the first visit
175+after that can take a short time while Grafana and ClickHouse start. The source
176+and starter dashboard are under `grafana/provisioning/`.
147 177
148 ## What it sends178 ## What it sends
149 179
modified modal_grafana.py +5 -3
@@ -2,7 +2,8 @@
22
33 Create a Modal Secret named ``atop-grafana`` before deploying. It needs:
44 ``CLICKHOUSE_HOST`` (hostname only), ``MODAL_KEY``, ``MODAL_SECRET``, and
5-``GF_SECURITY_ADMIN_PASSWORD``. Deploy with ``modal deploy modal_grafana.py``.
5+the Pocket ID Generic OAuth settings, including ``GF_SERVER_ROOT_URL``. Deploy
6+with ``modal deploy modal_grafana.py``.
67 """
78
89 import subprocess
@@ -17,7 +18,7 @@ app = modal.App(APP_NAME)
1718 grafana_secret = modal.Secret.from_name(
1819 "atop-grafana",
1920 required_keys=[
20- "CLICKHOUSE_HOST", "MODAL_KEY", "MODAL_SECRET", "GF_SECURITY_ADMIN_PASSWORD",
21+ "CLICKHOUSE_HOST", "MODAL_KEY", "MODAL_SECRET", "GF_SERVER_ROOT_URL",
2122 "GF_AUTH_GENERIC_OAUTH_CLIENT_ID", "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET",
2223 "GF_AUTH_GENERIC_OAUTH_AUTH_URL", "GF_AUTH_GENERIC_OAUTH_TOKEN_URL",
2324 "GF_AUTH_GENERIC_OAUTH_API_URL", "GF_AUTH_GENERIC_OAUTH_JWK_SET_URL",
@@ -32,10 +33,11 @@ image = (
3233 "GF_AUTH_DISABLE_LOGIN_FORM": "true",
3334 "GF_AUTH_GENERIC_OAUTH_ENABLED": "true",
3435 "GF_AUTH_GENERIC_OAUTH_NAME": "Pocket ID",
35- "GF_AUTH_GENERIC_OAUTH_SCOPES": "openid profile email",
36+ "GF_AUTH_GENERIC_OAUTH_SCOPES": "openid profile email offline_access",
3637 "GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP": "true",
3738 "GF_AUTH_GENERIC_OAUTH_AUTO_LOGIN": "true",
3839 "GF_AUTH_GENERIC_OAUTH_USE_PKCE": "true",
40+ "GF_AUTH_GENERIC_OAUTH_USE_REFRESH_TOKEN": "true",
3941 "GF_AUTH_GENERIC_OAUTH_VALIDATE_ID_TOKEN": "true",
4042 }
4143 )
@@ -2,7 +2,8 @@
2 2
3 Create a Modal Secret named ``atop-grafana`` before deploying. It needs:3 Create a Modal Secret named ``atop-grafana`` before deploying. It needs:
4 ``CLICKHOUSE_HOST`` (hostname only), ``MODAL_KEY``, ``MODAL_SECRET``, and4 ``CLICKHOUSE_HOST`` (hostname only), ``MODAL_KEY``, ``MODAL_SECRET``, and
5-``GF_SECURITY_ADMIN_PASSWORD``. Deploy with ``modal deploy modal_grafana.py``.5+the Pocket ID Generic OAuth settings, including ``GF_SERVER_ROOT_URL``. Deploy
6+with ``modal deploy modal_grafana.py``.
6 """7 """
7 8
8 import subprocess9 import subprocess
@@ -17,7 +18,7 @@ app = modal.App(APP_NAME)
17 grafana_secret = modal.Secret.from_name(18 grafana_secret = modal.Secret.from_name(
18 "atop-grafana",19 "atop-grafana",
19 required_keys=[20 required_keys=[
20- "CLICKHOUSE_HOST", "MODAL_KEY", "MODAL_SECRET", "GF_SECURITY_ADMIN_PASSWORD",21+ "CLICKHOUSE_HOST", "MODAL_KEY", "MODAL_SECRET", "GF_SERVER_ROOT_URL",
21 "GF_AUTH_GENERIC_OAUTH_CLIENT_ID", "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET",22 "GF_AUTH_GENERIC_OAUTH_CLIENT_ID", "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET",
22 "GF_AUTH_GENERIC_OAUTH_AUTH_URL", "GF_AUTH_GENERIC_OAUTH_TOKEN_URL",23 "GF_AUTH_GENERIC_OAUTH_AUTH_URL", "GF_AUTH_GENERIC_OAUTH_TOKEN_URL",
23 "GF_AUTH_GENERIC_OAUTH_API_URL", "GF_AUTH_GENERIC_OAUTH_JWK_SET_URL",24 "GF_AUTH_GENERIC_OAUTH_API_URL", "GF_AUTH_GENERIC_OAUTH_JWK_SET_URL",
@@ -32,10 +33,11 @@ image = (
32 "GF_AUTH_DISABLE_LOGIN_FORM": "true",33 "GF_AUTH_DISABLE_LOGIN_FORM": "true",
33 "GF_AUTH_GENERIC_OAUTH_ENABLED": "true",34 "GF_AUTH_GENERIC_OAUTH_ENABLED": "true",
34 "GF_AUTH_GENERIC_OAUTH_NAME": "Pocket ID",35 "GF_AUTH_GENERIC_OAUTH_NAME": "Pocket ID",
35- "GF_AUTH_GENERIC_OAUTH_SCOPES": "openid profile email",36+ "GF_AUTH_GENERIC_OAUTH_SCOPES": "openid profile email offline_access",
36 "GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP": "true",37 "GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP": "true",
37 "GF_AUTH_GENERIC_OAUTH_AUTO_LOGIN": "true",38 "GF_AUTH_GENERIC_OAUTH_AUTO_LOGIN": "true",
38 "GF_AUTH_GENERIC_OAUTH_USE_PKCE": "true",39 "GF_AUTH_GENERIC_OAUTH_USE_PKCE": "true",
40+ "GF_AUTH_GENERIC_OAUTH_USE_REFRESH_TOKEN": "true",
39 "GF_AUTH_GENERIC_OAUTH_VALIDATE_ID_TOKEN": "true",41 "GF_AUTH_GENERIC_OAUTH_VALIDATE_ID_TOKEN": "true",
40 }42 }
41 )43 )
added systemd/atop-modal-export.service +15 -0
new file mode 100644
@@ -0,0 +1,15 @@
1+[Unit]
2+Description=Export completed atop history to Modal ClickHouse
3+Wants=network-online.target
4+After=network-online.target atop.service
5+
6+[Service]
7+Type=oneshot
8+User=nandi
9+Group=nandi
10+WorkingDirectory=/home/nandi/code/atop-analyze
11+Environment=HOME=/home/nandi
12+Environment=PATH=/home/nandi/.nix-profile/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
13+# The prior day's file is complete after the midnight atop rotation. %% escapes
14+# the date-format percent signs from systemd's unit-specifier expansion.
15+ExecStart=/bin/bash -lc 'exec /home/nandi/.nix-profile/bin/modal run modal_atop_analyze.py --date "$$(/usr/bin/date -d yesterday +%%Y%%m%%d)" --initialize'
new file mode 100644
@@ -0,0 +1,15 @@
1+[Unit]
2+Description=Export completed atop history to Modal ClickHouse
3+Wants=network-online.target
4+After=network-online.target atop.service
5+
6+[Service]
7+Type=oneshot
8+User=nandi
9+Group=nandi
10+WorkingDirectory=/home/nandi/code/atop-analyze
11+Environment=HOME=/home/nandi
12+Environment=PATH=/home/nandi/.nix-profile/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
13+# The prior day's file is complete after the midnight atop rotation. %% escapes
14+# the date-format percent signs from systemd's unit-specifier expansion.
15+ExecStart=/bin/bash -lc 'exec /home/nandi/.nix-profile/bin/modal run modal_atop_analyze.py --date "$$(/usr/bin/date -d yesterday +%%Y%%m%%d)" --initialize'
added systemd/atop-modal-export.timer +11 -0
new file mode 100644
@@ -0,0 +1,11 @@
1+[Unit]
2+Description=Daily export of completed atop history to Modal ClickHouse
3+
4+[Timer]
5+OnCalendar=*-*-* 00:15:00
6+Persistent=true
7+RandomizedDelaySec=2m
8+Unit=atop-modal-export.service
9+
10+[Install]
11+WantedBy=timers.target
new file mode 100644
@@ -0,0 +1,11 @@
1+[Unit]
2+Description=Daily export of completed atop history to Modal ClickHouse
3+
4+[Timer]
5+OnCalendar=*-*-* 00:15:00
6+Persistent=true
7+RandomizedDelaySec=2m
8+Unit=atop-modal-export.service
9+
10+[Install]
11+WantedBy=timers.target