rickub
nandi/atop-analyzepublic⑂ Fork 0
⑂ 4edeea9
Branches
main View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/nandi/atop-analyze.git
git clone ssh://git@rickub.com/nandi/atop-analyze.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Harden Grafana and ClickHouse deployment configuration

nandithebull committed 2026-09-21T21:52:09-07:00 Browse files
4edeea9 parent: e5a43e4
modified README.md +28 -3
@@ -71,7 +71,7 @@ ClickHouse merges parts.
7171
7272 For low-volume ingestion, [modal_clickhouse.py](modal_clickhouse.py) runs the
7373 ClickHouse HTTP API as a protected Modal Web Function. It starts only when an
74-HTTP request arrives and is eligible to scale to zero after 60 seconds idle.
74+HTTP request arrives and is eligible to scale to zero after five minutes idle.
7575 Its data directory is retained in the named `atop-clickhouse-data` Modal
7676 Volume. It intentionally limits Modal to one container because the Volume must
7777 not have concurrent ClickHouse writers.
@@ -92,7 +92,9 @@ Modal Proxy Token credentials protect the endpoint at the edge; the collector
9292 passes them as `Modal-Key` and `Modal-Secret`. They may also be supplied with
9393 `--clickhouse-modal-key` and `--clickhouse-modal-secret` instead of environment
9494 variables. Expect a cold-start delay on the first request after the endpoint
95-has scaled down.
95+has scaled down. Grafana's ClickHouse data source allows for that startup delay
96+with a 120-second connection timeout and a 180-second query timeout; a failed
97+startup still surfaces as an error rather than an indefinite loading state.
9698
9799 ### Modal ingestion app
98100
@@ -159,6 +161,15 @@ and top processes). Grafana is public at the HTTP layer but authenticates every
159161 visitor through Pocket ID; its Modal Proxy Token stays in a Modal Secret and is
160162 never sent to the browser.
161163
164+Before deploying, create a ClickHouse user named `grafana_reader` with a
165+nonempty password. Grant it `SELECT` on `default.atop_samples` and
166+`default.atop_processes`, plus `SHOW TABLES` and `SHOW DATABASES` for the
167+`default` database. Store the matching password in a separate Modal Secret
168+named `atop-grafana-clickhouse-auth` under the key
169+`CLICKHOUSE_GRAFANA_PASSWORD`. Grafana requires that Secret at deployment;
170+its `default` ClickHouse user remains passwordless for ingestion but is not
171+used by Grafana. Do not put the password in the repository.
172+
162173 ```sh
163174 # In Pocket ID, create a confidential OIDC client and restrict it to the users
164175 # or groups that may view Grafana. Its redirect URI must exactly be:
@@ -185,10 +196,24 @@ modal secret create atop-grafana \
185196 modal deploy modal_grafana.py
186197 ```
187198
199+The deployment creates a persistent Modal Volume named `atop-grafana-data`
200+and mounts it at `/var/lib/grafana`, where Grafana stores its SQLite database
201+and saved settings. The ClickHouse plugin stays in the image at
202+`/opt/grafana/plugins`, so the data mount does not hide it. Keep Grafana at one
203+container while using SQLite; back up the Volume before replacing or deleting
204+it. For example, run
205+`modal volume get atop-grafana-data grafana.db ./grafana.db.backup` and store
206+that backup securely because the database may contain credentials. To recover,
207+stop Grafana, upload the backup with
208+`modal volume put --force atop-grafana-data ./grafana.db.backup grafana.db`,
209+then redeploy. If an older deployment already has dashboards or other changes
210+not in provisioning, migrate its database before deploying this new mount.
211+
188212 Or run `python create_grafana_secret.py` to fetch those four public endpoint
189213 values from Pocket ID's discovery document and securely prompt for the client
190214 credentials and Modal token. It creates the same `atop-grafana` secret without
191-printing the private values.
215+printing the private values. It does not create the separate ClickHouse user or
216+password Secret.
192217
193218 Grafana redirects visitors to Pocket ID and does not expose a local password
194219 form. It uses PKCE, validates the signed ID token with Pocket ID’s JWKS, and
@@ -71,7 +71,7 @@ ClickHouse merges parts.
71 71
72 For low-volume ingestion, [modal_clickhouse.py](modal_clickhouse.py) runs the72 For low-volume ingestion, [modal_clickhouse.py](modal_clickhouse.py) runs the
73 ClickHouse HTTP API as a protected Modal Web Function. It starts only when an73 ClickHouse HTTP API as a protected Modal Web Function. It starts only when an
74-HTTP request arrives and is eligible to scale to zero after 60 seconds idle.74+HTTP request arrives and is eligible to scale to zero after five minutes idle.
75 Its data directory is retained in the named `atop-clickhouse-data` Modal75 Its data directory is retained in the named `atop-clickhouse-data` Modal
76 Volume. It intentionally limits Modal to one container because the Volume must76 Volume. It intentionally limits Modal to one container because the Volume must
77 not have concurrent ClickHouse writers.77 not have concurrent ClickHouse writers.
@@ -92,7 +92,9 @@ Modal Proxy Token credentials protect the endpoint at the edge; the collector
92 passes them as `Modal-Key` and `Modal-Secret`. They may also be supplied with92 passes them as `Modal-Key` and `Modal-Secret`. They may also be supplied with
93 `--clickhouse-modal-key` and `--clickhouse-modal-secret` instead of environment93 `--clickhouse-modal-key` and `--clickhouse-modal-secret` instead of environment
94 variables. Expect a cold-start delay on the first request after the endpoint94 variables. Expect a cold-start delay on the first request after the endpoint
95-has scaled down.95+has scaled down. Grafana's ClickHouse data source allows for that startup delay
96+with a 120-second connection timeout and a 180-second query timeout; a failed
97+startup still surfaces as an error rather than an indefinite loading state.
96 98
97 ### Modal ingestion app99 ### Modal ingestion app
98 100
@@ -159,6 +161,15 @@ and top processes). Grafana is public at the HTTP layer but authenticates every
159 visitor through Pocket ID; its Modal Proxy Token stays in a Modal Secret and is161 visitor through Pocket ID; its Modal Proxy Token stays in a Modal Secret and is
160 never sent to the browser.162 never sent to the browser.
161 163
164+Before deploying, create a ClickHouse user named `grafana_reader` with a
165+nonempty password. Grant it `SELECT` on `default.atop_samples` and
166+`default.atop_processes`, plus `SHOW TABLES` and `SHOW DATABASES` for the
167+`default` database. Store the matching password in a separate Modal Secret
168+named `atop-grafana-clickhouse-auth` under the key
169+`CLICKHOUSE_GRAFANA_PASSWORD`. Grafana requires that Secret at deployment;
170+its `default` ClickHouse user remains passwordless for ingestion but is not
171+used by Grafana. Do not put the password in the repository.
172+
162 ```sh173 ```sh
163 # In Pocket ID, create a confidential OIDC client and restrict it to the users174 # In Pocket ID, create a confidential OIDC client and restrict it to the users
164 # or groups that may view Grafana. Its redirect URI must exactly be:175 # or groups that may view Grafana. Its redirect URI must exactly be:
@@ -185,10 +196,24 @@ modal secret create atop-grafana \
185 modal deploy modal_grafana.py196 modal deploy modal_grafana.py
186 ```197 ```
187 198
199+The deployment creates a persistent Modal Volume named `atop-grafana-data`
200+and mounts it at `/var/lib/grafana`, where Grafana stores its SQLite database
201+and saved settings. The ClickHouse plugin stays in the image at
202+`/opt/grafana/plugins`, so the data mount does not hide it. Keep Grafana at one
203+container while using SQLite; back up the Volume before replacing or deleting
204+it. For example, run
205+`modal volume get atop-grafana-data grafana.db ./grafana.db.backup` and store
206+that backup securely because the database may contain credentials. To recover,
207+stop Grafana, upload the backup with
208+`modal volume put --force atop-grafana-data ./grafana.db.backup grafana.db`,
209+then redeploy. If an older deployment already has dashboards or other changes
210+not in provisioning, migrate its database before deploying this new mount.
211+
188 Or run `python create_grafana_secret.py` to fetch those four public endpoint212 Or run `python create_grafana_secret.py` to fetch those four public endpoint
189 values from Pocket ID's discovery document and securely prompt for the client213 values from Pocket ID's discovery document and securely prompt for the client
190 credentials and Modal token. It creates the same `atop-grafana` secret without214 credentials and Modal token. It creates the same `atop-grafana` secret without
191-printing the private values.215+printing the private values. It does not create the separate ClickHouse user or
216+password Secret.
192 217
193 Grafana redirects visitors to Pocket ID and does not expose a local password218 Grafana redirects visitors to Pocket ID and does not expose a local password
194 form. It uses PKCE, validates the signed ID token with Pocket ID’s JWKS, and219 form. It uses PKCE, validates the signed ID token with Pocket ID’s JWKS, and
modified create_grafana_secret.py +2 -0
@@ -4,6 +4,8 @@
44 Pocket ID exposes public OIDC endpoint metadata through its discovery document,
55 but intentionally never exposes an OAuth client secret. This helper fetches the
66 public metadata and prompts locally for the credentials that must remain private.
7+Grafana also requires the separate ``atop-grafana-clickhouse-auth`` Secret and
8+its matching read-only ClickHouse user; this helper does not create either.
79 """
810
911 import argparse
@@ -4,6 +4,8 @@
4 Pocket ID exposes public OIDC endpoint metadata through its discovery document,4 Pocket ID exposes public OIDC endpoint metadata through its discovery document,
5 but intentionally never exposes an OAuth client secret. This helper fetches the5 but intentionally never exposes an OAuth client secret. This helper fetches the
6 public metadata and prompts locally for the credentials that must remain private.6 public metadata and prompts locally for the credentials that must remain private.
7+Grafana also requires the separate ``atop-grafana-clickhouse-auth`` Secret and
8+its matching read-only ClickHouse user; this helper does not create either.
7 """9 """
8 10
9 import argparse11 import argparse
modified grafana/provisioning/dashboards/json/atop-overview.json +3 -3
@@ -7,7 +7,7 @@
77 "fieldConfig": {"defaults": {"unit": "percent"}, "overrides": []},
88 "gridPos": {"h": 9, "w": 12, "x": 0, "y": 0},
99 "id": 1,
10- "targets": [{"format": 1, "query": "SELECT epoch AS time, argMax(busy_pct, ingest_version) AS busy, argMax(wait_pct, ingest_version) AS iowait FROM atop_samples WHERE $__timeFilter(epoch) GROUP BY host, epoch ORDER BY epoch", "refId": "A"}],
10+ "targets": [{"editorType": "sql", "format": 0, "queryType": "timeseries", "rawSql": "SELECT epoch AS time, argMax(busy_pct, ingest_version) AS busy, argMax(wait_pct, ingest_version) AS iowait FROM atop_samples WHERE $__timeFilter(epoch) GROUP BY host, epoch ORDER BY epoch", "refId": "A"}],
1111 "title": "CPU busy and I/O wait",
1212 "type": "timeseries"
1313 },
@@ -16,7 +16,7 @@
1616 "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []},
1717 "gridPos": {"h": 9, "w": 12, "x": 12, "y": 0},
1818 "id": 2,
19- "targets": [{"format": 1, "query": "SELECT epoch AS time, argMax(load1, ingest_version) AS load_1m, argMax(load5, ingest_version) AS load_5m, argMax(load15, ingest_version) AS load_15m FROM atop_samples WHERE $__timeFilter(epoch) GROUP BY host, epoch ORDER BY epoch", "refId": "A"}],
19+ "targets": [{"editorType": "sql", "format": 0, "queryType": "timeseries", "rawSql": "SELECT epoch AS time, argMax(load1, ingest_version) AS load_1m, argMax(load5, ingest_version) AS load_5m, argMax(load15, ingest_version) AS load_15m FROM atop_samples WHERE $__timeFilter(epoch) GROUP BY host, epoch ORDER BY epoch", "refId": "A"}],
2020 "title": "Load average",
2121 "type": "timeseries"
2222 },
@@ -26,7 +26,7 @@
2626 "gridPos": {"h": 9, "w": 24, "x": 0, "y": 9},
2727 "id": 3,
2828 "options": {"showHeader": true},
29- "targets": [{"format": 1, "query": "SELECT name AS process, round(avg(cpu_pct), 2) AS avg_cpu_pct, round(max(cpu_pct), 2) AS peak_cpu_pct FROM (SELECT host, epoch, pid, name, argMax(cpu_pct, ingest_version) AS cpu_pct FROM atop_processes WHERE $__timeFilter(epoch) GROUP BY host, epoch, pid, name) GROUP BY name ORDER BY avg_cpu_pct DESC LIMIT 15", "refId": "A"}],
29+ "targets": [{"editorType": "sql", "format": 1, "queryType": "table", "rawSql": "SELECT name AS process, round(avg(cpu_pct), 2) AS avg_cpu_pct, round(max(cpu_pct), 2) AS peak_cpu_pct FROM (SELECT host, epoch, pid, name, argMax(cpu_pct, ingest_version) AS cpu_pct FROM atop_processes WHERE $__timeFilter(epoch) GROUP BY host, epoch, pid, name) GROUP BY name ORDER BY avg_cpu_pct DESC LIMIT 15", "refId": "A"}],
3030 "title": "Top processes",
3131 "type": "table"
3232 }
@@ -7,7 +7,7 @@
7 "fieldConfig": {"defaults": {"unit": "percent"}, "overrides": []},7 "fieldConfig": {"defaults": {"unit": "percent"}, "overrides": []},
8 "gridPos": {"h": 9, "w": 12, "x": 0, "y": 0},8 "gridPos": {"h": 9, "w": 12, "x": 0, "y": 0},
9 "id": 1,9 "id": 1,
10- "targets": [{"format": 1, "query": "SELECT epoch AS time, argMax(busy_pct, ingest_version) AS busy, argMax(wait_pct, ingest_version) AS iowait FROM atop_samples WHERE $__timeFilter(epoch) GROUP BY host, epoch ORDER BY epoch", "refId": "A"}],10+ "targets": [{"editorType": "sql", "format": 0, "queryType": "timeseries", "rawSql": "SELECT epoch AS time, argMax(busy_pct, ingest_version) AS busy, argMax(wait_pct, ingest_version) AS iowait FROM atop_samples WHERE $__timeFilter(epoch) GROUP BY host, epoch ORDER BY epoch", "refId": "A"}],
11 "title": "CPU busy and I/O wait",11 "title": "CPU busy and I/O wait",
12 "type": "timeseries"12 "type": "timeseries"
13 },13 },
@@ -16,7 +16,7 @@
16 "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []},16 "fieldConfig": {"defaults": {"unit": "short"}, "overrides": []},
17 "gridPos": {"h": 9, "w": 12, "x": 12, "y": 0},17 "gridPos": {"h": 9, "w": 12, "x": 12, "y": 0},
18 "id": 2,18 "id": 2,
19- "targets": [{"format": 1, "query": "SELECT epoch AS time, argMax(load1, ingest_version) AS load_1m, argMax(load5, ingest_version) AS load_5m, argMax(load15, ingest_version) AS load_15m FROM atop_samples WHERE $__timeFilter(epoch) GROUP BY host, epoch ORDER BY epoch", "refId": "A"}],19+ "targets": [{"editorType": "sql", "format": 0, "queryType": "timeseries", "rawSql": "SELECT epoch AS time, argMax(load1, ingest_version) AS load_1m, argMax(load5, ingest_version) AS load_5m, argMax(load15, ingest_version) AS load_15m FROM atop_samples WHERE $__timeFilter(epoch) GROUP BY host, epoch ORDER BY epoch", "refId": "A"}],
20 "title": "Load average",20 "title": "Load average",
21 "type": "timeseries"21 "type": "timeseries"
22 },22 },
@@ -26,7 +26,7 @@
26 "gridPos": {"h": 9, "w": 24, "x": 0, "y": 9},26 "gridPos": {"h": 9, "w": 24, "x": 0, "y": 9},
27 "id": 3,27 "id": 3,
28 "options": {"showHeader": true},28 "options": {"showHeader": true},
29- "targets": [{"format": 1, "query": "SELECT name AS process, round(avg(cpu_pct), 2) AS avg_cpu_pct, round(max(cpu_pct), 2) AS peak_cpu_pct FROM (SELECT host, epoch, pid, name, argMax(cpu_pct, ingest_version) AS cpu_pct FROM atop_processes WHERE $__timeFilter(epoch) GROUP BY host, epoch, pid, name) GROUP BY name ORDER BY avg_cpu_pct DESC LIMIT 15", "refId": "A"}],29+ "targets": [{"editorType": "sql", "format": 1, "queryType": "table", "rawSql": "SELECT name AS process, round(avg(cpu_pct), 2) AS avg_cpu_pct, round(max(cpu_pct), 2) AS peak_cpu_pct FROM (SELECT host, epoch, pid, name, argMax(cpu_pct, ingest_version) AS cpu_pct FROM atop_processes WHERE $__timeFilter(epoch) GROUP BY host, epoch, pid, name) GROUP BY name ORDER BY avg_cpu_pct DESC LIMIT 15", "refId": "A"}],
30 "title": "Top processes",30 "title": "Top processes",
31 "type": "table"31 "type": "table"
32 }32 }
modified grafana/provisioning/datasources/clickhouse.yaml +4 -1
@@ -11,7 +11,9 @@ datasources:
1111 port: 443
1212 protocol: http
1313 secure: true
14- username: default
14+ dialTimeout: 120
15+ queryTimeout: 180
16+ username: grafana_reader
1517 defaultDatabase: default
1618 httpHeaders:
1719 - name: Modal-Key
@@ -19,5 +21,6 @@ datasources:
1921 - name: Modal-Secret
2022 secure: true
2123 secureJsonData:
24+ password: $__env{CLICKHOUSE_GRAFANA_PASSWORD}
2225 secureHttpHeaders.Modal-Key: $__env{MODAL_KEY}
2326 secureHttpHeaders.Modal-Secret: $__env{MODAL_SECRET}
@@ -11,7 +11,9 @@ datasources:
11 port: 44311 port: 443
12 protocol: http12 protocol: http
13 secure: true13 secure: true
14- username: default14+ dialTimeout: 120
15+ queryTimeout: 180
16+ username: grafana_reader
15 defaultDatabase: default17 defaultDatabase: default
16 httpHeaders:18 httpHeaders:
17 - name: Modal-Key19 - name: Modal-Key
@@ -19,5 +21,6 @@ datasources:
19 - name: Modal-Secret21 - name: Modal-Secret
20 secure: true22 secure: true
21 secureJsonData:23 secureJsonData:
24+ password: $__env{CLICKHOUSE_GRAFANA_PASSWORD}
22 secureHttpHeaders.Modal-Key: $__env{MODAL_KEY}25 secureHttpHeaders.Modal-Key: $__env{MODAL_KEY}
23 secureHttpHeaders.Modal-Secret: $__env{MODAL_SECRET}26 secureHttpHeaders.Modal-Secret: $__env{MODAL_SECRET}
modified modal_clickhouse.py +4 -4
@@ -18,7 +18,7 @@ import modal
1818
1919 APP_NAME = "atop-clickhouse"
2020 DATA_VOLUME_NAME = "atop-clickhouse-data"
21-IDLE_SECONDS = 60
21+IDLE_SECONDS = 300
2222
2323 app = modal.App(APP_NAME)
2424 data_volume = modal.Volume.from_name(DATA_VOLUME_NAME, create_if_missing=True)
@@ -35,13 +35,13 @@ image = modal.Image.from_registry(
3535 scaledown_window=IDLE_SECONDS,
3636 timeout=10 * 60,
3737 )
38-@modal.web_server(8123, startup_timeout=120, requires_proxy_auth=True)
38+@modal.web_server(8123, startup_timeout=240, requires_proxy_auth=True)
3939 def clickhouse() -> None:
4040 """Start ClickHouse's native HTTP API; Modal proxies requests to port 8123."""
4141 process = subprocess.Popen(
4242 ["clickhouse-server", "--config-file=/etc/clickhouse-server/config.xml"]
4343 )
44- deadline = time.monotonic() + 90
44+ deadline = time.monotonic() + 210
4545 while time.monotonic() < deadline:
4646 if process.poll() is not None:
4747 raise RuntimeError(f"ClickHouse exited during startup ({process.returncode})")
@@ -51,4 +51,4 @@ def clickhouse() -> None:
5151 return
5252 time.sleep(0.2)
5353 process.terminate()
54- raise RuntimeError("ClickHouse did not open port 8123 within 90 seconds")
54+ raise RuntimeError("ClickHouse did not open port 8123 within 210 seconds")
@@ -18,7 +18,7 @@ import modal
18 18
19 APP_NAME = "atop-clickhouse"19 APP_NAME = "atop-clickhouse"
20 DATA_VOLUME_NAME = "atop-clickhouse-data"20 DATA_VOLUME_NAME = "atop-clickhouse-data"
21-IDLE_SECONDS = 6021+IDLE_SECONDS = 300
22 22
23 app = modal.App(APP_NAME)23 app = modal.App(APP_NAME)
24 data_volume = modal.Volume.from_name(DATA_VOLUME_NAME, create_if_missing=True)24 data_volume = modal.Volume.from_name(DATA_VOLUME_NAME, create_if_missing=True)
@@ -35,13 +35,13 @@ image = modal.Image.from_registry(
35 scaledown_window=IDLE_SECONDS,35 scaledown_window=IDLE_SECONDS,
36 timeout=10 * 60,36 timeout=10 * 60,
37 )37 )
38-@modal.web_server(8123, startup_timeout=120, requires_proxy_auth=True)38+@modal.web_server(8123, startup_timeout=240, requires_proxy_auth=True)
39 def clickhouse() -> None:39 def clickhouse() -> None:
40 """Start ClickHouse's native HTTP API; Modal proxies requests to port 8123."""40 """Start ClickHouse's native HTTP API; Modal proxies requests to port 8123."""
41 process = subprocess.Popen(41 process = subprocess.Popen(
42 ["clickhouse-server", "--config-file=/etc/clickhouse-server/config.xml"]42 ["clickhouse-server", "--config-file=/etc/clickhouse-server/config.xml"]
43 )43 )
44- deadline = time.monotonic() + 9044+ deadline = time.monotonic() + 210
45 while time.monotonic() < deadline:45 while time.monotonic() < deadline:
46 if process.poll() is not None:46 if process.poll() is not None:
47 raise RuntimeError(f"ClickHouse exited during startup ({process.returncode})")47 raise RuntimeError(f"ClickHouse exited during startup ({process.returncode})")
@@ -51,4 +51,4 @@ def clickhouse() -> None:
51 return51 return
52 time.sleep(0.2)52 time.sleep(0.2)
53 process.terminate()53 process.terminate()
54- raise RuntimeError("ClickHouse did not open port 8123 within 90 seconds")54+ raise RuntimeError("ClickHouse did not open port 8123 within 210 seconds")
modified modal_grafana.py +38 -41
@@ -2,20 +2,22 @@
22
33 Create a Modal Secret named ``atop-grafana`` before deploying. It needs:
44 ``CLICKHOUSE_HOST`` (hostname only), ``MODAL_KEY``, ``MODAL_SECRET``, and
5-the Pocket ID Generic OAuth settings, including ``GF_SERVER_ROOT_URL``. Deploy
6-with ``modal deploy modal_grafana.py``.
5+the Pocket ID Generic OAuth settings, including ``GF_SERVER_ROOT_URL``. A
6+separate ``atop-grafana-clickhouse-auth`` Secret must contain the nonempty
7+``CLICKHOUSE_GRAFANA_PASSWORD`` for the read-only ``grafana_reader`` user.
8+Deploy with ``modal deploy modal_grafana.py``.
79 """
810
9-import os
10-import socket
1111 import subprocess
1212 import time
1313
1414 import modal
1515
1616 APP_NAME = "atop-grafana"
17+DATA_VOLUME_NAME = "atop-grafana-data"
1718 IDLE_SECONDS = 300
1819 app = modal.App(APP_NAME)
20+data_volume = modal.Volume.from_name(DATA_VOLUME_NAME, create_if_missing=True)
1921 grafana_secret = modal.Secret.from_name(
2022 "atop-grafana",
2123 required_keys=[
@@ -25,23 +27,44 @@ grafana_secret = modal.Secret.from_name(
2527 "GF_AUTH_GENERIC_OAUTH_API_URL", "GF_AUTH_GENERIC_OAUTH_JWK_SET_URL",
2628 ],
2729 )
30+clickhouse_auth_secret = modal.Secret.from_name(
31+ "atop-grafana-clickhouse-auth",
32+ required_keys=["CLICKHOUSE_GRAFANA_PASSWORD"],
33+)
2834 image = (
29- modal.Image.from_dockerfile("grafana/Dockerfile")
35+ modal.Image.debian_slim(python_version="3.12")
36+ .apt_install("adduser", "ca-certificates", "curl", "libfontconfig1", "musl")
37+ .run_commands(
38+ "curl --fail --location --silent --show-error "
39+ "https://dl.grafana.com/oss/release/grafana_11.6.0_amd64.deb "
40+ "--output /tmp/grafana.deb "
41+ "&& echo 'c5159ce58c1eb7b4a47474951f0334055dd1a4b3325ffc432ef1761da6474b2a /tmp/grafana.deb' "
42+ "| sha256sum --check "
43+ "&& apt-get update "
44+ "&& apt-get install --yes --no-install-recommends /tmp/grafana.deb "
45+ "&& /usr/share/grafana/bin/grafana cli --homepath=/usr/share/grafana "
46+ "--pluginsDir=/opt/grafana/plugins "
47+ "plugins install grafana-clickhouse-datasource"
48+ )
3049 .add_local_dir("grafana/provisioning", "/etc/grafana/provisioning")
3150 )
3251
3352
3453 @app.function(
3554 image=image,
36- secrets=[grafana_secret],
55+ secrets=[grafana_secret, clickhouse_auth_secret],
56+ volumes={"/var/lib/grafana": data_volume},
3757 min_containers=0,
3858 max_containers=1,
3959 scaledown_window=IDLE_SECONDS,
4060 timeout=15 * 60,
4161 env={
42- # Modal's web-server proxy connects over IPv4; Grafana otherwise binds
43- # to [::], which can leave the public endpoint waiting indefinitely.
4462 "GF_SERVER_HTTP_ADDR": "0.0.0.0",
63+ "GF_PATHS_DATA": "/var/lib/grafana",
64+ "GF_PATHS_LOGS": "/var/log/grafana",
65+ "GF_PATHS_PLUGINS": "/opt/grafana/plugins",
66+ "GF_DATABASE_WAL": "false",
67+ "GF_PATHS_PROVISIONING": "/etc/grafana/provisioning",
4568 "GF_AUTH_DISABLE_LOGIN_FORM": "true",
4669 "GF_AUTH_GENERIC_OAUTH_ENABLED": "true",
4770 "GF_AUTH_GENERIC_OAUTH_NAME": "Pocket ID",
@@ -54,39 +77,13 @@ image = (
5477 },
5578 )
5679 @modal.concurrent(max_inputs=100)
57-@modal.web_server(8080, startup_timeout=120)
80+@modal.web_server(3000, startup_timeout=120)
5881 def grafana() -> None:
5982 """Run a public Grafana login page; its data-source credentials stay server-side."""
60- process = subprocess.Popen(
61- ["grafana", "server", "--homepath=/usr/share/grafana", "--config=/etc/grafana/grafana.ini"],
62- # Pass this through the spawned process as well as Modal's function
63- # environment, since the web-server proxy reaches the service over IPv4.
64- env={**os.environ, "GF_SERVER_HTTP_ADDR": "0.0.0.0"},
65- )
66- # Start the external listener only after Grafana accepts connections.
67- for _ in range(60):
68- if process.poll() is not None:
69- raise RuntimeError(f"Grafana exited during startup ({process.returncode})")
70- with socket.socket() as probe:
71- probe.settimeout(0.25)
72- if probe.connect_ex(("127.0.0.1", 3000)) == 0:
73- print("Grafana backend ready on port 3000", flush=True)
74- break
75- time.sleep(0.5)
76- else:
77- raise RuntimeError("Grafana did not listen on port 3000 within 30 seconds")
78-
79- proxy = subprocess.Popen([
80- "socat", "TCP-LISTEN:8080,bind=0.0.0.0,reuseaddr,fork",
81- "TCP:127.0.0.1:3000",
83+ grafana_process = subprocess.Popen([
84+ "/usr/share/grafana/bin/grafana", "server",
85+ "--homepath=/usr/share/grafana", "--config=/etc/grafana/grafana.ini",
8286 ])
83- for _ in range(20):
84- if proxy.poll() is not None:
85- raise RuntimeError(f"Grafana proxy exited during startup ({proxy.returncode})")
86- with socket.socket() as probe:
87- probe.settimeout(0.25)
88- if probe.connect_ex(("127.0.0.1", 8080)) == 0:
89- print("Grafana proxy ready on port 8080", flush=True)
90- return
91- time.sleep(0.5)
92- raise RuntimeError("Grafana proxy did not listen on port 8080 within 10 seconds")
87+ time.sleep(1)
88+ if grafana_process.poll() is not None:
89+ raise RuntimeError(f"Grafana exited during startup ({grafana_process.returncode})")
@@ -2,20 +2,22 @@
2 2
3 Create a Modal Secret named ``atop-grafana`` before deploying. It needs:3 Create a Modal Secret named ``atop-grafana`` before deploying. It needs:
4 ``CLICKHOUSE_HOST`` (hostname only), ``MODAL_KEY``, ``MODAL_SECRET``, and4 ``CLICKHOUSE_HOST`` (hostname only), ``MODAL_KEY``, ``MODAL_SECRET``, and
5-the Pocket ID Generic OAuth settings, including ``GF_SERVER_ROOT_URL``. Deploy5+the Pocket ID Generic OAuth settings, including ``GF_SERVER_ROOT_URL``. A
6-with ``modal deploy modal_grafana.py``.6+separate ``atop-grafana-clickhouse-auth`` Secret must contain the nonempty
7+``CLICKHOUSE_GRAFANA_PASSWORD`` for the read-only ``grafana_reader`` user.
8+Deploy with ``modal deploy modal_grafana.py``.
7 """9 """
8 10
9-import os
10-import socket
11 import subprocess11 import subprocess
12 import time12 import time
13 13
14 import modal14 import modal
15 15
16 APP_NAME = "atop-grafana"16 APP_NAME = "atop-grafana"
17+DATA_VOLUME_NAME = "atop-grafana-data"
17 IDLE_SECONDS = 30018 IDLE_SECONDS = 300
18 app = modal.App(APP_NAME)19 app = modal.App(APP_NAME)
20+data_volume = modal.Volume.from_name(DATA_VOLUME_NAME, create_if_missing=True)
19 grafana_secret = modal.Secret.from_name(21 grafana_secret = modal.Secret.from_name(
20 "atop-grafana",22 "atop-grafana",
21 required_keys=[23 required_keys=[
@@ -25,23 +27,44 @@ grafana_secret = modal.Secret.from_name(
25 "GF_AUTH_GENERIC_OAUTH_API_URL", "GF_AUTH_GENERIC_OAUTH_JWK_SET_URL",27 "GF_AUTH_GENERIC_OAUTH_API_URL", "GF_AUTH_GENERIC_OAUTH_JWK_SET_URL",
26 ],28 ],
27 )29 )
30+clickhouse_auth_secret = modal.Secret.from_name(
31+ "atop-grafana-clickhouse-auth",
32+ required_keys=["CLICKHOUSE_GRAFANA_PASSWORD"],
33+)
28 image = (34 image = (
29- modal.Image.from_dockerfile("grafana/Dockerfile")35+ modal.Image.debian_slim(python_version="3.12")
36+ .apt_install("adduser", "ca-certificates", "curl", "libfontconfig1", "musl")
37+ .run_commands(
38+ "curl --fail --location --silent --show-error "
39+ "https://dl.grafana.com/oss/release/grafana_11.6.0_amd64.deb "
40+ "--output /tmp/grafana.deb "
41+ "&& echo 'c5159ce58c1eb7b4a47474951f0334055dd1a4b3325ffc432ef1761da6474b2a /tmp/grafana.deb' "
42+ "| sha256sum --check "
43+ "&& apt-get update "
44+ "&& apt-get install --yes --no-install-recommends /tmp/grafana.deb "
45+ "&& /usr/share/grafana/bin/grafana cli --homepath=/usr/share/grafana "
46+ "--pluginsDir=/opt/grafana/plugins "
47+ "plugins install grafana-clickhouse-datasource"
48+ )
30 .add_local_dir("grafana/provisioning", "/etc/grafana/provisioning")49 .add_local_dir("grafana/provisioning", "/etc/grafana/provisioning")
31 )50 )
32 51
33 52
34 @app.function(53 @app.function(
35 image=image,54 image=image,
36- secrets=[grafana_secret],55+ secrets=[grafana_secret, clickhouse_auth_secret],
56+ volumes={"/var/lib/grafana": data_volume},
37 min_containers=0,57 min_containers=0,
38 max_containers=1,58 max_containers=1,
39 scaledown_window=IDLE_SECONDS,59 scaledown_window=IDLE_SECONDS,
40 timeout=15 * 60,60 timeout=15 * 60,
41 env={61 env={
42- # Modal's web-server proxy connects over IPv4; Grafana otherwise binds
43- # to [::], which can leave the public endpoint waiting indefinitely.
44 "GF_SERVER_HTTP_ADDR": "0.0.0.0",62 "GF_SERVER_HTTP_ADDR": "0.0.0.0",
63+ "GF_PATHS_DATA": "/var/lib/grafana",
64+ "GF_PATHS_LOGS": "/var/log/grafana",
65+ "GF_PATHS_PLUGINS": "/opt/grafana/plugins",
66+ "GF_DATABASE_WAL": "false",
67+ "GF_PATHS_PROVISIONING": "/etc/grafana/provisioning",
45 "GF_AUTH_DISABLE_LOGIN_FORM": "true",68 "GF_AUTH_DISABLE_LOGIN_FORM": "true",
46 "GF_AUTH_GENERIC_OAUTH_ENABLED": "true",69 "GF_AUTH_GENERIC_OAUTH_ENABLED": "true",
47 "GF_AUTH_GENERIC_OAUTH_NAME": "Pocket ID",70 "GF_AUTH_GENERIC_OAUTH_NAME": "Pocket ID",
@@ -54,39 +77,13 @@ image = (
54 },77 },
55 )78 )
56 @modal.concurrent(max_inputs=100)79 @modal.concurrent(max_inputs=100)
57-@modal.web_server(8080, startup_timeout=120)80+@modal.web_server(3000, startup_timeout=120)
58 def grafana() -> None:81 def grafana() -> None:
59 """Run a public Grafana login page; its data-source credentials stay server-side."""82 """Run a public Grafana login page; its data-source credentials stay server-side."""
60- process = subprocess.Popen(83+ grafana_process = subprocess.Popen([
61- ["grafana", "server", "--homepath=/usr/share/grafana", "--config=/etc/grafana/grafana.ini"],84+ "/usr/share/grafana/bin/grafana", "server",
62- # Pass this through the spawned process as well as Modal's function85+ "--homepath=/usr/share/grafana", "--config=/etc/grafana/grafana.ini",
63- # environment, since the web-server proxy reaches the service over IPv4.
64- env={**os.environ, "GF_SERVER_HTTP_ADDR": "0.0.0.0"},
65- )
66- # Start the external listener only after Grafana accepts connections.
67- for _ in range(60):
68- if process.poll() is not None:
69- raise RuntimeError(f"Grafana exited during startup ({process.returncode})")
70- with socket.socket() as probe:
71- probe.settimeout(0.25)
72- if probe.connect_ex(("127.0.0.1", 3000)) == 0:
73- print("Grafana backend ready on port 3000", flush=True)
74- break
75- time.sleep(0.5)
76- else:
77- raise RuntimeError("Grafana did not listen on port 3000 within 30 seconds")
78-
79- proxy = subprocess.Popen([
80- "socat", "TCP-LISTEN:8080,bind=0.0.0.0,reuseaddr,fork",
81- "TCP:127.0.0.1:3000",
82 ])86 ])
83- for _ in range(20):87+ time.sleep(1)
84- if proxy.poll() is not None:88+ if grafana_process.poll() is not None:
85- raise RuntimeError(f"Grafana proxy exited during startup ({proxy.returncode})")89+ raise RuntimeError(f"Grafana exited during startup ({grafana_process.returncode})")
86- with socket.socket() as probe:
87- probe.settimeout(0.25)
88- if probe.connect_ex(("127.0.0.1", 8080)) == 0:
89- print("Grafana proxy ready on port 8080", flush=True)
90- return
91- time.sleep(0.5)
92- raise RuntimeError("Grafana proxy did not listen on port 8080 within 10 seconds")
modified modal_pocket_id.py +2 -1
@@ -43,7 +43,8 @@ image = (
4343 "ALLOW_INSECURE_CALLBACK_URLS": "false",
4444 # Modal Volumes are network filesystems; SQLite's WAL mode is unsafe
4545 # there. Keep a single writer and use DELETE journaling instead.
46- "DB_CONNECTION_STRING": "data/pocket-id.db?_journal_mode=DELETE",
46+ "DB_CONNECTION_STRING": "/app/data/pocket-id.db?_journal_mode=DELETE",
47+ "UPLOAD_PATH": "/app/data/uploads",
4748 },
4849 )
4950 @modal.concurrent(max_inputs=100)
@@ -43,7 +43,8 @@ image = (
43 "ALLOW_INSECURE_CALLBACK_URLS": "false",43 "ALLOW_INSECURE_CALLBACK_URLS": "false",
44 # Modal Volumes are network filesystems; SQLite's WAL mode is unsafe44 # Modal Volumes are network filesystems; SQLite's WAL mode is unsafe
45 # there. Keep a single writer and use DELETE journaling instead.45 # there. Keep a single writer and use DELETE journaling instead.
46- "DB_CONNECTION_STRING": "data/pocket-id.db?_journal_mode=DELETE",46+ "DB_CONNECTION_STRING": "/app/data/pocket-id.db?_journal_mode=DELETE",
47+ "UPLOAD_PATH": "/app/data/uploads",
47 },48 },
48 )49 )
49 @modal.concurrent(max_inputs=100)50 @modal.concurrent(max_inputs=100)