rickub
nandi/atop-analyzepublic Fork
⑂ a4ffaee
Branches
main View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/nandi/atop-analyze.git
git clone ssh://git@rickub.com/nandi/atop-analyze.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Download ZIP

Add memory metrics and incident replay dashboard panels

nandithebull committed 2026-09-21T22:38:52-07:00 Browse files
a4ffaee parent: 4edeea9
modified README.md +10 -7
@@ -56,16 +56,19 @@ export CLICKHOUSE_PASSWORD='…'
5656
5757 `--clickhouse-init` creates `atop_samples` (one host interval per row) and
5858 `atop_processes` (one observed process per interval) in the selected database.
59+It also applies additive, non-destructive schema upgrades to existing tables.
5960 Use `--clickhouse-database`, `--clickhouse-sample-table`, and
60-`--clickhouse-process-table` to change those names. Existing tables are never
61-modified; omit `--clickhouse-init` once the schema exists.
61+`--clickhouse-process-table` to change those names. Run it once after upgrading
62+the collector, then omit it during routine imports.
6263
6364 The sample table records CPU mode percentages, load averages, context switches,
64-interrupts, host, epoch, core count, and interval length. The process table
65-records host, epoch, PID, process name, whole-machine CPU percentage, and
66-interval length. Tables use `ReplacingMergeTree`, ordered by their natural
67-sample keys, so reimporting the same log converges to one row per key after
68-ClickHouse merges parts.
65+interrupts, and host memory in KiB: total, free, available, page cache, buffers,
66+and slab. The process table records host, epoch, PID, process name,
67+whole-machine CPU percentage, and per-process virtual memory, RSS, shared text,
68+swap, memory growth, and page faults. Per-process memory columns are nullable
69+when an atop record does not include a matching process-memory entry. Tables use
70+`ReplacingMergeTree`, ordered by their natural sample keys, so reimporting the
71+same log converges to one row per key after ClickHouse merges parts.
6972
7073 ### Scale-to-zero Modal endpoint
7174
@@ -56,16 +56,19 @@ export CLICKHOUSE_PASSWORD='…'
56 56
57 `--clickhouse-init` creates `atop_samples` (one host interval per row) and57 `--clickhouse-init` creates `atop_samples` (one host interval per row) and
58 `atop_processes` (one observed process per interval) in the selected database.58 `atop_processes` (one observed process per interval) in the selected database.
59+It also applies additive, non-destructive schema upgrades to existing tables.
59 Use `--clickhouse-database`, `--clickhouse-sample-table`, and60 Use `--clickhouse-database`, `--clickhouse-sample-table`, and
60-`--clickhouse-process-table` to change those names. Existing tables are never61+`--clickhouse-process-table` to change those names. Run it once after upgrading
61-modified; omit `--clickhouse-init` once the schema exists.62+the collector, then omit it during routine imports.
62 63
63 The sample table records CPU mode percentages, load averages, context switches,64 The sample table records CPU mode percentages, load averages, context switches,
64-interrupts, host, epoch, core count, and interval length. The process table65+interrupts, and host memory in KiB: total, free, available, page cache, buffers,
65-records host, epoch, PID, process name, whole-machine CPU percentage, and66+and slab. The process table records host, epoch, PID, process name,
66-interval length. Tables use `ReplacingMergeTree`, ordered by their natural67+whole-machine CPU percentage, and per-process virtual memory, RSS, shared text,
67-sample keys, so reimporting the same log converges to one row per key after68+swap, memory growth, and page faults. Per-process memory columns are nullable
68-ClickHouse merges parts.69+when an atop record does not include a matching process-memory entry. Tables use
70+`ReplacingMergeTree`, ordered by their natural sample keys, so reimporting the
71+same log converges to one row per key after ClickHouse merges parts.
69 72
70 ### Scale-to-zero Modal endpoint73 ### Scale-to-zero Modal endpoint
71 74
modified atop_analyze.py +90 -10
@@ -48,7 +48,7 @@ IDENTIFIER_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
4848 logger = logging.getLogger(__name__)
4949 logger.setLevel(logging.INFO)
5050
51-WANTED_LABELS = {"CPU", "CPL", "PRC"}
51+WANTED_LABELS = {"CPU", "CPL", "MEM", "PRC", "PRM"}
5252 DATE_RE = re.compile(r"^\d{4}/\d{2}/\d{2}$")
5353
5454 # Offsets into the label-specific fields (i.e. after the 6 common header fields).
@@ -71,6 +71,17 @@ CPL_LOAD1, CPL_LOAD5, CPL_LOAD15, CPL_CTXSW, CPL_INTR = 1, 2, 3, 4, 5
7171 PRC_PID, PRC_NAME, PRC_STATE, PRC_HERTZ, PRC_UTIME, PRC_STIME = 0, 1, 2, 3, 4, 5
7272 PRC_ISPROC = 13
7373
74+# MEM: pagesize physmem freemem cachemem buffermem slabmem ... availablemem
75+# Memory quantities emitted by atop are KiB, apart from pagesize.
76+MEM_TOTAL, MEM_FREE, MEM_CACHE, MEM_BUFFER, MEM_SLAB, MEM_AVAILABLE = 1, 2, 3, 4, 5, 24
77+
78+# PRM: pid name state pagesize vmem rss sharedtext vgrow rgrow minflt majflt
79+# ... swap ... tgid is_process
80+# Like MEM, memory quantities are KiB. The growth counters are signed.
81+PRM_PID, PRM_NAME, PRM_VMEM, PRM_RSS, PRM_SHARED_TEXT = 0, 1, 4, 5, 6
82+PRM_VMEM_GROWTH, PRM_RSS_GROWTH, PRM_MINFLT, PRM_MAJFLT = 7, 8, 9, 10
83+PRM_SWAP, PRM_ISPROC = 14, 16
84+
7485
7586 @dataclass
7687 class Sample:
@@ -88,7 +99,14 @@ class Sample:
8899 load15: float = 0.0
89100 ctxsw: int = 0
90101 intr: int = 0
102+ mem_total_kib: int = 0
103+ mem_free_kib: int = 0
104+ mem_available_kib: int = 0
105+ mem_cache_kib: int = 0
106+ mem_buffer_kib: int = 0
107+ mem_slab_kib: int = 0
91108 procs: list[tuple[str, int, float]] = field(default_factory=list)
109+ proc_memory: dict[tuple[int, str], dict[str, int]] = field(default_factory=dict)
92110
93111 @property
94112 def total_ticks(self) -> int:
@@ -122,7 +140,7 @@ def log_path(date: str | None, logdir: str) -> str:
122140 def run_atop(path: str, begin: str | None, end: str | None) -> str:
123141 if not shutil.which("atop"):
124142 raise AtopError("atop is not installed (pacman -S atop)")
125- cmd = ["atop", "-r", path, "-Z", "-P", "CPU,CPL,PRC"]
143+ cmd = ["atop", "-r", path, "-Z", "-P", "CPU,CPL,MEM,PRC,PRM"]
126144 if begin:
127145 cmd += ["-b", begin]
128146 if end:
@@ -188,6 +206,13 @@ def parse(raw: str) -> list[Sample]:
188206 current.load15 = float(rest[CPL_LOAD15])
189207 current.ctxsw = int(rest[CPL_CTXSW])
190208 current.intr = int(rest[CPL_INTR])
209+ elif label == "MEM":
210+ current.mem_total_kib = int(rest[MEM_TOTAL])
211+ current.mem_free_kib = int(rest[MEM_FREE])
212+ current.mem_available_kib = int(rest[MEM_AVAILABLE])
213+ current.mem_cache_kib = int(rest[MEM_CACHE])
214+ current.mem_buffer_kib = int(rest[MEM_BUFFER])
215+ current.mem_slab_kib = int(rest[MEM_SLAB])
191216 elif label == "PRC":
192217 if rest[PRC_ISPROC] != "y":
193218 continue # thread line, not a process
@@ -197,8 +222,24 @@ def parse(raw: str) -> list[Sample]:
197222 capacity = hertz * interval * max(current.ncpu or 1, 1)
198223 pct = 100.0 * ticks / capacity if capacity else 0.0
199224 current.procs.append((rest[PRC_NAME], int(rest[PRC_PID]), pct))
225+ elif label == "PRM":
226+ if rest[PRM_ISPROC] != "y":
227+ continue # thread line, not a process
228+ current.proc_memory[(int(rest[PRM_PID]), rest[PRM_NAME])] = {
229+ "vmem_kib": int(rest[PRM_VMEM]),
230+ "rss_kib": int(rest[PRM_RSS]),
231+ "shared_text_kib": int(rest[PRM_SHARED_TEXT]),
232+ "vmem_growth_kib": int(rest[PRM_VMEM_GROWTH]),
233+ "rss_growth_kib": int(rest[PRM_RSS_GROWTH]),
234+ "minflt": int(rest[PRM_MINFLT]),
235+ "majflt": int(rest[PRM_MAJFLT]),
236+ "swap_kib": int(rest[PRM_SWAP]),
237+ }
200238 except (IndexError, ValueError):
201- continue # a label whose layout this atop version changed
239+ raise AtopError(
240+ f"cannot parse {label} output from this atop version; "
241+ "the required parsable field layout is missing or invalid"
242+ )
202243
203244 if current and not skip_this:
204245 samples.append(current)
@@ -291,12 +332,19 @@ def write_clickhouse(
291332 ncpu UInt16, user_pct Float32, sys_pct Float32, nice_pct Float32,
292333 idle_pct Float32, wait_pct Float32, irq_pct Float32, softirq_pct Float32,
293334 steal_pct Float32, busy_pct Float32, load1 Float32, load5 Float32,
294- load15 Float32, ctxsw UInt64, intr UInt64, ingest_version UInt64
335+ load15 Float32, ctxsw UInt64, intr UInt64,
336+ mem_total_kib UInt64, mem_free_kib UInt64, mem_available_kib UInt64,
337+ mem_cache_kib UInt64, mem_buffer_kib UInt64, mem_slab_kib UInt64,
338+ ingest_version UInt64
295339 ) ENGINE = ReplacingMergeTree(ingest_version) ORDER BY (host, epoch)""", "", user, password,
296340 modal_key, modal_secret)
297341 clickhouse_request(url, f"""CREATE TABLE IF NOT EXISTS {processes_target} (
298342 host LowCardinality(String), epoch DateTime, pid UInt32,
299343 name LowCardinality(String), cpu_pct Float32, interval_seconds UInt32,
344+ vmem_kib Nullable(UInt64), rss_kib Nullable(UInt64),
345+ shared_text_kib Nullable(UInt64), vmem_growth_kib Nullable(Int64),
346+ rss_growth_kib Nullable(Int64), minflt Nullable(UInt64),
347+ majflt Nullable(UInt64), swap_kib Nullable(UInt64),
300348 ingest_version UInt64
301349 ) ENGINE = ReplacingMergeTree(ingest_version) ORDER BY (host, epoch, pid, name)""", "", user, password,
302350 modal_key, modal_secret)
@@ -307,6 +355,24 @@ def write_clickhouse(
307355 "", user, password, modal_key, modal_secret)
308356 clickhouse_request(url, f"ALTER TABLE {processes_target} ADD COLUMN IF NOT EXISTS ingest_version UInt64 DEFAULT 0",
309357 "", user, password, modal_key, modal_secret)
358+ for target, definition in (
359+ (samples_target, "mem_total_kib UInt64 DEFAULT 0"),
360+ (samples_target, "mem_free_kib UInt64 DEFAULT 0"),
361+ (samples_target, "mem_available_kib UInt64 DEFAULT 0"),
362+ (samples_target, "mem_cache_kib UInt64 DEFAULT 0"),
363+ (samples_target, "mem_buffer_kib UInt64 DEFAULT 0"),
364+ (samples_target, "mem_slab_kib UInt64 DEFAULT 0"),
365+ (processes_target, "vmem_kib Nullable(UInt64)"),
366+ (processes_target, "rss_kib Nullable(UInt64)"),
367+ (processes_target, "shared_text_kib Nullable(UInt64)"),
368+ (processes_target, "vmem_growth_kib Nullable(Int64)"),
369+ (processes_target, "rss_growth_kib Nullable(Int64)"),
370+ (processes_target, "minflt Nullable(UInt64)"),
371+ (processes_target, "majflt Nullable(UInt64)"),
372+ (processes_target, "swap_kib Nullable(UInt64)"),
373+ ):
374+ clickhouse_request(url, f"ALTER TABLE {target} ADD COLUMN IF NOT EXISTS {definition}",
375+ "", user, password, modal_key, modal_secret)
310376
311377 sample_rows = [
312378 {
@@ -317,16 +383,30 @@ def write_clickhouse(
317383 "softirq_pct": s.pct("softirq"), "steal_pct": s.pct("steal"),
318384 "busy_pct": s.busy_pct, "load1": s.load1, "load5": s.load5,
319385 "load15": s.load15, "ctxsw": s.ctxsw, "intr": s.intr,
386+ "mem_total_kib": s.mem_total_kib, "mem_free_kib": s.mem_free_kib,
387+ "mem_available_kib": s.mem_available_kib, "mem_cache_kib": s.mem_cache_kib,
388+ "mem_buffer_kib": s.mem_buffer_kib, "mem_slab_kib": s.mem_slab_kib,
320389 }
321390 for s in samples
322391 ]
323392 for row in sample_rows:
324393 row["ingest_version"] = ingest_version(row)
325- process_rows = [
326- {"host": s.host, "epoch": s.epoch, "pid": pid, "name": name,
327- "cpu_pct": pct, "interval_seconds": s.interval}
328- for s in samples for name, pid, pct in s.procs
329- ]
394+ process_rows = []
395+ for s in samples:
396+ for name, pid, pct in s.procs:
397+ memory = s.proc_memory.get((pid, name))
398+ process_rows.append({
399+ "host": s.host, "epoch": s.epoch, "pid": pid, "name": name,
400+ "cpu_pct": pct, "interval_seconds": s.interval,
401+ "vmem_kib": memory["vmem_kib"] if memory else None,
402+ "rss_kib": memory["rss_kib"] if memory else None,
403+ "shared_text_kib": memory["shared_text_kib"] if memory else None,
404+ "vmem_growth_kib": memory["vmem_growth_kib"] if memory else None,
405+ "rss_growth_kib": memory["rss_growth_kib"] if memory else None,
406+ "minflt": memory["minflt"] if memory else None,
407+ "majflt": memory["majflt"] if memory else None,
408+ "swap_kib": memory["swap_kib"] if memory else None,
409+ })
330410 for row in process_rows:
331411 row["ingest_version"] = ingest_version(row)
332412 clickhouse_request(url, f"INSERT INTO {samples_target} FORMAT JSONEachRow",
@@ -500,7 +580,7 @@ def main() -> int:
500580 p.add_argument("--clickhouse-modal-secret", default=os.environ.get("MODAL_SECRET"),
501581 help="Modal Proxy Token secret for a protected Modal endpoint (default: $MODAL_SECRET)")
502582 p.add_argument("--clickhouse-init", action="store_true",
503- help="create the ClickHouse database and tables if missing")
583+ help="create missing ClickHouse schema and apply additive upgrades")
504584 args = p.parse_args()
505585
506586 try:
@@ -48,7 +48,7 @@ IDENTIFIER_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
48 logger = logging.getLogger(__name__)48 logger = logging.getLogger(__name__)
49 logger.setLevel(logging.INFO)49 logger.setLevel(logging.INFO)
50 50
51-WANTED_LABELS = {"CPU", "CPL", "PRC"}51+WANTED_LABELS = {"CPU", "CPL", "MEM", "PRC", "PRM"}
52 DATE_RE = re.compile(r"^\d{4}/\d{2}/\d{2}$")52 DATE_RE = re.compile(r"^\d{4}/\d{2}/\d{2}$")
53 53
54 # Offsets into the label-specific fields (i.e. after the 6 common header fields).54 # Offsets into the label-specific fields (i.e. after the 6 common header fields).
@@ -71,6 +71,17 @@ CPL_LOAD1, CPL_LOAD5, CPL_LOAD15, CPL_CTXSW, CPL_INTR = 1, 2, 3, 4, 5
71 PRC_PID, PRC_NAME, PRC_STATE, PRC_HERTZ, PRC_UTIME, PRC_STIME = 0, 1, 2, 3, 4, 571 PRC_PID, PRC_NAME, PRC_STATE, PRC_HERTZ, PRC_UTIME, PRC_STIME = 0, 1, 2, 3, 4, 5
72 PRC_ISPROC = 1372 PRC_ISPROC = 13
73 73
74+# MEM: pagesize physmem freemem cachemem buffermem slabmem ... availablemem
75+# Memory quantities emitted by atop are KiB, apart from pagesize.
76+MEM_TOTAL, MEM_FREE, MEM_CACHE, MEM_BUFFER, MEM_SLAB, MEM_AVAILABLE = 1, 2, 3, 4, 5, 24
77+
78+# PRM: pid name state pagesize vmem rss sharedtext vgrow rgrow minflt majflt
79+# ... swap ... tgid is_process
80+# Like MEM, memory quantities are KiB. The growth counters are signed.
81+PRM_PID, PRM_NAME, PRM_VMEM, PRM_RSS, PRM_SHARED_TEXT = 0, 1, 4, 5, 6
82+PRM_VMEM_GROWTH, PRM_RSS_GROWTH, PRM_MINFLT, PRM_MAJFLT = 7, 8, 9, 10
83+PRM_SWAP, PRM_ISPROC = 14, 16
84+
74 85
75 @dataclass86 @dataclass
76 class Sample:87 class Sample:
@@ -88,7 +99,14 @@ class Sample:
88 load15: float = 0.099 load15: float = 0.0
89 ctxsw: int = 0100 ctxsw: int = 0
90 intr: int = 0101 intr: int = 0
102+ mem_total_kib: int = 0
103+ mem_free_kib: int = 0
104+ mem_available_kib: int = 0
105+ mem_cache_kib: int = 0
106+ mem_buffer_kib: int = 0
107+ mem_slab_kib: int = 0
91 procs: list[tuple[str, int, float]] = field(default_factory=list)108 procs: list[tuple[str, int, float]] = field(default_factory=list)
109+ proc_memory: dict[tuple[int, str], dict[str, int]] = field(default_factory=dict)
92 110
93 @property111 @property
94 def total_ticks(self) -> int:112 def total_ticks(self) -> int:
@@ -122,7 +140,7 @@ def log_path(date: str | None, logdir: str) -> str:
122 def run_atop(path: str, begin: str | None, end: str | None) -> str:140 def run_atop(path: str, begin: str | None, end: str | None) -> str:
123 if not shutil.which("atop"):141 if not shutil.which("atop"):
124 raise AtopError("atop is not installed (pacman -S atop)")142 raise AtopError("atop is not installed (pacman -S atop)")
125- cmd = ["atop", "-r", path, "-Z", "-P", "CPU,CPL,PRC"]143+ cmd = ["atop", "-r", path, "-Z", "-P", "CPU,CPL,MEM,PRC,PRM"]
126 if begin:144 if begin:
127 cmd += ["-b", begin]145 cmd += ["-b", begin]
128 if end:146 if end:
@@ -188,6 +206,13 @@ def parse(raw: str) -> list[Sample]:
188 current.load15 = float(rest[CPL_LOAD15])206 current.load15 = float(rest[CPL_LOAD15])
189 current.ctxsw = int(rest[CPL_CTXSW])207 current.ctxsw = int(rest[CPL_CTXSW])
190 current.intr = int(rest[CPL_INTR])208 current.intr = int(rest[CPL_INTR])
209+ elif label == "MEM":
210+ current.mem_total_kib = int(rest[MEM_TOTAL])
211+ current.mem_free_kib = int(rest[MEM_FREE])
212+ current.mem_available_kib = int(rest[MEM_AVAILABLE])
213+ current.mem_cache_kib = int(rest[MEM_CACHE])
214+ current.mem_buffer_kib = int(rest[MEM_BUFFER])
215+ current.mem_slab_kib = int(rest[MEM_SLAB])
191 elif label == "PRC":216 elif label == "PRC":
192 if rest[PRC_ISPROC] != "y":217 if rest[PRC_ISPROC] != "y":
193 continue # thread line, not a process218 continue # thread line, not a process
@@ -197,8 +222,24 @@ def parse(raw: str) -> list[Sample]:
197 capacity = hertz * interval * max(current.ncpu or 1, 1)222 capacity = hertz * interval * max(current.ncpu or 1, 1)
198 pct = 100.0 * ticks / capacity if capacity else 0.0223 pct = 100.0 * ticks / capacity if capacity else 0.0
199 current.procs.append((rest[PRC_NAME], int(rest[PRC_PID]), pct))224 current.procs.append((rest[PRC_NAME], int(rest[PRC_PID]), pct))
225+ elif label == "PRM":
226+ if rest[PRM_ISPROC] != "y":
227+ continue # thread line, not a process
228+ current.proc_memory[(int(rest[PRM_PID]), rest[PRM_NAME])] = {
229+ "vmem_kib": int(rest[PRM_VMEM]),
230+ "rss_kib": int(rest[PRM_RSS]),
231+ "shared_text_kib": int(rest[PRM_SHARED_TEXT]),
232+ "vmem_growth_kib": int(rest[PRM_VMEM_GROWTH]),
233+ "rss_growth_kib": int(rest[PRM_RSS_GROWTH]),
234+ "minflt": int(rest[PRM_MINFLT]),
235+ "majflt": int(rest[PRM_MAJFLT]),
236+ "swap_kib": int(rest[PRM_SWAP]),
237+ }
200 except (IndexError, ValueError):238 except (IndexError, ValueError):
201- continue # a label whose layout this atop version changed239+ raise AtopError(
240+ f"cannot parse {label} output from this atop version; "
241+ "the required parsable field layout is missing or invalid"
242+ )
202 243
203 if current and not skip_this:244 if current and not skip_this:
204 samples.append(current)245 samples.append(current)
@@ -291,12 +332,19 @@ def write_clickhouse(
291 ncpu UInt16, user_pct Float32, sys_pct Float32, nice_pct Float32,332 ncpu UInt16, user_pct Float32, sys_pct Float32, nice_pct Float32,
292 idle_pct Float32, wait_pct Float32, irq_pct Float32, softirq_pct Float32,333 idle_pct Float32, wait_pct Float32, irq_pct Float32, softirq_pct Float32,
293 steal_pct Float32, busy_pct Float32, load1 Float32, load5 Float32,334 steal_pct Float32, busy_pct Float32, load1 Float32, load5 Float32,
294- load15 Float32, ctxsw UInt64, intr UInt64, ingest_version UInt64335+ load15 Float32, ctxsw UInt64, intr UInt64,
336+ mem_total_kib UInt64, mem_free_kib UInt64, mem_available_kib UInt64,
337+ mem_cache_kib UInt64, mem_buffer_kib UInt64, mem_slab_kib UInt64,
338+ ingest_version UInt64
295 ) ENGINE = ReplacingMergeTree(ingest_version) ORDER BY (host, epoch)""", "", user, password,339 ) ENGINE = ReplacingMergeTree(ingest_version) ORDER BY (host, epoch)""", "", user, password,
296 modal_key, modal_secret)340 modal_key, modal_secret)
297 clickhouse_request(url, f"""CREATE TABLE IF NOT EXISTS {processes_target} (341 clickhouse_request(url, f"""CREATE TABLE IF NOT EXISTS {processes_target} (
298 host LowCardinality(String), epoch DateTime, pid UInt32,342 host LowCardinality(String), epoch DateTime, pid UInt32,
299 name LowCardinality(String), cpu_pct Float32, interval_seconds UInt32,343 name LowCardinality(String), cpu_pct Float32, interval_seconds UInt32,
344+ vmem_kib Nullable(UInt64), rss_kib Nullable(UInt64),
345+ shared_text_kib Nullable(UInt64), vmem_growth_kib Nullable(Int64),
346+ rss_growth_kib Nullable(Int64), minflt Nullable(UInt64),
347+ majflt Nullable(UInt64), swap_kib Nullable(UInt64),
300 ingest_version UInt64348 ingest_version UInt64
301 ) ENGINE = ReplacingMergeTree(ingest_version) ORDER BY (host, epoch, pid, name)""", "", user, password,349 ) ENGINE = ReplacingMergeTree(ingest_version) ORDER BY (host, epoch, pid, name)""", "", user, password,
302 modal_key, modal_secret)350 modal_key, modal_secret)
@@ -307,6 +355,24 @@ def write_clickhouse(
307 "", user, password, modal_key, modal_secret)355 "", user, password, modal_key, modal_secret)
308 clickhouse_request(url, f"ALTER TABLE {processes_target} ADD COLUMN IF NOT EXISTS ingest_version UInt64 DEFAULT 0",356 clickhouse_request(url, f"ALTER TABLE {processes_target} ADD COLUMN IF NOT EXISTS ingest_version UInt64 DEFAULT 0",
309 "", user, password, modal_key, modal_secret)357 "", user, password, modal_key, modal_secret)
358+ for target, definition in (
359+ (samples_target, "mem_total_kib UInt64 DEFAULT 0"),
360+ (samples_target, "mem_free_kib UInt64 DEFAULT 0"),
361+ (samples_target, "mem_available_kib UInt64 DEFAULT 0"),
362+ (samples_target, "mem_cache_kib UInt64 DEFAULT 0"),
363+ (samples_target, "mem_buffer_kib UInt64 DEFAULT 0"),
364+ (samples_target, "mem_slab_kib UInt64 DEFAULT 0"),
365+ (processes_target, "vmem_kib Nullable(UInt64)"),
366+ (processes_target, "rss_kib Nullable(UInt64)"),
367+ (processes_target, "shared_text_kib Nullable(UInt64)"),
368+ (processes_target, "vmem_growth_kib Nullable(Int64)"),
369+ (processes_target, "rss_growth_kib Nullable(Int64)"),
370+ (processes_target, "minflt Nullable(UInt64)"),
371+ (processes_target, "majflt Nullable(UInt64)"),
372+ (processes_target, "swap_kib Nullable(UInt64)"),
373+ ):
374+ clickhouse_request(url, f"ALTER TABLE {target} ADD COLUMN IF NOT EXISTS {definition}",
375+ "", user, password, modal_key, modal_secret)
310 376
311 sample_rows = [377 sample_rows = [
312 {378 {
@@ -317,16 +383,30 @@ def write_clickhouse(
317 "softirq_pct": s.pct("softirq"), "steal_pct": s.pct("steal"),383 "softirq_pct": s.pct("softirq"), "steal_pct": s.pct("steal"),
318 "busy_pct": s.busy_pct, "load1": s.load1, "load5": s.load5,384 "busy_pct": s.busy_pct, "load1": s.load1, "load5": s.load5,
319 "load15": s.load15, "ctxsw": s.ctxsw, "intr": s.intr,385 "load15": s.load15, "ctxsw": s.ctxsw, "intr": s.intr,
386+ "mem_total_kib": s.mem_total_kib, "mem_free_kib": s.mem_free_kib,
387+ "mem_available_kib": s.mem_available_kib, "mem_cache_kib": s.mem_cache_kib,
388+ "mem_buffer_kib": s.mem_buffer_kib, "mem_slab_kib": s.mem_slab_kib,
320 }389 }
321 for s in samples390 for s in samples
322 ]391 ]
323 for row in sample_rows:392 for row in sample_rows:
324 row["ingest_version"] = ingest_version(row)393 row["ingest_version"] = ingest_version(row)
325- process_rows = [394+ process_rows = []
326- {"host": s.host, "epoch": s.epoch, "pid": pid, "name": name,395+ for s in samples:
327- "cpu_pct": pct, "interval_seconds": s.interval}396+ for name, pid, pct in s.procs:
328- for s in samples for name, pid, pct in s.procs397+ memory = s.proc_memory.get((pid, name))
329- ]398+ process_rows.append({
399+ "host": s.host, "epoch": s.epoch, "pid": pid, "name": name,
400+ "cpu_pct": pct, "interval_seconds": s.interval,
401+ "vmem_kib": memory["vmem_kib"] if memory else None,
402+ "rss_kib": memory["rss_kib"] if memory else None,
403+ "shared_text_kib": memory["shared_text_kib"] if memory else None,
404+ "vmem_growth_kib": memory["vmem_growth_kib"] if memory else None,
405+ "rss_growth_kib": memory["rss_growth_kib"] if memory else None,
406+ "minflt": memory["minflt"] if memory else None,
407+ "majflt": memory["majflt"] if memory else None,
408+ "swap_kib": memory["swap_kib"] if memory else None,
409+ })
330 for row in process_rows:410 for row in process_rows:
331 row["ingest_version"] = ingest_version(row)411 row["ingest_version"] = ingest_version(row)
332 clickhouse_request(url, f"INSERT INTO {samples_target} FORMAT JSONEachRow",412 clickhouse_request(url, f"INSERT INTO {samples_target} FORMAT JSONEachRow",
@@ -500,7 +580,7 @@ def main() -> int:
500 p.add_argument("--clickhouse-modal-secret", default=os.environ.get("MODAL_SECRET"),580 p.add_argument("--clickhouse-modal-secret", default=os.environ.get("MODAL_SECRET"),
501 help="Modal Proxy Token secret for a protected Modal endpoint (default: $MODAL_SECRET)")581 help="Modal Proxy Token secret for a protected Modal endpoint (default: $MODAL_SECRET)")
502 p.add_argument("--clickhouse-init", action="store_true",582 p.add_argument("--clickhouse-init", action="store_true",
503- help="create the ClickHouse database and tables if missing")583+ help="create missing ClickHouse schema and apply additive upgrades")
504 args = p.parse_args()584 args = p.parse_args()
505 585
506 try:586 try:
modified grafana/provisioning/dashboards/json/atop-overview.json +32 -0
@@ -29,6 +29,38 @@
2929 "targets": [{"editorType": "sql", "format": 1, "queryType": "table", "rawSql": "SELECT name AS process, round(avg(cpu_pct), 2) AS avg_cpu_pct, round(max(cpu_pct), 2) AS peak_cpu_pct FROM (SELECT host, epoch, pid, name, argMax(cpu_pct, ingest_version) AS cpu_pct FROM atop_processes WHERE $__timeFilter(epoch) GROUP BY host, epoch, pid, name) GROUP BY name ORDER BY avg_cpu_pct DESC LIMIT 15", "refId": "A"}],
3030 "title": "Top processes",
3131 "type": "table"
32+ },
33+ {
34+ "datasource": {"type": "grafana-clickhouse-datasource", "uid": "atop-clickhouse"},
35+ "description": "Drag across a CPU spike in the chart above to narrow the dashboard time range. This table shows the five busiest processes in each captured interval, alongside the host's CPU busy percentage.",
36+ "fieldConfig": {"defaults": {}, "overrides": []},
37+ "gridPos": {"h": 10, "w": 24, "x": 0, "y": 18},
38+ "id": 4,
39+ "options": {"showHeader": true},
40+ "targets": [{"editorType": "sql", "format": 1, "queryType": "table", "rawSql": "SELECT time, host, round(host_busy_pct, 2) AS host_busy_pct, pid, process, round(process_cpu_pct, 2) AS process_cpu_pct FROM (SELECT p.epoch AS time, p.host, p.pid, p.name AS process, p.cpu_pct AS process_cpu_pct, s.busy_pct AS host_busy_pct, row_number() OVER (PARTITION BY p.host, p.epoch ORDER BY p.cpu_pct DESC, p.pid) AS process_rank FROM (SELECT host, epoch, pid, name, argMax(cpu_pct, ingest_version) AS cpu_pct FROM atop_processes WHERE $__timeFilter(epoch) GROUP BY host, epoch, pid, name) AS p INNER JOIN (SELECT host, epoch, argMax(busy_pct, ingest_version) AS busy_pct FROM atop_samples WHERE $__timeFilter(epoch) GROUP BY host, epoch) AS s ON p.host = s.host AND p.epoch = s.epoch) WHERE process_rank <= 5 ORDER BY time DESC, host, process_rank LIMIT 500", "refId": "A"}],
41+ "title": "Incident replay: top processes by interval",
42+ "type": "table"
43+ },
44+ {
45+ "datasource": {"type": "grafana-clickhouse-datasource", "uid": "atop-clickhouse"},
46+ "description": "Physical-memory counters from atop. Values are recorded as KiB in ClickHouse and displayed here as bytes.",
47+ "fieldConfig": {"defaults": {"unit": "bytes"}, "overrides": []},
48+ "gridPos": {"h": 9, "w": 12, "x": 0, "y": 28},
49+ "id": 5,
50+ "targets": [{"editorType": "sql", "format": 0, "queryType": "timeseries", "rawSql": "SELECT epoch AS time, argMax(mem_total_kib, ingest_version) * 1024 AS total, argMax(mem_available_kib, ingest_version) * 1024 AS available, argMax(mem_free_kib, ingest_version) * 1024 AS free, argMax(mem_cache_kib, ingest_version) * 1024 AS page_cache, argMax(mem_buffer_kib, ingest_version) * 1024 AS buffers, argMax(mem_slab_kib, ingest_version) * 1024 AS slab FROM atop_samples WHERE $__timeFilter(epoch) GROUP BY host, epoch ORDER BY epoch", "refId": "A"}],
51+ "title": "Host memory",
52+ "type": "timeseries"
53+ },
54+ {
55+ "datasource": {"type": "grafana-clickhouse-datasource", "uid": "atop-clickhouse"},
56+ "description": "Processes ranked by peak resident set size (RSS) in the selected time range. Rows with unavailable process-memory data are excluded.",
57+ "fieldConfig": {"defaults": {}, "overrides": []},
58+ "gridPos": {"h": 9, "w": 12, "x": 12, "y": 28},
59+ "id": 6,
60+ "options": {"showHeader": true},
61+ "targets": [{"editorType": "sql", "format": 1, "queryType": "table", "rawSql": "SELECT name AS process, round(avg(rss_kib) / 1024, 2) AS avg_rss_mib, round(max(rss_kib) / 1024, 2) AS peak_rss_mib, round(max(vmem_kib) / 1024, 2) AS peak_vmem_mib, round(max(swap_kib) / 1024, 2) AS peak_swap_mib FROM (SELECT host, epoch, pid, name, argMax(rss_kib, ingest_version) AS rss_kib, argMax(vmem_kib, ingest_version) AS vmem_kib, argMax(swap_kib, ingest_version) AS swap_kib FROM atop_processes WHERE $__timeFilter(epoch) GROUP BY host, epoch, pid, name) WHERE rss_kib IS NOT NULL GROUP BY name ORDER BY peak_rss_mib DESC LIMIT 15", "refId": "A"}],
62+ "title": "Top processes by RSS",
63+ "type": "table"
3264 }
3365 ],
3466 "schemaVersion": 39,
@@ -29,6 +29,38 @@
29 "targets": [{"editorType": "sql", "format": 1, "queryType": "table", "rawSql": "SELECT name AS process, round(avg(cpu_pct), 2) AS avg_cpu_pct, round(max(cpu_pct), 2) AS peak_cpu_pct FROM (SELECT host, epoch, pid, name, argMax(cpu_pct, ingest_version) AS cpu_pct FROM atop_processes WHERE $__timeFilter(epoch) GROUP BY host, epoch, pid, name) GROUP BY name ORDER BY avg_cpu_pct DESC LIMIT 15", "refId": "A"}],29 "targets": [{"editorType": "sql", "format": 1, "queryType": "table", "rawSql": "SELECT name AS process, round(avg(cpu_pct), 2) AS avg_cpu_pct, round(max(cpu_pct), 2) AS peak_cpu_pct FROM (SELECT host, epoch, pid, name, argMax(cpu_pct, ingest_version) AS cpu_pct FROM atop_processes WHERE $__timeFilter(epoch) GROUP BY host, epoch, pid, name) GROUP BY name ORDER BY avg_cpu_pct DESC LIMIT 15", "refId": "A"}],
30 "title": "Top processes",30 "title": "Top processes",
31 "type": "table"31 "type": "table"
32+ },
33+ {
34+ "datasource": {"type": "grafana-clickhouse-datasource", "uid": "atop-clickhouse"},
35+ "description": "Drag across a CPU spike in the chart above to narrow the dashboard time range. This table shows the five busiest processes in each captured interval, alongside the host's CPU busy percentage.",
36+ "fieldConfig": {"defaults": {}, "overrides": []},
37+ "gridPos": {"h": 10, "w": 24, "x": 0, "y": 18},
38+ "id": 4,
39+ "options": {"showHeader": true},
40+ "targets": [{"editorType": "sql", "format": 1, "queryType": "table", "rawSql": "SELECT time, host, round(host_busy_pct, 2) AS host_busy_pct, pid, process, round(process_cpu_pct, 2) AS process_cpu_pct FROM (SELECT p.epoch AS time, p.host, p.pid, p.name AS process, p.cpu_pct AS process_cpu_pct, s.busy_pct AS host_busy_pct, row_number() OVER (PARTITION BY p.host, p.epoch ORDER BY p.cpu_pct DESC, p.pid) AS process_rank FROM (SELECT host, epoch, pid, name, argMax(cpu_pct, ingest_version) AS cpu_pct FROM atop_processes WHERE $__timeFilter(epoch) GROUP BY host, epoch, pid, name) AS p INNER JOIN (SELECT host, epoch, argMax(busy_pct, ingest_version) AS busy_pct FROM atop_samples WHERE $__timeFilter(epoch) GROUP BY host, epoch) AS s ON p.host = s.host AND p.epoch = s.epoch) WHERE process_rank <= 5 ORDER BY time DESC, host, process_rank LIMIT 500", "refId": "A"}],
41+ "title": "Incident replay: top processes by interval",
42+ "type": "table"
43+ },
44+ {
45+ "datasource": {"type": "grafana-clickhouse-datasource", "uid": "atop-clickhouse"},
46+ "description": "Physical-memory counters from atop. Values are recorded as KiB in ClickHouse and displayed here as bytes.",
47+ "fieldConfig": {"defaults": {"unit": "bytes"}, "overrides": []},
48+ "gridPos": {"h": 9, "w": 12, "x": 0, "y": 28},
49+ "id": 5,
50+ "targets": [{"editorType": "sql", "format": 0, "queryType": "timeseries", "rawSql": "SELECT epoch AS time, argMax(mem_total_kib, ingest_version) * 1024 AS total, argMax(mem_available_kib, ingest_version) * 1024 AS available, argMax(mem_free_kib, ingest_version) * 1024 AS free, argMax(mem_cache_kib, ingest_version) * 1024 AS page_cache, argMax(mem_buffer_kib, ingest_version) * 1024 AS buffers, argMax(mem_slab_kib, ingest_version) * 1024 AS slab FROM atop_samples WHERE $__timeFilter(epoch) GROUP BY host, epoch ORDER BY epoch", "refId": "A"}],
51+ "title": "Host memory",
52+ "type": "timeseries"
53+ },
54+ {
55+ "datasource": {"type": "grafana-clickhouse-datasource", "uid": "atop-clickhouse"},
56+ "description": "Processes ranked by peak resident set size (RSS) in the selected time range. Rows with unavailable process-memory data are excluded.",
57+ "fieldConfig": {"defaults": {}, "overrides": []},
58+ "gridPos": {"h": 9, "w": 12, "x": 12, "y": 28},
59+ "id": 6,
60+ "options": {"showHeader": true},
61+ "targets": [{"editorType": "sql", "format": 1, "queryType": "table", "rawSql": "SELECT name AS process, round(avg(rss_kib) / 1024, 2) AS avg_rss_mib, round(max(rss_kib) / 1024, 2) AS peak_rss_mib, round(max(vmem_kib) / 1024, 2) AS peak_vmem_mib, round(max(swap_kib) / 1024, 2) AS peak_swap_mib FROM (SELECT host, epoch, pid, name, argMax(rss_kib, ingest_version) AS rss_kib, argMax(vmem_kib, ingest_version) AS vmem_kib, argMax(swap_kib, ingest_version) AS swap_kib FROM atop_processes WHERE $__timeFilter(epoch) GROUP BY host, epoch, pid, name) WHERE rss_kib IS NOT NULL GROUP BY name ORDER BY peak_rss_mib DESC LIMIT 15", "refId": "A"}],
62+ "title": "Top processes by RSS",
63+ "type": "table"
32 }64 }
33 ],65 ],
34 "schemaVersion": 39,66 "schemaVersion": 39,
modified modal_grafana.py +1 -0
@@ -56,6 +56,7 @@ image = (
5656 volumes={"/var/lib/grafana": data_volume},
5757 min_containers=0,
5858 max_containers=1,
59+ nonpreemptible=True,
5960 scaledown_window=IDLE_SECONDS,
6061 timeout=15 * 60,
6162 env={
@@ -56,6 +56,7 @@ image = (
56 volumes={"/var/lib/grafana": data_volume},56 volumes={"/var/lib/grafana": data_volume},
57 min_containers=0,57 min_containers=0,
58 max_containers=1,58 max_containers=1,
59+ nonpreemptible=True,
59 scaledown_window=IDLE_SECONDS,60 scaledown_window=IDLE_SECONDS,
60 timeout=15 * 60,61 timeout=15 * 60,
61 env={62 env={