rickub
nandi/memwatchpublic⑂ Fork 0
⑂ a2ab39b
Branches
main View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/nandi/memwatch.git
git clone ssh://git@rickub.com/nandi/memwatch.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

memwatch: PSI-driven memory stall watchdog

Triggers on /proc/pressure/memory stall time rather than free-memory
percentage, which never fires on a box with RAM-sized zram. Freezes the
offending cgroup before drawing the dialog so the desktop stays paintable,
then kills via cgroup.kill if the user doesn't answer or pressure keeps
climbing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nandi committed 2026-09-20T15:48:50-07:00 Browse files
a2ab39b
added README.md +116 -0
new file mode 100644
@@ -0,0 +1,116 @@
1+# memwatch
2+
3+A memory watchdog that triggers on **stall time**, not on free-memory percentage.
4+It freezes the offender first, then asks, then kills.
5+
6+## Why the obvious design doesn't work on this machine
7+
8+The instinct is "watch `MemAvailable`, kill when it drops below N%". On a box
9+with 32G RAM and 31.2G of zram swap, that never fires:
10+
11+- zram lives *in RAM*. Pages pushed to it are compressed, not evicted.
12+- So `MemAvailable` degrades gently forever instead of hitting a wall.
13+- So the kernel OOM killer never gets an allocation failure to react to.
14+ Confirmed: **zero OOM kills in 30 days** on a machine that freezes regularly.
15+- Meanwhile every task is blocked in page reclaim. That *is* the freeze.
16+
17+The quantity that actually corresponds to "the machine shat the bed" is
18+`/proc/pressure/memory` — PSI, the kernel's measure of time spent stalled.
19+`full avg10 = 40%` means 40% of the last ten seconds, *every* runnable task was
20+stuck waiting on memory. That is a number you can threshold on, and the kernel
21+will push you an event the moment it is crossed.
22+
23+Measured on this box: a hog capped at 2G drove global `full avg10` to 17.9%.
24+A real runaway sits far above that.
25+
26+## The freeze-first trick
27+
28+A dialog asking "kill this?" is worthless if the machine is too wedged to draw
29+it. So on the warn tier memwatchd does **not** show a dialog first. It:
30+
31+1. Writes `1` to the offending cgroup's `cgroup.freeze`. The entire process
32+ tree halts atomically — nothing forks away, nothing escapes.
33+2. Allocation rate instantly goes to zero, reclaim catches up, pressure drops.
34+ *Measured: `full avg10` fell 39.1% → 26.6% within 4 seconds of freezing.*
35+3. Now the compositor gets scheduled again, and the dialog paints.
36+4. You pick **Kill it** or **Resume it**. Resume thaws it and walks away.
37+
38+If you don't answer in 20s, it kills. If pressure keeps climbing *even with the
39+top consumer frozen* — meaning we picked wrong, or it's systemic — it stops
40+waiting on you and kills immediately.
41+
42+## Tiers
43+
44+| tier | condition | action |
45+|---|---|---|
46+| warn | kernel PSI trigger: `full` stall ≥ 100ms in any 1s window | freeze top consumer, show dialog |
47+| kill | `full avg10` ≥ 25% held 4 consecutive seconds | kill, no dialog |
48+| — | 15s cooldown after any action | prevents kill storms |
49+
50+The 4-second hold is what answers "it tends to spike": a build, a big paste, or
51+a page load will blow through 25% for a second or two and must be ignored. Four
52+straight seconds of total stall is not a spike, it's a machine going down.
53+
54+## Safety
55+
56+Victim selection is **structural, not name-based**, because name-based is one
57+bad guess away from logging you out. Permanently ineligible:
58+
59+- `session-N.scope` — the whole login session. Killing it is a logout. (This
60+ was a real bug caught in testing: `niri` at 808MB was ranked 4th and *would*
61+ have been selectable.)
62+- `init.scope`, `user@N.service` — your systemd user manager.
63+- any `*.slice` — only leaves inside a slice are ever targeted.
64+
65+On top of that, `protect=` in the config is a name blocklist for the compositor,
66+audio stack, dbus and sshd.
67+
68+Processes are aggregated **by cgroup**, not individually — a browser is 40
69+processes in one scope, and both the honest number and the correct kill unit is
70+the scope. `cgroup.kill` reaps the tree atomically, so no orphaned renderers.
71+
72+The daemon protects itself: `mlockall(MCL_CURRENT|MCL_FUTURE)`,
73+`OOMScoreAdjust=-1000`, `ManagedOOMPreference=omit`, `MemorySwapMax=0`,
74+`SCHED_RR` priority 10. If the watchdog has to fault its own pages back in from
75+zram to make a decision, it arrives after the freeze it was meant to prevent.
76+
77+## Install
78+
79+```bash
80+run0 ./install.sh
81+```
82+
83+Ships with `dry_run = 1`. It will log exactly what it *would* freeze and kill
84+and touch nothing. Watch it through a few real pressure events:
85+
86+```bash
87+journalctl -fu memwatch
88+```
89+
90+When the picks look right, set `dry_run = 0` in `/etc/memwatch.conf` and
91+`systemctl restart memwatch`.
92+
93+## Tuning
94+
95+All in `/etc/memwatch.conf`.
96+
97+- Woken during normal heavy builds → raise `warn_stall_us` to `200000` (20%).
98+- Want more time to react → raise `dialog_timeout_s`.
99+- Want it to kill sooner and stop asking → lower `kill_avg10` to `15`.
100+- Want it to never kill unattended → set `kill_avg10 = 100`. The dialog still
101+ appears and the freeze still happens; nothing dies without a click.
102+
103+## Two things to do alongside this
104+
105+1. **`optional/10-user-pressure.conf`** — systemd-oomd is running on this box
106+ but is disarmed for your applications (`user.slice` has
107+ `ManagedOOMMemoryPressure=auto`, which inherits from root, which is off).
108+ Arming it gives you a backstop that survives memwatchd being dead or wrong.
109+ It is slower and dumber and kills without asking; that is the point.
110+
111+2. **`optional/README-zram.md`** — 31.2G of zram on 32G of RAM is the
112+ underlying cause. Shrinking it to `ram / 4` or adding real disk swap makes
113+ every layer here work better.
114+
115+memwatchd does not require either. It triggers on stall, which is the right
116+signal regardless of how swap is set up.
new file mode 100644
@@ -0,0 +1,116 @@
1+# memwatch
2+
3+A memory watchdog that triggers on **stall time**, not on free-memory percentage.
4+It freezes the offender first, then asks, then kills.
5+
6+## Why the obvious design doesn't work on this machine
7+
8+The instinct is "watch `MemAvailable`, kill when it drops below N%". On a box
9+with 32G RAM and 31.2G of zram swap, that never fires:
10+
11+- zram lives *in RAM*. Pages pushed to it are compressed, not evicted.
12+- So `MemAvailable` degrades gently forever instead of hitting a wall.
13+- So the kernel OOM killer never gets an allocation failure to react to.
14+ Confirmed: **zero OOM kills in 30 days** on a machine that freezes regularly.
15+- Meanwhile every task is blocked in page reclaim. That *is* the freeze.
16+
17+The quantity that actually corresponds to "the machine shat the bed" is
18+`/proc/pressure/memory` — PSI, the kernel's measure of time spent stalled.
19+`full avg10 = 40%` means 40% of the last ten seconds, *every* runnable task was
20+stuck waiting on memory. That is a number you can threshold on, and the kernel
21+will push you an event the moment it is crossed.
22+
23+Measured on this box: a hog capped at 2G drove global `full avg10` to 17.9%.
24+A real runaway sits far above that.
25+
26+## The freeze-first trick
27+
28+A dialog asking "kill this?" is worthless if the machine is too wedged to draw
29+it. So on the warn tier memwatchd does **not** show a dialog first. It:
30+
31+1. Writes `1` to the offending cgroup's `cgroup.freeze`. The entire process
32+ tree halts atomically — nothing forks away, nothing escapes.
33+2. Allocation rate instantly goes to zero, reclaim catches up, pressure drops.
34+ *Measured: `full avg10` fell 39.1% → 26.6% within 4 seconds of freezing.*
35+3. Now the compositor gets scheduled again, and the dialog paints.
36+4. You pick **Kill it** or **Resume it**. Resume thaws it and walks away.
37+
38+If you don't answer in 20s, it kills. If pressure keeps climbing *even with the
39+top consumer frozen* — meaning we picked wrong, or it's systemic — it stops
40+waiting on you and kills immediately.
41+
42+## Tiers
43+
44+| tier | condition | action |
45+|---|---|---|
46+| warn | kernel PSI trigger: `full` stall ≥ 100ms in any 1s window | freeze top consumer, show dialog |
47+| kill | `full avg10` ≥ 25% held 4 consecutive seconds | kill, no dialog |
48+| — | 15s cooldown after any action | prevents kill storms |
49+
50+The 4-second hold is what answers "it tends to spike": a build, a big paste, or
51+a page load will blow through 25% for a second or two and must be ignored. Four
52+straight seconds of total stall is not a spike, it's a machine going down.
53+
54+## Safety
55+
56+Victim selection is **structural, not name-based**, because name-based is one
57+bad guess away from logging you out. Permanently ineligible:
58+
59+- `session-N.scope` — the whole login session. Killing it is a logout. (This
60+ was a real bug caught in testing: `niri` at 808MB was ranked 4th and *would*
61+ have been selectable.)
62+- `init.scope`, `user@N.service` — your systemd user manager.
63+- any `*.slice` — only leaves inside a slice are ever targeted.
64+
65+On top of that, `protect=` in the config is a name blocklist for the compositor,
66+audio stack, dbus and sshd.
67+
68+Processes are aggregated **by cgroup**, not individually — a browser is 40
69+processes in one scope, and both the honest number and the correct kill unit is
70+the scope. `cgroup.kill` reaps the tree atomically, so no orphaned renderers.
71+
72+The daemon protects itself: `mlockall(MCL_CURRENT|MCL_FUTURE)`,
73+`OOMScoreAdjust=-1000`, `ManagedOOMPreference=omit`, `MemorySwapMax=0`,
74+`SCHED_RR` priority 10. If the watchdog has to fault its own pages back in from
75+zram to make a decision, it arrives after the freeze it was meant to prevent.
76+
77+## Install
78+
79+```bash
80+run0 ./install.sh
81+```
82+
83+Ships with `dry_run = 1`. It will log exactly what it *would* freeze and kill
84+and touch nothing. Watch it through a few real pressure events:
85+
86+```bash
87+journalctl -fu memwatch
88+```
89+
90+When the picks look right, set `dry_run = 0` in `/etc/memwatch.conf` and
91+`systemctl restart memwatch`.
92+
93+## Tuning
94+
95+All in `/etc/memwatch.conf`.
96+
97+- Woken during normal heavy builds → raise `warn_stall_us` to `200000` (20%).
98+- Want more time to react → raise `dialog_timeout_s`.
99+- Want it to kill sooner and stop asking → lower `kill_avg10` to `15`.
100+- Want it to never kill unattended → set `kill_avg10 = 100`. The dialog still
101+ appears and the freeze still happens; nothing dies without a click.
102+
103+## Two things to do alongside this
104+
105+1. **`optional/10-user-pressure.conf`** — systemd-oomd is running on this box
106+ but is disarmed for your applications (`user.slice` has
107+ `ManagedOOMMemoryPressure=auto`, which inherits from root, which is off).
108+ Arming it gives you a backstop that survives memwatchd being dead or wrong.
109+ It is slower and dumber and kills without asking; that is the point.
110+
111+2. **`optional/README-zram.md`** — 31.2G of zram on 32G of RAM is the
112+ underlying cause. Shrinking it to `ram / 4` or adding real disk swap makes
113+ every layer here work better.
114+
115+memwatchd does not require either. It triggers on stall, which is the right
116+signal regardless of how swap is set up.
added install.sh +23 -0
new file mode 100755
@@ -0,0 +1,23 @@
1+#!/usr/bin/env bash
2+set -euo pipefail
3+[[ $EUID -eq 0 ]] || { echo "run as root: run0 ./install.sh" >&2; exit 1; }
4+src="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
5+
6+install -Dm755 "$src/src/memwatchd.py" /usr/local/lib/memwatch/memwatchd.py
7+install -Dm644 "$src/units/memwatch.service" /etc/systemd/system/memwatch.service
8+install -Dm644 "$src/README.md" /usr/share/doc/memwatch/README.md
9+if [[ -e /etc/memwatch.conf ]]; then
10+ install -Dm644 "$src/memwatch.conf" /etc/memwatch.conf.new
11+ echo "kept your /etc/memwatch.conf; new defaults at /etc/memwatch.conf.new"
12+else
13+ install -Dm644 "$src/memwatch.conf" /etc/memwatch.conf
14+fi
15+
16+systemctl daemon-reload
17+systemctl enable --now memwatch.service
18+echo
19+systemctl --no-pager --lines=15 status memwatch.service || true
20+echo
21+echo "Running in dry_run mode. Watch it with: journalctl -fu memwatch"
22+echo "When the picks look right, set dry_run = 0 in /etc/memwatch.conf"
23+echo "and run: systemctl restart memwatch"
new file mode 100755
@@ -0,0 +1,23 @@
1+#!/usr/bin/env bash
2+set -euo pipefail
3+[[ $EUID -eq 0 ]] || { echo "run as root: run0 ./install.sh" >&2; exit 1; }
4+src="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
5+
6+install -Dm755 "$src/src/memwatchd.py" /usr/local/lib/memwatch/memwatchd.py
7+install -Dm644 "$src/units/memwatch.service" /etc/systemd/system/memwatch.service
8+install -Dm644 "$src/README.md" /usr/share/doc/memwatch/README.md
9+if [[ -e /etc/memwatch.conf ]]; then
10+ install -Dm644 "$src/memwatch.conf" /etc/memwatch.conf.new
11+ echo "kept your /etc/memwatch.conf; new defaults at /etc/memwatch.conf.new"
12+else
13+ install -Dm644 "$src/memwatch.conf" /etc/memwatch.conf
14+fi
15+
16+systemctl daemon-reload
17+systemctl enable --now memwatch.service
18+echo
19+systemctl --no-pager --lines=15 status memwatch.service || true
20+echo
21+echo "Running in dry_run mode. Watch it with: journalctl -fu memwatch"
22+echo "When the picks look right, set dry_run = 0 in /etc/memwatch.conf"
23+echo "and run: systemctl restart memwatch"
added memwatch.conf +32 -0
new file mode 100644
@@ -0,0 +1,32 @@
1+# /etc/memwatch.conf -- see README for the reasoning behind each number.
2+
3+# --- when to wake up --------------------------------------------------------
4+# Kernel PSI trigger. Fires when *every* runnable task was stalled on memory
5+# for warn_stall_us out of each warn_window_us. 100ms/1s == 10% of wall time
6+# lost to reclaim, which is roughly where a desktop starts feeling sticky.
7+# Raise warn_stall_us if you get woken during normal heavy builds.
8+warn_stall_us = 100000
9+warn_window_us = 1000000
10+
11+# --- when to stop asking and just kill --------------------------------------
12+# full avg10, as a percentage, sustained for kill_hold_s consecutive seconds.
13+# 25% held for 4s is not a spike, it is a machine going down.
14+kill_avg10 = 25.0
15+kill_hold_s = 4
16+
17+# How long the dialog waits for you. Timing out means kill.
18+dialog_timeout_s = 20
19+
20+# Don't act twice inside this window.
21+cooldown_s = 15
22+
23+# --- who is eligible --------------------------------------------------------
24+min_victim_mb = 300
25+scan_slices = /user.slice
26+protect = niri,cosmic-comp,cosmic-session,cosmic-greeter,cosmic-panel,cosmic-bg,systemd,dbus-daemon,dbus-broker,pipewire,wireplumber,sshd,memwatchd,Xwayland
27+
28+# Leave blank to autodetect the owner of the graphical session.
29+notify_user =
30+
31+# Log what it *would* do and touch nothing. Start here.
32+dry_run = 1
new file mode 100644
@@ -0,0 +1,32 @@
1+# /etc/memwatch.conf -- see README for the reasoning behind each number.
2+
3+# --- when to wake up --------------------------------------------------------
4+# Kernel PSI trigger. Fires when *every* runnable task was stalled on memory
5+# for warn_stall_us out of each warn_window_us. 100ms/1s == 10% of wall time
6+# lost to reclaim, which is roughly where a desktop starts feeling sticky.
7+# Raise warn_stall_us if you get woken during normal heavy builds.
8+warn_stall_us = 100000
9+warn_window_us = 1000000
10+
11+# --- when to stop asking and just kill --------------------------------------
12+# full avg10, as a percentage, sustained for kill_hold_s consecutive seconds.
13+# 25% held for 4s is not a spike, it is a machine going down.
14+kill_avg10 = 25.0
15+kill_hold_s = 4
16+
17+# How long the dialog waits for you. Timing out means kill.
18+dialog_timeout_s = 20
19+
20+# Don't act twice inside this window.
21+cooldown_s = 15
22+
23+# --- who is eligible --------------------------------------------------------
24+min_victim_mb = 300
25+scan_slices = /user.slice
26+protect = niri,cosmic-comp,cosmic-session,cosmic-greeter,cosmic-panel,cosmic-bg,systemd,dbus-daemon,dbus-broker,pipewire,wireplumber,sshd,memwatchd,Xwayland
27+
28+# Leave blank to autodetect the owner of the graphical session.
29+notify_user =
30+
31+# Log what it *would* do and touch nothing. Start here.
32+dry_run = 1
added optional/10-user-pressure.conf +12 -0
new file mode 100644
@@ -0,0 +1,12 @@
1+# Drop-in for user.slice: /etc/systemd/system/user.slice.d/10-user-pressure.conf
2+#
3+# Out of the box user.slice has ManagedOOMMemoryPressure=auto, which inherits
4+# from the root slice, which is off. The result is that systemd-oomd watches
5+# system.slice and completely ignores every application you actually run.
6+#
7+# This arms it at 60% pressure sustained over oomd's default 30s window. It is
8+# deliberately slower and dumber than memwatchd -- it is the backstop for the
9+# case where memwatchd is dead, wedged, or wrong. It kills without asking.
10+[Slice]
11+ManagedOOMMemoryPressure=kill
12+ManagedOOMMemoryPressureLimit=60%
new file mode 100644
@@ -0,0 +1,12 @@
1+# Drop-in for user.slice: /etc/systemd/system/user.slice.d/10-user-pressure.conf
2+#
3+# Out of the box user.slice has ManagedOOMMemoryPressure=auto, which inherits
4+# from the root slice, which is off. The result is that systemd-oomd watches
5+# system.slice and completely ignores every application you actually run.
6+#
7+# This arms it at 60% pressure sustained over oomd's default 30s window. It is
8+# deliberately slower and dumber than memwatchd -- it is the backstop for the
9+# case where memwatchd is dead, wedged, or wrong. It kills without asking.
10+[Slice]
11+ManagedOOMMemoryPressure=kill
12+ManagedOOMMemoryPressureLimit=60%
added optional/README-zram.md +32 -0
new file mode 100644
@@ -0,0 +1,32 @@
1+# The zram sizing problem
2+
3+ NAME TYPE SIZE USED PRIO
4+ /dev/zram0 partition 31.2G 0B 100
5+
6+31.2G of zram on a 32G machine. zram is *not* swap in the usual sense -- it is
7+a compressed block device that lives in RAM. Pages "swapped" to it are still
8+occupying physical memory, just compressed (typically 2-3x with zstd).
9+
10+Sizing it at 100% of RAM means the kernel believes it has 64G of address space
11+to hand out, while physically it has 32G. Under real pressure it will keep
12+promising memory, keep compressing, and keep spending CPU on reclaim, and the
13+allocation failure that would trigger the OOM killer never arrives. You get an
14+unbounded thrash instead of a kill. This matches the evidence: zero OOM kills
15+in 30 days on a machine that reportedly freezes.
16+
17+Two ways out, pick one:
18+
19+1. Shrink zram to 25-50% of RAM (8-16G here). Compressed pages still help, but
20+ the kernel's accounting stays closer to physical truth.
21+ # /etc/systemd/zram-generator.conf
22+ [zram0]
23+ zram-size = ram / 4
24+ compression-algorithm = zstd
25+
26+2. Add a real disk swap file *in addition*, at lower priority than zram. Cold
27+ pages get a place to actually leave RAM, which restores the reclaim
28+ gradient the kernel needs.
29+
30+Either makes memwatchd's job easier. Neither is required for it to work --
31+memwatchd triggers on stall time, which is correct regardless of how swap is
32+configured.
new file mode 100644
@@ -0,0 +1,32 @@
1+# The zram sizing problem
2+
3+ NAME TYPE SIZE USED PRIO
4+ /dev/zram0 partition 31.2G 0B 100
5+
6+31.2G of zram on a 32G machine. zram is *not* swap in the usual sense -- it is
7+a compressed block device that lives in RAM. Pages "swapped" to it are still
8+occupying physical memory, just compressed (typically 2-3x with zstd).
9+
10+Sizing it at 100% of RAM means the kernel believes it has 64G of address space
11+to hand out, while physically it has 32G. Under real pressure it will keep
12+promising memory, keep compressing, and keep spending CPU on reclaim, and the
13+allocation failure that would trigger the OOM killer never arrives. You get an
14+unbounded thrash instead of a kill. This matches the evidence: zero OOM kills
15+in 30 days on a machine that reportedly freezes.
16+
17+Two ways out, pick one:
18+
19+1. Shrink zram to 25-50% of RAM (8-16G here). Compressed pages still help, but
20+ the kernel's accounting stays closer to physical truth.
21+ # /etc/systemd/zram-generator.conf
22+ [zram0]
23+ zram-size = ram / 4
24+ compression-algorithm = zstd
25+
26+2. Add a real disk swap file *in addition*, at lower priority than zram. Cold
27+ pages get a place to actually leave RAM, which restores the reclaim
28+ gradient the kernel needs.
29+
30+Either makes memwatchd's job easier. Neither is required for it to work --
31+memwatchd triggers on stall time, which is correct regardless of how swap is
32+configured.
added src/__pycache__/memwatchd.cpython-314.pyc +0 -0
new file mode 100644
Binary files /dev/null and b/src/__pycache__/memwatchd.cpython-314.pyc differ
new file mode 100644
Binary files /dev/null and b/src/__pycache__/memwatchd.cpython-314.pyc differBinary files /dev/null and b/src/__pycache__/memwatchd.cpython-314.pyc differ
added src/memwatchd.py +566 -0
new file mode 100755
@@ -0,0 +1,566 @@
1+#!/usr/bin/env python3
2+"""
3+memwatchd - a PSI-driven memory stall watchdog.
4+
5+Why PSI and not "free memory %":
6+ On a box with zram swap, MemAvailable never reaches zero -- the kernel keeps
7+ compressing pages into RAM instead of failing an allocation. So the classic
8+ "kill when free < 5%" heuristic never fires, and neither does the kernel OOM
9+ killer. What actually happens to the user is *stall*: every task blocked in
10+ page reclaim. /proc/pressure/memory measures exactly that, in microseconds of
11+ stall per window, and the kernel can push us an event the moment it crosses a
12+ line. That is the signal that corresponds to "the machine shat the bed".
13+
14+The freeze-first trick:
15+ A GUI that asks "kill this?" is useless if the machine is too wedged to paint
16+ it. So on the warn tier we SIGSTOP (cgroup.freeze) the top offender *first*.
17+ Its allocation rate instantly goes to zero, pressure drops, the desktop comes
18+ back, and only then do we draw the dialog. The user picks Kill or Resume.
19+ If the dialog times out or pressure keeps climbing anyway, we kill.
20+"""
21+
22+import ctypes
23+import errno
24+import os
25+import re
26+import select
27+import signal
28+import subprocess
29+import sys
30+import threading
31+import time
32+
33+CONFIG_PATH = "/etc/memwatch.conf"
34+PSI_PATH = "/proc/pressure/memory"
35+CGROUP_ROOT = "/sys/fs/cgroup"
36+
37+DEFAULTS = {
38+ # PSI kernel trigger: fire when "full" stall exceeds this many us per window.
39+ # 100ms out of 1s == 10% of wall time with *every* task stalled on memory.
40+ "warn_stall_us": 100000,
41+ "warn_window_us": 1000000,
42+ # Escalate to an unattended kill at this sustained full-avg10 percentage.
43+ "kill_avg10": 25.0,
44+ # ...held for this many consecutive 1s samples. Spikes shorter than this are
45+ # the normal cost of doing business (a build, a big paste) and are ignored.
46+ "kill_hold_s": 4,
47+ # How long the GUI waits for a human before it decides on its own.
48+ "dialog_timeout_s": 20,
49+ # Don't act again for this long after acting. Stops kill storms.
50+ "cooldown_s": 15,
51+ # Ignore anything smaller than this; killing a 40MB process helps nobody.
52+ "min_victim_mb": 300,
53+ # Never select these as victims (substring match on comm / unit name).
54+ "protect": "niri,cosmic-comp,cosmic-session,cosmic-greeter,cosmic-panel,gdm,sddm,systemd,"
55+ "dbus-daemon,dbus-broker,pipewire,wireplumber,sshd,memwatchd,"
56+ "Xwayland,gnome-shell,kwin_wayland,plasmashell",
57+ # Which cgroup subtrees are eligible. Keeps us out of system.slice by default.
58+ "scan_slices": "/user.slice",
59+ "notify_user": "", # empty = autodetect the graphical session owner
60+ "dry_run": "0",
61+}
62+
63+MCL_CURRENT, MCL_FUTURE = 1, 2
64+
65+
66+def log(msg):
67+ sys.stdout.write(msg + "\n")
68+ sys.stdout.flush()
69+
70+
71+def load_config():
72+ cfg = dict(DEFAULTS)
73+ try:
74+ with open(CONFIG_PATH) as fh:
75+ for line in fh:
76+ line = line.split("#", 1)[0].strip()
77+ if not line or "=" not in line:
78+ continue
79+ k, v = line.split("=", 1)
80+ k, v = k.strip(), v.strip()
81+ if k in cfg:
82+ cfg[k] = v
83+ else:
84+ log(f"config: ignoring unknown key {k!r}")
85+ except FileNotFoundError:
86+ log(f"config: {CONFIG_PATH} absent, using defaults")
87+ for k, proto in DEFAULTS.items():
88+ if isinstance(proto, float):
89+ cfg[k] = float(cfg[k])
90+ elif isinstance(proto, int):
91+ cfg[k] = int(cfg[k])
92+ cfg["protect"] = [p.strip() for p in str(cfg["protect"]).split(",") if p.strip()]
93+ cfg["scan_slices"] = [s.strip() for s in str(cfg["scan_slices"]).split(",") if s.strip()]
94+ cfg["dry_run"] = str(cfg["dry_run"]).lower() in ("1", "true", "yes", "on")
95+ return cfg
96+
97+
98+def pin_self():
99+ """Stay resident and runnable while everything else is stalled.
100+
101+ If the watchdog itself has to fault pages back in from zram to make a
102+ decision, it arrives after the freeze it was supposed to prevent.
103+ """
104+ try:
105+ libc = ctypes.CDLL("libc.so.6", use_errno=True)
106+ if libc.mlockall(MCL_CURRENT | MCL_FUTURE) != 0:
107+ log(f"mlockall failed: {os.strerror(ctypes.get_errno())}")
108+ else:
109+ log("mlockall: resident")
110+ except Exception as exc:
111+ log(f"mlockall unavailable: {exc}")
112+ try:
113+ with open(f"/proc/self/oom_score_adj", "w") as fh:
114+ fh.write("-1000")
115+ except OSError as exc:
116+ log(f"oom_score_adj failed: {exc}")
117+ try:
118+ os.sched_setscheduler(0, os.SCHED_RR, os.sched_param(10))
119+ log("sched: SCHED_RR/10")
120+ except (OSError, AttributeError) as exc:
121+ log(f"realtime priority unavailable: {exc}")
122+
123+
124+# --- PSI --------------------------------------------------------------------
125+
126+def read_psi():
127+ """Return (some_avg10, full_avg10) as percentages."""
128+ some = full = 0.0
129+ try:
130+ with open(PSI_PATH) as fh:
131+ for line in fh:
132+ parts = line.split()
133+ if not parts:
134+ continue
135+ vals = dict(p.split("=", 1) for p in parts[1:] if "=" in p)
136+ if parts[0] == "some":
137+ some = float(vals.get("avg10", 0))
138+ elif parts[0] == "full":
139+ full = float(vals.get("avg10", 0))
140+ except OSError:
141+ pass
142+ return some, full
143+
144+
145+def open_psi_trigger(stall_us, window_us):
146+ """Register a kernel PSI trigger and return an fd that polls PRI on breach.
147+
148+ This is the difference between reacting in ~1 second and reacting after
149+ avg10 has had ten seconds to catch up -- by which point you are already
150+ looking at a frozen cursor.
151+ """
152+ fd = os.open(PSI_PATH, os.O_RDWR | os.O_NONBLOCK)
153+ os.write(fd, f"full {stall_us} {window_us}".encode())
154+ return fd
155+
156+
157+# --- process / cgroup inspection -------------------------------------------
158+
159+_MEM_RE = re.compile(rb"^(VmRSS|VmSwap):\s+(\d+) kB", re.M)
160+
161+
162+class Victim:
163+ __slots__ = ("pid", "comm", "rss_kb", "swap_kb", "cgroup", "unit", "_top_kb", "nproc")
164+
165+ def __init__(self, pid, comm, rss_kb, swap_kb, cgroup, unit):
166+ self.pid, self.comm = pid, comm
167+ self.rss_kb, self.swap_kb = rss_kb, swap_kb
168+ self.cgroup, self.unit = cgroup, unit
169+ self._top_kb = rss_kb
170+ self.nproc = 1
171+
172+ @property
173+ def total_kb(self):
174+ return self.rss_kb + self.swap_kb
175+
176+ @property
177+ def total_mb(self):
178+ return self.total_kb // 1024
179+
180+ def __repr__(self):
181+ return (f"<{self.comm} pid={self.pid} {self.total_mb}MB "
182+ f"nproc={self.nproc} unit={self.unit}>")
183+
184+
185+# Structural exclusions. Name-based protection is not enough: these cgroups
186+# are fatal to kill regardless of what happens to be the fattest process
187+# inside them at the time.
188+# session-N.scope -- the whole login session; killing it is a logout
189+# init.scope -- the user's systemd manager
190+# user@N.service -- ditto, the manager unit itself
191+# Only leaf app scopes and user services are ever eligible.
192+def eligible_cgroup(cg, unit):
193+ if unit.startswith("session-") and unit.endswith(".scope"):
194+ return False
195+ if unit in ("init.scope", "user.slice", "-.slice"):
196+ return False
197+ if unit.startswith("user@") and unit.endswith(".service"):
198+ return False
199+ if unit.endswith(".slice"):
200+ return False # never target a slice, only the leaves inside one
201+ return unit.endswith(".scope") or unit.endswith(".service")
202+
203+
204+def proc_cgroup(pid):
205+ try:
206+ with open(f"/proc/{pid}/cgroup", "rb") as fh:
207+ for line in fh:
208+ if line.startswith(b"0::"):
209+ return line[3:].strip().decode("utf-8", "replace")
210+ except OSError:
211+ pass
212+ return ""
213+
214+
215+def scan_candidates(cfg):
216+ """Walk /proc once and return eligible victims, largest first.
217+
218+ We aggregate by cgroup: a browser is 40 processes in one scope, and the
219+ interesting number is what the *scope* costs, not what its biggest renderer
220+ costs. Killing the scope is also the only way to not leave orphans behind.
221+ """
222+ by_cgroup = {}
223+ self_pid = os.getpid()
224+ for entry in os.listdir("/proc"):
225+ if not entry.isdigit():
226+ continue
227+ pid = int(entry)
228+ if pid == self_pid or pid == 1:
229+ continue
230+ try:
231+ with open(f"/proc/{pid}/status", "rb") as fh:
232+ blob = fh.read(2048)
233+ except OSError:
234+ continue # died mid-scan, or a kernel thread
235+ found = dict(_MEM_RE.findall(blob))
236+ if b"VmRSS" not in found:
237+ continue # kernel thread
238+ rss = int(found[b"VmRSS"])
239+ swap = int(found.get(b"VmSwap", b"0"))
240+ nl = blob.find(b"\n")
241+ comm = blob[6:nl].decode("utf-8", "replace").strip() if blob.startswith(b"Name:") else ""
242+ cg = proc_cgroup(pid)
243+ if not cg or not any(cg.startswith(s) for s in cfg["scan_slices"]):
244+ continue
245+ unit = cg.rsplit("/", 1)[-1]
246+ if not eligible_cgroup(cg, unit):
247+ continue
248+ if any(p in comm or p in unit for p in cfg["protect"]):
249+ continue
250+ agg = by_cgroup.get(cg)
251+ if agg is None:
252+ by_cgroup[cg] = Victim(pid, comm, rss, swap, cg, unit)
253+ else:
254+ agg.rss_kb += rss
255+ agg.swap_kb += swap
256+ agg.nproc += 1
257+ # The cgroup gets named after its fattest process -- for a browser
258+ # that is the tab actually eating the box, which is what you want
259+ # to read on the dialog.
260+ if rss > agg._top_kb:
261+ agg._top_kb, agg.comm, agg.pid = rss, comm, pid
262+ out = [v for v in by_cgroup.values() if v.total_mb >= cfg["min_victim_mb"]]
263+ out.sort(key=lambda v: v.total_kb, reverse=True)
264+ return out
265+
266+
267+# --- acting -----------------------------------------------------------------
268+
269+def cg_path(victim, leaf):
270+ return os.path.join(CGROUP_ROOT, victim.cgroup.lstrip("/"), leaf)
271+
272+
273+def freeze(victim, on=True):
274+ """Halt the whole cgroup atomically. Nothing escapes, nothing forks away."""
275+ path = cg_path(victim, "cgroup.freeze")
276+ try:
277+ with open(path, "w") as fh:
278+ fh.write("1" if on else "0")
279+ return True
280+ except OSError as exc:
281+ log(f"freeze({victim.unit}, {on}) failed: {exc}; falling back to signals")
282+ sig = signal.SIGSTOP if on else signal.SIGCONT
283+ return signal_tree(victim, sig)
284+
285+
286+def signal_tree(victim, sig):
287+ ok = False
288+ try:
289+ with open(cg_path(victim, "cgroup.procs")) as fh:
290+ pids = [int(x) for x in fh.read().split()]
291+ except OSError:
292+ pids = [victim.pid]
293+ for pid in pids:
294+ try:
295+ os.kill(pid, sig)
296+ ok = True
297+ except OSError as exc:
298+ if exc.errno != errno.ESRCH:
299+ log(f"kill({pid},{sig}) failed: {exc}")
300+ return ok
301+
302+
303+def kill_victim(victim, cfg, reason):
304+ if cfg["dry_run"]:
305+ log(f"DRY-RUN would kill {victim!r} ({reason})")
306+ return True
307+ # cgroup.kill is atomic and reaps the entire tree; a frozen cgroup still
308+ # honours it, so we never need to thaw-then-race the process.
309+ try:
310+ with open(cg_path(victim, "cgroup.kill"), "w") as fh:
311+ fh.write("1")
312+ log(f"KILLED cgroup {victim.unit} ({victim.comm}, {victim.total_mb}MB) -- {reason}")
313+ return True
314+ except OSError as exc:
315+ log(f"cgroup.kill unavailable for {victim.unit}: {exc}; using SIGKILL")
316+ freeze(victim, False)
317+ ok = signal_tree(victim, signal.SIGKILL)
318+ log(f"{'KILLED' if ok else 'FAILED to kill'} {victim!r} -- {reason}")
319+ return ok
320+
321+
322+# --- talking to the human ---------------------------------------------------
323+
324+_GUI_ENV_KEYS = ("WAYLAND_DISPLAY", "DISPLAY", "XDG_RUNTIME_DIR",
325+ "DBUS_SESSION_BUS_ADDRESS", "XAUTHORITY", "XDG_SESSION_TYPE")
326+
327+
328+def find_session(cfg):
329+ """Locate a graphical session and lift its environment.
330+
331+ We run as a system service, which has no idea how to reach a Wayland
332+ compositor. The reliable way is to read the env of a process that is
333+ already inside the session.
334+ """
335+ want_user = cfg["notify_user"] or None
336+ try:
337+ out = subprocess.run(
338+ ["loginctl", "list-sessions", "--no-legend"],
339+ capture_output=True, text=True, timeout=5).stdout
340+ except Exception:
341+ return None
342+ for line in out.splitlines():
343+ f = line.split()
344+ if len(f) < 3:
345+ continue
346+ sid, uid, user = f[0], f[1], f[2]
347+ if want_user and user != want_user:
348+ continue
349+ try:
350+ show = subprocess.run(["loginctl", "show-session", sid,
351+ "-p", "Type", "-p", "Leader", "-p", "State"],
352+ capture_output=True, text=True, timeout=5).stdout
353+ except Exception:
354+ continue
355+ props = dict(l.split("=", 1) for l in show.splitlines() if "=" in l)
356+ if props.get("Type") not in ("wayland", "x11"):
357+ continue
358+ leader = props.get("Leader")
359+ if not leader:
360+ continue
361+ env = harvest_env(int(leader)) or harvest_env_from_uid(int(uid))
362+ if env:
363+ return int(uid), user, env
364+ return None
365+
366+
367+def harvest_env(pid):
368+ try:
369+ with open(f"/proc/{pid}/environ", "rb") as fh:
370+ raw = fh.read()
371+ except OSError:
372+ return None
373+ env = {}
374+ for item in raw.split(b"\0"):
375+ if b"=" not in item:
376+ continue
377+ k, v = item.decode("utf-8", "replace").split("=", 1)
378+ if k in _GUI_ENV_KEYS:
379+ env[k] = v
380+ return env if ("WAYLAND_DISPLAY" in env or "DISPLAY" in env) else None
381+
382+
383+def harvest_env_from_uid(uid):
384+ """Fallback: sweep the user's processes for one that knows the display."""
385+ for entry in os.listdir("/proc"):
386+ if not entry.isdigit():
387+ continue
388+ try:
389+ if os.stat(f"/proc/{entry}").st_uid != uid:
390+ continue
391+ except OSError:
392+ continue
393+ env = harvest_env(int(entry))
394+ if env:
395+ return env
396+ return None
397+
398+
399+def ask_human(victim, cfg, session, psi_full):
400+ """Show the dialog. Returns (kill: bool, reason: str)."""
401+ if session is None:
402+ log("no graphical session found; escalating without asking")
403+ return True, "no graphical session to ask"
404+ uid, user, env = session
405+ env = dict(env)
406+ env.setdefault("XDG_RUNTIME_DIR", f"/run/user/{uid}")
407+ env["HOME"] = os.path.expanduser(f"~{user}")
408+ env["PATH"] = "/usr/local/bin:/usr/bin:/bin"
409+
410+ text = (
411+ f"<b>Memory pressure: {psi_full:.0f}% stalled</b>\n\n"
412+ f"Largest consumer has been <b>paused</b> so the desktop stays usable:\n\n"
413+ f" <b>{victim.comm or victim.unit}</b>\n"
414+ f" {victim.total_mb} MB ({victim.rss_kb//1024} MB RAM + {victim.swap_kb//1024} MB zram)\n"
415+ f" {victim.unit}\n\n"
416+ f"Kill it, or resume it and ride it out?\n"
417+ f"<i>No answer in {cfg['dialog_timeout_s']}s = kill.</i>"
418+ )
419+ cmd = [
420+ "yad", "--title=Memory pressure", "--window-icon=dialog-warning",
421+ "--image=dialog-warning", "--borders=16", "--width=460",
422+ "--text", text, "--text-align=left", "--on-top", "--center",
423+ "--sticky", "--undecorated", "--skip-taskbar", "--no-escape",
424+ f"--timeout={cfg['dialog_timeout_s']}", "--timeout-indicator=bottom",
425+ "--button=Resume it!media-playback-start:1",
426+ "--button=Kill it!process-stop:0",
427+ ]
428+ try:
429+ rc = subprocess.run(
430+ ["setpriv", "--reuid", str(uid), "--regid", str(uid), "--init-groups", "--"] + cmd,
431+ env=env, timeout=cfg["dialog_timeout_s"] + 10).returncode
432+ except FileNotFoundError:
433+ log("yad or setpriv missing; escalating without asking")
434+ return True, "no dialog binary available"
435+ except subprocess.TimeoutExpired:
436+ log("dialog wedged; escalating")
437+ return True, "dialog stopped responding"
438+ # 0 = Kill pressed. 1 = Resume. 70 = our --timeout elapsed.
439+ if rc == 1:
440+ return False, "user chose resume"
441+ if rc == 0:
442+ return True, "user chose kill"
443+ if rc == 70:
444+ return True, f"no answer in {cfg['dialog_timeout_s']}s"
445+ return True, f"dialog dismissed (rc={rc})"
446+
447+
448+def notify(session, summary, body):
449+ if session is None:
450+ return
451+ uid, user, env = session
452+ env = dict(env)
453+ env.setdefault("XDG_RUNTIME_DIR", f"/run/user/{uid}")
454+ try:
455+ subprocess.run(
456+ ["setpriv", "--reuid", str(uid), "--regid", str(uid), "--init-groups", "--",
457+ "notify-send", "-u", "critical", "-i", "dialog-warning", summary, body],
458+ env=env, timeout=10, check=False)
459+ except Exception:
460+ pass
461+
462+
463+# --- main loop --------------------------------------------------------------
464+
465+def handle_pressure(cfg, psi_fd):
466+ """One full incident, from trigger to resolution."""
467+ some, full = read_psi()
468+ log(f"pressure event: some_avg10={some:.1f}% full_avg10={full:.1f}%")
469+
470+ candidates = scan_candidates(cfg)
471+ if not candidates:
472+ log("no eligible victim above min_victim_mb; nothing to do")
473+ return
474+ victim = candidates[0]
475+ log(f"candidates: {candidates[:3]}")
476+
477+ # Freeze first. This is what buys us the ability to draw a dialog at all:
478+ # the offender stops allocating immediately, reclaim catches up, and the
479+ # compositor gets scheduled again.
480+ frozen = freeze(victim, True) if not cfg["dry_run"] else False
481+ log(f"froze {victim.unit}: {frozen}")
482+
483+ session = find_session(cfg)
484+
485+ # Escalation watchdog: if pressure keeps climbing even with the top
486+ # consumer frozen, we picked wrong or it's systemic. Don't wait on a human.
487+ decided = None
488+ held = 0
489+ deadline = time.monotonic() + cfg["dialog_timeout_s"]
490+
491+ result = {}
492+
493+ def asker():
494+ result["kill"], result["why"] = ask_human(victim, cfg, session, full)
495+
496+ t = threading.Thread(target=asker, daemon=True)
497+ t.start()
498+ while t.is_alive() and time.monotonic() < deadline + 5:
499+ time.sleep(1)
500+ _, f_now = read_psi()
501+ if f_now >= cfg["kill_avg10"]:
502+ held += 1
503+ if held >= cfg["kill_hold_s"]:
504+ decided = "escalated: full_avg10 stayed >= %.0f%% for %ds" % (
505+ cfg["kill_avg10"], held)
506+ break
507+ else:
508+ held = 0
509+
510+ if decided is None:
511+ t.join(timeout=2)
512+ if result.get("kill", True):
513+ decided = result.get("why", "dialog never returned")
514+ else:
515+ log(f"user spared {victim.unit}; thawing")
516+ freeze(victim, False)
517+ notify(session, "Resumed", f"{victim.comm or victim.unit} is running again.")
518+ return
519+
520+ kill_victim(victim, cfg, decided)
521+ notify(session, "Killed to recover memory",
522+ f"{victim.comm or victim.unit} ({victim.total_mb} MB) — {decided}")
523+
524+
525+def main():
526+ cfg = load_config()
527+ log(f"memwatchd starting: warn at {cfg['warn_stall_us']}us/{cfg['warn_window_us']}us, "
528+ f"auto-kill at full_avg10>={cfg['kill_avg10']}% held {cfg['kill_hold_s']}s, "
529+ f"dry_run={cfg['dry_run']}")
530+ pin_self()
531+
532+ if not os.path.exists(PSI_PATH):
533+ log("FATAL: no /proc/pressure/memory (kernel needs CONFIG_PSI=y psi=1)")
534+ return 1
535+ try:
536+ psi_fd = open_psi_trigger(cfg["warn_stall_us"], cfg["warn_window_us"])
537+ except OSError as exc:
538+ log(f"FATAL: cannot register PSI trigger: {exc}")
539+ return 1
540+ log("armed; waiting on kernel pressure events")
541+
542+ poller = select.poll()
543+ poller.register(psi_fd, select.POLLPRI)
544+ last_action = 0.0
545+
546+ while True:
547+ events = poller.poll()
548+ for _fd, ev in events:
549+ if ev & select.POLLERR:
550+ log("FATAL: PSI trigger revoked")
551+ return 1
552+ if not (ev & select.POLLPRI):
553+ continue
554+ now = time.monotonic()
555+ if now - last_action < cfg["cooldown_s"]:
556+ continue
557+ last_action = now
558+ try:
559+ handle_pressure(cfg, psi_fd)
560+ except Exception as exc:
561+ log(f"incident handler blew up: {exc!r}")
562+ last_action = time.monotonic()
563+
564+
565+if __name__ == "__main__":
566+ sys.exit(main())
new file mode 100755
@@ -0,0 +1,566 @@
1+#!/usr/bin/env python3
2+"""
3+memwatchd - a PSI-driven memory stall watchdog.
4+
5+Why PSI and not "free memory %":
6+ On a box with zram swap, MemAvailable never reaches zero -- the kernel keeps
7+ compressing pages into RAM instead of failing an allocation. So the classic
8+ "kill when free < 5%" heuristic never fires, and neither does the kernel OOM
9+ killer. What actually happens to the user is *stall*: every task blocked in
10+ page reclaim. /proc/pressure/memory measures exactly that, in microseconds of
11+ stall per window, and the kernel can push us an event the moment it crosses a
12+ line. That is the signal that corresponds to "the machine shat the bed".
13+
14+The freeze-first trick:
15+ A GUI that asks "kill this?" is useless if the machine is too wedged to paint
16+ it. So on the warn tier we SIGSTOP (cgroup.freeze) the top offender *first*.
17+ Its allocation rate instantly goes to zero, pressure drops, the desktop comes
18+ back, and only then do we draw the dialog. The user picks Kill or Resume.
19+ If the dialog times out or pressure keeps climbing anyway, we kill.
20+"""
21+
22+import ctypes
23+import errno
24+import os
25+import re
26+import select
27+import signal
28+import subprocess
29+import sys
30+import threading
31+import time
32+
33+CONFIG_PATH = "/etc/memwatch.conf"
34+PSI_PATH = "/proc/pressure/memory"
35+CGROUP_ROOT = "/sys/fs/cgroup"
36+
37+DEFAULTS = {
38+ # PSI kernel trigger: fire when "full" stall exceeds this many us per window.
39+ # 100ms out of 1s == 10% of wall time with *every* task stalled on memory.
40+ "warn_stall_us": 100000,
41+ "warn_window_us": 1000000,
42+ # Escalate to an unattended kill at this sustained full-avg10 percentage.
43+ "kill_avg10": 25.0,
44+ # ...held for this many consecutive 1s samples. Spikes shorter than this are
45+ # the normal cost of doing business (a build, a big paste) and are ignored.
46+ "kill_hold_s": 4,
47+ # How long the GUI waits for a human before it decides on its own.
48+ "dialog_timeout_s": 20,
49+ # Don't act again for this long after acting. Stops kill storms.
50+ "cooldown_s": 15,
51+ # Ignore anything smaller than this; killing a 40MB process helps nobody.
52+ "min_victim_mb": 300,
53+ # Never select these as victims (substring match on comm / unit name).
54+ "protect": "niri,cosmic-comp,cosmic-session,cosmic-greeter,cosmic-panel,gdm,sddm,systemd,"
55+ "dbus-daemon,dbus-broker,pipewire,wireplumber,sshd,memwatchd,"
56+ "Xwayland,gnome-shell,kwin_wayland,plasmashell",
57+ # Which cgroup subtrees are eligible. Keeps us out of system.slice by default.
58+ "scan_slices": "/user.slice",
59+ "notify_user": "", # empty = autodetect the graphical session owner
60+ "dry_run": "0",
61+}
62+
63+MCL_CURRENT, MCL_FUTURE = 1, 2
64+
65+
66+def log(msg):
67+ sys.stdout.write(msg + "\n")
68+ sys.stdout.flush()
69+
70+
71+def load_config():
72+ cfg = dict(DEFAULTS)
73+ try:
74+ with open(CONFIG_PATH) as fh:
75+ for line in fh:
76+ line = line.split("#", 1)[0].strip()
77+ if not line or "=" not in line:
78+ continue
79+ k, v = line.split("=", 1)
80+ k, v = k.strip(), v.strip()
81+ if k in cfg:
82+ cfg[k] = v
83+ else:
84+ log(f"config: ignoring unknown key {k!r}")
85+ except FileNotFoundError:
86+ log(f"config: {CONFIG_PATH} absent, using defaults")
87+ for k, proto in DEFAULTS.items():
88+ if isinstance(proto, float):
89+ cfg[k] = float(cfg[k])
90+ elif isinstance(proto, int):
91+ cfg[k] = int(cfg[k])
92+ cfg["protect"] = [p.strip() for p in str(cfg["protect"]).split(",") if p.strip()]
93+ cfg["scan_slices"] = [s.strip() for s in str(cfg["scan_slices"]).split(",") if s.strip()]
94+ cfg["dry_run"] = str(cfg["dry_run"]).lower() in ("1", "true", "yes", "on")
95+ return cfg
96+
97+
98+def pin_self():
99+ """Stay resident and runnable while everything else is stalled.
100+
101+ If the watchdog itself has to fault pages back in from zram to make a
102+ decision, it arrives after the freeze it was supposed to prevent.
103+ """
104+ try:
105+ libc = ctypes.CDLL("libc.so.6", use_errno=True)
106+ if libc.mlockall(MCL_CURRENT | MCL_FUTURE) != 0:
107+ log(f"mlockall failed: {os.strerror(ctypes.get_errno())}")
108+ else:
109+ log("mlockall: resident")
110+ except Exception as exc:
111+ log(f"mlockall unavailable: {exc}")
112+ try:
113+ with open(f"/proc/self/oom_score_adj", "w") as fh:
114+ fh.write("-1000")
115+ except OSError as exc:
116+ log(f"oom_score_adj failed: {exc}")
117+ try:
118+ os.sched_setscheduler(0, os.SCHED_RR, os.sched_param(10))
119+ log("sched: SCHED_RR/10")
120+ except (OSError, AttributeError) as exc:
121+ log(f"realtime priority unavailable: {exc}")
122+
123+
124+# --- PSI --------------------------------------------------------------------
125+
126+def read_psi():
127+ """Return (some_avg10, full_avg10) as percentages."""
128+ some = full = 0.0
129+ try:
130+ with open(PSI_PATH) as fh:
131+ for line in fh:
132+ parts = line.split()
133+ if not parts:
134+ continue
135+ vals = dict(p.split("=", 1) for p in parts[1:] if "=" in p)
136+ if parts[0] == "some":
137+ some = float(vals.get("avg10", 0))
138+ elif parts[0] == "full":
139+ full = float(vals.get("avg10", 0))
140+ except OSError:
141+ pass
142+ return some, full
143+
144+
145+def open_psi_trigger(stall_us, window_us):
146+ """Register a kernel PSI trigger and return an fd that polls PRI on breach.
147+
148+ This is the difference between reacting in ~1 second and reacting after
149+ avg10 has had ten seconds to catch up -- by which point you are already
150+ looking at a frozen cursor.
151+ """
152+ fd = os.open(PSI_PATH, os.O_RDWR | os.O_NONBLOCK)
153+ os.write(fd, f"full {stall_us} {window_us}".encode())
154+ return fd
155+
156+
157+# --- process / cgroup inspection -------------------------------------------
158+
159+_MEM_RE = re.compile(rb"^(VmRSS|VmSwap):\s+(\d+) kB", re.M)
160+
161+
162+class Victim:
163+ __slots__ = ("pid", "comm", "rss_kb", "swap_kb", "cgroup", "unit", "_top_kb", "nproc")
164+
165+ def __init__(self, pid, comm, rss_kb, swap_kb, cgroup, unit):
166+ self.pid, self.comm = pid, comm
167+ self.rss_kb, self.swap_kb = rss_kb, swap_kb
168+ self.cgroup, self.unit = cgroup, unit
169+ self._top_kb = rss_kb
170+ self.nproc = 1
171+
172+ @property
173+ def total_kb(self):
174+ return self.rss_kb + self.swap_kb
175+
176+ @property
177+ def total_mb(self):
178+ return self.total_kb // 1024
179+
180+ def __repr__(self):
181+ return (f"<{self.comm} pid={self.pid} {self.total_mb}MB "
182+ f"nproc={self.nproc} unit={self.unit}>")
183+
184+
185+# Structural exclusions. Name-based protection is not enough: these cgroups
186+# are fatal to kill regardless of what happens to be the fattest process
187+# inside them at the time.
188+# session-N.scope -- the whole login session; killing it is a logout
189+# init.scope -- the user's systemd manager
190+# user@N.service -- ditto, the manager unit itself
191+# Only leaf app scopes and user services are ever eligible.
192+def eligible_cgroup(cg, unit):
193+ if unit.startswith("session-") and unit.endswith(".scope"):
194+ return False
195+ if unit in ("init.scope", "user.slice", "-.slice"):
196+ return False
197+ if unit.startswith("user@") and unit.endswith(".service"):
198+ return False
199+ if unit.endswith(".slice"):
200+ return False # never target a slice, only the leaves inside one
201+ return unit.endswith(".scope") or unit.endswith(".service")
202+
203+
204+def proc_cgroup(pid):
205+ try:
206+ with open(f"/proc/{pid}/cgroup", "rb") as fh:
207+ for line in fh:
208+ if line.startswith(b"0::"):
209+ return line[3:].strip().decode("utf-8", "replace")
210+ except OSError:
211+ pass
212+ return ""
213+
214+
215+def scan_candidates(cfg):
216+ """Walk /proc once and return eligible victims, largest first.
217+
218+ We aggregate by cgroup: a browser is 40 processes in one scope, and the
219+ interesting number is what the *scope* costs, not what its biggest renderer
220+ costs. Killing the scope is also the only way to not leave orphans behind.
221+ """
222+ by_cgroup = {}
223+ self_pid = os.getpid()
224+ for entry in os.listdir("/proc"):
225+ if not entry.isdigit():
226+ continue
227+ pid = int(entry)
228+ if pid == self_pid or pid == 1:
229+ continue
230+ try:
231+ with open(f"/proc/{pid}/status", "rb") as fh:
232+ blob = fh.read(2048)
233+ except OSError:
234+ continue # died mid-scan, or a kernel thread
235+ found = dict(_MEM_RE.findall(blob))
236+ if b"VmRSS" not in found:
237+ continue # kernel thread
238+ rss = int(found[b"VmRSS"])
239+ swap = int(found.get(b"VmSwap", b"0"))
240+ nl = blob.find(b"\n")
241+ comm = blob[6:nl].decode("utf-8", "replace").strip() if blob.startswith(b"Name:") else ""
242+ cg = proc_cgroup(pid)
243+ if not cg or not any(cg.startswith(s) for s in cfg["scan_slices"]):
244+ continue
245+ unit = cg.rsplit("/", 1)[-1]
246+ if not eligible_cgroup(cg, unit):
247+ continue
248+ if any(p in comm or p in unit for p in cfg["protect"]):
249+ continue
250+ agg = by_cgroup.get(cg)
251+ if agg is None:
252+ by_cgroup[cg] = Victim(pid, comm, rss, swap, cg, unit)
253+ else:
254+ agg.rss_kb += rss
255+ agg.swap_kb += swap
256+ agg.nproc += 1
257+ # The cgroup gets named after its fattest process -- for a browser
258+ # that is the tab actually eating the box, which is what you want
259+ # to read on the dialog.
260+ if rss > agg._top_kb:
261+ agg._top_kb, agg.comm, agg.pid = rss, comm, pid
262+ out = [v for v in by_cgroup.values() if v.total_mb >= cfg["min_victim_mb"]]
263+ out.sort(key=lambda v: v.total_kb, reverse=True)
264+ return out
265+
266+
267+# --- acting -----------------------------------------------------------------
268+
269+def cg_path(victim, leaf):
270+ return os.path.join(CGROUP_ROOT, victim.cgroup.lstrip("/"), leaf)
271+
272+
273+def freeze(victim, on=True):
274+ """Halt the whole cgroup atomically. Nothing escapes, nothing forks away."""
275+ path = cg_path(victim, "cgroup.freeze")
276+ try:
277+ with open(path, "w") as fh:
278+ fh.write("1" if on else "0")
279+ return True
280+ except OSError as exc:
281+ log(f"freeze({victim.unit}, {on}) failed: {exc}; falling back to signals")
282+ sig = signal.SIGSTOP if on else signal.SIGCONT
283+ return signal_tree(victim, sig)
284+
285+
286+def signal_tree(victim, sig):
287+ ok = False
288+ try:
289+ with open(cg_path(victim, "cgroup.procs")) as fh:
290+ pids = [int(x) for x in fh.read().split()]
291+ except OSError:
292+ pids = [victim.pid]
293+ for pid in pids:
294+ try:
295+ os.kill(pid, sig)
296+ ok = True
297+ except OSError as exc:
298+ if exc.errno != errno.ESRCH:
299+ log(f"kill({pid},{sig}) failed: {exc}")
300+ return ok
301+
302+
303+def kill_victim(victim, cfg, reason):
304+ if cfg["dry_run"]:
305+ log(f"DRY-RUN would kill {victim!r} ({reason})")
306+ return True
307+ # cgroup.kill is atomic and reaps the entire tree; a frozen cgroup still
308+ # honours it, so we never need to thaw-then-race the process.
309+ try:
310+ with open(cg_path(victim, "cgroup.kill"), "w") as fh:
311+ fh.write("1")
312+ log(f"KILLED cgroup {victim.unit} ({victim.comm}, {victim.total_mb}MB) -- {reason}")
313+ return True
314+ except OSError as exc:
315+ log(f"cgroup.kill unavailable for {victim.unit}: {exc}; using SIGKILL")
316+ freeze(victim, False)
317+ ok = signal_tree(victim, signal.SIGKILL)
318+ log(f"{'KILLED' if ok else 'FAILED to kill'} {victim!r} -- {reason}")
319+ return ok
320+
321+
322+# --- talking to the human ---------------------------------------------------
323+
324+_GUI_ENV_KEYS = ("WAYLAND_DISPLAY", "DISPLAY", "XDG_RUNTIME_DIR",
325+ "DBUS_SESSION_BUS_ADDRESS", "XAUTHORITY", "XDG_SESSION_TYPE")
326+
327+
328+def find_session(cfg):
329+ """Locate a graphical session and lift its environment.
330+
331+ We run as a system service, which has no idea how to reach a Wayland
332+ compositor. The reliable way is to read the env of a process that is
333+ already inside the session.
334+ """
335+ want_user = cfg["notify_user"] or None
336+ try:
337+ out = subprocess.run(
338+ ["loginctl", "list-sessions", "--no-legend"],
339+ capture_output=True, text=True, timeout=5).stdout
340+ except Exception:
341+ return None
342+ for line in out.splitlines():
343+ f = line.split()
344+ if len(f) < 3:
345+ continue
346+ sid, uid, user = f[0], f[1], f[2]
347+ if want_user and user != want_user:
348+ continue
349+ try:
350+ show = subprocess.run(["loginctl", "show-session", sid,
351+ "-p", "Type", "-p", "Leader", "-p", "State"],
352+ capture_output=True, text=True, timeout=5).stdout
353+ except Exception:
354+ continue
355+ props = dict(l.split("=", 1) for l in show.splitlines() if "=" in l)
356+ if props.get("Type") not in ("wayland", "x11"):
357+ continue
358+ leader = props.get("Leader")
359+ if not leader:
360+ continue
361+ env = harvest_env(int(leader)) or harvest_env_from_uid(int(uid))
362+ if env:
363+ return int(uid), user, env
364+ return None
365+
366+
367+def harvest_env(pid):
368+ try:
369+ with open(f"/proc/{pid}/environ", "rb") as fh:
370+ raw = fh.read()
371+ except OSError:
372+ return None
373+ env = {}
374+ for item in raw.split(b"\0"):
375+ if b"=" not in item:
376+ continue
377+ k, v = item.decode("utf-8", "replace").split("=", 1)
378+ if k in _GUI_ENV_KEYS:
379+ env[k] = v
380+ return env if ("WAYLAND_DISPLAY" in env or "DISPLAY" in env) else None
381+
382+
383+def harvest_env_from_uid(uid):
384+ """Fallback: sweep the user's processes for one that knows the display."""
385+ for entry in os.listdir("/proc"):
386+ if not entry.isdigit():
387+ continue
388+ try:
389+ if os.stat(f"/proc/{entry}").st_uid != uid:
390+ continue
391+ except OSError:
392+ continue
393+ env = harvest_env(int(entry))
394+ if env:
395+ return env
396+ return None
397+
398+
399+def ask_human(victim, cfg, session, psi_full):
400+ """Show the dialog. Returns (kill: bool, reason: str)."""
401+ if session is None:
402+ log("no graphical session found; escalating without asking")
403+ return True, "no graphical session to ask"
404+ uid, user, env = session
405+ env = dict(env)
406+ env.setdefault("XDG_RUNTIME_DIR", f"/run/user/{uid}")
407+ env["HOME"] = os.path.expanduser(f"~{user}")
408+ env["PATH"] = "/usr/local/bin:/usr/bin:/bin"
409+
410+ text = (
411+ f"<b>Memory pressure: {psi_full:.0f}% stalled</b>\n\n"
412+ f"Largest consumer has been <b>paused</b> so the desktop stays usable:\n\n"
413+ f" <b>{victim.comm or victim.unit}</b>\n"
414+ f" {victim.total_mb} MB ({victim.rss_kb//1024} MB RAM + {victim.swap_kb//1024} MB zram)\n"
415+ f" {victim.unit}\n\n"
416+ f"Kill it, or resume it and ride it out?\n"
417+ f"<i>No answer in {cfg['dialog_timeout_s']}s = kill.</i>"
418+ )
419+ cmd = [
420+ "yad", "--title=Memory pressure", "--window-icon=dialog-warning",
421+ "--image=dialog-warning", "--borders=16", "--width=460",
422+ "--text", text, "--text-align=left", "--on-top", "--center",
423+ "--sticky", "--undecorated", "--skip-taskbar", "--no-escape",
424+ f"--timeout={cfg['dialog_timeout_s']}", "--timeout-indicator=bottom",
425+ "--button=Resume it!media-playback-start:1",
426+ "--button=Kill it!process-stop:0",
427+ ]
428+ try:
429+ rc = subprocess.run(
430+ ["setpriv", "--reuid", str(uid), "--regid", str(uid), "--init-groups", "--"] + cmd,
431+ env=env, timeout=cfg["dialog_timeout_s"] + 10).returncode
432+ except FileNotFoundError:
433+ log("yad or setpriv missing; escalating without asking")
434+ return True, "no dialog binary available"
435+ except subprocess.TimeoutExpired:
436+ log("dialog wedged; escalating")
437+ return True, "dialog stopped responding"
438+ # 0 = Kill pressed. 1 = Resume. 70 = our --timeout elapsed.
439+ if rc == 1:
440+ return False, "user chose resume"
441+ if rc == 0:
442+ return True, "user chose kill"
443+ if rc == 70:
444+ return True, f"no answer in {cfg['dialog_timeout_s']}s"
445+ return True, f"dialog dismissed (rc={rc})"
446+
447+
448+def notify(session, summary, body):
449+ if session is None:
450+ return
451+ uid, user, env = session
452+ env = dict(env)
453+ env.setdefault("XDG_RUNTIME_DIR", f"/run/user/{uid}")
454+ try:
455+ subprocess.run(
456+ ["setpriv", "--reuid", str(uid), "--regid", str(uid), "--init-groups", "--",
457+ "notify-send", "-u", "critical", "-i", "dialog-warning", summary, body],
458+ env=env, timeout=10, check=False)
459+ except Exception:
460+ pass
461+
462+
463+# --- main loop --------------------------------------------------------------
464+
465+def handle_pressure(cfg, psi_fd):
466+ """One full incident, from trigger to resolution."""
467+ some, full = read_psi()
468+ log(f"pressure event: some_avg10={some:.1f}% full_avg10={full:.1f}%")
469+
470+ candidates = scan_candidates(cfg)
471+ if not candidates:
472+ log("no eligible victim above min_victim_mb; nothing to do")
473+ return
474+ victim = candidates[0]
475+ log(f"candidates: {candidates[:3]}")
476+
477+ # Freeze first. This is what buys us the ability to draw a dialog at all:
478+ # the offender stops allocating immediately, reclaim catches up, and the
479+ # compositor gets scheduled again.
480+ frozen = freeze(victim, True) if not cfg["dry_run"] else False
481+ log(f"froze {victim.unit}: {frozen}")
482+
483+ session = find_session(cfg)
484+
485+ # Escalation watchdog: if pressure keeps climbing even with the top
486+ # consumer frozen, we picked wrong or it's systemic. Don't wait on a human.
487+ decided = None
488+ held = 0
489+ deadline = time.monotonic() + cfg["dialog_timeout_s"]
490+
491+ result = {}
492+
493+ def asker():
494+ result["kill"], result["why"] = ask_human(victim, cfg, session, full)
495+
496+ t = threading.Thread(target=asker, daemon=True)
497+ t.start()
498+ while t.is_alive() and time.monotonic() < deadline + 5:
499+ time.sleep(1)
500+ _, f_now = read_psi()
501+ if f_now >= cfg["kill_avg10"]:
502+ held += 1
503+ if held >= cfg["kill_hold_s"]:
504+ decided = "escalated: full_avg10 stayed >= %.0f%% for %ds" % (
505+ cfg["kill_avg10"], held)
506+ break
507+ else:
508+ held = 0
509+
510+ if decided is None:
511+ t.join(timeout=2)
512+ if result.get("kill", True):
513+ decided = result.get("why", "dialog never returned")
514+ else:
515+ log(f"user spared {victim.unit}; thawing")
516+ freeze(victim, False)
517+ notify(session, "Resumed", f"{victim.comm or victim.unit} is running again.")
518+ return
519+
520+ kill_victim(victim, cfg, decided)
521+ notify(session, "Killed to recover memory",
522+ f"{victim.comm or victim.unit} ({victim.total_mb} MB) — {decided}")
523+
524+
525+def main():
526+ cfg = load_config()
527+ log(f"memwatchd starting: warn at {cfg['warn_stall_us']}us/{cfg['warn_window_us']}us, "
528+ f"auto-kill at full_avg10>={cfg['kill_avg10']}% held {cfg['kill_hold_s']}s, "
529+ f"dry_run={cfg['dry_run']}")
530+ pin_self()
531+
532+ if not os.path.exists(PSI_PATH):
533+ log("FATAL: no /proc/pressure/memory (kernel needs CONFIG_PSI=y psi=1)")
534+ return 1
535+ try:
536+ psi_fd = open_psi_trigger(cfg["warn_stall_us"], cfg["warn_window_us"])
537+ except OSError as exc:
538+ log(f"FATAL: cannot register PSI trigger: {exc}")
539+ return 1
540+ log("armed; waiting on kernel pressure events")
541+
542+ poller = select.poll()
543+ poller.register(psi_fd, select.POLLPRI)
544+ last_action = 0.0
545+
546+ while True:
547+ events = poller.poll()
548+ for _fd, ev in events:
549+ if ev & select.POLLERR:
550+ log("FATAL: PSI trigger revoked")
551+ return 1
552+ if not (ev & select.POLLPRI):
553+ continue
554+ now = time.monotonic()
555+ if now - last_action < cfg["cooldown_s"]:
556+ continue
557+ last_action = now
558+ try:
559+ handle_pressure(cfg, psi_fd)
560+ except Exception as exc:
561+ log(f"incident handler blew up: {exc!r}")
562+ last_action = time.monotonic()
563+
564+
565+if __name__ == "__main__":
566+ sys.exit(main())
added units/memwatch.service +46 -0
new file mode 100644
@@ -0,0 +1,46 @@
1+[Unit]
2+Description=PSI memory-stall watchdog (freeze, ask, kill)
3+Documentation=file:/usr/share/doc/memwatch/README.md
4+DefaultDependencies=no
5+After=sysinit.target
6+Before=multi-user.target
7+Conflicts=shutdown.target
8+Before=shutdown.target
9+
10+[Service]
11+Type=simple
12+ExecStart=/usr/local/lib/memwatch/memwatchd.py
13+Restart=always
14+RestartSec=2
15+
16+# The watchdog must be the last thing on the box to suffer. If it gets
17+# reclaimed, descheduled, or OOM-killed, it arrives after the freeze it exists
18+# to prevent.
19+OOMScoreAdjust=-1000
20+ManagedOOMPreference=omit
21+MemoryMin=48M
22+MemorySwapMax=0
23+CPUWeight=10000
24+IOWeight=10000
25+Nice=-10
26+
27+# mlockall() and SCHED_RR, plus the cgroup writes used to freeze and kill.
28+AmbientCapabilities=CAP_IPC_LOCK CAP_SYS_NICE CAP_KILL CAP_SYS_PTRACE
29+CapabilityBoundingSet=CAP_IPC_LOCK CAP_SYS_NICE CAP_KILL CAP_SYS_PTRACE CAP_SETUID CAP_SETGID CAP_DAC_OVERRIDE
30+LimitMEMLOCK=infinity
31+LimitRTPRIO=99
32+
33+NoNewPrivileges=no
34+ProtectSystem=strict
35+ProtectHome=read-only
36+PrivateTmp=yes
37+ProtectKernelModules=yes
38+ProtectControlGroups=no
39+RestrictRealtime=no
40+
41+StandardOutput=journal
42+StandardError=journal
43+SyslogIdentifier=memwatchd
44+
45+[Install]
46+WantedBy=multi-user.target
new file mode 100644
@@ -0,0 +1,46 @@
1+[Unit]
2+Description=PSI memory-stall watchdog (freeze, ask, kill)
3+Documentation=file:/usr/share/doc/memwatch/README.md
4+DefaultDependencies=no
5+After=sysinit.target
6+Before=multi-user.target
7+Conflicts=shutdown.target
8+Before=shutdown.target
9+
10+[Service]
11+Type=simple
12+ExecStart=/usr/local/lib/memwatch/memwatchd.py
13+Restart=always
14+RestartSec=2
15+
16+# The watchdog must be the last thing on the box to suffer. If it gets
17+# reclaimed, descheduled, or OOM-killed, it arrives after the freeze it exists
18+# to prevent.
19+OOMScoreAdjust=-1000
20+ManagedOOMPreference=omit
21+MemoryMin=48M
22+MemorySwapMax=0
23+CPUWeight=10000
24+IOWeight=10000
25+Nice=-10
26+
27+# mlockall() and SCHED_RR, plus the cgroup writes used to freeze and kill.
28+AmbientCapabilities=CAP_IPC_LOCK CAP_SYS_NICE CAP_KILL CAP_SYS_PTRACE
29+CapabilityBoundingSet=CAP_IPC_LOCK CAP_SYS_NICE CAP_KILL CAP_SYS_PTRACE CAP_SETUID CAP_SETGID CAP_DAC_OVERRIDE
30+LimitMEMLOCK=infinity
31+LimitRTPRIO=99
32+
33+NoNewPrivileges=no
34+ProtectSystem=strict
35+ProtectHome=read-only
36+PrivateTmp=yes
37+ProtectKernelModules=yes
38+ProtectControlGroups=no
39+RestrictRealtime=no
40+
41+StandardOutput=journal
42+StandardError=journal
43+SyslogIdentifier=memwatchd
44+
45+[Install]
46+WantedBy=multi-user.target