memwatch

A memory watchdog that triggers on stall time, not on free-memory percentage.
It freezes the offender first, then asks, then kills.

Why the obvious design doesn't work on this machine

The instinct is "watch MemAvailable, kill when it drops below N%". On a box
with 32G RAM and 31.2G of zram swap, that never fires:

zram lives in RAM. Pages pushed to it are compressed, not evicted.
So MemAvailable degrades gently forever instead of hitting a wall.
So the kernel OOM killer never gets an allocation failure to react to.
Confirmed: zero OOM kills in 30 days on a machine that freezes regularly.
Meanwhile every task is blocked in page reclaim. That is the freeze.

The quantity that actually corresponds to "the machine shat the bed" is
/proc/pressure/memory — PSI, the kernel's measure of time spent stalled.
full avg10 = 40% means 40% of the last ten seconds, every runnable task was
stuck waiting on memory. That is a number you can threshold on, and the kernel
will push you an event the moment it is crossed.

Measured on this box: a hog capped at 2G drove global full avg10 to 17.9%.
A real runaway sits far above that.

The freeze-first trick

A dialog asking "kill this?" is worthless if the machine is too wedged to draw
it. So on the warn tier memwatchd does not show a dialog first. It:

Writes 1 to the offending cgroup's cgroup.freeze. The entire process
tree halts atomically — nothing forks away, nothing escapes.
Allocation rate instantly goes to zero, reclaim catches up, pressure drops.
Measured: full avg10 fell 39.1% → 26.6% within 4 seconds of freezing.
Now the compositor gets scheduled again, and the dialog paints.
You pick Kill it or Resume it. Resume thaws it and walks away.

If you don't answer in 20s, it kills. If pressure keeps climbing even with the
top consumer frozen — meaning we picked wrong, or it's systemic — it stops
waiting on you and kills immediately.

Tiers

tier	condition	action
warn	kernel PSI trigger: `full` stall ≥ 100ms in any 1s window	freeze top consumer, show dialog
kill	`full avg10` ≥ 25% held 4 consecutive seconds	kill, no dialog
—	15s cooldown after any action	prevents kill storms

The 4-second hold is what answers "it tends to spike": a build, a big paste, or
a page load will blow through 25% for a second or two and must be ignored. Four
straight seconds of total stall is not a spike, it's a machine going down.

Safety

Victim selection is structural, not name-based, because name-based is one
bad guess away from logging you out. Permanently ineligible:

session-N.scope — the whole login session. Killing it is a logout. (This
was a real bug caught in testing: niri at 808MB was ranked 4th and would
have been selectable.)
init.scope, user@N.service — your systemd user manager.
any *.slice — only leaves inside a slice are ever targeted.

On top of that, protect= in the config is a name blocklist for the compositor,
audio stack, dbus and sshd.

Processes are aggregated by cgroup, not individually — a browser is 40
processes in one scope, and both the honest number and the correct kill unit is
the scope. cgroup.kill reaps the tree atomically, so no orphaned renderers.

The daemon protects itself: mlockall(MCL_CURRENT|MCL_FUTURE),
OOMScoreAdjust=-1000, ManagedOOMPreference=omit, MemorySwapMax=0,
SCHED_RR priority 10. If the watchdog has to fault its own pages back in from
zram to make a decision, it arrives after the freeze it was meant to prevent.

Install

run0 ./install.sh

Ships with dry_run = 1. It will log exactly what it would freeze and kill
and touch nothing. Watch it through a few real pressure events:

journalctl -fu memwatch

When the picks look right, set dry_run = 0 in /etc/memwatch.conf and
systemctl restart memwatch.

Tuning

All in /etc/memwatch.conf.

Woken during normal heavy builds → raise warn_stall_us to 200000 (20%).
Want more time to react → raise dialog_timeout_s.
Want it to kill sooner and stop asking → lower kill_avg10 to 15.
Want it to never kill unattended → set kill_avg10 = 100. The dialog still
appears and the freeze still happens; nothing dies without a click.

Two things to do alongside this

optional/10-user-pressure.conf — systemd-oomd is running on this box
but is disarmed for your applications (user.slice has
ManagedOOMMemoryPressure=auto, which inherits from root, which is off).
Arming it gives you a backstop that survives memwatchd being dead or wrong.
It is slower and dumber and kills without asking; that is the point.
optional/README-zram.md — 31.2G of zram on 32G of RAM is the
underlying cause. Shrinking it to ram / 4 or adding real disk swap makes
every layer here work better.

memwatchd does not require either. It triggers on stall, which is the right
signal regardless of how swap is set up.