The other entries in this repository show controls that looked protected and weren't. This one is about what comes next: you fixed everything — how do you know, six months later, that it's still fixed?
The uncomfortable answer is that most people don't. They find out by accident, or they don't find out. The case behind this entry: two mitigations that had been applied and verified sat undone for two weeks without a single indicator changing appearance.
Across the findings in this repository, not one was "the control isn't installed". Every one was a control present, active, and silently not enforcing:
| Symptom | What a naive check showed |
|---|---|
| Application firewall with default action "allow" | active (running) |
| Enforcement daemon in a crash loop | unit exists, stale hook still listed |
Unit masked without --now, socket alive in the kernel |
Loaded: masked |
Mask applied to the system instance only, not --user |
masked (on the wrong instance) |
LSM installed but absent from the lsm= cmdline |
service active, zero enforcement |
| Confinement profiles loaded in permissive mode | profile "loaded" |
| WAF in detection-only mode | rules loaded and matching |
All of them pass is-active and is-enabled. The gap to cover is the distance
between declared configuration and runtime/kernel truth — which neither a
file integrity checker (rkhunter/AIDE class) nor an attack surface audit
covers. Three different questions:
- did the file change? → integrity
- what is exposed? → surface
- is what I applied still applying? → this entry
Three states, never two: OK / FAIL / INDETERMINATE — and indeterminate never collapses into OK.
Without privilege, commands like listing BPF programs, printing a firewall
ruleset, or reading confinement profiles from securityfs return empty in a way
that looks successful. A monitor reporting "all clear" because it couldn't read
anything is worse than no monitor: it manufactures confidence with nothing behind
it.
Practical consequences:
- capture output before processing it.
command | grep -c patternfollowed by an exit-status test checks thegrep, which returns 1 when the count is zero — so "no hooks loaded" (the exact case you're trying to catch) becomes "couldn't verify"; - the summary counts indeterminates separately, and the exit code distinguishes them from failures;
- every possible state emits a line. An
if/elifwith noelsemakes the check vanish from the report when a unit is in an unexpected state, and a missing line reads as "not checked" to any human.
This is the most reusable result. Of ~20 checks, the ones that failed had one thing in common, and it wasn't importance:
| Survived months of updates | Where they live |
|---|---|
| masked units | symlinks you created under /etc/systemd/system |
kernel parameters (sysctl) |
your own file in /etc/sysctl.d/ |
| per-cgroup firewall rule | your own unit |
| LSM enabled at boot | bootloader configuration |
| Silently reverted | Where they live |
|---|---|
| application firewall default action | package-owned config file |
| perimeter firewall template rules | package-owned files |
| network alias file | package-owned directory |
Drift risk correlates with where a mitigation is stored, not with how important it is. Every mitigation living in a package-owned file is exposed to being undone by an update or reinstall; every mitigation in a file you created is out of that vector's reach.
This is actionable and forward-looking: list your mitigation files, find which belong to packages, and you have the exact watchlist — before any incident.
Instead of one check per specific rule, cross the list of files you hand-edited against what the package manager reports:
# Arch/pacman — adapt for dpkg --verify, rpm -V, etc.
pacman -Qii <package> | grep '<file path>'Two distinct signals:
- file reported as unmodified → your edit was overwritten;
- file outside the package's backup array → no
.pacnew/.dpkg-distis possible, the next update overwrites silently. Open vector even when today's content is correct.
Honest limitation, measured in practice: this catches total reversion and the "no backup" vector. It is blind to partial reversion of a file that has any other local divergence — the manager says "modified", which tells you nothing about whether your specific line survived. A content check that actually reads the line stays necessary. Package metadata is a cheap signal; what declares a mitigation alive is reading the content.
Most package managers let you declare that a file must never be overwritten
(NoUpgrade in pacman.conf, dpkg-divert, %config(noreplace)). The effect is
that the manager writes the new version alongside instead of on top. Pair it with
a check that warns when a new version is pending review.
The two categories above (package file vs. your own file) don't exhaust the ways a mitigation can disappear. A third case, seen in practice: a documented baseline had four processes with a permanent network rule. One of them had a stale binary path (the same pattern as the "rule derived from historical log" finding above) and broke an essential tool. The fix script, written under pressure to get the tool back online, only rewrote the rule for the process that broke — fixed the symptom, confirmed "fixed", and moved on.
The other three processes from the same documented baseline were never recreated. No alarm fired, because nothing was "broken" in the obvious sense — each one kept working, just through repeated manual approval (a popup) instead of a permanent rule. The gap stayed invisible for weeks: no symptom, just a recurring annoyance (approve the same popup again) easy to blame on "that process that always asks" rather than recognize as baseline regression.
Why this isn't caught by the pacman -Qii/dpkg --verify check from the
previous section: that check detects a file reverted from the package. This
case is different — the file itself was never recreated after being deleted
during an earlier incident; there's no "original version" to compare against via
package metadata, because the source of truth is a decision/document, not a file
any manager tracks.
Mitigation that generalizes: after any incident-fix script, the final step isn't "the symptom is gone" — it's diffing the resulting state against the entire documented baseline, not just the item that broke. If the baseline says "N processes with a permanent rule", the verification script lists the N expected names and fails explicitly if any is missing, instead of only testing that the fixed process now works:
# bad: only confirms the symptom is gone
command-that-depended-on-the-fixed-process --version
# better: checks the whole baseline, naming each expected item
for rule in item-a item-b item-c item-d; do
test -f "/path/to/rules/${rule}.json" || echo "MISSING: $rule"
doneAfter turning on a restrictive policy, the instinct is to test that "what should work, works". That proves nothing — it would work identically under the permissive policy.
What proves it is the pair, at the same moment, against the same destination:
# process ON the allowlist
curl -sS -o /dev/null -w "%{http_code}\n" https://public-example/ # → 200
# process NOT on the list, same host, same port
python3 -c "import socket;s=socket.socket();s.settimeout(10);s.connect(('public-example',443))"
# → TimeoutErrorThe contrast is the value of the test. Testing only the side that should work is the same sampling error that makes someone declare "zero false positives" over an observation window with no traffic in it.
These cost real time, and none would have been caught by reviewing code — they only surface when the program meets the machine's actual state.
Requiring every base chain with hook input to be drop produces an
immediate false positive. In nftables several base chains coexist on the same hook
and all are traversed; policy accept on one means only "this chain does not
object", and a drop anywhere ends the packet. Application firewalls using an
nftables backend create their own tables with chains at accept — that's where
they inject their rules, and it is normal operation.
Check the chain that actually carries the default posture (with the iptables-nft
backend, the INPUT chain of the ip filter and ip6 filter tables), and treat
the rest as informational. Check both families: a parser that stops at the
first match reports OK with IPv4 at drop even when IPv6 is wide open.
An allowlist built from logs weeks old contained the path of a binary that no
longer existed — the tool had migrated from a distro package to its own
versioned installer under $HOME. The rule was syntactically perfect, the daemon
loaded it without complaint, and it matched no process at all. Result: that
program silently lost network access.
Two lessons. First, validate against today's disk before applying — a loop
that confirms each path exists and aborts before touching policy. Second, if the
binary lives in a versioned directory, a fixed path will break again at the next
update; use a pattern covering any version, and know that allowing a path under
$HOME is weaker than one under /usr/bin, because it is user-writable.
Scheduling the monitor as a user timer and then installing the script as
root:root "so nothing running as you can change what executes" protects
nothing. What decides what runs is the unit file's ExecStart=, and a user
timer's unit file lives in a user-writable directory. Rewriting it is enough.
The vector only closes with unit and script out of the user's write reach —
that is, a system unit. As a bonus, running as root removes the need for a
NOPASSWD sudoers rule for the privileged checks, which would be a larger
escalation surface than the problem it solves.
systemctl --user run as root talks to root's own instance. Pointing the
runtime variable at another user is not enough, because the bus authenticates the
peer. The supported path is --machine=<user>@.host, which does not depend on
sudo — relevant because NoNewPrivileges= on the unit would block setuid
binaries anyway. And if the user session does not exist, the verdict is
indeterminate, never failure: you do not assert that a protection dropped when
you could not even ask.
When comparing against a baseline, distinguish a value that changed from a value that came into existence. A previously indeterminate check that is now measurable — or a new check added to the script — is not drift. Counting it as such produces a daily alarm about instrumentation changes, and a monitor that cries without cause is one you learn to ignore.
If the baseline is frozen by hand while the scheduled run executes with different environment variables, checks that one can measure and the other cannot will diverge forever. Provide an acceptance path that uses exactly the same environment definition as the scheduled run.
Every drift monitor needs an "accept this state as the new normal" command — without it, noise accumulates and the tool is abandoned within two weeks. And that same command is what can destroy the tool's value.
Accept only when both conditions hold:
- the monitor flagged a difference;
- you know that difference was intentional.
For drift you did not cause, the path is to investigate and fix. Once fixed, the value matches the baseline again on its own — no acceptance needed. Accepting there freezes the problem as the expected state, and the monitor now actively guarantees nobody will complain about it again.
The parallel is the integrity tool installed but whose reference database was never initialized: present, taking up space, producing no signal.
A report that exists only in the system log reproduces the original condition — information sitting somewhere with nothing pushing it forward.
A cheap solution, without building notification plumbing: map exit codes so the scheduler treats failure as failure.
# exit 0 = everything verified and OK -> success
# exit 1 = failure or drift -> unit FAILS, deliberately
# exit 2 = nothing failed, but INDET exist -> success
SuccessExitStatus=0 2The 2 must be success because it is the normal result of an unprivileged run;
treating it as an error leaves the unit permanently red until the signal becomes
noise. The 1 makes the unit fail, so it shows up in
systemctl list-units --failed and in any status bar that reads failed units.
- Does each check have three states, and is "couldn't verify" never reported as OK?
- Is privileged command output captured before being processed by
grep -cor similar? - Does every
if/elifpath emit a report line, including unexpected states? - Have you listed which of your mitigation files are package-owned?
- Are those files protected from overwrite, and is there a check for a pending new version?
- Is there a content check for each critical mitigation, beyond package-manager metadata?
- Was enforcement verified by a differential test (one case that must pass and one that must fail, at the same moment)?
- Were allowlists validated against the current disk, not just the log they came from?
- If the monitor is scheduled, are unit and script out of the user's write reach?
- Does the firewall policy check cover IPv4 and IPv6?
- Is the baseline accepted in the same environment the scheduled run uses?
- Does a monitor failure produce a visible signal, or does it stay in the log?
- After any incident fix, was the entire documented baseline checked — not just the item that broke?