vhotplug keeps its boot-time config: passthrough rule changes are inert until reboot

[brianmcgillion]

Summary

ghaf.hardware.passthrough.vhotplug writes its rules to /etc/vhotplug.conf, but
systemd.services.vhotplug declares no restartTriggers. NixOS restarts a unit when its unit
file changes, and rewriting a file under environment.etc does not change the unit file — so
after nixos-rebuild switch the running vhotplug keeps the rules it parsed at boot.

The result is a silent no-op. Devices continue to be routed by the previous rules while every
artefact you would think to check says the change is live: the generation moved,
/etc/vhotplug.conf has the new content, the store path resolves, nix flake check passes. Only
the process is stale.

Reproduce

1. On a running laptop-x86 target, add a prependUsbRules entry routing a specific USB device to an
   app VM — anything that changes usbPassthrough.
2. nixos-rebuild switch.
3. Confirm the deployed config is correct:

   $ jq '.usbPassthrough[0]' /etc/vhotplug.conf
   { "description": "...", "targetVm": "<app-vm>", "allow": [ { "vendorId": "...", "productId": "..." } ] }

4. Plug the device in.

Expected: attached to <app-vm>.
Actual: attached according to the previous rules — for a USB drive, gui-vm, via the default
interfaceClass 8 / interfaceSubclass 6 rule in modules/hardware/passthrough/usb-rules.nix.

The tell:

$ systemctl show vhotplug -p ExecMainStartTimestamp -p MainPID
ExecMainStartTimestamp=Thu 2026-08-06 23:30:52 +04
MainPID=933
$ stat -c '%y' /etc/vhotplug.conf
2026-08-06 23:59:25 +0400

The unit predates the config by 29 minutes and was never restarted.

systemctl restart vhotplug followed by a replug routes the device correctly on the first attempt.

Root cause

Three things compose badly:

1. vhotplug parses its config exactly once. vhotplug/vhotplug.py:171 builds
   config = Config(args.config), and Config.__init__ (vhotplug/config.py:37) calls load(),
   a single json.load at line 41. There is no re-read and no SIGHUP handler.

2. Its FileWatcher does not cover the config. vhotplug already uses inotify, but only on VM
   socket paths — vhotplug/vhotplug.py:180 adds vm["socket"] for each VM, to detect VM
   restarts so devices can be re-attached. The config file is never watched. (Worth stating
   explicitly: the presence of a file watcher in the tree makes it look as though this is already
   handled.)

3. The unit has no restartTriggers. modules/hardware/passthrough/vhotplug.nix:184 defines
   systemd.services.vhotplug with ExecStart = "... -a -c /etc/vhotplug.conf" (line 194) and no
   restart trigger for the file written at line 161.

Proposed fix

systemd.services.vhotplug.restartTriggers = [
  config.environment.etc."vhotplug.conf".source
];

This embeds the config's store path into X-Restart-Triggers=, so the unit file changes whenever
the rules change and switch restarts the daemon.

Caveats worth deciding on before merging

• A restart does not re-home already-attached devices. vhotplug runs with -a, but a device
  already attached to a VM is skipped, so a device plugged in before the switch stays where the
  old rules put it. Replugging re-routes it. This is probably the right behaviour — detaching a
  disk from a running VM during activation would be worse — but the fix should not be described as
  making switch alone sufficient.

• Restarting mid-session appeared harmless in our testing: PCI devices logged
  is already attached to the VM with id vhp-pci-N and nothing moved. That is a single
  observation on one machine, not a guarantee, and it is the main reason to consider the
  alternative below.

• Alternative: reload instead of restart. vhotplug could watch its own config and re-read it,
  which avoids restart side effects entirely and reuses the inotify plumbing already present. That
  is a change to tiiuae/vhotplug rather than to ghaf. If that is preferred, this issue is really
  "ghaf should restart it until vhotplug can reload itself".

Impact

Any change to device routing on a running machine: ghaf.hardware.passthrough.usb.guivmRules,
netvmRules, audiovmRules, an app VM's usbPassthrough, prependUsbRules / postpendUsbRules,
and the PCI, evdev and ACPI rule lists — all are written to the same file and all are affected.

The failure mode is the expensive kind: nothing errors, nothing logs, and the natural conclusion is
that the rule itself is wrong. We came close to concluding that prependUsbRules does not take
precedence over the default rules, and re-planning around a bug that does not exist. What actually
identified it was comparing ExecMainStartTimestamp against the config's mtime.

Environment

• ghaf d9328b9 — line numbers in this report refer to that revision
• vhotplug 4a0f407b2f96fc169f3cc83cda1924c456c169da, as pinned by ghaf's flake.lock
• An x86 laptop target with ghaf.profiles.laptop-x86 and hardware.passthrough.mode = "dynamic"