Our brutal solution for dealing with our server's flaky WiFi connection.
This repository contains firmware for the "WiFi Killer", which can power-cycle our USB WiFi stick via a relay. It also contains the wifi-watchdog systemd service, which monitors the server's WiFi connection and performs soft resets or, if unavoidable, hard resets through the WiFi Killer when the connection stalls.
This was developed due to the ever-growing unreliability of our server's WiFi connection. In some cases, this was caused by a stale eduroam connection. In other cases, the WiFi stick got "wedged", meaning it refused to reconnect until it was power-cycled.
- Hardware Setup
- WiFi Killer Firmware Installation
- Watchdog Installation
- Technical details
- AI Disclaimer
The hardware consists of an Arduino Nano with an ATmega168, connected to a relay that toggles the WiFi stick's power line.
The relay is configured so that the power line is normally closed:
Arduino LOW = power connected
Arduino HIGH = power interrupted
A relay was used instead of a transistor because it can be isolated and therefore does not require a common ground connection.
The firmware is built with PlatformIO. After installing PlatformIO, build the firmware with:
pio run -e nanoatmega168Upload it to the connected Nano with:
pio run -e nanoatmega168 -t uploadFirst, clone this repository onto the server:
git clone https://github.com/TuDo-Makerspace/server-wifi-watchdog.git
cd server-wifi-watchdogThe watchdog expects a stable serial path for the Arduino, normally:
/dev/stick-killer
That path is created by the included udev rule:
udev/99-stick-killer.rules
Ensure that the WiFi Killer is plugged in. Then install and reload the rule that maps it to /dev/stick-killer:
sudo install -m 0644 udev/99-stick-killer.rules /etc/udev/rules.d/99-stick-killer.rules
sudo udevadm control --reload-rules
sudo udevadm trigger --subsystem-match=tty
ls -l /dev/stick-killerIf the WiFi Killer does not appear under /dev, it may be plugged into a different USB port. In that case, the KERNELS=="..." match in udev/99-stick-killer.rules may need to be updated to match the correct port.
You can determine the correct USB path with commands such as:
dmesg
udevadm info
lsusb -tAfter updating the rule, reload it again using the commands above.
To install the watchdog, run:
sudo ./scripts/install-watchdog-service.sh --start --enableThe installer deploys the following files:
/usr/local/sbin/wifi-watchdog.py
/etc/systemd/system/wifi-watchdog.service
/etc/wifi-watchdog.env
/usr/local/share/wifi-watchdog/README.md
On update, existing values in /etc/wifi-watchdog.env are preserved, and the old file is backed up.
Review the deployed configuration with:
sudoedit /etc/wifi-watchdog.envImportant settings:
| Setting | Meaning |
|---|---|
WIFI_STICK_IFACE |
WiFi interface to monitor, for example wlx40a5efd86cc5. |
RELAY_SERIAL |
Stable serial path for the Arduino, normally /dev/stick-killer. |
MIN_SCAN_FAILURES |
Number of recent scan failures required to trigger recovery. |
MIN_STALE_EVENTS |
Number of recent WPA churn events required to trigger recovery. |
UNHEALTHY_TIMEOUT |
Time a non-connected interface may remain unhealthy before recovery. |
SOFT_RECOVERY_COOLDOWN |
Minimum time in seconds between soft recovery attempts. |
RECOVERY_COOLDOWN |
Minimum time in seconds between hard relay recovery attempts. |
Inspect the watchdog service with:
sudo systemctl status wifi-watchdog.service --no-pager
sudo journalctl -u wifi-watchdog.service -fYou can trigger a real forced hard recovery test with:
sudo touch /run/wifi-watchdog/force-dead-
For a technical overview of the wifi-killer firmware, please refer to docs/firmware.pdf.
-
For a technical overview of the wifi-watchdog service, please refer to docs/watchdog.pdf
Much of this project was written with the help of AI, or "vibe coded". The firmware and watchdog code are tested against a test suite (see test/).
Nevertheless, this disclaimer is here to warn you that slop code is to be expected.