Hardware time distribution: beacon TX-egress TSF + TsfSync, and the measured PTP reality

.github/workflows/cmake-multi-platform.yml

@@ -149,7 +149,7 @@ jobs:
           ToneMaskSelftest BfReportDecodeSelftest SweepSpecSelftest
           TxPowerQuantSelftest LinkHealthSelftest TxCapsSelftest TxPktPwrSelftest
           RxQualitySelftest AdapterHealthSelftest LogEventSelftest
-          AdapterCapsSelftest TdmaTsfSelftest TimesyncSelftest
+          AdapterCapsSelftest TdmaTsfSelftest TimesyncSelftest TsfSyncSelftest
 
       - name: Test
         working-directory: build

CMakeLists.txt

@@ -564,6 +564,16 @@ target_link_libraries(AdapterCapsSelftest PRIVATE devourer)
 
 add_test(NAME adapter_caps_derive COMMAND AdapterCapsSelftest)
 
+# Headless guard for the TsfSync one-way time-distribution fit (src/TsfSync.h):
+# skew/offset recovery from beacon {egress, arrival} TSF pairs, clock
+# translation, the local 32-bit TSF wrap, and the Packet::TxEgressTsf pick.
+add_executable(TsfSyncSelftest
+    tests/tsf_sync_selftest.cpp
+)
+target_link_libraries(TsfSyncSelftest PRIVATE devourer)
+
+add_test(NAME tsf_sync_math COMMAND TsfSyncSelftest)
+
 # Headless guard for the Jaguar2 per-packet TX-power quantizer
 # (jaguar2::txpkt_pwr_step_for_db) — dB delta -> the descriptor TXPWR_OFSET LUT
 # step. Gated on Jaguar2 being built (the header is behind DEVOURER_HAVE_JAGUAR2).

docs/logging.md

@@ -86,7 +86,7 @@ Emitters: L = library, RX/TX/... = demo. Optional fields in [brackets];
 | ev | emitter | fields |
 |---|---|---|
 | `rx.pkt` | RX | n, len (first 10 + every 100th frame) |
-| `rx.frame` | RX (`DEVOURER_STREAM_OUT`), duplex | rate, len, crc, icv, rssi[2], evm[2], snr[2], seq, tsfl, bw, stbc, ldpc, sgi, body hex |
+| `rx.frame` | RX (`DEVOURER_STREAM_OUT`), duplex | rate, len, crc, icv, rssi[2], evm[2], snr[2], seq, tsfl, bw, stbc, ldpc, sgi, body hex; `tx_tsf` (sender's hardware egress TSF) on beacons/probe-responses only |
 | `rx.body` | RX (`DEVOURER_DUMP_BODY`) | rate, rssi[2], evm[2], snr[2], crc, len, body hex |
 | `rx.corrupt` | RX (`DEVOURER_RX_DUMP_ALL`) | len, crc, icv, rate, bw, stbc, ldpc, sgi, rssi[2], evm[2], snr[2] |
 | `rx.txhit` | RX, TX | hits, total_rx, len — canonical-SA (57:42:75:05:d6:00) matcher |

docs/timing-accuracy.md

@@ -15,14 +15,16 @@ methodology, and the honest caveats. Reproduce the transport half with the
   `chronyd -Q` on the client measures the offset to a `local`-clock chrony server
   on the other node over the Wi-Fi path, without disciplining the clock. Software
   timestamps (see PTP below).
-- **PTP**: attempted with `ptp4l -S` on the same link.
+- **PTP**: `ptp4l` on the same link. The hardware-timestamped mode PTP needs
+  for its precision is unsupported (`ethtool -T` → `PTP Hardware Clock: none`),
+  so it cannot start; software mode (`-S`) is no better than the NTP row.
 
 ## Results
 
 | Method | Precision over Wi-Fi (measured) | Timestamp source |
 |---|---|---|
 | **NTP** (`chrony`, software TS) | **~0.76 ms** RMS (quiet link); tens of ms under power-save/contention | app/kernel software |
-| **PTP** (`ptp4l`) | **does not run** — see below | (would need HW/driver TS) |
+| **PTP** (`ptp4l`) on the Wi-Fi radio | **no hardware mode** — stock driver exposes no HW timestamps; TX egress unavailable even patched (see below) | 802.11 MAC (RX only) |
 | **devourer TSF downlink** | **~0.25 µs** RMS residual | 802.11 MAC hardware (TSF) |
 
 So the TSF downlink is roughly **~3000×** tighter than NTP over the same Wi-Fi,
@@ -42,18 +44,121 @@ NTP/PTP. devourer reads that per-frame RX TSF (`RxPacket.tsfl`) and the 64-bit
 timer directly — which is exactly what a PTP hardware-timestamping NIC would
 provide.
 
-### Why PTP can't run here at all
+### Why stock PTP can't run on the Wi-Fi radio
+
+Out of the box, `ptp4l` reports *"interface does not support requested
+timestamping mode"* and `ethtool -T` shows `PTP Hardware Clock: none`: neither the
+vendor drivers under `reference/` nor the in-tree `rtw88` expose the hardware
+timestamp through the kernel timestamping API, even though both read the TSF
+internally (the RX-descriptor `TSFL`). The hardware timestamp exists; the
+OS-facing drivers just don't surface it — which is what devourer does
+(`RxPacket.tsfl`).
+
+But surfacing the RX timestamp is only half of PTP. Separating the two halves is
+what makes the picture honest:
+
+- **RX** is a genuine per-frame MAC timestamp, latched below the contention
+  layer, and it *can* be surfaced through the standard kernel API — attaching it
+  to each received frame lets an ordinary `SO_TIMESTAMPING` socket read it, and
+  it advances in step with real airtime. This holds on both transports (the USB
+  full-MAC path and the PCIe/`rtw88` `mac80211` path).
+- **TX egress** — the instant the frame actually leaves the antenna — is the
+  wall. The Realtek firmware reports no on-chip TX-egress timestamp for data
+  frames, so a two-way PTP endpoint could only guess the send instant from a
+  host- or bus-completion time, reintroducing exactly the jitter PTP exists to
+  remove. This is a firmware limitation, not a driver one.
+
+So PTP fails on the Wi-Fi radio for the TX half, not the clock. For contrast, a
+dedicated PTP NIC on the *same board* — the Intel I226 Ethernet on the Radxa X4 —
+runs `ptp4l` as a hardware grandmaster with **both** TX and RX hardware
+timestamps and its own PHC. It has the TX-egress timestamp the Wi-Fi silicon
+does not.
+
+### The TX-egress wall — can it be cracked?
+
+The one-way broadcast below is the ceiling because no Realtek part reports *when
+a frame actually leaves the antenna*. This was checked across the vendor and
+in-tree drivers, the firmware TX feedback, the 802.11 ranging path, and the
+literature; the wall is in the silicon, not the driver:
+
+- **The only TX feedback Realtek firmware emits is an ACK-status report, not a
+  timestamp.** The rtw88/rtw89 C2H "TX report" carries transmit status, a retry
+  count and a sequence number — no TSF or egress-time field — and it feeds the
+  standard mac80211 ACK-status path, not a timestamp path.
+- **The mac80211/cfg80211 hardware-timestamping framework exists** (kernel
+  ~6.5+), but it is scoped to Timing-Measurement / Fine-Timing-Measurement
+  action frames, and only Intel and NXP drivers implement it; the Realtek
+  drivers register none of the required operations.
+- **Fine Timing Measurement is not a backdoor.** Where Realtek exposes ranging
+  at all it returns only a firmware-computed distance — the underlying TX/RX
+  timestamps are never surfaced.
+- **The one commodity Wi-Fi chip that _can_ do two-way hardware PTP is the
+  Atheros AR9388 (ath9k)**, which latches the TSF on both TX and RX and reports
+  the TX-status TSF to the host — the exact capability Realtek withholds. That
+  is the lone published exception, not a general result.
+- **Beacons are the one exception — and they prove the rule.** The Realtek MAC
+  *does* hardware-stamp a single class of frame at egress: it inserts the live
+  TSF into a beacon's timestamp field at the instant of transmission. Measured
+  against an independent receiver's hardware RX timestamp, that egress stamp
+  lands at **0.289 µs** RMS. But it is embedded in the outgoing frame for
+  receivers to read, not reported back to the host — so it powers a one-way
+  broadcast, not a two-way endpoint.
+
+So two-way `ptp4l` over Wi-Fi needs ath9k-class silicon. On Realtek the honest
+ceiling is a receive-only reference-broadcast scheme — timestamp a periodic AP
+broadcast at reception via the TSF, no transmit timestamp required — which is
+what devourer's TSF downlink already is, and what the published state of the art
+for commodity Wi-Fi (OpenWiFiSync / RBIS) independently settled on, at
+one-to-two-digit-µs accuracy.
+
+The tempting host-side shortcut — reading the TSF register right after handing a
+frame to the radio, as a stand-in for egress — does not close the gap. Measured
+against a witness receiver's hardware RX timestamp, that read lands **~117 µs**
+RMS from true air on a quiet 5 GHz channel and **~473 µs** on a busy 2.4 GHz
+one — 400–1600× the 0.289 µs the beacon's hardware egress stamp achieves,
+because it inherits exactly the USB-queue and CSMA-backoff jitter it was meant to
+avoid. So the broadcast path (hardware-stamped beacon, read at reception) is not
+just convenient — it is the only sub-µs option on this silicon.
+
+If a future firmware or driver ever surfaces a TX-egress timestamp, this wall
+falls and the RX timestamping already in place becomes a full two-way endpoint.
+
+Evidence behind the current state (so it can be re-checked if the situation
+changes):
+
+- rtw88/rtw89 C2H TX-report contents — transmit status / retry count only, no
+  timestamp:
+  <https://lkml.org/lkml/2025/9/20/269>,
+  <https://lists.openwall.net/linux-kernel/2025/11/04/1050>
+- rtw89 registers no `set_hw_timestamp` / TSF timestamping ops:
+  <https://github.com/lwfinger/rtw89/blob/main/mac80211.c>
+- mac80211 hardware-timestamping API, scoped to TM/FTM frames (Intel/NXP only):
+  <https://patchwork.kernel.org/project/linux-wireless/patch/20230223114629.2286397aa296.Iccc08869ea8156f1c71a3111a47f86dd56234bd0@changeid/>,
+  <https://docs.kernel.org/driver-api/80211/mac80211.html>,
+  <https://github.com/torvalds/linux/blob/master/include/net/mac80211.h>
+- Realtek ranging HAL returns only a firmware-computed distance (no t1–t4/TSF):
+  <https://github.com/kimocoder/android_wifi_hal/blob/master/hardware/realtek/wlan/wifi_hal/rtw_wifi_rtt.cpp>
+- Atheros ath9k hardware-PTP-over-Wi-Fi precedent (USENIX ATC '21):
+  <https://www.usenix.org/system/files/atc21-chen.pdf>
+- OpenWiFiSync / RBIS one-way broadcast sync (IEEE ETFA 2024):
+  <https://arxiv.org/pdf/2410.08742>
+
+### Wi-Fi TSF vs real PTP hardware — measured on one board
+
+Because that I226 is a true IEEE-1588 clock, it doubles as a local reference:
+cross-comparing the Wi-Fi MAC TSF against it — both exposed as PTP hardware
+clocks and disciplined slave-to-master on the same machine, no network in
+between — puts a number on how good the TSF really is.
+
+| Quantity | Measured | Reading |
+|---|---|---|
+| TSF vs I226 offset | **~290 ns** RMS (mean ≈ 0) | sub-µs tracking of genuine PTP hardware |
+| TSF crystal offset | **~+42 ppm** | the Wi-Fi reference oscillator's raw frequency error |
 
-`ptp4l` reports *"interface does not support requested timestamping mode"*, and
-`ethtool -T` shows `PTP Hardware Clock: none` with no timestamp modes. This is not
-specific to the in-tree `rtw88`: grepping the **vendor** drivers under
-`reference/` for `SOF_TIMESTAMPING` / `SIOCSHWTSTAMP` / `skb_hwtstamps` /
-`get_ts_info` / `ptp_clock` finds **nothing** — neither the vendor nor the
-mainline driver exposes the hardware timestamp through the kernel timestamping
-API, even though both read the TSF internally (`CONFIG_HW_P0_TSF_SYNC`, the
-RX-descriptor `TSFL`). The hardware timestamp exists; the OS-facing drivers just
-don't surface it. devourer does — which is what makes TSF-based sub-µs sync
-possible where PTP returns an error.
+The ~290 ns residual is essentially the TSF's own 1 µs-resolution quantization
+floor (a uniform ±0.5 µs rounds to ~289 ns RMS): once the +42 ppm crystal offset
+is taken out, the Wi-Fi MAC clock is as steady as the reference — the servo has
+nothing left to chase but the last microsecond of resolution.
 
 ## Transport latency — USB vs PCIe (the actuator floor)
 
@@ -72,6 +177,13 @@ actuator collapses from ~0.5–1.2 ms of jitter over USB toward ~µs over PCIe 
 the transport, not the method, is the closed-loop-TA floor, and a PCIe path would
 tighten the ~1.3 ms uplink residual toward tens of µs.
 
+The same floor is what makes a *clock read* practical over PCIe. A PTP hardware
+clock backed by the 8821CE TSF, read over that MMIO window, measures a gettime
+read window of **~3.7 µs** (p99 ~5.6 µs) — cheap and steady enough to discipline
+against, where the equivalent read over USB would cost tens of µs of jittery
+vendor-control transfer. That is why a genuine PHC is tractable on the PCIe part
+and not on the USB ones.
+
 ## Honest caveats
 
 - devourer's ~0.25 µs is the **precision/stability** of a *one-way* broadcast fit

examples/common/caps_event.h

@@ -67,6 +67,8 @@ inline void emit_adapter_caps(EventSink &sink, IRtlDevice *dev) {
       .f("narrowband", c.narrowband_ok ? 1 : 0)
       .f("fastretune", c.fastretune_ok ? 1 : 0)
       .f("per_chain_rssi", c.per_chain_rssi ? 1 : 0)
+      .f("hw_rx_tsf", c.hw_rx_timestamp ? 1 : 0)
+      .f("hw_beacon_txtsf", c.hw_beacon_txtsf ? 1 : 0)
       .f("xtal_cap_max", c.xtal_cap_max)
       .f("xtal_cap_default", c.xtal_cap_default);
 }

examples/rx/main.cpp

@@ -484,8 +484,8 @@ static void packetProcessor(const Packet &packet) {
         const int snr[2] = {packet.RxAtrib.snr[0], packet.RxAtrib.snr[1]};
         const size_t body_len =
             packet.Data.size() > 24 ? packet.Data.size() - 24 : 0;
-        devourer::Ev(*g_ev, "rx.frame")
-            .f("rate", packet.RxAtrib.data_rate)
+        auto ev = devourer::Ev(*g_ev, "rx.frame");
+        ev.f("rate", packet.RxAtrib.data_rate)
             .f("len", packet.Data.size())
             .f("crc", packet.RxAtrib.crc_err ? 1 : 0)
             .f("icv", packet.RxAtrib.icv_err ? 1 : 0)
@@ -497,8 +497,13 @@ static void packetProcessor(const Packet &packet) {
             .f("bw", packet.RxAtrib.bw)
             .f("stbc", packet.RxAtrib.stbc)
             .f("ldpc", packet.RxAtrib.ldpc)
-            .f("sgi", packet.RxAtrib.sgi)
-            .hex("body", packet.Data.data() + 24, body_len);
+            .f("sgi", packet.RxAtrib.sgi);
+        /* tx_tsf: the sender's hardware TX-egress TSF (beacons / probe responses
+         * only). Pair with tsfl — the local hardware RX timestamp above — for
+         * one-way hardware time sync with no host-clock jitter on either end. */
+        if (auto tx = packet.TxEgressTsf())
+          ev.f("tx_tsf", (unsigned long long)*tx);
+        ev.hex("body", packet.Data.data() + 24, body_len);
       }
       if (dump_body && hits <= 5) {
         /* Tier-2 health diagnostics alongside the byte mirror: rate (0x04 =

src/AdapterCaps.h

@@ -119,6 +119,16 @@ struct AdapterCaps {
   uint8_t xtal_cap_default = 0;    /* efuse/default crystal-cap code */
   bool fastretune_ok = false;      /* lean FastRetune override exists */
   bool per_chain_rssi = false;     /* frame parser fills per-chain rssi (>=2ch) */
+  /* Hardware timing. hw_rx_timestamp: every received frame is stamped with the
+   * MAC's microsecond TSF at receive (RxPacket.RxAtrib.tsfl) — true on all
+   * generations. hw_beacon_txtsf: this adapter, as a transmitter, inserts its
+   * live hardware TSF into the beacons it airs at the instant of transmission
+   * (a genuine sub-µs TX-egress timestamp a receiver reads via
+   * Packet::TxEgressTsf) — requires the hardware beacon function (StartBeacon),
+   * so Jaguar2/Jaguar3 only. Together they are the primitives for one-way
+   * hardware time distribution (see TsfSync). */
+  bool hw_rx_timestamp = false;
+  bool hw_beacon_txtsf = false;
 };
 
 inline void set_standard_freq_ranges(AdapterCaps &c) {

src/RxPacket.h

@@ -2,6 +2,7 @@
 #define DEVOURER_RX_PACKET_H
 
 #include <cstdint>
+#include <optional>
 #include <span>
 
 /* Parsed RX packet types, shared across chip generations. The per-generation
@@ -71,6 +72,35 @@ struct Packet
 {
     rx_pkt_attrib RxAtrib;
     std::span<uint8_t> Data;
+
+    /* The transmitter's hardware TX-egress TSF, when the frame carries one.
+     *
+     * For the frames the 802.11 MAC regenerates itself — beacons and probe
+     * responses — the sender's MAC overwrites the 8-byte timestamp field (MPDU
+     * bytes 24-31) with its live TSF at the instant the frame is clocked onto
+     * the air. That is a genuine hardware TX-egress timestamp, latched below the
+     * CSMA/queueing layer (bench-measured sub-µs against an independent
+     * receiver, vs ~100+ µs for any host-side "read the clock after send"
+     * approximation). Paired with RxAtrib.tsfl (this receiver's hardware RX
+     * timestamp), each such frame yields the {remote egress, local arrival}
+     * pair that one-way hardware time distribution is built on — with no
+     * host-clock jitter on either end, and against any beaconing AP, not just a
+     * devourer transmitter.
+     *
+     * Returns nullopt for frames the MAC does not stamp (data frames and mgmt
+     * frames other than beacon / probe-response). */
+    std::optional<uint64_t> TxEgressTsf() const
+    {
+        if (Data.size() < 32)
+            return std::nullopt;
+        const uint8_t fc0 = Data[0];      /* proto=0, type=mgmt, subtype in [7:4] */
+        if (fc0 != 0x80 /* beacon */ && fc0 != 0x50 /* probe response */)
+            return std::nullopt;
+        uint64_t tsf = 0;
+        for (int i = 0; i < 8; ++i)
+            tsf |= static_cast<uint64_t>(Data[24 + i]) << (8 * i);
+        return tsf;
+    }
 };
 
 #endif /* DEVOURER_RX_PACKET_H */