Timing accuracy: NTP/PTP-over-Wi-Fi comparison + USB-vs-PCIe transport-latency bench

CMakeLists.txt

@@ -302,6 +302,12 @@ if(DEVOURER_PCIE)
     target_link_libraries(pcieprobe PUBLIC devourer)
 endif()
 
+# reglat — register round-trip latency microbench (USB ctrl-xfer vs PCIe MMIO).
+# Times RtlAdapter::rtw_read32; the PCIe path compiles only with DEVOURER_PCIE
+# (inherited via the devourer PUBLIC DEVOURER_HAVE_PCIE define). See its header.
+add_executable(reglat tests/reglat.cpp)
+target_link_libraries(reglat PUBLIC devourer PRIVATE PkgConfig::libusb)
+
 add_executable(rxdemo
     examples/rx/main.cpp
     examples/common/env_config.cpp

README.md

@@ -191,6 +191,9 @@ one `IRtlDevice` interface covers all three generations.
 - [Time distribution](docs/time-distribution.md) — LTE-eNB-style over-the-air
   clock distribution off the hardware beacon TSF: sub-µs downlink, TSF adoption,
   µs-fine TBTT steering and a converging closed-loop uplink timing advance.
+- [Timing accuracy](docs/timing-accuracy.md) — measured comparison vs NTP/PTP over
+  Wi-Fi (why the hardware TSF beats software timestamps ~3000×, why PTP can't run),
+  and the USB-vs-PCIe transport-latency microbench (`tests/reglat.cpp`).
 - [AP mode](docs/ap-mode.md) — devourer as an infrastructure access point a real
   Linux station associates with: beacon → probe/auth/assoc → DHCP/ARP/ICMP → ping,
   open or WPA2-PSK (4-way handshake + software CCMP), validated against rtw88.

docs/timing-accuracy.md

@@ -0,0 +1,87 @@
+# Timing accuracy — devourer's TSF sync vs NTP / PTP over Wi-Fi
+
+A common question: how does devourer's hardware-TSF time distribution
+(`docs/time-distribution.md`) compare with running NTP or PTP over the same
+Wi-Fi? Below are **measured** numbers from a two-node bench (this rig), the
+methodology, and the honest caveats. Reproduce the transport half with the
+`reglat` microbench; the NTP half with `chronyd -Q` as described.
+
+## Bench
+
+- **devourer TSF downlink**: master beacon TSF (inserted by the MAC at TX) vs
+  slave arrival `tsfl` (latched by the MAC at RX), least-squares fit over many
+  beacons. Both timestamps are taken *in the MAC hardware*.
+- **NTP**: two separate machines associated to the same AP (WPA2, 5 GHz);
+  `chronyd -Q` on the client measures the offset to a `local`-clock chrony server
+  on the other node over the Wi-Fi path, without disciplining the clock. Software
+  timestamps (see PTP below).
+- **PTP**: attempted with `ptp4l -S` on the same link.
+
+## Results
+
+| Method | Precision over Wi-Fi (measured) | Timestamp source |
+|---|---|---|
+| **NTP** (`chrony`, software TS) | **~0.76 ms** RMS (quiet link); tens of ms under power-save/contention | app/kernel software |
+| **PTP** (`ptp4l`) | **does not run** — see below | (would need HW/driver TS) |
+| **devourer TSF downlink** | **~0.25 µs** RMS residual | 802.11 MAC hardware (TSF) |
+
+So the TSF downlink is roughly **~3000×** tighter than NTP over the same Wi-Fi,
+and the reason is the same one that stops PTP entirely.
+
+## Why: the Wi-Fi MAC jitter, and the one timestamp that's immune
+
+NTP/PTP precision over Wi-Fi is bounded by the **round-trip delay variation** of
+the medium. Measured on this link (ICMP, node↔node): RTT swung **1.6 ms → 110 ms**
+(mean 44 ms, mdev 45 ms) under load, ~0.3 ms mdev when quiet — the CSMA
+backoff / retransmit / power-save latency of the 802.11 MAC. Software timestamps
+sit *above* that layer, so they inherit all of it.
+
+The 802.11 **TSF** is latched *inside* the MAC at the instant of TX/RX, *below*
+that queueing/contention layer, so it is immune to the jitter that limits
+NTP/PTP. devourer reads that per-frame RX TSF (`RxPacket.tsfl`) and the 64-bit
+timer directly — which is exactly what a PTP hardware-timestamping NIC would
+provide.
+
+### Why PTP can't run here at all
+
+`ptp4l` reports *"interface does not support requested timestamping mode"*, and
+`ethtool -T` shows `PTP Hardware Clock: none` with no timestamp modes. This is not
+specific to the in-tree `rtw88`: grepping the **vendor** drivers under
+`reference/` for `SOF_TIMESTAMPING` / `SIOCSHWTSTAMP` / `skb_hwtstamps` /
+`get_ts_info` / `ptp_clock` finds **nothing** — neither the vendor nor the
+mainline driver exposes the hardware timestamp through the kernel timestamping
+API, even though both read the TSF internally (`CONFIG_HW_P0_TSF_SYNC`, the
+RX-descriptor `TSFL`). The hardware timestamp exists; the OS-facing drivers just
+don't surface it. devourer does — which is what makes TSF-based sub-µs sync
+possible where PTP returns an error.
+
+## Transport latency — USB vs PCIe (the actuator floor)
+
+The closed-loop *uplink* timing advance is limited by the actuator, not the sync
+measurement: `AdjustBeaconTimingFine` is a TSF read-modify-write of ~5–7 register
+ops, and over USB each op is a vendor-control transfer. `tests/reglat.cpp` times a
+single `RtlAdapter::rtw_read32` over each transport:
+
+| Transport | mean / op | jitter (p99−p50) |
+|---|---|---|
+| USB (8822EU) | ~68 µs | ~64 µs |
+| **PCIe 8821CE (MMIO)** | **~2.2 µs** | **~0.08 µs** |
+
+PCIe MMIO is **~30× faster and ~600–900× lower jitter** per op, so the chained
+actuator collapses from ~0.5–1.2 ms of jitter over USB toward ~µs over PCIe — i.e.
+the transport, not the method, is the closed-loop-TA floor, and a PCIe path would
+tighten the ~1.3 ms uplink residual toward tens of µs.
+
+## Honest caveats
+
+- devourer's ~0.25 µs is the **precision/stability** of a *one-way* broadcast fit
+  (relative clock + slave-to-slave common view), not an independently-verified
+  *absolute* offset. NTP/PTP are two-way and cancel path asymmetry; the TSF is
+  1 µs resolution, so sub-µs comes from averaging the fit over many beacons.
+- The NTP figure here is `chrony`'s offset over a *healthy* link; it degrades to
+  tens of ms under Wi-Fi power-save/contention (the RTT data above). It is also
+  the practical daemon result, close to the RTT-jitter bound that NTP cannot beat.
+- FTM (802.11mc) round-trip ranging, where a driver exposes it, can beat a one-way
+  µs-resolution TSF on *absolute* accuracy; devourer's advantage is availability
+  (works on stock silicon today) and broadcast common-view, not a fundamentally
+  finer clock than dedicated PTP hardware.

tests/reglat.cpp

@@ -0,0 +1,88 @@
+// reglat.cpp — register round-trip latency of the transport: USB vendor-control
+// transfer vs PCIe BAR2 MMIO. Times RtlAdapter::rtw_read32 in a tight loop — the
+// SAME call on both buses, so it's a fair USB-vs-PCIe comparison of the per-op
+// latency and jitter that bound AdjustBeaconTimingFine (read-modify-write of the
+// TSF) and hence the closed-loop uplink timing advance. No bring-up: the read
+// works right after claim (USB) / vfio-map (PCIe).
+//
+//   USB:  sudo ./build/reglat                    (DEVOURER_PID/VID pick the adapter)
+//   PCIe: sudo DEVOURER_PCIE_BDF=0000:01:00.0 ./build/reglat   (vfio-bind first:
+//         sudo tests/pcie_vfio_bind.sh 0000:01:00.0)
+//
+// Bench numbers (this rig): USB 8822EU ~66 us/op (jitter p99-p50 ~48 us); PCIe
+// 8821CE MMIO ~2.2 us/op (jitter ~0.09 us) — ~30x faster, ~600x lower jitter.
+#include <algorithm>
+#include <chrono>
+#include <cstdint>
+#include <cstdio>
+#include <cstdlib>
+#include <memory>
+#include <vector>
+#include <libusb.h>
+#include "RtlAdapter.h"
+#include "UsbOpen.h"
+#include "logger.h"
+#if defined(DEVOURER_HAVE_PCIE)
+#include "PcieTransport.h"
+#endif
+
+static int N = 3000;
+static constexpr uint32_t kReg = 0x00F0;  // SYS_CFG1 — safe read-only, any state
+
+static void report(const char *what, RtlAdapter &a) {
+  std::vector<double> us(N);
+  for (int i = 0; i < N; i++) {
+    auto t0 = std::chrono::steady_clock::now();
+    volatile uint32_t v = a.rtw_read32(kReg);
+    (void)v;
+    us[i] = std::chrono::duration<double, std::micro>(
+                std::chrono::steady_clock::now() - t0)
+                .count();
+  }
+  std::sort(us.begin(), us.end());
+  double sum = 0;
+  for (double x : us) sum += x;
+  auto q = [&](double p) { return us[(int)(p * N)]; };
+  printf("%s rtw_read32(0x%04x) x%d: mean=%.3f p50=%.3f p90=%.3f p99=%.3f "
+         "min=%.3f max=%.3f us  jitter(p99-p50)=%.3f\n",
+         what, kReg, N, sum / N, q(.50), q(.90), q(.99), us.front(), us.back(),
+         q(.99) - q(.50));
+}
+
+int main() {
+  auto lg = std::make_shared<Logger>();
+  if (const char *n = std::getenv("DEVOURER_REGLAT_N")) {
+    int nn = atoi(n);  // (not std::max: <windows.h> defines a max() macro on MSVC)
+    N = nn > 100 ? nn : 100;
+  }
+
+#if defined(DEVOURER_HAVE_PCIE)
+  if (const char *bdf = std::getenv("DEVOURER_PCIE_BDF")) {
+    auto t = devourer::PcieTransport::Open(bdf, lg);
+    if (!t) {
+      fprintf(stderr, "pcie open failed for %s (vfio-bound?)\n", bdf);
+      return 1;
+    }
+    RtlAdapter a(t, lg, {});
+    report("PCIe MMIO", a);
+    return 0;
+  }
+#endif
+
+  libusb_context *ctx = nullptr;
+  libusb_init(&ctx);
+  libusb_set_option(ctx, LIBUSB_OPTION_LOG_LEVEL, LIBUSB_LOG_LEVEL_WARNING);
+  uint16_t vid = 0x0bda, pid = 0;
+  if (const char *v = std::getenv("DEVOURER_VID")) vid = (uint16_t)strtoul(v, 0, 0);
+  if (const char *p = std::getenv("DEVOURER_PID")) pid = (uint16_t)strtoul(p, 0, 0);
+  auto *h = libusb_open_device_with_vid_pid(ctx, vid, pid);
+  if (!h) {
+    fprintf(stderr, "usb open %04x:%04x failed (set DEVOURER_PID)\n", vid, pid);
+    return 1;
+  }
+  std::shared_ptr<devourer::UsbDeviceLock> lk;
+  if (devourer::claim_interface_then_reset(h, 0, lg, true, lk) != 0) return 1;
+  RtlAdapter a(h, lg, ctx, lk, {});
+  report("USB ctrl-xfer", a);
+  return 0;
+}