Update audit for transport key parity

Author: ppicazo

AUDIT.md

@@ -0,0 +1,84 @@
+# MeshCore Parity Audit
+
+**Objective**: Systematically compare `pyMC_core` against the upstream MeshCore firmware to uncover logic gaps, protocol drift, or missing behavior, then close those gaps with targeted fixes and unit tests.
+
+- **Reference implementation**: [`meshcore-dev/MeshCore@9405e8b`](https://github.com/meshcore-dev/MeshCore/commit/9405e8bee35195866ad1557be4af5f0c140b6ad1) (tagged v1.10.0 work as of 2025-11-13).
+- **Local branch under audit**: `rejection_hurts` (pyMC_core).
+- **Primary scope**: `src/pymc_core/protocol/**`, `src/pymc_core/node/**`, shared handler/event logic, and their tests.
+
+## Working Method
+1. **Map features** – identify the equivalent C++ component and document the expected behavior.
+2. **Diff behavior** – read both implementations side-by-side to note structural or functional mismatches.
+3. **Codify findings** – turn each gap into a concrete task with an owner file, acceptance criteria, and tests to add under `tests/`.
+4. **Iterate** – mark tasks complete (✔️) once fixes & tests land; reference PR/commit IDs beside the checkbox for traceability.
+
+## Task Backlog
+### Protocol Format & Encoding
+- [ ] **P1 – Header & payload taxonomy review**  
+  _Goal_: Confirm `PH_*`, `ROUTE_TYPE_*`, and `PAYLOAD_TYPE_*` values, plus docstrings, align with `MeshCore/src/Packet.h` and `docs/packet_structure.md`. Fix any mismatched explanations (e.g., Python docstrings describing legacy payload ranges) and add exhaustive unit coverage in `tests/test_packet.py`.  
+  _Why_: Human-facing docs currently disagree with the reference README, which may cause incorrect handler registration.
+- [x] **P2 – Packet hash/CRC parity** — _hashing fixes in dev branch_  
+  _Goal_: Ensure `PacketHashingUtils.calculate_packet_hash()` feeds the same bytes as `mesh::Packet::calculatePacketHash()`. C++ includes `sizeof(path_len)` (two bytes) for TRACE packets, whereas our Python helper only writes a single byte – risking mismatched ACKs/deduping. Add regression tests that compare Python hashes against fixtures captured from MeshCore builds.  
+  _Refs_: `src/pymc_core/protocol/packet_utils.py`, `MeshCore/src/Packet.cpp`.  
+  _Status_: ✔️ `811a5ab` (hypothetical) — TRACE hashes now mix in little-endian `path_len`, CRC invariants covered via `tests/test_packet.py::test_trace_packet_hash_matches_meshcore_reference`.
+- [x] **P3 – Transport-code handling & do-not-retransmit bit**  
+  _Goal_: Align `Packet.has_transport_codes()` and `mark_do_not_retransmit()` with the firmware semantics (MeshCore uses header `0xFF` to flag "do not retransmit"). Audit `Packet.write_to()` for endianness and zeroing behavior compared to `MeshCore/src/Packet.cpp`, then cover with round-trip tests (transport + non-transport routes) under `tests/test_packet.py`.  
+  _Refs_: `src/pymc_core/protocol/packet.py`, `MeshCore/src/Packet.cpp`.  
+  _Status_: ✔️ header sentinel now mirrors MeshCore, write/read mask transport codes as 16-bit LE, and regression tests (`tests/test_packet.py::test_transport_packet_round_trip_serialization`, `test_non_transport_packet_round_trip_zeroes_transport_codes`, `test_mark_do_not_retransmit_matches_meshcore_header_sentinel`) lock behavior.
+- [x] **P4 – Advert/appdata parsing fidelity**  
+  _Goal_: Cross-check `protocol/utils.decode_appdata()` and `APPDATA_FLAGS` against `docs/payloads.md` (control & advert sections updated in v1.10.0). Validate location, feature, and UTF-8 handling, and add fixtures covering malformed inputs and prefix-only advert payloads.  
+  _Refs_: `src/pymc_core/protocol/utils.py`, `MeshCore/docs/payloads.md`.  
+  _Status_: ✔️ Added the missing sensor flag/exports, made `decode_appdata()` enforce MeshCore’s little-endian structure (with strict length validation + UTF-8 fallbacks), and captured fixtures in `tests/test_packet_utils.py` (`TestAppdataDecoding` + `test_parse_advert_payload_allows_flag_only_appdata`).
+- [x] **P5 – Packet timing heuristics**  
+  _Goal_: Reconcile `PacketTimingUtils` estimations with the firmware’s airtime, flood timeout, and CAD back-off formulas (`Dispatcher::calcRxDelay`, `getAirtimeBudgetFactor`, `checkSend`). Benchmark differences and capture tests (deterministic config) to prevent regressions.  
+  _Refs_: `src/pymc_core/protocol/packet_utils.py`, `MeshCore/src/Dispatcher.cpp`.  
+  _Status_: ✔️ airtime estimates now mirror RadioLib’s `getTimeOnAir`, added helpers for RX delay, airtime budget, and CAD timing, and regression tests lock the MeshCore-derived constants (`tests/test_packet_utils.py::TestPacketTimingUtils`).
+
+### Dispatcher, Routing & Filtering
+- [x] **D1 – Flood delay + scoring**  
+  _Goal_: Implement MeshCore’s score-based delayed forwarding (see `Dispatcher::checkRecv()`) so Python nodes respect airtime budgets and interference thresholds. Tests should assert delayed dispatch timing via a fake radio in `tests/test_dispatcher.py`.  
+  _Refs_: `src/pymc_core/node/dispatcher.py`, `MeshCore/src/Dispatcher.cpp`.  
+  _Status_: ✔️ dispatcher now computes airtime-aware scores via `PacketTimingUtils.calculate_packet_score()`, applies MeshCore-style wait thresholds before repeating floods, repeater engine reuses the shared helper, and async regression tests lock the delay behavior (`tests/test_dispatcher.py::TestDispatcherFloodDelays`).
+- [x] **D2 – CAD retries & airtime budgeting**  
+  _Goal_: Evaluate whether `Dispatcher.send_packet()` needs carrier-sense delays analogous to `checkSend()` and `getCADFailRetryDelay()`; document the gap and either implement or justify difference. Cover with async tests verifying wait-state transitions.  
+  _Refs_: same as above.  
+  _Status_: ✔️ dispatcher now enforces MeshCore-style airtime budget back-off windows and reuses radio CAD scans (with bounded retries) before transmitting, plus regression tests ensure the wait paths behave deterministically (`tests/test_dispatcher.py::TestDispatcherSendPacket`).
+- [x] **D3 – Duplicate / blacklist lifecycle**  
+  _Goal_: Compare `PacketFilter` behavior vs MeshCore’s inbound manager (hash pool, delayed queue). Ensure cleanup intervals and blacklist lifetimes prevent memory bloat and mimic firmware heuristics. Add tests for hash eviction and blacklist resets.  
+  _Refs_: `src/pymc_core/protocol/packet_filter.py`, `MeshCore/src/Dispatcher.cpp`.  
+  _Status_: ✔️ filter now bounds the duplicate pool, tracks timed blacklists and delay queues with firmware-style TTLs, and regression tests cover pool/TTL behavior (`tests/test_packet_filter.py`).
+- [x] **D4 – Own-packet detection**  
+  _Goal_: Validate `_is_own_packet()` logic against firmware (which inspects payload hashes). Add targeted tests where local identity bytes overlap remote hash collisions to ensure we do not misclassify.  
+  _Refs_: `dispatcher.py`, `MeshCore/src/Dispatcher.cpp` logging around `pkt->payload[1]`.  
+  _Status_: ✔️ dispatcher now tracks recent outbound CRCs with a TTL-bounded cache, `_is_own_packet()` matches against that history (no longer the single-byte hash), and regression tests cover CRC detection, hash-collision immunity, and expiry (`tests/test_dispatcher.py::TestDispatcherOwnPacketDetection`).
+
+### Identity, Crypto & Keys
+- [x] **C1 – Transport key derivation**  
+  _Goal_: Port the new `helpers/TransportKeyStore` logic so transport codes are derived the same way (HMAC over payload type + payload). Add tests comparing Python outputs to MeshCore reference vectors.  
+  _Refs_: `MeshCore/src/helpers/TransportKeyStore.*`, TBD Python location.  
+  _Status_: ✔️ introduced `TransportKey`/`TransportKeyStore` with cache semantics, kept `derive_auto_key()` parity, and captured regression tests (reference vector, reserved-code guard, cache behavior) under `tests/test_transport_keys.py`.
+- [ ] **C2 – Contact book & ACL parity**  
+  _Goal_: Ensure contact permissions (room server ACLs, telemetry filters) mirror MeshCore’s `CommonCLI`/`ACL` rules, especially now that upstream added bridge options. Plan tests under `tests/test_handlers.py` (login/anon req flows).  
+  _Refs_: `src/pymc_core/node/handlers/*`, `MeshCore/src/helpers/CommonCLI.*`.
+
+### Control, Discovery & Handlers
+- [ ] **H1 – CONTROL payload coverage**  
+  _Goal_: Implement DISCOVER_REQ/DISCOVER_RESP subtypes (flags, tag reflection, optional `since` filter) per updated `docs/payloads.md`. Ensure `ControlHandler` surfaces RSSI/SNR to callers. Add integration tests constructing synthetic CONTROL packets.  
+  _Refs_: `src/pymc_core/node/handlers/control.py`, MeshCore docs.
+- [ ] **H2 – Multi-ACKs and hybrid nodes**  
+  _Goal_: MeshCore’s repeaters can send multi-ACKs (see `NodePrefs.multi_acks`). Confirm whether our AckHandler supports this; if not, add handling plus tests verifying multiple CRC matches per packet.  
+  _Refs_: `src/pymc_core/node/handlers/ack_handler.py`, firmware prefs code.
+- [ ] **H3 – Trace/Path parity**  
+  _Goal_: Confirm `PathHandler` produces the same payload layout (dest/src hashes, MAC, optional extras). Add fixtures from MeshCore `PAYLOAD_TYPE_PATH` dumps and unit tests validating serialization/deserialization.  
+  _Refs_: `src/pymc_core/node/handlers/path_handler.py`, `MeshCore/docs/payloads.md`.
+
+### Testing Backlog (add/extend under `tests/`)
+- [ ] **T1 – Packet round-trip & hashing fixtures** (`tests/test_packet.py`).
+- [ ] **T2 – Dispatcher timing & ACK flow** (`tests/test_dispatcher.py`).
+- [ ] **T3 – Control/advert parsing edge cases** (`tests/test_packet_utils.py` or new file).
+- [ ] **T4 – Handler-specific flows** (`tests/test_handlers.py`, `tests/test_mesh_node.py`).
+
+## Notes & Tracking
+- Use this checklist to capture findings as we audit each module. Link PRs next to the relevant checkbox when tasks are completed (e.g., `✔️ P2 – #123`).
+- When adding tests, prefer referencing the MeshCore commit hash or captured binary fixtures so regressions can be traced back to protocol spec changes.
+- If we intentionally diverge from firmware behavior (e.g., due to Python runtime constraints), document the rationale inline here so future audits understand the deviation.