Skip to content

Commit 645f0b0

Browse files
author
ainish-coder bot
committed
docs: introduce new documentation for changelog, data models, deployment, and frontend, alongside updates to architecture, correlation engine, and quick reference.
1 parent 697b994 commit 645f0b0

File tree

7 files changed

+3294
-454
lines changed

7 files changed

+3294
-454
lines changed

‎.signals/signals-architecture.md‎

Lines changed: 17 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -484,23 +484,24 @@ the same physical device using different MAC addresses (randomization).
484484
485485
### Summary
486486

487-
The `SignalCorrelator` links WiFi and BLE observations from the same
488-
physical device by scoring 7 independent signals:
487+
The `SignalCorrelator` (L1408–L2390 in `app.py`) uses **multi-dimensional
488+
clustering** across 6 phases to link WiFi and BLE observations:
489489

490-
| Signal | Weight | Algorithm |
491-
|--------|--------|-----------|
492-
| OUI Match | 0.10 | Same first 3 octets |
493-
| Probe Overlap | 0.15 | Jaccard of probed SSIDs |
494-
| PNL Match | 0.25 | PNL engine Jaccard similarity |
495-
| Temporal | 0.10 | Co-occurrence within time window |
496-
| RSSI Proximity | 0.25 | RSSI delta < damped tolerance |
497-
| Name Similarity | 0.10 | String matching |
498-
| Address Type | 0.05 | Same address classification |
490+
| Phase | Algorithm | Cluster Types Produced |
491+
|-------|-----------|------------------------|
492+
| 2 | OUI grouping → RSSI band sub-clustering | `manufacturer`, `manufacturer_rssi` |
493+
| 3 | PNL hash match → Jaccard overlap | `pnl_match`, `pnl_overlap`, `randomised` |
494+
| 3b | BT name grouping → RSSI sub-clustering | `bt_name`, `bt_rssi` |
495+
| 4 | WiFi↔BT cross-link (OUI+RSSI+name score ≥ 0.50) | `cross_linked` |
496+
| 5 | Cross-vendor PNL merge (Jaccard ≥ 0.70) | `pnl_cross_vendor` |
499497

500-
Pairs above threshold → union-find clustering → correlation clusters.
498+
RSSI tolerance uses **exponential convergence** (no oscillation):
499+
$$\text{gap}(n) = 1.0 + 19.0 \times 0.92^n$$
501500

502-
RSSI tolerance uses **damped oscillatory convergence**:
503-
$$\Delta(n) = \Delta_{\text{target}} + (\Delta_0 - \Delta_{\text{target}}) \cdot r^n \cdot |\cos(n \cdot \theta)|$$
501+
- WiFi target: **1.0 dBm** · BT target: **5.0 dBm** (WiFi + 4)
502+
- Converges at step 36 (~36 minutes)
503+
- 9 distinct cluster types with semantic meaning
504+
- ~315 clusters from ~250 devices in production
504505

505506
---
506507

@@ -695,4 +696,5 @@ On user actions:
695696

696697
---
697698

698-
*signals-architecture v1.0 | 2026-02 | Creeper Sweeper*
699+
*signals-architecture v2.0 | 2026-02-08 | Creeper Sweeper*
700+
*Updated: multi-dimensional clustering, 9 cluster types, exponential convergence*

‎.signals/signals-changelog.md‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# Changelog — Creeper Sweeper
2+
3+
> **Complete commit history with feature descriptions.**
4+
>
5+
> 36 commits from initial scaffold to HEAD (`7228c8e`).
6+
> All work spans February 7–8, 2026.
7+
8+
---
9+
10+
## Summary Statistics
11+
12+
| Metric | Value |
13+
|--------|-------|
14+
| Total commits | 36 |
15+
| Features | 21 |
16+
| Fixes | 11 |
17+
| Docs | 2 |
18+
| Refactors | 1 |
19+
| Performance | 1 |
20+
| Total source lines | ~11,968 (9 webapp files) |
21+
| Backend (`app.py`) | 3,986 lines |
22+
| Frontend JS | 3,421 lines (app.js + reference.js) |
23+
| Frontend CSS | 2,046 lines |
24+
| Supporting modules | 1,953 lines (bt_scanner + pnl_engine + gps_reader + triangulation) |
25+
26+
---
27+
28+
## Commit Log (Chronological — Oldest First)
29+
30+
### Phase 1 — Foundation (Feb 7, 03:00–04:41)
31+
32+
| Hash | Type | Description |
33+
|------|------|-------------|
34+
| `77f1243` | feat | **Initial scaffold** — HTML structure, CSS dark theme, basic dashboard layout |
35+
| `b949228` | feat(sensors) | **GPS, BT LE, triangulation modules** — `gps_reader.py` (gpsd integration), `bt_scanner.py` (btmgmt wrapper), `triangulation.py` (multi-node RSSI positioning) |
36+
37+
### Phase 2 — Core Features (Feb 7, 21:15–22:27)
38+
39+
| Hash | Type | Description |
40+
|------|------|-------------|
41+
| `cea28c9` | feat | **Surveillance database** — OUI-based surveillance vendor matching, unit tests |
42+
| `0cd98c2` | feat(webapp) | **Single dashboard layout** — Alert sidebar (right column), instant tier alerts, device name removal, 2-column grid |
43+
| `4fd7122` | feat(time) | **NTP + timezone** — `timedatectl` in deploy.sh, Eastern timezone, timezone-aware timestamps throughout |
44+
| `d94d59b` | feat(reference) | **SIGINT reference module** — `reference.js` with ~16 surveillance vendors, 9 sub-tabs of educational content |
45+
| `a790e77` | feat(webapp) | **Live analyzer sidebar** — MAC parser + BT decoder + RSSI distance calculator, dual-panel device modal, expanded vendor DB |
46+
| `1234464` | fix(webapp) | **Layout restoration** — 2-column layout fix, floating analyzer positioning, Reference tab rendering |
47+
48+
### Phase 3 — Correlation & Hardware (Feb 7, 23:02–23:33)
49+
50+
| Hash | Type | Description |
51+
|------|------|-------------|
52+
| `e0d0f93` | feat(correlation) | **Signal correlation system** — Probe request tracking, initial weighted scoring algorithm |
53+
| `eb326a0` | feat(hardware) | **gpsd integration** — `/etc/default/gpsd` config, udev rules for U-blox7 (`/dev/gps0`), hardware docs |
54+
| `7abf4bc` | fix(systemd) | **Template unit fix** — `%i` instead of `%I` to prevent dash-to-slash unescaping in usernames |
55+
| `f8c2559` | fix(systemd) | **Path fix** — Replace `%h` with `/home/%i` for system service WorkingDirectory |
56+
| `660c227` | fix(systemd) | **Capabilities fix** — Remove `NoNewPrivileges`, use `AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW` for btmgmt |
57+
| `1e6f9bc` | fix(systemd) | **Config dir fix** — Pre-create `~/.local/share/creeper-sweeper/` to prevent `NAMESPACE` startup failure |
58+
| `b56bf11` | feat(hardware) | **Hardware auto-detection** — `hardware.py` module, USB VID:PID scanning for GPS/BT/WiFi adapters |
59+
60+
### Phase 4 — Detection Tuning & BT Pipeline (Feb 7 23:33–Feb 8 01:41)
61+
62+
| Hash | Type | Description |
63+
|------|------|-------------|
64+
| `b529187` | feat(correlation) | **Identification workflow** — Cluster identify + rename, bulk ignore (all MACs + SSIDs in cluster) |
65+
| `e581756` | fix(detection) | **Drive-by tuning** — Adjust persist threshold and alert cooldowns for vehicle-speed targets |
66+
| `abc3ff5` | fix(bluetooth) | **BT pipeline fix** — Parse both stdout+stderr from btmgmt, feed BT devices into detection/alert pipeline |
67+
| `8896287` | fix(bluetooth) | **PTY wrapper** — `script -qc "btmgmt find -l" /dev/null` to solve systemd PTY buffering |
68+
| `435ae1a` | docs(signals,llms) | **Documentation** — Comprehensive BT/WiFi scanning documentation in `.signals/` |
69+
70+
### Phase 5 — Dashboard Evolution (Feb 8, 01:57–04:26)
71+
72+
| Hash | Type | Description |
73+
|------|------|-------------|
74+
| `cc70816` | fix(webapp) | **Alert UX** — Auto-dismiss tier 1/2/4 alerts after 5s, recompute confirmation dialog |
75+
| `c5e1601` | feat(webapp) | **GPS-tagged detections** — Attach lat/lon to every detection from live GPS fix |
76+
| `6183e48` | feat(webapp) | **Location fallback** — Use SSID's last-known location when GPS unavailable, boost RSSI correlation |
77+
| `53236d1` | feat(webapp) | **Settings upgrades** — Sound toggle, unconstrained RSSI default (-127 = no limit) |
78+
| `6bdd77d` | feat(correlator) | **RSSI convergence** — Damped oscillatory tolerance convergence (later replaced by pure exponential) |
79+
| `5164384` | feat(webapp) | **Scrollable correlation** — Scrollable cluster cards with `max-height` constraints |
80+
| `155f8f1` | fix(webapp) | **Viewport constraint** — Enforce scrollable layout for all tabs, prevent page overflow |
81+
| `ac0c099` | fix(webapp) | **Dashboard restore** — Scrollable correlation only (not all tabs), popup toggle setting |
82+
| `2576ed6` | fix(webapp) | **Layout fix** — Remove flex from correlation tab, fix clipped cluster cards |
83+
| `9622411` | feat(webapp) | **RSSI setting UX** — Positive dBm scale display in settings, default 127 |
84+
85+
### Phase 6 — Advanced Features (Feb 8, 04:55–06:10)
86+
87+
| Hash | Type | Description |
88+
|------|------|-------------|
89+
| `8b2b753` | feat(detection) | **PNL fingerprinting** — `pnl_engine.py`, BLE address type classification (LE Public/Random/BR-EDR), directed probe extraction, PNL hash computation |
90+
| `09d1f06` | feat(webapp) | **CSV export/import** — 34-column CSV export merging WiFi+BT+fingerprints, CSV import with `action` field (ignore/name/both) |
91+
| `d247af4` | feat(webapp) | **Watch alerts** — WatchList system (7 match types, cooldown, enable/disable), settings persistence (7 fields to JSON), fix min RSSI handling |
92+
| `ce7b49d` | docs(signals) | **Knowledge base update** — Comprehensive `.signals/` documentation refresh |
93+
| `9082ecd` | refactor(correlation) | **Multi-dimensional clustering** — Replace union-find + 7-signal weighted scoring with 6-phase pipeline, 9 cluster types, pure exponential RSSI convergence |
94+
| `7228c8e` | perf(correlator) | **RSSI gap tightening** — `RSSI_GAP_TARGET` 8→1 dBm, deployed to Pi, recomputed 315 clusters |
95+
96+
---
97+
98+
## Key Architectural Milestones
99+
100+
| Commit | Milestone |
101+
|--------|-----------|
102+
| `77f1243` | First HTML/CSS scaffold |
103+
| `b949228` | Sensor modules (GPS, BT, triangulation) |
104+
| `0cd98c2` | Single-page dashboard with alert sidebar |
105+
| `e0d0f93` | First correlation engine |
106+
| `8896287` | BT scanning working under systemd (PTY fix) |
107+
| `6bdd77d` | RSSI convergence algorithm |
108+
| `8b2b753` | PNL fingerprinting engine |
109+
| `9082ecd` | Multi-dimensional clustering (final algorithm) |
110+
| `7228c8e` | Production-tuned RSSI gap (1 dBm) — 315 clusters |
111+
112+
---
113+
114+
## File Evolution
115+
116+
| File | Current Lines | First Commit | Major Changes |
117+
|------|--------------|--------------|---------------|
118+
| `app.py` | 3,986 | `0cd98c2` | Every phase added features; correlation engine is largest section |
119+
| `app.js` | 2,464 | `77f1243` | Started ~200 lines → 92 functions across 6 phases |
120+
| `style.css` | 2,046 | `77f1243` | Dark theme, modals, scrollable layouts, analyzer styles |
121+
| `reference.js` | 957 | `d94d59b` | 16 vendors → 30 vendors (14 added in `a790e77`) |
122+
| `index.html` | 562 | `77f1243` | 9 tabs, 4 modals, CDN imports, cache-busting `?v=6` |
123+
| `pnl_engine.py` | 613 | `8b2b753` | PNL profiling, Jaccard similarity, hash computation |
124+
| `triangulation.py` | 590 | `b949228` | Multi-node RSSI positioning, weighted centroid |
125+
| `bt_scanner.py` | 401 | `b949228` | btmgmt wrapper → PTY fix (`8896287`) → regex parser |
126+
| `gps_reader.py` | 349 | `b949228` | gpsd socket client, fix quality parsing |
127+
| `deploy.sh` | 168 | `4fd7122` | 9-step deploy with timezone, gpsd, udev, systemd |
128+
129+
---
130+
131+
## Deployment History
132+
133+
| Date | Target | Action | Result |
134+
|------|--------|--------|--------|
135+
| 2026-02-07 | Pi `192.168.1.243` | Initial deploy | Dashboard running, WiFi scanning |
136+
| 2026-02-08 | Pi `192.168.1.243` | BT pipeline fix | BT devices appearing in dashboard |
137+
| 2026-02-08 | Pi `192.168.1.243` | Multi-dim clustering + RSSI gap 1 dBm | 315 clusters computed |
138+
139+
---
140+
141+
*signals-changelog v1.0 | 2026-02-08 | Creeper Sweeper*

0 commit comments

Comments
 (0)