Skip to content

Commit 2763819

Browse files
authored
feat: add cable segment utilization module for FritzBox (#188)
* feat: add segment utilization storage class * feat: add segment utilization collector with AVM-SID auth * feat: update fritzbox_cable manifest to integration type with collector * feat: rewrite fritzbox_cable routes to serve stored segment data * feat: add i18n for segment utilization (EN/DE/FR/ES) * feat: rewrite fritzbox_cable tab for stored segment data * feat: rewrite fritzbox_cable JS for stored segment data with dual-series charts * feat: update fritzbox_cable CSS for range tabs and legend * feat: add segment utilization source to correlation timeline * feat: add segment utilization overlay to correlation chart * fix: add switchView callback for fritzbox_cable module tab * fix: default segment utilization range to 'all' * feat: add skeleton loading animation for segment utilization tab * feat: store all historical samples from FritzBox segment API * fix: use fixed 0-100% Y-axis scale for segment utilization charts Match FRITZ!Box presentation instead of auto-scaling to max value, which made small fluctuations appear disproportionately large. * feat: auto-enable segment utilization for FritzBox modems Remove segment_utilization_enabled config flag. The module is now automatically active when modem_type is fritzbox. Users can still disable it via Settings > Modules toggle. * refactor: migrate segment utilization charts to uPlot chart-engine Replace custom SVG rendering with the shared renderChart() API used throughout DOCSight. Provides tooltip, zoom modal, responsive resize, and consistent dark/light theme support out of the box. * fix: resolve skeleton shimmer on direct hash navigation Module script loads after deferred view routing, so auto-load call needs setTimeout(0) to defer until DOM elements are available. * style: add subtle fill under total utilization line Support dataset-level fill property for line charts in chart-engine. Adds purple gradient fill under segment utilization total line. * feat: add DB downsampling and periodic maintenance for segment utilization - Add downsample() to SegmentUtilizationStorage: aggregates samples older than 7 days into 5-min averages, older than 30 days into 15-min averages - Add periodic maintenance in collector: runs downsample + cleanup once per day after successful data collection - Prevents unbounded DB growth (525k rows/year at 1 sample/min) * refactor: move segment utilization from module system into core - Storage: app/modules/fritzbox_cable/storage.py -> app/storage/segment_utilization.py - Collector: app/modules/fritzbox_cable/collector.py -> app/collectors/segment_utilization.py - Routes: app/modules/fritzbox_cable/routes.py -> app/blueprints/segment_bp.py - Template: moved to app/templates/segment_utilization_tab.html - Static: moved to app/static/js/ and app/static/css/ - i18n: merged into core translations with seg_ prefix - Collector registered directly in discover_collectors() for fritzbox modems - Nav item, view div, switchView callback added to index.html - Deleted app/modules/fritzbox_cable/ entirely - All 1329 tests passing * fix: CM8200 session reuse to prevent brute-force lockout The CM8200A modem has a brute-force lockout that triggers after a few credential GETs in quick succession. Previously, every poll cycle sent fresh credentials, causing lockout after ~30 minutes (3 cycles at 900s). The driver now: - Probes the status page without credentials first (IP-based session reuse) - Only sends credentials when the session has actually expired - Checks Admin_Login_Lock.txt before credential auth to detect lockout - Falls back gracefully if the lockout check itself fails Reported-by: GitHub user via #168 * fix: i18n key mismatch in correlation segment overlay, add E2E tests - Fix correlation.js: use correct seg_correlation_ds/us i18n keys instead of non-existent correlation_segment_ds/us keys - Fix correlation.js: translate segment tooltip rows instead of hardcoded English strings - Fix test_dashboard.py: use .ring-title locator instead of text=Downstream which matched hidden glossary popovers first - Add 52 Playwright E2E tests for segment utilization covering navigation, loading, KPIs, charts, range tabs, API, i18n, theme switching, correlation, hash navigation, and JS errors - Add fritzbox_server fixture with seeded segment data to conftest.py --------- Co-authored-by: itsDNNS <itsDNNS@users.noreply.github.com>
1 parent f381028 commit 2763819

25 files changed

+2096
-26
lines changed

app/blueprints/__init__.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ def register_blueprints(app):
99
from .events_bp import events_bp
1010
from .modules_bp import modules_bp
1111
from .metrics_bp import metrics_bp
12+
from .segment_bp import segment_bp
1213

1314
app.register_blueprint(config_bp)
1415
app.register_blueprint(polling_bp)
@@ -17,3 +18,4 @@ def register_blueprints(app):
1718
app.register_blueprint(events_bp)
1819
app.register_blueprint(modules_bp)
1920
app.register_blueprint(metrics_bp)
21+
app.register_blueprint(segment_bp)

app/blueprints/segment_bp.py

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
"""Segment utilization API routes."""
2+
3+
import logging
4+
from datetime import datetime, timedelta, timezone
5+
6+
from flask import Blueprint, jsonify, request
7+
8+
from app.i18n import get_translations
9+
from app.web import get_config_manager, get_storage, require_auth
10+
11+
log = logging.getLogger("docsis.web.segment")
12+
13+
segment_bp = Blueprint("segment_bp", __name__)
14+
15+
_storage_instance = None
16+
17+
18+
def _get_lang():
19+
return request.cookies.get("lang", "en")
20+
21+
22+
def _get_storage():
23+
"""Lazy-init segment storage using core DB path."""
24+
global _storage_instance
25+
if _storage_instance is None:
26+
storage = get_storage()
27+
if storage:
28+
from app.storage.segment_utilization import SegmentUtilizationStorage
29+
_storage_instance = SegmentUtilizationStorage(storage.db_path)
30+
return _storage_instance
31+
32+
33+
RANGE_HOURS = {"24h": 24, "7d": 168, "30d": 720, "all": 0}
34+
35+
36+
@segment_bp.route("/api/fritzbox/segment-utilization")
37+
@require_auth
38+
def api_segment_utilization():
39+
"""Return stored segment utilization data for the tab view."""
40+
config = get_config_manager()
41+
t = get_translations(_get_lang())
42+
if not config:
43+
return jsonify({"error": t.get("seg_unavailable", "Configuration unavailable.")}), 503
44+
if config.get("modem_type") != "fritzbox":
45+
return jsonify({"error": t.get("seg_unsupported_driver", "This view is only available for FRITZ!Box cable devices.")}), 400
46+
47+
storage = _get_storage()
48+
if not storage:
49+
return jsonify({"error": "Storage unavailable"}), 503
50+
51+
range_key = request.args.get("range", "24h")
52+
hours = RANGE_HOURS.get(range_key, 24)
53+
54+
if hours > 0:
55+
start = (datetime.now(timezone.utc) - timedelta(hours=hours)).strftime("%Y-%m-%dT%H:%M:%SZ")
56+
else:
57+
start = "2000-01-01T00:00:00Z"
58+
end = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
59+
60+
return jsonify({
61+
"samples": storage.get_range(start, end),
62+
"latest": storage.get_latest(1),
63+
"stats": storage.get_stats(start, end),
64+
})
65+
66+
67+
@segment_bp.route("/api/fritzbox/segment-utilization/range")
68+
@require_auth
69+
def api_segment_utilization_range():
70+
"""Return segment data for a time range (used by correlation graph)."""
71+
storage = _get_storage()
72+
if not storage:
73+
return jsonify([])
74+
start = request.args.get("start", "")
75+
end = request.args.get("end", "")
76+
if not start or not end:
77+
return jsonify({"error": "start and end parameters required"}), 400
78+
return jsonify(storage.get_range(start, end))

app/collectors/__init__.py

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,16 @@ def discover_collectors(config_mgr, storage, event_detector, mqtt_pub, web, anal
108108
poll_interval=config["poll_interval"],
109109
notifier=notifier,
110110
))
111-
111+
112+
# Segment utilization collector (FritzBox only)
113+
if modem_type == "fritzbox":
114+
from .segment_utilization import SegmentUtilizationCollector
115+
collectors.append(SegmentUtilizationCollector(
116+
config_mgr=config_mgr,
117+
storage=storage,
118+
web=web,
119+
))
120+
112121
# ── Module collectors ──
113122
module_loader = web.get_module_loader() if hasattr(web, 'get_module_loader') else None
114123
if module_loader:
Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
"""Segment utilization collector for FritzBox cable modems."""
2+
3+
import logging
4+
import time
5+
6+
import requests
7+
8+
from app import fritzbox as fb
9+
from app.collectors.base import Collector, CollectorResult
10+
from app.storage.segment_utilization import SegmentUtilizationStorage
11+
12+
log = logging.getLogger("docsis.collector.segment_utilization")
13+
14+
MAINTENANCE_INTERVAL = 86400 # Run downsample + cleanup once per day
15+
16+
17+
def _last_non_null(values):
18+
"""Return the last non-None value from a list, or None if all are None/empty."""
19+
for v in reversed(values):
20+
if v is not None:
21+
return v
22+
return None
23+
24+
25+
class SegmentUtilizationCollector(Collector):
26+
"""Polls FritzBox /api/v0/monitor/segment/0 for cable segment utilization."""
27+
28+
def __init__(self, config_mgr, storage, web=None, **kwargs):
29+
super().__init__(poll_interval_seconds=300)
30+
self._config = config_mgr
31+
self._storage = SegmentUtilizationStorage(storage.db_path)
32+
self._web = web
33+
self._last_maintenance: float = 0.0
34+
35+
@property
36+
def name(self):
37+
return "segment_utilization"
38+
39+
def is_enabled(self):
40+
return self._config.get("modem_type") == "fritzbox"
41+
42+
def collect(self):
43+
url = self._config.get("modem_url")
44+
try:
45+
sid = fb.login(
46+
url,
47+
self._config.get("modem_user"),
48+
self._config.get("modem_password"),
49+
)
50+
except Exception as e:
51+
return CollectorResult.failure(self.name, str(e))
52+
53+
try:
54+
resp = requests.get(
55+
f"{url}/api/v0/monitor/segment/0",
56+
headers={"AUTHORIZATION": f"AVM-SID {sid}"},
57+
timeout=15,
58+
)
59+
resp.raise_for_status()
60+
except Exception as e:
61+
return CollectorResult.failure(self.name, f"API request failed: {e}")
62+
63+
try:
64+
from datetime import datetime, timezone
65+
66+
body = resp.json()
67+
data_items = body["data"]
68+
own = next(d for d in data_items if d["type"] == "own")
69+
total = next(d for d in data_items if d["type"] == "total")
70+
71+
last_sample_time = body.get("lastSampleTime", 0)
72+
sample_interval_ms = body.get("sampleInterval", 60000)
73+
sample_interval_s = sample_interval_ms / 1000
74+
n_samples = len(total["downstream"])
75+
76+
saved = 0
77+
for i in range(n_samples):
78+
ds_t = total["downstream"][i]
79+
us_t = total["upstream"][i]
80+
ds_o = own["downstream"][i]
81+
us_o = own["upstream"][i]
82+
if ds_t is None and us_t is None:
83+
continue
84+
sample_epoch = last_sample_time - (n_samples - 1 - i) * sample_interval_s
85+
ts = datetime.fromtimestamp(sample_epoch, tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
86+
self._storage.save_at(ts, ds_t, us_t, ds_o, us_o)
87+
saved += 1
88+
89+
ds_total = _last_non_null(total["downstream"])
90+
us_total = _last_non_null(total["upstream"])
91+
ds_own = _last_non_null(own["downstream"])
92+
us_own = _last_non_null(own["upstream"])
93+
94+
log.info(
95+
"Segment utilization: DS %.1f%% (own %.2f%%), US %.1f%% (own %.2f%%) [%d samples stored]",
96+
ds_total or 0, ds_own or 0, us_total or 0, us_own or 0, saved,
97+
)
98+
99+
self._run_maintenance()
100+
101+
return CollectorResult.ok(
102+
self.name,
103+
{"ds_total": ds_total, "us_total": us_total, "ds_own": ds_own, "us_own": us_own},
104+
)
105+
except Exception as e:
106+
return CollectorResult.failure(self.name, f"Parse failed: {e}")
107+
108+
def _run_maintenance(self):
109+
"""Run downsample + cleanup once per day."""
110+
now = time.time()
111+
if (now - self._last_maintenance) < MAINTENANCE_INTERVAL:
112+
return
113+
self._last_maintenance = now
114+
try:
115+
removed = self._storage.downsample()
116+
if removed:
117+
log.info("Downsampled segment utilization: %d rows aggregated", removed)
118+
deleted = self._storage.cleanup()
119+
if deleted:
120+
log.info("Cleaned up segment utilization: %d old rows deleted", deleted)
121+
except Exception as e:
122+
log.warning("Segment utilization maintenance failed: %s", e)

app/i18n/de.json

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -724,5 +724,32 @@
724724
"localization": "Lokalisierung",
725725
"localization_desc": "Sprache und Zeitzone",
726726
"dashboard_features": "Dashboard-Funktionen",
727-
"dashboard_features_desc": "Optionale Dashboard-Bereiche ein-/ausblenden"
727+
"dashboard_features_desc": "Optionale Dashboard-Bereiche ein-/ausblenden",
728+
"seg_title": "Segment-Auslastung",
729+
"seg_subtitle": "Kabel-Segmentauslastung aus dem FRITZ!Box-Monitoring.",
730+
"seg_loading": "Segment-Auslastung wird geladen...",
731+
"seg_no_data": "Noch keine Segment-Auslastungsdaten gesammelt. Aktiviere die Abfrage in den Einstellungen.",
732+
"seg_ds_total": "Downstream Gesamt",
733+
"seg_us_total": "Upstream Gesamt",
734+
"seg_ds_own": "Downstream Eigen",
735+
"seg_us_own": "Upstream Eigen",
736+
"seg_total": "Gesamt",
737+
"seg_own": "Eigener Anteil",
738+
"seg_downstream": "Downstream-Auslastung",
739+
"seg_upstream": "Upstream-Auslastung",
740+
"seg_status": "Status",
741+
"seg_status_polling": "Sammelt Daten",
742+
"seg_status_disabled": "Deaktiviert",
743+
"seg_avg": "Schnitt",
744+
"seg_min": "Min",
745+
"seg_max": "Max",
746+
"seg_range_24h": "24h",
747+
"seg_range_7d": "7d",
748+
"seg_range_30d": "30d",
749+
"seg_range_all": "Alle",
750+
"seg_note": "Die Auslastungswerte sind Schaetzungen der FRITZ!Box und koennen von der tatsaechlichen Segmentauslastung abweichen.",
751+
"seg_unsupported_driver": "Diese Ansicht ist nur fuer FRITZ!Box-Kabelgeraete verfuegbar.",
752+
"seg_unavailable": "Konfiguration nicht verfuegbar.",
753+
"seg_correlation_ds": "Segment DS Last (%)",
754+
"seg_correlation_us": "Segment US Last (%)"
728755
}

app/i18n/en.json

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -724,5 +724,32 @@
724724
"localization": "Localization",
725725
"localization_desc": "Language and timezone settings",
726726
"dashboard_features": "Dashboard Features",
727-
"dashboard_features_desc": "Toggle optional dashboard sections"
727+
"dashboard_features_desc": "Toggle optional dashboard sections",
728+
"seg_title": "Segment Utilization",
729+
"seg_subtitle": "Cable segment utilization from FRITZ!Box monitoring.",
730+
"seg_loading": "Loading segment utilization...",
731+
"seg_no_data": "No segment utilization data collected yet. Enable polling in Settings.",
732+
"seg_ds_total": "Downstream Total",
733+
"seg_us_total": "Upstream Total",
734+
"seg_ds_own": "Downstream Own",
735+
"seg_us_own": "Upstream Own",
736+
"seg_total": "Total",
737+
"seg_own": "Own Share",
738+
"seg_downstream": "Downstream Utilization",
739+
"seg_upstream": "Upstream Utilization",
740+
"seg_status": "Status",
741+
"seg_status_polling": "Collecting",
742+
"seg_status_disabled": "Disabled",
743+
"seg_avg": "Avg",
744+
"seg_min": "Min",
745+
"seg_max": "Max",
746+
"seg_range_24h": "24h",
747+
"seg_range_7d": "7d",
748+
"seg_range_30d": "30d",
749+
"seg_range_all": "All",
750+
"seg_note": "Segment utilization values are estimates provided by the FRITZ!Box and may differ from actual CMTS-wide load.",
751+
"seg_unsupported_driver": "This view is only available for FRITZ!Box cable devices.",
752+
"seg_unavailable": "Configuration unavailable.",
753+
"seg_correlation_ds": "Segment DS Load (%)",
754+
"seg_correlation_us": "Segment US Load (%)"
728755
}

app/i18n/es.json

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -719,5 +719,32 @@
719719
"localization": "Localización",
720720
"localization_desc": "Configuración de idioma y zona horaria",
721721
"dashboard_features": "Funciones del panel",
722-
"dashboard_features_desc": "Activar/desactivar secciones opcionales del panel"
722+
"dashboard_features_desc": "Activar/desactivar secciones opcionales del panel",
723+
"seg_title": "Utilizacion del segmento",
724+
"seg_subtitle": "Utilizacion del segmento de cable desde la monitorizacion FRITZ!Box.",
725+
"seg_loading": "Cargando utilizacion del segmento...",
726+
"seg_no_data": "No se han recopilado datos de utilizacion del segmento. Active la recopilacion en Configuracion.",
727+
"seg_ds_total": "Downstream total",
728+
"seg_us_total": "Upstream total",
729+
"seg_ds_own": "Downstream propio",
730+
"seg_us_own": "Upstream propio",
731+
"seg_total": "Total",
732+
"seg_own": "Parte propia",
733+
"seg_downstream": "Utilizacion downstream",
734+
"seg_upstream": "Utilizacion upstream",
735+
"seg_status": "Estado",
736+
"seg_status_polling": "Recopilando",
737+
"seg_status_disabled": "Desactivado",
738+
"seg_avg": "Prom",
739+
"seg_min": "Min",
740+
"seg_max": "Max",
741+
"seg_range_24h": "24h",
742+
"seg_range_7d": "7d",
743+
"seg_range_30d": "30d",
744+
"seg_range_all": "Todo",
745+
"seg_note": "Los valores de utilizacion del segmento son estimaciones de la FRITZ!Box y pueden diferir de la carga real del CMTS.",
746+
"seg_unsupported_driver": "Esta vista solo esta disponible para dispositivos cable FRITZ!Box.",
747+
"seg_unavailable": "Configuracion no disponible.",
748+
"seg_correlation_ds": "Carga segmento DS (%)",
749+
"seg_correlation_us": "Carga segmento US (%)"
723750
}

app/i18n/fr.json

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -716,5 +716,32 @@
716716
"localization": "Localisation",
717717
"localization_desc": "Paramètres de langue et de fuseau horaire",
718718
"dashboard_features": "Fonctionnalités du tableau de bord",
719-
"dashboard_features_desc": "Activer/désactiver les sections optionnelles du tableau de bord"
719+
"dashboard_features_desc": "Activer/désactiver les sections optionnelles du tableau de bord",
720+
"seg_title": "Utilisation du segment",
721+
"seg_subtitle": "Utilisation du segment cable depuis la surveillance FRITZ!Box.",
722+
"seg_loading": "Chargement de l'utilisation du segment...",
723+
"seg_no_data": "Aucune donnee d'utilisation de segment collectee. Activez la collecte dans les parametres.",
724+
"seg_ds_total": "Downstream total",
725+
"seg_us_total": "Upstream total",
726+
"seg_ds_own": "Downstream propre",
727+
"seg_us_own": "Upstream propre",
728+
"seg_total": "Total",
729+
"seg_own": "Part propre",
730+
"seg_downstream": "Utilisation downstream",
731+
"seg_upstream": "Utilisation upstream",
732+
"seg_status": "Statut",
733+
"seg_status_polling": "Collecte",
734+
"seg_status_disabled": "Desactive",
735+
"seg_avg": "Moy",
736+
"seg_min": "Min",
737+
"seg_max": "Max",
738+
"seg_range_24h": "24h",
739+
"seg_range_7d": "7j",
740+
"seg_range_30d": "30j",
741+
"seg_range_all": "Tout",
742+
"seg_note": "Les valeurs d'utilisation du segment sont des estimations de la FRITZ!Box et peuvent differer de la charge reelle du CMTS.",
743+
"seg_unsupported_driver": "Cette vue n'est disponible que pour les appareils cable FRITZ!Box.",
744+
"seg_unavailable": "Configuration non disponible.",
745+
"seg_correlation_ds": "Charge segment DS (%)",
746+
"seg_correlation_us": "Charge segment US (%)"
720747
}

0 commit comments

Comments
 (0)