Skip to content

Commit f5e2cd6

Browse files
jsprague84claude
andcommitted
Add structured logging, metrics, comprehensive testing, and Loki stack
Major improvements to observability, testing, and operational reliability: ## 🔍 Structured Logging with Tracing - Replace all eprintln! calls with structured tracing macros across all services - Add environment-based log level control via RUST_LOG - Implement tracing-subscriber with env-filter, fmt, and json support - Add tracing-log bridge for backward compatibility with existing log:: usage - Services updated: common, weatherust, healthmon, speedynotify, updatemon, updatectl Benefits: - Structured logging with key-value pairs for better filtering - Dynamic log level control without recompilation - Consistent logging format across all services - Better debugging with contextual information ## 📊 Prometheus Metrics Infrastructure - Create comprehensive metrics module in common crate - Add metrics crate dependency across all services - Implement 12 different metric types: * Notification delivery tracking (success/failure rates) * Operation execution timing (weather API, speedtests, updates) * Server operation tracking (multi-server operations) * Container health metrics (CPU, memory, health status) * Update availability gauges (OS and Docker updates) * Speedtest result metrics (download, upload, ping, degradation) * Weather API performance (call count, response time) * Cleanup operation metrics (items removed, bytes reclaimed) * Webhook request metrics (endpoint, status, duration) Benefits: - Full operational visibility - Performance monitoring - Alerting capabilities (when integrated with Prometheus) - Historical trend analysis ## 🧪 Comprehensive Testing Suite - Add 29 new tests across 3 packages (20 → 49 total tests, +145%) - weatherust: 13 tests for ZIP detection, city parsing, location normalization - speedynotify: 7 tests for JSON parsing, speed degradation detection - common: 9 tests for all metrics recording functions - All tests passing with 100% pass rate Benefits: - Prevents regressions during refactoring - Documents expected behavior - Enables confident code changes - Faster debugging of parsing issues ## 🔧 Loki Logging Stack (Optional) - Add docker-compose.logging.yml for centralized log aggregation - Configure Loki 3.0 for log storage (7-day retention) - Configure Promtail 3.0 for automatic Docker log collection - Configure Grafana 11.0 with Loki pre-configured as datasource - Traefik integration for HTTPS access to Grafana - Comprehensive LOGGING_SETUP.md guide Benefits: - Centralized log viewing across all containers - Rich query capabilities with LogQL - Time-based log exploration - Export and analysis features ## 📝 Documentation - Add IMPROVEMENTS_2025.md with complete implementation summary - Add LOGGING_SETUP.md with Loki stack setup and usage guide - Update .env.example and .env.example.short with Grafana configuration - Migration guide for users and developers ## 🔧 Dependencies Added Runtime: - tracing 0.1 - tracing-subscriber 0.3 (env-filter, fmt, json) - tracing-log 0.2 - metrics 0.23 - metrics-exporter-prometheus 0.15 (updatectl only) Dev: - tokio-test 0.4 - mockito 1.5 (common only) - axum-test 15 (updatectl only) ## ✅ Quality - All workspace crates compile successfully - All 49 tests passing (0 failures) - Fully backward compatible - No breaking changes ## 🚀 Usage Set log level: RUST_LOG=debug weatherust --zip 52726 Deploy logging: docker compose -f docker-compose.yml -f docker-compose.logging.yml up -d View logs: https://grafana.js-node.app (or your configured domain) 🤖 Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent bcf2407 commit f5e2cd6

25 files changed

+2222
-125
lines changed

.env.example

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -343,3 +343,15 @@ UPDATECTL_WEBHOOK_URL=https://webhook.js-node.app
343343
# HEALTHMON_TAG=
344344
# UPDATEMON_TAG=
345345
# UPDATECTL_TAG=
346+
347+
# ============================================================================
348+
# Grafana & Loki Configuration (for centralized logging)
349+
# ============================================================================
350+
# Set a secure admin password for Grafana web UI
351+
# Default: admin (CHANGE THIS IN PRODUCTION!)
352+
GRAFANA_PASSWORD=admin
353+
354+
# Domain for Grafana access via Traefik
355+
# Example: grafana.js-node.app
356+
GRAFANA_DOMAIN=grafana.localhost
357+

.env.example.short

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,3 +74,8 @@ UPDATECTL_WEBHOOK_URL=https://webhook.example.com
7474
# HEALTHMON_TAG=
7575
# UPDATEMON_TAG=
7676
# UPDATECTL_TAG=
77+
78+
# Grafana & Loki (optional - for centralized logging)
79+
GRAFANA_PASSWORD=admin
80+
GRAFANA_DOMAIN=grafana.localhost
81+

0 commit comments

Comments
 (0)