2643 - add Redis-backed metrics cache for multi-instance deployments

[gcgoncalves]

🐛 Bug-fix PR

📌 Summary

Implements centralized metrics aggregation cache using Redis to solve metric fluctuation issue in multi-instance deployments behind load balancers. This change is behind the METRICS_CACHE_USE_REDIS feature flag.

The downside of this approach is that the metrics are updated on the UI only after the cache TTL is expired. The cache TTL is defined using the METRICS_CACHE_TTL_SECONDS environment setting.

Closes #2643

🔁 Reproduction Steps

1. Navigate to http://localhost:8080/admin/#metrics
2. Note the "Total Executions" value
3. Refresh the page (F5 or Ctrl+R)
4. Observe the value changes significantly (up or down)
5. Repeat several times - values appear random

#2643

🐞 Root Cause

• Each gateway instance had its own in-memory cache of PostgreSQL aggregates
• Load balancer routing to different instances showed different cached values
• Metrics fluctuated non-monotonically on page refresh (e.g., 32075 → 21858)

💡 Fix Description

• Shared Redis cache for aggregated query results across all instances
• Cache invalidation after buffer flush ensures consistency
• Automatic fallback to local cache if Redis unavailable
• Default enabled (METRICS_CACHE_USE_REDIS=true)

🧪 Verification

Check	Command	Status
Lint suite	make lint	✅
Unit tests	make test	✅
Coverage ≥ 80 %	make coverage	✅
Manual regression no longer fails	steps / screenshots	✅

✅ Checklist

• Code formatted (make black isort pre-commit)
• No secrets/credentials committed

[crivetimihai]

Thanks @gcgoncalves. Solid implementation — good use of dual-write (Redis + local) with automatic fallback, Prometheus hit/miss counters, and one-time warning for sync callers when Redis is active.

A few observations from the diff:

1. Thread safety: _total_hit_count, _redis_hit_count, etc. are incremented outside the lock in get_async. These are stats counters so races aren't critical, but consider threading.Lock or atomic counters if precision matters.
2. JSON serialization: Redis values are json.loads/json.dumps. The local cache stores Python dicts directly. Ensure the serialized forms are consistent (e.g., datetime values won't roundtrip through JSON).
3. Docs quality: The Compose and Kubernetes deployment docs are thorough, though the Compose example includes PLATFORM_ADMIN_PASSWORD: "changeme" and JWT_SECRET_KEY: "your-secret-key" — consider adding a comment warning not to use these in production.
4. Feature flag fallback: When METRICS_CACHE_USE_REDIS=true but Redis is unreachable at startup, does it fall back gracefully to local, or fail? The code shows redis_client is not None and REDIS_AVAILABLE — looks like it handles the import-missing case but not connection-refused at init.