Merge pull request #16 from agenixframework/feature/implement-gracefull-shutdown

Implement graceful shutdown, enhance Redis resilience, update docs, a…

Author: asuruceanu

.github/workflows/docs.yml

@@ -68,4 +68,12 @@ jobs:
         with:
           github_token: ${{ secrets.GITHUB_TOKEN }}
           publish_branch: gh-pages
+          # MkDocs outputs to ./site — deployed to the root of gh-pages.
+          # Configure GitHub Pages: Settings → Pages → Deploy from a branch → Branch: gh-pages, Folder: /
           publish_dir: ./site
+          # Keep gh-pages as a clean, artifact-only branch with no shared history from main.
+          # Avoids leftover files (e.g., a README) and keeps history small and deterministic.
+          force_orphan: true
+          # Create .nojekyll so GitHub Pages doesn't process the output with Jekyll.
+          # Prevents Jekyll from ignoring directories starting with underscores and from interfering with MkDocs assets.
+          enable_jekyll: false

docs/Graceful-Shutdown.md

@@ -0,0 +1,149 @@
+# Graceful Shutdown
+
+This document explains how Playwright Grid (Hub and Worker) behaves during shutdown, how to configure it, and how to integrate it with container orchestrators for zero‑surprise rollouts.
+
+Last updated: 2025-09-01
+
+## Summary
+- Hub stops accepting new borrows as soon as shutdown begins and reports not-ready on readiness checks.
+- Worker denies new borrows, drains active client WebSocket sessions up to a configurable timeout, cleans up Redis state, and force-terminates sidecars only if sessions remain after the timeout.
+- Both components surface clear readiness signals (HTTP 503) to allow load balancers/orchestrators to stop sending traffic before processes exit.
+
+## Hub behavior
+When ASP.NET Core triggers ApplicationStopping (e.g., SIGTERM), the Hub:
+- Immediately stops accepting new borrow requests.
+  - POST /session/borrow responds with 503 Service Unavailable.
+  - Response includes `Retry-After: 30` header to hint clients to retry later.
+- Readiness endpoint reflects shutdown:
+  - GET /health/ready returns 503 so containers are removed from load balancers.
+- Existing sessions are unaffected at Hub level; Hub is stateless for live WS proxying (the Worker owns the WebSocket lifecycle).
+
+Relevant code paths:
+- hub/Infrastructure/Web/EndpointMappingExtensions.cs
+  - Internal `_acceptingBorrows` flag flips to false on `ApplicationStopping`.
+  - `/session/borrow` returns 503 when not accepting borrows.
+  - `/health/ready` returns 503 when not accepting borrows.
+
+## Worker behavior
+When the Worker receives shutdown (ApplicationStopping):
+- Sets `_acceptingBorrows = false` to deny new borrows at `/borrow/{labelKey}` with 503 and `Retry-After: 30`.
+- Begins graceful drain of active WebSocket sessions:
+  - Waits up to `WORKER_DRAIN_TIMEOUT_SECONDS` (default 30s) for all active client WS connections to close.
+  - During this period, no new borrows are accepted.
+- After waiting:
+  - Performs cleanup of Redis lists/keys for this node and removes itself from `nodes` set.
+  - If any sessions are still active, logs a warning and force-kills remaining sidecar processes to ensure timely shutdown.
+- Readiness reflects shutdown while draining:
+  - GET `/health/ready` returns 503, signaling the orchestrator to stop routing new traffic.
+
+Relevant code paths:
+- worker/Services/WebServerHost.cs
+  - Graceful drain, denying borrows, readiness 503 during shutdown.
+- worker/Services/PoolManager.cs
+  - Tracks active WS connections per browserId and exposes `HasAnyActiveConnections()` for drain logic.
+  - Cleanup of Redis state and optional force-kill of sidecars.
+
+## HTTP status codes and headers
+- New borrows denied during shutdown:
+  - Hub: POST `/session/borrow` → 503 Service Unavailable, `Retry-After: 30`.
+  - Worker: POST `/borrow/{labelKey}` → 503 Service Unavailable, `Retry-After: 30`.
+- Readiness while shutting down:
+  - Hub: GET `/health/ready` → 503.
+  - Worker: GET `/health/ready` → 503.
+
+## Configuration
+Environment variables impacting shutdown behavior:
+- WORKER_DRAIN_TIMEOUT_SECONDS
+  - Default: 30.
+  - How long the Worker waits for all active WS sessions to close before force-killing sidecars.
+- REDIS_* timeouts (Hub and Worker)
+  - Control health ping timings; not shutdown-specific but influence `/health/ready` responsiveness.
+
+Defaults and safety:
+- If WORKER_DRAIN_TIMEOUT_SECONDS is not set or invalid, default 30s is used.
+- If drain times out, sidecars are force-terminated; this prevents hung shutdowns on orchestrators with hard SIGKILL deadlines.
+
+## Orchestrator integration
+
+### Docker / docker-compose
+- The built-in readiness endpoints and 503 responses during shutdown are sufficient for Compose to stop routing requests when using healthchecks or external LB.
+- Example healthcheck in docker-compose.yml:
+
+```yaml
+healthcheck:
+  test: ["CMD", "curl", "-fsS", "http://localhost:5000/health/ready"]
+  interval: 5s
+  timeout: 2s
+  retries: 3
+  start_period: 10s
+```
+
+Set a drain timeout:
+
+```yaml
+environment:
+  - WORKER_DRAIN_TIMEOUT_SECONDS=45
+```
+
+### Kubernetes
+Use readiness probes and preStop hooks to ensure in-flight sessions drain:
+
+```yaml
+readinessProbe:
+  httpGet:
+    path: /health/ready
+    port: 5000
+  periodSeconds: 5
+  timeoutSeconds: 2
+  failureThreshold: 1
+
+lifecycle:
+  preStop:
+    exec:
+      command: ["/bin/sh", "-c", "sleep 40"]
+```
+
+- Set `terminationGracePeriodSeconds` to be >= WORKER_DRAIN_TIMEOUT_SECONDS + probe buffer. Example: 60.
+- The app flips readiness to 503 on shutdown automatically; the preStop sleep gives LBs time to drain before SIGTERM deadlines.
+
+## Observability
+- Logs
+  - Hub: "[hub] ApplicationStopping: stop accepting new borrows".
+  - Worker: "[worker] ApplicationStopping: initiating graceful drain" and possible timeout message.
+- Metrics
+  - Standard HTTP/ASP.NET metrics are exposed (Prometheus). During shutdown, expect:
+    - Increased 503 counts on borrow endpoints.
+    - `/health/ready` 503 rate until container exits.
+- Dashboard
+  - Ongoing sessions should continue; new borrows will fail fast with 503 until workers come back.
+
+## Verification steps
+- Local manual test
+  1) Start the stack (docker compose up --build).
+  2) Borrow a session and connect a client.
+  3) Send SIGTERM to a worker container: `docker kill --signal=TERM <worker_container>`.
+  4) Observe logs: drain starts; `/health/ready` returns 503; connection persists until closed or timeout.
+- Automated tests
+  - Unit tests remain green. Integration tests can be extended in future to simulate shutdown; current grid tests rely on Testcontainers bootstrap and are compatible with the behavior.
+
+## Compatibility and client expectations
+- Clients should handle 503 responses on borrow and respect `Retry-After` header.
+- Existing WebSocket sessions can continue until user closes them or the drain timeout ends.
+- No API changes were introduced; the feature is backward compatible.
+
+## FAQ
+- Q: Will shutdown interrupt a running Playwright session?
+  - A: Not immediately. The worker attempts a graceful drain. If the session exceeds the configured drain timeout, the sidecar is force-terminated to allow shutdown to complete.
+- Q: Do I need to change probes?
+  - A: Ensure you’re using `/health/ready` for readiness. Liveness can stay on `/health`.
+- Q: Can I make drain longer than my platform’s termination grace period?
+  - A: You can, but the platform may send SIGKILL before drain ends. Align `terminationGracePeriodSeconds` (K8s) or stop timeout (Docker) with your drain setting.
+
+## References
+- Source files:
+  - hub/Infrastructure/Web/EndpointMappingExtensions.cs
+  - worker/Services/WebServerHost.cs
+  - worker/Services/PoolManager.cs
+- Related docs:
+  - Node Liveness and Sweeper (node TTLs and cleanup)
+  - Borrow TTL & Session Persistence

docs/assets/styles.css

@@ -1,35 +1,27 @@
-/* Minimal overrides for Material theme used by Playwright Grid docs */
-:root {
-  /* Brand-ish colors (tweak as needed) */
-  --md-primary-fg-color: #2563eb; /* blue-600 */
-  --md-accent-fg-color: #22c55e;  /* green-500 */
-}
+/* Minimal tweaks on top of Material theme */
 
-/* Typography */
-.md-typeset {
-  -webkit-font-smoothing: antialiased;
+/* Improve code block readability */
+:root {
+  --code-bg: #0b1220;
 }
-.md-typeset h1, .md-typeset h2, .md-typeset h3 {
-  letter-spacing: 0.2px;
+.md-typeset pre>code {
+  background-color: var(--code-bg);
+  color: #e6edf3;
 }
+
+/* Slightly tighter paragraphs */
 .md-typeset p {
-  margin: 0.5em 0 0.9em;
+  line-height: 1.55;
 }
 
-/* Inline code */
-.md-typeset :not(pre) > code {
-  padding: 0.1em 0.25em;
-  border-radius: 0.25rem;
-  white-space: break-spaces;
+/* Make tables scroll on small screens */
+.md-typeset table {
+  display: block;
+  overflow-x: auto;
+  white-space: nowrap;
 }
 
-/* Code blocks */
-.md-typeset pre {
-  margin: 0.75rem 0;
-}
-.md-typeset pre > code {
-  display: block;
-  padding: 0.75rem 1rem;
-  border-radius: 0.5rem;
-  white-space: pre-wrap; /* wrap long lines */
+/* Subtle link underline on hover */
+.md-typeset a:hover {
+  text-decoration: underline;
 }

docs/index.md

@@ -18,6 +18,7 @@ Use the links below to get started and dive into specific topics.
 - Capacity Queue (pending borrows, fairness, timeouts): Capacity-Queue.md
 - Borrow TTL & Session Persistence: Borrow-TTL-and-Session-Persistence.md
 - Node Liveness & Sweeper: Node-Liveness-and-Sweeper.md
+- Graceful Shutdown: Graceful-Shutdown.md
 - Session Distribution across workers: distribution.md
 
 ## Observability

docs/tasks.md

@@ -16,12 +16,12 @@ The following is an ordered, actionable checklist covering architectural and cod
 10. [X] Introduce a capacity queue in Hub for pending borrows with timeout and fairness (per-label and per-run caps) to reduce thundering herd.
 11. [X] Implement node heartbeat/liveness tracker with configurable timeout; evict stale nodes and reclaim/expire orphaned sessions.
 12. [X] Add borrow TTL and auto-return on timeout; persist session state to Redis to survive Hub restarts.
-13. [ ] Harden Redis usage: resilience (timeouts, retries with jitter, circuit breaker), connection settings, and health checks integrated into readiness.
+13. [X] Harden Redis usage: resilience (timeouts, retries with jitter, circuit breaker), connection settings, and health checks integrated into readiness.
 14. [ ] Support secret rotation: accept multiple HUB_RUNNER_SECRET/HUB_NODE_SECRET values (comma-separated) and log deprecation windows.
 15. [ ] Redact secrets and PII in logs; ensure headers and sensitive values never appear in structured logs.
 16. [ ] Add rate limiting (per IP and per runner id) on Hub borrow/return to protect from abuse; return 429 with Retry-After.
 17. [ ] Add optional IP allowlist or token-based auth (e.g., PAT via header) for Hub API alongside shared secrets.
-18. [ ] Implement graceful shutdown: Hub stops accepting new borrows; Worker drains sessions and returns cleanly on SIGTERM.
+18. [X] Implement graceful shutdown: Hub stops accepting new borrows; Worker drains sessions and returns cleanly on SIGTERM.
 19. [ ] Enforce maximum WebSocket message size and idle timeouts in Worker; send periodic pings and close dead connections.
 20. [ ] Add backpressure controls in Worker WS proxy (bounded channels, drop policy, and metrics for drops).
 21. [ ] Strengthen Worker sidecar management: sidecar health endpoint, restart/backoff strategy, and clear error surfacing to Hub.

hub/Dockerfile

@@ -23,5 +23,5 @@ COPY --from=build /app/publish .
 ENV ASPNETCORE_URLS=http://0.0.0.0:5000 \
     DOTNET_RUNNING_IN_CONTAINER=true
 EXPOSE 5000
-HEALTHCHECK --interval=10s --timeout=3s --retries=5 --start-period=20s CMD curl -fsSL http://127.0.0.1:5000/health || exit 1
+HEALTHCHECK --interval=10s --timeout=3s --retries=5 --start-period=20s CMD curl -fsSL http://127.0.0.1:5000/health/ready || exit 1
 ENTRYPOINT ["dotnet","PlaywrightHub.dll"]

hub/Infrastructure/Web/EndpointMappingExtensions.cs

@@ -190,6 +190,10 @@ private sealed class Waiter
 
 public static class EndpointMappingExtensions
 {
+    private static long _redisBreakerUntilTicks;
+    private static int _redisConsecutiveFailures;
+    private static volatile bool _acceptingBorrows = true;
+
     public static void MapHubEndpoints(this WebApplication app)
     {
         var config = app.Configuration;
@@ -212,6 +216,13 @@ public static void MapHubEndpoints(this WebApplication app)
         var resultsStore = services.GetRequiredService<IResultsStore>();
         var resultsHubCtx = services.GetRequiredService<IHubContext<ResultsHub, IResultsClient>>();
 
+        // Graceful shutdown: stop accepting new borrow requests when application is stopping
+        app.Lifetime.ApplicationStopping.Register(() =>
+        {
+            _acceptingBorrows = false;
+            try { Console.WriteLine("[hub] ApplicationStopping: stop accepting new borrows"); } catch { }
+        });
+
         var hubRunnerSecret = config["HUB_RUNNER_SECRET"] ?? "runner-secret";
         var hubNodeSecret = config["HUB_NODE_SECRET"] ?? "node-secret";
         var nodeTimeoutSeconds = int.TryParse(config["HUB_NODE_TIMEOUT"], out var t) ? t : 60;
@@ -420,6 +431,14 @@ public static void MapHubEndpoints(this WebApplication app)
                     return Results.Unauthorized();
                 }
 
+                // During graceful shutdown, deny new borrows with 503
+                if (!_acceptingBorrows)
+                {
+                    try { req.HttpContext.Response.Headers.Append("Retry-After", "30"); } catch { }
+                    try { borrowOutcomes.WithLabels("unknown", "denied").Inc(); } catch { }
+                    return Results.StatusCode(StatusCodes.Status503ServiceUnavailable);
+                }
+
                 var body = await req.ReadFromJsonAsync<Dictionary<string, string>>() ??
                            new Dictionary<string, string>();
                 if (!body.TryGetValue("labelKey", out var labelKey) || string.IsNullOrEmpty(labelKey))
@@ -2230,6 +2249,63 @@ static bool IsSecret(string key)
 
         app.MapGet("/nodes", () => Results.Ok(db.SetMembers("nodes").Select(x => x.ToString())));
         app.MapGet("/health", () => Results.Ok(new { status = "ok" }));
+        app.MapGet("/health/ready", async () =>
+        {
+            // During graceful shutdown, report not ready
+            if (!_acceptingBorrows)
+            {
+                return Results.StatusCode(StatusCodes.Status503ServiceUnavailable);
+            }
+            try
+            {
+                var timeoutMs = int.TryParse(config["REDIS_HEALTH_TIMEOUT_MS"], out var ms) ? Math.Max(100, ms) : 1000;
+                var breakerThreshold = int.TryParse(config["REDIS_BREAKER_THRESHOLD"], out var th) ? Math.Max(1, th) : 3;
+                var breakerCooldownMs = int.TryParse(config["REDIS_BREAKER_COOLDOWN_MS"], out var cd) ? Math.Max(500, cd) : 5000;
+
+                // Circuit breaker: if open, fail fast
+                var nowTicks = DateTime.UtcNow.Ticks;
+                if (System.Threading.Interlocked.Read(ref _redisBreakerUntilTicks) > nowTicks)
+                {
+                    return Results.StatusCode(StatusCodes.Status503ServiceUnavailable);
+                }
+
+                if (!mux.IsConnected)
+                {
+                    var until = nowTicks + (long)breakerCooldownMs * TimeSpan.TicksPerMillisecond;
+                    System.Threading.Interlocked.Exchange(ref _redisBreakerUntilTicks, until);
+                    System.Threading.Interlocked.Exchange(ref _redisConsecutiveFailures, 0);
+                    return Results.StatusCode(StatusCodes.Status503ServiceUnavailable);
+                }
+
+                var sw = System.Diagnostics.Stopwatch.StartNew();
+                var pingTask = db.PingAsync();
+                var completed = await Task.WhenAny(pingTask, Task.Delay(timeoutMs));
+                if (completed != pingTask)
+                {
+                    return Results.StatusCode(StatusCodes.Status503ServiceUnavailable);
+                }
+                await pingTask; // propagate any errors
+                sw.Stop();
+
+                // Success: reset failure counters and breaker
+                System.Threading.Interlocked.Exchange(ref _redisConsecutiveFailures, 0);
+                System.Threading.Interlocked.Exchange(ref _redisBreakerUntilTicks, 0);
+                return Results.Ok(new { status = "ready", redis = new { pingMs = sw.ElapsedMilliseconds } });
+            }
+            catch
+            {
+                var threshold = System.Threading.Interlocked.Increment(ref _redisConsecutiveFailures);
+                var breakerThreshold = int.TryParse(config["REDIS_BREAKER_THRESHOLD"], out var th2) ? Math.Max(1, th2) : 3;
+                var breakerCooldownMs = int.TryParse(config["REDIS_BREAKER_COOLDOWN_MS"], out var cd2) ? Math.Max(500, cd2) : 5000;
+                if (threshold >= breakerThreshold)
+                {
+                    var until = DateTime.UtcNow.Ticks + (long)breakerCooldownMs * TimeSpan.TicksPerMillisecond;
+                    System.Threading.Interlocked.Exchange(ref _redisBreakerUntilTicks, until);
+                    System.Threading.Interlocked.Exchange(ref _redisConsecutiveFailures, 0);
+                }
+                return Results.StatusCode(StatusCodes.Status503ServiceUnavailable);
+            }
+        });
     }
 
     private static bool CheckSecret(HttpRequest req, string header, string expected)

hub/Services/HubServiceRunner.cs

@@ -107,7 +107,22 @@ public static async Task RunAsync(string[] args)
 
 
         var redisUrl = builder.Configuration["REDIS_URL"] ?? "redis:6379";
-        var mux = await ConnectionMultiplexer.ConnectAsync(redisUrl);
+        // Configure Redis connection with resilience and timeouts
+        var redisOptions = ConfigurationOptions.Parse(redisUrl, true);
+        redisOptions.AbortOnConnectFail = false; // keep retrying
+        redisOptions.ConnectRetry = 3;
+        redisOptions.KeepAlive = 15;
+        int GetInt(string key, int def)
+        {
+            return int.TryParse(builder.Configuration[key], out var v) ? Math.Max(0, v) : def;
+        }
+        redisOptions.ConnectTimeout = GetInt("REDIS_CONNECT_TIMEOUT_MS", 5000);
+        redisOptions.SyncTimeout = GetInt("REDIS_SYNC_TIMEOUT_MS", 5000);
+        redisOptions.AsyncTimeout = GetInt("REDIS_ASYNC_TIMEOUT_MS", 5000);
+        // Exponential reconnect backoff policy (includes jitter internally)
+        redisOptions.ReconnectRetryPolicy = new ExponentialRetry(5000);
+
+        var mux = await ConnectionMultiplexer.ConnectAsync(redisOptions);
         var db = mux.GetDatabase();
 
         // Services for dashboard integration

mkdocs.yml

@@ -58,6 +58,7 @@ nav:
   - Capacity Queue: Capacity-Queue.md
   - Borrow TTL & Session Persistence: Borrow-TTL-and-Session-Persistence.md
   - Node Liveness and Sweeper: Node-Liveness-and-Sweeper.md
+  - Graceful Shutdown: Graceful-Shutdown.md
   - "Playwright .NET pw:api": PlaywrightDotNet-pw-api.md
   - Presentation: presentation-why-playwright-grid.md
   - Metrics and Grafana: Metrics-and-Grafana.md

worker/Services/PoolManager.cs

@@ -60,6 +60,12 @@ public bool HasActiveConnection(string browserId)
         return _activeWs.TryGetValue(browserId, out var v) && v > 0;
     }
 
+    public bool HasAnyActiveConnections()
+    {
+        try { return !_activeWs.IsEmpty; }
+        catch { return _activeWs.Count > 0; }
+    }
+
     private static string NormalizeBrowser(string s)
     {
         return string.IsNullOrWhiteSpace(s) ? "Chromium" : s.Trim();