[server] ReadinessProbe: add redis as dependency

Tool: gitpod/catfood.gitpod.cloud

Author: geropl

components/server/src/liveness/readiness-controller.ts

@@ -11,11 +11,13 @@ import { SpiceDBClientProvider } from "../authorization/spicedb";
 import { log } from "@gitpod/gitpod-protocol/lib/util/logging";
 import { ReadSchemaRequest } from "@authzed/authzed-node/dist/src/v1";
 import { getExperimentsClientForBackend } from "@gitpod/gitpod-protocol/lib/experiments/configcat-server";
+import { Redis } from "ioredis";
 
 @injectable()
 export class ReadinessController {
     @inject(TypeORM) protected readonly typeOrm: TypeORM;
     @inject(SpiceDBClientProvider) protected readonly spiceDBClientProvider: SpiceDBClientProvider;
+    @inject(Redis) protected readonly redis: Redis;
 
     get apiRouter(): express.Router {
         const router = express.Router();
@@ -55,7 +57,15 @@ export class ReadinessController {
                     return;
                 }
 
-                // Both connections are good
+                // Check Redis connection
+                const redisConnection = await this.checkRedisConnection();
+                if (!redisConnection) {
+                    log.warn("Readiness check failed: Redis connection failed");
+                    res.status(503).send("Redis connection failed");
+                    return;
+                }
+
+                // All connections are good
                 res.status(200).send("Ready");
             } catch (error) {
                 log.error("Readiness check failed", error);
@@ -91,4 +101,16 @@ export class ReadinessController {
             return false;
         }
     }
+
+    private async checkRedisConnection(): Promise<boolean> {
+        try {
+            // Simple PING command to verify connection is working
+            const result = await this.redis.ping();
+            log.debug("Redis connection check successful", { result });
+            return result === "PONG";
+        } catch (error) {
+            log.error("Redis connection check failed", error);
+            return false;
+        }
+    }
 }

memory-bank/activeContext.md

@@ -149,11 +149,12 @@ Initial exploration of the Gitpod codebase has revealed:
     - Manages complex dependencies between components
 - **Server Health Checks**: The Gitpod server uses two distinct health check mechanisms:
   - **Liveness Probe**: Checks the event loop lag to determine if the server is functioning properly
-  - **Readiness Probe**: Checks database and SpiceDB connectivity to ensure the server is ready to handle requests
+  - **Readiness Probe**: Checks database, SpiceDB, and Redis connectivity to ensure the server is ready to handle requests
     - Controlled by a ConfigCat feature flag `server_readiness_probe` (default: true) that can bypass the actual checks
 - **Critical Dependencies**: The server has critical external dependencies that must be operational:
   - **Database (TypeORM)**: Used for persistent storage
   - **SpiceDB**: Used for authorization and permission management
+  - **Redis**: Used for caching, pub/sub messaging, and distributed locking
 - **Server Architecture Patterns**:
   - The server uses dependency injection (Inversify) for component management
   - Components are registered in `container-module.ts` and injected where needed

memory-bank/progress.md

@@ -212,11 +212,16 @@ No specific blockers or dependencies have been identified yet. This section will
     - Created PRD document for the readiness probe implementation
     - Updated Kubernetes deployment configuration to add the readiness probe
     - Added ConfigCat feature flag `server_readiness_probe` to control readiness checks
+  - Enhanced server readiness probe with Redis health check:
+    - Added Redis connectivity check to the ReadinessController
+    - Updated PRD document to include Redis in the list of checked dependencies
+    - Ensured the feature flag continues to work with the new Redis check
   - Updated memory bank with new learnings:
     - Added information about server health checks and critical dependencies
     - Documented server architecture patterns and dependency injection
     - Added information about Kubernetes deployment configuration
     - Documented feature flag implementation for readiness probe
+    - Added Redis as a critical dependency for the server
   - Established standardized development workflows:
     - Created `workflows.md` to document standardized development processes
     - Documented the Product Requirements Document (PRD) workflow

prd/001-readinessprobe-server.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-This document outlines the implementation of a readiness probe for the Gitpod server deployment. The readiness probe will ensure that the server is only considered ready when it has established connections to both the database and SpiceDB authorizer.
+This document outlines the implementation of a readiness probe for the Gitpod server deployment. The readiness probe will ensure that the server is only considered ready when it has established connections to the database, SpiceDB authorizer, and Redis.
 
 ## Background
 
@@ -13,6 +13,7 @@ Currently, the server deployment has a liveness probe that checks the event loop
 1. Create a readiness endpoint in the server that checks:
    - Database connectivity
    - SpiceDB authorizer connectivity
+   - Redis connectivity
 2. Configure the Kubernetes deployment to use this endpoint as a readiness probe
 
 ## Implementation Details
@@ -22,7 +23,8 @@ Currently, the server deployment has a liveness probe that checks the event loop
 We've created a new `ReadinessController` class in `components/server/src/liveness/readiness-controller.ts` that:
 - Checks database connectivity by executing a simple query
 - Checks SpiceDB connectivity by attempting to get a client
-- Returns a 200 status code only if both checks pass, otherwise returns a 503 status code
+- Checks Redis connectivity by executing a PING command
+- Returns a 200 status code only if all checks pass, otherwise returns a 503 status code
 
 ```typescript
 // components/server/src/liveness/readiness-controller.ts
@@ -147,8 +149,8 @@ ReadinessProbe: &corev1.Probe{
 
 The readiness probe should be tested to ensure:
 
-1. The server is only considered ready when both database and SpiceDB connections are established
-2. The server is not considered ready if either connection fails
+1. The server is only considered ready when database, SpiceDB, and Redis connections are established
+2. The server is not considered ready if any connection fails
 3. The server becomes ready again when connections are re-established
 
 ## Deployment Considerations
@@ -161,7 +163,7 @@ The readiness probe should be tested to ensure:
 
 The readiness probe implementation includes a ConfigCat feature flag called `server_readiness_probe` that controls whether the actual connectivity checks are performed:
 
-- When the flag is set to `true` (default): The readiness probe will always return a 200 status code, bypassing the actual database and SpiceDB connectivity checks
+- When the flag is set to `true` (default): The readiness probe will always return a 200 status code, bypassing the actual database, SpiceDB, and Redis connectivity checks
 - When the flag is set to `false`: The readiness probe will perform the actual checks and return the appropriate status code based on the results
 
 This feature flag provides several benefits:
@@ -173,9 +175,10 @@ This feature flag provides several benefits:
 ## Future Improvements
 
 - Add more sophisticated checks for SpiceDB connectivity, such as a simple permission check
+- Add more sophisticated checks for Redis connectivity, such as a simple key-value operation
 - Add metrics for readiness probe failures
 - Consider adding more dependencies to the readiness check as needed
 
 ## Conclusion
 
-This implementation ensures that the server is only considered ready when it has established connections to both the database and SpiceDB authorizer. This improves the reliability of the deployment by preventing traffic from being sent to instances that are not fully initialized.
+This implementation ensures that the server is only considered ready when it has established connections to the database, SpiceDB authorizer, and Redis. This improves the reliability of the deployment by preventing traffic from being sent to instances that are not fully initialized.