[server] Introduce server_readiness_probe feature flag so we can disable the ReadinessProbe if required

Tool: gitpod/catfood.gitpod.cloud

Author: geropl

components/server/src/liveness/readiness-controller.ts

@@ -10,6 +10,7 @@ import { TypeORM } from "@gitpod/gitpod-db/lib";
 import { SpiceDBClientProvider } from "../authorization/spicedb";
 import { log } from "@gitpod/gitpod-protocol/lib/util/logging";
 import { ReadSchemaRequest } from "@authzed/authzed-node/dist/src/v1";
+import { getExperimentsClientForBackend } from "@gitpod/gitpod-protocol/lib/experiments/configcat-server";
 
 @injectable()
 export class ReadinessController {
@@ -25,6 +26,19 @@ export class ReadinessController {
     protected addReadinessHandler(router: express.Router) {
         router.get("/", async (_, res) => {
             try {
+                // Check feature flag first
+                const readinessProbeEnabled = await getExperimentsClientForBackend().getValueAsync(
+                    "server_readiness_probe",
+                    true, // Default to readiness probe, skip if false
+                    {},
+                );
+
+                if (!readinessProbeEnabled) {
+                    log.debug("Readiness check skipped due to feature flag");
+                    res.status(200);
+                    return;
+                }
+
                 // Check database connection
                 const dbConnection = await this.checkDatabaseConnection();
                 if (!dbConnection) {

memory-bank/activeContext.md

@@ -147,6 +147,7 @@ Initial exploration of the Gitpod codebase has revealed:
 - **Server Health Checks**: The Gitpod server uses two distinct health check mechanisms:
   - **Liveness Probe**: Checks the event loop lag to determine if the server is functioning properly
   - **Readiness Probe**: Checks database and SpiceDB connectivity to ensure the server is ready to handle requests
+    - Controlled by a ConfigCat feature flag `server_readiness_probe` (default: true) that can bypass the actual checks
 - **Critical Dependencies**: The server has critical external dependencies that must be operational:
   - **Database (TypeORM)**: Used for persistent storage
   - **SpiceDB**: Used for authorization and permission management

memory-bank/progress.md

@@ -211,10 +211,12 @@ No specific blockers or dependencies have been identified yet. This section will
     - Updated server container module and route configuration
     - Created PRD document for the readiness probe implementation
     - Updated Kubernetes deployment configuration to add the readiness probe
+    - Added ConfigCat feature flag `server_readiness_probe` to control readiness checks
   - Updated memory bank with new learnings:
     - Added information about server health checks and critical dependencies
     - Documented server architecture patterns and dependency injection
     - Added information about Kubernetes deployment configuration
+    - Documented feature flag implementation for readiness probe
 
 ## Next Evaluation Point
 

prd/001-readinessprobe-server.md

@@ -157,6 +157,19 @@ The readiness probe should be tested to ensure:
 - The probe runs every 10 seconds
 - The probe allows up to 3 failures before marking the pod as not ready
 
+## Feature Flag Control
+
+The readiness probe implementation includes a ConfigCat feature flag called `server_readiness_probe` that controls whether the actual connectivity checks are performed:
+
+- When the flag is set to `true` (default): The readiness probe will always return a 200 status code, bypassing the actual database and SpiceDB connectivity checks
+- When the flag is set to `false`: The readiness probe will perform the actual checks and return the appropriate status code based on the results
+
+This feature flag provides several benefits:
+
+1. **Gradual Rollout**: Allows for a gradual rollout of the readiness probe, which is useful for testing in production environments
+2. **Emergency Override**: If the readiness probe causes issues in production, the flag can be quickly toggled to bypass the checks without requiring a code deployment
+3. **Environment-Specific Configuration**: Different environments (dev, staging, production) can have different settings for the readiness probe
+
 ## Future Improvements
 
 - Add more sophisticated checks for SpiceDB connectivity, such as a simple permission check