Redis Comprehensive Troubleshooting CRE - GitHub Issue #132

rules/cre-2025-0173/redis-connection-timeout.yaml

@@ -0,0 +1,87 @@
+rules:
+- cre:
+    id: CRE-2025-0173
+    severity: 0
+    title: "Redis Connection Timeout and Connectivity Issues"
+    category: "in-memory-database-problem"
+    author: Prequel Community
+    description: |
+      Detects Redis connection timeout errors and connectivity failures that prevent clients from establishing or maintaining connections to the Redis server. These issues commonly occur during high load, network problems, or server resource exhaustion.
+    cause: |
+      - Network latency or packet loss between client and Redis server
+      - Redis server CPU overload causing slow response times
+      - Client connection pool exhaustion or misconfiguration
+      - Firewall or security group blocking connections
+      - Redis server reached max clients limit
+      - DNS resolution failures
+      - Redis server process crashed or unresponsive
+    impact: |
+      - Application unable to read/write cache data
+      - Increased latency for user requests
+      - Potential data inconsistency if writes fail silently
+      - Backend database overload due to cache unavailability
+      - Service degradation or complete outage
+      - Connection pool exhaustion leading to thread blocking
+    impactScore: 10
+    tags:
+      - redis
+      - connection
+      - timeout
+      - connectivity
+      - network
+    mitigation: |
+      IMMEDIATE ACTIONS:
+      - Verify Redis server is running: `systemctl status redis`
+      - Test connectivity: `redis-cli -h <host> -p <port> ping`
+      - Check current connections: `redis-cli CLIENT LIST | wc -l`
+      - Review max clients setting: `redis-cli CONFIG GET maxclients`
+
+      RECOVERY:
+      - Restart Redis service if unresponsive:
+        `systemctl restart redis`
+      - Increase connection timeout in client:
+        `redis.conf: timeout 300`
+      - Kill idle connections:
+        `redis-cli CLIENT KILL TYPE normal`
+      - Increase max clients limit:
+        `redis-cli CONFIG SET maxclients 10000`
+
+      NETWORK TROUBLESHOOTING:
+      - Check firewall rules: `iptables -L -n`
+      - Test network connectivity: `telnet redis-host 6379`
+      - Verify DNS resolution: `nslookup redis-host`
+      - Check for packet loss: `ping -c 100 redis-host`
+
+      PREVENTION:
+      - Implement connection pooling with proper sizing
+      - Configure appropriate timeout values
+      - Monitor connection metrics and set alerts
+      - Use Redis Sentinel or Cluster for high availability
+      - Implement circuit breaker pattern in clients
+      - Regular load testing and capacity planning
+    mitigationScore: 7
+    references:
+      - https://redis.io/docs/latest/operate/oss_and_stack/management/troubleshooting/#latency-issues
+      - https://redis.io/commands/client-list/
+      - https://redis.io/docs/latest/develop/clients/
+    applications:
+      - name: redis
+        version: "*"
+      - name: redis-cli
+        version: "*"
+    reports: 89
+  metadata:
+    kind: prequel
+    id: Hf8NpQr4VxKmLw9TbYaZe6
+    gen: 1
+  rule:
+    set:
+      window: 180s
+      event:
+        source: cre.log.redis
+      match:
+        - regex: "Connection timeout"
+        - regex: "Unable to connect to Redis"
+        - regex: "Could not connect to Redis"
+        - regex: "redis connection timeout"
+        - regex: "Connection pool.*exhausted"

rules/cre-2025-0173/test.log

@@ -0,0 +1,10 @@
+[2024-01-15 11:00:01,123] ERROR [RedisClient] Connection timeout errors
+Connection timeout while connecting to redis server at 192.168.1.100:6379
+Unable to connect to Redis server
+Could not connect to Redis at localhost:6379: Connection refused
+redis connection timeout
+Timeout connecting to redis://cache.example.com:6379
+Failed to connect to Redis
+Redis is not reachable
+Connection pool exhausted for redis server
+Connection reset by peer while communicating with redis server

rules/cre-2025-0174/redis-auth-failure.yaml

@@ -0,0 +1,85 @@
+rules:
+  - metadata:
+      kind: prequel
+      id: Bx5MnWq8TdRpLk3YfNvGa7
+      hash: Jk9Pf4XsNmRw2QbVtHeLy6
+    cre:
+      id: CRE-2025-0174
+      severity: 0
+      title: "Redis Authentication Failures and ACL Permission Denials"
+      category: "in-memory-database-problem"
+      author: Prequel Community
+      description: |
+        Detects Redis authentication failures including wrong passwords, missing authentication, and ACL permission denials. These errors prevent legitimate clients from accessing Redis and may indicate security misconfigurations or attempted unauthorized access.
+      cause: |
+        - Incorrect password provided by client
+        - Redis requirepass configured but client not sending auth
+        - ACL user lacks required permissions for commands
+        - Password rotation without updating client configs
+        - Expired or disabled ACL user accounts
+        - Misconfigured Redis AUTH settings
+      impact: |
+        - Complete inability to access Redis cache/data
+        - Application features dependent on Redis fail
+        - Service outages if Redis is critical infrastructure
+        - Security risk if authentication is bypassed
+        - Potential data exposure if misconfigured
+      tags:
+        - redis
+        - authentication
+        - security
+        - acl
+        - wrongpass
+      mitigation: |
+        IMMEDIATE ACTIONS:
+        - Verify Redis auth configuration: `redis-cli CONFIG GET requirepass`
+        - Test authentication: `redis-cli -a <password> ping`
+        - Check ACL users: `redis-cli ACL LIST`
+        - Review client connection strings for correct credentials
+
+        RECOVERY:
+        - Update client password configuration
+        - Reset Redis password if needed:
+          `redis-cli CONFIG SET requirepass newpassword`
+        - Fix ACL permissions for user:
+          `redis-cli ACL SETUSER username +@all`
+        - Disable auth temporarily (UNSAFE):
+          `redis-cli CONFIG SET requirepass ""`
+
+        ACL TROUBLESHOOTING:
+        - List user permissions: `redis-cli ACL GETUSER username`
+        - Grant specific command access:
+          `redis-cli ACL SETUSER username +get +set +del`
+        - Create new user with full access:
+          `redis-cli ACL SETUSER newuser on >password +@all`
+
+        PREVENTION:
+        - Use environment variables for passwords
+        - Implement proper secret management
+        - Regular password rotation with coordination
+        - Monitor authentication failure rates
+        - Use ACL for fine-grained access control
+        - Document authentication requirements
+      references:
+        - https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/
+        - https://redis.io/commands/auth/
+        - https://redis.io/docs/latest/operate/oss_and_stack/management/security/
+      applications:
+        - name: redis
+          version: ">=6.0.0"
+      impactScore: 7
+      mitigationScore: 8
+      reports: 156
+    rule:
+      set:
+        window: 120s
+        event:
+          source: cre.log.redis
+        match:
+          - regex: "WRONGPASS invalid username-password pair"
+          - regex: "NOAUTH Authentication required"
+          - regex: "ERR invalid password"
+          - regex: "ERR wrong password"
+          - regex: "NOPERM.*has no permissions to run"
+          - regex: "ERR ACL.*permission denied"
+          - regex: "AUTH failed.*invalid.*credentials"

rules/cre-2025-0174/test.log

@@ -0,0 +1,8 @@
+2024-01-15 12:00:01.123 [ERROR] Redis authentication failed: WRONGPASS invalid username-password pair or user is disabled.
+2024-01-15 12:00:02.234 [ERROR] (error) NOAUTH Authentication required.
+2024-01-15 12:00:03.345 [ERROR] redis.exceptions.ResponseError: ERR invalid password
+2024-01-15 12:00:04.456 [ERROR] Command rejected: ERR wrong password provided
+2024-01-15 12:00:05.567 [ERROR] ACL violation: NOPERM User 'readonly' has no permissions to run the 'SET' command
+2024-01-15 12:00:06.678 [ERROR] ERR ACL permission denied for user 'app_user' on command 'FLUSHDB'
+2024-01-15 12:00:07.789 [ERROR] AUTH failed: invalid username/password credentials
+2024-01-15 12:00:08.890 [WARN] Redis server returned: NOAUTH Authentication required for this operation

rules/cre-2025-0175/redis-replication-failure.yaml

@@ -0,0 +1,91 @@
+rules:
+  - metadata:
+      kind: prequel
+      id: Qm7WxPr3NbKfLs9YhVaEz2
+      hash: Td5Gn8XqPmWsRf4BkLyVe3
+    cre:
+      id: CRE-2025-0175
+      severity: 0
+      title: "Redis Master-Replica Synchronization Failure"
+      category: "in-memory-database-problem"
+      author: Prequel Community
+      description: |
+        Detects failures in Redis master-replica synchronization including broken replication links, sync timeouts, and full resync loops. These issues compromise data consistency and high availability in Redis deployments.
+      cause: |
+        - Network partition between master and replica
+        - Replica unable to keep up with master write load
+        - Insufficient replica output buffer size
+        - Master rewrite of AOF/RDB during sync
+        - Replica disk I/O too slow for sync
+        - Version incompatibility between master and replica
+        - Replication backlog size too small
+      impact: |
+        - Replicas serve stale or inconsistent data
+        - Failover capability compromised
+        - Read scaling degraded with out-of-sync replicas
+        - Full resync causing performance impact
+        - Potential data loss during failover
+        - Increased load on master during resync attempts
+      tags:
+        - redis
+        - replication
+        - master-replica
+        - sync
+        - psync
+      mitigation: |
+        IMMEDIATE ACTIONS:
+        - Check replication status: `redis-cli INFO replication`
+        - Verify replica connectivity: `redis-cli -h replica ping`
+        - Monitor sync progress: `redis-cli INFO | grep master_sync`
+        - Check replication lag: `redis-cli INFO | grep master_repl_offset`
+
+        RECOVERY:
+        - Restart replication on replica:
+          ```
+          redis-cli REPLICAOF NO ONE
+          redis-cli REPLICAOF master-host master-port
+          ```
+        - Increase replication backlog:
+          `redis-cli CONFIG SET repl-backlog-size 256mb`
+        - Adjust replica output buffer:
+          `redis-cli CONFIG SET client-output-buffer-limit "replica 256mb 64mb 60"`
+        - Force full resync if partial sync fails:
+          `redis-cli PSYNC replicationid -1`
+
+        TROUBLESHOOTING:
+        - Check network latency: `ping -c 100 master-host`
+        - Monitor disk I/O: `iostat -x 1`
+        - Review Redis logs: `tail -f /var/log/redis/redis-server.log`
+        - Verify firewall rules allow port 6379
+
+        PREVENTION:
+        - Size replication backlog appropriately
+        - Monitor replication lag metrics
+        - Use dedicated network for replication
+        - Optimize disk I/O on replicas
+        - Regular testing of failover procedures
+        - Keep master and replica versions in sync
+      references:
+        - https://redis.io/docs/latest/operate/oss_and_stack/management/replication/
+        - https://redis.io/commands/psync/
+        - https://redis.io/topics/persistence
+      applications:
+        - name: redis
+          version: ">=2.8.0"
+      impactScore: 8
+      mitigationScore: 6
+      reports: 67
+    rule:
+      set:
+        window: 300s
+        event:
+          source: cre.log.redis
+        match:
+          - regex: "Unable to connect to MASTER"
+          - regex: "MASTER.*sync.*timeout"
+          - regex: "Partial resynchronization not accepted"
+          - regex: "SYNC failed.*Cannot allocate memory"
+          - regex: "Full resync.*aborted"
+          - regex: "Replication.*broken.*disconnected"
+          - regex: "Error condition on socket for SYNC"
+          - regex: "master_link_status:down"

rules/cre-2025-0175/test.log

@@ -0,0 +1,8 @@
+2024-01-15 13:00:01.123 [ERROR] Unable to connect to MASTER: connection refused
+2024-01-15 13:00:02.234 [ERROR] MASTER <-> REPLICA sync: timeout in receiving data from master
+2024-01-15 13:00:03.345 [WARN] Partial resynchronization not accepted: full resync required
+2024-01-15 13:00:04.456 [ERROR] SYNC failed: Cannot allocate memory for replication backlog
+2024-01-15 13:00:05.567 [ERROR] Full resync from master aborted: read error
+2024-01-15 13:00:06.678 [CRITICAL] Replication link broken: disconnected from master
+2024-01-15 13:00:07.789 [ERROR] Error condition on socket for SYNC: Connection reset by peer
+2024-01-15 13:00:08.890 [INFO] master_link_status:down master_link_down_since_seconds:45

rules/cre-2025-0176/redis-persistence-failure.yaml

@@ -0,0 +1,96 @@
+rules:
+  - metadata:
+      kind: prequel
+      id: Yx4NmQp7RdWfKs8LbHaVt9
+      hash: Pm3Xk6WsNbRq5TfGeLyVn2
+    cre:
+      id: CRE-2025-0176
+      severity: 0
+      title: "Redis Persistence Failure - MISCONF Disk Write Errors"
+      category: "in-memory-database-problem"
+      author: Prequel Community
+      description: |
+        Detects Redis MISCONF errors when the server cannot persist data to disk due to RDB/AOF write failures. This critical error prevents Redis from saving snapshots and may lead to data loss on restart.
+      cause: |
+        - Disk full or insufficient space for RDB/AOF files
+        - File system permissions preventing writes
+        - Disk I/O errors or hardware failures
+        - AOF file corruption
+        - Background save process (BGSAVE) failures
+        - Operating system resource limits reached
+        - File system mounted read-only
+      impact: |
+        - Redis stops accepting write commands (by default)
+        - Complete data loss on server restart
+        - Inability to create backups
+        - Replication to slaves may fail
+        - Application write operations blocked
+        - Service degradation or outage
+      tags:
+        - redis
+        - persistence
+        - misconf
+        - rdb
+        - aof
+        - disk
+      mitigation: |
+        IMMEDIATE ACTIONS:
+        - Check disk space: `df -h /var/lib/redis`
+        - Review Redis persistence status: `redis-cli INFO persistence`
+        - Check last save status: `redis-cli LASTSAVE`
+        - Verify file permissions: `ls -la /var/lib/redis/`
+
+        RECOVERY:
+        - Free disk space:
+          ```
+          # Clean old logs
+          find /var/log -name "*.gz" -delete
+          # Remove old backups
+          rm /var/lib/redis/dump.rdb.old
+          ```
+        - Fix permissions:
+          `chown redis:redis /var/lib/redis/*`
+        - Temporarily disable persistence (RISKY):
+          ```
+          redis-cli CONFIG SET save ""
+          redis-cli CONFIG SET stop-writes-on-bgsave-error no
+          ```
+        - Force manual save after fixing:
+          `redis-cli BGSAVE`
+
+        DISK TROUBLESHOOTING:
+        - Check disk errors: `dmesg | grep -i error`
+        - Verify filesystem: `fsck /dev/sda1`
+        - Monitor I/O: `iostat -x 1`
+        - Check mount options: `mount | grep redis`
+
+        PREVENTION:
+        - Monitor disk usage with alerts at 80% capacity
+        - Regular disk cleanup automation
+        - Separate partition for Redis data
+        - Configure appropriate save intervals
+        - Use both RDB and AOF for redundancy
+        - Regular backup verification
+      references:
+        - https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/
+        - https://redis.io/commands/bgsave/
+        - https://redis.io/topics/problems#background-saving-fails-with-a-fork-error
+      applications:
+        - name: redis
+          version: "*"
+      impactScore: 9
+      mitigationScore: 7
+      reports: 234
+    rule:
+      set:
+        window: 180s
+        event:
+          source: cre.log.redis
+        match:
+          - regex: "MISCONF Redis is configured to save RDB snapshots.*unable to persist.*disk"
+          - regex: "Can't save in background"
+          - regex: "Failed opening.*rdb for saving"
+          - regex: "Write error saving DB on disk"
+          - regex: "AOF.*write error"
+          - regex: "Error moving temp.*file.*final destination"
+          - regex: "BGSAVE.*failed.*No space"