diff --git a/rules/cre-2025-0173/redis-connection-timeout.yaml b/rules/cre-2025-0173/redis-connection-timeout.yaml new file mode 100644 index 0000000..1c01f01 --- /dev/null +++ b/rules/cre-2025-0173/redis-connection-timeout.yaml @@ -0,0 +1,87 @@ +rules: +- cre: + id: CRE-2025-0173 + severity: 0 + title: "Redis Connection Timeout and Connectivity Issues" + category: "in-memory-database-problem" + author: Prequel Community + description: | + Detects Redis connection timeout errors and connectivity failures that prevent clients from establishing or maintaining connections to the Redis server. These issues commonly occur during high load, network problems, or server resource exhaustion. + cause: | + - Network latency or packet loss between client and Redis server + - Redis server CPU overload causing slow response times + - Client connection pool exhaustion or misconfiguration + - Firewall or security group blocking connections + - Redis server reached max clients limit + - DNS resolution failures + - Redis server process crashed or unresponsive + impact: | + - Application unable to read/write cache data + - Increased latency for user requests + - Potential data inconsistency if writes fail silently + - Backend database overload due to cache unavailability + - Service degradation or complete outage + - Connection pool exhaustion leading to thread blocking + impactScore: 10 + tags: + - redis + - connection + - timeout + - connectivity + - network + mitigation: | + IMMEDIATE ACTIONS: + - Verify Redis server is running: `systemctl status redis` + - Test connectivity: `redis-cli -h -p ping` + - Check current connections: `redis-cli CLIENT LIST | wc -l` + - Review max clients setting: `redis-cli CONFIG GET maxclients` + + RECOVERY: + - Restart Redis service if unresponsive: + `systemctl restart redis` + - Increase connection timeout in client: + `redis.conf: timeout 300` + - Kill idle connections: + `redis-cli CLIENT KILL TYPE normal` + - Increase max clients limit: + `redis-cli CONFIG SET maxclients 10000` + + NETWORK TROUBLESHOOTING: + - Check firewall rules: `iptables -L -n` + - Test network connectivity: `telnet redis-host 6379` + - Verify DNS resolution: `nslookup redis-host` + - Check for packet loss: `ping -c 100 redis-host` + + PREVENTION: + - Implement connection pooling with proper sizing + - Configure appropriate timeout values + - Monitor connection metrics and set alerts + - Use Redis Sentinel or Cluster for high availability + - Implement circuit breaker pattern in clients + - Regular load testing and capacity planning + mitigationScore: 7 + references: + - https://redis.io/docs/latest/operate/oss_and_stack/management/troubleshooting/#latency-issues + - https://redis.io/commands/client-list/ + - https://redis.io/docs/latest/develop/clients/ + applications: + - name: redis + version: "*" + - name: redis-cli + version: "*" + reports: 89 + metadata: + kind: prequel + id: Hf8NpQr4VxKmLw9TbYaZe6 + gen: 1 + rule: + set: + window: 180s + event: + source: cre.log.redis + match: + - regex: "Connection timeout" + - regex: "Unable to connect to Redis" + - regex: "Could not connect to Redis" + - regex: "redis connection timeout" + - regex: "Connection pool.*exhausted" \ No newline at end of file diff --git a/rules/cre-2025-0173/test.log b/rules/cre-2025-0173/test.log new file mode 100644 index 0000000..25e199f --- /dev/null +++ b/rules/cre-2025-0173/test.log @@ -0,0 +1,10 @@ +[2024-01-15 11:00:01,123] ERROR [RedisClient] Connection timeout errors +Connection timeout while connecting to redis server at 192.168.1.100:6379 +Unable to connect to Redis server +Could not connect to Redis at localhost:6379: Connection refused +redis connection timeout +Timeout connecting to redis://cache.example.com:6379 +Failed to connect to Redis +Redis is not reachable +Connection pool exhausted for redis server +Connection reset by peer while communicating with redis server \ No newline at end of file diff --git a/rules/cre-2025-0174/redis-auth-failure.yaml b/rules/cre-2025-0174/redis-auth-failure.yaml new file mode 100644 index 0000000..63de2ff --- /dev/null +++ b/rules/cre-2025-0174/redis-auth-failure.yaml @@ -0,0 +1,85 @@ +rules: + - metadata: + kind: prequel + id: Bx5MnWq8TdRpLk3YfNvGa7 + hash: Jk9Pf4XsNmRw2QbVtHeLy6 + cre: + id: CRE-2025-0174 + severity: 0 + title: "Redis Authentication Failures and ACL Permission Denials" + category: "in-memory-database-problem" + author: Prequel Community + description: | + Detects Redis authentication failures including wrong passwords, missing authentication, and ACL permission denials. These errors prevent legitimate clients from accessing Redis and may indicate security misconfigurations or attempted unauthorized access. + cause: | + - Incorrect password provided by client + - Redis requirepass configured but client not sending auth + - ACL user lacks required permissions for commands + - Password rotation without updating client configs + - Expired or disabled ACL user accounts + - Misconfigured Redis AUTH settings + impact: | + - Complete inability to access Redis cache/data + - Application features dependent on Redis fail + - Service outages if Redis is critical infrastructure + - Security risk if authentication is bypassed + - Potential data exposure if misconfigured + tags: + - redis + - authentication + - security + - acl + - wrongpass + mitigation: | + IMMEDIATE ACTIONS: + - Verify Redis auth configuration: `redis-cli CONFIG GET requirepass` + - Test authentication: `redis-cli -a ping` + - Check ACL users: `redis-cli ACL LIST` + - Review client connection strings for correct credentials + + RECOVERY: + - Update client password configuration + - Reset Redis password if needed: + `redis-cli CONFIG SET requirepass newpassword` + - Fix ACL permissions for user: + `redis-cli ACL SETUSER username +@all` + - Disable auth temporarily (UNSAFE): + `redis-cli CONFIG SET requirepass ""` + + ACL TROUBLESHOOTING: + - List user permissions: `redis-cli ACL GETUSER username` + - Grant specific command access: + `redis-cli ACL SETUSER username +get +set +del` + - Create new user with full access: + `redis-cli ACL SETUSER newuser on >password +@all` + + PREVENTION: + - Use environment variables for passwords + - Implement proper secret management + - Regular password rotation with coordination + - Monitor authentication failure rates + - Use ACL for fine-grained access control + - Document authentication requirements + references: + - https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/ + - https://redis.io/commands/auth/ + - https://redis.io/docs/latest/operate/oss_and_stack/management/security/ + applications: + - name: redis + version: ">=6.0.0" + impactScore: 7 + mitigationScore: 8 + reports: 156 + rule: + set: + window: 120s + event: + source: cre.log.redis + match: + - regex: "WRONGPASS invalid username-password pair" + - regex: "NOAUTH Authentication required" + - regex: "ERR invalid password" + - regex: "ERR wrong password" + - regex: "NOPERM.*has no permissions to run" + - regex: "ERR ACL.*permission denied" + - regex: "AUTH failed.*invalid.*credentials" \ No newline at end of file diff --git a/rules/cre-2025-0174/test.log b/rules/cre-2025-0174/test.log new file mode 100644 index 0000000..8f6840e --- /dev/null +++ b/rules/cre-2025-0174/test.log @@ -0,0 +1,8 @@ +2024-01-15 12:00:01.123 [ERROR] Redis authentication failed: WRONGPASS invalid username-password pair or user is disabled. +2024-01-15 12:00:02.234 [ERROR] (error) NOAUTH Authentication required. +2024-01-15 12:00:03.345 [ERROR] redis.exceptions.ResponseError: ERR invalid password +2024-01-15 12:00:04.456 [ERROR] Command rejected: ERR wrong password provided +2024-01-15 12:00:05.567 [ERROR] ACL violation: NOPERM User 'readonly' has no permissions to run the 'SET' command +2024-01-15 12:00:06.678 [ERROR] ERR ACL permission denied for user 'app_user' on command 'FLUSHDB' +2024-01-15 12:00:07.789 [ERROR] AUTH failed: invalid username/password credentials +2024-01-15 12:00:08.890 [WARN] Redis server returned: NOAUTH Authentication required for this operation \ No newline at end of file diff --git a/rules/cre-2025-0175/redis-replication-failure.yaml b/rules/cre-2025-0175/redis-replication-failure.yaml new file mode 100644 index 0000000..a8a5242 --- /dev/null +++ b/rules/cre-2025-0175/redis-replication-failure.yaml @@ -0,0 +1,91 @@ +rules: + - metadata: + kind: prequel + id: Qm7WxPr3NbKfLs9YhVaEz2 + hash: Td5Gn8XqPmWsRf4BkLyVe3 + cre: + id: CRE-2025-0175 + severity: 0 + title: "Redis Master-Replica Synchronization Failure" + category: "in-memory-database-problem" + author: Prequel Community + description: | + Detects failures in Redis master-replica synchronization including broken replication links, sync timeouts, and full resync loops. These issues compromise data consistency and high availability in Redis deployments. + cause: | + - Network partition between master and replica + - Replica unable to keep up with master write load + - Insufficient replica output buffer size + - Master rewrite of AOF/RDB during sync + - Replica disk I/O too slow for sync + - Version incompatibility between master and replica + - Replication backlog size too small + impact: | + - Replicas serve stale or inconsistent data + - Failover capability compromised + - Read scaling degraded with out-of-sync replicas + - Full resync causing performance impact + - Potential data loss during failover + - Increased load on master during resync attempts + tags: + - redis + - replication + - master-replica + - sync + - psync + mitigation: | + IMMEDIATE ACTIONS: + - Check replication status: `redis-cli INFO replication` + - Verify replica connectivity: `redis-cli -h replica ping` + - Monitor sync progress: `redis-cli INFO | grep master_sync` + - Check replication lag: `redis-cli INFO | grep master_repl_offset` + + RECOVERY: + - Restart replication on replica: + ``` + redis-cli REPLICAOF NO ONE + redis-cli REPLICAOF master-host master-port + ``` + - Increase replication backlog: + `redis-cli CONFIG SET repl-backlog-size 256mb` + - Adjust replica output buffer: + `redis-cli CONFIG SET client-output-buffer-limit "replica 256mb 64mb 60"` + - Force full resync if partial sync fails: + `redis-cli PSYNC replicationid -1` + + TROUBLESHOOTING: + - Check network latency: `ping -c 100 master-host` + - Monitor disk I/O: `iostat -x 1` + - Review Redis logs: `tail -f /var/log/redis/redis-server.log` + - Verify firewall rules allow port 6379 + + PREVENTION: + - Size replication backlog appropriately + - Monitor replication lag metrics + - Use dedicated network for replication + - Optimize disk I/O on replicas + - Regular testing of failover procedures + - Keep master and replica versions in sync + references: + - https://redis.io/docs/latest/operate/oss_and_stack/management/replication/ + - https://redis.io/commands/psync/ + - https://redis.io/topics/persistence + applications: + - name: redis + version: ">=2.8.0" + impactScore: 8 + mitigationScore: 6 + reports: 67 + rule: + set: + window: 300s + event: + source: cre.log.redis + match: + - regex: "Unable to connect to MASTER" + - regex: "MASTER.*sync.*timeout" + - regex: "Partial resynchronization not accepted" + - regex: "SYNC failed.*Cannot allocate memory" + - regex: "Full resync.*aborted" + - regex: "Replication.*broken.*disconnected" + - regex: "Error condition on socket for SYNC" + - regex: "master_link_status:down" \ No newline at end of file diff --git a/rules/cre-2025-0175/test.log b/rules/cre-2025-0175/test.log new file mode 100644 index 0000000..0dfda20 --- /dev/null +++ b/rules/cre-2025-0175/test.log @@ -0,0 +1,8 @@ +2024-01-15 13:00:01.123 [ERROR] Unable to connect to MASTER: connection refused +2024-01-15 13:00:02.234 [ERROR] MASTER <-> REPLICA sync: timeout in receiving data from master +2024-01-15 13:00:03.345 [WARN] Partial resynchronization not accepted: full resync required +2024-01-15 13:00:04.456 [ERROR] SYNC failed: Cannot allocate memory for replication backlog +2024-01-15 13:00:05.567 [ERROR] Full resync from master aborted: read error +2024-01-15 13:00:06.678 [CRITICAL] Replication link broken: disconnected from master +2024-01-15 13:00:07.789 [ERROR] Error condition on socket for SYNC: Connection reset by peer +2024-01-15 13:00:08.890 [INFO] master_link_status:down master_link_down_since_seconds:45 \ No newline at end of file diff --git a/rules/cre-2025-0176/redis-persistence-failure.yaml b/rules/cre-2025-0176/redis-persistence-failure.yaml new file mode 100644 index 0000000..74092dc --- /dev/null +++ b/rules/cre-2025-0176/redis-persistence-failure.yaml @@ -0,0 +1,96 @@ +rules: + - metadata: + kind: prequel + id: Yx4NmQp7RdWfKs8LbHaVt9 + hash: Pm3Xk6WsNbRq5TfGeLyVn2 + cre: + id: CRE-2025-0176 + severity: 0 + title: "Redis Persistence Failure - MISCONF Disk Write Errors" + category: "in-memory-database-problem" + author: Prequel Community + description: | + Detects Redis MISCONF errors when the server cannot persist data to disk due to RDB/AOF write failures. This critical error prevents Redis from saving snapshots and may lead to data loss on restart. + cause: | + - Disk full or insufficient space for RDB/AOF files + - File system permissions preventing writes + - Disk I/O errors or hardware failures + - AOF file corruption + - Background save process (BGSAVE) failures + - Operating system resource limits reached + - File system mounted read-only + impact: | + - Redis stops accepting write commands (by default) + - Complete data loss on server restart + - Inability to create backups + - Replication to slaves may fail + - Application write operations blocked + - Service degradation or outage + tags: + - redis + - persistence + - misconf + - rdb + - aof + - disk + mitigation: | + IMMEDIATE ACTIONS: + - Check disk space: `df -h /var/lib/redis` + - Review Redis persistence status: `redis-cli INFO persistence` + - Check last save status: `redis-cli LASTSAVE` + - Verify file permissions: `ls -la /var/lib/redis/` + + RECOVERY: + - Free disk space: + ``` + # Clean old logs + find /var/log -name "*.gz" -delete + # Remove old backups + rm /var/lib/redis/dump.rdb.old + ``` + - Fix permissions: + `chown redis:redis /var/lib/redis/*` + - Temporarily disable persistence (RISKY): + ``` + redis-cli CONFIG SET save "" + redis-cli CONFIG SET stop-writes-on-bgsave-error no + ``` + - Force manual save after fixing: + `redis-cli BGSAVE` + + DISK TROUBLESHOOTING: + - Check disk errors: `dmesg | grep -i error` + - Verify filesystem: `fsck /dev/sda1` + - Monitor I/O: `iostat -x 1` + - Check mount options: `mount | grep redis` + + PREVENTION: + - Monitor disk usage with alerts at 80% capacity + - Regular disk cleanup automation + - Separate partition for Redis data + - Configure appropriate save intervals + - Use both RDB and AOF for redundancy + - Regular backup verification + references: + - https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/ + - https://redis.io/commands/bgsave/ + - https://redis.io/topics/problems#background-saving-fails-with-a-fork-error + applications: + - name: redis + version: "*" + impactScore: 9 + mitigationScore: 7 + reports: 234 + rule: + set: + window: 180s + event: + source: cre.log.redis + match: + - regex: "MISCONF Redis is configured to save RDB snapshots.*unable to persist.*disk" + - regex: "Can't save in background" + - regex: "Failed opening.*rdb for saving" + - regex: "Write error saving DB on disk" + - regex: "AOF.*write error" + - regex: "Error moving temp.*file.*final destination" + - regex: "BGSAVE.*failed.*No space" \ No newline at end of file diff --git a/rules/cre-2025-0176/test.log b/rules/cre-2025-0176/test.log new file mode 100644 index 0000000..41fa7ea --- /dev/null +++ b/rules/cre-2025-0176/test.log @@ -0,0 +1,8 @@ +2024-01-15 14:00:01.123 [ERROR] MISCONF Redis is configured to save RDB snapshots, but it's currently unable to persist to disk +2024-01-15 14:00:02.234 [ERROR] Can't save in background: fork: Cannot allocate memory +2024-01-15 14:00:03.345 [ERROR] Failed opening dump.rdb for saving: No space left on device +2024-01-15 14:00:04.456 [ERROR] Write error saving DB on disk: No space left on device +2024-01-15 14:00:05.567 [ERROR] AOF rewrite: write error - no space left on device +2024-01-15 14:00:06.678 [ERROR] Error moving temp DB file dump.rdb.tmp on the final destination dump.rdb +2024-01-15 14:00:07.789 [ERROR] BGSAVE failed: No space left on device +2024-01-15 14:00:08.890 [CRITICAL] Redis persistence disabled due to repeated write failures \ No newline at end of file diff --git a/rules/cre-2025-0177/redis-slow-query.yaml b/rules/cre-2025-0177/redis-slow-query.yaml new file mode 100644 index 0000000..d3b5012 --- /dev/null +++ b/rules/cre-2025-0177/redis-slow-query.yaml @@ -0,0 +1,94 @@ +rules: + - metadata: + kind: prequel + id: Kp9XmNw4RbTfGs7YqLaVe3 + hash: Hm5Wd8PsKnRq2BfVtXyLm6 + cre: + id: CRE-2025-0177 + severity: 0 + title: "Redis Slow Query Performance Degradation" + category: "in-memory-database-problem" + author: Prequel Community + description: | + Detects slow query execution in Redis that exceeds configured thresholds. Slow queries indicate performance problems that can impact overall Redis responsiveness and application latency. + cause: | + - Large dataset operations (KEYS *, SMEMBERS on huge sets) + - Inefficient Lua scripts with complex logic + - Blocking operations on large lists/sets/sorted sets + - O(N) operations on large datasets + - High CPU contention from concurrent operations + - Memory swapping causing disk I/O + - Unoptimized data structures + impact: | + - Increased application response times + - Redis becomes unresponsive during slow operations + - Connection timeouts for other clients + - Queue buildup and memory pressure + - Potential cascading failures + - Poor user experience + tags: + - redis + - performance + - slowlog + - latency + - slow-query + mitigation: | + IMMEDIATE ACTIONS: + - Check slowlog: `redis-cli SLOWLOG GET 10` + - Monitor current operations: `redis-cli --latency` + - Identify blocking clients: `redis-cli CLIENT LIST` + - Check CPU usage: `redis-cli INFO cpu` + + OPTIMIZATION: + - Replace KEYS with SCAN: + ``` + # Bad: KEYS pattern* + # Good: SCAN 0 MATCH pattern* COUNT 100 + ``` + - Use pagination for large collections: + ``` + ZRANGE key start stop + LRANGE key start stop + ``` + - Optimize Lua scripts: + - Minimize Redis calls within scripts + - Avoid complex computations + - Break large operations into smaller batches + + CONFIGURATION: + - Adjust slowlog threshold: + `redis-cli CONFIG SET slowlog-log-slower-than 10000` + - Increase slowlog size: + `redis-cli CONFIG SET slowlog-max-len 128` + - Enable latency monitoring: + `redis-cli CONFIG SET latency-monitor-threshold 100` + + PREVENTION: + - Regular slowlog analysis + - Use appropriate data structures + - Implement query timeouts in clients + - Cache computation results + - Use Redis modules for complex operations + - Monitor and alert on slow query patterns + references: + - https://redis.io/commands/slowlog/ + - https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/latency/ + - https://redis.io/docs/latest/develop/use/patterns/ + applications: + - name: redis + version: ">=2.2.12" + impactScore: 6 + mitigationScore: 8 + reports: 145 + rule: + set: + window: 300s + event: + source: cre.log.redis + match: + - regex: "Slow log.*microseconds|slowlog.*exceeded" + - regex: "command.*took.*milliseconds|Query execution.*slow" + - regex: "SLOWLOG.*latency.*threshold" + - regex: "Latency spike detected" + - regex: "Operation.*exceeded.*timeout" + - regex: "Redis.*high latency.*detected" \ No newline at end of file diff --git a/rules/cre-2025-0177/test.log b/rules/cre-2025-0177/test.log new file mode 100644 index 0000000..0b1b443 --- /dev/null +++ b/rules/cre-2025-0177/test.log @@ -0,0 +1,8 @@ +2024-01-15 15:00:01.123 [WARN] Slow log entry: command KEYS * took 15234 microseconds +2024-01-15 15:00:02.234 [WARN] Redis slowlog threshold exceeded for command: SMEMBERS large_set +2024-01-15 15:00:03.345 [WARN] Query execution slow: ZRANGE bigset 0 -1 took 8500 milliseconds +2024-01-15 15:00:04.456 [WARN] SLOWLOG: latency threshold exceeded (5000 microseconds) +2024-01-15 15:00:05.567 [WARN] Latency spike detected: 12000 microseconds for GET operation +2024-01-15 15:00:06.678 [ERROR] Operation LRANGE exceeded timeout of 5000ms +2024-01-15 15:00:07.789 [WARN] Redis server high latency detected: avg response time 850ms +2024-01-15 15:00:08.890 [WARN] Lua script execution took 25000 microseconds \ No newline at end of file diff --git a/rules/cre-2025-0178/redis-readonly-replica.yaml b/rules/cre-2025-0178/redis-readonly-replica.yaml new file mode 100644 index 0000000..000b416 --- /dev/null +++ b/rules/cre-2025-0178/redis-readonly-replica.yaml @@ -0,0 +1,92 @@ +rules: + - metadata: + kind: prequel + id: Ws3YnPm6KbRfTq8XdLaGe5 + hash: Vk7Nf4QsWmPr2BxLtHeYn9 + cre: + id: CRE-2025-0178 + severity: 0 + title: "Redis Read-Only Replica Write Attempt Error" + category: "in-memory-database-problem" + author: Prequel Community + description: | + Detects attempts to perform write operations on Redis read-only replicas. This error indicates application misconfiguration where clients are incorrectly routing write commands to replica instances instead of the master. + cause: | + - Client connected to replica instead of master + - Load balancer incorrectly routing writes to replicas + - Failover not properly handled by application + - Misconfigured connection strings + - Redis Sentinel not properly directing traffic + - Application unaware of Redis topology changes + impact: | + - Write operations fail with READONLY errors + - Data inconsistency if retried on different nodes + - Application functionality degraded + - User transactions may fail + - Potential data loss if not properly handled + tags: + - redis + - readonly + - replica + - replication + - write-error + mitigation: | + IMMEDIATE ACTIONS: + - Verify connection target: `redis-cli INFO replication` + - Check if connected to replica: `redis-cli INFO | grep role` + - Find master instance: `redis-cli INFO | grep master_host` + - List all replicas: `redis-cli --cluster nodes` + + RECOVERY: + - Redirect clients to master: + ``` + # Update connection string to master + redis://master-host:6379 + ``` + - For Redis Sentinel setups: + ``` + # Connect via Sentinel for automatic master discovery + sentinel://sentinel-host:26379/mymaster + ``` + - Temporary replica promotion (if master failed): + `redis-cli REPLICAOF NO ONE` + + CLIENT CONFIGURATION: + - Configure read/write splitting: + ``` + # Writes to master + master_client = Redis(host='master') + # Reads from replica + replica_client = Redis(host='replica') + ``` + - Use Redis Cluster aware clients + - Implement retry logic with master discovery + + PREVENTION: + - Use Redis Sentinel for automatic failover + - Implement proper connection pooling + - Separate read and write connection pools + - Monitor topology changes + - Document Redis architecture clearly + - Regular failover testing + references: + - https://redis.io/docs/latest/operate/oss_and_stack/management/replication/#read-only-replica + - https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/ + - https://redis.io/commands/readonly/ + applications: + - name: redis + version: ">=2.6.0" + impactScore: 5 + mitigationScore: 9 + reports: 89 + rule: + set: + window: 120s + event: + source: cre.log.redis + match: + - regex: "READONLY You can't write against a read only replica" + - regex: "READONLY You cannot write against a read only replica" + - regex: "-READONLY.*write.*replica" + - regex: "ERR.*operation not permitted.*readonly" + - regex: "Cannot execute write command on readonly replica" \ No newline at end of file diff --git a/rules/cre-2025-0178/test.log b/rules/cre-2025-0178/test.log new file mode 100644 index 0000000..692d25f --- /dev/null +++ b/rules/cre-2025-0178/test.log @@ -0,0 +1,8 @@ +2024-01-15 16:00:01.123 [ERROR] READONLY You can't write against a read only replica. +2024-01-15 16:00:02.234 [ERROR] Redis command failed: READONLY You cannot write against a read only replica +2024-01-15 16:00:03.345 [ERROR] -READONLY error: Cannot write to replica instance +2024-01-15 16:00:04.456 [ERROR] ERR operation not permitted on readonly replica +2024-01-15 16:00:05.567 [ERROR] Cannot execute write command on readonly replica: SET key value +2024-01-15 16:00:06.678 [WARN] Write operation rejected: READONLY You can't write against a read only replica +2024-01-15 16:00:07.789 [ERROR] Application error: Attempted DEL on readonly replica +2024-01-15 16:00:08.890 [ERROR] Transaction failed: READONLY You cannot write against a read only replica \ No newline at end of file diff --git a/rules/cre-2025-0180/redis-aof-corruption.yaml b/rules/cre-2025-0180/redis-aof-corruption.yaml new file mode 100644 index 0000000..20e41bc --- /dev/null +++ b/rules/cre-2025-0180/redis-aof-corruption.yaml @@ -0,0 +1,108 @@ +rules: + - metadata: + kind: prequel + id: Rm7WpNx4KfQbTs9YeLaGd3 + hash: Xk5Nf8PmWsRq2VbLtHeYn6 + cre: + id: CRE-2025-0180 + severity: 0 + title: "Redis AOF File Corruption and Recovery Failure" + category: "in-memory-database-problem" + author: Prequel Community + description: | + Detects Redis Append-Only File (AOF) corruption that prevents Redis from starting or causes data loss. AOF corruption typically occurs due to unexpected shutdowns, disk errors, or incomplete writes during crashes. + cause: | + - Unexpected server shutdown during AOF write + - Disk errors or bad sectors + - Filesystem corruption + - Incomplete AOF rewrite operation + - Power failure during persistence + - Manual AOF file modification + - Disk full during AOF append + - Memory corruption affecting AOF buffer + impact: | + - Redis fails to start with corrupted AOF + - Potential data loss up to corruption point + - Service complete unavailability + - Recovery requires manual intervention + - Possible loss of recent transactions + - Extended downtime for large AOF files + tags: + - redis + - aof + - corruption + - persistence + - recovery + mitigation: | + IMMEDIATE ACTIONS: + - Check Redis logs: `tail -100 /var/log/redis/redis-server.log` + - Verify AOF file integrity: `redis-check-aof /var/lib/redis/appendonly.aof` + - Backup corrupted AOF before repair: + `cp /var/lib/redis/appendonly.aof /var/lib/redis/appendonly.aof.backup` + + RECOVERY: + - Option 1: Repair AOF file (may lose some data): + ``` + redis-check-aof --fix /var/lib/redis/appendonly.aof + systemctl start redis + ``` + - Option 2: Truncate corrupted portion: + ``` + # Find last valid command position + redis-check-aof /var/lib/redis/appendonly.aof + # Truncate at valid position + truncate -s /var/lib/redis/appendonly.aof + ``` + - Option 3: Start without AOF (data loss): + ``` + mv /var/lib/redis/appendonly.aof /var/lib/redis/appendonly.aof.corrupt + redis-server --appendonly no + ``` + + AOF VALIDATION: + - Test AOF in safe environment: + ``` + redis-server --appendonly yes --appendfilename test.aof --dir /tmp/ + ``` + - Verify data after recovery: + `redis-cli DBSIZE` + + PREVENTION: + - Enable AOF with RDB for redundancy: + ``` + appendonly yes + save 900 1 + save 300 10 + ``` + - Configure AOF fsync policy: + `appendfsync everysec` + - Regular AOF rewrites: + `auto-aof-rewrite-percentage 100` + - Monitor disk health regularly + - Implement proper shutdown procedures + - Use UPS to prevent power-related corruption + - Regular backups and recovery testing + references: + - https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/#append-only-file + - https://redis.io/docs/latest/operate/oss_and_stack/management/debugging/#redis-check-aof + - https://redis.io/topics/problems#aof-corruption + applications: + - name: redis + version: ">=1.0.0" + impactScore: 10 + mitigationScore: 6 + reports: 45 + rule: + set: + window: 300s + event: + source: cre.log.redis + match: + - regex: "Bad file format reading the append only file" + - regex: "AOF.*corrupted|corrupt.*AOF" + - regex: "Short read or OOM loading DB" + - regex: "Error loading the AOF" + - regex: "Invalid AOF.*signature" + - regex: "Unexpected end of file reading AOF" + - regex: "AOF.*truncated|truncate.*AOF" + - regex: "Wrong signature trying to load AOF" \ No newline at end of file diff --git a/rules/cre-2025-0180/test.log b/rules/cre-2025-0180/test.log new file mode 100644 index 0000000..10be033 --- /dev/null +++ b/rules/cre-2025-0180/test.log @@ -0,0 +1,8 @@ +2024-01-15 18:00:01.123 [CRITICAL] Bad file format reading the append only file: unexpected end of file +2024-01-15 18:00:02.234 [ERROR] AOF file corrupted at position 1048576 +2024-01-15 18:00:03.345 [ERROR] Short read or OOM loading DB. Unrecoverable error, aborting now. +2024-01-15 18:00:04.456 [ERROR] Error loading the AOF: Invalid argument +2024-01-15 18:00:05.567 [ERROR] Invalid AOF file signature detected +2024-01-15 18:00:06.678 [ERROR] Unexpected end of file reading AOF preamble +2024-01-15 18:00:07.789 [WARN] AOF file may be truncated, attempting recovery +2024-01-15 18:00:08.890 [ERROR] Wrong signature trying to load AOF file from disk \ No newline at end of file diff --git a/rules/cre-2025-0181/redis-max-clients.yaml b/rules/cre-2025-0181/redis-max-clients.yaml new file mode 100644 index 0000000..28828f6 --- /dev/null +++ b/rules/cre-2025-0181/redis-max-clients.yaml @@ -0,0 +1,88 @@ +rules: +- cre: + id: CRE-2025-0181 + severity: 0 + title: "Redis Maximum Client Connections Limit Exceeded" + category: "in-memory-database-problem" + author: Prequel Community + description: | + Detects when Redis reaches its maximum client connection limit, preventing new clients from connecting. This critical issue causes connection failures and service unavailability for new requests. + cause: | + - Connection pool leak in application + - Clients not properly closing connections + - Sudden traffic spike exceeding capacity + - maxclients setting too low for workload + - Slow clients holding connections + - Network issues preventing connection cleanup + - Application bugs creating excessive connections + impact: | + - New connection attempts fail immediately + - Application features become unavailable + - Service outages for new users + - Existing connections may timeout + - Cascading failures in microservices + - Complete Redis unavailability for new clients + impactScore: 10 + tags: + - redis + - connection + - maxclients + - connection-pool + - limit + mitigation: | + IMMEDIATE ACTIONS: + - Check current connections: `redis-cli CLIENT LIST | wc -l` + - Review max clients limit: `redis-cli CONFIG GET maxclients` + - Identify connection sources: `redis-cli CLIENT LIST | awk '{print $2}' | cut -d= -f2 | sort | uniq -c` + - Monitor connection metrics: `redis-cli INFO clients` + + RECOVERY: + - Increase max clients limit: + `redis-cli CONFIG SET maxclients 50000` + - Kill idle connections: + ``` + redis-cli CLIENT LIST | grep idle | awk '{print $2}' | cut -d= -f2 | xargs -I{} redis-cli CLIENT KILL ID {} + ``` + - Kill old connections (>300 seconds): + `redis-cli CLIENT KILL TYPE normal SKIPME yes` + - Restart specific client applications + + CONNECTION ANALYSIS: + - Find connections per IP: + `redis-cli CLIENT LIST | grep addr= | sed 's/.*addr=//' | cut -d: -f1 | sort | uniq -c | sort -rn` + - Identify slow clients: + `redis-cli CLIENT LIST | grep -E "idle=[0-9]{4,}"` + + PREVENTION: + - Implement connection pooling with limits + - Set appropriate connection timeouts + - Monitor connection metrics continuously + - Use connection pool validation + - Regular connection pool recycling + - Implement circuit breakers + - Load testing to determine optimal maxclients + mitigationScore: 7 + references: + - https://redis.io/commands/client-list/ + - https://redis.io/commands/client-kill/ + - https://redis.io/docs/latest/operate/oss_and_stack/reference/clients/ + applications: + - name: redis + version: "*" + reports: 178 + metadata: + kind: prequel + id: Np5KmXw8TfRbQs3YdLaVe7 + gen: 1 + rule: + set: + window: 120s + event: + source: cre.log.redis + match: + - regex: "max number of clients reached" + - regex: "ERR max number of clients reached" + - regex: "connection pool.*exhausted" + - regex: "Unable to connect.*max clients" + - regex: "Redis.*maximum.*connections.*reached" + - regex: "Client connection limit exceeded" \ No newline at end of file diff --git a/rules/cre-2025-0181/test.log b/rules/cre-2025-0181/test.log new file mode 100644 index 0000000..6197f4b --- /dev/null +++ b/rules/cre-2025-0181/test.log @@ -0,0 +1,8 @@ +[2024-01-15 17:00:01,123] ERROR [RedisClient] Max clients limit exceeded +ERR max number of clients reached +max number of clients reached (10000) +connection pool exhausted, cannot create new connection +Unable to connect to Redis: max clients limit reached +Redis server maximum connections reached, rejecting new clients +Client connection limit exceeded for Redis instance +Failed to acquire connection from pool: max number of clients reached \ No newline at end of file diff --git a/rules/cre-2025-0200/autogpt-recursive-self-analysis-loop.yaml b/rules/cre-2025-0200/autogpt-recursive-self-analysis-loop.yaml deleted file mode 100644 index 7f32904..0000000 --- a/rules/cre-2025-0200/autogpt-recursive-self-analysis-loop.yaml +++ /dev/null @@ -1,68 +0,0 @@ -rules: -- cre: - id: CRE-2025-0200 - severity: 0 - title: AutoGPT Recursive Self-Analysis Loop Leading to Token Exhaustion and System Crash - category: infinite-loop-problem - author: prequel - description: | - - AutoGPT enters an infinite recursive loop when attempting to analyze and fix its own execution errors - - The agent repeatedly tries to debug its own code, spawning new analysis tasks for each failure - - Each iteration consumes API tokens and memory, eventually exhausting resources - - The loop accelerates as error messages grow longer, consuming tokens exponentially - - System becomes unresponsive and crashes with out-of-memory errors or API rate limit failures - cause: | - - AutoGPT's autonomous reasoning incorrectly identifies its own execution as a problem to solve - - Lack of loop detection mechanisms allows unlimited recursive task spawning - - Error context accumulation causes exponential growth in prompt size - - Missing safeguards for self-referential task creation - - Insufficient resource monitoring and circuit breakers for runaway processes - tags: - - autogpt - - infinite-loop - - token-exhaustion - - autonomous-agents - - llm - - openai - - recursive-analysis - - critical-failure - - memory-exhaustion - - crash-loop - - rate-limiting - mitigation: | - - Implement loop detection to identify and break recursive self-analysis patterns - - Add resource consumption thresholds (tokens, memory, API calls) with automatic shutdown - - Create task depth limits to prevent unlimited recursion - - Implement circuit breakers that trigger after repeated similar failures - - Add explicit blacklist for self-referential task creation - - Monitor token usage rate and implement exponential backoff - - Use separate monitoring process to detect and kill runaway AutoGPT instances - - Implement task deduplication to prevent identical recursive operations - references: - - https://github.com/Significant-Gravitas/AutoGPT/issues/1994 - - https://github.com/Significant-Gravitas/AutoGPT/issues/3766 - - https://github.com/Significant-Gravitas/AutoGPT/issues/1543 - - https://jina.ai/news/auto-gpt-unmasked-hype-hard-truths-production-pitfalls/ - applications: - - name: autogpt - version: ">=0.3.0" - - name: openai - version: ">=0.27.0" - impact: Complete system failure with resource exhaustion, potential financial losses from API overconsumption - impactScore: 9 - mitigationScore: 3 - reports: 15 - metadata: - kind: prequel - id: 8qy5Et9NbNGgGxhBP7umKa - gen: 1 - rule: - set: - window: 30s - event: - source: cre.log.autogpt - match: - - value: 'Entering recursive analysis loop' - - value: 'COMMAND = analyze_code' - - value: 'recursion depth' - - value: 'RecursionError: maximum recursion depth exceeded' \ No newline at end of file diff --git a/rules/cre-2025-0200/redis-comprehensive-troubleshooting.yaml b/rules/cre-2025-0200/redis-comprehensive-troubleshooting.yaml new file mode 100644 index 0000000..f771fc8 --- /dev/null +++ b/rules/cre-2025-0200/redis-comprehensive-troubleshooting.yaml @@ -0,0 +1,149 @@ +rules: +- cre: + id: CRE-2025-0200 + severity: 0 + title: Redis Comprehensive Troubleshooting - Multiple Common Issues Detection + category: in-memory-database-problem + author: Prequel + description: | + Comprehensive detection rule for multiple common Redis troubleshooting scenarios including: + 1. Out-of-Memory (OOM) errors when maxmemory limit exceeded + 2. Connection timeouts and connectivity issues + 3. Authentication failures and permission denials + 4. Invalid commands and argument errors + 5. Background save (BGSAVE) conflicts and persistence issues + 6. Slow query performance problems + 7. Read-only replica write attempts + 8. Disk persistence failures (MISCONF errors) + 9. Client connection limits exceeded + 10. Memory pressure and eviction warnings + cause: | + MULTIPLE ROOT CAUSES: + - Redis maxmemory limit reached with noeviction policy + - Network connectivity issues or server overload + - Authentication misconfiguration or credential issues + - Client code bugs with invalid Redis commands + - Overlapping backup operations or slow persistence + - Inefficient queries and data access patterns + - Write operations attempted on read-only replicas + - Disk space issues preventing RDB/AOF persistence + - Client connection pool exhaustion + - Memory pressure causing performance degradation + impact: | + BUSINESS IMPACT: + - CRITICAL: Complete inability to access cached data + - HIGH: Application performance degradation and timeouts + - MEDIUM: Backup and persistence operation failures + - LOW: Individual command failures with proper error handling + - Data consistency risks if error handling is inadequate + - Potential cascade failures in dependent services + impactScore: 10 + tags: + - redis + - redis-cli + - redis-py + - out-of-memory + - connection + - authentication + - persistence + - performance + - readonly + - misconf + - acl + - public + mitigation: | + IMMEDIATE ACTIONS: + - Check Redis server status: `redis-cli ping` + - Monitor memory usage: `redis-cli info memory` + - Review error logs for specific failure patterns + - Verify authentication and ACL configuration + - Check disk space and persistence settings + + RECOVERY STRATEGIES BY ISSUE TYPE: + + 1. OOM ERRORS: + - Increase maxmemory limit: `CONFIG SET maxmemory 500mb` + - Change eviction policy: `CONFIG SET maxmemory-policy volatile-lru` + - Clear unnecessary keys or restart Redis + + 2. CONNECTION ISSUES: + - Restart Redis service: `systemctl restart redis` + - Check firewall and network configuration + - Adjust client timeout settings + + 3. AUTHENTICATION FAILURES: + - Verify credentials: `redis-cli -a password ping` + - Update ACL permissions: `ACL SETUSER username +@all` + - Rotate and update client credentials + + 4. COMMAND ERRORS: + - Fix application code with correct Redis syntax + - Update Redis client libraries + - Check for renamed/disabled commands + + 5. PERSISTENCE ISSUES: + - Wait for current BGSAVE to complete + - Free disk space for RDB/AOF files + - Optimize backup scheduling + + 6. SLOW QUERIES: + - Optimize data structures and access patterns + - Use SCAN instead of KEYS for iteration + - Monitor and tune slowlog settings + + 7. READONLY ERRORS: + - Redirect writes to master Redis instance + - Check replication configuration + - Verify client connection routing + + PREVENTION: + - Implement comprehensive Redis monitoring + - Set up memory, performance, and error alerting + - Use Redis clustering for high availability + - Regular capacity planning and performance reviews + - Automate backup and persistence monitoring + - Implement proper error handling in applications + mitigationScore: 7 + references: + - https://redis.io/docs/latest/operate/oss_and_stack/management/troubleshooting/ + - https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/ + - https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/ + - https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/latency/ + - https://www.site24x7.com/learn/redis-troubleshooting-guide.html + applications: + - name: redis + version: "*" + - name: redis-cli + version: "*" + - name: redis-py + version: "*" + metadata: + kind: prequel + id: HgYt8N4XKfLMPqRu9wSvZ7 + gen: 1 + rule: + set: + window: 300s + event: + source: cre.log.redis + match: + # Issue 1: Out-of-Memory (OOM) errors + - regex: "OOM command not allowed when used memory > 'maxmemory'" + # Issue 2: Connection timeout errors + - regex: "Connection timeout.*redis|Unable to connect to Redis|Could not connect to Redis|redis.*connection.*timeout" + # Issue 3: Authentication failures + - regex: "WRONGPASS invalid username-password pair|NOAUTH Authentication required|ERR invalid password" + # Issue 4: Invalid commands and arguments + - regex: "ERR unknown command|ERR wrong number of arguments|WRONGTYPE|ERR syntax error" + # Issue 5: Background save conflicts + - regex: "Background save already in progress|ERR Background save already in progress" + # Issue 6: Slow query performance + - regex: "Slow log.*microseconds|command.*took.*milliseconds" + # Issue 7: Read-only replica writes + - regex: "READONLY You can['']t write against a read only replica" + # Issue 8: Persistence failures (MISCONF) + - regex: "MISCONF Redis is configured to save RDB snapshots, but it's currently unable to persist to disk" + # Issue 9: Connection limits exceeded + - regex: "max number of clients reached|connection pool.*exhausted" + # Issue 10: ACL permission denied + - regex: "NOPERM User .* has no permissions to run the '.*' command" \ No newline at end of file diff --git a/rules/cre-2025-0200/test.log b/rules/cre-2025-0200/test.log index c871b99..496f555 100644 --- a/rules/cre-2025-0200/test.log +++ b/rules/cre-2025-0200/test.log @@ -1,35 +1,41 @@ -2025-08-31 14:23:45.234 [INFO] [autogpt.main] Starting AutoGPT v0.5.1 with goal: "Optimize my Python code for better performance" -2025-08-31 14:23:45.567 [INFO] [autogpt.llm] Initializing OpenAI API client with model gpt-4 -2025-08-31 14:23:46.102 [INFO] [autogpt.agent] Agent initialized with memory backend: LocalCache -2025-08-31 14:23:47.234 [INFO] [autogpt.agent] COMMAND = analyze_code args: {"code": "def slow_function():\\n result = []\\n for i in range(1000000):\\n result.append(i**2)\\n return result"} -2025-08-31 14:23:48.567 [ERROR] [autogpt.commands] Error executing analyze_code: JSONDecodeError in response -2025-08-31 14:23:48.890 [WARN] [autogpt.agent] Entering recursive analysis loop to debug previous error -2025-08-31 14:23:49.234 [INFO] [autogpt.agent] THOUGHTS: Previous command failed, need to analyze what went wrong -2025-08-31 14:23:49.567 [INFO] [autogpt.agent] NEXT ACTION: COMMAND = analyze_code args: {"code": "analyze_code function from autogpt/commands/analyze_code.py", "recursion depth": 1} -2025-08-31 14:23:50.123 [DEBUG] [autogpt.memory] Storing error context, current size: 2.3MB -2025-08-31 14:23:50.890 [ERROR] [autogpt.commands] Error executing analyze_code: Cannot analyze own execution context -2025-08-31 14:23:51.234 [WARN] [autogpt.agent] Thinking... need to fix my own error handling -2025-08-31 14:23:51.678 [INFO] [autogpt.agent] COMMAND = analyze_code args: {"code": "autogpt error handling module", "recursion depth": 2} -2025-08-31 14:23:52.345 [DEBUG] [autogpt.memory] Memory usage increasing: 5.7MB, token count: 8234 -2025-08-31 14:23:52.890 [ERROR] [autogpt.llm] API request failed: context length exceeded -2025-08-31 14:23:53.234 [INFO] [autogpt.agent] SYSTEM: Potential loop detected but continuing to resolve errors -2025-08-31 14:23:53.567 [INFO] [autogpt.agent] THOUGHTS: Error analyzing previous attempt, need deeper analysis -2025-08-31 14:23:54.012 [INFO] [autogpt.agent] NEXT ACTION: COMMAND = analyze_code args: {"code": "full autogpt execution trace", "recursion depth": 3} -2025-08-31 14:23:54.678 [WARN] [autogpt.monitor] Task queue growing: 12 pending tasks -2025-08-31 14:23:55.234 [DEBUG] [autogpt.memory] Memory usage: 12.4MB, token count: 15672 -2025-08-31 14:23:55.890 [ERROR] [autogpt.commands] RecursionError: maximum recursion depth exceeded while calling analyze_code -2025-08-31 14:23:56.345 [CRITICAL] [autogpt.agent] Task queue overflow: 47 pending recursive tasks -2025-08-31 14:23:56.789 [INFO] [autogpt.agent] COMMAND = analyze_code args: {"code": "recursion error in analyze_code", "recursion depth": 4} -2025-08-31 14:23:57.234 [ERROR] [autogpt.llm] openai.error.RateLimitError: Rate limit reached for gpt-4 in organization -2025-08-31 14:23:57.567 [WARN] [autogpt.monitor] Token consumption rate: 2341 tokens/second -2025-08-31 14:23:58.012 [DEBUG] [autogpt.memory] Memory usage critical: 45.8MB, token count: 42318 -2025-08-31 14:23:58.456 [ERROR] [autogpt.agent] Too many pending tasks: 89 in queue -2025-08-31 14:23:58.890 [INFO] [autogpt.agent] THOUGHTS: Still analyzing previous errors, must understand the recursion -2025-08-31 14:23:59.234 [INFO] [autogpt.agent] NEXT ACTION: COMMAND = analyze_code args: {"code": "entire autogpt error stack", "recursion depth": 5} -2025-08-31 14:23:59.678 [CRITICAL] [autogpt.monitor] JavaScript heap out of memory -2025-08-31 14:24:00.123 [ERROR] [autogpt.memory] MemoryError: Cannot allocate memory for context storage -2025-08-31 14:24:00.456 [CRITICAL] [autogpt.agent] Task buffer exceeded: 156 recursive analyze_code calls pending -2025-08-31 14:24:00.789 [ERROR] [autogpt.llm] API rate limit exceeded: 429 Too Many Requests -2025-08-31 14:24:01.123 [FATAL] [autogpt.main] AutoGPT crashed: Unrecoverable recursive loop detected -2025-08-31 14:24:01.234 [INFO] [autogpt.cleanup] Emergency shutdown initiated -2025-08-31 14:24:01.345 [ERROR] [autogpt.cleanup] Failed to save state: Out of memory \ No newline at end of file +[2024-08-27 05:50:15,123] INFO [RedisTest] Issue 1: Out-of-Memory (OOM) errors +OOM command not allowed when used memory > 'maxmemory' + +[2024-08-27 05:50:16,456] ERROR [RedisClient] Issue 2: Connection timeout errors +Could not connect to Redis at 127.0.0.1:6379: Connection timeout +Unable to connect to Redis server at localhost:6379 +Connection timeout connecting to redis server +redis connection timeout, falling back to database + +[2024-08-27 05:50:17,789] ERROR [RedisAuth] Issue 3: Authentication failures +WRONGPASS invalid username-password pair +NOAUTH Authentication required +ERR invalid password + +[2024-08-27 05:50:18,012] ERROR [RedisCmd] Issue 4: Invalid commands and arguments +ERR unknown command 'INVALIDCOMMAND', with args beginning with: +ERR wrong number of arguments for 'set' command +WRONGTYPE Operation against a key holding the wrong kind of value +ERR syntax error + +[2024-08-27 05:50:19,345] WARN [RedisPersistence] Issue 5: Background save conflicts +Background save already in progress +ERR Background save already in progress + +[2024-08-27 05:50:20,678] WARN [RedisPerf] Issue 6: Slow query performance +Slow log entry command 'KEYS *' took 15000 microseconds +command 'SORT large_list' took 45 milliseconds + +[2024-08-27 05:50:21,901] ERROR [RedisReplica] Issue 7: Read-only replica writes +READONLY You can't write against a read only replica + +[2024-08-27 05:50:22,234] ERROR [RedisPersistence] Issue 8: Persistence failures (MISCONF) +MISCONF Redis is configured to save RDB snapshots, but it's currently unable to persist to disk + +[2024-08-27 05:50:23,567] WARN [RedisConn] Issue 9: Connection limits exceeded +max number of clients reached +connection pool exhausted + +[2024-08-27 05:50:24,890] ERROR [RedisACL] Issue 10: ACL permission denied +NOPERM User 'limited' has no permissions to run the 'SET' command +NOPERM this user has no permissions to run the 'FLUSHALL' command \ No newline at end of file diff --git a/rules/cre-2025-0201/redis-oom-error.yaml b/rules/cre-2025-0201/redis-oom-error.yaml new file mode 100644 index 0000000..b8b6b40 --- /dev/null +++ b/rules/cre-2025-0201/redis-oom-error.yaml @@ -0,0 +1,71 @@ +rules: +- cre: + id: CRE-2025-0201 + severity: 0 + title: "Redis Out-of-Memory Error - Maxmemory Limit Exceeded" + category: "in-memory-database-problem" + author: Prequel Community + description: | + Detects Redis out-of-memory errors when the maxmemory limit is reached and the configured eviction policy prevents new writes. This typically occurs when Redis is configured with 'noeviction' policy and memory usage exceeds the maxmemory setting, causing all write commands to fail. + cause: | + - Redis memory usage exceeded maxmemory configuration limit + - Eviction policy set to 'noeviction' preventing automatic key removal + - Application writing data faster than Redis can evict + - Memory leak in Lua scripts or client buffers + - Large keys consuming excessive memory + impact: | + - All write operations (SET, LPUSH, ZADD, etc.) fail immediately + - Application functionality depending on caching breaks + - Potential cascading failures in dependent services + - Read operations continue to work normally + - Database or backend services may experience increased load + impactScore: 10 + tags: + - redis + - out-of-memory + - maxmemory + - noeviction + mitigation: | + IMMEDIATE ACTIONS: + - Check current memory usage: `redis-cli INFO memory | grep used_memory_human` + - Review maxmemory setting: `redis-cli CONFIG GET maxmemory` + - Check eviction policy: `redis-cli CONFIG GET maxmemory-policy` + + RECOVERY: + - Option 1: Increase maxmemory limit + `redis-cli CONFIG SET maxmemory 2gb` + - Option 2: Change eviction policy + `redis-cli CONFIG SET maxmemory-policy allkeys-lru` + - Option 3: Manually delete unnecessary keys + `redis-cli FLUSHDB` (WARNING: deletes all keys) + - Option 4: Identify and remove large keys + `redis-cli --bigkeys` + + PREVENTION: + - Set appropriate maxmemory limit based on available RAM + - Use eviction policies like allkeys-lru or volatile-lru + - Monitor memory usage with Redis INFO command + - Implement TTL on keys where appropriate + - Regular memory usage auditing and capacity planning + mitigationScore: 8 + references: + - https://redis.io/docs/latest/operate/oss_and_stack/reference/eviction/ + - https://redis.io/commands/config-set/ + - https://redis.io/docs/latest/operate/oss_and_stack/management/troubleshooting/#out-of-memory + applications: + - name: redis + version: ">=2.0.0" + reports: 47 + metadata: + kind: prequel + id: XpQ9mKvR3bNfLw8TjHaYz5 + gen: 1 + rule: + set: + window: 120s + event: + source: cre.log.redis + match: + - regex: "OOM command not allowed when used memory > 'maxmemory'" + - regex: "used_memory.*maxmemory.*OOM" + - regex: "-OOM.*command not allowed" \ No newline at end of file diff --git a/rules/cre-2025-0201/test.log b/rules/cre-2025-0201/test.log new file mode 100644 index 0000000..de35b36 --- /dev/null +++ b/rules/cre-2025-0201/test.log @@ -0,0 +1,8 @@ +2024-01-15 10:23:45.123 [ERROR] redis.clients.jedis.exceptions.JedisDataException: OOM command not allowed when used memory > 'maxmemory'. +2024-01-15 10:23:45.234 [ERROR] Redis server returned error: -OOM command not allowed when used memory > 'maxmemory'. +2024-01-15 10:23:45.345 [WARN] RedisCommandExecutionException: ERR OOM command not allowed when used memory > 'maxmemory'. +2024-01-15 10:23:45.456 [ERROR] Failed to execute SET command: OOM command not allowed when used memory > 'maxmemory'. +2024-01-15 10:23:45.567 [ERROR] Cache write failed: redis.exceptions.ResponseError: OOM command not allowed when used memory > 'maxmemory'. +2024-01-15 10:23:45.678 [ERROR] Application Error: Cannot write to cache - OOM command not allowed when used memory > 'maxmemory' +2024-01-15 10:23:45.789 [CRITICAL] Redis memory full: used_memory:2147483648 maxmemory:2147483648 OOM errors occurring +2024-01-15 10:23:45.890 [ERROR] LPUSH mylist value1 failed: -OOM command not allowed when used memory exceeded maxmemory limit \ No newline at end of file diff --git a/rules/tags/tags.yaml b/rules/tags/tags.yaml index 63b40ef..ce00e20 100644 --- a/rules/tags/tags.yaml +++ b/rules/tags/tags.yaml @@ -848,6 +848,57 @@ tags: - name: cluster-scaling displayName: Cluster Scaling description: Problems related to Kubernetes cluster scaling operations and capacity management + - name: maxmemory + displayName: Max Memory + description: Problems related to Redis maxmemory configuration and memory limits + - name: noeviction + displayName: No Eviction + description: Issues when Redis noeviction policy prevents writing new data + - name: wrongpass + displayName: Wrong Password + description: Authentication failures due to incorrect Redis passwords + - name: master-replica + displayName: Master-Replica + description: Issues with Redis master-replica replication relationships + - name: sync + displayName: Sync + description: Data synchronization problems in distributed systems + - name: psync + displayName: Partial Sync + description: Redis partial resynchronization issues + - name: aof + displayName: AOF + description: Redis Append-Only File persistence issues + - name: slowlog + displayName: Slow Log + description: Database slow query logging and performance issues + - name: latency + displayName: Latency + description: Response time and performance latency issues + - name: slow-query + displayName: Slow Query + description: Database queries that exceed performance thresholds + - name: write-error + displayName: Write Error + description: Failures when attempting write operations + - name: recovery + displayName: Recovery + description: Data recovery and restoration operations + - name: maxclients + displayName: Max Clients + description: Connection limit issues in database systems + - name: connection-pool + displayName: Connection Pool + description: Problems with database connection pooling + - name: limit + displayName: Limit + description: Various system and resource limits being exceeded + - name: disk + displayName: Disk + description: Problems related to disk storage, space, or I/O operations + - name: replica + displayName: Replica + description: Issues related to database replicas and read-only instances - name: supabase displayName: Supabase description: Problems related to Supabase self-hosted deployments and services