Skip to content

Commit 6dd45d4

Browse files
authored
Nat storage exhaustion (CRE-2025-0088) (#82)
* nats slow consumer * updates
1 parent e301590 commit 6dd45d4

File tree

3 files changed

+68
-0
lines changed

3 files changed

+68
-0
lines changed
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
rules:
2+
- metadata:
3+
kind: prequel
4+
id: 9KdL2nQpR3sWyZzF4GxHkM
5+
gen: 1
6+
cre:
7+
id: CRE-2025-0088
8+
severity: 1
9+
title: NATS JetStream Storage Exhaustion Detection
10+
category: message-queue-problems
11+
author: Community
12+
description: |
13+
Detects NATS JetStream storage exhaustion conditions when streams reach configured
14+
storage limits (maximum bytes, maximum messages) causing message storage failures.
15+
These patterns indicate insufficient stream storage capacity relative to message
16+
production rate, leading to message rejection and potential data loss.
17+
cause: |
18+
JetStream streams configured with insufficient storage limits (max_bytes, max_msgs)
19+
relative to message production rate and retention requirements. Messages exceed
20+
configured stream storage capacity faster than they can be consumed or expired.
21+
impact: |
22+
Message storage failures, potential data loss, stream unavailability, producer
23+
errors, degraded application performance, and inability to persist critical messages
24+
in JetStream streams.
25+
impactScore: 9
26+
tags:
27+
- nats
28+
- jetstream
29+
- storage-exhaustion
30+
- message-storage-failure
31+
- capacity-exceeded
32+
- data-loss-risk
33+
mitigation: |
34+
Increase JetStream stream storage limits (max_bytes, max_msgs), implement stream
35+
retention policies, scale consumer processing capacity, monitor stream storage
36+
utilization, implement producer flow control, and configure appropriate discard
37+
policies for stream overflow scenarios.
38+
mitigationScore: 8
39+
references:
40+
- https://docs.nats.io/nats-concepts/jetstream/streams
41+
- https://docs.nats.io/nats-concepts/jetstream/administration
42+
- https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin
43+
applications:
44+
- name: nats-server
45+
version: ">=2.2.0"
46+
47+
rule:
48+
set:
49+
event:
50+
source: cre.log.nats.jetstream
51+
match:
52+
- regex: '(?i)jetstream\s+failed\s+to\s+store.*maximum\s+bytes\s+exceeded'
53+
count: 1%

rules/CRE-2025-0088/test.log

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[1] 2025/06/29 11:57:58.502311 [INF] Name: nats-jetstream-storage-test
2+
[1] 2025/06/29 11:58:16.221858 [DBG] JetStream failed to store a msg on stream '$G > STORAGE_TEST': maximum bytes exceeded
3+
[1] 2025/06/29 11:58:17.622012 [DBG] JetStream failed to store a msg on stream '$G > STORAGE_TEST': maximum bytes exceeded

rules/tags/tags.yaml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -702,3 +702,15 @@ tags:
702702
- name: critical-infrastructure
703703
displayName: Critical Infrastructure
704704
description: Issues affecting mission-critical infrastructure components that require immediate attention to prevent widespread outages
705+
- name: storage-exhaustion
706+
displayName: Storage Exhaustion
707+
description: Problems where storage limits are exceeded causing service failures or data rejection
708+
- name: message-storage-failure
709+
displayName: Message Storage Failure
710+
description: Failures to persist messages due to storage constraints or system limitations
711+
- name: capacity-exceeded
712+
displayName: Capacity Exceeded
713+
description: Problems where system capacity limits (storage, memory, connections) are exceeded
714+
- name: data-loss-risk
715+
displayName: Data Loss Risk
716+
description: Conditions that may lead to data loss if not addressed promptly

0 commit comments

Comments
 (0)