Skip to content

Commit 89f303a

Browse files
authored
MongoDB Crash Loop(CRE-2025-0120) (#109)
* mongodb crash loop * fix * fix * fix tags * fix tags * fix: reference link * fix: conflict * fix: test * fix: duplicate
1 parent 5ebcf6c commit 89f303a

File tree

4 files changed

+75
-5
lines changed

4 files changed

+75
-5
lines changed
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
rules:
2+
- metadata:
3+
kind: prequel
4+
id: DQbJmYTnPqvL6HdX2Xs7Wh
5+
cre:
6+
id: CRE-2025-0093
7+
severity: 0
8+
title: MongoDB Crash Loop Due to WiredTiger Metadata Corruption
9+
category: mongodb-startup-failure
10+
author: Community
11+
description: |
12+
Detects MongoDB failures where the server enters a crash loop due to corrupted WiredTiger metadata.
13+
This prevents startup and often requires manual repair or restoration from backups.
14+
cause: |
15+
WiredTiger metadata corruption caused by unclean shutdowns, disk failures, or filesystem issues.
16+
MongoDB is unable to initialize its storage engine due to invalid or unreadable metadata.
17+
impact: |
18+
Database remains unavailable due to repeated startup failures.
19+
Services relying on MongoDB experience outages and potential data loss.
20+
Errors referencing WiredTiger metadata corruption are observed in the logs.
21+
impactScore: 10
22+
tags:
23+
- mongodb
24+
- wiredtiger
25+
- metadata
26+
- corruption
27+
- crash-loop
28+
- startup-failure
29+
mitigation: |
30+
Prevent unclean shutdowns, monitor disk and filesystem health, run regular backups.
31+
If corruption occurs, attempt recovery with mongod --repair, or restore from a known good backup.
32+
mitigationScore: 9
33+
references:
34+
- https://www.mongodb.com/docs/manual/administration/repair/
35+
- https://jira.mongodb.org/browse/SERVER-37978
36+
- https://www.mongodb.com/community/forums/t/failed-to-start-wiredtiger-after-system-upgrade/13088
37+
applications:
38+
- name: mongod
39+
version: ">=4.2"
40+
41+
rule:
42+
set:
43+
event:
44+
source: cre.log.mongodb
45+
match:
46+
- regex: '(?i)WiredTiger.*metadata.*corrupt'
47+
count: 1

rules/cre-2025-0129/test.log

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
2025-07-06 18:34:21.262 [INF] Starting mongod
2+
2025-07-06 18:34:21.264 [INF] db version v4.4.26
3+
2025-07-06 18:34:21.264 [INF] git version: 971c204888dd1b6d1072442a3f958e3e52a5d2d0
4+
2025-07-06 18:34:21.265 [INF] build environment: Ubuntu 20.04
5+
2025-07-06 18:34:21.265 [INF] Using storage engine: WiredTiger
6+
2025-07-06 18:34:21.265 [INF] WiredTiger message: [1674512923:123456][1:0x7f8b4e7d0700], file:WiredTiger.wt, WT_SESSION.open_cursor: __wt_btree_open, 508: WiredTiger metadata file is corrupt
7+
2025-07-06 18:34:21.266 [ERR] WiredTiger metadata file is corrupt
8+
2025-07-06 18:34:21.266 [ERR] WiredTiger: [WT_SESSION] Failed to open WiredTiger metadata
9+
2025-07-06 18:34:21.266 [ERR] WiredTiger failed to initialize due to metadata corruption
10+
2025-07-06 18:34:21.267 [ERR] MongoDB encountered fatal error during startup
11+
2025-07-06 18:34:21.267 [ERR] MongoDB crash loop due to WiredTiger metadata corruption
12+
2025-07-06 18:34:21.268 [INF] For more information, see: https://www.mongodb.com/docs/manual/administration/repair/
13+
2025-07-06 18:34:21.268 [INF] Attempting automatic shutdown due to unrecoverable error

rules/tags/categories.yaml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -239,4 +239,8 @@ categories:
239239
- name: database-availability
240240
displayName: Database Availability
241241
description: Critical failures affecting database service availability, including crashes, OOM kills, and unplanned terminations that require immediate recovery
242-
242+
- name: mongodb-startup-failure
243+
displayName: MongoDB Startup Failure
244+
description: |
245+
Failures that prevent MongoDB from starting successfully due to corrupted metadata, invalid configurations,
246+
or unrecoverable internal errors (e.g., WiredTiger metadata corruption). These failures often require manual repair or backup restoration.

rules/tags/tags.yaml

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -645,9 +645,6 @@ tags:
645645
- name: assertion
646646
displayName: Assertion
647647
description: Assertion failures indicating internal consistency checks or invariants violated at runtime.
648-
- name: crash-loop
649-
displayName: Crash Loop
650-
description: Repeated process restarts due to persistent failures causing operational impact.
651648
- name: redpanda-startup
652649
displayName: Redpanda Startup
653650
description: Problems related to Redpanda startup, such as configuration issues or missing state
@@ -814,7 +811,16 @@ tags:
814811
description: Problems related to MongoDB database system
815812
- name: wiredtiger
816813
displayName: WiredTiger
817-
description: Problems related to MongoDB's WiredTiger storage engine
814+
description: Issues related to MongoDB's default storage engine, WiredTiger, including corruption, crash loops, and initialization errors.
815+
- name: metadata
816+
displayName: Metadata
817+
description: Failures caused by corrupted or unreadable metadata files used by the application to track state.
818+
- name: crash-loop
819+
displayName: Crash Loop
820+
description: Repeated failure to start or recover due to a persistent internal problem, such as corrupted state or misconfiguration.
821+
- name: corruption
822+
displayName: Corruption
823+
description: Issues involving data corruption, including damaged files, invalid data structures, or inconsistencies caused by software or hardware faults.
818824
- name: oom-kill
819825
displayName: OOM Kill
820826
description: Problems where processes are terminated by the OS Out-Of-Memory killer

0 commit comments

Comments
 (0)