Skip to content

[Bug]: fault-remediation module is stuck with error :"Failed to watch change stream" after v0.7.1 upgrade #955

@deesharma24

Description

@deesharma24

Prerequisites

  • I searched existing issues
  • I can reproduce this issue

Bug Description

when fault-remediation comes up, it is not able to watch the mongodb changestream and is throwing the below error

{"time":"2026-02-26T11:55:55.346439016Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}

it needs to be checked how best to remediate it.

Component

Health Monitor

Steps to Reproduce

After upgrade to v0.7.1, this issue is coming. This is seen only once and there is no mechanism to repro this yet.

Environment

  • NVSentinel version: V0.7.1
  • Kubernetes version:v1.29.15
  • Deployment method: helm chart

Logs/Output

{"time":"2026-02-26T11:55:55.346433518Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346435161Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346437384Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346439016Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346440729Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346442412Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346444104Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346445787Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346447439Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}

Metadata

Metadata

Labels

bugSomething isn't working

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions