BB-727: Fix ZookeeperConfigManager event listener leaks #2687

BourgoisMickael · 2025-11-14T17:58:44Z

improve some logs
warn early for event listener leaks
avoid send a remove to zk when a watcher notify the removed bucket config
separate list and get configs

Below a sequence diagram showing in red a few problems fixed here

bert-e · 2025-11-14T17:58:49Z

Hello bourgoismickael,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

codecov · 2025-11-14T18:09:33Z

Codecov Report

❌ Patch coverage is 82.22222% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.05%. Comparing base (02e18ef) to head (f4614f2).

Files with missing lines	Patch %	Lines
...tification/configManager/ZookeeperConfigManager.js	84.61%	4 Missing ⚠️
lib/clients/ZookeeperManager.js	78.94%	4 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
...ensions/notification/NotificationQueuePopulator.js	`98.18% <ø> (ø)`
lib/BackbeatConsumer.js	`92.37% <ø> (ø)`
...tification/configManager/ZookeeperConfigManager.js	`82.74% <84.61%> (-0.77%)`	⬇️
lib/clients/ZookeeperManager.js	`93.87% <78.94%> (-3.60%)`	⬇️

Components	Coverage Δ
Bucket Notification	`80.25% <84.61%> (-0.16%)`	⬇️
Core Library	`80.30% <78.94%> (-0.01%)`	⬇️
Ingestion	`70.28% <ø> (ø)`
Lifecycle	`78.67% <ø> (ø)`
Oplog Populator	`85.06% <ø> (ø)`
Replication	`58.62% <ø> (ø)`
Bucket Scanner	`85.76% <ø> (ø)`

@@               Coverage Diff                @@
##           development/9.0    #2687   +/-   ##
================================================
  Coverage            74.05%   74.05%           
================================================
  Files                  201      201           
  Lines                13449    13477   +28     
================================================
+ Hits                  9959     9980   +21     
- Misses                3480     3487    +7     
  Partials                10       10

Flag	Coverage Δ
api:retry	`9.43% <0.00%> (-0.03%)`	⬇️
api:routes	`9.24% <0.00%> (-0.02%)`	⬇️
bucket-scanner	`85.76% <ø> (ø)`
ft_test:queuepopulator	`8.90% <0.00%> (-0.02%)`	⬇️
ingestion	`12.50% <0.00%> (-0.03%)`	⬇️
lib	`7.45% <0.00%> (-0.02%)`	⬇️
lifecycle	`18.65% <0.00%> (-0.04%)`	⬇️
notification	`1.04% <0.00%> (-0.01%)`	⬇️
replication	`18.44% <0.00%> (-0.04%)`	⬇️
unit	`50.10% <82.22%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull Request Overview

This PR addresses event listener leaks in the ZookeeperConfigManager by improving watcher management and separating list/get config operations. The key improvements include early detection of watcher leaks, preventing redundant ZooKeeper remove operations when watchers notify of deleted bucket configs, and enhanced logging for better debugging.

Added early warning system for ZooKeeper watcher leaks to detect issues before hitting EventEmitter limits
Separated getConfig and listConfigs event handlers to avoid unnecessary bucket list operations
Modified removeConfig to optionally skip ZooKeeper emission when triggered by a watcher notification

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
lib/BackbeatConsumer.js	Enhanced logging to safely convert Buffer keys to strings for better debugging
extensions/notification/configManager/ZookeeperConfigManager.js	Core fixes for watcher leak detection, separated list/get config handlers, and added conditional ZK emission for removeConfig
extensions/notification/NotificationQueuePopulator.js	Corrected typo in comment from "deleter" to "deleted"

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Fix `key: { type: 'Buffer', data: [ 104, 101, 108, 108, 111 ] }` To `key: 'hello'`

The watcher NODE_DELETED already calls removeConfig with false Because it's already removed from zk if the event is received There is no need to send a request to zk

This kind of error can happen if data is undefined when the read is triggered too late when the znode is already removed

Otherwise we need a few buckets and bucket operations to trigger the event emitter listener limit warning at 10 listeners

The getConfig should not trigger a listing with watcher as it duplicates an already existing watcher. This can have an exponential effect of adding even more watchers when events are triggered. A clear separation prevent this effect: - only NODE_CHILDREN_CHANGED should trigger a new listing to reapply a watcher - getConfig should be contained to a single bucket (discovered by listing, or from watcher NODE_DATA_CHANGED)

bert-e · 2026-01-05T10:44:14Z

Incorrect fix version

The Fix Version/s in issue BB-727 contains:

9.0.21
9.1.5

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.0.22
9.1.5

Please check the Fix Version/s of BB-727, or the target
branch of this pull request.

bert-e · 2026-01-05T10:52:57Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

BourgoisMickael · 2026-01-05T10:59:58Z

/create_integration_branches

bert-e · 2026-01-05T11:00:09Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/BB-727-zk-config into
development/9.0
w/9.1/improvement/BB-727-zk-config will be merged into development/9.1

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.6

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2026-01-05T11:00:10Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

bert-e · 2026-01-05T14:27:49Z

History mismatch

Merge commit #7edd3977408e2654642035c90a030202a5b764bc on the integration branch
w/9.1/improvement/BB-727-zk-config is merging a branch which is neither the current
branch improvement/BB-727-zk-config nor the development branch
development/9.1.

It is likely due to a rebase of the branch improvement/BB-727-zk-config and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

The following options are set: create_integration_branches

Atomic mkdirp to prevent trigger a watcher for znode creation before data is set on the znode

BourgoisMickael · 2026-01-05T17:45:34Z

@bert-e reset

bert-e · 2026-01-05T17:45:43Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

bert-e · 2026-01-05T17:45:54Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/BB-727-zk-config into
development/9.0
w/9.1/improvement/BB-727-zk-config will be merged into development/9.1

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.6

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2026-01-05T17:45:55Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

scality deleted a comment from bert-e Nov 14, 2025

BourgoisMickael requested a review from Copilot November 14, 2025 18:10

Copilot started reviewing on behalf of BourgoisMickael November 14, 2025 18:11 View session

Copilot finished reviewing on behalf of BourgoisMickael November 14, 2025 18:12

Copilot AI reviewed Nov 14, 2025

View reviewed changes

BourgoisMickael force-pushed the improvement/BB-727-zk-config branch from 62f6510 to dc1fe12 Compare November 14, 2025 18:14

nicolas2bert approved these changes Nov 18, 2025

View reviewed changes

BourgoisMickael added 8 commits January 5, 2026 11:43

BB-727: Fix comment typo

0e418b4

BB-727: Better print buffer in logs

e665d0f

Fix `key: { type: 'Buffer', data: [ 104, 101, 108, 108, 111 ] }` To `key: 'hello'`

BB-727: removeConfig only locally on NODE_DELETED

c8cd7e6

The watcher NODE_DELETED already calls removeConfig with false Because it's already removed from zk if the event is received There is no need to send a request to zk

BB-727: Log bucket for invalid config error

5361492

This kind of error can happen if data is undefined when the read is triggered too late when the znode is already removed

BB-727: Check and warn event emitter leak early

cea258a

Otherwise we need a few buckets and bucket operations to trigger the event emitter listener limit warning at 10 listeners

BB-727: Log as debug the list of buckets

c663635

BB-727: Watch created config in queuePopulator

66d8b1a

BourgoisMickael force-pushed the improvement/BB-727-zk-config branch from 694086a to 66d8b1a Compare January 5, 2026 10:44

scality deleted a comment from bert-e Jan 5, 2026

BourgoisMickael force-pushed the improvement/BB-727-zk-config branch from 7edd397 to 80072f1 Compare January 5, 2026 14:27

BB-727: Improve mkdirp to create znode with data

3a1a21c

Atomic mkdirp to prevent trigger a watcher for znode creation before data is set on the znode

BourgoisMickael force-pushed the improvement/BB-727-zk-config branch from 80072f1 to 3dc379d Compare January 5, 2026 14:55

BourgoisMickael added 2 commits January 5, 2026 17:17

BB-727: Update tests to use ZookeeperManager

bdfdaff

BB-727: Unit test mkdirpWithChildDataOnly

f4614f2

BourgoisMickael force-pushed the improvement/BB-727-zk-config branch from 3dc379d to f4614f2 Compare January 5, 2026 16:17

BB-727: Fix ZookeeperConfigManager event listener leaks #2687

Are you sure you want to change the base?

BB-727: Fix ZookeeperConfigManager event listener leaks #2687

Uh oh!

Conversation

BourgoisMickael commented Nov 14, 2025

Uh oh!

bert-e commented Nov 14, 2025

Hello bourgoismickael,

Uh oh!

codecov bot commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

bert-e commented Jan 5, 2026

Incorrect fix version

Uh oh!

bert-e commented Jan 5, 2026

Request integration branches

Uh oh!

BourgoisMickael commented Jan 5, 2026

Uh oh!

bert-e commented Jan 5, 2026

Integration data created

Uh oh!

bert-e commented Jan 5, 2026

Waiting for approval

Uh oh!

bert-e commented Jan 5, 2026

History mismatch

Uh oh!

BourgoisMickael commented Jan 5, 2026

Uh oh!

bert-e commented Jan 5, 2026

Reset complete

Uh oh!

bert-e commented Jan 5, 2026

Integration data created

Uh oh!

bert-e commented Jan 5, 2026

Waiting for approval

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Nov 14, 2025 •

edited

Loading