-
-
Notifications
You must be signed in to change notification settings - Fork 8.6k
[grid] Add event bus heartbeat to prevent steal connection #16444
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: Viet Nguyen Duc <[email protected]>
PR Compliance Guide 🔍Below is a summary of compliance checks for this PR:
Compliance status legend🟢 - Fully Compliant🟡 - Partial Compliant 🔴 - Not Compliant ⚪ - Requires Further Human Verification 🏷️ - Compliance label |
PR Code Suggestions ✨Latest suggestions up to cc71cd0
Previous suggestions✅ Suggestions up to commit 7a60828
|
Signed-off-by: Viet Nguyen Duc <[email protected]>
Signed-off-by: Viet Nguyen Duc <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds a configurable ZeroMQ heartbeat mechanism to Selenium Grid's EventBus to prevent stale connection issues, particularly after long idle periods (8+ hours) where nodes may fail to register with the distributor.
Key changes:
- Introduces
--eventbus-heartbeat-period
CLI flag with 60-second default - Implements heartbeat configuration for all ZMQ socket types (XPUB, XSUB, PUB, SUB)
- Adds heartbeat utilities with interval clamping and validation
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
File | Description |
---|---|
EventBusOptions.java | Adds heartbeat period configuration option with 60-second default |
EventBusFlags.java | Adds CLI parameter for configuring heartbeat period |
ZmqUtils.java | New utility class for configuring ZMQ socket heartbeat settings |
ZeroMqEventBus.java | Updates factory methods to accept heartbeat period parameter |
UnboundZmqEventBus.java | Configures heartbeat on PUB/SUB sockets and fixes isReady() method |
BoundZmqEventBus.java | Configures heartbeat on XPUB/XSUB sockets |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
java/src/org/openqa/selenium/events/zeromq/UnboundZmqEventBus.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Viet Nguyen Duc <[email protected]>
User description
🔗 Related Issues
💥 What does this PR do?
Fixes SeleniumHQ/docker-selenium#2991 - Distributor Requires Restart to Register New Nodes After Hours of Inactivity.
Implementation adds configurable ZeroMQ heartbeat support to prevent stale connection issues in Selenium Grid's EventBus, particularly for Node-to-Hub registration after long idle periods.
After extended idle periods (e.g., 8 hours), ZeroMQ connections can become stale. Nodes send registration requests, but the Distributor doesn't recognize them because the ZMQ socket layer is broken, even though network connections appear fine.
Prevents Stale Connections: ZMQ detects and recovers from stale connections automatically
Configurable: Can be tuned based on network conditions and requirements
Backward Compatible: Defaults to 60 seconds if not specified
Automatic Recovery: ZMQ heartbeat mechanism handles connection health monitoring
Refer to zeromq/jeromq#364
Heartbeat Settings
With a 60-second heartbeat period:
These settings ensure:
🔧 Implementation Notes
💡 Additional Considerations
🔄 Types of changes
PR Type
Bug fix, Enhancement
Description
Adds configurable ZeroMQ heartbeat mechanism to EventBus sockets to prevent stale connections after idle periods
Implements heartbeat configuration with interval, timeout (3x interval), and TTL (6x interval) settings
Introduces
--eventbus-heartbeat-period
CLI flag with 60-second default valueApplies heartbeat settings to all ZMQ socket types (XPUB, XSUB, PUB, SUB)
Diagram Walkthrough
File Walkthrough
BoundZmqEventBus.java
Add heartbeat configuration to bound EventBus sockets
java/src/org/openqa/selenium/events/zeromq/BoundZmqEventBus.java
Duration heartbeatPeriod
parameter to constructorconfigureHeartbeat()
method for XPUB and XSUB socketsUnboundZmqEventBus
UnboundZmqEventBus.java
Add heartbeat configuration to unbound EventBus sockets
java/src/org/openqa/selenium/events/zeromq/UnboundZmqEventBus.java
Duration heartbeatPeriod
parameter to constructorconfigureHeartbeat()
method for PUB and SUB socketsZeroMqEventBus.java
Add heartbeat period parameter to EventBus factory methods
java/src/org/openqa/selenium/events/zeromq/ZeroMqEventBus.java
Duration heartbeatPeriod
parameter tocreate()
methodseventbus-heartbeat-period
config value with 60-second defaultEventBusFlags.java
Add CLI flag for EventBus heartbeat period configuration
java/src/org/openqa/selenium/grid/server/EventBusFlags.java
--eventbus-heartbeat-period
CLI parameterEventBusOptions.java
Add heartbeat period configuration option to EventBusOptions
java/src/org/openqa/selenium/grid/server/EventBusOptions.java
DEFAULT_HEARTBEAT_PERIOD
constant set to 60 secondsgetHeartbeatPeriod()
method to read config valueDuration
object from configured heartbeat period