Reinstate watch count check in tests with busy/wait #123979
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lukewhiting
added
>test-failure
Contributor
There was a problem hiding this comment.
PR Overview
This PR reinstates a watch count check in integration tests by reintroducing a busy/wait pattern to handle temporary unavailability of watcher stats during shard moves and watcher service restarts.
- Reintroduces assertBusy checks to ensure the watch count is 1L before further test execution.
- Updates three test methods in WatchAckTests.java to use WatcherStatsRequestBuilder for polling watcher stats.
Reviewed Changes
| File | Description |
|---|---|
| x-pack/plugin/watcher/src/internalClusterTest/java/org/elasticsearch/xpack/watcher/test/integration/WatchAckTests.java | Adds busy/wait assertions using WatcherStatsRequestBuilder to ensure proper watch count |
Copilot reviewed 1 out of 1 changed files in this pull request and generated no comments.
elasticsearchmachine
added
Team:Data Management
Collaborator
|
Pinging @elastic/es-data-management (Team:Data Management) |
lukewhiting
added
>test
lukewhiting
force-pushed
the
es-9782-watcher-stats-reset
branch
from
d86373c to
6b6a869
Compare
masseyke
approved these changes
Mar 5, 2025
Member
masseyke
left a comment
masseyke
left a comment
There was a problem hiding this comment.
LGTM. We can always revert again if it causes problems.
This was referenced Mar 6, 2025
Collaborator
lukewhiting
added a commit
to lukewhiting/elasticsearch
that referenced
this pull request
Mar 6, 2025
elasticsearchmachine
pushed a commit
that referenced
this pull request
Mar 6, 2025
elasticsearchmachine
pushed a commit
that referenced
this pull request
Mar 6, 2025
georgewallace
pushed a commit
to georgewallace/elasticsearch
that referenced
this pull request
Mar 11, 2025
costin
pushed a commit
to costin/elasticsearch
that referenced
this pull request
Mar 15, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses a failing test initially flagged by #109679 by reverting the initial workaround created by #111435. It wraps the request in a busy/wait to deal with the stats temporarily being unavailable during a move of the watcher shards between nodes and the subsequent restarting of the watcher stats service.
Looking into how that's stats are generated, it's not so much that they reset on move but that they just take a short amount of time to repopulate as watches are loaded in again by the newly started instance of the watcher service on the node the shards have been moved to. Wrapping this request in a wait loop allows the tests to cope with this move.
I let the change soak by running the affected tests on a loop for 2 hours on my laptop and no failures occurred during that time.
Fixes ES-9782