Increase node up timeout in AbstractLocalClusterFactory #129639
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
elasticsearchmachine
added
needs:triage
gmarouli
added
>test
elasticsearchmachine
added
Team:Delivery
Collaborator
|
Pinging @elastic/es-delivery (Team:Delivery) |
This was
linked to
issues
Jun 19, 2025
brianseeders
approved these changes
Jun 19, 2025
Collaborator
💚 Backport successful
|
gmarouli
added a commit
to gmarouli/elasticsearch
that referenced
this pull request
Jun 19, 2025
In the last two months a lot of tests were converted to use the newer rest test framework. Some tests start 1 node, other start 3 nodes, others even more, the framework runs tests in parallel but it doesn't know how many nodes its tests needs meaning that running 3 tests in parallel, for example, can be very different when they are single node clusters or 3 node clusters etc. During this execution we saw the 3x more CPU load than what we would want to have ideally. Currently there is no good solution for this because if dial down the concurrency we will use the nodes inefficiently, but if we keep the concurrency to where it is we risk longer start up times. Considering that the starting time of elasticsearch is not related to this test, we choose to increase the timeout to reduce the noise.
elasticsearchmachine
pushed a commit
that referenced
this pull request
Jun 19, 2025
…29720) In the last two months a lot of tests were converted to use the newer rest test framework. Some tests start 1 node, other start 3 nodes, others even more, the framework runs tests in parallel but it doesn't know how many nodes its tests needs meaning that running 3 tests in parallel, for example, can be very different when they are single node clusters or 3 node clusters etc. During this execution we saw the 3x more CPU load than what we would want to have ideally. Currently there is no good solution for this because if dial down the concurrency we will use the nodes inefficiently, but if we keep the concurrency to where it is we risk longer start up times. Considering that the starting time of elasticsearch is not related to this test, we choose to increase the timeout to reduce the noise.
kderusso
pushed a commit
to kderusso/elasticsearch
that referenced
this pull request
Jun 23, 2025
In the last two months a lot of tests were converted to use the newer rest test framework. Some tests start 1 node, other start 3 nodes, others even more, the framework runs tests in parallel but it doesn't know how many nodes its tests needs meaning that running 3 tests in parallel, for example, can be very different when they are single node clusters or 3 node clusters etc. During this execution we saw the 3x more CPU load than what we would want to have ideally. Currently there is no good solution for this because if dial down the concurrency we will use the nodes inefficiently, but if we keep the concurrency to where it is we risk longer start up times. Considering that the starting time of elasticsearch is not related to this test, we choose to increase the timeout to reduce the noise.
mridula-s109
pushed a commit
to mridula-s109/elasticsearch
that referenced
this pull request
Jun 25, 2025
In the last two months a lot of tests were converted to use the newer rest test framework. Some tests start 1 node, other start 3 nodes, others even more, the framework runs tests in parallel but it doesn't know how many nodes its tests needs meaning that running 3 tests in parallel, for example, can be very different when they are single node clusters or 3 node clusters etc. During this execution we saw the 3x more CPU load than what we would want to have ideally. Currently there is no good solution for this because if dial down the concurrency we will use the nodes inefficiently, but if we keep the concurrency to where it is we risk longer start up times. Considering that the starting time of elasticsearch is not related to this test, we choose to increase the timeout to reduce the noise.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Recently we have been seeing an increase on the cluster initialisation failures, for example:
It appears that starting a node exceeds the 3 minute time out:
In the last two months a lot of tests were converted to use the newer rest test framework. Some tests start 1 node, other start 3 nodes, others even more, the framework runs tests in parallel but it doesn't know how many nodes its tests needs meaning that running 3 tests in parallel, for example, can be very different when they are single node clusters or 3 node clusters etc. During this execution we saw the 3x more CPU load than what we would want to have ideally.
Currently there is no good solution for this because if dial down the concurrency we will use the nodes inefficiently, but if we keep the concurrency to where it is we risk longer start up times. Considering that the starting time of elasticsearch is not related to this test, we choose to increase the timeout to reduce the noise.
Fixes: