roachtest: beef up jobs/stress roachtest #158304
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
This change is |
msbutler
force-pushed
the
butler-job-stress-steroids
branch
2 times, most recently
from
10ef427 to
3006d24
Compare
Potential Bug(s) DetectedThe three-stage Claude Code analysis has identified potential bug(s) in this PR that may warrant investigation. Next Steps: Note: When viewing the workflow output, scroll to the bottom to find the Final Analysis Summary. After you review the findings, please tag the issue as follows:
|
github-actions
bot
added
the
o-AI-Review-Potential-Issue-Detected
msbutler
force-pushed
the
butler-job-stress-steroids
branch
from
897a4bf to
c2a36a1
Compare
msbutler
changed the title
Collaborator
Author
|
this test sufficiently stresses the job system, as the job adoption rate is significantly decreased (but not flatlined) we also see plenty of logs indicating claim query timeouts (added in #160084), for example: |
This patch reconfigures the jobs/stress roachtest logic in a few ways: - now runs on a 20 node cluster in the weekly suite - lowers the base interval to 0.1, to simulate internal job query contention at a 200 node scale - refines the job control loop logic to pause 20% of running changefeeds, resume 20% of paused changefeeds, recreate up to 200 canceled changefeeds, per iteration. - fails the roachtest if a job stays unclaimed for more than 5 minutes. Informs: cockroachdb#158976 Release note: none
msbutler
force-pushed
the
butler-job-stress-steroids
branch
from
c2a36a1 to
90b6904
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch reconfigures the jobs/stress roachtest logic in a few ways:
a 200 node scale
resume 20% of paused changefeeds, recreate up to 200 canceled changefeeds,
per iteration.
Informs: #158976
Release note: none