[ML] Add retry logic for datafeed missing data checks #132991
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Exclude .claude-flow/, .swarm/ directories and temporary test/analysis files from version control to keep the repository clean.
Implement retry mechanism with exponential backoff for the checkForMissingDataIfNecessary method in DatafeedJob. This improves resilience when checking for missing data encounters transient failures. Changes: - Add retry logic with up to 3 retries using exponential backoff - Log warnings on retry attempts - Issue audit warning if all retries fail - Continue datafeed operation even if missing data check fails - Add comprehensive unit tests for retry scenarios The backoff delays are: 100ms, 200ms, 400ms for successive retries. Fixes elastic#107873 🤖 Generated with Claude Code Co-Authored-By: Claude <[email protected]>
elasticsearchmachine
added
external-contributor
szybia
added
:ml
Collaborator
|
Pinging @elastic/ml-core (Team:ML) |
Member
|
@elasticmachine test this please |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes issue #107873 by adding retry logic to the
checkForMissingDataIfNecessarymethod inDatafeedJob. The change improves resilience when checking for missing data encounters transient failures.Problem
The datafeed's missing data check could fail due to transient issues (network problems, temporary unavailability of indices, etc.), causing the entire check to be skipped. This meant that delayed data detection would not work properly during these transient failures.
Solution
Implemented a retry mechanism with exponential backoff that:
detectMissingData()throws an exceptionChanges
checkForMissingDataIfNecessary()methodTesting
Added unit tests that verify:
Related Issues
Fixes #107873
Checklist
Notes
The retry mechanism is conservative with a maximum of 3 retries and relatively short backoff delays to avoid significantly delaying the datafeed operation. The total maximum delay from retries is approximately 700ms (100 + 200 + 400ms).
🤖 Generated with Claude Code
Co-Authored-By: Claude [email protected]