Skip to content

Conversation

l-trotta
Copy link
Contributor

Adds retry logic to the bulk ingester, allowing it to to retry operations that failed with error 429 (too many requests), hoping that the error will recover and the request will go through. Users can configure the desired backoff policy using the backoffPolicy() method in the bulk ingester builder, which will look like this:

BulkIngester ingester = BulkIngester.of(b -> b
            .client(client)
            ...
            .listener(listener)
            .flushInterval(1000, TimeUnit.MILLISECONDS)
            .backoffPolicy(BackoffPolicy.constantBackoff(50L, 8)) 

This is an example of constant backoff, meaning the single failed operation will be retried 8 times every 50 milliseconds.

Copy link
Member

@swallez swallez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good overall, except for some potentially costly list operations. Bulk ingestion is used in high load contexts where performance is critical, and the number of operations per request (and thus list size) can be large.

Copy link
Member

@swallez swallez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Great stuff!

@l-trotta l-trotta merged commit e7120d4 into main Feb 21, 2025
8 checks passed
@l-trotta l-trotta deleted the bulk-ingester-retry-policy branch February 21, 2025 16:50
github-actions bot pushed a commit that referenced this pull request Feb 21, 2025
* WIP: added backoff policy logic to operations

* all basic key points implemented

* bugfixes

* retry - main implementation done

* working unit test, bugfixes

* more tests

* unit tests done

* refactoring

* more refactoring

* checkstyle

* refactoring and optimizations
github-actions bot pushed a commit that referenced this pull request Feb 21, 2025
* WIP: added backoff policy logic to operations

* all basic key points implemented

* bugfixes

* retry - main implementation done

* working unit test, bugfixes

* more tests

* unit tests done

* refactoring

* more refactoring

* checkstyle

* refactoring and optimizations
l-trotta added a commit that referenced this pull request Feb 21, 2025
* WIP: added backoff policy logic to operations

* all basic key points implemented

* bugfixes

* retry - main implementation done

* working unit test, bugfixes

* more tests

* unit tests done

* refactoring

* more refactoring

* checkstyle

* refactoring and optimizations

Co-authored-by: Laura Trotta <[email protected]>
l-trotta added a commit that referenced this pull request Feb 21, 2025
* WIP: added backoff policy logic to operations

* all basic key points implemented

* bugfixes

* retry - main implementation done

* working unit test, bugfixes

* more tests

* unit tests done

* refactoring

* more refactoring

* checkstyle

* refactoring and optimizations

Co-authored-by: Laura Trotta <[email protected]>
l-trotta added a commit that referenced this pull request Apr 4, 2025
* WIP: added backoff policy logic to operations

* all basic key points implemented

* bugfixes

* retry - main implementation done

* working unit test, bugfixes

* more tests

* unit tests done

* refactoring

* more refactoring

* checkstyle

* refactoring and optimizations

Co-authored-by: Laura Trotta <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants