-
Notifications
You must be signed in to change notification settings - Fork 583
feat: support backoff/retry in OTLP #3126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3126 +/- ##
=======================================
+ Coverage 79.6% 80.8% +1.2%
=======================================
Files 124 128 +4
Lines 23174 23090 -84
=======================================
+ Hits 18456 18676 +220
+ Misses 4718 4414 -304 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
3847b26
to
fb141db
Compare
603c31e
to
75f0d71
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR implements retry logic with exponential backoff and jitter for OTLP exporters to handle transient failures gracefully, addressing issue #3081. The implementation supports both HTTP and gRPC protocols with protocol-specific error classification and server-provided throttling hints.
- Adds a new
retry
module toopentelemetry-sdk
with configurable retry policies and exponential backoff - Implements protocol-specific error classification in
opentelemetry-otlp
for HTTP and gRPC responses - Integrates retry functionality into all OTLP exporters (traces, metrics, logs) for both HTTP and gRPC transports
Reviewed Changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.
Show a summary per file
File | Description |
---|---|
opentelemetry-sdk/src/retry.rs | Core retry module with exponential backoff, jitter, and error classification |
opentelemetry-otlp/src/retry_classification.rs | Protocol-specific error classification for HTTP and gRPC responses |
opentelemetry-otlp/src/exporter/tonic/*.rs | gRPC exporter integration with retry functionality |
opentelemetry-otlp/src/exporter/http/*.rs | HTTP exporter integration with retry functionality |
opentelemetry-otlp/Cargo.toml | Feature flags and dependencies for retry support |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
af933a2
to
f1636a0
Compare
ba1f9e2
to
ff33723
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the HTTP exporters look good now. Love all those red lines.
c0dba75
to
059943f
Compare
059943f
to
693765b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍🏼 for the HTTP code. I can't see a clear way to reuse more of the Tonic code.
Sorry for delay. I would like to review during this week - assigning to myself. |
af8f832
to
27d303a
Compare
8cb6a43
to
01e6526
Compare
… we don't want retry
01e6526
to
9ba3a06
Compare
Fixes #3081, building on the work started by @AaronRM 🤝
Changes
A new retry module added to
opentelemetry-sdk
Models the sorts of retry an operation may request (retry / can't retry / throttle), and provides a helper
retry_with_backoff
mechanism that can be used to wrap up a retryable operation and retry it. The helper relies onexperimental_async_runtime
for its runtime abstraction, to provide the actual pausing. It also takes a lambda to classify the error, so the caller can inform the retry mechanism if a retry is required.A new retry_classification module added to
opentelemetry-otlp
This bit takes the actual error responses that we get back over OTLP and maps them back to the retry model. Because this is OTLP-specific stuff it belongs here rather than alongside the retry code.
Retry binding
... happens in each one of the concrete exporters to tie it all together.
Also ...
Open Questions
Merge requirement checklist
CHANGELOG.md
files updated for non-trivial, user-facing changes