-
Notifications
You must be signed in to change notification settings - Fork 501
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Retry to OTLP Exporter #2727
base: main
Are you sure you want to change the base?
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2727 +/- ##
======================================
Coverage 79.3% 79.3%
======================================
Files 123 124 +1
Lines 22670 22800 +130
======================================
+ Hits 17986 18093 +107
- Misses 4684 4707 +23 ☔ View full report in Codecov by Sentry. |
use std::future::Future; | ||
use std::time::{Duration, SystemTime}; | ||
use opentelemetry::otel_warn; | ||
use tokio::time::sleep; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
tokio
runtime won't be available here. Should we move this code under exporter/tonic
, or if we want to keep it generic, one option could be to make the delay function configurable as an argument to retry_with_exponential_backoff
- such that retry function can be callable from both async and blocking code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The idea was the retry logic should be generic and used across all the OTLP exporters. Going to try that approach first and try to avoid the direct tokio dependency.
Err(err) if attempt < policy.max_retries => { | ||
attempt += 1; | ||
// Log the error and retry after a delay with jitter | ||
otel_warn!(name: "OtlpRetry", message = format!("Retrying operation {:?} due to error: {:?}", operation_name, err)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
otel_warn!(name: "OtlpRetry", message = format!("Retrying operation {:?} due to error: {:?}", operation_name, err)); | |
otel_debug!(name: "OtlpRetry", message = format!("Retrying operation {:?} due to error: {:?}", operation_name, err)); |
Fixes #
Design discussion issue (if applicable) #
Changes
Please provide a brief description of the changes here.
Merge requirement checklist
CHANGELOG.md
files updated for non-trivial, user-facing changes