fix(async-processor): concurrent exports actually serialised #3028

alexbrt · 2025-06-18T14:19:11Z

#2685 unintentionally broke parallel exports by awaiting the export() future directly in opentelemetry-sdk/src/trace/span_processor_with_async_runtime.rs, rather than passing it to the runtime for concurrent polling. As a result, OTEL_BSP_MAX_CONCURRENT_EXPORTS became ineffective, serialising all exports and increasing the risk of dropped spans under load.

This PR restores true parallelism by respecting max_concurrent_exports, and adds tests to verify:

Exports run in parallel when max_concurrent_exports > 1
Exports are serialised when max_concurrent_exports == 1

Merge requirement checklist

CONTRIBUTING guidelines followed
Unit tests added/updated (if applicable)
Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
Changes in public API reviewed (if applicable)

linux-foundation-easycla · 2025-06-18T14:19:15Z

The committers listed above are authorized under a signed CLA.

✅ login: TommyCpp / name: Zhongyang Wu (8346e30)
✅ login: alexbrt / name: Alexandru-Liviu Bratosin (1905b50, d8e046e, 2076120, f7faf80)

codecov · 2025-06-18T14:22:04Z

Codecov Report

Attention: Patch coverage is 94.73684% with 5 lines in your changes missing coverage. Please review.

Project coverage is 80.2%. Comparing base (5e447d0) to head (8346e30).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...sdk/src/trace/span_processor_with_async_runtime.rs	94.7%	5 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #3028     +/-   ##
=======================================
+ Coverage   80.0%   80.2%   +0.1%     
=======================================
  Files        126     126             
  Lines      21879   21949     +70     
=======================================
+ Hits       17519   17604     +85     
+ Misses      4360    4345     -15

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull Request Overview

This PR restores true parallel exports by scheduling export() futures on the async runtime instead of awaiting them directly, and adds tests to verify behavior under different max_concurrent_exports settings.

Refactored export calls to use a static async function with RwLock-wrapped exporter and FuturesUnordered for concurrency.
Updated shutdown and resource-setting to acquire a write lock on the exporter.
Added tokio-based tests (TrackingExporter) to ensure exports are parallel when allowed and serialized when limited.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
opentelemetry-sdk/src/trace/span_processor_with_async_runtime.rs	Refactor export logic for concurrent scheduling and lock-protect the exporter; update shutdown and set_resource; add concurrency tests.
opentelemetry-sdk/Cargo.toml	Include the `rt-tokio` feature in `experimental_trace_batch_span_processor_with_async_runtime`.

opentelemetry-sdk/src/trace/span_processor_with_async_runtime.rs

lalitb · 2025-06-18T21:32:37Z

opentelemetry-sdk/src/trace/span_processor_with_async_runtime.rs

@@ -188,13 +190,19 @@ struct BatchSpanProcessorInternal<E, R> {
    spans: Vec<SpanData>,
    export_tasks: FuturesUnordered<BoxFuture<'static, OTelSdkResult>>,
    runtime: R,
-    exporter: E,
+    exporter: Arc<RwLock<E>>,


Going forward, the SpanExporter trait should be redesigned to use immutable references (&self) for all methods. This would allow us to remove the RwLock and use just Arc<E> for sharing the exporter across concurrent tasks - similar to how LogExporter is implemented.

Agreed — my goal was mainly to minimise the scope of the change, but you're right that using a RwLock is... questionable.

Can you add a TODO comment for this so someone can work on it? I'll also file an issue. Thanks.

lalitb

LGTM. Thanks for adding tests for concurrent export.

opentelemetry-sdk/src/trace/span_processor_with_async_runtime.rs

lalitb · 2025-06-19T16:13:05Z

opentelemetry-sdk/src/trace/span_processor_with_async_runtime.rs

@@ -188,13 +190,19 @@ struct BatchSpanProcessorInternal<E, R> {
    spans: Vec<SpanData>,
    export_tasks: FuturesUnordered<BoxFuture<'static, OTelSdkResult>>,
    runtime: R,
-    exporter: E,
+    exporter: Arc<RwLock<E>>,


Can you add a TODO comment for this so someone can work on it? I'll also file an issue. Thanks.

opentelemetry-sdk/Cargo.toml

cijothomas

Thanks for catching this and fixing! Can you add a changelog entry too?

(Concurrent Export support is something we need to support everywhere, and is pending some spec-level discussions. But given this functionality was already there and got removed unintentionally, no problem adding it back)

alexbrt · 2025-06-24T10:37:25Z

@lalitb @cijothomas Thanks again for all the reviews! 🙏 Is there anything else outstanding that needs to be addressed before this can be merged?

alexbrt · 2025-07-11T15:40:04Z

Any updates here? 👀

TommyCpp

Redesign the SpanExporter trait to use immutable references (&self)

OK to merge this PR as it is but we should fix this before next release so we don't cause a preformance regression

TommyCpp · 2025-07-14T01:55:44Z

opentelemetry-sdk/src/trace/span_processor_with_async_runtime.rs

+    // TODO: Redesign the `SpanExporter` trait to use immutable references (`&self`)
+    // for all methods. This would allow us to remove the `RwLock` and just use `Arc<E>`,
+    // similar to how `crate::logs::LogExporter` is implemented.
+    exporter: Arc<RwLock<E>>,


Emmm if we need to add RwLock to support concurrent export maybe we should consider offer two favor of span processor so users that don't use concurrent export don't have to pay the cost

lalitb

LGTM. RwLock (and so the dependency on tokio/sync) should be removed in subsequent iteration. Will create a tracking issue for same.

EDIT: issue #3065

alexbrt requested a review from a team as a code owner June 18, 2025 14:19

alexbrt force-pushed the alxbrt/concurrent-exports branch 2 times, most recently from 290d3be to 06fa8ae Compare June 18, 2025 14:36

lalitb requested a review from Copilot June 18, 2025 20:11

Copilot AI reviewed Jun 18, 2025

View reviewed changes

opentelemetry-sdk/src/trace/span_processor_with_async_runtime.rs Outdated Show resolved Hide resolved

fix(async-processor): concurrent exports actually serialised

1905b50

alexbrt force-pushed the alxbrt/concurrent-exports branch from 06fa8ae to 1905b50 Compare June 18, 2025 20:15

lalitb reviewed Jun 18, 2025

View reviewed changes

opentelemetry-sdk/src/trace/span_processor_with_async_runtime.rs Outdated Show resolved Hide resolved

lalitb reviewed Jun 18, 2025

View reviewed changes

address review

f7faf80

alexbrt requested a review from lalitb June 19, 2025 13:22

lalitb self-assigned this Jun 19, 2025

lalitb approved these changes Jun 19, 2025

View reviewed changes

cijothomas reviewed Jun 19, 2025

View reviewed changes

opentelemetry-sdk/Cargo.toml Outdated Show resolved Hide resolved

cijothomas reviewed Jun 19, 2025

View reviewed changes

alexbrt added 2 commits June 19, 2025 17:52

address review 2

d8e046e

only pull in tokio::sync

2076120

alexbrt requested a review from cijothomas June 19, 2025 22:01

alexbrt requested a review from lalitb July 1, 2025 17:03

Merge branch 'main' into alxbrt/concurrent-exports

8346e30

TommyCpp approved these changes Jul 14, 2025

View reviewed changes

lalitb approved these changes Jul 14, 2025

View reviewed changes

lalitb mentioned this pull request Jul 14, 2025

Modify SpanExporter methods to use immutable reference to &self #3065

Open

Copilot AI mentioned this pull request Jul 14, 2025

fix: Modify SpanExporter trait to use immutable references for shutdown and force_flush methods #3066

Draft

lalitb merged commit 0631070 into open-telemetry:main Jul 14, 2025
27 checks passed

cijothomas mentioned this pull request Jul 22, 2025

Stabilization - OTLP #3061

Open

fix(async-processor): concurrent exports actually serialised #3028

fix(async-processor): concurrent exports actually serialised #3028

Conversation

alexbrt commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge requirement checklist

Uh oh!

linux-foundation-easycla bot commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

lalitb Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

alexbrt Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

lalitb Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lalitb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lalitb Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cijothomas left a comment

Choose a reason for hiding this comment

Uh oh!

alexbrt commented Jun 24, 2025

Uh oh!

alexbrt commented Jul 11, 2025

Uh oh!

TommyCpp left a comment

Choose a reason for hiding this comment

Uh oh!

TommyCpp Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

lalitb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alexbrt commented Jun 18, 2025 •

edited

Loading

linux-foundation-easycla bot commented Jun 18, 2025 •

edited

Loading

codecov bot commented Jun 18, 2025 •

edited

Loading

lalitb Jun 19, 2025 •

edited

Loading

lalitb Jun 19, 2025 •

edited

Loading

lalitb left a comment •

edited

Loading