fix: handle shutdown in logs exporter

[stefanobaghino]

Fixes #2777

Changes

This follows the approach suggested here by @scottgerring:

The issue is in TonicLogsClient which uses tokio::sync::Mutex (async) but the LogExporter::shutdown_with_timeout() trait method is synchronous. This prevents calling .lock().await, so the current implementation has a TODO and just returns Ok() without actually shutting down the gRPC client.

Comparing shutdown on TonicMetricsClient , and in particular the type of the mutex used for the inner client and locked in the shutdown, seems to show a workable pattern.

Merge requirement checklist

• CONTRIBUTING guidelines followed
• Unit tests added/updated (if applicable)
• Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
• Changes in public API reviewed (if applicable)

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: scottgerring / name: Scott Gerring (4df15de)
• ✅ login: stefanobaghino / name: Stefano Baghino (4df15de, 28b45dd, 8181574, e6fae39)

[codecov]

Codecov Report

❌ Patch coverage is 0% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.7%. Comparing base (df412fe) to head (4df15de).

Files with missing lines	Patch %	Lines
opentelemetry-otlp/src/exporter/tonic/logs.rs	0.0%	22 Missing ⚠️
opentelemetry-otlp/src/exporter/tonic/metrics.rs	0.0%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #3255     +/-   ##
=======================================
- Coverage   80.8%   80.7%   -0.1%     
=======================================
  Files        129     129             
  Lines      23203   23212      +9     
=======================================
  Hits       18750   18750             
- Misses      4453    4462      +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[scottgerring]

Thanks! LGTM

[scottgerring]

I think this is helpful; I don't think the user would necessarily see which failed otherwise

[scottgerring]

@stefanobaghino that was a quick turnaround :D

[cijothomas]

are there any repercussions of moving to std Mutex instead of tokio one inside the async export call?

[scottgerring]

I think it's fine, and its the same pattern used in TonicsMetricsClient for the last couple years, which as far as I can tell we've not had problems with (linky to blame).

In the export, we hold the lock only so long as to clone the client and call the interceptor - and significantly, not across any await points, which is the main contra-indicator for std mutexes:

                let (mut client, metadata, extensions) = self
                    .inner
                    .lock()
                    .map_err(|e| tonic::Status::internal(format!("Failed to acquire lock: {e:?}")))
                    .and_then(|mut inner| match &mut *inner {
Some(inner) => {
let (m, e, _) = inner
.interceptor
.call(Request::new(()))
.map_err(|e| {
// Convert interceptor errors to tonic::Status for retry classification
tonic::Status::internal(format!("interceptor error: {e:?}"))
})? // lock released

I think if you called shutdown_with_timeout on a single-threaded tokio runtime while the runtime is still trying to do an export, and the export was at that exact moment in the code I highlighted above, then you could potentially deadlock it, but this relies on amazing timing, and I think we already warn against doing exactly this:

opentelemetry-rust/opentelemetry-sdk/src/logs/batch_log_processor.rs

Lines 99 to 102 in df412fe

	/// **Warning**: When using tokio's current-thread runtime, `shutdown()`, which
	/// is a blocking call ,should not be called from your main thread. This can
	/// cause deadlock. Instead, call `shutdown()` from a separate thread or use
	/// tokio's `spawn_blocking`.

I note there are some other suggestions (e.g. spawning a thread) to do this particular thing in the linked issue, but this feels a bit overwrought.

What are you thinking @cijothomas ?