Code stuck on exit

[jiri-blahos-tfs]

Describe your environment

I use opentelemetry-cpp 1.18.0 built on Windows 11 (conaninfo file at the bottom).

Steps to reproduce

1. Initialize otel:
   opentelemetry::trace::TracerProvider
   opentelemetry::sdk::trace::SimpleSpanProcessor
   opentelemetry::exporter::otlp::OtlpGrpcExporter

I set tracer provider using:

    auto exporter = CreateExporter(configuration);
    m_Exporter = exporter.get();
    auto processor = CreateProcessor(std::move(exporter), configuration);
    m_Processor = processor.get();

    auto const resource_attributes = resource::ResourceAttributes{---redacted---};
    auto const provider = nostd::shared_ptr<trace_api::TracerProvider>(
        new trace_sdk::TracerProvider(std::move(processor), resource_instance));
    opentelemetry::trace::Provider::SetTracerProvider(provider);

2. Export some traces (we are exporing to elastic over grpc, it works fine)
3. Exit the app
4. App does not exit but hangs

What is the expected behavior?
App exits immediately

What is the actual behavior?
App hangs for several minutes

Additional context
Feel free to ask for additional info, I don't know what might be relevant for this issue. I know the info I provide is not enough for proper reproduction, I just want to start somewhere without writing too much, see if anyone might have some tips what to look for.

What I do on app exit (calling flush and shutdown on everything out of paranoia):

    {
        if (m_Exporter != nullptr)
        {
            // flush with 1s timeout
            m_Exporter->ForceFlush(std::chrono::seconds(1));
            m_Exporter->Shutdown();
        }
        if (m_Processor != nullptr)
        {
            // flush with 1s timeout
            m_Processor->ForceFlush(std::chrono::seconds(1));
            m_Processor->Shutdown();
        }
        auto const provider_api = trace_api::Provider::GetTracerProvider();
        auto const provider_sdk_raw = dynamic_cast<trace_sdk::TracerProvider*>(provider_api.get());
        if (provider_sdk_raw != nullptr)
        {
            provider_sdk_raw->ForceFlush(std::chrono::seconds(1));
            provider_sdk_raw->Shutdown();
        }
    }
    // reset tracer provider to noop
    trace_api::Provider::SetTracerProvider(nostd::shared_ptr(new trace_api::NoopTracerProvider()));

All this shutdown code finishes with no errors.
(I keep raw pointer references to the span processor and exporter after their creation)

Stack trace:

ntdll.dll!00007ffc562d4844() Unknown
ntdll.dll!00007ffc5629981b() Unknown
KernelBase.dll!00007ffc5340f849() Unknown
CodeTracing.dll!absl::lts_20220623::synchronization_internal::Waiter::Wait(class absl::lts_20220623::synchronization_internal::KernelTimeout) Unknown
CodeTracing.dll!AbslInternalPerThreadSemWait_lts_20220623() Unknown
CodeTracing.dll!absl::lts_20220623::synchronization_internal::PerThreadSem::Wait(class absl::lts_20220623::synchronization_internal::KernelTimeout) Unknown
CodeTracing.dll!absl::lts_20220623::Mutex::DecrementSynchSem(class absl::lts_20220623::Mutex *,struct absl::lts_20220623::base_internal::PerThreadSynch *,class absl::lts_20220623::synchronization_internal::KernelTimeout) Unknown
CodeTracing.dll!absl::lts_20220623::CondVar::WaitCommon(class absl::lts_20220623::Mutex *,class absl::lts_20220623::synchronization_internal::KernelTimeout) Unknown
CodeTracing.dll!absl::lts_20220623::CondVar::Wait(class absl::lts_20220623::Mutex *) Unknown
CodeTracing.dll!gpr_cv_wait() C++
CodeTracing.dll!grpc_timer_manager_tick(void) C++
CodeTracing.dll!grpc_shutdown_internal_locked(void) C++
CodeTracing.dll!grpc_shutdown() C++
CodeTracing.dll!grpc::Channel::~Channel(void) C++
CodeTracing.dll!std::_Ref_count::_Destroy(void) C++
CodeTracing.dll!std::_Ref_count_base::_Decref() Line 1107 C++
CodeTracing.dll!std::_Ptr_base::_Decref(void) Unknown
CodeTracing.dll!std::shared_ptr::~shared_ptr(void) Unknown
CodeTracing.dll!opentelemetry::proto::collector::trace::v1::TraceService::Stub::~Stub(void) Unknown
CodeTracing.dll!opentelemetry::proto::collector::trace::v1::TraceService::Stub::vector deleting destructor'(unsigned int) Unknown CodeTracing.dll!std::default_delete<class opentelemetry::proto::collector::trace::v1::TraceService::StubInterface>::operator()(class opentelemetry::proto::collector::trace::v1::TraceService::StubInterface *) Unknown CodeTracing.dll!std::_Ref_count_resource<class opentelemetry::proto::collector::trace::v1::TraceService::StubInterface *,struct std::default_delete<class opentelemetry::proto::collector::trace::v1::TraceService::StubInterface> >::_Destroy(void) Unknown CodeTracing.dll!std::_Ref_count_base::_Decref() Line 1107 C++ CodeTracing.dll!std::_Ptr_base<class opentelemetry::proto::collector::trace::v1::TraceService::StubInterface>::_Decref(void) Unknown CodeTracing.dll!std::shared_ptr<class opentelemetry::proto::collector::trace::v1::TraceService::StubInterface>::~shared_ptr<class opentelemetry::proto::collector::trace::v1::TraceService::StubInterface>(void) Unknown CodeTracing.dll!opentelemetry::v1::exporter::otlp::OtlpGrpcExporter::~OtlpGrpcExporter(void) Unknown CodeTracing.dll!opentelemetry::v1::exporter::otlp::OtlpGrpcExporter::scalar deleting destructor'(unsigned int) C++
CodeTracing.dll!std::default_deleteopentelemetry::v1::sdk::trace::SpanExporter::operator()(opentelemetry::v1::sdk::trace::SpanExporter * _Ptr) Line 3090 C++
CodeTracing.dll!std::unique_ptr<opentelemetry::v1::sdk::trace::SpanExporter,std::default_deleteopentelemetry::v1::sdk::trace::SpanExporter>::~unique_ptr<opentelemetry::v1::sdk::trace::SpanExporter,std::default_deleteopentelemetry::v1::sdk::trace::SpanExporter>() Line 3200 C++
CodeTracing.dll!opentelemetry::v1::sdk::trace::SimpleSpanProcessor::~SimpleSpanProcessor() Line 88 C++
CodeTracing.dll!opentelemetry::v1::sdk::trace::SimpleSpanProcessor::scalar deleting destructor'(unsigned int) C++ CodeTracing.dll!std::default_delete<opentelemetry::v1::sdk::trace::SpanProcessor>::operator()(opentelemetry::v1::sdk::trace::SpanProcessor * _Ptr) Line 3090 C++ CodeTracing.dll!std::unique_ptr<opentelemetry::v1::sdk::trace::SpanProcessor,std::default_delete<opentelemetry::v1::sdk::trace::SpanProcessor>>::~unique_ptr<opentelemetry::v1::sdk::trace::SpanProcessor,std::default_delete<opentelemetry::v1::sdk::trace::SpanProcessor>>() Line 3200 C++ CodeTracing.dll!opentelemetry::v1::sdk::trace::MultiSpanProcessor::ProcessorNode::~ProcessorNode(void) Unknown CodeTracing.dll!opentelemetry::v1::sdk::trace::MultiSpanProcessor::ProcessorNode::scalar deleting destructor'(unsigned int) Unknown
CodeTracing.dll!opentelemetry::v1::sdk::trace::MultiSpanProcessor::Cleanup(void) Unknown
CodeTracing.dll!opentelemetry::v1::sdk::trace::MultiSpanProcessor::~MultiSpanProcessor(void) Unknown
CodeTracing.dll!opentelemetry::v1::sdk::trace::MultiSpanProcessor::vector deleting destructor'(unsigned int) Unknown CodeTracing.dll!std::default_delete<opentelemetry::v1::sdk::trace::SpanProcessor>::operator()(opentelemetry::v1::sdk::trace::SpanProcessor * _Ptr) Line 3090 C++ CodeTracing.dll!std::unique_ptr<opentelemetry::v1::sdk::trace::SpanProcessor,std::default_delete<opentelemetry::v1::sdk::trace::SpanProcessor>>::~unique_ptr<opentelemetry::v1::sdk::trace::SpanProcessor,std::default_delete<opentelemetry::v1::sdk::trace::SpanProcessor>>() Line 3200 C++ CodeTracing.dll!opentelemetry::v1::sdk::trace::TracerContext::~TracerContext(void) Unknown CodeTracing.dll!opentelemetry::v1::sdk::trace::TracerContext::vector deleting destructor'(unsigned int) Unknown
CodeTracing.dll!std::_Destroy_in_place(class opentelemetry::v1::sdk::trace::TracerContext &) Unknown
CodeTracing.dll!std::_Ref_count_obj2::_Destroy(void) Unknown
CodeTracing.dll!std::_Ref_count_base::_Decref() Line 1107 C++
CodeTracing.dll!std::_Ptr_base::_Decref(void) Unknown
CodeTracing.dll!std::shared_ptr::~shared_ptr(void) Unknown
CodeTracing.dll!opentelemetry::v1::sdk::trace::Tracer::~Tracer(void) Unknown
CodeTracing.dll!opentelemetry::v1::sdk::trace::Tracer::vector deleting destructor'(unsigned int) Unknown CodeTracing.dll!std::_Ref_count<class opentelemetry::v1::sdk::trace::Tracer>::_Destroy(void) Unknown CodeTracing.dll!std::_Ref_count_base::_Decref() Line 1107 C++ CodeTracing.dll!std::_Ptr_base<class opentelemetry::v1::sdk::trace::Tracer>::_Decref(void) Unknown CodeTracing.dll!std::shared_ptr<class opentelemetry::v1::sdk::trace::Tracer>::~shared_ptr<class opentelemetry::v1::sdk::trace::Tracer>(void) Unknown CodeTracing.dll!std::shared_ptr<class opentelemetry::v1::sdk::trace::Tracer>::scalar deleting destructor'(unsigned int) Unknown
CodeTracing.dll!std::_Default_allocator_traits<class std::allocator<class std::shared_ptr > >::destroy<class std::shared_ptr >(class std::allocator<class std::shared_ptr > &,class std::shared_ptr * const) Unknown
CodeTracing.dll!std::Destroy_range<class std::allocator<class std::shared_ptr > >(class std::shared_ptr *,class std::shared_ptr * const,class std::allocator<class std::shared_ptr > &) Unknown
CodeTracing.dll!std::vector<class std::shared_ptr,class std::allocator<class std::shared_ptr > >::Destroy(class std::shared_ptr *,class std::shared_ptr *) Unknown
CodeTracing.dll!std::vector<class std::shared_ptr,class std::allocator<class std::shared_ptr > >::Tidy(void) Unknown
CodeTracing.dll!std::vector<class std::shared_ptr,class std::allocator<class std::shared_ptr > >::~vector<class std::shared_ptr,class std::allocator<class std::shared_ptr > >(void) Unknown
CodeTracing.dll!opentelemetry::v1::sdk::trace::TracerProvider::~TracerProvider(void) Unknown
CodeTracing.dll!opentelemetry::v1::sdk::trace::TracerProvider::vector deleting destructor'(unsigned int) Unknown CodeTracing.dll!std::_Ref_count<opentelemetry::v1::trace::TracerProvider>::_Destroy() Line 1133 C++ CodeTracing.dll!std::_Ref_count_base::_Decref() Line 1107 C++ CodeTracing.dll!std::_Ptr_base<opentelemetry::v1::trace::TracerProvider>::_Decref() Line 1333 C++ CodeTracing.dll!std::shared_ptr<opentelemetry::v1::trace::TracerProvider>::~shared_ptr<opentelemetry::v1::trace::TracerProvider>() Line 1615 C++ CodeTracing.dll!opentelemetry::v1::nostd::shared_ptr<opentelemetry::v1::trace::TracerProvider>::shared_ptr_wrapper::~shared_ptr_wrapper() Line 50 C++ CodeTracing.dll!opentelemetry::v1::nostd::shared_ptr<opentelemetry::v1::trace::TracerProvider>::shared_ptr_wrapper::scalar deleting destructor'(unsigned int) C++
CodeTracing.dll!opentelemetry::v1::nostd::shared_ptropentelemetry::v1::trace::TracerProvider::operator=(const opentelemetry::v1::nostd::shared_ptropentelemetry::v1::trace::TracerProvider & other) Line 136 C++
CodeTracing.dll!opentelemetry::v1::trace::Provider::SetTracerProvider(opentelemetry::v1::nostd::shared_ptropentelemetry::v1::trace::TracerProvider tp) Line 43 C++
CodeTracing.dll!CodeTracing::OpenTelemetryLifetimeManager::Finalize() Line 61 C++
CodeTracing.dll!CodeTracing::OpenTelemetryLifetimeManager::~OpenTelemetryLifetimeManager() Line 25 C++
CodeTracing.dll!CodeTracing::OpenTelemetryLifetimeManager::scalar deleting destructor'(unsigned int) C++ CodeTracing.dll!std::default_delete<CodeTracing::OpenTelemetryLifetimeManager>::operator()(CodeTracing::OpenTelemetryLifetimeManager * _Ptr) Line 3090 C++ CodeTracing.dll!std::unique_ptr<CodeTracing::OpenTelemetryLifetimeManager,std::default_delete<CodeTracing::OpenTelemetryLifetimeManager>>::~unique_ptr<CodeTracing::OpenTelemetryLifetimeManager,std::default_delete<CodeTracing::OpenTelemetryLifetimeManager>>() Line 3200 C++ CodeTracing.dll!CodeTracing::ServiceProvider::~ServiceProvider() C++ CodeTracing.dll!CodeTracing::API::GetInstance'::2'::dynamic atexit destructor for 'serviceProvider''() C++
ucrtbased.dll!00007ffba68b2b99() Unknown
ucrtbased.dll!00007ffba68b25c5() Unknown
ucrtbased.dll!00007ffba68b2717() Unknown
ucrtbased.dll!00007ffba68b2d44() Unknown
CodeTracing.dll!scrt_dllmain_uninitialize_c() Line 399 C++
CodeTracing.dll!dllmain_crt_process_detach(const bool is_terminating) Line 182 C++
CodeTracing.dll!dllmain_crt_dispatch(HINSTANCE * const instance, const unsigned long reason, void * const reserved) Line 220 C++
CodeTracing.dll!dllmain_dispatch(HINSTANCE * const instance, const unsigned long reason, void * const reserved) Line 293 C++
CodeTracing.dll!DllMainCRTStartup(HINSTANCE * const instance, const unsigned long reason, void * const reserved) Line 335 C++
ntdll.dll!00007ffc56258bcf() Unknown
ntdll.dll!00007ffc562815d6() Unknown
ntdll.dll!00007ffc562811cd() Unknown
kernel32.dll!00007ffc54987fcb() Unknown
ucrtbased.dll!00007ffba68b219b() Unknown
ucrtbased.dll!00007ffba68b214f() Unknown
ucrtbased.dll!00007ffba68b2456() Unknown
CodeTracingTest.exe!__scrt_common_main_seh() Line 297 C++
CodeTracingTest.exe!__scrt_common_main() Line 331 C++
CodeTracingTest.exe!mainCRTStartup(void * __formal) Line 17 C++
kernel32.dll!00007ffc5498259d() Unknown
ntdll.dll!00007ffc5628af78() Unknown

conaninfo.txt:

[settings]
arch=x86_64
build_type=Debug
compiler=Visual Studio
compiler.runtime=MDd
compiler.version=16
os=Windows

[requires]
abseil/20220623.Y.Z
grpc/1.Y.Z
libcurl/8.Y.Z
nlohmann_json/3.Y.Z
openssl/3.Y.Z
protobuf/3.Y.Z

[options]
shared=False
with_abseil=True
with_async_export_preview=False
with_elasticsearch=False
with_etw=False
with_gsl=False
with_jaeger=False
with_logs_preview=False
with_metrics_exemplar_preview=False
with_no_deprecated_code=False
with_no_getenv=False
with_otlp=True
with_otlp_grpc=True
with_otlp_http=True
with_prometheus=False
with_stl=False
with_zipkin=False
with_zpages=False

[full_settings]
arch=x86_64
build_type=Debug
compiler=Visual Studio
compiler.runtime=MDd
compiler.version=16
os=Windows

[full_requires]
--redacted--

[full_options]
shared=False
with_abseil=True
with_async_export_preview=False
with_elasticsearch=False
with_etw=False
with_gsl=False
with_jaeger=False
with_logs_preview=False
with_metrics_exemplar_preview=False
with_no_deprecated_code=False
with_no_getenv=False
with_otlp=True
with_otlp_grpc=True
with_otlp_http=True
with_prometheus=False
with_stl=False
with_zipkin=False
with_zpages=False
abseil:shared=False
c-ares:shared=False
c-ares:tools=True
grpc:codegen=True
grpc:cpp_plugin=True
grpc:csharp_ext=False
grpc:csharp_plugin=True
grpc:node_plugin=True
grpc:objective_c_plugin=True
grpc:php_plugin=True
grpc:python_plugin=True
grpc:ruby_plugin=True
grpc:secure=False
grpc:shared=False
libcurl:shared=False
libcurl:with_brotli=False
libcurl:with_c_ares=False
libcurl:with_ca_bundle=auto
libcurl:with_ca_fallback=False
libcurl:with_ca_path=auto
libcurl:with_cookies=True
libcurl:with_crypto_auth=True
libcurl:with_dict=True
libcurl:with_docs=False
libcurl:with_file=True
libcurl:with_ftp=True
libcurl:with_gopher=True
libcurl:with_http=True
libcurl:with_imap=True
libcurl:with_ipv6=True
libcurl:with_largemaxwritesize=False
libcurl:with_ldap=False
libcurl:with_libidn=False
libcurl:with_libpsl=False
libcurl:with_librtmp=False
libcurl:with_libssh2=False
libcurl:with_mqtt=True
libcurl:with_nghttp2=False
libcurl:with_ntlm=True
libcurl:with_ntlm_wb=True
libcurl:with_pop3=True
libcurl:with_proxy=True
libcurl:with_rtsp=True
libcurl:with_smb=True
libcurl:with_smtp=True
libcurl:with_ssl=openssl
libcurl:with_symbol_hiding=False
libcurl:with_telnet=True
libcurl:with_tftp=True
libcurl:with_threaded_resolver=True
libcurl:with_unix_sockets=True
libcurl:with_verbose_debug=True
libcurl:with_verbose_strings=True
libcurl:with_zlib=True
libcurl:with_zstd=False
openssl:386=False
openssl:capieng_dialog=False
openssl:enable_capieng=False
openssl:enable_weak_ssl_ciphers=False
openssl:no_aria=False
openssl:no_asm=False
openssl:no_async=False
openssl:no_bf=False
openssl:no_blake2=False
openssl:no_camellia=False
openssl:no_cast=False
openssl:no_chacha=False
openssl:no_cms=False
openssl:no_comp=False
openssl:no_ct=False
openssl:no_deprecated=False
openssl:no_des=False
openssl:no_dgram=False
openssl:no_dh=False
openssl:no_dsa=False
openssl:no_dso=False
openssl:no_ec=False
openssl:no_ecdh=False
openssl:no_ecdsa=False
openssl:no_engine=False
openssl:no_filenames=False
openssl:no_fips=False
openssl:no_gost=False
openssl:no_idea=False
openssl:no_legacy=False
openssl:no_md2=True
openssl:no_md4=False
openssl:no_mdc2=False
openssl:no_module=False
openssl:no_ocsp=False
openssl:no_pinshared=False
openssl:no_rc2=False
openssl:no_rc4=False
openssl:no_rc5=False
openssl:no_rfc3779=False
openssl:no_rmd160=False
openssl:no_seed=False
openssl:no_sm2=False
openssl:no_sm3=False
openssl:no_sm4=False
openssl:no_sock=False
openssl:no_srp=False
openssl:no_srtp=False
openssl:no_sse2=False
openssl:no_ssl=False
openssl:no_ssl3=False
openssl:no_stdio=False
openssl:no_threads=False
openssl:no_tls1=False
openssl:no_ts=False
openssl:no_whirlpool=False
openssl:no_zlib=False
openssl:openssldir=None
openssl:shared=False
protobuf:debug_suffix=True
protobuf:lite=False
protobuf:shared=False
protobuf:with_rtti=True
protobuf:with_zlib=True
re2:shared=False
zlib:shared=False

[recipe_hash]
8a2f3cbdf794dfcdb31b9626bc0714bd

[env]

[malkia]

I was going to try this, but is there any more complete sample settings this up (just the .cpp file)?

[jblahos]

Hello, after further investigation, it seems that the issue only happens when opentelemetry-cpp is wrapped in a DLL. Once the DLL is being unloaded during program shutdown, this hang happens. Solution is to call shutdown on the tracer provider manually before that, so something like:

auto const provider_api = trace_api::Provider::GetTracerProvider();
auto const provider_sdk_raw = dynamic_cast<trace_sdk::TracerProvider*>(provider_api.get());
if (provider_sdk_raw != nullptr)
{
    provider_sdk_raw->ForceFlush(std::chrono::seconds(1));
    provider_sdk_raw->Shutdown();
}

I am closing the issue, since I found a solution that works for me.