Stack smashing crash in batch span processing

[dluongiop]

Describe your environment

Rocky Linux 9
version of otel-cpp = 1.9.1

cmake3 .. -DBUILD_TESTING=OFF -DWITH_EXAMPLES=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
 -DBUILD_SHARED_LIBS=ON -DWITH_OTLP=ON -DWITH_OTLP_HTTP=ON -DWITH_PROMETHEUS=ON -DBUILD_PACKAGE=ON

Backtrace:

#0  0x00007ff97bd1294c in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007ff97bcc5646 in raise () from /lib64/libc.so.6
#2  0x00007ff97bcaf7f3 in abort () from /lib64/libc.so.6
#3  0x00007ff97bcb0130 in __libc_message.cold () from /lib64/libc.so.6
#4  0x00007ff97bda477b in __fortify_fail () from /lib64/libc.so.6
#5  0x00007ff97bda4756 in __stack_chk_fail () from /lib64/libc.so.6
#6  0x00007ff97a04accd in opentelemetry::proto::trace::v1::ResourceSpans::~ResourceSpans() () from /lib64/libopentelemetry_proto.so
#7  0x00007ff97a04aced in opentelemetry::proto::trace::v1::ResourceSpans::~ResourceSpans() () from /lib64/libopentelemetry_proto.so
#8 0x00007ff97a0537dd in google::protobuf::RepeatedPtrField<opentelemetry::proto::trace::v1::ResourceSpans>::~RepeatedPtrField() () from /lib64/libopentelemetry_proto.so
#9 0x00007ff97a83b6a7 in opentelemetry::v1::exporter::otlp::OtlpHttpExporter::Export(opentelemetry::v1::nostd::span<std::unique_ptr<opentelemetry::v1::sdk::trace::Recordable, std::default_delete<opentelemetry::v1::sdk::trace::Recordable> >, 18446744073709551615ul> const&) () from /lib64/libopentelemetry_exporter_otlp_http.so
#10 0x00007ff97a89de3d in opentelemetry::v1::sdk::trace::BatchSpanProcessor::Export() () from /lib64/libopentelemetry_trace.so
#11 0x00007ff97a89cce9 in opentelemetry::v1::sdk::trace::BatchSpanProcessor::DoBackgroundWork() () from /lib64/libopentelemetry_trace.so
#12 0x00007ff97bf88ad4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#13 0x00007ff97bd10c02 in start_thread () from /lib64/libc.so.6
#14 0x00007ff97bd95c40 in clone3 () from /lib64/libc.so.6

Additional context
The error is only happening with otel tracing enabled software. I've look for similar bug reports but didn't see anything that's relevant. However, I suspect there might be an issue with synchronous span processing that we're currently using.

[dluongiop]

Update:
I since found this bug report: #2191
Would the stack smashing be fixed by this?

In the meantime, we've upgraded to 1.16.1 so that included the fix above.

[marcalff]

Please double check how/if span attributes are set on a span, and if/how resource attributes are set on a resource.

When setting a string attribute in particular, a common error is to pass a pointer to a string that is later de allocated, for example passing a pointer to a local variable, or using a string_view on something that goes out of scope.

By the time the opentelemetry-cpp code needs to read the string value, it can read garbage instead, causing corruption.

Tools like valgrind are very useful to detect the root cause for errors like this.

[dluongiop]

We don't have anything like passage of pointer to a string which is out of scope.

We've been testing with 1.16.1 and so far no crash. We did manage to recreate the crash with 1.9.1 very consistently with the following technique:

Point otlp http client to an endpoint: http//localhost:some_port. When telemetry is received, don't respond and let request timeout. Spans must be generated pretty frequently for this to happen.

I think this issue can be closed.