segfault when creating an active span in an exit handler

[mwaw-osl]

Using the default RuntimeContextStorage (ThreadLocalContextStorage) causes a segfault when a span is made active in an exit handler registered with atexit(). This is due to thread_local storage being cleaned up before exit handlers are called, destroying the context storage Stack and causing the next Context push (in the exit handler) to segfault.

A lightly modified version of otel's simple example demonstrates this:

#include <iostream>

#include "opentelemetry/exporters/ostream/span_exporter_factory.h"
#include "opentelemetry/sdk/trace/exporter.h"
#include "opentelemetry/sdk/trace/processor.h"
#include "opentelemetry/sdk/trace/provider.h"
#include "opentelemetry/sdk/trace/simple_processor_factory.h"
#include "opentelemetry/sdk/trace/tracer_provider.h"
#include "opentelemetry/sdk/trace/tracer_provider_factory.h"
#include "opentelemetry/trace/tracer_provider.h"
#include "opentelemetry/trace/provider.h"

namespace trace_api      = opentelemetry::trace;
namespace trace_sdk      = opentelemetry::sdk::trace;
namespace trace_exporter = opentelemetry::exporter::trace;

namespace
{
void InitTracer()
{
  auto exporter  = trace_exporter::OStreamSpanExporterFactory::Create();
  auto processor = trace_sdk::SimpleSpanProcessorFactory::Create(std::move(exporter));

  std::shared_ptr<opentelemetry::sdk::trace::TracerProvider> sdk_provider =
      trace_sdk::TracerProviderFactory::Create(std::move(processor));

  const std::shared_ptr<opentelemetry::trace::TracerProvider> &api_provider = sdk_provider;
  trace_sdk::Provider::SetTracerProvider(api_provider);
}

void CleanupTracer()
{
  std::shared_ptr<opentelemetry::trace::TracerProvider> noop;
  trace_sdk::Provider::SetTracerProvider(noop);
}

void tracedFunction() {
auto tracer = opentelemetry::trace::Provider::GetTracerProvider()->GetTracer("simpleTest");
auto span = tracer->StartSpan("testSpan");
auto scope = tracer->WithActiveSpan(span);
}

void exitHandler() {
    std::cout << "Exit handler" << std::endl;
    tracedFunction();  // THIS WILL CAUSE A SEGFAULT
    CleanupTracer();
}
}  // namespace

int main()
{
  InitTracer();

  tracedFunction();
  atexit(exitHandler);
  tracedFunction();
}

This is seen using otel 1.19.0 with g++ 13.3.0. Is opentelemetry expected to crash a program if an exit handler is registered that happens to do something that activates a span? One work around we're considering (to avoid the segfault) is using a sentinel thread_local bool that is set when the Stack is destructed to signal the end of thread_local storage (and then either create ad hoc storage or stop trying to interact with the Stack; it's not clear we'll be able to collect any meaningful trace information at this point anyways).

[owent]

We can't create any object of otel-cpp when th program is exiting. Because the singleton instances may be destroyed. We will keep the destroying order of components in otel-cpp, but we can't ensure other objects to destroy before otel-cpp.

[lalitb]

closed as mentioned by @owent above.

[mwaw-osl]

So the answer to my question is that crashing is the expected behaviour for trying to access the default RuntimeContextStorage in an exit handler?

[ThomsonTan]

I think the SEGFAULT is expected when calling our tracing API during exiting, because we rely some thread local storage to manage context which could have been destructed, then we could just access the invalid/freed object like context::RuntimeContext::GetCurrent().

In the specific example, it is WithActiveSpan which relies on Context but not available during exiting time.

[mwaw-osl]

Thanks for the clarification Tom!

In that case we'll continue to test our work-around where we include a sentinel that is set in the destructor of the thread_local Stack object created by the ThreadLocalContextStorage to signal that the stack is destroyed even if the thread is still alive (which I think can only happen in exit handlers, where the compiler might clean up thread_local storage before calling the handlers).