[BUG] OS 3.1 not showing node level traces for DBQ query

[Deepti24]

Describe the bug

Setup: Created a local setup of Open Search 3.1 with jaegar to view traces. I am able to see http level traces but not detailed traces at node/shard level to debug slow DBQ queries. Configs used:

telemetry.feature.tracer.enabled - true
opensearch.experimental.feature.telemetry.enabled - true
telemetry.tracer.enabled - true
telemetry.tracer.sampler.probability - 1
telemetry.otel.tracer.span.exporter.class - io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter

Plugin: Open Telemetry plugin

Attaching screenshot for the same

Related component

Plugins

To Reproduce

1. Run OS version 3.1 in local and enable configs as mentioned above. Also start jaegar so that you are able to visualise traces.
2. Run DBQ query at URL: https://localhost:8443/test-index-dbq-tracing/_delete_by_query?wait_for_completion=true&requests_per_second=500&conflicts=proceed
3. Check jaegar UI with relevant trace id. We will not see scroll or node level traces. Just http level traces.

Expected behavior

We should see traces at node and scroll level.

Additional Details

Plugins
telemetry-otel

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):
Os version 3.1

Additional context
Add any other context about the problem here.

[Deepti24]

It is not working with Security plugin enabled. Without it, its working fine

[andrross]

Catch All Triage - 1

@cwperks Can you take a quick look here? Looks like maybe the security plugin is interfering with tracing somehow?

[Deepti24]

I have raised above PR for main branch as I was able to find this issue even in latest version.
I will want this fix to be back-ported to 3.1 version as well
Will raise request for that once this PR is reviewed

[cwperks]

@Deepti24 thanks for the deep dive and PR on this one. I'll get feedback on the PR asap.