How to setup to consider probability instead of sample flag in traceparent header for web requests

[abalabolov]

Issue: Overwhelm an application with tracing overhead
https://www.w3.org/TR/trace-context-1/#denial-of-service

Env: Spring Boot 3.2.3, micrometer-tracing-bridge-otel + opentelemetry-exporter-zipkin
System: We have desktop, web client and monolitic server (for now) behind AWS ALB with autoscaling.

What I would like to achive:

1. The possibility to set trace and span id from any Web client and take them into account on the BE side for other internal purposes.
2. Make a final decision about Sampling on the BE side taking into account specified probability on the BE side and SampleFlag header from the Web client side. I.e. to follow this https://www.w3.org/TR/trace-context-1/#trace-flags

I set tracing.sampling.probability: 0.1 and added Aspect (for testing purpose) that wraps several packages with the following code:

Span parentSpan = tracer.currentSpan();
Span newSpan = tracer.nextSpan(parentSpan)
	.name(SpanNameUtil.toLowerHyphen(methodName))
	.tag("class.name", className)
	.tag("input.params", Arrays.toString(parameters));

Object result;

try (Tracer.SpanInScope ws = this.tracer.withSpan(newSpan.start())) {
	result = joinPoint.proceed();
	
	if (nonNull(result)) {
		newSpan.tag("return.value", result.toString());
	}
}
finally {
	newSpan.end();
}

During startup I see that OTel creates TraceIdRatioBasedSampler and wraps it in ParentBasedSampler

When I specify traceparent header with sample flag = 01, I face with the following:

1. All web requests are catched in PropagatingReceiverTracingObservationHandler.onStart() that uses parentTraceContext with PropagatedSpanwithtraceFlags=01`.
2. Then SpanBuilder.startSpan() has tracerSharedState with ParentBasedSampler and calls shouldSample() on behalf of ParentBasedSampler that checks traceFlags from PropagatedSpan. So, it is more SampleFlagBased rather parentBased
3. PropagatedSpan.traceFlags = 01 so the decision is RECORD_AND_SAMPLE every time
4. If I specify traceparent header with sample flag = 00, then the result is DROP every time.
5. In any cases above, the probability doesn't make an effect. Even I set it to 0 or 0.0, all spans still appear in Zipkin

I think this behavior was implemented for comminication between isolated microservices.

So, my questions:

1. How to prevent against attack from Web by calling our server with SampleFlag=01
2. How to make probability work with default out of the box settings
3. How to properly configure tracing to consider probability instead of SampledFlag
4. In case custom implementations, How to keep the possibility to support default behaviour for microservices.

I am loking for the right way to resolve them and think about one of the following:

1. [Selected by me for now] Create Root OTel Sampler with only traceIdRatioBased sampler. Trusting the following convention: if Span is considered to be Sampled, other calls to check shouldSample with the same params will return the same result. I have checked, this works as I described above.

@Bean
Sampler otelSampler() {
    return Sampler.traceIdRatioBased(0.10f);
}

2. Replace fields in ParentBasedSampler: remoteParentSampled(Sampler), localParentSampled(Sampler) to TraceIdRatioBasedSampler. Based on the "running the sampler on any child Span will produce the same decision"
3. Create own Sampler
4. Implement OncePerRequest filter and replace SampleFlag 01 to 00 if request is not from some "White list" that is loaded at startup to the memory.
5. Implement custom W3CTraceContextPropagator and the same propagators: check if the header is allowed from the caller see option 4).

There is a topic that has the similar question but the answer is absent.
open-telemetry/opentelemetry-dotnet#3594

[trask]

hi @abalabolov! I'd recommend asking in the micrometer-tracing-bridge-otel project

[abalabolov]

Created micrometer-metrics/tracing#626

I didn't find micrometer-tracing-bridge-otel project