[SpringBoot Starter] Ability to exclude specific KafkaListener from traces

[yddimitrov]

Is your feature request related to a problem? Please describe.

In our applications we have some Kafka Listeners that run on Application start and bootstrap caches by reading topics from the beginning.
This causes enormous spike in unnecessary traces during application boot. The Listeners are stopped after initial bootstrap so they do not provide any meaning after the boot is completed anyway.

Describe the solution you'd like

We want to have the ability to exclude specific Kafka Listeners or specific classes by Spring Configuration and/or a specific annotation.

Proposed solutions:

• Ideally we want to have the ability to exclude classes and/or methods from producing traces.
  E.g.: otel.exclude-classes="my.package.MyClass,my.package2.*"
  E.g 2: otel.instrumentation.kafka.exclude-classes="my.package.MyClass,my.package2.*"

• Have the ability to annotate methods and/or classes that should be ignored by OTEL exporters
  E.g: @WithoutSpan or @IgnoreSpan

Or any alternative that does not include JavaAgent and is compatible with the SpringBoot Starter for OTEL.

Describe alternatives you've considered

We considered migrating over to JavaAgent but that is not an ideal solution ATM.

Additional context

No response

[trask]

hi @yddimitrov, have you already looked at registering a custom sampler to filter them out?

[yddimitrov]

hi @yddimitrov, have you already looked at registering a custom sampler to filter them out?

I was thinking about this as a workaround. Is there any documentation about samplers? I'm unable to find it.

Thank you!

[trask]

check out https://opentelemetry.io/docs/languages/java/sdk/#sampler

[yddimitrov]

check out https://opentelemetry.io/docs/languages/java/sdk/#sampler

I am registering a custom sampler that excludes classes from tracing based on a pre-refined list but the shouldSample method is not invoked.

I have a SpringBoot application with Java21 and SpringBoot 3.3.4 and I tried registering a bean via configuration or directly in the Main() method of the application in GlobalOpenTelemetry object and in both cases the custom sampler is registered and added but when tracing starts the shouldSample method is not invoked.

Custom Sampler class:

@Slf4j
public class CustomSampler implements Sampler {
    private final Sampler delegateSampler;
    private final Set<String> excludedClasses;

    public CustomSampler(Sampler delegateSampler, Set<String> excludedClasses) {
        this.delegateSampler = delegateSampler;
        this.excludedClasses = excludedClasses;
    }

    @Override
    public SamplingResult shouldSample(Context parentContext, String traceId, String name, SpanKind spanKind, Attributes attributes, List<LinkData> parentLinks) {
        if (excludedClasses.contains(name)) {
            log.info("Dropping class {} from sampling with traceId: {}", name, traceId);
            return SamplingResult.drop();
        }
        log.info("Sampling class: {} with traceId: {}", name, traceId);
        return delegateSampler.shouldSample(parentContext, traceId, name, spanKind, attributes, parentLinks);
    }

    @Override
    public String getDescription() {
        return "CustomClassExclusionSampler";
    }
}

Sampler Configuration:

@Configuration
public class OpenTelemetryConfig {
    
    @Bean
    public SdkTracerProvider sdkTracerProvider() {
        Set<String> excludedClasses = Set.of("com.example.ExcludedClass");
        Sampler customSampler = new CustomSampler(Sampler.alwaysOn(), excludedClasses);

        var sdkTracerProvider =  SdkTracerProvider.builder()
                .setSampler(customSampler)
                .build();

        GlobalOpenTelemetry.set(OpenTelemetrySdk.builder()
                .setTracerProvider(sdkTracerProvider)
                .build());

        return sdkTracerProvider;
    }
}

[trask]

Since you're using the OTel Spring Boot Starter, I think you can register your custom sampler by exposing it as a bean

cc @jeanbisutti @zeitlinger

[jeanbisutti]

See https://opentelemetry.io/docs/zero-code/java/spring-boot-starter/sdk-configuration/#exclude-actuator-endpoints-from-tracing

[yddimitrov]

See https://opentelemetry.io/docs/zero-code/java/spring-boot-starter/sdk-configuration/#exclude-actuator-endpoints-from-tracing

Thank you @trask and @jeanbisutti for the hints.
I managed to register a custom sampler successfully by the example:

 @Bean
    public AutoConfigurationCustomizerProvider otelCustomizer() {
        return p -> p.addSamplerCustomizer((fallback, config) -> new CustomSamplerCustomizer(fallback));
    }

In the custom sampler I am excluding (dropping) samples from specific consumer groups based on the consumer group attribute. Not very elegant solution but a working one and I will close this issue.