Support Sampling for Long-Duration Span and Erroneous Span

[pepeshore]

Is your feature request related to a problem? Please describe.

100% sampling rate is not suitable for production environments because of high overhead,but non-100% sampling may lead to the loss of certain important spans,such as long-Duration span and erroneous span. I try to provide a custom io.opentelemetry.sdk.trace.SpanProcessor which will hold the spans associated with the same traceId util LocalRootSpan is ended. But I found that when isRecording(samplingDecision) is false, a io.opentelemetry.api.trace.PropagatedSpan is returned instead of io.opentelemetry.sdk.trace.SdkSpan which lost a lot of important information

Describe the solution you'd like

Two solution

1. Return a SDKSpan even if isRecording(samplingDecision) is false, by this, I can give a custom io.opentelemetry.sdk.trace.SpanProcessor to implement Sampling for Long-Duration Span and Erroneous Span
2. if sampling for long-duration span、erroneous span or other span with specified feature is a common need，can the agent give a native support

Describe alternatives you've considered

No response

Additional context

No response

[trask]

hi @pepeshore! have you seen the Collector Tail-Based Sampler?

[pepeshore]

hi @pepeshore! have you seen the Collector Tail-Based Sampler?
yes，it's really a helpful approach, tks

[pepeshore]

hen the collector is working in a separate cluster, the number of agent reports is too large ? Can the agent implement exception samplin
See Tail-Based Sampler in SpanProcessor， But trace that cross process boundaries will be lost

[trask]

hi @pepeshore, here's a sketch of how you could implement this today:

/**
 * A sampler that delegates to another sampler but converts RECORD_AND_SAMPLE 
 * decisions to RECORD_ONLY. This allows spans to be recorded locally with full
 * data, but defers the filter decision to a SpanProcessor based on runtime
 * characteristics (duration, error status).
 */
public class DeferredDecisionSampler implements Sampler {
    
    private final Sampler delegateSampler;
    
    public DeferredDecisionSampler(Sampler delegateSampler) {
        this.delegateSampler = delegateSampler;
    }
    
    @Override
    public SamplingResult shouldSample(
            Context parentContext,
            String traceId,
            String name,
            SpanKind spanKind,
            Attributes attributes,
            List<LinkData> parentLinks) {
        
        // Get the decision from the delegate sampler
        SamplingResult delegateResult = delegateSampler.shouldSample(
            parentContext, traceId, name, spanKind, attributes, parentLinks);
        
        SamplingDecision decision = delegateResult.getDecision();
        
        // Convert DROP to RECORD_ONLY
        // This creates a full SdkSpan that can be conditionally filtered for export later
        if (decision == SamplingDecision.DROP) {
            return SamplingResult.create(
                SamplingDecision.RECORD_ONLY, 
                delegateResult.getAttributes());
        }
        
        // Pass through other decisions (DROP, RECORD_ONLY)
        return delegateResult;
    }
}

/**
 * A SpanProcessor that conditionally filters spans based on runtime characteristics:
 * - Spans that exceed a duration threshold
 * - Spans with error status
 * - Spans that were already sampled (RECORD_AND_SAMPLE)
 * 
 * This processor wraps a BatchSpanProcessor and only delegates spans that pass the filter.
 */
public class FilteringSpanProcessor implements SpanProcessor {
    
    private final SpanProcessor delegate;
    private final long durationThresholdNanos;
    
    public FilteringSpanProcessor(SpanProcessor delegate, Duration durationThreshold) {
        this.delegate = delegate;
        this.durationThresholdNanos = durationThreshold.toNanos();
    }
    
    @Override
    public void onStart(Context parentContext, ReadWriteSpan span) {
        delegate.onStart(parentContext, span);
    }
    
    @Override
    public boolean isStartRequired() {
        return delegate.isStartRequired();
    }
    
    @Override
    public void onEnd(ReadableSpan span) {
        if (span == null) {
            return;
        }
        
        // Only pass to delegate (for export) if filter conditions are met
        if (shouldExport(span)) {
            delegate.onEnd(span);
        }
    }
    
    private boolean shouldExport(ReadableSpan span) {
        // Always export sampled spans (RECORD_AND_SAMPLE)
        if (span.getSpanContext().isSampled()) {
            return true;
        }
        
        // For RECORD_ONLY spans, check additional conditions
        
        // Check duration threshold using getLatencyNanos()
        if (span.getLatencyNanos() > durationThresholdNanos) {
            return true;
        }
        
        // Check for errors (requires toSpanData())
        SpanData spanData = span.toSpanData();
        if (spanData.getStatus().getStatusCode() == StatusCode.ERROR) {
            return true;
        }
        
        return false;
    }
    
    @Override
    public boolean isEndRequired() {
        return true;
    }
    
    @Override
    public CompletableResultCode shutdown() {
        return delegate.shutdown();
    }
    
    @Override
    public CompletableResultCode forceFlush() {
        return delegate.forceFlush();
    }
}

/**
 * SPI implementation that automatically registers the deferred decision sampler
 * and filtering span processor when OpenTelemetry is autoconfigured.
 * 
 * This works with both:
 * - AutoConfiguredOpenTelemetrySdk in application code
 * - OpenTelemetry Java Agent
 */
public class DeferredSamplingCustomizer implements AutoConfigurationCustomizerProvider {
    
    @Override
    public void customize(AutoConfigurationCustomizer autoConfiguration) {
        autoConfiguration.addSamplerCustomizer((sampler, configProperties) -> {
            return new DeferredDecisionSampler(sampler);
        });
        
        // Wrap BatchSpanProcessor with our filtering logic
        autoConfiguration.addSpanProcessorCustomizer((spanProcessor, configProperties) -> {
            if (spanProcessor instanceof BatchSpanProcessor) {
                long durationThresholdMillis = 
                    configProperties.getLong("otel.traces.deferred.duration.threshold.ms", 5000L);
                Duration durationThreshold = Duration.ofMillis(durationThresholdMillis);
                return new FilteringSpanProcessor(spanProcessor, durationThreshold);
            }
            return spanProcessor;
        });
    }
}

[github-actions]

This issue has been labeled as stale due to lack of activity and needing author feedback. It will be automatically closed if there is no further activity over the next 7 days.