Plan on more advanced sampler or sampler configuration?

[stnor]

Context

I'm currently using traceidratio sampling and a 1/200 ratio, and we have a Spring websocket endpoint that has a high invocation frequency.

Problem

Sampling rates are defined on a per-JVM level. If a JVM has two endpoints with vastly different throughputs, then its sampling rate will be driven on the high frequency endpoint, which may leave the low frequency endpoint never sampled. For example, if the frequency of the endpoints is different by a factor of 100, and the probability is set to 0.001, then the low frequency endpoints will have only 1 in 100,000 chance to be sampled.

Features I'd love to see in a sampler

1. Assigning sampling probabilities on a service + endpoint basis rather than just the service
2. Using a lower bound rate limiter to ensure that all endpoints are sampled with a certain minimal rate
3. Observing the impact of sampling rates on the overall number of traces sampled from a service and dynamically adjusting the per-endpoint sampling rates to meet certain target rates.

Perhaps allow more detailed configuration passed into the sampler argument to solve (1)
Perhaps a file ref to a JSON/YAML config with a default sample rate and the ability to override on an endpoint basis?

Something along the lines of this:

{ rates: 
 { default: 0.005,
   http: [
      uri: '/api/login/google': 0.5,
      uri: '/api/login/*': 0.2,
      uri: '/api/*': 0.1
   ],
   websocket: 0.0001
} 

This would totally solve my use-case. Wild-cards would be nice. I'm using Spring btw.

[jkwatson]

Of course, you are 100% free to implement any kind of sampler that is useful to you. And, new kinds of samplers would be a great contribution to the opentelemetry-java-contrib project. Any sampler that we'd add to the default SDK and the built-in configuration mechanisms would need to go through the specification process, so that it was done in the same way across all the languages.

[stnor]

It looks like io.opentelemetry.sdk.extension.trace.jaeger.sampler.PerOperationSampler and io.opentelemetry.sdk.extension.trace.jaeger.sampler.RateLimitingSampler does what I need pretty much.

https://www.jaegertracing.io/docs/1.21/sampling/#client-sampling-configuration

[stnor]

Sorry to bother again.

How do I plugin my own SamplerProvider? Do I need to roll my own javaagent jar? I haven't worked with javaagents before, and when just putting the classes in my application, the TracerProviderConfiguration doesn't pick up my classes.

[jkwatson]

Great question for the instrumentation repository, but right now, the only way to do it is to roll your own javaagent jar. But, the next release should include a new extension/plugin system that will let you do this much more easily. Follow this PR for details: open-telemetry/opentelemetry-java-instrumentation#2881