Service level downsampling and rate limit in jaeger-collector

[jianwu]

Seems current, the collector side only supports a global level down sampling. Is it possible to do service level down sampling configuration? This is to protect backend storage from misbehaved clients which sent a large number of traces causing logs of span drops.

Also ideally we should also be able to config a service level rate limit in addition to percentage-based downsampling to further protect backend storage.

[yurishkuro]

Applying downsampling at service level means you end up with broken traces. We used to do that at Uber in the earlier versions of Jaeger, as a last ditch overload protection, but subsequently moved onto adaptive sampling that can achieve the same results by varying sampling rates in the entry services while keeping the sampled traces intact.

[jianwu]

Got it. I thought service is at trace level instead of span level.

Just a thought, Is it possible to propagate root service to the child spans as a tag (such as rootService. so that we can based on the rootService do a consistent rootService sampling decision in the collector. I feel collect should have a reliable way to be protected from abnormal clients.

The problem with adaptive sampling is we have to trust all the clients to enable the remote config. In our case, our client is across many different languages and stacks. So it's hard to standardize it.

[jkowall]

You can also do smarter combinations of tailbased sampling allowing for more granular sampling decisions on the result of the trace. You can also easily use the filter processor to filter out anything you don't specifically want. Sampling on result code or performance is really handy.