How to combine OR in tail_sampling?

[gfan8w]

Hi,

I am working on setting up a tracing tool. The data flow would be - application -> otel-collector-> jaeger.
I want to log tracing data about

1. All tracing data : http.status_code is NOT 200, I refer this
2. All data that latency > 200ms,
3. Some proportion of tracing data that http.status_code IS 200 (still missing, I am still wondering how to write)

The collector config file collector.yaml is:

receivers:
    otlp:
        protocols:
            http:
                endpoint: 0.0.0.0:4318
            grpc:
                endpoint: 0.0.0.0:4317
processors:
    batch:
        timeout: 10s
    resource:
        attributes:
            - key: test.key
              value: "test-value"
              action: insert
#    probabilistic_sampler:
#        sampling_percentage: 1
    tail_sampling:
        policies:
          [
              {
                  name: always,
                  type: always_sample
              },
              {
                  name: error_http,
                  type: numeric_attribute,
                  numeric_attribute: { key: http.status_code, min_value: 300, max_value: 500 }
              },
              {
                  name: latency,
                  type: latency,
                  latency: { threshold_ms: 200 }
              },
          ]

exporters:
    logging:
        loglevel: debug
    jaeger:
        endpoint: jaeger:14250
        tls:
            insecure: true
extensions:
    health_check:
    pprof:
        endpoint: :1888
    zpages:
        endpoint: :55679
service:
    extensions: [pprof, zpages, health_check]
    pipelines:
        traces:
            receivers: [otlp]
            processors: [batch, resource, tail_sampling]
            exporters: [logging, jaeger]

[jpkrohling]

What are you seeing as the output of this policy, and what do you expect to see instead?

[gfan8w]

I modified the policy. Now it seems work as expected: all error and latency>200 are always logged, only some proportion of normal http requests are logged.

policies:
          [
#              {
#                  name: always,
#                  type: always_sample
#              },
              {
                  name: error_http,
                  type: numeric_attribute,
                  numeric_attribute: { key: http.status_code, min_value: 300, max_value: 500 }
              },
              {
                  name: latency,
                  type: latency,
                  latency: { threshold_ms: 200 }
              },
              {
                  name: ok_http_record_partial,
                  type: and,
                  and: {
                      and_sub_policy:
                        [
                            {
                                name: ok_http,
                                type: numeric_attribute,
                                numeric_attribute: { key: http.status_code, min_value: 200, max_value: 299 }
                            },
                            {
                                name: probabilistic-policy,
                                type: probabilistic,
                                probabilistic: { sampling_percentage: 10 }
                            },
                        ]
                  }
              },
          ]

[edenkoveshi]

I have some questions about this too
How exactly do the policies work? Are the spans compared to each policy until it finds a matching one and stops?
Second, if I use this to sample spans, will it not break the span-trace logic? If a span is kept, will all of the spans belong to the same trace be kept?
And is it possible to use resource attributes?
Also, I want to do something like that:
Suppose I have some very important services (or attributes, such as error) for which I want to keep 100% of the spans
Than I have some other "special" services for which I want to keep 50% or 5% of the spans
And then as a default keep 25% of the spans
How is this achievable?

[jpkrohling]

I opened a PR (#16544) with some clarifications. Could you please look and see if it answers some of your questions? Here are the answer to the other questions:

if I use this to sample spans, will it not break the span-trace logic?

Spans are not sampled, traces are. Decisions may be based on spans or the trace as a whole.

How is this achievable?

Take a look at the composite policy.

[edenkoveshi]

I am still struggling a bit actually.
What is the behaviour of the composition policy? If a trace matches any of the policies it is kept? Or is it forwarded to the next policy like a pipeline?
And what is the behaviour of simply writing them as an array?

[jpkrohling]

The composite policy follows the same decision framework as the processor itself. From the readme:

Each policy will result in a decision, and the processor will evaluate them to make a final decision:

• When there's an "inverted not sample" decision, the trace is not sampled;
• When there's a "sample" decision, the trace is sampled;
• When there's a "inverted sample" decision and no "not sample" decisions, the trace is sampled;
• In all other cases, the trace is NOT sampled