ComposableRuleBasedSampler configuration

[anuraaga]

There is a new rule-based sampler in the spec

https://opentelemetry.io/docs/specs/otel/trace/sdk/#composablerulebased

Unlike every other sampler, this one needs special care in configuration since we can't "just model args as JSON" - one of the args is an arbitrary predicate function. So we need to define what types of predicates should be supported specifically by declarative configuration. For starters, I suggest the following

• attribute_value: takes an attribute key and value and performs an exact match
• attribute_pattern: takes an attribute key and regex pattern and performs a regex match
• span_kind: takes a span kind enum name and performs an exact match
• is_root: no parameters, matches if the span is a root span.

We can allow all of these to be defined at the same time to perform an AND composition (it should probably be an error to have both attribute specs though). If no predicate is specified, it is catch-all - in practice, most rule sets will almost always have this in the end because the default for a non-match is to drop.

The rule would contain the predicates, and sampler, with the latter being identical to the current Sampler schema.

Example

sampler:
  rule_based:
    rules:
      - attribute_value:
          key: http.route
          value: /health
        sampler:
          always_off: 
      - span_kind: CLIENT
        is_root: true
        sampler:
          always_off:
      - sampler: # The default sampler, parent-based always-on

Let me know how that sounds. /cc @jack-berg

[jack-berg]

I think we introduce something like this there's going to be an inevitable ask to support logic more complex than conditions and-ed together. If we add an abstraction where these same sets of rules can be or-ed together, I think it gets a lot more expressive. Here's a quick sketch on what that might look like in JSON schema:

        "ExperimentalComposableRuleBasedSampler": {
            "type": ["object"],
            "additionalProperties": false,
            "properties": {
                "rules": {
                    "type": "array",
                    "minItems": 1,
                    "items": {
                        "$ref": "#/$defs/ExperimentalComposableRuleBasedSamplerRule"
                    }
                }
            },
            "required": [
                "rules"
            ]
        },
        "ExperimentalComposableRuleBasedSamplerRule": {
            "type": ["object"],
            "additionalProperties": false,
            "properties": {
                "sampler": {
                    "$ref": "#/$defs/Sampler"
                },
                "predicate": {
                    "$ref": "#/$defs/ExperimentalComposableRuleBasedSamplerPredicate"
                }
            }
        },
        "ExperimentalComposableRuleBasedSamplerPredicate": {
            "type": ["object"],
            "additionalProperties": false,
            "properties": {
                "and": {
                    "type": "array",
                    "minItems": 1,
                    "items": {
                        "$ref": "#/$defs/ExperimentalComposableRuleBasedSamplerPredicate"
                    }
                },
                "or": {
                    "type": "array",
                    "minItems": 1,
                    "items": {
                        "$ref": "#/$defs/ExperimentalComposableRuleBasedSamplerPredicate"
                    }
                },
                "attribute_value": {
                    "$ref": "#/$defs/ExperimentalComposableRuleBasedSamplerKeyValue"
                },
                "attribute_pattern": {
                    "$ref": "#/$defs/ExperimentalComposableRuleBasedSamplerKeyValue"
                },
                "is_root": {
                    "type": "boolean"
                },
                "span_kind": {
                    "$ref": "#/$defs/ExperimentalComposableRuleBasedSamplerSpanKind"
                }

            },
            "minProperties": 1,
            "maxProperties": 1
        },
        "ExperimentalComposableRuleBasedSamplerKeyValue": {
            "type": ["object"],
            "additionalProperties": false,
            "properties": {
                "key": {
                    "type": "string"
                },
                "value": {
                    "type": ["number", "bool"]
                }
            },
            "required": [
                "key", "value"
            ]
        },
        "ExperimentalComposableRuleBasedSamplerSpanKind": {
            "type": ["string"],
            "enum": ["CLIENT", "SERVER", "INTERNAL", "PRODUCER", "CONSUMER"]
        }

And some examples:

rules:
  # use always_off when http.route = /health
  - predicate:
      attribute_value:
        key: http.route
        value: /health
    sampler:
      always_off:
  # use always_off when http.route = /health AND span.kind = CLIENT
  - predicate:
      and:
        - attribute_value:
            key: http.route
            value: /health
        - span.kind: CLIENT
    sampler:
      always_off:
  # use always_off when http.route = /health OR span.kind = CLIENT
  - predicate:
      or:
        - attribute_value:
            key: http.route
            value: /health
        - span.kind: CLIENT
    sampler:
      always_off:
  # use always_off when (http.route = /health OR http.route = /users) AND span.kind = CLIENT
  - predicate:
      and:
        or:
          - attribute_value:
              key: http.route
              value: /health
          - attribute_value:
              key: http.route
              value: /users
        span.kind: CLIENT
    sampler:
      always_off:
``

[anuraaga]

It's definitely quite expressive - but I wonder if supporting something like or: and: or: or: and: is allowing some pretty gnarly rules. That being said it does seem reasonable to support it too and let users go wild. Some points to confirm

• For top level or, is it fine to suggest just use multiple rules? e.g. for /health and CLIENT suppression example above, it's pretty easy to use two rules
• For the and: or: example above, since we are matching a single attribute key, attribute_pattern with an OR regex would also work. How often would we expect that not to work and require or
• and still feels more common, especially with top-level or being modelable by multiple rules. Would it make sense to still have and be implicit without the and: block? If so, would we still also allow explicit and: to allow choice?
• If and at the top-level makes sense anyways, we could add or later when there is significant demand?

Personally have a preference to not doing or, especially if the answer to the 4th question is yes, but I'm ok with following that suggestion as-is too, just want to see what you think about these points.

[jack-berg]

For top level or, is it fine to suggest just use multiple rules? e.g. for /health and CLIENT suppression example above, it's pretty easy to use two

Yup. Probably many cases that you might express using or: could be translated into separate top level rules if you're careful.

For the and: or: example above, since we are matching a single attribute key, attribute_pattern with an OR regex would also work. How often would we expect that not to work and require or

I forgot to mention, we're avoiding regex patterns in the config scheme and have guidance to use glob pattern matching instead. The reason being that one of the main goals of declarative config is interoperability across languages and languages have varying support for different regex features.

and still feels more common, especially with top-level or being modelable by multiple rules. Would it make sense to still have and be implicit without the and: block? If so, would we still also allow explicit and: to allow choice?
If and at the top-level makes sense anyways, we could add or later when there is significant demand?

Yeah I think and will be more common. Given that these are experimental types / properties, we do have the ability to make breaking changes. But that hurts users, so ideally we would get the first attempt close enough to the final stable version that we can get to where we want to be via non-breaking additions. Having the explicit and keyword is beneficial because it makes the semantics more clear for users, and forces the data model into a shape which is more evolvable to include more logical operators. But if we don't end up needing those additional logical operators, it looks out of place and verbose.

I think the main question here is how far do we want to take this? Even with my suggestion of adding or, there are lots of other logic features a user might want to have. Things like having support for inverting the boolean (i.e. a not function), starts with, case insensitive compare, null comparison, and many many things you can easily do with a proper programming language. We see this desire play out in things like the collector's transform processor and OTTL and the recent support for common expression language syntax in a sampler in opentelemetry-java-contrib.

What's the principle that helps us decide how far to go? If we go with something lightweight / simple / lacking power, users will be forced to go use programmatic config. If we go with something complex / expressive / powerful, we increase the implementation burden across every language. Most users won't need that complexity, but a few will. It would be very frustrating to buy into using declarative config and be happy, then one day try to do something more expressive and run into a wall and are forced to go back to programmatic config. My intuition is that it will be hard to justify saying no to future calls for more expressiveness. Not sure though - definitely speculating.

(P.S. as I'm writing this I'm realizing how much expressiveness is solved by having regex support. Giving it up stings particularly bad in this case. And thoughts / insights here?)

[anuraaga]

The thought of not supporting regex didn't even come to mind - but I do understand the motivation for wanting configs to be compatible across any language SDK which regex can have issues with in theory. In practice, I guess most patterns will be fairly simple though - perhaps the edge cases (both using very advanced regexes and building multi-language systems) are low enough to push some of the burden to users.

I used Gemini to research what the existing practices are in this realm, mostly looking at routers including Traefik and Envoy, as well as the collector's tailsampling processor. I was surprised that there actually weren't any cases of a declarative or construct outside the top-level list of rules - I figured there's gotta be at least one. But the only ones were the ones that supported it through a DSL like Traefik or tailsampling processor's OTTL support.

Besides that, a restricted OR is accomplished either using regex (envoy, prometheus), or making the fields a list (apisix). The latter is inspiring - do we get enough by making the field values all lists (I think yaml would convert a single value into a list automatically?) and not needing explicit or statements? It definitely doesn't accomplish every conceivable use case but maybe we get enough.

I don't know what the vision of the project is, but I would expect declarative configuration to not aim to handle every single case possible. But I do understand the desire for it - one personal anecdote is I've found the more expressive a config is (envoy config, k8s), the more tedious it is to do those basic things that the 90% of the users need, and it can make the whole experience a bit frustrating. So if supporting or:, I would still advocate for having the simple mode that allows omitting and: to make sure the basic case is still easy to use (and read, personally I have trouble grokking the 4th rule example that is using and: or: without squinting because yaml doesn't have braces)

[anuraaga]

Yikes I accidentally hit Close issue instead of the text box to continue my paragraph, and don't have permission to Reopen issue. Can you reopen it (also a surprising contributor experience /cc @trask).

To continue... Edit: edited the above comment instead

[jack-berg]

The latter is inspiring - do we get enough by making the field values all lists (I think yaml would convert a single value into a list automatically?) and not needing explicit or statements? It definitely doesn't accomplish every conceivable use case but maybe we get enough.

Coincidentally (or maybe not!), the list of values OR'd together is the pattern we've adopted in this repo as guidance for where pattern matching is needed: https://github.com/open-telemetry/opentelemetry-configuration/blob/main/CONTRIBUTING.md#properties-which-pattern-matching

Not sure if YAML converts a single value into a list. Can test. But even if not, we can write the schema to accept either a single value or a list of values. This makes it a little harder for implementations, but improves the UX for users using the single value case, which will be the majority of the time.

But I do understand the desire for it - one personal anecdote is I've found the more expressive a config is (envoy config, k8s), the more tedious it is to do those basic things that the 90% of the users need, and it can make the whole experience a bit frustrating.

Yeah its true. And because of this we see the emergence of higher order abstractions like helm for simplifying complex k8s configs. Some knowledgeable person / group curates a simplified / terse set of configuration options, and a templating engine spits out a much more verbose and detailed configuration spec. This type of topic has already come up in a number of places, with this being the most recent.

Given that this is experimental, let's see how far we can get with a simplified configuration interface like the one we've been discussing (something where conditions are and'ed together, and each condition can support a single value or list of values / patterns which are or'd together). If we are going to increase the implementation burden, its better for it to be justified by demand. And any feedback can be used to inform this how far we need to take it.