How to send messages with multiple filter values for 3.13 feature server side filtering

[aeb-dev]

We are currently using topic exchange with the following routing key format: <key1>.<key2>.<key3>

And consumers can consume like *.<key2>.* | <key1>.*.* etc.

Soon amqp protocol will not be performant enough for us and also we need the replay feature of streams. So I was considering streams with server side filter but I could not find a way to publish with multiple filter values. Is there anyway?

[acogoluegnes]

It is not possible to associate a stream message with several filter values, only one is allowed. It is possible to create a stream consumer with several filter values though. Wildcards are not supported. Note that if you specified several values for a consumer to match e.g. the *.key2.key3 pattern, the order of keys would not be enforced.

Would you mind telling us more about your use case?

[acogoluegnes]

For your use case it's likely to increase the cardinality of filter values, which will make the internal bloom filter less efficient. See https://rabbitmq.github.io/rabbitmq-stream-java-client/snapshot/htmlsingle/#considerations-on-filtering.

[michaelklishin]

@aeb-dev think of it this way: not every feature is worth the potential overhead and complexity. We are starting small intentionally, and will then see how the minimalistic stream filtering feature is received.

[aeb-dev]

I see, I was expecting a technical reason but sure I understand, it is not up to me to decide

[kjnilsson]

@aeb-dev there is a technical reason for this as well. If we allow multiple filter values per messages it becomes very easy to saturate the bloom filter such that the false positivity rate becomes too high which effectively is the same as having no filtering at all. Not allowing multiple filter values means we steer usage towards it's optimal point.

As @acogoluegnes suggested you could use a coarse grained filter value to reduce the amount of data sent to the client then rely on client side logic for further filtering. You will always need client side filtering anyway as explained in the docs.

[aeb-dev]

there is a technical reason for this as well. If we allow multiple filter values per messages it becomes very easy to saturate the bloom filter such that the false positivity rate becomes too high which effectively is the same as having no filtering at all. Not allowing multiple filter values means we steer usage towards it's optimal point.

I was suspecting that as well, thank you.

Not to insist but to share an idea, does using multiple bloom filters for each key group help here?