RabbitMQ AMQP and Differentiated services QOS

[btbhanders]

I have searched docs/threads and even code to find if there is native support for Differentiated services (https://en.wikipedia.org/wiki/Differentiated_services) in the RabbitMQ ecosystem but ended up empty.
Does RabbitMQ broker and client libraries support Differentiated services out of the box using AMQP 0.91?

Specifically I'm looking for sending low latency messages.
My application uses different types of messages of which a subset requires a low latency connection and would therefore benefit from Diffserv functionality on my network.
One possible solution would be to use a separate connection (and exchange/queue) for the low latency traffic, but this requires the broker to also respect the Diffserv flags on the consumer side, i.e. tag the messages correctly when sending to consumer.

If this is not supported, are there plans for incorporating this feature?

(Question originally asked in users forum, but seems to get no attention there.)

[lukebakken]

https://groups.google.com/g/rabbitmq-users/c/Jpz16jQGk94/

Please be patient and note that we try to respond to rabbitmq-users questions within a week.

No, RabbitMQ does not support "Differentiated Services" as described in that article, due to the IP QoS feature being at a lower network layer than what AMQP operates at.

[lukebakken]

What do you know, see this document: https://www.erlang.org/doc/man/inet.html

It looks like you can set priority and tos on systems that support it. But, these are options that would have to be set per-socket and there is no way to configure that in RabbitMQ at this point. How would RabbitMQ know what values to use for an incoming connection? Source IP / host name? There is no feature of that sort in RabbitMQ at this point and as far as I know it has never been requested.

I'm just guessing but you may be able to use some sort of TCP proxy between RabbitMQ and your client applications that has the ability to specify these values.

cc @michaelklishin because he'll find this interesting.

[michaelklishin]

A mapping of some kind could be added but we'd need to see both more demand and some numbers that demonstrate the benefits of those advanced networking settings.

@mkuratczyk and perhaps @illotum would find this even more interesting than I do :)

[illotum]

It is interesting! We already expose some ranch options in config, tos is one of them. Running a dedicated low-latency broker, with the corresponding setting in config, might be the path of least friction.

Also agree, the setting's impact will likely be noticeable only in congested (public) networks. I presume IoT is the main use case.

[btbhanders]

First of all, thank you all for the attention.

To make QoS for RabbitMQ work, the socket options must be set on each socket involved in the connection, from producer to consumer - see attached figure.
(The COS and DSCP markers are set on the egress packets from the socket.)

The question is how to incorporate these settings in a manageable and logical way.
I was thinking to do that on Exchange level, but the link to the consumer is via a queue which might be a bit vague - how to know that the connection to the queue needs this setting?
The other common entity is the routing key, but I don't know if it is possible or even suitable to attach socket properties to that.
Another option is to link this to port usage. One might perhaps set aside certain ports for low latency traffic?

@illotum, @michaelklishin
DSCP and TOS bits will not work over public internet in general. These bits are in most cases ignored since
otherwise everyone would prioritize their own traffic.
In my case I have a plant with several robots and other wired devices performing different tasks (on a private plant LAN).
The broker is used for plant wide supervision and control (SCADA) and some soft real-time control.
Some hosts are performing tasks that do not accept extended latency when it comes to messages.
This is where the COS and DSCP shines.
As you might know, the packet latency on IP networks usually follow the Erlang distribution, and the function of QoS reduces the tail (removes the worst delays).

The principle is to differentiate latency sensitive traffic from e.g. bulk transfer of files, or web traffic etc.
If a large file is in transfer, the IP stack might be saturated with this traffic.
The low latency message could then end up last in the queue and it could take in the order of several hundreds of milliseconds before the packet is even on the wire.
However, if a the packet is tagged with COS or DSCP, it would be submitted to the NIC buffers ahead of low prio packets.

Another aspect is that the switches and routers may start to drop packets when traffic bursts/high load occurs.
Prioritized packets are less likely to be dropped in such situations.

All in all this leads to reduced latency and less jitter which in some cases is important.
I have observed plant networks with 75-80% load but still performs perfect without jitter for prioritized traffic.

[illotum]

I assumed in a private network it's gonna be easier to avoid congestion altogether.

Anyway, both producers and consumers talk over the same protocol, so @lukebakken's snippet should work. As long as you have the client applications setting those bits, and switches configured with the policies, you'll get an end to end channel.

[lukebakken]

@btbhanders I followed up on @illotum's suggestion - "Running a dedicated low-latency broker, with the corresponding setting in config, might be the path of least friction"

You can test out priority and tos now if you are willing to run a dedicated RabbitMQ broker. Create the /etc/rabbitmq/advanced.config file with this content (replace INTEGER with the values you would like to use):

[
    {rabbit, [
        {tcp_listeners, [5672]},
	{tcp_listen_options, [
            % Note:
            % these are the default values
            {backlog, 128},
            {nodelay, true},
            {linger, {true, 0}},
            {exit_on_close, false},
            % https://github.com/rabbitmq/rabbitmq-server/discussions/6519
            % https://www.erlang.org/doc/man/inet.html
            {priority, INTEGER},
            {tos, INTEGER}
        ]}
    ]}
].

If you are using /etc/rabbitmq/rabbitmq.conf to set any TCP listen options, they will be ignored.

The above settings will be applied to every AMQP connection to port 5672. Of course your client applications will have to make the same settings, I assume.

Let me know if you also need to apply priority and tos to inter-node distributed Erlang communications (if you're running a cluster). There's probably a way to do that, too.

[btbhanders]

@lukebakken , @illotum using two brokers is actually a really neat solution. Thank you for this proposal!
This also opens up for running the low latency brokers on higher priority in the operating system.

For inter-node communication I do believe priority is needed as well and it would be nice to have these settings on that link as well.
Having said that, I know that in Windows it is possible to set network priority using a policy, but this is unfortunately not so fine grained.
It is possible to set priority for a certain exe file, but in this case the same exe file will be running two instances with different settings.
I know it possible to also filter on port number so I guess a combination of policies would resolve this.
I will do a test setup as you suggested to see this in action as soon as possible.

[lukebakken]

I took a few minutes to see how the sockets are set up for Erlang distribution. I'm pretty sure that if you were using TLS for inter-node distribution, you could add tos and priority to the set of options passed (https://www.erlang.org/doc/apps/ssl/ssl_distribution.html).

If you're not using TLS, there is no way to pass options via configuration. However, you could do a custom build and add the options to f_setopts_pre_nodeup and f_setopts_post_nodeup here:

https://github.com/erlang/otp/blob/master/lib/kernel/src/inet_tcp_dist.erl#L78-L104

I'm reasonably sure that's the correct place 😸

[lukebakken]

It is possible to set priority for a certain exe file, but in this case the same exe file will be running two instances with different settings

You should not run two instances of RabbitMQ on the same server / VM, if you want predictable performance from RabbitMQ.