Default CongestionControl policy (Drop) cannot deliver all message sizes.

[ciprianf]

Describe the bug

Context: forked from the discussion on Discord https://discord.com/channels/914168414178779197/940584045287460885/1436370380628426936.

The CongestionControl policy Drop is dropping large messages, even if there is no network congestion. Sending a single 5MB message over a WiFi connection (200 Mbits up/down) always results in the message being dropped.

There are some configuration knobs that can be tweaked to increase internal timeouts (wait_before_drop, max_wait_before_drop_fragments), but one does not fundamentally solve the issue for an arbitrary sized message.

Observations:

• the Drop policy should not drop a large messages midway just because it is a large message and some internal timeout expires - it should try to deliver it and only drop new arriving messages.
• the Drop policy is blocking the calling thread, arguably with a timeout (max_wait_before_drop_fragments). This is very misleading. Why isn't the Drop policy just store the message in an internal queue for async dispatching, and release the calling thread?
• the default value for max_wait_before_drop_fragments is 50 milliseconds. For successful delivery of a 5MB message, the necessary network bandwidth should be at least 800Mbits. This seems excessive. Is this the best default value?

Taking a step back, the overall feedback is that we should aim for Zenoh to work out of the box for standard cases. The configuration knobs should be reserved solely for optimizing a traffic pattern.

To reproduce

Start two nodes in peer mode, connected via WiFi. Send a message of 5-10MB. The subscriber never receives any message.

System info

Zenoh Cpp 1.5.0
Ubuntu 24.04

[OlivierHecart]

The CongestionControl::BlockFirst policy has been introduced to mitigate this problem. It works roughly as you describe with a 1 msg length queue. Please use this policy if you want to make sure that at least some of the messages go through.

Zenoh has been designed to prioritized high throughput, low latency and high scalability.
For those reasons the Zenoh internal queues are bounded in size and not number of messages and the default values are quite low as memory consumption quickly rises in scalability scenarios.
For those same reasons we cannot afford to wait very long as the thread may have other messages to read/publish (it is the writing thread that directly serializes the message in bounded size buffers for higher throughput). 50 milliseconds is already quite large in some scenarios.

There is no single default configuration that works for all scenarios. Anyway the kind of scenario you describe is indeed quite common and already bite multiple users (thus the CongestionControl::BlockFirst policy). We will try to improve the out of the box experience regarding those scenarios.