Bug: spmc_broadcast::Sender::send with NoSpaceLeft error may be wrong

[PiotrKorkus]

Original report: https://github.com/qorix-group/inc_orchestrator_internal/issues/280

Describe the bug:

spmc_broadcast::channel::sender::send() behavior for the CommonErrors::NoSpaceLeft is potentially wrong.

Current send() documentation:

Sends a value to all connected Receivers.
Returns
...
Err(CommonErrors::NoSpaceLeft) - When no more space is available in the channel was accounted at least on single receiver.

In the current implementation when one of the receivers read queue is full, then message is send to all other receivers that have empty slots for the data and the sender returns NoSpaceLeft error. However, lagging receiver is unaware that it is missing some messages.

Tokio handles it in a way that sending will overrider the latest unread message, but the receiver will get RecvError::Lagged error code for the next read.

Few possible solutions:
a) Leave it as is, but clearly specify that receiver may not receive all the values without notification (IMO bad)
b) Add receiver read return code to inform receiver, that it is missing some messages (like in tokio)
c) Return NoSpaceLeft error and do not sent any values until there is empty slot in the channel (current implementation in the test)

The test creates 3 receivers where one of them won't read any values. Sender sends channel size messages and after that resend them.

Steps to reproduce the behavior:

1. git checkout igorostrowskiq_spmc_broadcast_channel_tests (will be replaced with main once tests are merged)
2. cd component_integration_tests/rust_test_scenarios
3. cargo run -- --name runtime.channel.spmc_broadcast_send_receive_one_lagging --input '{"runtime": {"task_queue_size": 256, "workers": 4}, "test": {"data_to_send": [11, 12, 13, 14], "receivers": ["receiver_1", "receiver_2", "receiver_3"], "max_receiver_count": 3}}' | grep INFO

Observed behavior:

Message is delivered to receivers with empty slots, sender returns NoSpaceLeft error, lagging receiver has no possibility to know that it missed some messages.

Expected behavior

To be specified. Right now test is expecting that after channel is filled, for NoSpaceLeft error none of the receivers will get a message until there is an empty slot.

Occurrence:

Always

Attachments / Logs:

failing_log.txt
expected_log.txt
note: order of the valid messages can be different