Secenario Discussion: multipath transportation on industrial routers with 4G and 5G modems

[alex-cheung92]

My program mainly depends on quic multipath extension.
I read the source code methods such as picoquic_prepare_next_packet_ex ,picoquic_prepare_packet_ex, picoquic_handle_send_paths and picoquic_select_next_path_tuple It seems when multipath works in round robin order as the comment says

* For each path:
*    - For each tuple:
*         - if a first challenge is required now, the path/tuple is selected immediately.
*         - if a response is required now, the path/tuple is selected immediately.
*         - if a challenge or a response is required after the current time:
*               - the next time is update.
* If there is not an immediate selection:
*    for each path in available mode. in round robin order:
*       - if there is data or ACK to send:
*           the path is selected, with first tuple.
* If no path selected:
*    select the default path or tuple.
 */

Months before I was testing my code in two 4G modems , two paths works round robin It's fine.

But when different path with different bandwith, 4G modem maybe 10-20Mbps, and 5G modem about 70-100Mbps, I want to give diffrent weight to diffrent path according to their bandwitdth.
picoquic_set_stream_path_affinity can't help because I don't known how many packets a TCP 4 tuples will send, and don't known how may streams concurrent , So I can't divide streams into different path by number of streams.

So is there any plan to support packet scheduler to decide which path to send. Or maybe I fork the source code and do some modifications in method picoquic_select_next_path_tuple, will it affect some where else in the whole picoquic stack ? Can you give me some ideas or suggestions ?

[huitema]

Picoquic sends packets when ask to "prepare a packet" in the API picoquic_prepare_next_packet_ex, which finds the next active connection and calls picoquic_prepare_packet_ex, which in turn will call prepare_segment and create a packet for the connection, based on what frames can be sent.

There are two parts in the scheduling of packets on multiple paths. First, instead of just choosing a connection in picoquic_prepare_next_packet_ex, the code must also decide which path to poll next in picoquic_handle_send_paths, and then use that path as a parameter in the call to prepare_segment. Then, when preparing a packet in picoquic_prepare_segment, the code will send the frames that can be sent on that path.

Not all frames can be sent on all paths. The path challenge and path response frames can only be sent on the path that they are testing. The acknowledgement frames are preferably sent on the shortest path. Some stream data frames can only be sent on the "affinity" path for the stream. This is implemented in the functions that package frames in packets, for example picoquic_prepare_packet_ready, which is called from prepare_segment when the connection is in the ready state.

The goal of picoquic_handle_send_paths is to find a path which is not blocked by flow control or congestion control, and on which something can be sent. This is implemented in picoquic_select_next_path_tuple, with the following constraints:

• as long as a path is not validated, only path control frames can be sent,
• if a path is marked as backup, it should only be used if no path marked available is validated.
• acknowledgement should be sent on the shortest path,
• data can only be sent if flow control and congestion control permit
• if a path is experiencing packet loss, wait until losses are corrected to send new data
• if multiple paths are available, use round robin.

In theory, if we have paths with different data rate, the congestion control will ensure that the round robin will be "weighted" according to the available bandwidth. But this is a complex algorithm, I have been making choices, those choices may not be the best for your specific application. I have had requests to produce an API and let developers and experimenters pick different algorithms, much like we can select different congestion algorithms today. Yours in just another such request.

So, yes, adding an API to select a multipath scheduler would be useful. But of course, doing that will take time, and requires developing tests. If you want to help develop the API and the tests, that would be great.

[alex-cheung92]

Thanks for your patience and explainatin.Constrains you mentioned above I will to examize the source code carefully.

Control frames such as path_challenge and path_reponse and ack and may be misc frames send as original logic , there few of them comparing to user data frames.

Data frames I disscuess here mainly refers to user data through picoquic_callback_prepare_to_send inject into stack or by calling picoquic_add_to_stream , these frames can be scheduler.

In method picoquic_prepare_packet_ready before call to user callback picoquic_callback_prepare_to_send, do many things such as check whether exists retransmit packets and carry them(picoquic_retransmit_needed eventually call picoquic_retransmit_needed_packet ),format ack frame(picoquic_format_ack_frame) and generate path_challegne (picoquic_prepare_path_challenge_frames) frame and other frames such as picoquic_format_ack_frame, picoquic_format_max_streams_frame_if_needed , picoquic_format_max_data_frame and finally picoquic_prepare_stream_and_datagrams, so in many cases a QUIC packet may contains many frames, some of them are control frames you mentioned and maybe most of them are user data payload.

If we suppose to scheduler packets (frames may be much precise) into different paths, we should first classify frames into two category , one category should send obey the contraints, these paths are known and specific, and the other category here the issue discussed should be scheduler by scheduler. Any mistackes?

[huitema]

Converting this to a discussion. Yes, there are several frames that are not scheduled in the same way as data, but these frames are rare. Regarding packet loss correction, most of these packets correct the loss of a stream frame. The repeated stream frames are sent at the same priority as the stream.

It might make sense to ensure that packets containing stream frames do not contain control frames, but that's a big change of code.