TCP over subg on BeagleconnectFreedom is unusable

[Ayush1325]

Describe the bug
After investigating UDP over subg on beagleconnect freedom (#68674), I have moved to testing TCP. However, I cannot seem to get TCP reliably working on the echo_server and echo_client.

Note: The subg overlays present in echo_server and echo_client disable TCP by default, so to reproduce these results, it should be enabled.

Here are the logs from echo_server:

*** Booting Zephyr OS build v3.6.0-1935-gf87143d9198a ***
[00:00:00.045,104] <inf> net_config: Initializing network
[00:00:00.155,487] <inf> net_config: IPv6 address: 2001:db8::1
[00:00:00.157,012] <inf> net_echo_server_sample: Run echo server
[00:00:00.157,073] <inf> net_echo_server_sample: Network connected
[00:00:00.157,135] <inf> net_echo_server_sample: Starting...
[00:00:00.157,592] <inf> net_echo_server_sample: Waiting for TCP connection on port 4242 (IPv6)...
[00:00:03.049,682] <inf> net_echo_server_sample: TCP (IPv6): Accepted connection
[00:00:03.050,506] <inf> net_echo_server_sample: Waiting for TCP connection on port 4242 (IPv6)...
[00:00:03.173,492] <dbg> net_echo_server_sample: handle_data: TCP (IPv6): Received and replied with 65 bytes
[00:00:03.715,881] <dbg> net_echo_server_sample: handle_data: TCP (IPv6): Received and replied with 65 bytes
[00:00:03.739,776] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.755,676] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.755,676] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.773,010] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.773,010] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.773,040] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.790,039] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.790,069] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.790,100] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.790,130] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.804,321] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.804,351] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.804,351] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.804,382] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.809,295] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.809,295] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.809,326] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.809,356] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.809,356] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:03.958,648] <dbg> net_echo_server_sample: handle_data: TCP (IPv6): Received and replied with 130 bytes
[00:00:04.676,635] <dbg> net_echo_server_sample: handle_data: TCP (IPv6): Received and replied with 8 bytes
[00:00:21.128,753] <err> net_echo_server_sample: TCP (IPv6): Connection error 116
[00:01:00.157,775] <inf> net_echo_server_sample: IPv6 TCP: Received 4 B/sec

And here are the logs from echo_client:

*** Booting Zephyr OS build v3.6.0-1935-gf87143d9198a ***
[00:00:00.046,569] <inf> net_config: Initializing network
[00:00:00.164,703] <inf> net_config: IPv6 address: 2001:db8::2
[00:00:00.164,825] <inf> net_echo_client_sample: Run echo client
[00:00:00.164,886] <inf> net_echo_client_sample: Network connected
[00:00:00.164,947] <inf> net_echo_client_sample: Starting...
[00:00:00.427,185] <dbg> net_echo_client_sample: send_tcp_data: IPv6 TCP: Sent 268 bytes
[00:00:01.144,378] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.155,883] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.155,883] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.162,139] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.162,170] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.178,253] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.178,283] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.178,283] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.202,453] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.202,453] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.202,484] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.202,514] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.220,947] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.220,947] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.220,977] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.220,977] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.221,008] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.243,560] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.243,560] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.243,591] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.243,591] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.243,621] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:01.243,652] <wrn> ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet
[00:00:17.344,635] <inf> net_echo_client_sample: Stopping...

Some additional config options I tried adding to fix this are as follows:

CONFIG_NET_L2_IEEE802154_FRAGMENT_REASS_CACHE_SIZE=32
CONFIG_IEEE802154_CC13XX_CC26XX_SUB_GHZ_NUM_RX_BUF=6

• Target: beagleconnect_freedom

To Reproduce
Steps to reproduce the behavior:

• Enable TCP in both server and client subg overlay
• Build and flash echo_server to one beagleconnect freedom

west build -b beagleconnect_freedom zephyr/samples/net/sockets/echo_server -p -- -DEXTRA_CONF_FILE=overlay-802154-subg.conf && west flash

• Build and flash echo_client to the other beagleconnect freedom

west build -b beagleconnect_freedom zephyr/samples/net/sockets/echo_client -p -- -DEXTRA_CONF_FILE=overlay-802154-subg.conf && west flash

• Open serial connection.

Expected behavior
All packets to be received by the server.

Environment (please complete the following information):

• OS: Linux
• Toolchain Zephyr SDK 0.16.5
• Zephyr Commit SHA: f87143d

[jukkar]

You are running out of network buffers according to the log output. The CONFIG_NET_TCP_MAX_SEND_WINDOW_SIZE option tuning could help the situation. See the TCP configuration guide for help
https://docs.zephyrproject.org/latest/connectivity/networking/net_config_guide.html#tcp-options

It would be useful to see sniffer data. Any TI Dev Board can be transformed into a sniffer via their PacketSniffer 2 firmware. Analyzing the wireshark protocols could help us identify the root cause.

[Ayush1325]

It would be useful to see sniffer data. Any TI Dev Board can be transformed into a sniffer via their PacketSniffer 2 firmware. Analyzing the wireshark protocols could help us identify the root cause.

Hi. Is there a way to use PacketSniffer 2 in Linux? I can only see stuff for windows.

[Ayush1325]

You are running out of network buffers according to the log output. The CONFIG_NET_TCP_MAX_SEND_WINDOW_SIZE option tuning could help the situation. See the TCP configuration guide for help https://docs.zephyrproject.org/latest/connectivity/networking/net_config_guide.html#tcp-options

Currently, it is set to 0, which means it will be calculated automatically.

Also, I did try increasing both CONFIG_NET_TCP_MAX_RECV_WINDOW_SIZE and CONFIG_NET_TCP_MAX_TX_WINDOW_SIZE to 20480 but no improvement.

Hi. Is there a way to use PacketSniffer 2 in Linux? I can only see stuff for windows.

I'm afraid that's true - several of TI's proprietary tools do not exist for the Linux platform. I use them in a Windows VM running on Linux which works well. As you're in the SubG range and TI's SUN SubG PHY settings are not 100% standard conforming, you're pretty much depending on their tools currently AFAIK.

I think there's a way to capture packages in Zephyr and forward them on another interface as alternative to using a dedicated sniffer, but I've never worked with that arrangement. And you'll only always see one side of the equation.

Re your TCP problems - it might just be that you don't get responses reliably, especially if you use ALOHA without ACK switched off. CSMA/CA and ACK are there for a reason... You'll see traces of collisions and missing packages when sniffing or debugging the TCP protocol details directly.

[jukkar]

Zephyr has packet capture support, see this document https://docs.zephyrproject.org/latest/connectivity/networking/network_monitoring.html
There is a sample that can show what can be done in your own application, https://docs.zephyrproject.org/latest/samples/net/capture/README.html
I recently created support for cooked mode capture in this PR #70926. It allows user to capture low level non-IP based packets. The PR has example for capturing HDLC frames in PPP traffic and send them for analysis to external systems, see this commit 7fdebb5 for details.
For 802.15.4 L2 packet capturing, some extra work would need to be done, basically adding packet capture calls to the proper places in 802.15.4 stack (similarly what I did for PPP), if one wants to see 802.15.4 L2 traffic. If normal IP traffic is enough, then that IP traffic can be captured already.
If the generic packet capture is setup and enabed (either from net-shell or directly from capture API), the packets are then automatically captured for analysis.
One caveat in the capturing packets inside a Zephyr device, is that it is not storing the packets locally but send them to outside system where Wireshark or similar could be used to show the data. This requires extra IP based network connection in the device. For normal IP traffic capturing, the same interface can be used for capturing and transferring the captured packets, but for cooked capture mode, the captured packets cannot be sent via the same interface (as that would cause recursion atm, there might be a way to avoid that but it requires some extra coding).

[Ayush1325]

Re your TCP problems - it might just be that you don't get responses reliably, especially if you use ALOHA without ACK switched off. CSMA/CA and ACK are there for a reason... You'll see traces of collisions and missing packages when sniffing or debugging the TCP protocol details directly.

The initial testing is being done with CSMA/CA and ACK since I was not really testing the performance in this issue (it has to work reliably before worrying about performance).

I will try sniffing and a git bisect to pin point the problem.

The initial testing is being done with CSMA/CA and ACK since I was not really testing the performance in this issue (it has to work reliably before worrying about performance).

Totally agree.

I will try sniffing and a git bisect to pin point the problem.

Nice. You may of course contact me any time on Discord as well if there's something you think I could help you with. I'd really love to see this hardware to be better supported.