NXP Ethernet eth_mcux poor performance

[DerekSnell]

Creating a new Issue, originally reported by @jameswalmsley at #51107. He reports that zperf shows only 1Mbps:

I have the mimxrt1064_evk board. So far I can build the following (from zephyr/main):

west build -b mimxrt1064_evk zephyr/samples/net/zperf/    

zephyr-shell

zperf udp download

linux

iperf -V -u -c fe80::4:9fff:fe39:3ca4%enp0s20f0u1u1
------------------------------------------------------------
Client connecting to fe80::4:9fff:fe39:3ca4%enp0s20f0u1u1, UDP port 5001
Sending 1450 byte datagrams, IPG target: 11062.62 us (kalman adjust)
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  1] local fe80::935:d1d6:7fda:bd83 port 49230 connected with fe80::4:9fff:fe39:3ca4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0119 sec  1.25 MBytes  1.05 Mbits/sec
[  1] Sent 906 datagrams
read failed: Connection refused
read failed: Connection refused
read failed: Connection refused

The speed seems very slow. I have tried to change many settings like dtcm, hardware acceleration etc, and I always get a very similar result of 1.05 Mbits/sec. I get the same result on our board too.

I have created a PR to fix some issues with the driver. #60073

The previous ENET_GetRxFrameSize() errors came from the eth_mcux driver only supporting the REFCLK being generated by the RT1064 (as in the ref-board).

I have added some changes in #60073 to enable the REFCLK as input, and support configuration of both 25MHz / 50MHz crystals on the PHYs.

I've also added other changes to disable cache maintenance in the HAL driver when DTCM is used for all buffers.

Unfortunately I was not able to find the source of the performance issue.

[danieldegrasse]

I've verified I can reproduce this on an RT1060 EVK. Performance also seems to have dropped for TCP RX by ~10Mbps since Zephyr 3.2. Moving the application to ITCM does not seem to help performance (and actually results in dropped packets in the TCP test)

[ssharks]

Could you check if setting CONFIG_NET_TCP_RECV_QUEUE_TIMEOUT=0 has any influence. Handling out of order data was significantly changed in 3.2.

[danieldegrasse]

Based on some tests ran on 3.4 as well as the latest main, it appears that the performance issues I encountered may have been caused by commits between 3.4 and main, by updating to the latest version of main I was able to resolve the issue. I did not observe poor performance on 3.4 either. @DerekSnell, would you like to keep this issue open?

[ssharks]

Based on some tests ran on 3.4 as well as the latest main, it appears that the performance issues I encountered may have been caused by commits between 3.4 and main, by updating to the latest version of main I was able to resolve the issue. I did not observe poor performance on 3.4 either. @DerekSnell, would you like to keep this issue open?

Could you explain / check the exact numbers in this comments? As I read it, the statement is in contradiction with itself.

[DerekSnell]

Based on some tests ran on 3.4 as well as the latest main, it appears that the performance issues I encountered may have been caused by commits between 3.4 and main, by updating to the latest version of main I was able to resolve the issue. I did not observe poor performance on 3.4 either. @DerekSnell, would you like to keep this issue open?

@jameswalmsley, does using the latest main resolve your performance issues? Thanks

[danieldegrasse]

Could you explain / check the exact numbers in this comments? As I read it, the statement is in contradiction with itself.

Sure, I'll do the best I can. When I ran some initial tests to validate that the issue existed, I was able to reproduce the performance problems mentioned in this issue. Later in the day, I sat down to bisect the issue (since I know Ethernet performance was acceptable at the 3.2 release point). As part of this, I updated my branch to the latest main. When I sat down to do additional performance validation, I was unable to reproduce the issue. Unfortunately, I did not record the initial test commands I used so I am not sure if there was an error there, or a commit between 3.4 and main had caused a temporary performance regression that was patched. I've attached the performance results from the testing I ran on 3.4 and main as a CSV to this comment.
Network Stack Testing.csv

[danieldegrasse]

@jameswalmsley, could you try with #60273 and see if this improves your performance? If you are running a custom application, you would need to integrate those code relocation directives into your CMakeLists file