NXP: driver: nxp,kinetis-lpuart: Async API issues at high speed on imxrt1040 series

[matt-wood-ct]

Describe the bug
Background in additional comments to keep this section short.

After chasing issues for a while we went back to basics and loaded up the test/drivers/uart/uart_async_api ZTEST to validate that the IMXRT1040's UART driver is operating well.
We configured it for 1,843,200 baud with loopback enabled on lpuart8 (overlay based on IMXRT EVK overlays) and the test start crashing/failing, we found any baudrate higher then 1,000,000 would fail (see log in log section).

At baud rates below 1 mega baud, the tests pass as expected (testing 460,800 & 700,000 both passed). I don't think the test is supposed to be baud rate sensitive but please tell me if this is normal and not indicative of a driver issue.

To Reproduce

1. Open tests/drivers/uart/uart_async_api
2. Add a relevant overlay for a imxrt10xx based board with a high baudrate and loopback enabled
3. Build and run the test on hardware

Expected behavior
Async and ISR based APIs to both work reliably at any supported baud rate, and the driver tests to pass at any supported baud rate.

Impact
This is preventing us from using the DMA with the IMXRTs UARTs which is causing various issues for us, our inter-processor UART links are unreliable on ISR based UART implementations owing to data overrun issues on data bursts. Additionally on our WS2812 UART based driver we are experiencing occasional LED glitches due to transmission hitching which occurs when the ISR can't keep up.

Logs and console output
Error:

*** Booting Zephyr OS build v3.7.0-2556-g884a4e5a35f3 ***
Running TESTSUITE uart_async_chain_read
===================================================================
E: No buffers to release from RX DMA!
E: No buffers to release from RX DMA!
START - test_chained_read
E: Got error : -5

    Assertion failed at WEST_TOPDIR/zephyr/tests/drivers/uart/uart_async_api/src/test_uart_async.c:348: test_chained_read_callback: rx_data_idx + evt->data.rx.len <= sizeof(chained_cpy_buf) is false

ASSERTION FAIL [0] @ WEST_TOPDIR/zephyr/kernel/sched.c:1430
        aborting essential thread 0x20002908
E: r0/a1:  0x00000004  r1/a2:  0x00000596  r2/a3:  0x40184000
E: r3/a4:  0x00000004 r12/ip:  0x00000000 r14/lr:  0x60009aa7
E:  xpsr:  0x21000000
E: Faulting instruction address (r15/pc): 0x6000bc44
E: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
E: Current thread: 0x20002908 (sysworkq)
E: Halting system

This error seems to imply the test received back more data when it sent implying there is extra bytes coming in or going out.

Environment (please complete the following information):

• OS: Windows
• Zephyr SDK
• 884a4e5

Additional Notes
During testing of a larger system which uses high speed UARTs we had issues using the UART driver in async mode, after more testing we found that the IMXRT1040 UARTs do not behave reliably at speeds in excess of 400,000 baud. In our example we were driving Neopixels using a UART TX with signal inversion at 8,000,000 baud (which works out as 1 frame per bit on a WS2812B) and we were seeing on the bus the first few transmission bursts had all the data as zero, but when we stepped through the code the transmissions would come out uncorrupted.

[hakehuang]

@matt-wood-ct do you test with latest version?

[SynchronicIT]

We run in the same problem. We need to capture a datastream from an external transceiver over UART and both the fifo and async interface fail to receive the data using 2,000,000 baudrate.

I ran the tests with today's version and no success at all.
For us, this is a critical issue and we need to completely redesign our hardware should this be not solvable. And therefore I would propose to change the priority to at least "high".

We're using Linux OS, Zephyr SDK 0.17.0, Zephyr 4.0.0 rc1 for reference.

[hakehuang]

We're using Linux OS, Zephyr SDK 0.17.0, Zephyr 4.0.0 rc1 for reference.

@SynchronicIT , @matt-wood-ct

can you paste the log? for mimxrt1040 you need use below command line to build

west build -b mimxrt1040_evk -p always tests/drivers/uart/uart_async_api/ -- -DCONFIG_DCACHE=y -DCONFIG_NOCACHE_MEMORY=y -DCONFIG_USERSPACE=n

[SynchronicIT]

I believe I've found the root cause of the issue:

In static void dma_callback(..) at zephyr/drivers/serial/uart_mcux_lpuart.c:697 the new DMA buffer is configured (by using dma_reload(..) and start(..). This is after the both the RX_RDY and RX_RELEASED callbacks are executed. At that time, the DMA is not running and all incoming bytes beyond the uart fifo are lost.
As the RT10xx SoC only has 4 bytes of fifo, this occurs very quickly at higher baud-rates.

As soon as a new buffer is consumed by the DMA, the next one should be provided and not at the end of a buffer.

[hakehuang]

As soon as a new buffer is consumed by the DMA, the next one should be provided and not at the end of a buffer.

if in this case if you define multiply big buffer in your application, instead of just two 8 bytes buffer will be an option? as the async is kind of interact between driver and application, in the line 697, the async framework calls the user callback to get next buffer, and copy data out, which consumes a lot. If speed is a key, then the test application need updated.

[SynchronicIT]

@hakehuang The issue is in the interaction between the lpuart driver and the dma interface. We work with buffers of 8K bytes and the point where we loose data is always at the boundry between the 2 buffers.

The driver code also shows that only a single rx dma channel is used. As soon as the complete rx buffer is consumed, the new dma transfer is started after the RX_RDY and RX_RELEASE is executed. Just the function calls, the irq handling and new dma start call will consume a few [us]'s. At 2Mbit on a uart, a byte takes 5 [us] and the fifo of the peripheral is 4 bytes. So anything longer than 15~20 [us] in total would be sufficient to loose bytes.
The lpuart driver should use 2 rx channels and use them as ping-pong buffers. As soon as the first buffer in channel 0 is done, the dma would continue on the second buffer coupled to channel 1, and the lpuart driver has the whole period up to the end of the second buffer to update the first buffer. (which can be the buffer provided by the third RX_REQUEST).

That way the buffer handling and keeping the DMA happy is decoupled (thus asynchronous) from the data handling.

I'm working on a patch to solve this issue as its a blocking issue for us.

[hakehuang]

I'm working on a patch to solve this issue as its a blocking issue for us.

maybe linked DMA can help as well, so when 1 dma is done, another will auto start, take a look at: tests\drivers\dma\chan_link_transfer

[SynchronicIT]

Thanks for the hint. That's exactly what I had in mind.
I only need to check if and how to update the channels while the transfer is already active, but with doing sufficient RX_REQUEST events, that should be well solvable.

[Raymond0225]

I am preparing a loop SG mode to solve issues like this one. I am also trying to address a similar issue of I2S driver.
#70777