Fix hugepage DMA mapping with IOMMU enabled

[halalboro]

Description

Commit 92d8a6b9 ("Fix memory mapping of user (DMA) buffers") introduced dma_map_single() for obtaining FPGA-accessible DMA addresses, which is necessary for IOMMU support. However, the hugepage path maps only the first 4KB page of each hugepage and then calculates the remaining sub-page IOVAs by adding PAGE_SIZE offsets. With an IOMMU enabled, IOVA space is not guaranteed to be contiguous, so these calculated addresses can point to unmapped or incorrect regions, causing DMA transfers to hang.

This fix maps each 4KB page individually via dma_map_single() regardless of hugepage backing, and updates all cleanup paths accordingly.

Type of change

• Bug fix
• New feature
• Documentation update
• A new research paper code implementation
• Other

Tests & Results

Tested with example 09_perf_rdma on Xilinx U280 with IOMMU enabled. Before the fix, RDMA transfers hang at 256 bytes and above when using hugepage-backed buffers. After the fix, all transfer sizes (64B–4MB) complete successfully.

Checklist

• I have commented my code and made corresponding changes to the documentation.
• I have added tests/results that prove my fix is effective or that my feature works.
• My changes generate no new warnings or errors & all tests successfully pass.

[bo3z]

Hi @halalboro,

Thank you for the contribution. Before reviewing/merging, I would like to clarify a couple of things.

Can you confirm that this issue is present on Example 1? RDMA can also fail due to re-transmissions. If the issue is truly due to memory mappings, it will also be present in Example 1. Additionally, 256 B seems too early to fail for this bug (as a page is 4 KiB).

We could verify this is using a series of tests, as follows:

• Running Example 1 (Hello World), 5 - 10 times with and without the fix, with huge pages.
• Running Example 9 (RDMA), 5 - 10 times with regular pages. If it fails can you note down the number of retransmissions by running cat /sys/kernel/coyote_sysfs_0/cyt_attr_nstats.
• Running Example 9 (RDMA), 5 - 10 times with huge pages, with and without and note if failures are more likely without the fix. In both cases, please note down the number of retransmissions.

For any tests that fail, can you please add the user-space outputs and the dmesg log.