Porting Coyote to Alveo U200, created a u200_static_floorplan, routing doesn't meet timing despite BUILD_OPT=1

[linusyukwong]

Hello FPGA @ Systems Group,

I am currently trying to port Coyote V2 to Alveo U200 based on the U250 constraints and scripts. I have forked Coyote and created a u200_port branch based on my best knowledge: https://github.com/linusyukwong/Coyote/tree/u200_port

Move of the u200 stuff can be directly copied from u250. The only deviations are the u200_static_floorplan, which I created based on eyeballing the u250 ones, and network_infrastructure.tcl, which I moved 1 quad higher (U200's QSFPs are 1 quad higher than U250: https://docs.amd.com/r/en-US/ds962-u200-u250/Network-Interfaces)

I have tried to build 09_perf_rdma using Vitis/Vivado 2024.1 with BUILD_STATIC=1, BUILD_SHELL=0, and BUILD_OPT=1. Sadly, the routing failed because of node overlaps and very high WNS. Please find the console log as attached u200_rdma_build_hw.log. I believe this is because of my incorrect u200_static_floorplan pblock constraints: https://github.com/linusyukwong/Coyote/blob/u200_port/hw/constraints/u200/fplan/u200_static_floorplan.xdc

Would you have any advice or guide on how to create the static floorplan or resolving the timing failure?

Thank you so much in advance.

[bo3z]

Hi @linusyukwong,

Thank you for opening the issue!

Looking at the large timing violation and the log you reported in the other issue, this doesn't look like a pblock constraint issue to me. Instead, it looks like a wrong pin assignment / unconstrained path or similar.

Disabling the floorplan & reconfiguration does not affect the driver insertion process (we have tested this on many designs across platforms). The log you reported in the other issue indicates that the XDMA wasn't brought up at all (no engines found, unable to read memory mapped registers such number of virtual regions etc.). This would likely be due to some incorrect constraint.

I can advise having a look at the Coyote v1 branch (for which supported the u200) and comparing it to your existing constraints: https://github.com/fpgasystems/Coyote/tree/coyote_v1/hw/constraints/u200.

[linusyukwong]

Hi @bo3z ,

Thank you so much for your reply!

I have been able to successfully compile and run 09_perf_rdma on U200 by specifying the GTY quad during the XDMA instantiation and removing the pin constraints. I also find that doing a warm reboot after flashing the bitstream works better compared to PCIe hot reset.

I will test a bit more on how to generate the static dcp, and hopefully open a pull request for U200.

[bo3z]

That's good to hear. I'll be looking out for the PR.

…

On Thu, 26 Feb 2026, 21:12 Linus Wong, ***@***.***> wrote: *linusyukwong* left a comment (fpgasystems/Coyote#195) <#195 (comment)> Hi @bo3z <https://github.com/bo3z> , Thank you so much for your reply! I have been able to successfully compile and run 09_perf_rdma on U200 by specifying the GTY quad during the XDMA instantiation and removing the pin constraints. I also find that doing a warm reboot after flashing the bitstream works better compared to PCIe hot reset. I will test a bit more on how to generate the static dcp, and hopefully open a pull request for U200. — Reply to this email directly, view it on GitHub <#195 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOIYLWZY3CKLKQEQXNSFSY34N5HTZAVCNFSM6AAAAACV3CAIDOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSNRYHE3DIOBRGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[yn224]

Hi @bo3z,

While we have successfully generated a bitstream for the U200 design, we have realized that running "write" demonstrates a similar behavior as that mentioned in PR #191. Specifically, the client seems to hang after the first couple of runs. Running with the default configuration of 10 test runs simply hangs after registering the client. Moreover, any subsequent attempts to re-run the example ("read" or "write") simply get stuck after the client is registered, unless we reprogram the FPGA and re-instantiate the driver.

The following shows an example output that we see with -r 1. With the default number of test runs (10) or re-running attempts, the output will be similar, except that no data will be printed out after the RDMA BENCHMARK: CLIENT header.

$ sudo ./test -i "10.0.20.30" -o 1 -r 1

-- CLI PARAMETERS:
-----------------------------------------------
Server's TCP address: 10.0.20.30
Benchmark operation: WRITE
Number of test runs: 1
Starting transfer size: 64
Ending transfer size: 1048576

Queue pair:
Local: : QPN 0x000000, PSN 0xd27567, VADDR 00007ff397000000, SIZE 00100000, IP 0xc0a80202
Remote: : QPN 0x000000, PSN 0x975bb5, VADDR 00007f909d800000, SIZE 00100000, IP 0xc0a80203
Client registered

-- RDMA BENCHMARK: CLIENT
-----------------------------------------------
Size:        64; Average throughput:  94.2264 MB/s; Average latency:    4.203 us
Size:      128; Average throughput:  194.389 MB/s; Average latency:    4.188 us
^C

Searching for a potential solution, we have landed on your most recent comment in PR #191. Following the suggestion, we have tried

1. Migrating our U200 patch (Coyote-U200-master-static.patch) and changing assign axis_ddr_wr.tlast = (cnt_ddr_wr == 1); to assign axis_ddr_wr.tlast = (cnt_C == 0); under master and recompiling the hardware
2. Disabling IOMMU on our machines through GRUB configuration

However, none of them seem to be successful so far. Would it be possible to know more in detail the key points that may be most relevant for solving this issue, assuming a clean start?

P.S. As we have two U200's directly connected to each other, RS-FEC setting was out of our concern. We have been using Vivado/Vitis 2024.1. The build configuration that we have been using is
cmake .. -DFDEV_NAME=u200 -DBUILD_STATIC=1 -DBUILD_SHELL=0

[bo3z]

Hi @yn224; have you correctly set-up the memory controller (DDR) on the U200? Can you run Example 1 with card memory (-s 0 flag in CLI).

The retrans module relies on DDR/HBM and if not set-up correctly, could lead to stalls.

Otherwise, I suggest checking the network statistics from sysfs (especially looking for TX/RX packages, retrans and PSN count) and adding ILAs to the network_stack module capturing signals going into and from the RoCE stack and looking for signs back-pressured (ready signal permanently going down).

[yn224]

Hi @bo3z,

Thank you for the help! In addition to checking DDR, we had to further fine-tune the floorplan a bit more to give more headroom for the static region. We have tested on the latest commit of yours, and the example output looks as follows. It seems that the problem of back-to-back write is also fixed for Example 09.

Example 1:

-- CLI PARAMETERS:
-----------------------------------------------
Enable hugepages: 1
Enable mapped pages: 1
Data stream: CARD
Number of test runs: 50
Starting transfer size: 64
Ending transfer size: 4194304


-- PERF LOCAL
-----------------------------------------------
Size:       64; Average throughput:  103.651 MB/s; Average latency:   2.6308 us
Size:      128; Average throughput:  209.907 MB/s; Average latency:  2.65552 us
Size:      256; Average throughput:  419.315 MB/s; Average latency:   2.6854 us
Size:      512; Average throughput:   834.22 MB/s; Average latency:  2.84864 us
Size:     1024; Average throughput:  1664.71 MB/s; Average latency:   2.8519 us
Size:     2048; Average throughput:   3305.7 MB/s; Average latency:  3.13646 us
Size:     4096; Average throughput:  6353.78 MB/s; Average latency:  3.83968 us
Size:     8192; Average throughput:  6822.05 MB/s; Average latency:  3.91246 us
Size:    16384; Average throughput:  7041.62 MB/s; Average latency:  4.55562 us
Size:    32768; Average throughput:  7139.45 MB/s; Average latency:  6.65426 us
Size:    65536; Average throughput:  7202.77 MB/s; Average latency:  10.8835 us
Size:   131072; Average throughput:  7229.73 MB/s; Average latency:  19.4951 us
Size:   262144; Average throughput:  7244.25 MB/s; Average latency:  36.6313 us
Size:   524288; Average throughput:  7251.24 MB/s; Average latency:  71.0867 us
Size:  1048576; Average throughput:  7254.71 MB/s; Average latency:  140.075 us
Size:  2097152; Average throughput:  7256.53 MB/s; Average latency:  277.861 us
Size:  4194304; Average throughput:  7257.68 MB/s; Average latency:  553.407 us

-- CLI PARAMETERS:
-----------------------------------------------
Enable hugepages: 1
Enable mapped pages: 1
Data stream: HOST
Number of test runs: 50
Starting transfer size: 64
Ending transfer size: 4194304


-- PERF LOCAL
-----------------------------------------------
Size:       64; Average throughput:  70.1326 MB/s; Average latency:  3.58462 us
Size:      128; Average throughput:  139.907 MB/s; Average latency:  3.63824 us
Size:      256; Average throughput:  280.156 MB/s; Average latency:  3.73646 us
Size:      512; Average throughput:  558.241 MB/s; Average latency:  3.84838 us
Size:     1024; Average throughput:  1111.79 MB/s; Average latency:  3.88854 us
Size:     2048; Average throughput:  2213.14 MB/s; Average latency:  3.90764 us
Size:     4096; Average throughput:  4289.98 MB/s; Average latency:   4.2847 us
Size:     8192; Average throughput:  6212.81 MB/s; Average latency:  4.72878 us
Size:    16384; Average throughput:  8392.81 MB/s; Average latency:  5.82956 us
Size:    32768; Average throughput:  10391.7 MB/s; Average latency:  7.26672 us
Size:    65536; Average throughput:  11588.6 MB/s; Average latency:   9.8485 us
Size:   131072; Average throughput:    11793 MB/s; Average latency:  15.1128 us
Size:   262144; Average throughput:    11899 MB/s; Average latency:  25.4511 us
Size:   524288; Average throughput:  11947.2 MB/s; Average latency:  46.4572 us
Size:  1048576; Average throughput:  11970.2 MB/s; Average latency:  87.9643 us
Size:  2097152; Average throughput:  11981.2 MB/s; Average latency:  171.392 us
Size:  4194304; Average throughput:  11986.3 MB/s; Average latency:  338.166 us

Example 9:

-- CLI PARAMETERS:
-----------------------------------------------
Server's TCP address: 10.0.20.30
Benchmark operation: WRITE
Number of test runs: 10
Starting transfer size: 64
Ending transfer size: 1048576

Queue pair:
Local: : QPN 0x000000, PSN 0x1da0a9, VADDR 00007f0f6aa00000, SIZE 00100000, IP 0xc0a80202
Remote: : QPN 0x000000, PSN 0x00ac2f, VADDR 00007fa685c00000, SIZE 00100000, IP 0xc0a80203
Client registered

-- RDMA BENCHMARK: CLIENT
-----------------------------------------------
Size:       64; Average throughput:  83.7971 MB/s; Average latency:  4.09465 us
Size:      128; Average throughput:  166.937 MB/s; Average latency:    4.065 us
Size:      256; Average throughput:  332.041 MB/s; Average latency:  4.21275 us
Size:      512; Average throughput:  622.599 MB/s; Average latency:  4.98055 us
Size:     1024; Average throughput:  1289.54 MB/s; Average latency:   5.1321 us
Size:     2048; Average throughput:  2532.83 MB/s; Average latency:   5.0276 us
Size:     4096; Average throughput:  4694.85 MB/s; Average latency:   5.6912 us
Size:     8192; Average throughput:  6780.41 MB/s; Average latency:   5.7346 us
Size:    16384; Average throughput:  8938.04 MB/s; Average latency:   7.2105 us
Size:    32768; Average throughput:  10234.6 MB/s; Average latency:  8.33425 us
Size:    65536; Average throughput:    10903 MB/s; Average latency:  11.1339 us
Size:   131072; Average throughput:  11133.6 MB/s; Average latency:  16.3768 us
Size:   262144; Average throughput:  11214.1 MB/s; Average latency:  27.9227 us
Size:   524288; Average throughput:  11280.8 MB/s; Average latency:  48.8922 us
Size:  1048576; Average throughput:  11304.6 MB/s; Average latency:   91.346 us
Successfully closed connection to the server

Currently, we have tested other examples as well (except for 06 and 10) and they seem to be okay. There is a slight hiccup on Example 08 but hopefully we can open a PR for U200 soon.

Thank you again.

[bo3z]

Thank you, sounds good!

May I ask what the hiccup with Example 8 is?

[yn224]

Yes, I have encountered the following error, where vfid could not be obtained.

~/Coyote/examples/08_multithreading/sw/build_sw$ sudo ./test

-- CLI PARAMETERS:
-----------------------------------------------
Number of Coyote threads: 1
Number of test runs: 50
Text size: 32108
terminate called after throwing an instance of 'std::runtime_error'
  what():  ERROR: cThread instance could not be obtained, vfid: 0
Aborted

However, I have tried reprogramming the FPGA, and it seems to be functional. Perhaps I have mistakenly programmed using a bitstream from a different example.

~/Coyote/examples/08_multithreading/sw/build_sw$ sudo ./test

-- CLI PARAMETERS:
-----------------------------------------------
Number of Coyote threads: 1
Number of test runs: 50
Text size: 32108

-- MULTI-THREADED AES ECB ENCRYPTION
-----------------------------------------------
Average throughput:  279.914 MB/s;

~/Coyote/examples/08_multithreading/sw/build_sw$ sudo ./test -t 4 -r 1000

-- CLI PARAMETERS:
-----------------------------------------------
Number of Coyote threads: 4
Number of test runs: 1000
Text size: 32108

-- MULTI-THREADED AES ECB ENCRYPTION
-----------------------------------------------
Average throughput:  1004.35 MB/s;

[bo3z]

It could be that the driver or bitstream wasn't inserted properly. If it's not reproducible, then it's probably to do with the driver or bitstream.