RmInitAdapter failed when GPU is installed in a PCIe Switch - PLEASE HELP!!!

[ykali-sv]

Hi,

I’m developing a pre-Si PCie Switch in emulation environment (FPGA) with a dedicated board that serves as a PCIe bridge connected to the emulator with RTL of the switch.
the GPU is Nvidia RTX 4000 Ada. the emulator is connected to a Supermicro X11DAi-N.
Nvidia Driver version is 550.107.02 on Fedora 37 system.
When the host boots, dmesg shows several issues with the GPU initialization process like: RmInitAdapter: Cannot initialize GSP firmware RM.
when the GPU is connected directly to the host - it works fine!
I’ve attached the dmesg , lspci dump of the GPU and the Nvidia debug log.
I think that maybe because the GPU is slow because its connected an emulator so there are issues with timeout…
Anyone can help me understand why we get these errors ?

gpu_debug_log.txt
gpu_lspci.txt

[mtijanic]

Hi there. From the log, it looks like you did manage to read some valid data from PCI already, so it could just be too slow. vbios could be large.

Maybe try something like this to diagnose further?

diff --git a/src/nvidia/src/kernel/gpu/gsp/arch/turing/kernel_gsp_vbios_tu102.c b/src/nvidia/src/kernel/gpu/gsp/arch/turing/kernel_gsp_vbios_tu102.c
index d179af79..b5c62165 100644
--- a/src/nvidia/src/kernel/gpu/gsp/arch/turing/kernel_gsp_vbios_tu102.c
+++ b/src/nvidia/src/kernel/gpu/gsp/arch/turing/kernel_gsp_vbios_tu102.c
@@ -527,6 +527,8 @@ kgspExtractVbiosFromRom_TU102
 
     biosSize = biosSizeFromRom;
 
+    NV_PRINTF(LEVEL_ERROR, "XXX: vbios size is %u bytes\n", biosSize);
+
     // Copy to system memory and populate pVbiosImg
     {
         NvU32 i;
@@ -542,6 +544,7 @@ kgspExtractVbiosFromRom_TU102
         biosSizeAligned = biosSize & (~0x3);
         for (i = 0; i < biosSizeAligned; i += 4)
         {
+            NV_PRINTF(LEVEL_ERROR, "XXX: Reading vbios word %u out of %u\n", i/4, biosSizeAligned/4);
             pImageDwords[i >> 2] = s_promRead32(pGpu, pciOffset + i);
         }

If that's the case, then maybe you can connect the card to a regular PCI bus, extract the vbios and save it to disk, and then patch up that function to return the data from disk instead. However, you'll likely hit other issues as the driver will have to copy much larger data structures eventually (such as uploading of gsp.bin).