Driver: loads with BAR warnings, no `etX_ops` devices

[rjzak]

I've compiled and installed the et-driver via dkms and was able to load it, but there are warnings and errors in dkms. How should I proceed?

[ 7804.881271] et_soc1: loading out-of-tree module taints kernel.
[ 7804.881280] et_soc1: module verification failed: signature and/or required key missing - tainting kernel
[ 7804.894332] ET 0000:52:00.0: enabling device (0140 -> 0142)
[ 7805.906518] ET 0000:52:00.0: Error Event Detected
               Level     : Warning
               Desc      : BAR0 size doesn't match BAR0 size exposed by DIRs!
               Count     : 1
               Syndrome  : BAR0 size detected by host: 0x1000000000
               BAR0 size exposed by DIRs: 0x800000000
[ 7805.907310] ET 0000:52:00.0: Error Event Detected
               Level     : Fatal
               Desc      : DIR discovered P2P region mapping failed!
               Count     : 1
               Syndrome  : 
               Device: Ops
               Region type: 1
[ 7805.907325] ET 0000:52:00.0: Ops: DIR Memory Regions mapping failed!
[ 7805.907373] ET 0000:52:00.0: Ops device initialization failed, errno: 95
[ 7805.907674] ET 0000:53:00.0: enabling device (0140 -> 0142)
[ 7806.930501] ET 0000:53:00.0: Error Event Detected
               Level     : Warning
               Desc      : BAR0 size doesn't match BAR0 size exposed by DIRs!
               Count     : 1
               Syndrome  : BAR0 size detected by host: 0x1000000000
               BAR0 size exposed by DIRs: 0x800000000
[ 7806.931273] ET 0000:53:00.0: Error Event Detected
               Level     : Fatal
               Desc      : DIR discovered P2P region mapping failed!
               Count     : 1
               Syndrome  : 
               Device: Ops
               Region type: 1
[ 7806.931286] ET 0000:53:00.0: Ops: DIR Memory Regions mapping failed!
[ 7806.931332] ET 0000:53:00.0: Ops device initialization failed, errno: 95
[ 7806.931595] ET 0000:56:00.0: enabling device (0140 -> 0142)
[ 7909.329587] ET 0000:56:00.0: Error Event Detected
               Level     : Fatal
               Desc      : DIRs discovery timed out!
               Count     : 1
               Syndrome  : 
               Device: Mgmt
               Boot status: 65535
[ 7909.329619] ET 0000:56:00.0: Mgmt device initialization failed
[ 7909.329890] ET 0000:56:00.0: PCIe initialization failed
[ 7909.329917] ET 0000:56:00.0: probe with driver ET failed with error -16
[ 7909.330074] ET 0000:57:00.0: enabling device (0140 -> 0142)
[ 7910.353612] ET 0000:57:00.0: Error Event Detected
               Level     : Warning
               Desc      : BAR0 size doesn't match BAR0 size exposed by DIRs!
               Count     : 1
               Syndrome  : BAR0 size detected by host: 0x1000000000
               BAR0 size exposed by DIRs: 0x800000000
[ 7910.354459] ET 0000:57:00.0: Error Event Detected
               Level     : Fatal
               Desc      : DIR discovered P2P region mapping failed!
               Count     : 1
               Syndrome  : 
               Device: Ops
               Region type: 1
[ 7910.354472] ET 0000:57:00.0: Ops: DIR Memory Regions mapping failed!
[ 7910.354732] ET 0000:57:00.0: Ops device initialization failed, errno: 95
[ 7910.355048] ET 0000:d1:00.0: enabling device (0140 -> 0142)
[ 8012.752725] ET 0000:d1:00.0: Error Event Detected
               Level     : Fatal
               Desc      : DIRs discovery timed out!
               Count     : 1
               Syndrome  : 
               Device: Mgmt
               Boot status: 65535
[ 8012.752753] ET 0000:d1:00.0: Mgmt device initialization failed
[ 8012.753010] ET 0000:d1:00.0: PCIe initialization failed
[ 8012.753038] ET 0000:d1:00.0: probe with driver ET failed with error -16
[ 8012.753199] ET 0000:d2:00.0: enabling device (0140 -> 0142)
[ 8013.776776] ET 0000:d2:00.0: Error Event Detected
               Level     : Warning
               Desc      : BAR0 size doesn't match BAR0 size exposed by DIRs!
               Count     : 1
               Syndrome  : BAR0 size detected by host: 0x1000000000
               BAR0 size exposed by DIRs: 0x800000000
[ 8013.777675] ET 0000:d2:00.0: Error Event Detected
               Level     : Fatal
               Desc      : DIR discovered P2P region mapping failed!
               Count     : 1
               Syndrome  : 
               Device: Ops
               Region type: 1
[ 8013.777691] ET 0000:d2:00.0: Ops: DIR Memory Regions mapping failed!
[ 8013.777734] ET 0000:d2:00.0: Ops device initialization failed, errno: 95
[ 8013.778039] ET 0000:d6:00.0: enabling device (0140 -> 0142)
[ 8116.175871] ET 0000:d6:00.0: Error Event Detected
               Level     : Fatal
               Desc      : DIRs discovery timed out!
               Count     : 1
               Syndrome  : 
               Device: Mgmt
               Boot status: 65535
[ 8116.175901] ET 0000:d6:00.0: Mgmt device initialization failed
[ 8116.176182] ET 0000:d6:00.0: PCIe initialization failed
[ 8116.176211] ET 0000:d6:00.0: probe with driver ET failed with error -16

etX_ops devices missing:

❯ ls -lah /dev/et*
crw-rw-rw- 1 root root 10, 261 Jan 15 15:27 /dev/et0_mgmt
crw-rw-rw- 1 root root 10, 262 Jan 15 15:27 /dev/et1_mgmt
crw-rw-rw- 1 root root 10, 263 Jan 15 15:28 /dev/et2_mgmt
crw-rw-rw- 1 root root 10, 264 Jan 15 15:30 /dev/et3_mgmt

I was able to compile and install ET's Llama.cpp port, and it fails quickly:

❯ ./llama-cli 
ET: Attempting to initialize PCIe hardware device
ggml_et: Exception message:Only Mgmt can be enabled in recovery mode
StackTrace:
	stack dump [1]  dbg::StackException::StackException(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x31
	stack dump [2]  /home/rjzak/llama.cpp/build/bin/libggml-et.so(+0x4f37f4) [0x7f611a4f37f4]
	stack dump [3]  dev::IDeviceLayer::createPcieDeviceLayer(bool, bool) + 0x2d
	stack dump [4]  /home/rjzak/llama.cpp/build/bin/libggml-et.so(+0x4f9635) [0x7f611a4f9635]
	stack dump [5]  /home/rjzak/llama.cpp/build/bin/libggml-et.so(+0x4f9d84) [0x7f611a4f9d84]
	stack dump [6]  ggml_backend_et_reg + 0x4d
	stack dump [7]  /home/rjzak/llama.cpp/build/bin/libggml.so(+0x3dc9) [0x7f611b3f8dc9]
	stack dump [8]  /home/rjzak/llama.cpp/build/bin/libggml.so(+0x578d) [0x7f611b3fa78d]
	stack dump [9]  ggml_backend_load_all_from_path + 0xa9
	stack dump [10]  ./llama-cli(+0x5f5d5) [0x55bc2b9ee5d5]
	stack dump [11]  ./llama-cli(+0x77df5) [0x55bc2ba06df5]
	stack dump [12]  ./llama-cli(+0x44a68) [0x55bc2b9d3a68]
	stack dump [13]  /lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7f611ac34ca8]
	stack dump [14]  __libc_start_main + 0x85
	stack dump [15]  ./llama-cli(+0x4aa21) [0x55bc2b9d9a21]
build: 6872 (f36eb795) with cc (Debian 14.2.0-19) 14.2.0 for x86_64-linux-gnu

Some hopefully useful output:

❯ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 13 (trixie)
Release:	13
Codename:	trixie

❯ uname -a
Linux box.localhost 6.12.63+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.63-1 (2025-12-30) x86_64 GNU/Linux

[glguida]

Thank you for this. Which machine is this happening on? i.e., aifoundry one or your private. Has nothing to do with the machine, most likely, but with the card. And this is just to understand how we should proceed debugging this.

[rjzak]

It's a private machine.

[glguida]

Could you share the kernel configuration? This can be due to a few issues, unfortuntely.

Are you on discord? We can continue there.

[rjzak]

What are you looking for kernel-wise? It's the default Debian 13 kernel.

I can continue on Discord but I'd prefer GitHub or email.