HELP: Working on a PR for an extension that packages the Xilinx DMA drivers

[nikolaishields]

Hi!
I'm working on contributing a PR to bring the Xilinx DMA drivers into talos as an extension. I'm confused on a few points:

• I've packaged the drivers into an extension, but the driver makefile is looking for this path:
  0.058 make -C /lib/modules/6.6.71/build M=/tmp/build/dma_ip_drivers-2023.2.1/XDMA/linux-kernel/xdma modules 0.060 make[1]: Entering directory '/tmp/build/dma_ip_drivers-2023.2.1/XDMA/linux-kernel/xdma' 0.060 make[1]: Leaving directory '/tmp/build/dma_ip_drivers-2023.2.1/XDMA/linux-kernel/xdma' 0.060 make[1]: *** /lib/modules/6.6.71/build: No such file or directory. Stop.
  I've tried using the kernel image as the base, the same way the v4l extension does with no luck. I'm certain I'm missing something here. By the looks of it this is no different from the other drivers, the closest of which may be the proprietary nvidia drivers.

Any and all advice welcomed.
References:
drivers
my branch

[frezbo]

the drivers are built in the pkgs which has access to the kernel-build stage repo, see this as an example https://github.com/siderolabs/pkgs/tree/main/gasket-driver

extensions repo just copies over the built driver modules from the pkgs repo

[nikolaishields]

@frezbo Thanks for the reply
I've got things building now and the local build of the files look to have everything I expect. However I'm curious as to the minimum I need to upgrade an existing node with the kernel/driver I've just packaged. After reading this it seems that I can build an installer image that overrides with the new kernel which in turn should contain the copy of my package by setting the PKG_KERNEL variable to my image? Is this correct? Or do I have to also make an extension and load that in?

Thanks!

[nikolaishields]

I'm trying to bring a set of upstream drivers that are packaged into a kernel module into the system. I'm upgrading an existing set of talos nodes using a custom installer, with the ultimate goal of validating the xdma kernel module works in talos. After following the tutorials detailed in the documentation:
https://www.talos.dev/v1.9/advanced/developing-talos/
https://www.talos.dev/v1.9/advanced/building-images/
https://www.talos.dev/v1.9/talos-guides/install/boot-assets/
https://www.talos.dev/v1.9/advanced/customizing-the-kernel/

I'm trying to test the following:

• Does the xdma module get loaded?
• Are the character devices created as expected?
• Is the xdma-drivers extension I've drafted being installed when upgrading a talos node?
• Are connected Xilinx devices detected and functioning properly?

Testing Procedure:
I've been performing the following:

1. Perform local build of the pkg and verify the drivers and resulting filesystem is as I expect

cd pkgs/
make local-xdma-driver-pkg PLATFORM=linux/amd64 USERNAME=nikolaishields DEST=_out/

2. Perform local build of the extension and verify the drivers and resulting filesystem is as I expect

cd extensions/
make local-xdma-driver PLATFORM=linux/amd64 USERNAME=nikolaishields DEST=_out/

3. Push the pkgs image

cd pkgs/
make xdma-driver-pkg PLATFORM=linux/amd64 USERNAME=nikolaishields PUSH=true

4. Push the extensions image for the driver

cd extensions/
make xdma-driver PLATFORM=linux/amd64 USERNAME=nikolaishields PUSH=true

5. generate an installer image with imager with my extensions

cd talos/
docker run --rm -t -v $PWD/_out:/out ghcr.io/siderolabs/imager:v1.9.5 installer --system-extension-image ghcr.io/nikolaishields/xdma-driver:aefa9a1@sha256:d82a58de41db539b0d107f70a2ac26246a562053b2aeb0e4a02fbc75c959475e --system-extension-image ghcr.io/siderolabs/gasket-driver:5815ee3-v1.9.5@sha256:a525db5589ce4b0f14c5c7a9f57103f4a03e80724b005d1766c16ef31e7cac2f

6. Push the image with crane

crane push _out/installer-amd64.tar ghcr.io/nikolaishields/g/installer:v1.9.5

7. Upgrade the node:

talosctl upgrade --image <image> 

At the end of the day all I want to do is use the kernel module and let talos do what it does best, which is get out of the way in an operational sense. If there's something I'm doing (or not doing) please let me know. All the better if there's something that can expedite the contribution upstream so that I can use this via the image factory.

Thanks in advance 🙂

[smira]

Let's cut some steps to make testing easier, but your steps certainly miss building/pushing kernel image, and that might be the problem today.

1. Build and push both your driver and upstream kernel. This should be done in one step, and kernel build should be cached.

make kernel xdma-driver-pkg USERNAME=nikolaishields PUSH=true PLATFORM=linux/amd64

2. As you're testing, skip the extension step and compile in your module into base Talos to simplify things.

Insert COPY from your xdma-driver-pkg package to the steps https://github.com/siderolabs/talos/blob/main/Dockerfile#L691-L692, so that you copy both standard and yours modules.

Add your module(s) to hack/modules-amd64.txt.

3. If you're testing on main, the build process is multi-step right now, on 1.9 it's a bit simpler:

For 1.9:

make installer PKG_KERNEL=<your build of kernel> USERNAME=... PUSH=true

For main:

make installer-base PKG_KERNEL=<your build of kernel> USERNAME=... PUSH=true
make imager PKG_KERNEL=<your build of kernel> USERNAME=... INSTALLERARCH=targetarch PUSH=true
make installer USERNAME=... PUSH=true

Anyways, in the end you should have your custom installer image, and use it do upgrade to.

[nikolaishields]

After attempting your instructions on v1.9.5 I received this error back:

 => CACHED [installer-image  4/10] COPY --chmod=0644 hack/extra-modules.conf /etc/m  0.0s
 => CANCELED [installer-image  5/10] COPY --from=pkg-grub / /                        0.0s
 => CANCELED [installer-image  6/10] COPY --from=pkg-grub-arm64 /usr/lib/grub /usr/  0.0s
 => CANCELED [installer-image  7/10] COPY --from=pkg-grub-amd64 /usr/lib/grub /usr/  0.0s
------
 > [embed-abbrev-generate 3/3] COPY --from=pkg-kernel /certs/signing_key.x509 _out/signing_key.x509:
------
------
 > [depmod-amd64 3/4] COPY --from=ghcr.io/nikolaishields/xdma-driver-pkg:8e51334-dirty /lib/modules lib/modules:
------
Dockerfile:381
--------------------
 379 |         echo TAG=${TAG} >> _out/talos-metadata && \
 380 |         echo EXTRAS=${EXTRAS} >> _out/talos-metadata
 381 | >>> COPY --from=pkg-kernel /certs/signing_key.x509 _out/signing_key.x509
 382 |     
 383 |     FROM scratch AS embed-abbrev
--------------------
ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref tiwjvsx61nduqg5ya0xbrs3tq::e61ebc5hlw9p45fkt3uwhp6eh: "/certs/signing_key.x509": not found
make[2]: *** [Makefile:333: target-installer] Error 1
make[2]: Leaving directory '/home/nshields/Code/git.dwavesys.local/do-2683/talos'
make[1]: *** [Makefile:350: registry-installer] Error 2
make[1]: Leaving directory '/home/nshields/Code/git.dwavesys.local/do-2683/talos'
make: *** [Makefile:397: installer] Error 2

Compilation exited abnormally with code 2 at Mon Mar 31 02:16:46

However when following the giving instructions from the main branch I get an installer but that image inevitably breaks the node upon upgrade.

make installer-base PKG_KERNEL=ghcr.io/nikolaishields/kernel:8e51334-dirty USERNAME=nikolaishields PLATFORM=linux/amd64 PUSH=true

make imager PKG_KERNEL=ghcr.io/nikolaishields/kernel:8e51334-dirty USERNAME=nikolaishields INSTALLER_ARCH=targetarch PLATFORM=linux/amd64 PUSH=true

0.284 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/drivers/virtio/virtio_mmio.ko': No such file or directory
0.285 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/drivers/virtio/virtio_pci.ko': No such file or directory
0.285 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/drivers/virtio/virtio_pci_legacy_dev.ko': No such file or directory
0.288 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/drivers/virtio/virtio_pci_modern_dev.ko': No such file or directory
0.288 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/drivers/watchdog/i6300esb.ko': No such file or directory
0.288 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/drivers/watchdog/iTCO_vendor_support.ko': No such file or directory
0.290 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/drivers/watchdog/iTCO_wdt.ko': No such file or directory
0.290 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/drivers/watchdog/sp5100_tco.ko': No such file or directory
0.291 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/drivers/watchdog/watchdog.ko': No such file or directory
0.292 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/drivers/watchdog/wdat_wdt.ko': No such file or directory
0.293 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/drivers/watchdog/xen_wdt.ko': No such file or directory
0.294 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/lib/objagg.ko': No such file or directory
0.294 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/lib/parman.ko': No such file or directory
0.295 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/lib/raid6/raid6_pq.ko': No such file or directory
0.297 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/net/ipv4/ip_gre.ko': No such file or directory
0.298 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/net/openvswitch/vport-gre.ko': No such file or directory
0.299 install: cannot stat 'usr/lib/modules/6.12.19-talos/kernel/net/tls/tls.ko': No such file or directory
------
Dockerfile:692
--------------------
 691 |     COPY --from=ghcr.io/nikolaishields/xdma-driver-pkg:8e51334-dirty /usr/lib/modules usr/lib/modules
 692 | >>> RUN <<EOF
 693 | >>> set -euo pipefail
 694 | >>>
 695 | >>> KERNEL_VERSION=$(ls usr/lib/modules)
 696 | >>>
 697 | >>> xargs -a modules-amd64.txt -I {} install -D usr/lib/modules/${KERNEL_VERSION}/{} /build/usr/lib/modules/${KERNEL_VERSION}/{}
 698 | >>>
 699 | >>> depmod -b /build/usr ${KERNEL_VERSION}
 700 | >>> EOF
 701 |
--------------------
ERROR: failed to solve: process "/bin/bash -c set -euo pipefail\n\nKERNEL_VERSION=$(ls usr/lib/modules)\n\nxargs -a modules-amd64.txt -I {} install -D usr/lib/modules/${KERNEL_VERSION}/{} /build/usr/lib/modules/${KERNEL_VERSION}/{}\n\ndepmod -b /build/usr ${KERNEL_VERSION}\n" did not complete successfully: exit code: 123
make[2]: *** [Makefile:344: target-imager] Error 1
make[2]: Leaving directory '/home/nshields/Code/git.dwavesys.local/do-2683/talos'
make[1]: *** [Makefile:361: registry-imager] Error 2
make[1]: Leaving directory '/home/nshields/Code/git.dwavesys.local/do-2683/talos'
make: *** [Makefile:412: imager] Error 2

I attempt to correct this error by copying from the kernel image instead and that seems to build a broken installer.

make installer USERNAME=nikolaishields PLATFORM=linux/amd64 PUSH=true

talosctl upgrade -n xdma-fit.internal --image ghcr.io/nikolaishields/installer:v1.10.0-alpha.3-8-gbf80079d4-dirty --preserve

I'd greatly appreciate any insight you can lend @smira

[smira]

you can't mix PKGS with Talos from main/release-1.9 (there's a different directory layout).

so I posted instructions for both, as I wasn't sure which one you're using.

All in all, it seems to be more painful for you than it should be, and probably just leave here a link to the pkgs PR, and we can look/verify/get it merged.

[nikolaishields]

Ah, I did build from main so that would explain why the 1.9 method errored out.

I speak for the company when I say we'd all greatly appreciate that:
Here's a link to the:
pkgs wip pr
link to fork for your convenience
extensions fork

I'll try once more from each tag myself, though I admit it has been more challenging than anticipated to bring an existing kernel module into the system. We're presently evaluating talos as a replacement to another vendor and having these drivers/modules is critical path for us, especially when looking at omni.

@smira a personal thank-you for your support as I've been working through this 👍

[nikolaishields]

@smira I was able to get a build up and have confirmed the modules are built, in the proper place and extension installed all from upstream artifacts published by sidero/talos. The module lands in /usr/lib/modules/6.12.21-talos/extras after upgrading with an image with the alpha3 system extension for the xdma-driver.

 talosctl ls /usr/lib/modules/6.12.21-talos/extras -n 192.168.1.109
NODE            NAME
192.168.1.109   .
192.168.1.109   xdma.ko

The last piece of the puzzle seems like a simple one that perhaps I'm missing:
How do I load the module? specifying the following in the machine config doesn't work and it seems that extras should be loaded automatically per https://github.com/siderolabs/talos/blob/main/hack/extra-modules.conf

    kernel:
        modules:
            - name: xdma 

[frezbo]

it needs to be loaded explicitly, please attach a support.zip after adding the machine config to load the modules

[smira]

Yes, xdma module loads successfully and registers, I verified that.

[nikolaishields]

Absolutely:
support.zip

Yes, xdma module loads successfully and registers, I verified that.

Fantastic, now I just need to be able to get this loaded and we'll be off to the races.

[nikolaishields]

@smira any chance you could attach the installer image/iso you tested to this thread so that I could try it?

[smira]

If you have trouble building from main, you can wait for Talos 1.10.0-beta.0 early next week, and you can download your requested image from the Image Factory.