CUDA image fails to build on Rocky Linux 9

[priteau]

We need to bump the timestamp of the EPEL repository to get a newer dkms. Alternatively it seems that 575-open installs fine?

    openstack.openhpc: TASK [cuda : Install nvidia drivers] *******************************************
    openstack.openhpc: task path: /home/rocky/cuda-bump/ansible/roles/cuda/tasks/install.yml:38
    openstack.openhpc: fatal: [default]: FAILED! => {
    openstack.openhpc:     "changed": true,
    openstack.openhpc:     "cmd": [
    openstack.openhpc:         "dnf",
    openstack.openhpc:         "module",
    openstack.openhpc:         "install",
    openstack.openhpc:         "-y",
    openstack.openhpc:         "nvidia-driver"
    openstack.openhpc:     ],
    openstack.openhpc:     "delta": "0:00:01.078464",
    openstack.openhpc:     "end": "2025-05-27 20:10:27.156802",
    openstack.openhpc:     "rc": 1,
    openstack.openhpc:     "start": "2025-05-27 20:10:26.078338"
    openstack.openhpc: }
    openstack.openhpc:
    openstack.openhpc: STDOUT:
    openstack.openhpc:
    openstack.openhpc: Last metadata expiration check: 0:00:05 ago on Tue 27 May 2025 08:10:21 PM UTC.
    openstack.openhpc: (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
    openstack.openhpc:
    openstack.openhpc:
    openstack.openhpc: STDERR:
    openstack.openhpc:
    openstack.openhpc: Error:
    openstack.openhpc:  Problem 1: cannot install the best candidate for the job
    openstack.openhpc:   - nothing provides dkms >= 3.1.8 needed by kmod-nvidia-open-dkms-3:570.148.08-1.el9.noarch from cuda-rhel9-x86_64
    openstack.openhpc:  Problem 2: package nvidia-kmod-common-3:570.148.08-1.el9.noarch from cuda-rhel9-x86_64 requires nvidia-kmod = 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - cannot install the best candidate for the job
    openstack.openhpc:   - package kmod-nvidia-latest-dkms-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 is filtered out by modular filtering
    openstack.openhpc:   - nothing provides dkms >= 3.1.8 needed by kmod-nvidia-latest-dkms-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64
    openstack.openhpc:   - nothing provides dkms >= 3.1.8 needed by kmod-nvidia-open-dkms-3:570.148.08-1.el9.noarch from cuda-rhel9-x86_64
    openstack.openhpc:  Problem 3: package nvidia-driver-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 requires nvidia-kmod-common = 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - package nvidia-kmod-common-3:570.148.08-1.el9.noarch from cuda-rhel9-x86_64 requires nvidia-kmod = 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - cannot install the best candidate for the job
    openstack.openhpc:   - package kmod-nvidia-latest-dkms-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 is filtered out by modular filtering
    openstack.openhpc:   - nothing provides dkms >= 3.1.8 needed by kmod-nvidia-latest-dkms-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64
    openstack.openhpc:   - nothing provides dkms >= 3.1.8 needed by kmod-nvidia-open-dkms-3:570.148.08-1.el9.noarch from cuda-rhel9-x86_64
    openstack.openhpc:  Problem 4: package nvidia-driver-cuda-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 requires nvidia-kmod-common = 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - package nvidia-kmod-common-3:570.148.08-1.el9.noarch from cuda-rhel9-x86_64 requires nvidia-kmod = 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - cannot install the best candidate for the job
    openstack.openhpc:   - package kmod-nvidia-latest-dkms-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 is filtered out by modular filtering
    openstack.openhpc:   - nothing provides dkms >= 3.1.8 needed by kmod-nvidia-latest-dkms-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64
    openstack.openhpc:   - nothing provides dkms >= 3.1.8 needed by kmod-nvidia-open-dkms-3:570.148.08-1.el9.noarch from cuda-rhel9-x86_64
    openstack.openhpc:  Problem 5: package nvidia-driver-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 requires nvidia-kmod-common = 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - package nvidia-settings-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 requires nvidia-driver(x86-64) = 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - package nvidia-kmod-common-3:570.148.08-1.el9.noarch from cuda-rhel9-x86_64 requires nvidia-kmod = 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - cannot install the best candidate for the job
    openstack.openhpc:   - package kmod-nvidia-latest-dkms-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 is filtered out by modular filtering
    openstack.openhpc:   - nothing provides dkms >= 3.1.8 needed by kmod-nvidia-latest-dkms-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64
    openstack.openhpc:   - nothing provides dkms >= 3.1.8 needed by kmod-nvidia-open-dkms-3:570.148.08-1.el9.noarch from cuda-rhel9-x86_64
    openstack.openhpc:  Problem 6: package nvidia-driver-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 requires nvidia-kmod-common = 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - package xorg-x11-nvidia-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 requires nvidia-driver(x86-64) = 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - package nvidia-kmod-common-3:570.148.08-1.el9.noarch from cuda-rhel9-x86_64 requires nvidia-kmod = 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - cannot install the best candidate for the job
    openstack.openhpc:   - package kmod-nvidia-latest-dkms-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 is filtered out by modular filtering
    openstack.openhpc:   - nothing provides dkms >= 3.1.8 needed by kmod-nvidia-latest-dkms-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64
    openstack.openhpc:   - nothing provides dkms >= 3.1.8 needed by kmod-nvidia-open-dkms-3:570.148.08-1.el9.noarch from cuda-rhel9-x86_64
    openstack.openhpc:  Problem 7: package xorg-x11-nvidia-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 requires nvidia-driver(x86-64) = 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - package nvidia-driver-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 requires nvidia-kmod-common = 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - package nvidia-xconfig-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 requires xorg-x11-nvidia(x86-64) >= 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - package nvidia-kmod-common-3:570.148.08-1.el9.noarch from cuda-rhel9-x86_64 requires nvidia-kmod = 3:570.148.08, but none of the providers can be installed
    openstack.openhpc:   - cannot install the best candidate for the job
    openstack.openhpc:   - package kmod-nvidia-latest-dkms-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64 is filtered out by modular filtering
    openstack.openhpc:   - nothing provides dkms >= 3.1.8 needed by kmod-nvidia-latest-dkms-3:570.148.08-1.el9.x86_64 from cuda-rhel9-x86_64
    openstack.openhpc:   - nothing provides dkms >= 3.1.8 needed by kmod-nvidia-open-dkms-3:570.148.08-1.el9.noarch from cuda-rhel9-x86_64
    openstack.openhpc:   - package xorg-x11-nvidia-3:575.51.03-1.el9.x86_64 from cuda-rhel9-x86_64 is filtered out by modular filtering

[sjpb]

Is this the cluster using OFED rather than DOCA?

[sjpb]

You can override the cuda version, if that works for you.

[sjpb]

Or PR to bump the epel timestamp

[priteau]

Is this the cluster using OFED rather than DOCA?

I checked and it also fails with doca,cuda on the current main branch. I am testing a bump of CUDA and the NVIDIA driver.

[sjpb]

I'm a bit confused how this suddently started failing. The nvidia version is pinned and epel etc are coming from Ark.

[sjpb]

So the problem is that although we pin the dnf module stream, we then just install the module from that stream, which gets the latest packages. And those have changed.

Apparently there is no way to install a specific version of a module, but you can install the pacakges from that module, and that is fine. So that's what #691 does.

[sjpb]

Ok so turns out you CAN install specific module versions using dnf module installe name:stream:version syntax, but it doesn't help in this case:

Name             : nvidia-driver
Stream           : 570-open [e] [a]
Version          : 20250527090231
Context          : f56de2b252
Architecture     : x86_64
Profiles         : default [d], fm, ks
Default profiles : default
Repo             : cuda-rhel9-x86_64
Summary          : Nvidia driver for 570-open branch
Description      : This package provides the most recent NVIDIA display driver which allows for hardware accelerated rendering with recent NVIDIA chipsets.
                 : For the full product support list, please consult the release notes for driver version 570.148.08.
Requires         : 
Artifacts        : cuda-drivers-fabricmanager-0:570.124.06-1.x86_64
                 : cuda-drivers-fabricmanager-0:570.133.20-1.x86_64
                 : cuda-drivers-fabricmanager-0:570.148.08-1.x86_64
                 : cuda-drivers-fabricmanager-0:570.86.10-1.x86_64
                 : cuda-drivers-fabricmanager-0:570.86.15-1.x86_64
                 : cuda-drivers-fabricmanager-570-0:570.124.06-1.x86_64
                 : cuda-drivers-fabricmanager-570-0:570.133.20-1.x86_64
                 : cuda-drivers-fabricmanager-570-0:570.148.08-1.x86_64
                 : cuda-drivers-fabricmanager-570-0:570.86.10-1.x86_64
                 : cuda-drivers-fabricmanager-570-0:570.86.15-1.x86_64
...

so one version of the stream contains multiple package versions...

[sjpb]

The last normal install of open-570 was this one: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/15228129032

for whcih the Install nvidia drivers task shows

 ================================================================================
    openstack.openhpc:  Package                 Arch   Version                 Repository         Size
    openstack.openhpc: ================================================================================
    openstack.openhpc: Installing group/module packages:
    openstack.openhpc:  kmod-nvidia-open-dkms   noarch 3:570.133.20-1.el9      cuda-rhel9-x86_64  12 M
    openstack.openhpc:  libnvidia-cfg           x86_64 3:570.133.20-1.el9      cuda-rhel9-x86_64 151 k
    openstack.openhpc:  libnvidia-fbc           x86_64 3:570.133.20-1.el9      cuda-rhel9-x86_64 102 k
    openstack.openhpc:  libnvidia-ml            x86_64 3:570.133.20-1.el9      cuda-rhel9-x86_64 642 k
    openstack.openhpc:  nvidia-driver           x86_64 3:570.133.20-1.el9      cuda-rhel9-x86_64 4.0 M
    openstack.openhpc:  nvidia-driver-cuda      x86_64 3:570.133.20-1.el9      cuda-rhel9-x86_64 473 k
    openstack.openhpc:  nvidia-driver-cuda-libs x86_64 3:570.133.20-1.el9      cuda-rhel9-x86_64  48 M
    openstack.openhpc:  nvidia-driver-libs      x86_64 3:570.133.20-1.el9      cuda-rhel9-x86_64 161 M
    openstack.openhpc:  nvidia-kmod-common      noarch 3:570.133.20-1.el9      cuda-rhel9-x86_64  63 M
    openstack.openhpc:  nvidia-libXNVCtrl       x86_64 3:570.133.20-1.el9      cuda-rhel9-x86_64  23 k
    openstack.openhpc:  nvidia-libXNVCtrl-devel x86_64 3:570.133.20-1.el9      cuda-rhel9-x86_64  51 k
    openstack.openhpc:  nvidia-modprobe         x86_64 3:570.133.20-1.el9      cuda-rhel9-x86_64  29 k
    openstack.openhpc:  nvidia-persistenced     x86_64 3:570.133.20-1.el9      cuda-rhel9-x86_64  33 k
    openstack.openhpc:  nvidia-settings         x86_64 3:570.133.20-1.el9      cuda-rhel9-x86_64 847 k
    openstack.openhpc:  nvidia-xconfig          x86_64 3:570.133.20-1.el9      cuda-rhel9-x86_64  93 k
    openstack.openhpc:  xorg-x11-nvidia         x86_64 3:570.133.20-1.el9      cuda-rhel9-x86_64 2.4 M
    openstack.openhpc: Installing dependencies:
    openstack.openhpc:  dkms                    noarch 3.1.6-1.el9             epel               84 k
    openstack.openhpc:  egl-gbm                 x86_64 2:1.1.2.1-1.el9         cuda-rhel9-x86_64  22 k
    openstack.openhpc:  egl-wayland             x86_64 1.1.19-2.el9            cuda-rhel9-x86_64  44 k
    openstack.openhpc:  egl-x11                 x86_64 1.0.1-6.el9             cuda-rhel9-x86_64  56 k
    openstack.openhpc:  kernel-devel-matched    x86_64 5.14.0-503.33.1.el9_5   appstream         2.0 M
    openstack.openhpc:  libglvnd-opengl         x86_64 1:1.3.4-1.el9           appstream          42 k
    openstack.openhpc:  libvdpau                x86_64 1.5-1.el9               appstream          16 k
    openstack.openhpc:  linux-firmware          noarch 20250212-146.4.el9_5    baseos            443 M
    openstack.openhpc:  linux-firmware-whence   noarch 20250212-146.4.el9_5    baseos            108 k
    openstack.openhpc:  ocl-icd                 x86_64 2.2.13-4.el9            appstream          51 k
    openstack.openhpc:  opencl-filesystem       noarch 1.0-15.el9              appstream         7.4 k
    openstack.openhpc: Installing module profiles:
    openstack.openhpc:  nvidia-driver/default                                                         

[sjpb]

Ok so:

[root@RL9-login-0 rocky]# dnf module info  --profile nvidia-driver:570-open
Last metadata expiration check: 0:30:51 ago on Thu 29 May 2025 10:07:32 AM UTC.
Name    : nvidia-driver:570-open:20250527090231:f56de2b252:x86_64
default : kmod-nvidia-open-dkms
        : libnvidia-cfg
        : libnvidia-fbc
        : libnvidia-ml
        : nvidia-driver
        : nvidia-driver-cuda
        : nvidia-driver-cuda-libs
        : nvidia-driver-libs
        : nvidia-kmod-common
        : nvidia-libXNVCtrl
        : nvidia-libXNVCtrl-devel
        : nvidia-modprobe
        : nvidia-persistenced
        : nvidia-settings
        : nvidia-xconfig
        : xorg-x11-nvidia

matches the above, i.e. that tells you what dnf module install will actually install.

[sjpb]

I think this is fixed properly by #703, which updates the install to follow the latest instructions, and, as a consequence, can now pin the driver version.