[spark-rapids.sh] Refactor NVIDIA driver installation for Rocky Linux 8 to use run file (#1359)

* Refactor NVIDIA driver installation for Rocky Linux 8

Updated the installation process for the NVIDIA GPU driver on Rocky Linux 8. The script now installs kernel development packages directly and downloads the CUDA installer run file, executing it in silent mode. The installer file is removed post-installation to clean up. This change simplifies the installation steps and ensures the correct driver version is used.

* feat: Enable spark-rapids on Dataproc 2.1 Rocky Linux 8

This commit integrates changes to enable the spark-rapids initialization action on Dataproc 2.1-rocky8 images.

- Updates the NVIDIA driver installation process in `spark-rapids.sh` for Rocky Linux:
  - Uses `curl` with retry and fail-fast options for downloading the CUDA installer.
  - Executes the NVIDIA installer with `--silent --driver --toolkit --no-opengl-libs` flags and wraps it in `execute_with_retries`.

- Modifies `test_spark_rapids.py` to enable tests for Rocky Linux on Dataproc 2.1 and below, while keeping them skipped for 2.2+ (Rocky 9).

This resolves the installation issues on Rocky 8. Further work is required to support Rocky 9 (Dataproc 2.2).

---------

Co-authored-by: C.J. Collier <[email protected]>

Author: SurajAralihalli

spark-rapids/spark-rapids.sh

@@ -501,17 +501,20 @@ function install_nvidia_gpu_driver() {
 
   elif is_rocky ; then
 
-    # Ensure the Correct Kernel Development Packages are Installed
-    execute_with_retries "dnf -y -q update --exclude=systemd*,kernel*"
-    execute_with_retries "dnf -y -q install pciutils kernel-devel gcc"
+    # Install kernel development packages
+    execute_with_retries "dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r)"
 
-    readonly NVIDIA_ROCKY_REPO_URL="${NVIDIA_REPO_URL}/cuda-${shortname}.repo"
-    execute_with_retries "dnf config-manager --add-repo ${NVIDIA_ROCKY_REPO_URL}"
-    execute_with_retries "dnf clean all"
-    configure_dkms_certs
-    execute_with_retries "dnf -y -q module install nvidia-driver:latest-dkms"
-    clear_dkms_key
-    execute_with_retries "dnf -y -q install cuda-toolkit"
+    # Download the CUDA installer run file
+    curl -fsSL --retry-connrefused --retry 3 --retry-max-time 30 -o driver.run \
+        "https://developer.download.nvidia.com/compute/cuda/${CUDA_VERSION}/local_installers/cuda_${CUDA_VERSION}_${NVIDIA_DRIVER_VERSION}_linux.run"
+    
+    # Run the installer in silent mode
+    execute_with_retries "bash driver.run --silent --driver --toolkit --no-opengl-libs"
+
+    # Remove the installer file after installation to clean up
+    rm driver.run
+
+    # Load the NVIDIA kernel module
     modprobe nvidia
 
   else

spark-rapids/test_spark_rapids.py

@@ -58,8 +58,8 @@ def verify_spark_job_sql(self):
                             ("STANDARD", ["w-0"], GPU_T4))
   def test_spark_rapids(self, configuration, machine_suffixes, accelerator):
 
-    if self.getImageOs() == "rocky":
-      self.skipTest("Not supported for Rocky OS")
+    if self.getImageVersion() > pkg_resources.parse_version("2.0") and self.getImageOs() == "rocky":
+      self.skipTest("Not supported for Rocky 9")
 
     if self.getImageVersion() <= pkg_resources.parse_version("2.0"):
       self.skipTest("Not supported in 2.0 and earlier images")
@@ -88,8 +88,8 @@ def test_spark_rapids(self, configuration, machine_suffixes, accelerator):
                             ("STANDARD", ["w-0"], GPU_T4))
   def test_spark_rapids_sql(self, configuration, machine_suffixes, accelerator):
 
-    if self.getImageOs() == "rocky":
-      self.skipTest("Not supported for Rocky OS")
+    if self.getImageVersion() > pkg_resources.parse_version("2.0") and self.getImageOs() == "rocky":
+      self.skipTest("Not supported for Rocky 9")
 
     if self.getImageVersion() <= pkg_resources.parse_version("2.0"):
       self.skipTest("Not supported in 2.0 and earlier images")
@@ -118,8 +118,8 @@ def test_spark_rapids_sql(self, configuration, machine_suffixes, accelerator):
   def test_non_default_cuda_versions(self, configuration, machine_suffixes,
                                      accelerator, cuda_version, driver_version):
 
-    if self.getImageOs() == "rocky":
-      self.skipTest("Not supported for Rocky OS")
+    if self.getImageVersion() > pkg_resources.parse_version("2.0") and self.getImageOs() == "rocky":
+      self.skipTest("Not supported for Rocky 9")
 
     if self.getImageVersion() <= pkg_resources.parse_version("2.0"):
       self.skipTest("Not supported in 2.0 and earlier images")