Upgrade dependencies

- Upgrade Slurm to version 24.11.6 (from 24.05.8).
- Upgrade EFA installer to 1.42.0 (from 1.41.0).
  - Efa-driver: efa-2.15.3-1
  - Efa-config: efa-config-1.18-1
  - Efa-profile: efa-profile-1.7-1
  - Libfabric-aws: libfabric-aws-2.1.0-3
  - Rdma-core: rdma-core-57.0-1
  - Open MPI: openmpi40-aws-4.1.7-2 and openmpi50-aws-5.0.6-11
- Upgrade Cinc Client to version to 18.4.12 from 18.2.7.
- Upgrade NVIDIA driver to version 570.172.08 (from 570.86.15) for all OSs except AL2.
- Upgrade CUDA Toolkit to version 12.8.1 (from 12.8.0) for all OSs except AL2.
- Upgrade DCGM to version 4.2.3 (from 3.3.6) for all OSs except AL2.
- Upgrade Python to 3.12.11 (from 3.12.8) for all OSs except AL2.
- Upgrade Intel MPI Library to 2021.16.0 (from 2021.13.1).

Among the above upgrade, DCGM is a major version upgrade (from version 3 to version 4)
This is a new change in DCGM 4:
```
Installation assets are no longer shipped in a single monolithic package. Instead, installation assets have been split among several packages, allowing clients to opt-out of the installation of assets not applicable to their use case.

  Component packages are as follows:

      datacenter-gpu-manager-4-core

              Provides nv-hostengine binary and other CUDA-agnostic installation assets available through the DCGM open source product

      datacenter-gpu-manager-4-cuda11

              Provides the CUDA11-specific binaries available through the DCGM open source product

      datacenter-gpu-manager-4-cuda12

              Provides the CUDA12-specific binaries available through the DCGM open source product

      datacenter-gpu-manager-4-proprietary

              Provides CUDA-agnostic installation assets not distributed as part of the DCGM open source product

      datacenter-gpu-manager-4-proprietary-cuda11

              Provides CUDA11 binaries not distributed as part of the DCGM open source product

      datacenter-gpu-manager-4-proprietary-cuda12

              Provides CUDA12 binaries not distributed as part of the DCGM open source product

      datacenter-gpu-manager-4-development

              Provides files necessary for the development of downstream software dependent on the DCGM library

```
https://docs.nvidia.com/datacenter/dcgm/latest/release-notes/changelog.html

Signed-off-by: Hanwen <[email protected]>

Signed-off-by: Hanwen <[email protected]>

Author: hanwen-cluster

CHANGELOG.md

@@ -11,7 +11,20 @@ This file is used to list changes made in each version of the AWS ParallelCluste
 
 **CHANGES**
 - Ubuntu 20.04 is no longer supported.
-- Upgrade Slurm to version 24.11.5.
+- Upgrade Slurm to version 24.11.6 (from 24.05.8).
+- Upgrade EFA installer to 1.42.0 (from 1.41.0).
+  - Efa-driver: efa-2.15.3-1
+  - Efa-config: efa-config-1.18-1
+  - Efa-profile: efa-profile-1.7-1
+  - Libfabric-aws: libfabric-aws-2.1.0-3
+  - Rdma-core: rdma-core-57.0-1
+  - Open MPI: openmpi40-aws-4.1.7-2 and openmpi50-aws-5.0.6-11
+- Upgrade Cinc Client to version to 18.4.12 from 18.2.7.
+- Upgrade NVIDIA driver to version 570.172.08 (from 570.86.15) for all OSs except AL2.
+- Upgrade CUDA Toolkit to version 12.8.1 (from 12.8.0) for all OSs except AL2.
+- Upgrade DCGM to version 4.2.3 (from 3.3.6) for all OSs except AL2.
+- Upgrade Python to 3.12.11 (from 3.12.8) for all OSs except AL2.
+- Upgrade Intel MPI Library to 2021.16.0 (from 2021.13.1).
 - Addressed cluster id mismatch known issue by deleting the file `/var/spool/slurm.state/clustername` before configuring Slurm accounting.
 - Upgrade DCV to version 2024.0-19030.
 - Remove `berkshelf`. All cookbooks are local and do not need `berkshelf` dependency management.

cookbooks/aws-parallelcluster-awsbatch/test/controls/awsbatch_virtualenv_spec.rb

@@ -13,7 +13,7 @@
 pyenv_dir = "#{base_dir}/pyenv"
 
 control 'tag:install_awsbatch_virtualenv_created' do
-  python_version = os_properties.alinux2? ? '3.9.20' : '3.12.8'
+  python_version = os_properties.alinux2? ? '3.9.20' : '3.12.11'
   title "awsbatch virtualenv should be created on #{python_version}"
   only_if { !os_properties.redhat? }
 

cookbooks/aws-parallelcluster-computefleet/kitchen.computefleet-config.yml

@@ -31,7 +31,7 @@ suites:
     attributes:
       cluster:
         custom_node_package: https://github.com/aws/aws-parallelcluster-node/archive/develop.tar.gz
-        python-version: 3.12.8
+        python-version: 3.12.11
         node_virtualenv_path: /opt/parallelcluster/pyenv/versions/node_virtualenv
   - name: fleet_status
     run_list:

cookbooks/aws-parallelcluster-environment/attributes/environment.rb

@@ -70,8 +70,8 @@
 
 default['cluster']['head_node_private_ip'] = nil
 
-default['cluster']['efa']['version'] = '1.41.0'
-default['cluster']['efa']['sha256'] = '3506354cdfbe31ff552fe75f5d0d9bb7efd29cf79bd99457347d29c751c38f9f'
+default['cluster']['efa']['version'] = '1.42.0'
+default['cluster']['efa']['sha256'] = '4114fe612905ee05083ae5cb391a00a012510f3abfecc642d86c9a5ae4be9008'
 
 default['cluster']['efs']['version'] = '2.3.1'
 default['cluster']['efs']['sha256'] = 'ced12f82e76f9740476b63f30c49bd76cc00b6375e12a9f5f7ba852635c49e15'

cookbooks/aws-parallelcluster-environment/spec/unit/resources/efa_spec.rb

@@ -2,8 +2,8 @@
 
 # parallelcluster default source dir defined in attributes
 source_dir = '/opt/parallelcluster/sources'
-efa_version = '1.41.0'
-efa_checksum = '3506354cdfbe31ff552fe75f5d0d9bb7efd29cf79bd99457347d29c751c38f9f'
+efa_version = '1.42.0'
+efa_checksum = '4114fe612905ee05083ae5cb391a00a012510f3abfecc642d86c9a5ae4be9008'
 
 class ConvergeEfa
   def self.setup(chef_run, efa_version: nil, efa_checksum: nil)

cookbooks/aws-parallelcluster-environment/test/controls/cfn_bootstrap_spec.rb

@@ -13,7 +13,7 @@
 pyenv_dir = "#{base_dir}/pyenv"
 
 control 'tag:install_cfnbootstrap_virtualenv_created' do
-  cfn_python_version = os_properties.alinux2? ? '3.9.20' : '3.12.8'
+  cfn_python_version = os_properties.alinux2? ? '3.9.20' : '3.12.11'
   title "cfnbootstrap virtualenv should be created on #{cfn_python_version}"
   only_if { !os_properties.redhat_on_docker? }
 

cookbooks/aws-parallelcluster-platform/attributes/platform.rb

@@ -16,10 +16,11 @@
 
 # NVidia
 default['cluster']['nvidia']['enabled'] = 'no'
-default['cluster']['nvidia']['driver_version'] = '570.86.15'
-default['cluster']['nvidia']['dcgm_version'] = '3.3.6'
+default['cluster']['nvidia']['driver_version'] = '570.172.08'
+default['cluster']['nvidia']['dcgm_version'] = '4.2.3-2'
 if platform?('amazon') && node['platform_version'] == "2"
   default['cluster']['nvidia']['driver_version'] = '550.127.08'
+  default['cluster']['nvidia']['dcgm_version'] = '3.3.6-1'
 end
 
 # DCV

cookbooks/aws-parallelcluster-platform/recipes/install/cuda.rb

@@ -20,9 +20,9 @@
 # Cuda installer from https://developer.nvidia.com/cuda-toolkit-archive
 # Cuda installer naming: cuda_11.8.0_520.61.05_linux
 cuda_version = '12.8'
-cuda_patch = '0'
+cuda_patch = '1'
 cuda_complete_version = "#{cuda_version}.#{cuda_patch}"
-cuda_version_suffix = '570.86.10'
+cuda_version_suffix = '570.124.06'
 cuda_samples_version = '12.8'
 if platform?('amazon') && node['platform_version'] == "2"
   cuda_version = '12.4'

cookbooks/aws-parallelcluster-platform/recipes/install/intel_mpi.rb

@@ -16,7 +16,7 @@
 # limitations under the License.
 
 intelmpi_supported = !arm_instance?
-intelmpi_version = '2021.13'
+intelmpi_version = '2021.16'
 
 node.default['conditions']['intel_mpi_supported'] = intelmpi_supported
 node.default['cluster']['intelmpi']['version'] = intelmpi_version
@@ -25,9 +25,9 @@
 
 return unless intelmpi_supported
 
-intelmpi_full_version = "#{intelmpi_version}.1.769"
+intelmpi_full_version = "#{intelmpi_version}.0.443"
 intelmpi_installation_path = "/opt/intel/mpi/#{intelmpi_version}"
-intelmpi_installer = "l_mpi_oneapi_p_#{intelmpi_full_version}_offline.sh"
+intelmpi_installer = "intel-mpi-#{intelmpi_full_version}_offline.sh"
 intelmpi_installer_path = "#{node['cluster']['sources_dir']}/#{intelmpi_installer}"
 intelmpi_installer_url = "#{node['cluster']['artifacts_s3_url']}/impi/#{intelmpi_installer}"
 intelmpi_qt_version = '6.5.3'

cookbooks/aws-parallelcluster-platform/resources/nvidia_dcgm/partial/_nvidia_dcgm_debian.rb

@@ -13,34 +13,45 @@
 # See the License for the specific language governing permissions and limitations under the License.
 
 action :install_package do
-  remote_file "#{node['cluster']['sources_dir']}/#{dcgm_package}-#{package_version}.deb" do
-    source "#{dcgm_url}"
-    mode '0644'
-    retries 3
-    retry_delay 5
-    action :create_if_missing
-  end
+  packages_urls_list = if package_version.start_with?("3.")
+                         [dcgm_package]
+                       else
+                         [dcgm4_core_package, dcgm4_package]
+                       end
+  packages_urls_list.each do |package|
+    remote_file "#{node['cluster']['sources_dir']}/#{package}-#{package_version}.deb" do
+      source "#{node['cluster']['artifacts_s3_url']}/dependencies/nvidia_dcgm/#{platform}/#{package}_#{package_version}_#{arch_suffix}.deb"
+      mode '0644'
+      retries 3
+      retry_delay 5
+      action :create_if_missing
+    end
 
-  bash "Install #{dcgm_package}" do
-    user 'root'
-    cwd node['cluster']['sources_dir']
-    code <<-DCGM_INSTALL
-    set -e
-    dpkg -i #{dcgm_package}-#{package_version}.deb
-    DCGM_INSTALL
-    retries 3
-    retry_delay 5
+    bash "Install #{package}" do
+      user 'root'
+      cwd node['cluster']['sources_dir']
+      code <<-DCGM_INSTALL
+      set -e
+      dpkg -i #{package}-#{package_version}.deb
+      DCGM_INSTALL
+      retries 3
+      retry_delay 5
+    end
   end
 end
 
-def dcgm_url
-  "#{node['cluster']['artifacts_s3_url']}/dependencies/nvidia_dcgm/#{platform}/#{dcgm_package}_#{package_version}_#{arch_suffix}.deb"
-end
-
 def dcgm_package
   'datacenter-gpu-manager'
 end
 
+def dcgm4_package
+  "#{dcgm_package}-4-cuda12"
+end
+
+def dcgm4_core_package
+  "#{dcgm_package}-4-core"
+end
+
 def arch_suffix
   arm_instance? ? 'arm64' : 'amd64'
 end