TF 2.18 Training SM pin nccl to version<13 (#5193)

* Test heavy efa

* enable efa logging

* enable ap

* pin nccl version

* fix nccl version

* revert libnccl

* test efa with usr lib

* add nvidia debug

* add logging

* fix debug call

* remove -x

* fix log

* set +x

* enable set -x

* print output

* try tf 2.19

* print output

* run 2.18

* install aws-ofi-nccl efa and mpi from amazon

* fix ofi

* test pytorch

* print output

* test tf with correct logging

* use prod image

* build nccl cuda 12

* build test all

* revert toml and formatting

Author: sirutBuasai

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,13 +1,14 @@
 *GitHub Issue #, if available:*
 
-**Note**: 
-- If merging this PR should also close the associated Issue, please also add that Issue # to the Linked Issues section on the right. 
+**Note**:
+- If merging this PR should also close the associated Issue, please also add that Issue # to the Linked Issues section on the right.
 
 - All PR's are checked weekly for staleness. This PR will be closed if not updated in 30 days.
 
 ### Description
 
 ### Tests Run
+
 By default, docker image builds and tests are disabled. Two ways to run builds and tests:
 1. Using dlc_developer_config.toml
 2. Using this PR description (currently only supported for PyTorch, TensorFlow, vllm, and base images)
@@ -16,7 +17,7 @@ By default, docker image builds and tests are disabled. Two ways to run builds a
 <summary>How to use the helper utility for updating dlc_developer_config.toml</summary>
 
 Assuming your remote is called `origin` (you can find out more with `git remote -v`)...
-  
+
 - Run default builds and tests for a particular buildspec - also commits and pushes changes to remote; Example:
 
 `python src/prepare_dlc_dev_environment.py -b </path/to/buildspec.yml> -cp origin`
@@ -42,7 +43,7 @@ Use the code block below to uncomment commands and run the PR CodeBuild jobs. Th
 
 - `# /buildspec <buildspec_path>`
   - e.g.: `# /buildspec pytorch/training/buildspec.yml`
-  - If this line is commented out, dlc_developer_config.toml will be used. 
+  - If this line is commented out, dlc_developer_config.toml will be used.
 - `# /tests <test_list>`
   - e.g.: `# /tests sanity security ec2`
   - If this line is commented out, it will run the default set of tests (same as the defaults in dlc_developer_config.toml): `sanity, security, ec2, ecs, eks, sagemaker, sagemaker-local`.
@@ -51,13 +52,13 @@ Use the code block below to uncomment commands and run the PR CodeBuild jobs. Th
 
 ```
 # /buildspec <buildspec_path>
-# /tests <test_list> 
+# /tests <test_list>
 ```
 
 ### Formatting
 - [ ] I have run `black -l 100` on my code (formatting tool: https://black.readthedocs.io/en/stable/getting_started.html)
 
-### PR Checklist 
+### PR Checklist
 <details>
 <summary>Expand</summary>
 

tensorflow/training/docker/2.18/py3/cu125/Dockerfile.gpu

@@ -106,8 +106,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends --allow-unauthe
  && apt-get install -y --no-install-recommends --allow-unauthenticated --allow-change-held-packages \
    libcublas-dev-${CUDA_DASH} \
    libcublas-${CUDA_DASH} \
-   libnccl2 \
-   libnccl-dev \
+   libnccl2=2.27.7-1+cuda12.9 \
+   libnccl-dev=2.27.7-1+cuda12.9 \
  && rm -rf /var/lib/apt/lists/* \
  && apt-get clean \
  && mkdir -p /var/run/sshd
@@ -220,8 +220,8 @@ RUN ${PIP} install --no-cache-dir -U \
    h5py \
    absl-py \
    werkzeug \
-   urllib3 
-   
+   urllib3
+
 # Install AWS OFI NCCL plug-in
 RUN apt-get update && apt-get install -y \
     autoconf \
@@ -256,7 +256,7 @@ RUN mkdir -p /tmp/nvjpeg \
 && rm -rf /tmp/nvjpeg \
 # patch cuobjdump and nvdisasm
 && rm -rf /usr/local/cuda/bin/cuobjdump* \
-&& rm -rf /usr/local/cuda/bin/nvdisasm* 
+&& rm -rf /usr/local/cuda/bin/nvdisasm*
 
 # Allow OpenSSH to talk to containers without asking for confirmation
 RUN cat /etc/ssh/ssh_config | grep -v StrictHostKeyChecking > /etc/ssh/ssh_config.new \
@@ -295,7 +295,7 @@ RUN ${PIP} install --no-cache-dir -U \
    ${TF_URL} \
    "tensorflow-io==0.37.*" \
    "tensorflow-datasets==4.9.7" \
-   opencv-python 
+   opencv-python
 
 RUN HOME_DIR=/root \
    && curl -o ${HOME_DIR}/oss_compliance.zip https://aws-dlinfra-utilities.s3.amazonaws.com/oss_compliance.zip \
@@ -381,7 +381,7 @@ RUN $PYTHON -m pip install --no-cache-dir -U \
 
 RUN $PYTHON -m pip install --no-cache-dir -U \
     sagemaker-experiments==0.1.45
-   
+
 RUN $PYTHON -m pip install --no-cache-dir -U \
     sagemaker-training
 
@@ -392,11 +392,11 @@ RUN $PYTHON -m pip install --no-cache-dir -U \
     sagemaker-studio-analytics-extension==0.1.4
 
 RUN $PYTHON -m pip install --no-cache-dir -U \
-    sagemaker-studio-sparkmagic-lib==0.2.0 
+    sagemaker-studio-sparkmagic-lib==0.2.0
 
 RUN $PYTHON -m pip install --no-cache-dir -U \
     sparkmagic==0.21.0 \
-    smclarify 
+    smclarify
 
 
 # install boost