fix: upgrade Intel MPI to 2021.14 and fix CI race condition

[Syed-Suhaan]

Upgrades Intel MPI version from 2021.13 to 2021.14.

Adds a 5-second initialization delay in entrypoint.sh to prevent SSH handshake failures (kex_exchange_identification) observed in CI environments.

Fixes Issue #678.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign terrytangyuan for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tenzen-y]

Thank you for working on this issue 👍

[tenzen-y]

Instead of this sleep, what about increasing the backoff to 1 or more?

[Syed-Suhaan]

Replaced sleep with backoff=1 as suggested.

[tenzen-y]

Could you use the previous indent?

[Syed-Suhaan]

Fixed the indentation to match the previous style.

[tenzen-y]

ditto

[Syed-Suhaan]

Fixed the indentation to match the previous style.

[Syed-Suhaan]

I've applied the feedback: replaced sleep with backoff logic and fixed Dockerfile indentation. Verified locally

[tenzen-y]

Let's see if the errors have gone away in CI.

[tenzen-y]

It seems that you forgot to push the new commit to this PR?

Suggested change

	libstdc++-12-dev binutils procps clang \
	intel-oneapi-compiler-dpcpp-cpp \
	intel-oneapi-mpi-devel-2021.14 \
	libstdc++-12-dev binutils procps clang \
	intel-oneapi-compiler-dpcpp-cpp \
	intel-oneapi-mpi-devel-2021.14 \

[tenzen-y]

Suggested change

	dnsutils \
	intel-oneapi-mpi-2021.14 \
	dnsutils \
	intel-oneapi-mpi-2021.14 \

[Syed-Suhaan]

Fixed entrypoint.sh race condition by switching to native bash backoff (removing awk dependency), and corrected Dockerfile indentation.

[tenzen-y]

Suggested change

	backoff=1
	backoff=3

@Syed-Suhaan It seems 1 sec is not enough: https://github.com/kubeflow/mpi-operator/actions/runs/20128945073/job/57765870629?pr=757

Let's increase it to 3 sec.

[tenzen-y]

@Syed-Suhaan Uhm, 3 sec seems still not enough time. Did 3 sec work well in your local?
If not, we might need to increase it to 5 sec.

[Syed-Suhaan]

Yes, 3s works locally, but my local environment resolves instantly so it doesn't reproduce the race condition. Since the CI logs show the environment is noticeably slower, I've updated it to 5s as suggested to ensure we clear that race condition reliably.