[CI] Remove the xpu env source for linux binary validate (pytorch#154409)

[CI] Remove the xpu env source for linux binary validate (pytorch#150138)

Due to we have enabled the xpu runtime pypi packages as dependencies directly

Pull Request resolved: pytorch#150138
Approved by: https://github.com/atalman

(cherry picked from commit 35ff508)

Co-authored-by: Wang, Chuanqi <[email protected]>

Author: pytorchbot

.ci/pytorch/check_binary.sh

@@ -59,16 +59,6 @@ else
   export install_root="$(dirname $(which python))/../lib/python${py_dot}/site-packages/torch/"
 fi
 
-###############################################################################
-# Setup XPU ENV
-###############################################################################
-if [[ "$DESIRED_CUDA" == 'xpu' ]]; then
-  set +u
-  # Refer https://www.intel.com/content/www/us/en/developer/articles/tool/pytorch-prerequisites-for-intel-gpus.html
-  source /opt/intel/oneapi/compiler/latest/env/vars.sh
-  source /opt/intel/oneapi/pti/latest/env/vars.sh
-fi
-
 ###############################################################################
 # Check GCC ABI
 ###############################################################################