[E2E] Align env with torch-xpu-ops; uninstall fbgemm_gpu_nightly-cpu (#5523)

Signed-off-by: Anatoly Myachev <[email protected]>

Author: anmyachev

.github/workflows/e2e-reusable.yml

@@ -194,6 +194,7 @@ jobs:
           # for dlrm
           pip install pyre-extensions
           curl -fsSL https://raw.githubusercontent.com/facebookresearch/dlrm/refs/heads/torchrec-dlrm/requirements.txt |xargs pip install
+          pip uninstall -y fbgemm_gpu_nightly-cpu
           # for soft_actor_critic, temp fix
           pip install git+https://github.com/nocoding03/gym@fix-np