Doc update for r2.6 (#3453)

ZailiWang · Xia-Weiwen · web-flow · commit c392d59395cc · 2025-01-20T13:06:39.000+08:00
* doc update for r2.6

* source build DeepSpeed with the specified commit

* empty DS version and add note to reminder users about env_setup.sh param change

* deprecate serving examples

* Update deepspeed UT for deepspeed upgrade

---------

Co-authored-by: Xia, Weiwen &lt;weiwen.xia@intel.com&gt;
diff --git a/README.md b/README.md
@@ -5,7 +5,7 @@ Intel® Extension for PyTorch\*
 
 </div>
 
-**CPU** [💻main branch](https://github.com/intel/intel-extension-for-pytorch/tree/main)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[🌱Quick Start](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/getting_started.html)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[📖Documentations](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[🏃Installation](https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=cpu&version=v2.5.0%2Bcpu)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[💻LLM Example](https://github.com/intel/intel-extension-for-pytorch/tree/main/examples/cpu/llm) <br>
+**CPU** [💻main branch](https://github.com/intel/intel-extension-for-pytorch/tree/main)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[🌱Quick Start](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/getting_started.html)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[📖Documentations](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[🏃Installation](https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=cpu&version=v2.6.0%2Bcpu)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[💻LLM Example](https://github.com/intel/intel-extension-for-pytorch/tree/release/2.6/examples/cpu/llm) <br>
 **GPU** [💻main branch](https://github.com/intel/intel-extension-for-pytorch/tree/xpu-main)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[🌱Quick Start](https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/getting_started.html)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[📖Documentations](https://intel.github.io/intel-extension-for-pytorch/xpu/latest/)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[🏃Installation](https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=gpu)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[💻LLM Example](https://github.com/intel/intel-extension-for-pytorch/tree/xpu-main/examples/gpu/llm)<br>  
 
 Intel® Extension for PyTorch\* extends PyTorch\* with up-to-date features optimizations for an extra performance boost on Intel hardware. Optimizations take advantage of Intel® Advanced Vector Extensions 512 (Intel® AVX-512) Vector Neural Network Instructions (VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel X<sup>e</sup> Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs through the PyTorch* xpu device.
diff --git a/dependency_version.json b/dependency_version.json
@@ -20,8 +20,8 @@
     "version": "2.6.0+cpu"
   },
   "deepspeed": {
-    "version": "0.16.0",
-    "commit": "v0.16.0"
+    "version": "",
+    "commit": "018ece5af2d89a11a4a235f81f94496c78b4f990"
   },
   "oneCCL": {
     "commit": "2021.12"
diff --git a/docker/Dockerfile.prebuilt b/docker/Dockerfile.prebuilt
@@ -35,11 +35,11 @@ RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 100
 
 WORKDIR /root
 
-ARG IPEX_VERSION=2.5.0
-ARG TORCHCCL_VERSION=2.5.0
-ARG PYTORCH_VERSION=2.5.0
-ARG TORCHAUDIO_VERSION=2.5.0
-ARG TORCHVISION_VERSION=0.20.0
+ARG IPEX_VERSION=2.6.0
+ARG TORCHCCL_VERSION=2.6.0
+ARG PYTORCH_VERSION=2.6.0
+ARG TORCHAUDIO_VERSION=2.6.0
+ARG TORCHVISION_VERSION=0.21.0
 RUN . ./venv/bin/activate && \
     python -m pip --no-cache-dir install --upgrade \
     pip \
diff --git a/docker/README.md b/docker/README.md
@@ -10,24 +10,25 @@
 
   ```console
   $ cd $DOCKERFILE_DIR
-  $ DOCKER_BUILDKIT=1 docker build -f Dockerfile.prebuilt -t intel-extension-for-pytorch:main .
+  $ DOCKER_BUILDKIT=1 docker build -f Dockerfile.prebuilt -t intel-extension-for-pytorch:2.6.0 .
   ```
 
   Run the following commands to build a `conda` based container with Intel® Extension for PyTorch\* compiled from source:
 
   ```console
   $ git clone https://github.com/intel/intel-extension-for-pytorch.git
   $ cd intel-extension-for-pytorch
+  $ git checkout v2.6.0+cpu
   $ git submodule sync
   $ git submodule update --init --recursive
-  $ DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.compile -t intel-extension-for-pytorch:main .
+  $ DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.compile -t intel-extension-for-pytorch:2.6.0 .
   ```
 
 * Sanity Test
 
   When a docker image is built out, Run the command below to launch into a container:
   ```console
-  $ docker run --rm -it intel-extension-for-pytorch:main bash
+  $ docker run --rm -it intel-extension-for-pytorch:2.6.0 bash
   ```
 
   Then run the command below inside the container to verify correct installation.
diff --git a/docs/tutorials/getting_started.md b/docs/tutorials/getting_started.md
@@ -1,6 +1,6 @@
 # Quick Start
 
-The following instructions assume you have installed the Intel® Extension for PyTorch\*. For installation instructions, refer to [Installation](../../../index.html#installation?platform=cpu&version=main).
+The following instructions assume you have installed the Intel® Extension for PyTorch\*. For installation instructions, refer to [Installation](../../../index.html#installation?platform=cpu&version=v2.6.0%2Bcpu).
 
 To start using the Intel® Extension for PyTorch\* in your code, you need to make the following changes:
 
@@ -157,4 +157,4 @@ with torch.inference_mode(), torch.cpu.amp.autocast(enabled=amp_enabled):
     print(gen_text, total_new_tokens, flush=True)
 ```
 
-More LLM examples, including usage of low precision data types are available in the [LLM Examples](https://github.com/intel/intel-extension-for-pytorch/tree/main/examples/cpu/llm) section.
+More LLM examples, including usage of low precision data types are available in the [LLM Examples](https://github.com/intel/intel-extension-for-pytorch/tree/release/2.6/examples/cpu/llm) section.
diff --git a/docs/tutorials/installation.md b/docs/tutorials/installation.md
@@ -1,8 +1,8 @@
 Installation
 ============
 
-Select your preferences and follow the installation instructions provided on the [Installation page](../../../index.html#installation?platform=cpu&version=v2.5.0%2Bcpu).
+Select your preferences and follow the installation instructions provided on the [Installation page](../../../index.html#installation?platform=cpu&version=v2.6.0%2Bcpu).
 
 After successful installation, refer to the [Quick Start](getting_started.md) and [Examples](examples.md) sections to start using the extension in your code.
 
-**NOTE:** For detailed instructions on installing and setting up the environment for Large Language Models (LLM), as well as example scripts, refer to the [LLM best practices](https://github.com/intel/intel-extension-for-pytorch/tree/v2.5.0%2Bcpu/examples/cpu/llm).
+**NOTE:** For detailed instructions on installing and setting up the environment for Large Language Models (LLM), as well as example scripts, refer to the [LLM best practices](https://github.com/intel/intel-extension-for-pytorch/tree/v2.6.0%2Bcpu/examples/cpu/llm).
diff --git a/docs/tutorials/introduction.rst b/docs/tutorials/introduction.rst
@@ -16,7 +16,7 @@ the `Large Language Models (LLM) <llm.html>`_ section.
 
 Get Started
 -----------
-- `Installation <../../../index.html#installation?platform=cpu&version=v2.5.0%2Bcpu>`_
+- `Installation <../../../index.html#installation?platform=cpu&version=v2.6.0%2Bcpu>`_
 - `Quick Start <getting_started.md>`_
 - `Examples <examples.md>`_
 
diff --git a/docs/tutorials/llm.rst b/docs/tutorials/llm.rst
@@ -30,7 +30,7 @@ Verified for distributed inference mode via DeepSpeed
 
 *Note*: The above verified models (including other models in the same model family, like "codellama/CodeLlama-7b-hf" from LLAMA family) are well supported with all optimizations like indirect access KV cache, fused ROPE, and customized linear kernels. We are working in progress to better support the models in the tables with various data types. In addition, more models will be optimized in the future.
 
-Please check `LLM best known practice <https://github.com/intel/intel-extension-for-pytorch/tree/main/examples/cpu/llm>`_ for instructions to install/setup environment and example scripts.
+Please check `LLM best known practice <https://github.com/intel/intel-extension-for-pytorch/tree/release/2.6/examples/cpu/llm>`_ for instructions to install/setup environment and example scripts.
 
 Module Level Optimization API for customized LLM (Prototype)
 ------------------------------------------------------------
diff --git a/examples/cpu/inference/cpp/README.md b/examples/cpu/inference/cpp/README.md
@@ -16,15 +16,15 @@ We can have `libtorch` and `libintel-ext-pt` installed via the following command
 Download zip file of `libtorch` and decompress it:
 
 ```bash
-wget https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-2.5.0%2Bcpu.zip
-unzip libtorch-cxx11-abi-shared-with-deps-2.5.0+cpu.zip
+wget https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-2.6.0%2Bcpu.zip
+unzip libtorch-cxx11-abi-shared-with-deps-2.6.0+cpu.zip
 ```
 
 Download and execute `libintel-ext-pt` installation script:
 
 ```bash
-wget https://intel-extension-for-pytorch.s3.amazonaws.com/libipex/cpu/libintel-ext-pt-cxx11-abi-2.5.0%2Bcpu.run
-bash libintel-ext-pt-cxx11-abi-2.5.0+cpu.run install ./libtorch
+wget https://intel-extension-for-pytorch.s3.amazonaws.com/libipex/cpu/libintel-ext-pt-cxx11-abi-2.6.0%2Bcpu.run
+bash libintel-ext-pt-cxx11-abi-2.6.0+cpu.run install ./libtorch
 ```
 
 *Note:* If your C++ project has pre-C\+\+11 library dependencies,
@@ -59,4 +59,4 @@ Run the executable file:
 ./example-app ../resnet50.pt
 ```
 
-Please view the [c++ example in Intel® Extension for PyTorch\* online document](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/examples.html#c) for more information.
+Please view the [C++ example in Intel® Extension for PyTorch\* online document](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/examples.html#c) for more information.
diff --git a/examples/cpu/llm/Dockerfile b/examples/cpu/llm/Dockerfile
@@ -43,7 +43,7 @@ COPY . ./intel-extension-for-pytorch
 RUN . ~/miniforge3/bin/activate && conda create -y -n compile_py310 python=3.10 && conda activate compile_py310 && \
     cd intel-extension-for-pytorch/examples/cpu/llm && \
     export CC=gcc && export CXX=g++ && \
-    if [ -z ${COMPILE} ]; then bash tools/env_setup.sh 6; else bash tools/env_setup.sh 2; fi && \
+    if [ -z ${COMPILE} ]; then bash tools/env_setup.sh 14; else bash tools/env_setup.sh 10; fi && \
     unset CC && unset CXX
 
 FROM base AS deploy
@@ -60,7 +60,7 @@ COPY --from=dev /root/intel-extension-for-pytorch/tools/get_libstdcpp_lib.sh ./l
 RUN . ~/miniforge3/bin/activate && conda create -y -n py310 python=3.10 && conda activate py310 && \
     cd /usr/lib/x86_64-linux-gnu/ && ln -s libtcmalloc.so.4 libtcmalloc.so && cd && \
     cd ./llm && \
-    bash tools/env_setup.sh 1 && \
+    bash tools/env_setup.sh 9 && \
     python -m pip cache purge && \
     mv ./oneCCL_release /opt/oneCCL && \
     chown -R root:root /opt/oneCCL && \
diff --git a/examples/cpu/llm/README.md b/examples/cpu/llm/README.md
@@ -7,24 +7,75 @@ And a set of data types are supported for various scenarios, including FP32, BF1
 
 # 2. Environment Setup
 
-*Note*: The instructions in this section will setup an environment with a recent PyTorch\* nightly build and **a latest source build of IPEX**. 
-If you would like to use stable PyTorch\* and IPEX release versions, please refer to the instructions [in the release branch](https://github.com/intel/intel-extension-for-pytorch/blob/v2.5.0%2Bcpu/examples/cpu/llm/README.md#2-environment-setup), in which IPEX is installed via prebuilt wheels using `pip install` rather than source code building.
+**Note**: Please be aware that in order to enable the latest optimizations for MoE models (DeepSeek, Mixtral, etc.) in `DeepSpeed`,
+we are setting a different argument for `env_setup.sh` in IPEX v2.6.0+cpu comparing with previous versions,
+in order to build `DeepSpeed` from source code with a recent commit.
 
-## 2.1 [Recommended] Docker-based environment setup with compilation from source
+## 2.1 [RECOMMENDED] Docker-based environment setup with pre-built wheels
 
 ```bash
 # Get the Intel® Extension for PyTorch\* source code
 git clone https://github.com/intel/intel-extension-for-pytorch.git
 cd intel-extension-for-pytorch
+git checkout v2.6.0+cpu
+git submodule sync
+git submodule update --init --recursive
+
+# Build an image with the provided Dockerfile by installing from Intel® Extension for PyTorch\* prebuilt wheel files
+# To have a custom ssh server port for multi-nodes run, please add --build-arg PORT_SSH=<CUSTOM_PORT> ex: 2345, otherwise use the default 22 SSH port
+DOCKER_BUILDKIT=1 docker build -f examples/cpu/llm/Dockerfile --build-arg PORT_SSH=2345 -t ipex-llm:2.6.0 .
+
+# Run the container with command below
+docker run --rm -it --privileged -v /dev/shm:/dev/shm ipex-llm:2.6.0 bash
+
+# When the command prompt shows inside the docker container, enter llm examples directory
+cd llm
+
+# Activate environment variables
+# set bash script argument to "inference" or "fine-tuning" for different usages
+source ./tools/env_activate.sh [inference|fine-tuning]
+```
+
+## 2.2 Conda-based environment setup with pre-built wheels
+
+```bash
+# Get the Intel® Extension for PyTorch\* source code
+git clone https://github.com/intel/intel-extension-for-pytorch.git
+cd intel-extension-for-pytorch
+git checkout v2.6.0+cpu
+git submodule sync
+git submodule update --init --recursive
+
+# GCC 12.3 is required. Installation can be taken care of by the environment configuration script.
+# Create a conda environment
+conda create -n llm python=3.10 -y
+conda activate llm
+
+# Setup the environment with the provided script
+cd examples/cpu/llm
+bash ./tools/env_setup.sh 15
+
+# Activate environment variables
+# set bash script argument to "inference" or "fine-tuning" for different usages
+source ./tools/env_activate.sh [inference|fine-tuning]
+```
+
+## 2.3 Docker-based environment setup with compilation from source
+
+```bash
+# Get the Intel® Extension for PyTorch\* source code
+git clone https://github.com/intel/intel-extension-for-pytorch.git
+cd intel-extension-for-pytorch
+git checkout v2.6.0+cpu
 git submodule sync
 git submodule update --init --recursive
 
 # Build an image with the provided Dockerfile by compiling Intel® Extension for PyTorch\* from source
 # To have a custom ssh server port for multi-nodes run, please add --build-arg PORT_SSH=<CUSTOM_PORT> ex: 2345, otherwise use the default 22 SSH port
-docker build -f examples/cpu/llm/Dockerfile --build-arg COMPILE=ON --build-arg PORT_SSH=2345 -t ipex-llm:main .
+docker build -f examples/cpu/llm/Dockerfile --build-arg COMPILE=ON --build-arg PORT_SSH=2345 -t ipex-llm:2.6.0 .
 
 # Run the container with command below
-docker run --rm -it --net host --privileged -v /dev/shm:/dev/shm ipex-llm:main bash
+docker run --rm -it --net host --privileged -v /dev/shm:/dev/shm ipex-llm:2.6.0 bash
 
 # When the command prompt shows inside the docker container, enter llm examples directory
 cd llm
@@ -34,12 +85,13 @@ cd llm
 source ./tools/env_activate.sh [inference|fine-tuning]
 ```
 
-## 2.2 Conda-based environment setup with compilation from source
+## 2.4 Conda-based environment setup with compilation from source
 
 ```bash
 # Get the Intel® Extension for PyTorch\* source code
 git clone https://github.com/intel/intel-extension-for-pytorch.git
 cd intel-extension-for-pytorch
+git checkout v2.6.0+cpu
 git submodule sync
 git submodule update --init --recursive
 
@@ -50,7 +102,7 @@ conda activate llm
 
 # Setup the environment with the provided script
 cd examples/cpu/llm
-bash ./tools/env_setup.sh
+bash ./tools/env_setup.sh 8
 
 # Activate environment variables
 # set bash script argument to "inference" or "fine-tuning" for different usages
diff --git a/examples/cpu/llm/fine-tuning/README.md b/examples/cpu/llm/fine-tuning/README.md
@@ -34,6 +34,8 @@ mv alpaca.json ./templates
 export LOCAL_BATCH_SIZE=32 #32 is default one, you can choose per need
 export MODEL_NAME_OR_PATH="YOUR LOCAL PATH or MODEL_ID (HF)"
 export MAXSTEP=-1 #default is -1, which means running all steps
+mkdir ./outputs
+export OUTPUT_DIR=./outputs
 
 #[optional] you may need to get access to llama2 weights from HF
 Apply the access in this page [LLaMA2 7B](https://huggingface.co/meta-llama/Llama-2-7b-hf) with your huggingface account
diff --git a/examples/cpu/llm/inference/README.md b/examples/cpu/llm/inference/README.md
@@ -239,7 +239,7 @@ The docker image built in [the environment setup tutorial](../README.md#2-enviro
 Use the command below on all machines to launch the docker containers. This command uses the host network interfaces inside the docker container. Thus, you need to put the host ip addresses into the `hostfile.txt`. Do NOT launch multiple docker containers on one single machine from the same docker image. These docker containers listen on the same machine on the same port, will result in unpredicable ssh connections.
 
 ```bash
-docker run --rm -it --privileged -v /dev/shm:/dev/shm --net host ipex-llm:main bash
+docker run --rm -it --privileged -v /dev/shm:/dev/shm --net host ipex-llm:2.6.0 bash
 ```
 
 **Note:** For models on HuggingFace require access privileges, you need to run the `huggingface-cli login` command in each docker container to config a HuggingFace access token.
@@ -508,7 +508,7 @@ To convert model and run inference with INT4 checkpoints, you can run the follow
 OMP_NUM_THREADS=<physical cores num> numactl -m <node N> -C <cpu list> python run.py -m <MODEL_ID> --benchmark --ipex-weight-only-quantization --quant-with-amp --low-precision-checkpoint <PATH_TO_CHECKPOINT>
 ```
 
-To run accuracy tests, please follow the instructions in the [Accuracy Test](#32-accuracy-test) part
+To run accuracy tests, please follow the instructions in the [Accuracy Test](#32-accuracy-test) part.
 
 **Example**
 
diff --git a/examples/cpu/serving/torchserve/README.md b/examples/cpu/serving/torchserve/README.md
@@ -1,4 +1,4 @@
-# Serving ResNet50 INT8 model with TorchServe and Intel® Extension for PyTorch optimizations
+# Serving ResNet50 INT8 model with TorchServe and Intel® Extension for PyTorch optimizations (Deprecated)
 
 ## Description
 This sample provides code to integrate Intel® Extension for PyTorch with TorchServe. This project quantizes a ResNet50 model to use the INT8 Precision to improve performance on CPU.
diff --git a/examples/cpu/serving/triton/README.md b/examples/cpu/serving/triton/README.md
@@ -1,4 +1,4 @@
-# Serving BERT models with Triton Server and Intel® Extension for PyTorch optimizations
+# Serving BERT models with Triton Server and Intel® Extension for PyTorch optimizations (Deprecated)
 
 ## Description
 This sample provide code to integrate Intel® Extension for PyTorch with Triton Inference Server framework. This project provides custom Python backend for Intel® Extension for PyTorch and additional dynamic batching algorithm to improve the performance. This code can be used as performance benchmark for Bert-Base and Bert-Large models.
diff --git a/examples/cpu/training/python-scripts/README.md b/examples/cpu/training/python-scripts/README.md
@@ -27,6 +27,7 @@ Running ResNet50 distributed training example:
 
 ```bash
 source $(python -c "import intel_extension_for_pytorch;import oneccl_bindings_for_pytorch as torch_ccl;print(torch_ccl.cwd)")/env/setvars.sh
+export FI_TCP_IFACE="$(ip -o -4 route show to default | awk '{print $5}')"
 # This example command would utilize all the numa sockets of the processor, taking each socket as a rank.
 ipexrun --nnodes 1 distributed_data_parallel_training.py
 ```
diff --git a/scripts/compile_bundle.sh b/scripts/compile_bundle.sh
@@ -1,7 +1,7 @@
 #!/bin/bash
 set -eo pipefail
 
-VER_IPEX=main
+VER_IPEX=v2.6.0+cpu
 
 # Mode: Select which components to install. PyTorch and Intel® Extension for PyTorch* are always installed.
 # High bit: 8 7 6 5 4 3 2 1 :Low bit
diff --git a/tests/cpu/test_deepspeed.py b/tests/cpu/test_deepspeed.py
