Skip to content

Commit b8c2371

Browse files
ZailiWangjianan-gu
ZailiWang
and
jianan-gu
authored
rel 2.5 documentation update (#3339)
* rel2.5 doc update * r2.5 doc update 2 * validated DDP example fix and update BKC * update LLM table and some expressions * Update README.md * Update README.md --------- Co-authored-by: jianan-gu <[email protected]>
1 parent 584a4e2 commit b8c2371

File tree

13 files changed

+239
-198
lines changed

13 files changed

+239
-198
lines changed

‎README.md‎

Lines changed: 32 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ Intel® Extension for PyTorch\*
55

66
</div>
77

8-
**CPU** [💻main branch](https://github.com/intel/intel-extension-for-pytorch/tree/main)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[🌱Quick Start](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/getting_started.html)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[📖Documentations](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[🏃Installation](https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=cpu&version=v2.4.0%2Bcpu)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[💻LLM Example](https://github.com/intel/intel-extension-for-pytorch/tree/main/examples/cpu/llm) <br>
8+
**CPU** [💻main branch](https://github.com/intel/intel-extension-for-pytorch/tree/main)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[🌱Quick Start](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/getting_started.html)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[📖Documentations](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[🏃Installation](https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=cpu&version=v2.5.0%2Bcpu)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[💻LLM Example](https://github.com/intel/intel-extension-for-pytorch/tree/main/examples/cpu/llm) <br>
99
**GPU** [💻main branch](https://github.com/intel/intel-extension-for-pytorch/tree/xpu-main)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[🌱Quick Start](https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/getting_started.html)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[📖Documentations](https://intel.github.io/intel-extension-for-pytorch/xpu/latest/)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[🏃Installation](https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=gpu)&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;[💻LLM Example](https://github.com/intel/intel-extension-for-pytorch/tree/xpu-main/examples/gpu/llm)<br>
1010

1111
Intel® Extension for PyTorch\* extends PyTorch\* with up-to-date features optimizations for an extra performance boost on Intel hardware. Optimizations take advantage of Intel® Advanced Vector Extensions 512 (Intel® AVX-512) Vector Neural Network Instructions (VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel X<sup>e</sup> Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs through the PyTorch* xpu device.
@@ -18,49 +18,47 @@ In the current technological landscape, Generative AI (GenAI) workloads and mode
1818

1919
| MODEL FAMILY | MODEL NAME (Huggingface hub) | FP32 | BF16 | Static quantization INT8 | Weight only quantization INT8 | Weight only quantization INT4 |
2020
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
21-
|LLAMA| meta-llama/Llama-2-7b-hf | 🟩 | 🟩 | 🟨 | 🟩 | 🟨 |
21+
|LLAMA| meta-llama/Llama-2-7b-hf | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
2222
|LLAMA| meta-llama/Llama-2-13b-hf | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
2323
|LLAMA| meta-llama/Llama-2-70b-hf | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
24-
|LLAMA| meta-llama/Meta-Llama-3-8B | 🟩 | 🟩 | 🟨 | 🟩 | 🟨 |
25-
|LLAMA| meta-llama/Meta-Llama-3-70B | 🟩 | 🟩 | 🟨 | 🟩 | 🟩 |
26-
|LLAMA| meta-llama/Meta-Llama-3.1-8B-Instruct | 🟩 | 🟩 | 🟨 | 🟩 | 🟩 |
24+
|LLAMA| meta-llama/Meta-Llama-3-8B | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
25+
|LLAMA| meta-llama/Meta-Llama-3-70B | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
26+
|LLAMA| meta-llama/Meta-Llama-3.1-8B-Instruct | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
27+
|LLAMA| meta-llama/Llama-3.2-3B-Instruct | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
28+
|LLAMA| meta-llama/Llama-3.2-11B-Vision-Instruct | 🟩 | 🟩 | | 🟩 | |
2729
|GPT-J| EleutherAI/gpt-j-6b | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
28-
|GPT-NEOX| EleutherAI/gpt-neox-20b | 🟩 | 🟨 | 🟨 | 🟩 | 🟨 |
29-
|DOLLY| databricks/dolly-v2-12b | 🟩 | 🟨 | 🟨 | 🟩 | 🟨 |
30-
|FALCON| tiiuae/falcon-7b | 🟩 | 🟩 | 🟩 | 🟩 | |
31-
|FALCON| tiiuae/falcon-11b | 🟩 | 🟩 | 🟩 | 🟩 | 🟨 |
30+
|GPT-NEOX| EleutherAI/gpt-neox-20b | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
31+
|DOLLY| databricks/dolly-v2-12b | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
32+
|FALCON| tiiuae/falcon-7b | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
33+
|FALCON| tiiuae/falcon-11b | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
3234
|FALCON| tiiuae/falcon-40b | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
33-
|OPT| facebook/opt-30b | 🟩 | 🟩 | 🟩 | 🟩 | 🟨 |
34-
|OPT| facebook/opt-1.3b | 🟩 | 🟩 | 🟩 | 🟩 | 🟨 |
35-
|Bloom| bigscience/bloom-1b7 | 🟩 | 🟨 | 🟩 | 🟩 | 🟨 |
35+
|OPT| facebook/opt-30b | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
36+
|OPT| facebook/opt-1.3b | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
37+
|Bloom| bigscience/bloom-1b7 | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
3638
|CodeGen| Salesforce/codegen-2B-multi | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
37-
|Baichuan| baichuan-inc/Baichuan2-7B-Chat | 🟩 | 🟩 | 🟩 | 🟩 | 🟨 |
38-
|Baichuan| baichuan-inc/Baichuan2-13B-Chat | 🟩 | 🟩 | 🟨 | 🟩 | 🟨 |
39-
|Baichuan| baichuan-inc/Baichuan-13B-Chat | 🟩 | 🟨 | 🟩 | 🟩 | 🟨 |
40-
|ChatGLM| THUDM/chatglm3-6b | 🟩 | 🟩 | 🟨 | 🟩 | 🟨 |
41-
|ChatGLM| THUDM/chatglm2-6b | 🟩 | 🟩 | 🟩 | 🟩 | 🟨 |
42-
|GPTBigCode| bigcode/starcoder | 🟩 | 🟩 | 🟨 | 🟩 | 🟨 |
43-
|T5| google/flan-t5-xl | 🟩 | 🟩 | 🟨 | 🟩 | |
39+
|Baichuan| baichuan-inc/Baichuan2-7B-Chat | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
40+
|Baichuan| baichuan-inc/Baichuan2-13B-Chat | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
41+
|Baichuan| baichuan-inc/Baichuan-13B-Chat | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
42+
|ChatGLM| THUDM/chatglm3-6b | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
43+
|ChatGLM| THUDM/chatglm2-6b | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
44+
|GPTBigCode| bigcode/starcoder | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
45+
|T5| google/flan-t5-xl | 🟩 | 🟩 | 🟩 | 🟩 | |
4446
|MPT| mosaicml/mpt-7b | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
45-
|Mistral| mistralai/Mistral-7B-v0.1 | 🟩 | 🟩 | 🟨 | 🟩 | 🟨 |
46-
|Mixtral| mistralai/Mixtral-8x7B-v0.1 | 🟩 | 🟩 | | 🟩 | 🟨 |
47-
|Stablelm| stabilityai/stablelm-2-1_6b | 🟩 | 🟩 | 🟨 | 🟩 | 🟨 |
48-
|Qwen| Qwen/Qwen-7B-Chat | 🟩 | 🟩 | 🟨 | 🟩 | 🟨 |
49-
|Qwen| Qwen/Qwen2-7B | 🟩 | 🟩 | 🟨 | 🟩 | 🟨 |
47+
|Mistral| mistralai/Mistral-7B-v0.1 | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
48+
|Mixtral| mistralai/Mixtral-8x7B-v0.1 | 🟩 | 🟩 | | 🟩 | 🟩 |
49+
|Stablelm| stabilityai/stablelm-2-1_6b | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
50+
|Qwen| Qwen/Qwen-7B-Chat | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
51+
|Qwen| Qwen/Qwen2-7B | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
5052
|LLaVA| liuhaotian/llava-v1.5-7b | 🟩 | 🟩 | | 🟩 | 🟩 |
5153
|GIT| microsoft/git-base | 🟩 | 🟩 | | 🟩 | |
52-
|Yuan| IEITYuan/Yuan2-102B-hf | 🟩 | 🟩 | | 🟨 | |
53-
|Phi| microsoft/phi-2 | 🟩 | 🟩 | 🟩 | 🟩 | 🟨 |
54-
|Phi| microsoft/Phi-3-mini-4k-instruct | 🟩 | 🟩 | 🟨 | 🟩 | 🟨 |
55-
|Phi| microsoft/Phi-3-mini-128k-instruct | 🟩 | 🟩 | 🟨 | 🟩 | 🟨 |
56-
|Phi| microsoft/Phi-3-medium-4k-instruct | 🟩 | 🟩 | 🟨 | 🟩 | 🟨 |
57-
|Phi| microsoft/Phi-3-medium-128k-instruct | 🟩 | 🟩 | 🟨 | 🟩 | 🟨 |
54+
|Yuan| IEITYuan/Yuan2-102B-hf | 🟩 | 🟩 | | 🟩 | |
55+
|Phi| microsoft/phi-2 | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
56+
|Phi| microsoft/Phi-3-mini-4k-instruct | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
57+
|Phi| microsoft/Phi-3-mini-128k-instruct | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
58+
|Phi| microsoft/Phi-3-medium-4k-instruct | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
59+
|Phi| microsoft/Phi-3-medium-128k-instruct | 🟩 | 🟩 | 🟩 | 🟩 | 🟩 |
5860
|Whisper| openai/whisper-large-v2 | 🟩 | 🟩 | 🟩 | 🟩 | |
5961

60-
- 🟩 signifies that the model can perform well and with good accuracy (<1% difference as compared with FP32).
61-
62-
- 🟨 signifies that the model can perform well while accuracy may not been in a perfect state (>1% difference as compared with FP32).
63-
6462
*Note*: The above verified models (including other models in the same model family, like "codellama/CodeLlama-7b-hf" from LLAMA family) are well supported with all optimizations like indirect access KV cache, fused ROPE, and customized linear kernels.
6563
We are working in progress to better support the models in the tables with various data types. In addition, more models will be optimized in the future.
6664

‎docker/README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,24 +10,25 @@
1010

1111
```console
1212
$ cd $DOCKERFILE_DIR
13-
$ DOCKER_BUILDKIT=1 docker build -f Dockerfile.prebuilt -t intel-extension-for-pytorch:main .
13+
$ DOCKER_BUILDKIT=1 docker build -f Dockerfile.prebuilt -t intel-extension-for-pytorch:2.5.0 .
1414
```
1515

1616
Run the following commands to build a `conda` based container with Intel® Extension for PyTorch\* compiled from source:
1717

1818
```console
1919
$ git clone https://github.com/intel/intel-extension-for-pytorch.git
2020
$ cd intel-extension-for-pytorch
21+
$ git checkout v2.5.0+cpu
2122
$ git submodule sync
2223
$ git submodule update --init --recursive
23-
$ DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.compile -t intel-extension-for-pytorch:main .
24+
$ DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.compile -t intel-extension-for-pytorch:2.5.0 .
2425
```
2526

2627
* Sanity Test
2728

2829
When a docker image is built out, Run the command below to launch into a container:
2930
```console
30-
$ docker run --rm -it intel-extension-for-pytorch:main bash
31+
$ docker run --rm -it intel-extension-for-pytorch:2.5.0 bash
3132
```
3233

3334
Then run the command below inside the container to verify correct installation.

‎docs/_static/htmls/tbl_deepspeed.html‎

Lines changed: 18 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,18 @@
4444
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
4545
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
4646
</tr>
47+
<tr class="row-even">
48+
<td><p>LLAMA</p></td>
49+
<td><p>meta-llama/Llama-3.2-3B-Instruct</p></td>
50+
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
51+
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
52+
</tr>
53+
<tr class="row-odd">
54+
<td><p>LLAMA</p></td>
55+
<td><p>meta-llama/Llama-3.2-11B-Vision-Instruct</p></td>
56+
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
57+
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
58+
</tr>
4759
<tr class="row-even">
4860
<td><p>GPT-J</p></td>
4961
<td><p>EleutherAI/gpt-j-6b</p></td>
@@ -53,13 +65,13 @@
5365
<tr class="row-odd">
5466
<td><p>GPT-NEOX</p></td>
5567
<td><p>EleutherAI/gpt-neox-20b</p></td>
56-
<td><p style="text-align: center; vertical-align: middle;">🟨</p></td>
68+
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
5769
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
5870
</tr>
5971
<tr class="row-even">
6072
<td><p>DOLLY</p></td>
6173
<td><p>databricks/dolly-v2-12b</p></td>
62-
<td><p style="text-align: center; vertical-align: middle;">🟨</p></td>
74+
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
6375
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
6476
</tr>
6577
<tr class="row-odd">
@@ -77,7 +89,7 @@
7789
<tr class="row-odd">
7890
<td><p>OPT</p></td>
7991
<td><p>facebook/opt-30b</p></td>
80-
<td><p style="text-align: center; vertical-align: middle;">🟨</p></td>
92+
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
8193
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
8294
</tr>
8395
<tr class="row-even">
@@ -89,7 +101,7 @@
89101
<tr class="row-odd">
90102
<td><p>Bloom</p></td>
91103
<td><p>bigscience/bloom-1b7</p></td>
92-
<td><p style="text-align: center; vertical-align: middle;">🟨</p></td>
104+
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
93105
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
94106
</tr>
95107
<tr class="row-even">
@@ -113,7 +125,7 @@
113125
<tr class="row-odd">
114126
<td><p>Baichuan</p></td>
115127
<td><p>baichuan-inc/Baichuan-13B-Chat</p></td>
116-
<td><p style="text-align: center; vertical-align: middle;">🟨</p></td>
128+
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
117129
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
118130
</tr>
119131
<tr class="row-even">
@@ -207,8 +219,4 @@
207219
<td><p style="text-align: center; vertical-align: middle;">🟩</p></td>
208220
</tr>
209221
</tbody>
210-
</table>
211-
<ul class="simple">
212-
<li><p>🟩 signifies that the model can perform well and with good accuracy (&lt;1% difference as compared with FP32).</p></li>
213-
<li><p>🟨 signifies that the model can perform well while accuracy may not been in a perfect state (&gt;1% difference as compared with FP32).</p></li>
214-
</ul>
222+
</table>

0 commit comments

Comments
 (0)