Skip to content

Commit 67c5b59

Browse files
Duyi-WangDuyi-Wang
authored
[Docs] Add english documents for BF16 and oneDNN optimization. (#554)
1 parent b7baf8a commit 67c5b59

File tree

4 files changed

+241
-0
lines changed

4 files changed

+241
-0
lines changed

‎docs/docs_en/Operator-Optimization.md‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Optimization of Operator
2+
3+
## Hardware and Software Configuration
4+
5+
Hardware: [Alibaba Cloud ECS general purpose instance family with high clock speeds - **ecs.hfg7.2xlarge**](https://help.aliyun.com/document_detail/25378.html?spm=5176.2020520101.vmBInfo.instanceType.4a944df5PvCcED#hfg7).
6+
7+
CPU number: 8 cores
8+
9+
Baseline version: Tensorflow v1.15.5
10+
11+
Optimized version: DeepRec
12+
13+
Gcc version 7.5.0
14+
15+
## Performance Data
16+
17+
| Op Name | Input Tensor Shape | Baseline Perf (latency/ms) | Optimized Perf (latency/ms) | Speedup |
18+
| ----------------- | -------------------------------------------------------- | -------------------------- | --------------------------- | ------- |
19+
| Select | condition: (1024, 64), x: (1024, 64), y: (1024, 64) | 2.080 | 0.564 | +3.68X |
20+
| Dynamic_stitch | indices: (40, 2500), data: (40, 2500, 64) | 82.14 | 24.77 | +3.31X |
21+
| Transpose | data: (1024, 64) | 1.504 | 0.366 | +4.11X |
22+
| Tile | input: (512, 50), multiples: (2, 50) | 1.68 | 0.125 | +13.44X |
23+
| BiasAddGrad | data: (51200, 512) | 26.84 | 1.67 | +16.07X |
24+
| SparseSegmentMean | data: (51200, 128), indices: (51200), seg index: (51200) | 1.93 | 0.445 | +4.34X |
25+
| Unique | | | | |
26+
| Gather | | | | |
27+
| BiasAdd | | | | |
28+
| where | | | | |
29+
| DynamicPartition | | | | |
30+
| SparseConcat | | | | |
31+
32+
## Case Studies：Select
33+
34+
The computing process of operator Select：
35+
36+
![select.png](../docs_zh/Operator-Optimization/select.png)
37+
38+
TensorFlow original implementation：Broadcast + Elementwise Select
39+
40+
```
41+
template <typename Device, typename T, int NDIMS>
42+
struct BCastSelectFunctorBase {
43+
void operator()(const Device& d,
44+
typename TTypes<T, NDIMS>::Tensor output_tensor,
45+
typename TTypes<bool, NDIMS>::ConstTensor cond_tensor,
46+
typename TTypes<T, NDIMS>::ConstTensor then_tensor,
47+
typename TTypes<T, NDIMS>::ConstTensor else_tensor,
48+
typename Eigen::array<Eigen::DenseIndex, NDIMS> cond_bcast,
49+
typename Eigen::array<Eigen::DenseIndex, NDIMS> then_bcast,
50+
typename Eigen::array<Eigen::DenseIndex, NDIMS> else_bcast) {
51+
output_tensor.device(d) = cond_tensor.broadcast(cond_bcast)
52+
.select(then_tensor.broadcast(then_bcast),
53+
else_tensor.broadcast(else_bcast));
54+
}
55+
};
56+
```
57+
58+
PAI-TF (Merged to Community)：Row Select ,Optimized redundant broadcast operations in the original TensorFlow version.。
59+
60+
```
61+
if (c[i]) {
62+
for (size_t j = 0; j < batch_size; ++j) {
63+
output[offset + j] = t[offset + j];
64+
}
65+
} else {
66+
for (size_t j = 0; j < batch_size; ++j) {
67+
output[offset + j] = e[offset + j];
68+
}
69+
}
70+
```
71+
72+
DeepRec: vectorized Row Select, used AVX512 mask vectorisation instructions for the further optimizing of select operation, which improved the performance of this operator by 3.68x.
73+
74+
```
75+
__mmask16 cmask = (c[i] == false) ? 0xffff : 0x0000; // select t/e
76+
size_t ofs = 0;
77+
78+
for (size_t j = 0; j < quotient; ++j) {
79+
__m512 src = _mm512_loadu_ps(t + offset + ofs);
80+
__m512 tmp = _mm512_mask_loadu_ps(src, cmask, e + offset + ofs);
81+
_mm512_storeu_ps(output + offset + ofs, tmp);
82+
ofs += float_alignment;
83+
}
84+
85+
if (remainder != 0) {
86+
__mmask16 mask = (remainder >= float_alignment)
87+
? 0xffff : 0xffff >> (float_alignment - remainder);
88+
cmask &= mask;
89+
__m512 src = _mm512_mask_loadu_ps(_mm512_setzero_ps(), mask, t + offset + ofs);
90+
__m512 tmp = _mm512_mask_loadu_ps(src, cmask, e + offset + ofs);
91+
_mm512_mask_storeu_ps(output + offset + ofs, mask, tmp);
92+
}
93+
```
94+

‎docs/docs_en/bf16.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# BFloat16
2+
3+
BFloat16 (BF16) is a computational format and the instruction for accelerating deep learning training and inference, which is supported on the third-generation Intel® Xeon® Scalable processor Cooper Lake [AliCloud hfg7 specification family](https://help.aliyun.com/document_detail/25378.html?spm=5176.2020520101.vmBInfo.instanceType.4a944df5PvCcED#hfg7) and its successor processors. Below shows the comparison with other commonly used data formats:
4+
5+
![img_1.png](../docs_zh/oneDNN/BF16.png)
6+
7+
## Requirments and methods
8+
9+
Requirements：The cloud instance requires to be the third-generation Intel® Xeon® Scalable processor Cooper Lake [AliCloud hfg7 specification family](https://help.aliyun.com/document_detail/25378.html?spm=5176.2020520101.vmBInfo.instanceType.4a944df5PvCcED#hfg7). It also requires to use DeepRec which is compiled and optimized by oneDNN in order to provide BF16 instruction acceleration, details of which can be found in the oneDNN section.
10+
11+
Method：As the recommended scenarios are extremely demanding in terms of model accuracy, in order to improve model performance while taking into account model accuracy, users could control the BF16 computing graph freely in the following way:
12+
13+
- Step 1: Add `.keep_weights(dtype=tf.float32)` after `tf.variable_scope(…)` to keep the current weights as FP32.
14+
- Step 2: Add `tf.cast(…, dtype=tf.bfloat16)` to transfer the input tensors to BF16 type.
15+
- Step3: Add `tf.cast(…, dtype=tf.float32)` to transfer the output tensors to FP32 type.
16+
17+
```
18+
with tf.variable_scope(…).keep_weights(dtype=tf.float32):
19+
inputs_bf16 = tf.cast(inputs, dtype=tf.bfloat16)
20+
… // BF16 graph, FP32 weights
21+
outputs = tf.cast(outputs_bf16, dtype=tf.float32)
22+
```
23+
24+
Example：
25+
26+
```
27+
import tensorflow as tf
28+
29+
inputs = tf.ones([4, 8], tf.float32)
30+
31+
with tf.variable_scope('dnn', reuse=tf.AUTO_REUSE).keep_weights(dtype=tf.float32):
32+
# cast inputs to BF16
33+
inputs = tf.cast(inputs, dtype=tf.bfloat16)
34+
outputs = tf.layers.dense(inputs, units=8, activation=tf.nn.relu)
35+
outputs = tf.layers.dense(inputs, units=1, activation=None)
36+
# cast ouputs to FP32
37+
outputs = tf.cast(outputs, dtype=tf.float32)
38+
39+
outputs = tf.nn.softmax(outputs)
40+
41+
with tf.Session() as sess:
42+
sess.run(tf.global_variables_initializer())
43+
print(sess.run(outputs))
44+
```
45+
46+
Special Reminder: according to the experience of parameters tuning, usually the last layer of DNN network in a multi-layer DNN network has the most impact on the accuracy of the model, which occupies a lower computational ratio. So the last layer of DNN network can be converted to FP32 type to run, which can improve the computational performance of the model training while preserving the accuracy of the model.
47+
48+
To maintain consistency with the accuracy of the model without BF16 optimization, DeepRec provides the `keep_weights(dtype=dtypes.float32)` method in variable_scope. With this method, all variables in this variable field will be saved in FP32 format, which significantly reduce the cumulative summation error of variables. And the cast operation is automatically added to the graph, converting it to BF16 format for computation. To reduce the extra computational overhead of the cast operation introduced, DeepRec automatically fuses the cast operator with its nearest operator to improve the operation performance. DeepRec will perform the following fusion operations on cast-related operators.
49+
50+
- MatMul + Cast
51+
- Concat + Cast
52+
- Split + Cast
53+
54+
## Performance comparison
55+
56+
Use models in DeepRec Modelzoo to compare the DeepRec with BF16 and FP32 to see the performance improvement. Models in Modelzoo can enable the BF16 feature by adding `--bf16` parameter.
57+
58+
Use Aliyun ECS cloud server as benchmark machine, Intel 3rd Xeon Scalable Processor(Cooper Lake) specification [ecs.hfg7.2xlarge](https://help.aliyun.com/document_detail/25378.html?spm=5176.2020520101.vmBInfo.instanceType.4a944df5PvCcED#hfg7)
59+
60+
- Hardware configuration：
61+
- Intel(R) Xeon(R) Platinum 8369HC CPU @ 3.30GHz
62+
- CPU(s): 8
63+
- Socket(s): 1
64+
- Core(s) per socket: 4
65+
- Thread(s) per core: 2
66+
- Memory: 32G
67+
- Software configuration：
68+
- kernel: 4.18.0-348.2.1.el8_5.x86_64
69+
- OS: CentOS Linux release 8.5.2111
70+
- GCC: 8.5.0
71+
- Docker: 20.10.12
72+
- Python: 3.6.8
73+
74+
Performance Result：
75+
76+
| **Throughput** | **WDL** | **DeepFM** | **DSSM** |
77+
| -------------- | -------- | ---------- | --------- |
78+
| FP32 | 15792.49 | 30718.6 | 114436.87 |
79+
| FP32+BF16 | 22633.8 | 34554.67 | 125995.83 |
80+
| Speedup | 1.43x | 1.12x | 1.10x |
81+
82+
BF16 has little effect on the AUC metric of model training, more details of the difference can be found in the documentation of each model in the model zoo.

‎docs/docs_en/index.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,20 @@ Sparse model is a type of deep learning model that accounts for a relatively hig
1212

1313
DeepRec has been deeply cultivated since 2016, which supports core businesses such as Taobao Search, recommendation and advertising. It precipitates a list of features on basic frameworks and has excellent performance in sparse models training. Facing a wide variety of external needs and the environment of deep learning framework embracing open source, DeepeRec open source is conducive to establishing standardized interfaces, cultivating user habits, greatly reducing the cost of external customers working on cloud and establishing the brand value.
1414

15+
# Getting started
16+
17+
# Features
18+
```{toctree}
19+
:maxdepth: 2
20+
:caption: Operator & Hardware Acceleration
21+
22+
oneDNN
23+
Operator-Optimization
24+
```
25+
26+
```{toctree}
27+
:maxdepth: 2
28+
:caption: Model Quantification
29+
30+
BFloat16
31+
```

‎docs/docs_en/oneDNN.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# oneDNN
2+
3+
## Introduction
4+
5+
[oneDNN](https://github.com/oneapi-src/oneDNN) is the open source cross-platform performance acceleration library for deep learning from Intel, The [documentation](https://oneapi-src.github.io/oneDNN/) guides you to find out which primitives are supported. OneDNN has been integrated into DeepRec, which can be enabled by adding the compiling option in the compile command. `--config=mkl_threadpool` is used to enable oneDNN accelerated arithmetic computation. Adding the compiling option `--config=opt` will enable the optimization of `--copt=-march=native`, which can further accelerate arithmetic performance on the CPU which supports AVX512, for example, Skylake, Caslake and Icelake.
6+
7+
8+
9+
Tips: MKL was first renamed as DNNL and then renamed as oneDNN. Tensorflow initially used MKL to accelerate the computation of the operators, and in subsequent versions of iteration, oneDNN gradually take the place of MKL, but the macro definitions were still retained.
10+
11+
12+
13+
Macro definition of oneDNN in DeepRec:
14+
15+
| Macro Definition | Values（Bold for Default） | Explanation |
16+
| :------------------------------- | --------------------------------------------- | ------------------------------------------------------------ |
17+
| TF_MKL_PRIMITIVE_ONLY_FOR_RECO | **1/true**, 0/false | 1: Only replace the [operators](https://github.com/alibaba/DeepRec/blob/main/tensorflow/core/graph/mkl_layout_pass.cc#L824-L840) which supported by oneDNN in recommendation models; 0: Replace all of the operators to that supported by oneDNN. |
18+
| TF_MKL_OPTIMIZE_PRIMITIVE_MEMUSE | **1/true**, 0/false | 1: Reduce the use of main memory by releasing the primitives; 0: Don't release primitives. |
19+
| TF_DISABLE_MKL | **0**, 1 | 0: Enable MKL; 1: Disable MKL |
20+
| TF_MKL_NUM_INTRAOP | Integer, such as 14 ,**Not set by default** | Integer：set the number of intra threads used by oneDNN；Not set：number of TF intra threads used most. |
21+
| ONEDNN_VERBOSE | **0**/1/2 | Print the [level](https://oneapi-src.github.io/oneDNN/dev_guide_verbose.html) of log output by oneDNN primitive. |
22+
| DNNL_MAX_CPU_ISA | **ALL**, AVX512_CORE_AMX, AVX512_CORE_BF16, … | The[ highest ISA](https://oneapi-src.github.io/oneDNN/v2.4/dev_guide_cpu_dispatcher_control.html#run-time-controls) used by oneDNN (for versions less than 2.5.0) |
23+
| ONEDNN_MAX_CPU_ISA | **ALL**, AVX512_CORE_AMX, AVX512_CORE_BF16, … | The [highest ISA](https://oneapi-src.github.io/oneDNN/v2.4/dev_guide_cpu_dispatcher_control.html#run-time-controlsused) by oneDNN (for versions more than or equal to 2.5.0) |
24+
25+
Primitives supported by oneDNN:
26+
27+
| Primitive | Available Types | Available Backward Operations |
28+
| -------------------------------------- | --------------------------- | --------------------------------- |
29+
| Matrix Multiplication | f32, bf16, f16, u8, s8 | Scale, Zero, Eltwise, Sum, Binary |
30+
| Inner Product | f32, bf16, f16, u8, s8 | Scale, Eltwise, Sum, Binary |
31+
| Layer Normalization | f32, bf16, f16 | / |
32+
| Batch Normalization | f32, bf16, f16, s8 | Eltwise |
33+
| Local Response Normalization (LRN) | f32, bf16, f16 | / |
34+
| Binary (+, =, *, /, >, <, min, max...) | f32, bf16, f16, u8, s8 | Scale, Eltwise, Sum, Binary |
35+
| Eltwise (relu, gelu, tanh, linear...) | f32, s32, bf16, f16, u8, s8 | Binary |
36+
| PReLU | f32, s32, bf16, s8, u8 | / |
37+
| Sum | f32, s32, bf16, f16, u8, s8 | / |
38+
| Reduction | f32, bf16, u8, s8 | Eltwise, Sum, Binary |
39+
| Softmax | f32, bf16, f16 | / |
40+
| LogSoftmax | f32, bf16 | / |
41+
| Reorder | f32, s32, bf16, f16, u8, s8 | Scale, Sum |
42+
| Concat | f32, s32, bf16, f16, u8, s8 | / |
43+
| Convolution | f32, bf16, f16, u8, s8 | Scale, Zero, Eltwise, Sum, Binary |
44+
| Pooling | f32, s32, bf16, f16, u8, s8 | Binary |
45+
| RNN (LSTM, GRU, Vanilla RNN...) | f32, bf16, f16, u8, s8 | / |
46+
| Resampling | f32, s32, bf16, f16, s8, u8 | Eltwise, Sum, Binary |
47+
| Shuffle | f32, s32, bf16, s8, u8 | / |
48+

0 commit comments

Comments
 (0)