Skip to content

Commit a849230

Browse files
hd9568hd9568
authored
[Fix Doc] [CI] 修复api差异文档的格式，提高CI鲁棒性 (#7554)
* 修复api差异文档的格式，提高CI鲁棒性 * 修复api差异文档的格式，提高CI鲁棒性
1 parent 541675d commit a849230

File tree

928 files changed

+1046
-43
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

928 files changed

+1046
-43
lines changed

‎ci_scripts/hooks/pre-doc-compile.sh‎

Lines changed: 109 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -5,55 +5,140 @@ SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
55

66
FLUIDDOCDIR=${FLUIDDOCDIR:=/FluidDoc}
77
DOCROOT=${FLUIDDOCDIR}/docs
8-
9-
10-
## 1. 获取API映射文件
118
APIMAPPING_ROOT=${DOCROOT}/guides/model_convert/convert_from_pytorch
129
TOOLS_DIR=${APIMAPPING_ROOT}/tools
1310

14-
# 确保tools目录存在
15-
mkdir -p ${TOOLS_DIR}
11+
# Create tools directory if not exists
12+
if [ ! -d "${TOOLS_DIR}" ]; then
13+
echo "INFO: Creating tools directory at ${TOOLS_DIR}"
14+
mkdir -p "${TOOLS_DIR}"
15+
if [ $? -ne 0 ]; then
16+
echo "ERROR: Failed to create directory ${TOOLS_DIR}"
17+
exit 1
18+
fi
19+
else
20+
echo "INFO: Tools directory ${TOOLS_DIR} already exists"
21+
fi
1622

17-
#下载的文件URL
23+
# Define API mapping files URLs
1824
API_ALIAS_MAPPING_URL="https://raw.githubusercontent.com/PaddlePaddle/PaConvert/master/paconvert/api_alias_mapping.json"
1925
API_MAPPING_URL="https://raw.githubusercontent.com/PaddlePaddle/PaConvert/master/paconvert/api_mapping.json"
2026
GLOBAL_VAR_URL="https://raw.githubusercontent.com/PaddlePaddle/PaConvert/master/paconvert/global_var.py"
2127
ATTRIBUTE_MAPPING_URL="https://raw.githubusercontent.com/PaddlePaddle/PaConvert/master/paconvert/attribute_mapping.json"
2228

23-
# 下载文件
29+
# Define backup URLs
30+
BACKUP_API_ALIAS_MAPPING_URL="https://paddle-paconvert.bj.bcebos.com/api_alias_mapping.json"
31+
BACKUP_API_MAPPING_URL="https://paddle-paconvert.bj.bcebos.com/api_mapping.json"
32+
BACKUP_GLOBAL_VAR_URL="https://paddle-paconvert.bj.bcebos.com/global_var.py"
33+
BACKUP_ATTRIBUTE_MAPPING_URL="https://paddle-paconvert.bj.bcebos.com/attribute_mapping.json"
34+
35+
# Check for proxy settings
2436
PROXY=""
2537
if [ -n "$https_proxy" ]; then
2638
PROXY="$https_proxy"
39+
echo "INFO: find proxy"
2740
elif [ -n "$http_proxy" ]; then
2841
PROXY="$http_proxy"
42+
echo "INFO: find proxy"
43+
else
44+
echo "INFO: No proxy detected, downloading directly."
2945
fi
3046

31-
# 构建 curl 代理参数
32-
CURL_PROXY_ARGS=""
47+
# Build curl proxy arguments
3348
if [ -n "$PROXY" ]; then
34-
CURL_PROXY_ARGS="--proxy $PROXY"
49+
CURL_PROXY_ARGS="--proxy ${PROXY}"
3550
else
36-
echo "No proxy detected, downloading directly."
51+
CURL_PROXY_ARGS=""
52+
fi
53+
54+
# Download API mapping files with retry
55+
download_file() {
56+
local url=$1
57+
local dest=$2
58+
local filename=$(basename "$dest")
59+
local max_retries=5
60+
local retry_count=0
61+
62+
echo "INFO: Starting download of ${filename} from ${url}"
63+
64+
while [ $retry_count -lt $max_retries ]; do
65+
retry_count=$((retry_count + 1))
66+
echo "INFO: Attempt $retry_count of $max_retries to download ${filename}"
67+
68+
if curl $CURL_PROXY_ARGS -o "${dest}" -s "${url}" > /dev/null 2>&1; then
69+
echo "SUCCESS: Successfully downloaded ${filename} to ${dest}"
70+
return 0
71+
else
72+
echo "WARNING: Failed to download ${filename} from ${url} (attempt $retry_count)"
73+
sleep 2 # Wait for 2 seconds before next retry
74+
fi
75+
done
76+
77+
echo "ERROR: Failed to download ${filename} after $max_retries attempts"
78+
return 1
79+
}
80+
81+
# Download each file with detailed logging
82+
echo "INFO: Downloading API alias mapping file"
83+
if ! download_file "${API_ALIAS_MAPPING_URL}" "${TOOLS_DIR}/api_alias_mapping.json"; then
84+
echo "INFO: Trying backup URL for API alias mapping file"
85+
if ! download_file "${BACKUP_API_ALIAS_MAPPING_URL}" "${TOOLS_DIR}/api_alias_mapping.json"; then
86+
echo "ERROR: API alias mapping download failed (both main and backup URLs). Exiting."
87+
exit 1
88+
fi
89+
fi
90+
91+
echo "INFO: Downloading API mapping file"
92+
if ! download_file "${API_MAPPING_URL}" "${TOOLS_DIR}/api_mapping.json"; then
93+
echo "INFO: Trying backup URL for API mapping file"
94+
if ! download_file "${BACKUP_API_MAPPING_URL}" "${TOOLS_DIR}/api_mapping.json"; then
95+
echo "ERROR: API mapping download failed (both main and backup URLs). Exiting."
96+
exit 1
97+
fi
3798
fi
3899

39-
# 执行下载
40-
curl $CURL_PROXY_ARGS -o "${TOOLS_DIR}/api_alias_mapping.json" -s "${API_ALIAS_MAPPING_URL}"
41-
curl $CURL_PROXY_ARGS -o "${TOOLS_DIR}/api_mapping.json" -s "${API_MAPPING_URL}"
42-
curl $CURL_PROXY_ARGS -o "${TOOLS_DIR}/global_var.py" -s "${GLOBAL_VAR_URL}"
43-
curl $CURL_PROXY_ARGS -o "${TOOLS_DIR}/attribute_mapping.json" -s "${ATTRIBUTE_MAPPING_URL}"
100+
echo "INFO: Downloading global variable file"
101+
if ! download_file "${GLOBAL_VAR_URL}" "${TOOLS_DIR}/global_var.py"; then
102+
echo "INFO: Trying backup URL for global variable file"
103+
if ! download_file "${BACKUP_GLOBAL_VAR_URL}" "${TOOLS_DIR}/global_var.py"; then
104+
echo "ERROR: Global variable download failed (both main and backup URLs). Exiting."
105+
exit 1
106+
fi
107+
fi
108+
109+
echo "INFO: Downloading attribute mapping file"
110+
if ! download_file "${ATTRIBUTE_MAPPING_URL}" "${TOOLS_DIR}/attribute_mapping.json"; then
111+
echo "INFO: Trying backup URL for attribute mapping file"
112+
if ! download_file "${BACKUP_ATTRIBUTE_MAPPING_URL}" "${TOOLS_DIR}/attribute_mapping.json"; then
113+
echo "ERROR: Attribute mapping download failed (both main and backup URLs). Exiting."
114+
exit 1
115+
fi
116+
fi
44117

45-
# 检查下载是否成功
46-
if [ $? -ne 0 ]; then
47-
echo "Error: Failed to download API mapping files"
118+
# Check if all files exist before proceeding
119+
if [ ! -f "${TOOLS_DIR}/api_alias_mapping.json" ] || \
120+
[ ! -f "${TOOLS_DIR}/api_mapping.json" ] || \
121+
[ ! -f "${TOOLS_DIR}/global_var.py" ] || \
122+
[ ! -f "${TOOLS_DIR}/attribute_mapping.json" ]; then
123+
echo "ERROR: One or more API mapping files are missing after download"
124+
echo "Missing files:"
125+
if [ ! -f "${TOOLS_DIR}/api_alias_mapping.json" ]; then echo " - api_alias_mapping.json"; fi
126+
if [ ! -f "${TOOLS_DIR}/api_mapping.json" ]; then echo " - api_mapping.json"; fi
127+
if [ ! -f "${TOOLS_DIR}/global_var.py" ]; then echo " - global_var.py"; fi
128+
if [ ! -f "${TOOLS_DIR}/attribute_mapping.json" ]; then echo " - attribute_mapping.json"; fi
48129
exit 1
49130
fi
50131

51-
## 3. Apply PyTorch-PaddlePaddle mapping using the new API mapping files
52-
python ${APIMAPPING_ROOT}/tools/get_api_difference_info.py
53-
python ${APIMAPPING_ROOT}/tools/generate_pytorch_api_mapping.py
132+
echo "INFO: All API mapping files successfully downloaded"
54133

134+
echo "INFO: Running get_api_difference_info.py"
135+
if ! python "${APIMAPPING_ROOT}/tools/get_api_difference_info.py"; then
136+
echo "ERROR: get_api_difference_info.py failed. Please check the script."
137+
exit 1
138+
fi
55139

56-
if [ $? -ne 0 ]; then
57-
echo "Error: API mapping generate script failed, please check changes in ${APIMAPPING_ROOT}"
140+
echo "INFO: Running generate_pytorch_api_mapping.py"
141+
if ! python "${APIMAPPING_ROOT}/tools/generate_pytorch_api_mapping.py"; then
142+
echo "ERROR: generate_pytorch_api_mapping.py failed. Please check the script."
58143
exit 1
59144
fi

‎docs/guides/model_convert/convert_from_pytorch/api_difference/args_default_value_diff/torch.alpha_dropout.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ paddle.nn.functional.alpha_dropout(x, p=0.5, training=True, name=None)
1212
其中 PyTorch 和 Paddle 功能一致，参数默认值不一致，具体如下：
1313

1414
### 参数映射
15+
1516
| PyTorch | PaddlePaddle | 备注 |
1617
| ------- | ------------ | -- |
1718
| input | x | 输入的多维 Tensor，仅参数名不一致。 |

‎docs/guides/model_convert/convert_from_pytorch/api_difference/args_default_value_diff/torch.cuda.amp.GradScaler.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ paddle.amp.GradScaler(enable=True, init_loss_scaling=65536.0, incr_ratio=2.0, de
1313
其中 Paddle 相比 PyTorch 支持更多其他参数且参数默认值不一致，具体如下：
1414

1515
### 参数映射
16+
1617
| PyTorch | PaddlePaddle | 备注 |
1718
| --------------- | ------------------------ |-----------------------------------------------------------------------------|
1819
| init_scale | init_loss_scaling | 初始 loss scaling 因子。Paddle 与 PyTorch 默认值不同，Paddle 应设置为 65536.0。 |

‎docs/guides/model_convert/convert_from_pytorch/api_difference/args_default_value_diff/torch.linalg.diagonal.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ paddle.diagonal(x,
1616
两者功能一致且参数用法一致，参数默认值不一致，具体如下：
1717

1818
### 参数映射
19+
1920
| PyTorch | PaddlePaddle | 备注 |
2021
| ------------- | ------------ | ------------------------------------------------------ |
2122
| <font color='red'> A </font> | <font color='red'> x </font> | 表示输入的 Tensor ，仅参数名不一致。 |

‎docs/guides/model_convert/convert_from_pytorch/api_difference/args_default_value_diff/torch.nn.functional.rrelu_.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ paddle.nn.functional.rrelu(x, lower=1./8., upper=1./3., training=True, name=None
1212
其中 PyTorch 和 Paddle 功能基本一致，前者会在不变更变量内存地址的情况下，直接改变变量的值，一般对网络训练结果影响不大。参数名与参数默认值不一致，具体如下：
1313

1414
### 参数映射
15+
1516
| PyTorch | PaddlePaddle | 备注 |
1617
| -------- | ------------ | -- |
1718
| input | x | 输入的 Tensor，仅参数名不一致。 |

‎docs/guides/model_convert/convert_from_pytorch/api_difference/args_default_value_diff/torch.nn.functional.threshold_.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ paddle.nn.functional.thresholded_relu_(x, threshold=1.0, value=0.0, name=None)
1212
两者功能一致，但参数名称与参数默认值不一致，具体如下：
1313

1414
### 参数映射
15+
1516
| PyTorch | PaddlePaddle | 备注 |
1617
| --------- | ------------ | --------------------------------------------------------------------------------------------------------------- |
1718
| input | x | 输入的 Tensor，仅参数名不一致。 |

‎docs/guides/model_convert/convert_from_pytorch/api_difference/args_default_value_diff/torch.rrelu.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ paddle.nn.functional.rrelu(x, lower=1./8, upper=1./3, training=True, name=None)
1212
其中 PyTorch 和 Paddle 功能一致，参数名与参数默认值不一致，具体如下：
1313

1414
### 参数映射
15+
1516
| PyTorch | PaddlePaddle | 备注 |
1617
| -------- | ------------ | ----------------------------------------- |
1718
| input | x | 输入的 Tensor，仅参数名不一致。 |

‎docs/guides/model_convert/convert_from_pytorch/api_difference/args_name_diff/torch.Tensor.addmm.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ paddle.Tensor.addmm(x, y, alpha=1.0, beta=1.0)
1212
两者功能一致且参数用法一致，仅参数名不一致，具体如下：
1313

1414
### 参数映射
15+
1516
| PyTorch | PaddlePaddle | 备注 |
1617
| -------- | ------------ | -------------------------------- |
1718
| mat1 | x | 表示输入的 Tensor，仅参数名不一致。 |

‎docs/guides/model_convert/convert_from_pytorch/api_difference/args_name_diff/torch.Tensor.addmm_.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ paddle.Tensor.addmm_(x, y, alpha=1.0, beta=1.0)
1212
两者功能一致且参数用法一致，仅参数名不一致，具体如下：
1313

1414
### 参数映射
15+
1516
| PyTorch | PaddlePaddle | 备注 |
1617
| -------- | ------------ | -------------------------------- |
1718
| mat1 | x | 表示输入的 Tensor，仅参数名不一致。 |

‎docs/guides/model_convert/convert_from_pytorch/api_difference/args_name_diff/torch.Tensor.as_strided.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ paddle.Tensor.as_strided(shape,
1616

1717
两者功能一致且参数用法一致，仅参数名不一致，具体如下：
1818
### 参数映射
19+
1920
| PyTorch | PaddlePaddle | 备注 |
2021
| ------------- | ------------ | ------------------------------------------------------ |
2122
| size | shape | 表示输出 Tensor 的维度, 仅参数名不一致。 |

0 commit comments

Comments
 (0)