Skip to content

Commit ce620fe

Browse files
myhlolimyhloli
authored
Merge pull request #4405 from myhloli/dev
Dev
2 parents c6543b4 + 8c1aed8 commit ce620fe

File tree

20 files changed

+617
-41
lines changed

20 files changed

+617
-41
lines changed

‎README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,9 @@
4545

4646
# Changelog
4747

48+
- 2026/01/23 2.7.2 Release
49+
- Cross-page table merging optimization, improving merge success rate and merge quality
50+
4851
- 2026/01/06 2.7.1 Release
4952
- fix bug: #4300
5053
- Updated pdfminer.six dependency version to resolve [CVE-2025-64512](https://github.com/advisories/GHSA-wf5f-4jwr-ppcp)

‎README_zh-CN.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,13 @@
4545

4646
# 更新记录
4747

48+
- 2026/01/23 2.7.2 发布
49+
- 新增国产算力平台适配:
50+
- [海光 DCU](https://opendatalab.github.io/MinerU/zh/usage/acceleration_cards/Hygon/)
51+
- [燧原 GCU](https://opendatalab.github.io/MinerU/zh/usage/acceleration_cards/Enflame/)
52+
- [摩尔线程 MUSA](https://opendatalab.github.io/MinerU/zh/usage/acceleration_cards/MooreThreads/)
53+
- 跨页表合并优化，提升合并成功率与合并效果
54+
4855
- 2026/01/06 2.7.1 发布
4956
- fix bug: #4300
5057
- 更新pdfminer.six的依赖版本以解决 [CVE-2025-64512](https://github.com/advisories/GHSA-wf5f-4jwr-ppcp)

‎docker/china/gcu.Dockerfile‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Base image containing the vLLM inference environment, requiring amd64(x86-64) CPU + Enflame GCU.
2+
FROM crpi-vofi3w62lkohhxsp.cn-shanghai.personal.cr.aliyuncs.com/opendatalab-mineru/gcu:docker_images_topsrider_i3x_3.6.20260106_vllm0.11_pytorch2.8.0
3+
4+
5+
# Install libgl for opencv support & Noto fonts for Chinese characters
6+
RUN echo 'deb http://mirrors.aliyun.com/ubuntu/ noble main restricted universe multiverse\n\
7+
deb http://mirrors.aliyun.com/ubuntu/ noble-updates main restricted universe multiverse\n\
8+
deb http://mirrors.aliyun.com/ubuntu/ noble-backports main restricted universe multiverse\n\
9+
deb http://mirrors.aliyun.com/ubuntu/ noble-security main restricted universe multiverse' > /tmp/aliyun-sources.list && \
10+
apt-get -o Dir::Etc::SourceList=/tmp/aliyun-sources.list update && \
11+
apt-get -o Dir::Etc::SourceList=/tmp/aliyun-sources.list install -y \
12+
fonts-noto-core \
13+
fonts-noto-cjk \
14+
fontconfig && \
15+
fc-cache -fv && \
16+
apt-get clean && \
17+
rm -rf /var/lib/apt/lists/* /tmp/aliyun-sources.list
18+
19+
# Install mineru latest
20+
RUN python3 -m pip install "mineru[core]>=2.7.2" \
21+
numpy==1.26.4 \
22+
opencv-python==4.11.0.86 \
23+
-i https://mirrors.aliyun.com/pypi/simple && \
24+
python3 -m pip cache purge
25+
26+
# Download models and update the configuration file
27+
RUN /bin/bash -c "mineru-models-download -s modelscope -m all"
28+
29+
# Set the entry point to activate the virtual environment and run the command line tool
30+
ENTRYPOINT ["/bin/bash", "-c", "export MINERU_MODEL_SOURCE=local && exec \"$@\"", "--"]

‎docker/china/musa.Dockerfile‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Base image containing the vLLM inference environment, requiring amd64(x86-64) CPU + MooreThreads GPU.
2+
FROM registry.mthreads.com/mcconline/vllm-musa-qy2-py310:v0.8.4-release
3+
4+
5+
# Install libgl for opencv support & Noto fonts for Chinese characters
6+
RUN apt-get update && \
7+
apt-get install -y \
8+
fonts-noto-core \
9+
fonts-noto-cjk \
10+
fontconfig \
11+
libgl1 && \
12+
fc-cache -fv && \
13+
apt-get clean && \
14+
rm -rf /var/lib/apt/lists/*
15+
16+
# Install mineru latest
17+
RUN python3 -m pip install -U pip -i https://mirrors.aliyun.com/pypi/simple && \
18+
git clone https://gitcode.com/gh_mirrors/vi/vision.git -b v0.20.0 --depth 1 && \
19+
cd vision && \
20+
python3 setup.py install && \
21+
python3 -m pip install "mineru[api,gradio]>=2.7.2" \
22+
"matplotlib>=3.10,<4" \
23+
"ultralytics>=8.3.48,<9" \
24+
"doclayout_yolo==0.0.4" \
25+
"ftfy>=6.3.1,<7" \
26+
"shapely>=2.0.7,<3" \
27+
"pyclipper>=1.3.0,<2" \
28+
"omegaconf>=2.3.0,<3" \
29+
numpy==1.26.4 \
30+
opencv-python==4.11.0.86 \
31+
-i https://mirrors.aliyun.com/pypi/simple && \
32+
python3 -m pip cache purge
33+
34+
# Download models and update the configuration file
35+
RUN /bin/bash -c "mineru-models-download -s modelscope -m all"
36+
37+
# Set the entry point to activate the virtual environment and run the command line tool
38+
ENTRYPOINT ["/bin/bash", "-c", "export MINERU_MODEL_SOURCE=local && exec \"$@\"", "--"]

‎docs/en/usage/cli_tools.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,3 +125,9 @@ Here are the environment variables and their descriptions:
125125
- `MINERU_HYBRID_FORCE_PIPELINE_ENABLE`:
126126
* Used to force the text extraction part in `hybrid-*` backends to be processed using small models.
127127
* Defaults to `false`. Can be set to `true` via environment variable to enable this feature, thereby reducing hallucinations in certain extreme cases.
128+
129+
- `MINERU_VL_MODEL_NAME`:
130+
* Used to specify the model name for the vlm/hybrid backend, allowing you to designate the model required for MinerU to run when multiple models exist on a remote openai-server.
131+
132+
- `MINERU_VL_API_KEY`:
133+
* Used to specify the API Key for the vlm/hybrid backend, enabling authentication on the remote openai-server.

‎docs/zh/usage/acceleration_cards/Enflame.md‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
## 1. 测试平台
2+
以下为本指南测试使用的平台信息，供参考：
3+
```
4+
os: Ubuntu 22.04.4 LTS
5+
cpu: Intel x86-64
6+
gcu: Enflame S60
7+
driver: 1.7.0.9
8+
docker: 28.0.1
9+
```
10+
11+
## 2. 环境准备
12+
13+
### 2.1 使用 Dockerfile 构建镜像
14+
15+
```bash
16+
wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/gcu.Dockerfile
17+
docker build --network=host -t mineru:gcu-vllm-latest -f gcu.Dockerfile .
18+
```
19+
20+
21+
## 3. 启动 Docker 容器
22+
23+
```bash
24+
docker run -u root --name mineru_docker \
25+
--network=host \
26+
--ipc=host \
27+
--privileged \
28+
-e MINERU_MODEL_SOURCE=local \
29+
-it mineru:gcu-vllm-latest \
30+
/bin/bash
31+
```
32+
33+
执行该命令后，您将进入到Docker容器的交互式终端，您可以直接在容器内运行MinerU相关命令来使用MinerU的功能。
34+
您也可以直接通过替换`/bin/bash`为服务启动命令来启动MinerU服务，详细说明请参考[通过命令启动服务](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuihttp-clientserver)。
35+
36+
37+
## 4. 注意事项
38+
39+
不同环境下，MinerU对Enflame加速卡的支持情况如下表所示：
40+
41+
<table border="1">
42+
<thead>
43+
<tr>
44+
<th rowspan="2" colspan="2">使用场景</th>
45+
<th colspan="2">容器环境</th>
46+
</tr>
47+
<tr>
48+
<th>vllm</th>
49+
</tr>
50+
</thead>
51+
<tbody>
52+
<tr>
53+
<td rowspan="3">命令行工具(mineru)</td>
54+
<td>pipeline</td>
55+
<td>🟢</td>
56+
</tr>
57+
<tr>
58+
<td>&lt;vlm/hybrid&gt;-auto-engine</td>
59+
<td>🟢</td>
60+
</tr>
61+
<tr>
62+
<td>&lt;vlm/hybrid&gt;-http-client</td>
63+
<td>🟢</td>
64+
</tr>
65+
<tr>
66+
<td rowspan="3">fastapi服务(mineru-api)</td>
67+
<td>pipeline</td>
68+
<td>🟢</td>
69+
</tr>
70+
<tr>
71+
<td>&lt;vlm/hybrid&gt;-auto-engine</td>
72+
<td>🟢</td>
73+
</tr>
74+
<tr>
75+
<td>&lt;vlm/hybrid&gt;-http-client</td>
76+
<td>🟢</td>
77+
</tr>
78+
<tr>
79+
<td rowspan="3">gradio界面(mineru-gradio)</td>
80+
<td>pipeline</td>
81+
<td>🟢</td>
82+
</tr>
83+
<tr>
84+
<td>&lt;vlm/hybrid&gt;-auto-engine</td>
85+
<td>🟢</td>
86+
</tr>
87+
<tr>
88+
<td>&lt;vlm/hybrid&gt;-http-client</td>
89+
<td>🟢</td>
90+
</tr>
91+
<tr>
92+
<td colspan="2">openai-server服务（mineru-openai-server）</td>
93+
<td>🟢</td>
94+
</tr>
95+
<tr>
96+
<td colspan="2">数据并行 (--data-parallel-size)</td>
97+
<td>🔴</td>
98+
</tr>
99+
</tbody>
100+
</table>
101+
102+
注：
103+
🟢: 支持，运行较稳定，精度与Nvidia GPU基本一致
104+
🟡: 支持但较不稳定，在某些场景下可能出现异常，或精度存在一定差异
105+
🔴: 不支持，无法运行，或精度存在较大差异
106+
107+
>[!TIP]
108+
>GCU加速卡指定可用加速卡的方式与NVIDIA GPU类似，请参考[使用指定GPU设备](https://opendatalab.github.io/MinerU/zh/usage/advanced_cli_parameters/#cuda_visible_devices)章节说明,
109+
>将环境变量`CUDA_VISIBLE_DEVICES`替换为`TOPS_VISIBLE_DEVICES`即可。

‎docs/zh/usage/acceleration_cards/MooreThreads.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
## 1. 测试平台
2+
以下为本指南测试使用的平台信息，供参考：
3+
```
4+
os: Ubuntu 22.04.4 LTS
5+
cpu: Intel x86-64
6+
dcu: MTT S4000
7+
driver: 3.0.0-rc-KuaE2.0
8+
docker: 24.0.7
9+
```
10+
11+
## 2. 环境准备
12+
13+
### 2.1 使用 Dockerfile 构建镜像
14+
15+
```bash
16+
wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/musa.Dockerfile
17+
docker build --network=host -t mineru:musa-vllm-latest -f musa.Dockerfile .
18+
```
19+
20+
21+
## 3. 启动 Docker 容器
22+
23+
```bash
24+
docker run -u root --name mineru_docker \
25+
--network=host \
26+
--ipc=host \
27+
--shm-size=80g \
28+
--privileged \
29+
-e MTHREADS_VISIBLE_DEVICES=all \
30+
-e MINERU_MODEL_SOURCE=local \
31+
-it mineru:musa-vllm-latest \
32+
/bin/bash
33+
```
34+
35+
执行该命令后，您将进入到Docker容器的交互式终端，您可以直接在容器内运行MinerU相关命令来使用MinerU的功能。
36+
您也可以直接通过替换`/bin/bash`为服务启动命令来启动MinerU服务，详细说明请参考[通过命令启动服务](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuihttp-clientserver)。
37+
38+
39+
## 4. 注意事项
40+
41+
不同环境下，MinerU对MooreThreads加速卡的支持情况如下表所示：
42+
43+
>[!NOTE]
44+
> **兼容性说明**：由于摩尔线程（MooreThreads）目前对 vLLM v1 引擎的支持尚待完善，MinerU 现阶段采用 v0 引擎作为适配方案。
45+
> 受此限制，vLLM 的异步引擎（Async Engine）功能存在兼容性问题，可能导致部分使用场景无法正常运行。
46+
> 我们将持续跟进摩尔线程对 vLLM v1 引擎的支持进展，并及时在 MinerU 中进行相应的适配与优化。
47+
48+
<table border="1">
49+
<thead>
50+
<tr>
51+
<th rowspan="2" colspan="2">使用场景</th>
52+
<th colspan="2">容器环境</th>
53+
</tr>
54+
<tr>
55+
<th>vllm</th>
56+
</tr>
57+
</thead>
58+
<tbody>
59+
<tr>
60+
<td rowspan="3">命令行工具(mineru)</td>
61+
<td>pipeline</td>
62+
<td>🟢</td>
63+
</tr>
64+
<tr>
65+
<td>&lt;vlm/hybrid&gt;-auto-engine</td>
66+
<td>🟢</td>
67+
</tr>
68+
<tr>
69+
<td>&lt;vlm/hybrid&gt;-http-client</td>
70+
<td>🟢</td>
71+
</tr>
72+
<tr>
73+
<td rowspan="3">fastapi服务(mineru-api)</td>
74+
<td>pipeline</td>
75+
<td>🟢</td>
76+
</tr>
77+
<tr>
78+
<td>&lt;vlm/hybrid&gt;-auto-engine</td>
79+
<td>🔴</td>
80+
</tr>
81+
<tr>
82+
<td>&lt;vlm/hybrid&gt;-http-client</td>
83+
<td>🟢</td>
84+
</tr>
85+
<tr>
86+
<td rowspan="3">gradio界面(mineru-gradio)</td>
87+
<td>pipeline</td>
88+
<td>🟢</td>
89+
</tr>
90+
<tr>
91+
<td>&lt;vlm/hybrid&gt;-auto-engine</td>
92+
<td>🔴</td>
93+
</tr>
94+
<tr>
95+
<td>&lt;vlm/hybrid&gt;-http-client</td>
96+
<td>🟢</td>
97+
</tr>
98+
<tr>
99+
<td colspan="2">openai-server服务（mineru-openai-server）</td>
100+
<td>🟢</td>
101+
</tr>
102+
<tr>
103+
<td colspan="2">数据并行 (--data-parallel-size)</td>
104+
<td>🔴</td>
105+
</tr>
106+
</tbody>
107+
</table>
108+
109+
注：
110+
🟢: 支持，运行较稳定，精度与Nvidia GPU基本一致
111+
🟡: 支持但较不稳定，在某些场景下可能出现异常，或精度存在一定差异
112+
🔴: 不支持，无法运行，或精度存在较大差异
113+
114+
>[!TIP]
115+
>MooreThreads加速卡指定可用加速卡的方式与NVIDIA GPU类似，请参考[GPU 枚举](https://docs.mthreads.com/cloud-native/cloud-native-doc-online/install_guide/#gpu-%E6%9E%9A%E4%B8%BE)

‎docs/zh/usage/cli_tools.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -119,4 +119,10 @@ MinerU命令行工具的某些参数存在相同功能的环境变量配置，
119119

120120
- `MINERU_HYBRID_FORCE_PIPELINE_ENABLE`：
121121
* 用于强制将 hybrid-* 后端中的 文本提取部分使用 小模型 进行处理
122-
* 默认为`false`，可通过环境变量设置为`true`来启用该功能，从而在某些极端情况下减少幻觉的发生。
122+
* 默认为`false`，可通过环境变量设置为`true`来启用该功能，从而在某些极端情况下减少幻觉的发生。
123+
124+
- `MINERU_VL_MODEL_NAME`：
125+
* 用于指定 vlm/hybrid 后端使用的模型名称，这将允许您在同时存在多个模型的远程openai-server中指定 MinerU 运行所需的模型。
126+
127+
- `MINERU_VL_API_KEY`:
128+
* 用于指定 vlm/hybrid 后端使用的API Key，这将允许您在远程openai-server中进行身份验证。

‎docs/zh/usage/index.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,8 @@
1313
* [平头哥 T-Head](acceleration_cards/THead.md) 🚀
1414
* [沐曦 METAX](acceleration_cards/METAX.md) 🚀
1515
* [海光 Hygon](acceleration_cards/Hygon.md) 🚀
16+
* [燧原 Enflame](acceleration_cards/Enflame.md) 🚀
17+
* [摩尔线程 MooreThreads](acceleration_cards/MooreThreads.md) 🚀
1618
* [AMD](acceleration_cards/AMD.md) [#3662](https://github.com/opendatalab/MinerU/discussions/3662) ❤️
1719
* [太初元碁 Tecorigin](acceleration_cards/Tecorigin.md) [#3767](https://github.com/opendatalab/MinerU/pull/3767) ❤️
1820
* [寒武纪 Cambricon](acceleration_cards/Cambricon.md) [#4004](https://github.com/opendatalab/MinerU/discussions/4004) ❤️

‎mineru/backend/vlm/utils.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,10 @@ def enable_custom_logits_processors() -> bool:
1818
compute_capability = f"{major}.{minor}"
1919
elif hasattr(torch, 'npu') and torch.npu.is_available():
2020
compute_capability = "8.0"
21+
elif hasattr(torch, 'gcu') and torch.gcu.is_available():
22+
compute_capability = "8.0"
23+
elif hasattr(torch, 'musa') and torch.musa.is_available():
24+
compute_capability = "8.0"
2125
else:
2226
logger.info("CUDA not available, disabling custom_logits_processors")
2327
return False

0 commit comments

Comments
 (0)