PaddleX/latest/pipeline_deploy/serving #3524
giscus[bot]
bot
Replies: 29 comments 114 replies
-
|
您好，paddlex --serve --pipeline image_classification这样启动服务之后，关闭服务的命令是什么呀？ |
Beta Was this translation helpful? Give feedback.
-
|
ctrl+c |
Beta Was this translation helpful? Give feedback.
-
|
paddlex --serve --pipeline {产线名称或产线配置文件路径} [{其他命令行选项}] 如果在部署多个产线， 是否要启动多个服务吗？ |
Beta Was this translation helpful? Give feedback.
-
|
您好，需要启动多个服务 |
Beta Was this translation helpful? Give feedback.
-
|
服务化部署支持实时数据的高性能推理？ |
Beta Was this translation helpful? Give feedback.
-
|
暂时没有支持流式返回的服务。不过，对于实时数据，可以考虑多实例+高性能推理增加并发、降低延迟～ |
Beta Was this translation helpful? Give feedback.
-
|
我在测试‘高稳定性服务化部署’，使用通用OCR SDK，本地docker启动成功，client.py测试GRPCInferenceService通过，Metrics Service访问成功，但是HTTPService请求失败，一直报错400 Bad Request。想问一下HTTPService的接口调用文档有没有，请求参数是什么，怎么才能正确访问HTTPService |
Beta Was this translation helpful? Give feedback.
-
|
你好，我们暂时没有提供HTTPService的调用文档，目前只支持gRPC～ |
Beta Was this translation helpful? Give feedback.
-
|
SDK的下载链接都失效了，麻烦修复下吧 |
Beta Was this translation helpful? Give feedback.
-
|
你好，目前这块正在升级中，文档先更新了，最新的3.0.1 SDK预计今天之内上传～ |
Beta Was this translation helpful? Give feedback.
-
您好，想问下此次更新主要是更新了什么内容呢？ |
Beta Was this translation helpful? Give feedback.
-
|
这次主要是bug修复，去除了一些无效的参数，另外PP-ChatOCR产线也补充了一个 |
Beta Was this translation helpful? Give feedback.
-
|
您好 |
Beta Was this translation helpful? Give feedback.
-
|
前者是部署镜像，用于高稳定性服务化部署，内置了相关环境；后者是开发镜像，主要用来开发调试paddlex代码，如果对镜像体积不关心，也可以用在部署（高性能推理、基础服务化部署等）。 |
Beta Was this translation helpful? Give feedback.
-
|
docker部署的，为什么λ localhost ~/PaddleX paddlex --serve --pipeline OCR |
Beta Was this translation helpful? Give feedback.
-
|
可能需要先确认一下本地是否能跑通 |
Beta Was this translation helpful? Give feedback.
-
|
我使用阿里云或者腾讯云的linux系统就可以，同样的命令，但是使用麒麟系统的这个就不太行 是不是不太支持这个版本呀？ |
Beta Was this translation helpful? Give feedback.
-
|
建议可以提一个issue，我们会跟进这个问题 |
Beta Was this translation helpful? Give feedback.
-
|
高稳定性服务化部署如何通过http方式调用 |
Beta Was this translation helpful? Give feedback.
-
|
暂不支持，后续我们会考虑添加HTTP调用示例～ |
Beta Was this translation helpful? Give feedback.
-
|
文档中说到 手动构造http请求 的方式，好像不能用呀，端口都没通。 |
Beta Was this translation helpful? Give feedback.
-
|
请问具体报什么错呢？ |
Beta Was this translation helpful? Give feedback.
-
|
已解决，是我机器端口号冲突了。把--network host改为-p 7090:8000 -p 7091:8001 -p 7092:8002就可以了。 |
Beta Was this translation helpful? Give feedback.
-
|
感谢你的反馈，后续我们将考虑增加端口占用检查，并优化错误提示 |
Beta Was this translation helpful? Give feedback.
-
|
您好，这个SDK又不能下载了 |
Beta Was this translation helpful? Give feedback.
-
|
不好意思，这几天在更新中，预计今天之内会上传。 |
Beta Was this translation helpful? Give feedback.
-
|
您好，想请教下CUDA版本为12.4的话 怎么获取ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.1-gpu的版本 这个只支持CUDA版本为11.8的吧？ |
Beta Was this translation helpful? Give feedback.
-
|
CUDA驱动支持到12.4应该也可以运行CUDA 11.8的镜像，您可以直接尝试～我们目前暂时没有提供更高版本CUDA Runtime的镜像 |
Beta Was this translation helpful? Give feedback.
-
|
你好，我用paddlex3.0.0rc1-gpu版本能跑成功了 但是OCR识别的结果还是和cpu一样 需要10秒左右；显卡是Tesla T4 请问还有其它调优方式吗？config_gpu.pbtxt配置是这样： |
Beta Was this translation helpful? Give feedback.
-
|
请使用paddlex和paddle框架最新版本吧 3.0.0rc1应该是很早以前的版本～ |
Beta Was this translation helpful? Give feedback.
-
|
之前只是paddlex3.1-gpu会报如下错误：Traceback (most recent call last): |
Beta Was this translation helpful? Give feedback.
-
|
这个问题看起来是因为我们底层使用的pypdfium2库无法正确解析这个PDF，建议可以到pypdfium2库提一个issue～ |
Beta Was this translation helpful? Give feedback.
-
|
服务化部署如何开启高性能推理？ |
Beta Was this translation helpful? Give feedback.
-
|
基础服务化部署在启动服务时指定参数 |
Beta Was this translation helpful? Give feedback.
-
|
高性能推理插件已经安装，但是加上这个参数启动时报错，不加这个参数是可以正常启动的。 paddlex --serve --use_hpip --pipeline OCRCreating model: ('PP-LCNet_x1_0_doc_ori', None) |
Beta Was this translation helpful? Give feedback.
-
|
请确认你使用的是最新版本的paddlex～ |
Beta Was this translation helpful? Give feedback.
-
|
执行 1.1 paddlex --install serving 一直报错啊，os 版本 AlmaLinux9.6， 在linux 上你们强烈建议使用 docker安装paddlex ，这里怎么没有使用docekr 安装的paddlex 执行服务部署的教程呀， 请大佬指教， 谢谢！ Using cached future-1.0.0-py3-none-any.whl (491 kB) [notice] A new release of pip is available: 25.0.1 -> 25.1.1 During handling of the above exception, another exception occurred: Traceback (most recent call last): |
Beta Was this translation helpful? Give feedback.
-
|
好的，谢谢大佬， 对于 2.4.2 可以使用grpc的方式吗， 监听的有端口， 是不是按http 的方式传入json就可以呢， 如果前台要出图片流， 或者 base64 字符， grpc 如何调用呢，谢谢大佬指导 |
Beta Was this translation helpful? Give feedback.
-
|
不好意思，我刚贴的文档是之前的版本的……可以参考这里： grpc的话，建议用Python客户端来调用；HTTP的话，可以手动构造请求。 |
Beta Was this translation helpful? Give feedback.
-
|
谢谢大佬指导，我们的业务环境比较特殊， 我们前台app 是安卓自动循环调用相机每帧去识别的，所以有大量的请求发送到server端，而且每次识别要尽量快，既然容器监听的有grpc的端口， 那么这种场景grpc 应该比http 性能要好很多吧，可以指导一下grpc的方式吗？ |
Beta Was this translation helpful? Give feedback.
-
|
这个版本 ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.0.3-gpu \ 我运行正常，但是运行 docker run E0716 02:02:04.653384 7 model_repository_manager.cc:1890] Poll failed for model directory 'ocr': instance group ocr_0 of model ocr has kind KIND_GPU but server does not support GPUs I0716 02:02:04.653444 7 server.cc:549] I0716 02:02:04.653449 7 server.cc:592] I0716 02:02:04.653482 7 tritonserver.cc:1920] I0716 02:02:04.653493 7 server.cc:252] Waiting for in-flight requests to complete. |
Beta Was this translation helpful? Give feedback.
-
|
因为你使用的是CPU镜像～请使用GPU镜像吧（以 |
Beta Was this translation helpful? Give feedback.
-
|
https://paddlepaddle.github.io/PaddleX/latest/pipeline_deploy/serving.html#23 调整后的 pipeline_config.yaml pipeline_name: OCR text_type: general use_doc_preprocessor: True SubPipelines: SubModules: |
Beta Was this translation helpful? Give feedback.
-
|
明白了，看起来是因为部分模型使用的后端（比如paddle）不支持这个配置，所以出现了上面的报错，在不手动固定后端的情况下，可能还是需要逐模型配置～ |
Beta Was this translation helpful? Give feedback.
-
|
{ 大佬，ocr识别的话，这个file必须必须是个url地址吗？ |
Beta Was this translation helpful? Give feedback.
-
|
不是，请参考产线使用文档吧，其中有详细的API参数说明。 |
Beta Was this translation helpful? Give feedback.
-
|
启用hpi时，启动服务发生以下错误 Creating model: ('PP-DocLayout_plus-L', None) |
Beta Was this translation helpful? Give feedback.
-
|
如果这个错误不影响正常使用（只是程序卡住打印这些错误），可以忽略～ |
Beta Was this translation helpful? Give feedback.
-
|
对于这种启动服务后还需下载的模型，因为网络原因无法下载怎么办，可以手动挂载吗 I0718 07:56:17.887946 7 grpc_server.cc:4117] Started GRPCInferenceService at 0.0.0.0:8001 The above exception was the direct cause of the following exception: Traceback (most recent call last): The above exception was the direct cause of the following exception: Traceback (most recent call last): |
Beta Was this translation helpful? Give feedback.
-
|
再问一下，字体文件又该挂载到哪个位置？ |
Beta Was this translation helpful? Give feedback.
-
|
解决了 |
Beta Was this translation helpful? Give feedback.
-
|
2.4.2 手动构造 HTTP 请求，构造的请求体格式不对，"data"中的json应该是String类型。 |
Beta Was this translation helpful? Give feedback.
-
|
pipeline_name: OCR text_type: general use_doc_preprocessor: True SubPipelines: SubModules: Inference backend config: precision='fp16' use_dynamic_shapes=True dynamic_shapes={'x': [[1, 3, 224, 224], [1, 3, 224, 224], [8, 3, 224, 224]]} pipeline_config.yaml配置修改下模型为PP-OCRv5_server_det, PP-OCRv5_server_rec 启动报以上错误 原来使用的mobile模型， 我就修改了这个地方，起订报错了，这两个模型不可用吗？ |
Beta Was this translation helpful? Give feedback.
-
|
NFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(572)::BuildTrtEngine Start to building TensorRT Engine... [INFO] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(572)::BuildTrtEngine Start to building TensorRT Engine... 字后还有两个报错，ERROR] ultra_infer/runtime/backends/tensorrt/trt_backend.cc(239)::log 4: Could not find any implementation for node Conv.140 + BatchNormalization.86 + Relu.59 due to insufficient workspace. 这个是空间不足吗？ |
Beta Was this translation helpful? Give feedback.
-
|
这些错误如果不影响正常直行的话可以忽略～ |
Beta Was this translation helpful? Give feedback.
-
|
我这边调用报错了，没法调用 如果ocr mobile5模型有些内容的话识别不出来，换成ocr server5的话，识别精度会不会好点呢 |
Beta Was this translation helpful? Give feedback.
-
|
请问是在PaddleX官方镜像中安装和使用高性能推理插件的吗？ server模型的识别精度更高，不过可能需要消耗更多的计算资源。 |
Beta Was this translation helpful? Give feedback.
-
|
请大佬有空了帮忙看看这个问题：
以上运行后，有两个问题。
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
明白了，感谢，useDocUnwarping确实OK了。 图片无法还原的问题我也在找到了，返回了三张图片，而不是一张 |
Beta Was this translation helpful? Give feedback.
-
|
使用 |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
https://paddlepaddle.github.io/PaddleX/latest/en/support_list/models_list.html?h=small+object+detection+yaml#object-detection-module |
Beta Was this translation helpful? Give feedback.
-
|
vim ./my_path/small_object_detection.yaml 改small_object_detection.yaml中的model_name配置就好了，会自动下载的 |
Beta Was this translation helpful? Give feedback.
-
|
高稳定性服务化部署可以不通过docker的方式部署吗？能不能出个具体的教程 |
Beta Was this translation helpful? Give feedback.
-
|
你好，目前暂不支持docker以外的方式 |
Beta Was this translation helpful? Give feedback.
-
|
你好，我想在一个triton中部署两个不同的OCR模型（通用和单独训练过的），我要问一下如何修改一下server.sh这个脚本？我看上面配置了pipe_line_config.yaml这个路径，但是两个模型，存在两个流水线文件，如何配置？还是需要启动两个docker？ |
Beta Was this translation helpful? Give feedback.
-
|
目前建议启动两个docker容器来实现这一点 |
Beta Was this translation helpful? Give feedback.
-
|
好的，希望能够解决这个问题，要不然很不方便。另外paddlex_hps_server.whl文件哪里可以获取到？ |
Beta Was this translation helpful? Give feedback.
-
|
高稳定性服务化部署的部分现在在这里开源了，可以参考～ |
Beta Was this translation helpful? Give feedback.
-
|
谢谢 |
Beta Was this translation helpful? Give feedback.
-
|
我用paddleX微调了Cascade-FasterRCNN-ResNet50-FPN模型，单模型评估后AP：63，然后通过服务化部署了object_detection产线，模型路径用的是自定义路径。在C#客户端通过http调用这个端口，单张3072*2048的图片，耗时2890毫秒，我们用的平台是windows，不太想用WSL的方式安装高性能推理，想请问一下，有没有其他的方式可以提高推理的速度？能直接在推理电脑里面能直接加载是做好的，类似于以前PaddelX GUI版本的，可以用原生库推理的，后面新版本的PaddleX会支持吗？ |
Beta Was this translation helpful? Give feedback.
-
|
更新完，测试OK,感谢解答 |
Beta Was this translation helpful? Give feedback.
-
|
但我很好奇，连最简单的通用图像分类网络，都无法加载高性能插件，我都怀疑是不是我的配置有问题，还是这个高性能插件就只支持个 例的模型，我测试几个产线，都是报The Paddle Inference backend is selected with the default configuration. This may not provide optimal performance |
Beta Was this translation helpful? Give feedback.
-
|
考虑到你使用的是飞桨框架官方镜像，而不是PaddleX官方镜像，可能需要手动安装一些依赖才可以使用完整的高性能推理功能。详情请参考高性能推理指南。 |
Beta Was this translation helpful? Give feedback.
-
|
好的，我参考高性能推理指南 重新部署PadddleX官方镜像 验证一下 |
Beta Was this translation helpful? Give feedback.
-
zh这次是用PaddleX的官方镜像，这次高性能插件安装启动了吗？
|
Beta Was this translation helpful? Give feedback.
-
|
现在docker pull ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.2-gpu这个3.2版本是支持cuda12的吗 |
Beta Was this translation helpful? Give feedback.
-
|
镜像已更新，大家可以尝试重新拉取 |
Beta Was this translation helpful? Give feedback.
-
目前镜像还是11.8，是还没有更新吗？
|
Beta Was this translation helpful? Give feedback.
-
|
自定义dockerfile的话，需要cuda12版本的ultra_infer，目前gpu_hpi.txt里面的版本是ultra-infer-gpu-python @ https://paddle-model-ecology.bj.bcebos.com/paddlex/PaddleX3.0/deploy/hpi/ultra_infer/releases/new_hpi/v1.2.0/ultra_infer_gpu_python-1.2.0-cp310-cp310-linux_x86_64.whl这个，是基于cuda11.8的，请问是否有cuda12版本的ultra_infer |
Beta Was this translation helpful? Give feedback.
-
|
在这里可以找到：https://github.com/PaddlePaddle/PaddleX/blob/release/3.2/paddlex/hpip_links_cu12.html |
Beta Was this translation helpful? Give feedback.
-
|
谢谢，已成功构建 |
Beta Was this translation helpful? Give feedback.
-
|
请教一下，现在通过服务化部署产线，在容器里面显示的Time taken 都在100ms左右，但我在C#通过http通信耗时在400ms左右，我想了解一下这个耗时会有哪些因素影响？怎么可以调整达到容器的推理的速度呢？ C#客户端的：推理耗时:456.5396毫秒 |
Beta Was this translation helpful? Give feedback.
-
|
这样的话考虑是通信的耗时，请问推理的图像是否较大，是否关闭结果可视化，以及服务端和客户端之间的网络带宽是否充足？ |
Beta Was this translation helpful? Give feedback.
-
|
图像是1000*888的，关闭了结果可视化，你说的网络带宽最小应该多少呢？ |
Beta Was this translation helpful? Give feedback.
-
|
可以打印一下客户端发送请求的timestamp，以及服务端接受到请求的timstamp（在SDK的model.py中添加打印代码），计算一下两者之间的差值，如果说这个耗时比较长，说明主要是网络通信的开销，可能需要考虑传递URL、在客户端缩减图像尺寸或者优化网络通信。 |
Beta Was this translation helpful? Give feedback.
-
|
shape参数有什么讲究吗？哪里可以看详细？ |
Beta Was this translation helpful? Give feedback.
-
|
这是TensorRT的功能，可以参考TensorRT相关文档了解更多信息～ |
Beta Was this translation helpful? Give feedback.
-
|
请问目前是不支持arm64版本进行高性能部署使用吗？ 对应的paddle信息： 启动命令：nohup paddlex --serve --pipeline OCR --port 38080 --use_hpip > /logs/ocr.log 2>&1 & 异常问题： |
Beta Was this translation helpful? Give feedback.
-
|
root@localhost:/work# paddlex --pipeline OCR \
Creating model: ('PP-LCNet_x1_0_doc_ori', None) 图片为什么使用官方的命令OCR识别图片也没有有看到文字提取是那个地方有问题？，但是如果不使用NPU是可以解析对应的图片的：https://img1.baidu.com/it/u=1855070411,442203363&fm=253&app=138&f=JPEG?w=800&h=1422 图片地址如下 |
Beta Was this translation helpful? Give feedback.
-
|
直接用命令对比就可以看出问题 请问是什么地方有问题？ root@localhost:/work# paddlex --pipeline OCR \
Creating model: ('PP-LCNet_x1_0_doc_ori', None) C++ Traceback (most recent call last):No stack trace in paddle, may be caused by external reasons. Error Message Summary:FatalError: Aborted (core dumped)
Creating model: ('PP-LCNet_x1_0_doc_ori', None) corrupted size vs. prev_size |
Beta Was this translation helpful? Give feedback.
-
|
经验证H20使用ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.2-gpu镜像（cuda11.8），会出现无法识别图片的情况（即：识别的图片是空的、返回的rec_texts也是空的），请问是否有支持cuda12.6的镜像可以使用？ |
Beta Was this translation helpful? Give feedback.
-
|
有12.6镜像用的，链接和11.8在一起的呀 |
Beta Was this translation helpful? Give feedback.
-
|
不行，用这个镜像跑server.sh命令的时候，会报server.sh: line 27: exec: tritonserver: not found |
Beta Was this translation helpful? Give feedback.
-
|
12.6的要自己自定义构建dockerfile的，目前官方hps镜像只支持cuda11.8的 |
Beta Was this translation helpful? Give feedback.
-
|
目前 50 系显卡是不是依旧不支持高性能推理以及高稳定性部署？ |
Beta Was this translation helpful? Give feedback.
-
|
是的，暂不支持 |
Beta Was this translation helpful? Give feedback.
-
|
你好，我正在v100服务器上使用高稳定性服务化部署OCR服务，实例配置情况为： 1 backend: "python"
2 max_batch_size: 16
3 input [
4 {
5 name: "input"
6 data_type: TYPE_STRING
7 dims: [ 1 ]
8 }
9 ]
10 output [
11 {
12 name: "output"
13 data_type: TYPE_STRING
14 dims: [ 1 ]
15 }
16 ]
17 instance_group [
18 {
19 count: 2
20 kind: KIND_GPU
21 gpus: [ 2, 3, 4 ]
22 }
23 ]实例可以正常运行。但当我使用nvidia-smi实时监控gpu占用时，发现在异步并发调用接口（grpc方式）时，始终只能在一张卡上进行推理计算，现象是：在gpu 2上推理几秒钟，切到gpu 3上推理几秒钟……始终无法在多块gpu上并行计算，请问这个是什么原因，有无解决办法呢？ |
Beta Was this translation helpful? Give feedback.
-
|
请问可以提供一下调用脚本吗？ |
Beta Was this translation helpful? Give feedback.
-
|
可以的，我是采用fastapi封装成api服务进行使用的，每次调用接口时传入一张图片的base64编码字符串。理论上fastapi接口可以支持多个请求同时访问接口，应该是并发独立调用send_post函数。 def send_post(file: str):
data_json: dict = {"file": file, "fileType": 1, "visualize": False}
output = triton_request(client, "ocr", data_json)
if output["errorCode"] != 0:
return [], []
result = output["result"]["ocrResults"][0]["prunedResult"]
return result["rec_texts"], result["dt_polys"]
@app.post("/realtime/ocrrec", response_model=RspImg)
async def ocr_rec(item: Item):
content, poly = send_post(item.img)
cif = ChineseInfoValidator()
ocrs: list[dict] = []
for c, p in zip(content, poly):
ocrs.append({"content": c, "position": p})
entities = cif.extract_and_validate("\n".join(content))
return {"ocr": ocrs, "entities": entities} |
Beta Was this translation helpful? Give feedback.
-
|
我猜测有可能是client限制了并发，可以考虑尝试创建多个client，看看是否可以实现并发。 |
Beta Was this translation helpful? Give feedback.
-
|
谢谢🤣。 async def async_triton_request(client, model_name, data, *, request_kwargs=None):
loop = asyncio.get_running_loop()
future = loop.create_future()
def callback(result, error):
if error:
loop.call_soon_threadsafe(future.set_exception, RuntimeError(error))
else:
loop.call_soon_threadsafe(future.set_result, result)
if request_kwargs is None:
request_kwargs = {}
input_ = triton_grpc.InferInput(constants.INPUT_NAME, [1, 1], "BYTES")
input_.set_data_from_numpy(_create_triton_input(data))
client.async_infer(model_name, inputs=[input_], callback=callback, **request_kwargs)
result = await future
output = result.as_numpy(constants.OUTPUT_NAME)
return _parse_triton_output(output)问题得到了解决。 |
Beta Was this translation helpful? Give feedback.
-
|
你说得对，之前在异步函数里调用同步方法，而且是比较耗时的同步方法，会阻塞住事件循环。另一种可能的解决方案是使用 |
Beta Was this translation helpful? Give feedback.
and others
Uh oh!
There was an error while loading. Please reload this page.
-
PaddleX/latest/pipeline_deploy/serving
https://paddlepaddle.github.io/PaddleX/latest/pipeline_deploy/serving.html
Beta Was this translation helpful? Give feedback.
All reactions