Skip to content

Conversation

DrRyanHuang
Copy link
Contributor

@DrRyanHuang DrRyanHuang commented Aug 28, 2025

本 PR 依赖 Paddle 主框架的两个PR:


#3302 添加了 append_attention_with_output 但是开启后存在打断,本PR消除 full_cuda_graph=false 时的打断

动态图下运行的 cpp_extensions,都是不需要 key_cache_outvalue_cache_out
本PR移除自定义算子注册的 key_cache_outvalue_cache_out,与动态图对齐

另外静态图没有 place,SOT转静的时候会打断:

90afbd244795e119d2a92ca04622f8bb

故而移除 .to(qkv.place)

PS: 目前CUDAGraph + 子图切分的脚本:

source /workspace/env/.py3.10/bin/activate
export MODEL=/ssd1/EB_MODELS/ERNIE-4.5-0.3B-Paddle
rm -rf log/*

export http_proxy=xxx.com:xx
export https_proxy=xxx.com:xx
export no_proxy=xxxx

export FLAGS_cuda_graph_blacklist="custom_op.static_op_append_attention_with_output_"
export PYTHONPATH=/workspace/FastDeploy:/workspace/Paddle/build/python:$PYTHONPATH
export CUDA_VISIBLE_DEVICES=3
export PYTHON_EXCUTOR=/workspace/env/.py3.10/bin/python
export PORT=9905

python -m fastdeploy.entrypoints.openai.api_server \
  --model $MODEL \
  --metrics-port 9717 \
  --port 9718 \
  --engine-worker-queue-port 9719 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --max-num-seqs 128 \
  --quantization wint4 \
  --graph-optimization-config '{"graph_opt_level": 1, "use_cudagraph": true, "full_cuda_graph": false}' \

cc @SigureMo @zyfncg @gongshaotian

Copy link

paddle-bot bot commented Aug 28, 2025

Thanks for your contribution!

gongshaotian
gongshaotian previously approved these changes Aug 28, 2025
Copy link
Collaborator

@gongshaotian gongshaotian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codecov-commenter
Copy link

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (develop@808b548). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff            @@
##             develop   #3694   +/-   ##
=========================================
  Coverage           ?   0.00%           
=========================================
  Files              ?       1           
  Lines              ?       3           
  Branches           ?       0           
=========================================
  Hits               ?       0           
  Misses             ?       3           
  Partials           ?       0           
Flag Coverage Δ
diff 0.00% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@DrRyanHuang DrRyanHuang changed the title [SOT] Eliminate BreakGraph caused by #3302 [SOT] Eliminate BreakGraph caused by #3302 && update CustomOp Sep 4, 2025
@DrRyanHuang DrRyanHuang changed the title [SOT] Eliminate BreakGraph caused by #3302 && update CustomOp [SOT][Cudagraph] Remove BreakGraph of #3302 && update CustomOp Sep 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants