Skip to content

[BugFix] fix too many open files problem #3319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: release/2.0.2
Choose a base branch
from

Conversation

ltd0924
Copy link
Collaborator

@ltd0924 ltd0924 commented Aug 11, 2025

Add parameters to control concurrency, max-concurrency and max-waiting_time. Only support in online mode.
Description:

  • max-waiting-time: maximum wait time for establishing a connection upon service request. Default: -1 (indicates no wait time limit).
  • max-concurrency: the actual number of connections established by the service, default 512.

If the number of requests is bigger than max-concurrency, the service will return error response 429 (Too Many Requests).

If the waiting time is bigger than max-waiting-time , the service will return error response 408 (Request Timeout).

Usage:

python -m fastdeploy.entrypoints.openai.api_server \
--model xxx
--max-concurrency 1024 \
--max-waiting-time 300

Copy link

paddle-bot bot commented Aug 11, 2025

Thanks for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant