Skip to content

【Refactored】Use shared memory for request information management and significantly reducing CPU overhead.#716

Merged
hiworldwzj merged 157 commits intomainfrom
new_router
Feb 6, 2025
Merged

【Refactored】Use shared memory for request information management and significantly reducing CPU overhead.#716
hiworldwzj merged 157 commits intomainfrom
new_router

Conversation

@hiworldwzj
Copy link
Collaborator

Main improvements:

  1. Utilized the ctypes library and shared memory operations to pass request information and token output between processes.
  2. Improved the allocation and management of tokens and request objects in the mem_manager and req_manager.
  3. Reduced CPU overhead.

@hiworldwzj hiworldwzj changed the title 【Refactored】Use shared memory for request information management while significantly reducing CPU overhead. 【Refactored】Use shared memory for request information management and significantly reducing CPU overhead. Feb 6, 2025
@hiworldwzj hiworldwzj merged commit b8cfd70 into main Feb 6, 2025
1 check passed
@shihaobai shihaobai deleted the new_router branch May 29, 2025 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants