-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Open
Copy link
Labels
AutoDeploy<NV> AutoDeploy Backend<NV> AutoDeploy Backendfeature requestNew feature or request. This includes new model, dtype, functionality supportNew feature or request. This includes new model, dtype, functionality support
Description
🚀 The feature, motivation and pitch
Now that PT LlmArgs have mostly stabilized, let's see if we can more closely align AD LlmArgs with PT LlmArgs:
- Deprecate
AutoDeployConfigas separate class and just move it all into AD'sLlmArgsclass - Align any field in AD's LlmArgs if there is a corresponding field in PT
- Consider inheriting from PT's LlmArgs instead of BaseLlmArgs
This help ensuring AD can serve as a drop-in replacement for PT backend where needed and simplify integration management (trtllm-serve, trtllm-bench, dynamo, NIM, ...)
Alternatives
No response
Additional context
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
AutoDeploy<NV> AutoDeploy Backend<NV> AutoDeploy Backendfeature requestNew feature or request. This includes new model, dtype, functionality supportNew feature or request. This includes new model, dtype, functionality support
Type
Projects
Status
Ready