Skip to content

[Feature]: huggingface revision (aka tag) for serve command #9463

@oplatek

Description

@oplatek

🚀 The feature, motivation and pitch

The HugginFace Hub support revisions (git tags) for model versions.

You can now use the command

# download manually the exact revision
huggingface-cli download YourOrg/YourHuggingFaceModel \
  --revision YourRevision \
  --local-dir ./YourHuggingFaceModel-YourRevision

# serve the exact revision from local file
trtllm-serve ./YourHuggingFaceModel-YourRevision 

I can now serve the model directly from the hugginface but only from the main default branch

trtllm-serve YourOrg/YourHuggingFaceModel

It will be great to add to start serving the exact version directly (very useful for Docker deployment)

trtllm-serve YourOrg/YourHuggingFaceModel --revision YourRevision  # add the revision flag
trtllm-serve YourOrg/YourHuggingFaceModel:YourRevision  # or Docker style revision specification

See HF docs about model version.

See Olmo model checkpoints versions named after step number

Alternatives

The revision parameter is supported in the llmapi

revision: Optional[str] = Field(

Additional context

No response

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Metadata

Metadata

Assignees

Labels

feature requestNew feature or request. This includes new model, dtype, functionality support

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions