Skip to content

Conversation

parfeniukink
Copy link
Contributor

@parfeniukink parfeniukink commented Aug 30, 2024

Summary

This PR extends the PR: Deepsparse Backend implementation.
The base branch is parfeniukink/features/deepsparse-backend.

  • vllm is added to optional dependencies
  • The VllmBackend class encapsulates the vLLM integration.
  • The guidellm/backend/vllm is available only if the Python version and the runtime platform pass the validation.
  • vllm tests are skipped in case the platform is not Linux

Usage

This is an example of a command you can use in your terminal:

  • --data=openai_humaneval: determines the dataset
  • --model=/local/path/my_model: determines the local path to the model object. If not specified - the env variable will be used.
python -m src.guidellm.main --data=openai_humaneval --max-requests=1 --max-seconds=20 --rate-type=constant --rate=1.0 --backend=vllm --model=/local-path

Environment configuration

The model could also be set with GUIDELLM__LLM_MODEL. If the CLI value or environment variable is not set, then the default will be used. Currently, the default model is: mistralai/Mistral-7B-Instruct-v0.3.

Dmytro Parfeniuk added 2 commits August 30, 2024 13:22
* backend/test_openai_backend.py -> backend/test_openai.py
* backend/test_deepsparse_backend.py -> backend/test_deepsparse.py
@parfeniukink parfeniukink self-assigned this Aug 30, 2024
@parfeniukink parfeniukink requested a review from markurtz August 30, 2024 11:10
@parfeniukink parfeniukink marked this pull request as ready for review September 10, 2024 07:44
@markurtz markurtz closed this Apr 22, 2025
@sjmonson sjmonson deleted the parfeniukink/features/vllm-backend branch September 29, 2025 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants