✨ vLLM Backend integration #42
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR extends the PR: Deepsparse Backend implementation.
The base branch is
parfeniukink/features/deepsparse-backend
.vllm
is added to optional dependenciesVllmBackend
class encapsulates the vLLM integration.guidellm/backend/vllm
is available only if the Python version and the runtime platform pass the validation.vllm
tests are skipped in case the platform is not LinuxUsage
This is an example of a command you can use in your terminal:
--data=openai_humaneval
: determines the dataset--model=/local/path/my_model
: determines the local path to the model object. If not specified - the env variable will be used.Environment configuration
The model could also be set with
GUIDELLM__LLM_MODEL
. If the CLI value or environment variable is not set, then the default will be used. Currently, the default model is:mistralai/Mistral-7B-Instruct-v0.3
.