feat: Adds a vllm backend #122
Merged
+2,602
−406
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Merge ProtectionsYour pull request matches the following merge protections and will not be merged until they are valid. 🟢 Enforce conventional commitWonderful, this rule succeeded.Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
guicho271828
force-pushed
the
masa/vllm-backend
branch
6 times, most recently
from
ba3bb52 to
300c148
Compare
jakelorocco
reviewed
Sep 5, 2025
guicho271828
force-pushed
the
masa/vllm-backend
branch
3 times, most recently
from
93a191a to
69d3be8
Compare
jakelorocco
reviewed
Sep 17, 2025
jakelorocco
requested changes
Sep 17, 2025
Contributor
jakelorocco
left a comment
jakelorocco
left a comment
There was a problem hiding this comment.
looks mostly good to me; I left a few comments and looks like there's some pre-commit checks failing as well
guicho271828
force-pushed
the
masa/vllm-backend
branch
from
87cbc96 to
9701c08
Compare
jakelorocco
reviewed
Sep 23, 2025
Contributor
jakelorocco
left a comment
jakelorocco
left a comment
There was a problem hiding this comment.
I think this looks good. There will be conflicts / changes required based on my async PR: #137. I'm happy to make those commits though once the async stuff gets merged and this branch gets rebased.
guicho271828
force-pushed
the
masa/vllm-backend
branch
5 times, most recently
from
e84ff46 to
048e90d
Compare
guicho271828
force-pushed
the
masa/vllm-backend
branch
4 times, most recently
from
61d6384 to
291b459
Compare
Contributor
Author
|
async support is already implemented |
Signed-off-by: Masataro Asai <[email protected]>
Signed-off-by: Masataro Asai <[email protected]>
Signed-off-by: Masataro Asai <[email protected]>
guicho271828
force-pushed
the
masa/vllm-backend
branch
from
5009659 to
186af0f
Compare
guicho271828
force-pushed
the
masa/vllm-backend
branch
from
186af0f to
55397ad
Compare
jakelorocco
reviewed
Oct 22, 2025
…PR that migrates to V1
jakelorocco
approved these changes
Oct 22, 2025
Contributor
|
I just merged main. If tests do not predictably pass in the CI/CD pipeline, we should mark those as non-cicd tests (ask @avinash2692 if you need a pointer on how to do this). Please run the full test suite locally after the main merge and ensure all tests are passing locally. Other than that, LGTM. |
tuliocoppola
pushed a commit
to tuliocoppola/mellea
that referenced
this pull request
Nov 5, 2025
* feat: added smaller qwen models for debugging Signed-off-by: Masataro Asai <[email protected]> * feat(vllm): copied from huggingface Signed-off-by: Masataro Asai <[email protected]> * fix(vllm): remove alora and cache Signed-off-by: Masataro Asai <[email protected]> * fix(vllm): remove tool calls Signed-off-by: Masataro Asai <[email protected]> * fix(vllm): finished the implementation with limited functionality: free-form and constrained generation Signed-off-by: Masataro Asai <[email protected]> * fix(vllm): passing mypy and linter Signed-off-by: Masataro Asai <[email protected]> * fix(vllm): added vllm optional dep in pyproject.toml Signed-off-by: Masataro Asai <[email protected]> * feat(vllm test): copied from huggingface Signed-off-by: Masataro Asai <[email protected]> * fix(vllm test): implemented the test Signed-off-by: Masataro Asai <[email protected]> * test: require V0 in vllm test Signed-off-by: Masataro Asai <[email protected]> * refactor: ctx to chat conversion function * refactor: use_alora function * refactor: moved _extract_model_tool_requests to mellea.backends.utils * feat(vllm): added tool calls * test(tools): run test with mistral * fix(vllm): rename model_options -> engine_args * fix(vllm): use FancyLogger * fix(vllm): ignore type checking for vllm and msgspec * fix(vllm): fixed the backend name in the log * feat(vllm): asynchronous call support * test(vllm): asynchronous call support * fix(vllm): avoid unnecessary incremental processing in non-streaming mode * fix(vllm): fix for the new return format * fix(vllm): fixed vllm test for the new contexts * fix(vllm): addressed minor comments * fix(vllm): uv lock * fix(vllm): mark V0 api test qualitative; will be removed in a future PR that migrates to V1 --------- Signed-off-by: Masataro Asai <[email protected]> Co-authored-by: MASATARO ASAI [email protected] <[email protected]> Co-authored-by: Nathan Fulton <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
basic VLLM backend without tool and alora support.
VLLM_V1=0to be set when running.