fix: some minor fixes #223

jakelorocco · 2025-11-04T18:43:01Z

I ran into enough issues where I decided to fix them all. All tests pass locally now.

Fixes: #222 (comment)

List of changes:

vllm versioning: the vllm package was defaulting to a lower version when installing with .[all] vs .[vllm]; I could not figure out why but forcing the version works.
vllm bug with format resposne
- some vllm servers don't like it when you just specify text, removed it from openai and litellm
vllm issues
- changed generate_from_raw to utilize the event_loop helper like Ollama
- added some minor event loop handling to vllm so that it can be utilized between both async and sync mfunc calls
vllm tests
- Changed the skip condition to run at the fixture level and skip all other tests if it fails
- Fixed the calls to generate_from_raw that got missed when the refactor happened
add ctx to val result
- added example for accessing and utilizing these values
- fixed issue with aloras not getting added to context properly
- added tests
remove the β tests that always fail
add warning to huggingface generate_from_raw with mps
- there's a but with our version of pytorch that causes batched requests to only populate the last item in the batch
fixed a doc folder that had .py in its name
removed pytest-xdist since it wasn't helping and was causing issues when running locally

mergify · 2025-11-04T18:43:36Z

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert|release)(?:\(.+\))?:

avinash2692 · 2025-11-04T19:10:25Z

mellea/backends/huggingface.py

+            FancyLogger.get_logger().warning(
+                "utilizing device mps with a `generate_from_raw` request; you may see issues when submitting batches of prompts to a huggingface backend; ensure all ModelOutputThunks have non-empty values."
+            )
+


Ah that's what this was!! I was having issues when I was running hf tests for this but it disappeared when I stepped into the while debugging. Thanks for adding this!

avinash2692

LGTM except not sure why the we're moving response_format into extra_params. Let me know if there is a reasoning for it and it can be documented here for posterity.

avinash2692 · 2025-11-04T19:33:54Z

mellea/backends/litellm.py

+        extra_params: dict[str, Any] = {}
        if _format is not None:
-            response_format = {
+            extra_params["response_format"] = {


any reason for the additional abstraction?

I agree with avi; response_format = None is better if the old value causes errors

response_format = None sometimes causes issues as well with some backends (at least with the OpenAI backend I believe it used to). It's best to just not pass a response_format parameter if possible.

mellea/backends/openai.py

avinash2692 · 2025-11-04T19:54:29Z

Fixes issues related to vllm-project/vllm#26639

guicho271828

approved, but its nice to fix minor requests

guicho271828 · 2025-11-04T19:06:26Z

mellea/backends/vllm.py

+        # if switching between async and sync calls.
+        if el != self._event_loop:
+            self._underlying_model.shutdown_background_loop()
+            self._underlying_model.start_background_loop()


optional: assert el == self._underlying_model.background_loop ?

cf. https://docs.vllm.ai/en/v0.6.3/_modules/vllm/engine/async_llm_engine.html#AsyncLLMEngine.start_background_loop

They call that a background_loop but it's not an event loop, it's actually a Future. Even the _background_loop_unshielded is a Task object.

I think it's fine to manage the reference to the event loop on our side. We only ever have the one AsyncLLMEngine per LocalVLLMBackend so there shouldn't be issues with us tracking it this way. Happy to change it if it causes issues later on.

guicho271828 · 2025-11-04T20:34:45Z

mellea/backends/litellm.py

+        extra_params: dict[str, Any] = {}
        if _format is not None:
-            response_format = {
+            extra_params["response_format"] = {


I agree with avi; response_format = None is better if the old value causes errors

* fix: enforce minimum vllm version * fix: remove tests that look for "β" * fix: remove default response_format from litellm and openai backends * fix: remove xdist from pytests * fix: fix vllm tests * fix: vllm async event loop * feat: add contexts to validation results * fix: add warning for mps with huggingface generate from raw * fix: remove .py from folder name * fix: remove pytest-xdist specific args * fix: add exception with vllm backend when env var not set

jakelorocco added 8 commits November 4, 2025 09:06

fix: enforce minimum vllm version

e0ddfe2

fix: remove tests that look for "β"

70bac6a

fix: remove default response_format from litellm and openai backends

b0bd8ee

fix: remove xdist from pytests

f2c067c

fix: fix vllm tests

e521377

fix: vllm async event loop

7d27532

feat: add contexts to validation results

70d92f1

fix: add warning for mps with huggingface generate from raw

fa4c4c6

jakelorocco added 2 commits November 4, 2025 13:44

fix: remove .py from folder name

b8582e6

fix: remove pytest-xdist specific args

46bc909

jakelorocco requested review from avinash2692, guicho271828 and nrfulton November 4, 2025 18:51

jakelorocco marked this pull request as ready for review November 4, 2025 18:52

avinash2692 reviewed Nov 4, 2025

View reviewed changes

fix: add exception with vllm backend when env var not set

692b395

avinash2692 reviewed Nov 4, 2025

View reviewed changes

avinash2692 approved these changes Nov 4, 2025

View reviewed changes

guicho271828 approved these changes Nov 4, 2025

View reviewed changes

jakelorocco merged commit 7fa0891 into main Nov 4, 2025
3 of 4 checks passed

jakelorocco deleted the jal/minor-fixes branch November 4, 2025 21:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: some minor fixes #223

fix: some minor fixes #223

jakelorocco commented Nov 4, 2025 •

edited

Loading

Uh oh!

mergify bot commented Nov 4, 2025

Uh oh!

avinash2692 Nov 4, 2025

Uh oh!

avinash2692 left a comment

Uh oh!

avinash2692 Nov 4, 2025

Uh oh!

guicho271828 Nov 4, 2025

Uh oh!

jakelorocco Nov 4, 2025

Uh oh!

Uh oh!

avinash2692 commented Nov 4, 2025

Uh oh!

guicho271828 left a comment

Uh oh!

guicho271828 Nov 4, 2025

Uh oh!

jakelorocco Nov 4, 2025

Uh oh!

guicho271828 Nov 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fix: some minor fixes #223

fix: some minor fixes #223

Conversation

jakelorocco commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify bot commented Nov 4, 2025

Merge Protections

🟢 Enforce conventional commit

Uh oh!

avinash2692 Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

avinash2692 left a comment

Choose a reason for hiding this comment

Uh oh!

avinash2692 Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

guicho271828 Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

jakelorocco Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

avinash2692 commented Nov 4, 2025

Uh oh!

guicho271828 left a comment

Choose a reason for hiding this comment

Uh oh!

guicho271828 Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

jakelorocco Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

guicho271828 Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jakelorocco commented Nov 4, 2025 •

edited

Loading