-
Notifications
You must be signed in to change notification settings - Fork 624
Description
Describe the issue as clearly as possible:
when using outlines[transformers] with pytorch models, i get torch compilation warnings about missing setuptools. the warnings indicate that pytorch can't compile certain kernels and falls back to slower execution. adding setuptools manually fixes this but it should be included in the transformers extra since pytorch compilation features require it at runtime.
Steps/code to reproduce the bug:
# mkdir /tmp/outlines-transformers; cd /tmp/outlines-transformers; uv init; uv add 'outlines[transformers]'
# paste contents below (from outlines transformers docs) in default hello.py file
# uv run hello.py
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer
from pydantic import BaseModel
from typing import List
class Character(BaseModel):
name: str
age: int
skills: List[str]
model = outlines.from_transformers(
AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
)
result = model("Create a character.", output_type=Character, max_new_tokens=200, repetition_penalty=0.5)
print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']
Expected result:
model should run without any warnings and with full pytorch compilation optimizations enabled
Error message:
W0826 15:14:55.338000 682 .venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py:1339] WON'T CONVERT _apply_token_bitmask_inplace_kernel /.../.venv/lib/python3.11/site-packages/outlines_core/kernels/torch.py line 43
W0826 15:14:55.338000 682 .venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py:1339] due to:
W0826 15:14:55.338000 682 .venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py:1339] Traceback (most recent call last):
[... long traceback ...]
W0826 15:14:55.338000 682 .venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py:1339] torch._inductor.exc.InductorError: ModuleNotFoundError: No module named 'setuptools'
Outlines/Python version information:
Version information
Context for the issue:
i believe this affects performance since pytorch can't compile the optimized kernels and falls back to slower execution. users installing outlines[transformers] expect all necessary dependencies for optimal performance to be included.
the issue will affect anyone using uv
since it only includes what is declared in dependencies. since outlines is specifically using pytorch compilation features, setuptools should be declared as a dependency in the transformers extra.