[Feature]: Support transformers>=5

[Whadup]

🚀 The feature, motivation and pitch

vllm requires transformers<5, but some new models already require transformers 5.0.0rc0.
The combination of vllm>=0.12 and transformers>=5.0.0rc0 cannot be resolved.

Ignoring the constraint and installing transformers 5 anyways works, so hopefully supporting it officially is not a lot of work.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

@hmellor has put in a lot of work to get vLLM to work with Transformers main branch, but since Transformers v5 is still not released on PyPI yet, we cannot officially guarantee compatibility at the moment.

[ArthurZucker]

Final v5 should be ~1 month away, we really want to make sure inter-compatibility is good and we don't break stuff that is not supposed to be broken! 🤗 we'll push hard!

[jiangwu300]

@ArthurZucker v5+ seems to break for Kimi K2.5, not from vllm serve, but offline vllm mode and from Ray Data LLM. Wondering if anyone has any insight on this?

[hmellor]

Could you please make an issue detailing the issue and how to reproduce it and tag me? I'll look into it

[ArthurZucker]

Yep happy to have a look!

[jiangwu300]

@hmellor @ArthurZucker I'm running the following: https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/batch_llm_inference.py with nightly Ray and vllm 0.15.1. I force upgrade transformers to 5.0.0, and run the same snippet and the code will break (after fixing the dependency of TRANSFORMERS_CACHE in Ray. Replace all of

from transformers.utils.hub import TRANSFORMERS_CACHE

with

from huggingface_hub.constants import default_cache_path as TRANSFORMERS_CACHE

since TRANSFORMERS_CACHE is deprecated in v5. The following error occurs (I just tested with new moonshotai/Kimi-K2.5 so your changes should already be in this.):

Traceback (most recent call last):
  File "/home/ray/test.py", line 76, in <module>
    ds.show(limit=1)
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/data/dataset.py", line 3760, in show
    for row in self.take(limit):
               ^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/data/dataset.py", line 3682, in take
    for row in limited_ds.iter_rows():
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/data/iterator.py", line 285, in _wrapped_iterator
    for batch in batch_iterable:
                 ^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/data/iterator.py", line 216, in _create_iterator
    ) = self._to_ref_bundle_iterator()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/data/_internal/iterator/iterator_impl.py", line 37, in _to_ref_bundle_iterator
    ) = self._base_dataset._execute_to_iterator()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/data/dataset.py", line 6813, in _execute_to_iterator
    bundle_iter, stats, executor = self._plan.execute_to_iterator()
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/data/exceptions.py", line 87, in handle_trace
    raise e.with_traceback(None)
ray.exceptions.RayTaskError(UserCodeException): ray::MapWorker(MapBatches(TokenizeUDF)).submit() (pid=29970, ip=172.17.0.2, actor_id=4102814384d00ea911a68fcc01000000, repr=MapWorker(MapBatches(TokenizeUDF)))
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/llm/_internal/batch/stages/base.py", line 198, in __call__
    async for output in self.udf(normal_rows):
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/llm/_internal/batch/stages/tokenize_stage.py", line 59, in udf
    self.tokenizer([row["prompt"] for row in batch])["input_ids"],
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2456, in __call__
    encodings = self._encode_plus(
                ^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/site-packages/transformers/tokenization_python.py", line 785, in _encode_plus
    batch_outputs = self.pad(
                    ^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2630, in pad
    raise ValueError(
ValueError: type of None unknown: <class 'NoneType'>. Should be one of a python, numpy, or pytorch object.

Let me know if you want more information on reproducing or if you have any questions, thanks!

[hmellor]

Thanks for the details.

@zucchini-nlp has found the same issue elsewhere and is working on fixing it in huggingface/transformers#43768

It's because by default we we're adding cls-bos special tokens but not all tokenizers have those ids. Therefore, we end up having None token ids in the encoded sequence