[feat] Enable mm caching for transformers backend

[zucchini-nlp]

As per title, I didn't find a reason for the two caching to be tied. After this change, we can also stop asking users to explicitly set disable_mm_caching=True in transformers backend and always use the no-cache code path when processing

After it is merged, this blogpost has to be updated as well: vllm-project/vllm-project.github.io#61

cc @hmellor

[gemini-code-assist]

Code Review

This pull request correctly decouples multi-modal (MM) caching from prefix caching. The changes are logical and well-targeted. By switching the condition in need_extra_keys from request.mm_positions to request.mm_hashes, the prefix caching for MM inputs is now correctly triggered only when MM hashes are available. The removal of the error-raising check in the transformers backend complements this by allowing it to gracefully opt-out of MM caching. The documentation update is also consistent with these changes. The fix appears solid and addresses the described issue effectively.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

Could you elaborate a bit? Do you mean that because you return mm_hashes=None, the multi-modal cache is automatically disabled so there is no need for users to set the flag?

[zucchini-nlp]

Do you mean that because you return mm_hashes=None, the multi-modal cache is automatically disabled so there is no need for users to set the flag?

Exactly! In current version there is a dependency if disable_mm_cache=True then we must as well enable_prefix_cache=False, otherwise transformers model fails. It is due to the updated lines we assume that mm cache is present if mm positions are in the request. The bug wasn't seen before because I had disabled prefix caching and without it cache logic is entirely skipped

[DarkLight1337]

Hmm, then what happens if the user explicitly sets disable_mm_cache after this change? Won't they still get this error?

[DarkLight1337]

Btw, why do we have to skip computing the hashes in the first place? Even if it is not used in the multimodal processor, we can still use it for caching multimodal embeddings which is used for chunked prefill.

[zucchini-nlp]

Btw, why do we have to skip computing the hashes in the first place?

We can't cache during processing the same way as vLLM processors do, because vLLM applies text-only/mm-only processing and then adds placeholders manually. Transformers doesn't support it per model and it will be too much work to implement, so yesterday we decided processor caching will not an option for Transformers backend with @hmellor

Even if it is not used in the multimodal processor, we can still use it for caching multimodal embeddings which is used for chunked prefill.

Do we still need to compute caches from processor code for that? Hmm, I assumed prefix caching and chunked prefill re-compute the cache again. In that case, how is mm caching linked to other parts of vLLM?

Hmm, then what happens if the user explicitly sets disable_mm_cache after this change? Won't they still get this error?

It won't change anything for users, disable_mm_cache will become a no-op. It is also no-op now, but we just enforce users to pass flags every time

[DarkLight1337]

IIRC we only compute the cache once in the multimodal processor in P0 and pass the hash directly to P1 where the model is run. So it is still helpful to compute the hashes

[DarkLight1337]

@ywang96 can help verify in case something changed. It has been a while since I last looked at that code and I'm outside rn

[zucchini-nlp]

Ah oke, lemme see, would be great to add a test with prefix caching as well

[zucchini-nlp]

Inference works fine, though I don't see any special prefix caching tests on multimodal models. Should I add a new testcase or existing ones are enough?

[Isotr0py]

LGTM!

[DarkLight1337]

Inference works fine, though I don't see any special prefix caching tests on multimodal models. Should I add a new testcase or existing ones are enough?

It's enabled by default so no need to add tests

[Isotr0py]

Inference works fine, though I don't see any special prefix caching tests on multimodal models. Should I add a new testcase or existing ones are enough?

We have batching same images inputs test case to cover prefix-caching, so existing ones is enough:

vllm/tests/models/multimodal/generation/vlm_utils/types.py

Line 42 in 10904e6

IMAGE_SIZE_FACTORS = [(), (1.0, ), (1.0, 1.0, 1.0), (0.25, 0.5, 1.0)]

[DarkLight1337]

The transformers models tests passed, merging early