fix chameleonvision UT failure

[yao-matrix]

as discussed in huggingface/diffusers#11690, ut pytest -rA tests/models/chameleon/test_modeling_chameleon.py::ChameleonVision2SeqModelTest::test_model_parallel_beam_search w/ 2 cards, the error log is "RuntimeError: Expected all tensors to be on the same device, but found at least two devices,src/transformers/models/chameleon/modeling_chameleon.py", the reason is even residual is in the same device as hidden_states at the beginning, but after they went through some operators as both input and output, they finally placed to different device, but when they come to + which is not a nn.Module(so accelerate cannot pre-hook it), error happens.

so, update no_split_modules in ChameleonVQVAE to avoid it. @SunMarc, pls help review, thx.

[SunMarc]

Thanks !

[SunMarc]

Thanks ! we can also put instead ChameleonVQVAEEncoder if these blocks are small enough

[yao-matrix]

seems ChameleonVQVAEEncoder is a mega block w/ multiple ChameleonVQVAEEncoderResnetBlock, ChameleonVQVAEEncoderConvDownsample and ChameleonVQVAEEncoderAttnBlock(

transformers/src/transformers/models/chameleon/modeling_chameleon.py

Line 650 in 4abb053

class ChameleonVQVAEEncoder(nn.Module):

), so maybe better not to put ChameleonVQVAEEncoder

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: chameleon