Bug introduced in `from_pretrained` `v4.48.3`..`v4.49.0`

[hlky]

Hi 🤗

Diffusers 🧨 noticed some failing tests starting with v4.49.0 in Kolors, one of our models that uses a custom text encoder.

Reproduction

This is working on v4.48.3.

from transformers import AutoModel

model = AutoModel.from_pretrained("hf-internal-testing/tiny-random-chatglm3-6b", trust_remote_code=True)

On v4.49.0:

TypeError: empty() received an invalid combination of arguments - got (tuple, dtype=str, device=str), but expected one of:
 * (tuple of ints size, *, tuple of names names, torch.memory_format memory_format = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
 * (tuple of ints size, *, torch.memory_format memory_format = None, Tensor out = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)

The issue seems to be that the config in the test model and checkpoints like Kwai-Kolors/Kolors-diffusers contain torch_dtype as a string.

On Diffusers end explicitly setting torch_dtype when using ChatGLMModel and setting a default torch_dtype for from_pretrained paths is working huggingface/diffusers#10816 and it's mainly internal effects as torch_dtype wasn't passed for some tests, should be ok for end users as they would generally pass torch_dtype.

[zucchini-nlp]

Hey @hlky ! Thanks for reporting that. Seems to have been broken after the recent addition of dtype for composite configs. I will submit a PR to fix it

[wycoal]

same problem ， how to fix? thanks

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.