draft to test dduf compatibility with file explorer

[Wauplin]

Related to huggingface/huggingface_hub#2692, huggingface/diffusers#10037 and #35093.

And more precisely to huggingface/huggingface_hub#2703 which aims at unifying the interface to handle a filesystem (e.g. a local directory) and a zip file (e.g. a DDUF file). Still a lot of changes to make if we go in this direction. I preferred to open the PR early for discussions.

Reproduction steps:

Install

pip install git+https://github.com/huggingface/transformers.git@dduf-compatibility-with-file-explorer
pip install git+https://github.com/huggingface/huggingface_hub.git@draft-file-explorer

Run on test DDUF file

from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

dduf_path = hf_hub_download("DDUF/Flux.1-Dev-Tnf4-TE2NF4", "Flux.1-Dev-Tnf4-TE2NF4.dduf")

tokenizer = AutoTokenizer.from_pretrained(dduf_path, subfolder="tokenizer")

[Wauplin]

It was fun for the science but let's go for a simple solution 😄

(discussed offline but changing stuff in transformers for the sake of diffusers' DDUF is meeh 🤷‍♂️ maintenance will be a huge burden, been able to load from zip affects many internal methods for low impact, it might bring breaking changes and many untested variations => let's drop it and let diffusers do the rest. Worth case we unzip the transformers's components before loading them. We want a v0 to ship, not a perfect integration aux petits oignons)