From a21ad398d7a355e989c06c751f54c43dd030abe8 Mon Sep 17 00:00:00 2001 From: stevhliu Date: Thu, 28 Aug 2025 15:17:18 -0700 Subject: [PATCH 1/8] init --- docs/source/en/_toctree.yml | 2 +- .../en/using-diffusers/other-formats.md | 494 +++--------------- 2 files changed, 81 insertions(+), 415 deletions(-) diff --git a/docs/source/en/_toctree.yml b/docs/source/en/_toctree.yml index 14dbfe3ea1d3..329d1b5471e6 100644 --- a/docs/source/en/_toctree.yml +++ b/docs/source/en/_toctree.yml @@ -29,7 +29,7 @@ - local: using-diffusers/scheduler_features title: Scheduler features - local: using-diffusers/other-formats - title: Model files and layouts + title: Model formats - local: using-diffusers/push_to_hub title: Sharing pipelines and models diff --git a/docs/source/en/using-diffusers/other-formats.md b/docs/source/en/using-diffusers/other-formats.md index 59835bbf2622..2d47cdaa6d32 100644 --- a/docs/source/en/using-diffusers/other-formats.md +++ b/docs/source/en/using-diffusers/other-formats.md @@ -10,504 +10,170 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o specific language governing permissions and limitations under the License. --> -# Model files and layouts - [[open-in-colab]] -Diffusion models are saved in various file types and organized in different layouts. Diffusers stores model weights as safetensors files in *Diffusers-multifolder* layout and it also supports loading files (like safetensors and ckpt files) from a *single-file* layout which is commonly used in the diffusion ecosystem. - -Each layout has its own benefits and use cases, and this guide will show you how to load the different files and layouts, and how to convert them. - -## Files +# Model formats -PyTorch model weights are typically saved with Python's [pickle](https://docs.python.org/3/library/pickle.html) utility as ckpt or bin files. However, pickle is not secure and pickled files may contain malicious code that can be executed. This vulnerability is a serious concern given the popularity of model sharing. To address this security issue, the [Safetensors](https://hf.co/docs/safetensors) library was developed as a secure alternative to pickle, which saves models as safetensors files. - -### safetensors +Diffusion models are typically stored in the Diffusers format or single-file format. Model files can be stored in various file types such as safetensors, dduf, or ckpt. > [!TIP] -> Learn more about the design decisions and why safetensor files are preferred for saving and loading model weights in the [Safetensors audited as really safe and becoming the default](https://blog.eleuther.ai/safetensors-security-audit/) blog post. +> Format refers to the directory structure and file refers to the file type. -[Safetensors](https://hf.co/docs/safetensors) is a safe and fast file format for securely storing and loading tensors. Safetensors restricts the header size to limit certain types of attacks, supports lazy loading (useful for distributed setups), and has generally faster loading speeds. +This guide will show you how to load pipelines and models from these formats and files. -Make sure you have the [Safetensors](https://hf.co/docs/safetensors) library installed. +## Diffusers format -```py -!pip install safetensors -``` +The Diffusers format stores each model (UNet, transformer, text encoder) in a separate subfolder. There are several benefits to storing models separately. + +- Faster overall pipeline initialization because you can load the individual model you need or load them all in parallel. +- Reduced memory usage because you don't need to load all the pipeline components if you only need one model. [Reuse](./loading#reusing-models-in-multiple-pipelines) a model that is shared between multiple pipelines. +- Lower storage requirements because common models shared between multiple pipelines are only downloaded once. +- Flexibility to use new or improved models in a pipeline. -Safetensors stores weights in a safetensors file. Diffusers loads safetensors files by default if they're available and the Safetensors library is installed. There are two ways safetensors files can be organized: +## Single file format -1. Diffusers-multifolder layout: there may be several separate safetensors files, one for each pipeline component (text encoder, UNet, VAE), organized in subfolders (check out the [stable-diffusion-v1-5/stable-diffusion-v1-5](https://hf.co/stable-diffusion-v1-5/stable-diffusion-v1-5/tree/main) repository as an example) -2. single-file layout: all the model weights may be saved in a single file (check out the [WarriorMama777/OrangeMixs](https://hf.co/WarriorMama777/OrangeMixs/tree/main/Models/AbyssOrangeMix) repository as an example) +A single-file format stores *all* the model (UNet, transformer, text encoder) weights in a single file. Benefits of single-file formats include the following. - - +- Greater compatibility with [ComfyUI](https://github.com/comfyanonymous/ComfyUI) or [Automatic1111](https://github.com/AUTOMATIC1111/stable-diffusion-webui). +- Easier to download and share a single file. -Use the [`~DiffusionPipeline.from_pretrained`] method to load a model with safetensors files stored in multiple folders. +Use [`~loaders.FromSingleFileMixin.from_single_file`] to load a single file. ```py +import torch from diffusers import DiffusionPipeline -pipeline = DiffusionPipeline.from_pretrained( - "stable-diffusion-v1-5/stable-diffusion-v1-5", - use_safetensors=True +pipeline = DiffusionPipeline.from_single_file( + "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors", + torch_dtype=torch.float16, + device_map="cuda" ) ``` - - - -Use the [`~loaders.FromSingleFileMixin.from_single_file`] method to load a model with all the weights stored in a single safetensors file. +The [`~loaders.FromSingleFileMixin.from_single_file`] method also supports passing new models or schedulers. ```py -from diffusers import StableDiffusionPipeline +import torch +from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler -pipeline = StableDiffusionPipeline.from_single_file( - "https://huggingface.co/WarriorMama777/OrangeMixs/blob/main/Models/AbyssOrangeMix/AbyssOrangeMix.safetensors" +scheduler = DPMSolverMultistepScheduler() +pipeline = DiffusionPipeline.from_single_file( + "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors", + scheduler=scheduler, + torch_dtype=torch.float16, + device_map="cuda" ) ``` - - +### Configuration options + -#### LoRAs +## File types -[LoRAs](../tutorials/using_peft_for_inference) are lightweight checkpoints fine-tuned to generate images or video in a specific style. If you are using a checkpoint trained with a Diffusers training script, the LoRA configuration is automatically saved as metadata in a safetensors file. When the safetensors file is loaded, the metadata is parsed to correctly configure the LoRA and avoids missing or incorrect LoRA configurations. +Models can be stored in several file types. Safetensors is the most common file type but you may encounter other file types on the Hub or diffusion community. + +### safetensors -The easiest way to inspect the metadata, if available, is by clicking on the Safetensors logo next to the weights. +[Safetensors](https://hf.co/docs/safetensors) is a safe and fast file type for securely storing and loading tensors. It restricts the header size to limit certain types of attacks, supports lazy loading (useful for distributed setups), and generally loads faster. -
- -
+Diffusers loads safetensors file by default if they are available and the Safetensors library is installed. -For LoRAs that aren't trained with Diffusers, you can still save metadata with the `transformer_lora_adapter_metadata` and `text_encoder_lora_adapter_metadata` arguments in [`~loaders.FluxLoraLoaderMixin.save_lora_weights`] as long as it is a safetensors file. +Use [`~DiffusionPipeline.from_pretrained`] or [`~loaders.FromSingleFileMixin.from_single_file`] to load safetensor files. ```py import torch -from diffusers import FluxPipeline - -pipeline = FluxPipeline.from_pretrained( - "black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16 -).to("cuda") -pipeline.load_lora_weights("linoyts/yarn_art_Flux_LoRA") -pipeline.save_lora_weights( - transformer_lora_adapter_metadata={"r": 16, "lora_alpha": 16}, - text_encoder_lora_adapter_metadata={"r": 8, "lora_alpha": 8} +from diffusers import DiffusionPipeline + +pipeline = DiffusionPipeline.from_pretrained( + "stabilityai/stable-diffusion-xl-base-1.0", + torch.dtype=torch.float16, + device_map="cuda" +) + +pipeline = DiffusionPipeline.from_single_file( + "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors", + torch_dtype=torch.float16, + device_map="cuda" ) ``` ### ckpt -> [!WARNING] -> Pickled files may be unsafe because they can be exploited to execute malicious code. It is recommended to use safetensors files instead where possible, or convert the weights to safetensors files. +Older model weights are commonly saved with Python's [pickle](https://docs.python.org/3/library/pickle.html) utility in a ckpt file. -PyTorch's [torch.save](https://pytorch.org/docs/stable/generated/torch.save.html) function uses Python's [pickle](https://docs.python.org/3/library/pickle.html) utility to serialize and save models. These files are saved as a ckpt file and they contain the entire model's weights. +Pickled files may be unsafe because they can be exploited to execute malicious code. It is recommended to use safetensors files or convert the weights to safetensors files. -Use the [`~loaders.FromSingleFileMixin.from_single_file`] method to directly load a ckpt file. +Use [`~loaders.FromSingleFileMixin.from_single_file`] to load a ckpt file. ```py -from diffusers import StableDiffusionPipeline +from diffusers import DiffusionPipeline -pipeline = StableDiffusionPipeline.from_single_file( +pipeline = DiffusionPipeline.from_single_file( "https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5/blob/main/v1-5-pruned.ckpt" ) ``` -## Storage layout - -There are two ways model files are organized, either in a Diffusers-multifolder layout or in a single-file layout. The Diffusers-multifolder layout is the default, and each component file (text encoder, UNet, VAE) is stored in a separate subfolder. Diffusers also supports loading models from a single-file layout where all the components are bundled together. +### dduf -### Diffusers-multifolder +> [~TIP] +> DDUF is an experimental file type and the API may change. Refer to the DDUF [docs](https://huggingface.co/docs/hub/dduf) to learn more. -The Diffusers-multifolder layout is the default storage layout for Diffusers. Each component's (text encoder, UNet, VAE) weights are stored in a separate subfolder. The weights can be stored as safetensors or ckpt files. +DDUF is a file type designed to unify different diffusion model distribution methods and weight-saving formats. It is a standardized and flexible method to package all components of a diffusion model into a single file, providing a balance between the Diffusers and single-file formats. -
-
- -
multifolder layout
-
-
- -
UNet subfolder
-
-
- -To load from Diffusers-multifolder layout, use the [`~DiffusionPipeline.from_pretrained`] method. +Use the `dduf_file` argument in [`~DiffusionPipeline.from_pretrained`] to load a DDUF file. You can also load quantized dduf files as long as they are stored in the Diffusers format. ```py +import torch from diffusers import DiffusionPipeline pipeline = DiffusionPipeline.from_pretrained( - "stabilityai/stable-diffusion-xl-base-1.0", - torch_dtype=torch.float16, - variant="fp16", - use_safetensors=True, -).to("cuda") + "DDUF/FLUX.1-dev-DDUF", + dduf_file="FLUX.1-dev.dduf", + torch_dtype=torch.bfloat16, + device_map="cuda" +) ``` -Benefits of using the Diffusers-multifolder layout include: - -1. Faster to load each component file individually or in parallel. -2. Reduced memory usage because you only load the components you need. For example, models like [SDXL Turbo](https://hf.co/stabilityai/sdxl-turbo), [SDXL Lightning](https://hf.co/ByteDance/SDXL-Lightning), and [Hyper-SD](https://hf.co/ByteDance/Hyper-SD) have the same components except for the UNet. You can reuse their shared components with the [`~DiffusionPipeline.from_pipe`] method without consuming any additional memory (take a look at the [Reuse a pipeline](./loading#reuse-a-pipeline) guide) and only load the UNet. This way, you don't need to download redundant components and unnecessarily use more memory. - - ```py - import torch - from diffusers import StableDiffusionXLPipeline, UNet2DConditionModel, EulerDiscreteScheduler - - # download one model - sdxl_pipeline = StableDiffusionXLPipeline.from_pretrained( - "stabilityai/stable-diffusion-xl-base-1.0", - torch_dtype=torch.float16, - variant="fp16", - use_safetensors=True, - ).to("cuda") - - # switch UNet for another model - unet = UNet2DConditionModel.from_pretrained( - "stabilityai/sdxl-turbo", - subfolder="unet", - torch_dtype=torch.float16, - variant="fp16", - use_safetensors=True - ) - # reuse all the same components in new model except for the UNet - turbo_pipeline = StableDiffusionXLPipeline.from_pipe( - sdxl_pipeline, unet=unet, - ).to("cuda") - turbo_pipeline.scheduler = EulerDiscreteScheduler.from_config( - turbo_pipeline.scheduler.config, - timestep_spacing="trailing" - ) - image = turbo_pipeline( - "an astronaut riding a unicorn on mars", - num_inference_steps=1, - guidance_scale=0.0, - ).images[0] - image - ``` - -3. Reduced storage requirements because if a component, such as the SDXL [VAE](https://hf.co/madebyollin/sdxl-vae-fp16-fix), is shared across multiple models, you only need to download and store a single copy of it instead of downloading and storing it multiple times. For 10 SDXL models, this can save ~3.5GB of storage. The storage savings is even greater for newer models like PixArt Sigma, where the [text encoder](https://hf.co/PixArt-alpha/PixArt-Sigma-XL-2-1024-MS/tree/main/text_encoder) alone is ~19GB! -4. Flexibility to replace a component in the model with a newer or better version. - - ```py - from diffusers import DiffusionPipeline, AutoencoderKL - - vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16, use_safetensors=True) - pipeline = DiffusionPipeline.from_pretrained( - "stabilityai/stable-diffusion-xl-base-1.0", - vae=vae, - torch_dtype=torch.float16, - variant="fp16", - use_safetensors=True, - ).to("cuda") - ``` - -5. More visibility and information about a model's components, which are stored in a [config.json](https://hf.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/unet/config.json) file in each component subfolder. - -### Single-file - -The single-file layout stores all the model weights in a single file. All the model components (text encoder, UNet, VAE) weights are kept together instead of separately in subfolders. This can be a safetensors or ckpt file. - -
- -
- -To load from a single-file layout, use the [`~loaders.FromSingleFileMixin.from_single_file`] method. +To save a pipeline as a dduf file, use the [`~huggingface_hub.export_folder_as_dduf`] utility. ```py import torch -from diffusers import StableDiffusionXLPipeline - -pipeline = StableDiffusionXLPipeline.from_single_file( - "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors", - torch_dtype=torch.float16, - variant="fp16", - use_safetensors=True, -).to("cuda") -``` - -Benefits of using a single-file layout include: - -1. Easy compatibility with diffusion interfaces such as [ComfyUI](https://github.com/comfyanonymous/ComfyUI) or [Automatic1111](https://github.com/AUTOMATIC1111/stable-diffusion-webui) which commonly use a single-file layout. -2. Easier to manage (download and share) a single file. - -### DDUF - -> [!WARNING] -> DDUF is an experimental file format and APIs related to it can change in the future. - -DDUF (**D**DUF **D**iffusion **U**nified **F**ormat) is a file format designed to make storing, distributing, and using diffusion models much easier. Built on the ZIP file format, DDUF offers a standardized, efficient, and flexible way to package all parts of a diffusion model into a single, easy-to-manage file. It provides a balance between Diffusers multi-folder format and the widely popular single-file format. - -Learn more details about DDUF on the Hugging Face Hub [documentation](https://huggingface.co/docs/hub/dduf). - -Pass a checkpoint to the `dduf_file` parameter to load it in [`DiffusionPipeline`]. - -```py from diffusers import DiffusionPipeline -import torch - -pipe = DiffusionPipeline.from_pretrained( - "DDUF/FLUX.1-dev-DDUF", dduf_file="FLUX.1-dev.dduf", torch_dtype=torch.bfloat16 -).to("cuda") -image = pipe( - "photo a cat holding a sign that says Diffusers", num_inference_steps=50, guidance_scale=3.5 -).images[0] -image.save("cat.png") -``` - -To save a pipeline as a `.dduf` checkpoint, use the [`~huggingface_hub.export_folder_as_dduf`] utility, which takes care of all the necessary file-level validations. - -```py from huggingface_hub import export_folder_as_dduf -from diffusers import DiffusionPipeline -import torch -pipe = DiffusionPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16) +pipeline = DiffusionPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16) save_folder = "flux-dev" -pipe.save_pretrained("flux-dev") +pipeline.save_pretrained("flux-dev") export_folder_as_dduf("flux-dev.dduf", folder_path=save_folder) ``` -> [!TIP] -> Packaging and loading quantized checkpoints in the DDUF format is supported as long as they respect the multi-folder structure. - -## Convert layout and files - -Diffusers provides many scripts and methods to convert storage layouts and file formats to enable broader support across the diffusion ecosystem. +## Converting formats and files -Take a look at the [diffusers/scripts](https://github.com/huggingface/diffusers/tree/main/scripts) collection to find a script that fits your conversion needs. +Diffusers provides scripts and methods to convert format and files to enable broader support across the diffusion ecosystem. -> [!TIP] -> Scripts that have "`to_diffusers`" appended at the end mean they convert a model to the Diffusers-multifolder layout. Each script has their own specific set of arguments for configuring the conversion, so make sure you check what arguments are available! +Take a look at the [diffusers/scripts](https://github.com/huggingface/diffusers/tree/main/scripts) folder to find a conversion script. Scripts with `"to_diffusers` appended at the end converts a model to the Diffusers format. Each script has a specific set of arguments for configuring the conversion. Make sure you check what arguments are available. -For example, to convert a Stable Diffusion XL model stored in Diffusers-multifolder layout to a single-file layout, run the [convert_diffusers_to_original_sdxl.py](https://github.com/huggingface/diffusers/blob/main/scripts/convert_diffusers_to_original_sdxl.py) script. Provide the path to the model to convert, and the path to save the converted model to. You can optionally specify whether you want to save the model as a safetensors file and whether to save the model in half-precision. +The example below converts a model stored in Diffusers format to a single-file format. Provide the path to the model to convert and where to save the converted model. You can optionally specify what file type and data type to save the model as. ```bash python convert_diffusers_to_original_sdxl.py --model_path path/to/model/to/convert --checkpoint_path path/to/save/model/to --use_safetensors ``` -You can also save a model to Diffusers-multifolder layout with the [`~DiffusionPipeline.save_pretrained`] method. This creates a directory for you if it doesn't already exist, and it also saves the files as a safetensors file by default. +The [`~DiffusionPipeline.save_pretrained`] method also saves a model in Diffusers format and takes care of creating subfolders for each model. It saves the files as safetensor files by default. ```py -from diffusers import StableDiffusionXLPipeline +from diffusers import DiffusionPipeline -pipeline = StableDiffusionXLPipeline.from_single_file( +pipeline = DiffusionPipeline.from_single_file( "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors", ) pipeline.save_pretrained() ``` -Lastly, there are also Spaces, such as [SD To Diffusers](https://hf.co/spaces/diffusers/sd-to-diffusers) and [SD-XL To Diffusers](https://hf.co/spaces/diffusers/sdxl-to-diffusers), that provide a more user-friendly interface for converting models to Diffusers-multifolder layout. This is the easiest and most convenient option for converting layouts, and it'll open a PR on your model repository with the converted files. However, this option is not as reliable as running a script, and the Space may fail for more complicated models. - -## Single-file layout usage - -Now that you're familiar with the differences between the Diffusers-multifolder and single-file layout, this section shows you how to load models and pipeline components, customize configuration options for loading, and load local files with the [`~loaders.FromSingleFileMixin.from_single_file`] method. - -### Load a pipeline or model +Finally, you can use a Space like [SD To Diffusers](https://hf.co/spaces/diffusers/sd-to-diffusers) or [SD-XL To Diffusers](https://hf.co/spaces/diffusers/sdxl-to-diffusers) to convert models to the Diffusers format. It'll open a PR on your model repository with the converted files. This is the easiest way to convert a model, but it may fail for more complicated models. Using a conversion script is more reliable. -Pass the file path of the pipeline or model to the [`~loaders.FromSingleFileMixin.from_single_file`] method to load it. - - - - -```py -from diffusers import StableDiffusionXLPipeline - -ckpt_path = "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors" -pipeline = StableDiffusionXLPipeline.from_single_file(ckpt_path) -``` +## Resources - - +- Learn more about the design decisions and why safetensor files are preferred for saving and loading model weights in the [Safetensors audited as really safe and becoming the default](https://blog.eleuther.ai/safetensors-security-audit/) blog post. -```py -from diffusers import StableCascadeUNet - -ckpt_path = "https://huggingface.co/stabilityai/stable-cascade/blob/main/stage_b_lite.safetensors" -model = StableCascadeUNet.from_single_file(ckpt_path) -``` - - - - -Customize components in the pipeline by passing them directly to the [`~loaders.FromSingleFileMixin.from_single_file`] method. For example, you can use a different scheduler in a pipeline. - -```py -from diffusers import StableDiffusionXLPipeline, DDIMScheduler - -ckpt_path = "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors" -scheduler = DDIMScheduler() -pipeline = StableDiffusionXLPipeline.from_single_file(ckpt_path, scheduler=scheduler) -``` - -Or you could use a ControlNet model in the pipeline. - -```py -from diffusers import StableDiffusionControlNetPipeline, ControlNetModel - -ckpt_path = "https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5/blob/main/v1-5-pruned-emaonly.safetensors" -controlnet = ControlNetModel.from_pretrained("lllyasviel/control_v11p_sd15_canny") -pipeline = StableDiffusionControlNetPipeline.from_single_file(ckpt_path, controlnet=controlnet) -``` - -### Customize configuration options - -Models have a configuration file that define their attributes like the number of inputs in a UNet. Pipelines configuration options are available in the pipeline's class. For example, if you look at the [`StableDiffusionXLInstructPix2PixPipeline`] class, there is an option to scale the image latents with the `is_cosxl_edit` parameter. - -These configuration files can be found in the models Hub repository or another location from which the configuration file originated (for example, a GitHub repository or locally on your device). - - - - -> [!TIP] -> The [`~loaders.FromSingleFileMixin.from_single_file`] method automatically maps the checkpoint to the appropriate model repository, but there are cases where it is useful to use the `config` parameter. For example, if the model components in the checkpoint are different from the original checkpoint or if a checkpoint doesn't have the necessary metadata to correctly determine the configuration to use for the pipeline. - -The [`~loaders.FromSingleFileMixin.from_single_file`] method automatically determines the configuration to use from the configuration file in the model repository. You could also explicitly specify the configuration to use by providing the repository id to the `config` parameter. - -```py -from diffusers import StableDiffusionXLPipeline - -ckpt_path = "https://huggingface.co/segmind/SSD-1B/blob/main/SSD-1B.safetensors" -repo_id = "segmind/SSD-1B" - -pipeline = StableDiffusionXLPipeline.from_single_file(ckpt_path, config=repo_id) -``` - -The model loads the configuration file for the [UNet](https://huggingface.co/segmind/SSD-1B/blob/main/unet/config.json), [VAE](https://huggingface.co/segmind/SSD-1B/blob/main/vae/config.json), and [text encoder](https://huggingface.co/segmind/SSD-1B/blob/main/text_encoder/config.json) from their respective subfolders in the repository. - - - - -The [`~loaders.FromSingleFileMixin.from_single_file`] method can also load the original configuration file of a pipeline that is stored elsewhere. Pass a local path or URL of the original configuration file to the `original_config` parameter. - -```py -from diffusers import StableDiffusionXLPipeline - -ckpt_path = "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors" -original_config = "https://raw.githubusercontent.com/Stability-AI/generative-models/main/configs/inference/sd_xl_base.yaml" - -pipeline = StableDiffusionXLPipeline.from_single_file(ckpt_path, original_config=original_config) -``` - -> [!TIP] -> Diffusers attempts to infer the pipeline components based on the type signatures of the pipeline class when you use `original_config` with `local_files_only=True`, instead of fetching the configuration files from the model repository on the Hub. This prevents backward breaking changes in code that can't connect to the internet to fetch the necessary configuration files. -> -> This is not as reliable as providing a path to a local model repository with the `config` parameter, and might lead to errors during pipeline configuration. To avoid errors, run the pipeline with `local_files_only=False` once to download the appropriate pipeline configuration files to the local cache. - - - - -While the configuration files specify the pipeline or models default parameters, you can override them by providing the parameters directly to the [`~loaders.FromSingleFileMixin.from_single_file`] method. Any parameter supported by the model or pipeline class can be configured in this way. - - - - -For example, to scale the image latents in [`StableDiffusionXLInstructPix2PixPipeline`] pass the `is_cosxl_edit` parameter. - -```python -from diffusers import StableDiffusionXLInstructPix2PixPipeline - -ckpt_path = "https://huggingface.co/stabilityai/cosxl/blob/main/cosxl_edit.safetensors" -pipeline = StableDiffusionXLInstructPix2PixPipeline.from_single_file(ckpt_path, config="diffusers/sdxl-instructpix2pix-768", is_cosxl_edit=True) -``` - - - - -For example, to upcast the attention dimensions in a [`UNet2DConditionModel`] pass the `upcast_attention` parameter. - -```python -from diffusers import UNet2DConditionModel - -ckpt_path = "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors" -model = UNet2DConditionModel.from_single_file(ckpt_path, upcast_attention=True) -``` - - - - -### Local files - -In Diffusers>=v0.28.0, the [`~loaders.FromSingleFileMixin.from_single_file`] method attempts to configure a pipeline or model by inferring the model type from the keys in the checkpoint file. The inferred model type is used to determine the appropriate model repository on the Hugging Face Hub to configure the model or pipeline. - -For example, any single file checkpoint based on the Stable Diffusion XL base model will use the [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) model repository to configure the pipeline. - -But if you're working in an environment with restricted internet access, you should download the configuration files with the [`~huggingface_hub.snapshot_download`] function, and the model checkpoint with the [`~huggingface_hub.hf_hub_download`] function. By default, these files are downloaded to the Hugging Face Hub [cache directory](https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache), but you can specify a preferred directory to download the files to with the `local_dir` parameter. - -Pass the configuration and checkpoint paths to the [`~loaders.FromSingleFileMixin.from_single_file`] method to load locally. - - - - -```python -from huggingface_hub import hf_hub_download, snapshot_download - -my_local_checkpoint_path = hf_hub_download( - repo_id="segmind/SSD-1B", - filename="SSD-1B.safetensors" -) - -my_local_config_path = snapshot_download( - repo_id="segmind/SSD-1B", - allow_patterns=["*.json", "**/*.json", "*.txt", "**/*.txt"] -) - -pipeline = StableDiffusionXLPipeline.from_single_file(my_local_checkpoint_path, config=my_local_config_path, local_files_only=True) -``` - - - - -```python -from huggingface_hub import hf_hub_download, snapshot_download - -my_local_checkpoint_path = hf_hub_download( - repo_id="segmind/SSD-1B", - filename="SSD-1B.safetensors" - local_dir="my_local_checkpoints" -) - -my_local_config_path = snapshot_download( - repo_id="segmind/SSD-1B", - allow_patterns=["*.json", "**/*.json", "*.txt", "**/*.txt"] - local_dir="my_local_config" -) - -pipeline = StableDiffusionXLPipeline.from_single_file(my_local_checkpoint_path, config=my_local_config_path, local_files_only=True) -``` - - - - -#### Local files without symlink - -> [!TIP] -> In huggingface_hub>=v0.23.0, the `local_dir_use_symlinks` argument isn't necessary for the [`~huggingface_hub.hf_hub_download`] and [`~huggingface_hub.snapshot_download`] functions. - -The [`~loaders.FromSingleFileMixin.from_single_file`] method relies on the [huggingface_hub](https://hf.co/docs/huggingface_hub/index) caching mechanism to fetch and store checkpoints and configuration files for models and pipelines. If you're working with a file system that does not support symlinking, you should download the checkpoint file to a local directory first, and disable symlinking with the `local_dir_use_symlink=False` parameter in the [`~huggingface_hub.hf_hub_download`] function and [`~huggingface_hub.snapshot_download`] functions. - -```python -from huggingface_hub import hf_hub_download, snapshot_download - -my_local_checkpoint_path = hf_hub_download( - repo_id="segmind/SSD-1B", - filename="SSD-1B.safetensors" - local_dir="my_local_checkpoints", - local_dir_use_symlinks=False -) -print("My local checkpoint: ", my_local_checkpoint_path) - -my_local_config_path = snapshot_download( - repo_id="segmind/SSD-1B", - allow_patterns=["*.json", "**/*.json", "*.txt", "**/*.txt"] - local_dir_use_symlinks=False, -) -print("My local config: ", my_local_config_path) -``` - -Then you can pass the local paths to the `pretrained_model_link_or_path` and `config` parameters. - -```python -pipeline = StableDiffusionXLPipeline.from_single_file(my_local_checkpoint_path, config=my_local_config_path, local_files_only=True) -``` From e580f327f4def28fd1f21ac874f745b366af1516 Mon Sep 17 00:00:00 2001 From: stevhliu Date: Tue, 2 Sep 2025 15:31:12 -0700 Subject: [PATCH 2/8] config --- docs/source/en/using-diffusers/loading.md | 3 + .../en/using-diffusers/other-formats.md | 108 +++++++++++++++++- 2 files changed, 110 insertions(+), 1 deletion(-) diff --git a/docs/source/en/using-diffusers/loading.md b/docs/source/en/using-diffusers/loading.md index 25b53d2f4d49..3fb608b1c26c 100644 --- a/docs/source/en/using-diffusers/loading.md +++ b/docs/source/en/using-diffusers/loading.md @@ -52,6 +52,9 @@ pipeline = QwenImagePipeline.from_pretrained( ) ``` +> [!TIP] +> Refer to the [Single file format](./other-formats#single-file-format) docs to learn how to load single file models. + ### Local pipelines Pipelines can also be run locally. Use [`~huggingface_hub.snapshot_download`] to download a model repository. diff --git a/docs/source/en/using-diffusers/other-formats.md b/docs/source/en/using-diffusers/other-formats.md index 2d47cdaa6d32..52cf4931ee54 100644 --- a/docs/source/en/using-diffusers/other-formats.md +++ b/docs/source/en/using-diffusers/other-formats.md @@ -67,6 +67,112 @@ pipeline = DiffusionPipeline.from_single_file( ### Configuration options +Models have a `config.json` file in their repositories with important attributes such as the number of layers and attention heads. The [`~loaders.FromSingleFileMixin.from_single_file`] method automatically determines the appropriate config to use from `config.json`. + +But if the models in a pipeline are different from the original implementation or if it doesn't have to necessary metadata to determine the correct config, then you need to use the `config` argument. + +```py +from diffusers import QwenImagePipeline + +ckpt_path = "https://huggingface.co/lightx2v/Qwen-Image-Lightning/blob/main/Qwen-Image-Lightning-8steps-V1.1-bf16.safetensors" + +pipeline = QwenImagePipeline.from_single_file( + ckpt_path, + config="lightx2v/Qwen-Image-Lightning" +) +``` + +You could also load a config file not stored on the Hub by passing a local path or URL of the config file to the `original_config` argument. + +```py +from diffusers import WanPipeline + +ckpt_path = "https://huggingface.co/lightx2v/Qwen-Image-Lightning/blob/main/Qwen-Image-Lightning-8steps-V1.1-bf16.safetensors" +original_config = "https://raw.githubusercontent.com/Wan-Video/Wan2.2/refs/heads/main/wan/configs/wan_ti2v_5B.py" + +pipeline = WanPipeline.from_single_file( + ckpt_path, + original_config=original_config +) +``` + +Diffusers attempts to infer the pipeline components based on the signature types of the pipeline class when using `original_config` with `local_files_only=True`. It won't download the config files from a Hub repository to avoid backward breaking changes when you can't connect to the internet. This method isn't as reliable as providing a path to a local model with the `config` argument and may lead to errors. You should run the pipeline with `local_files_only=False` to download the config files to the local cache to avoid errors. + +Override default configs by passing the arguments directly to [`~loaders.FromSingleFileMixin.from_single_file`]. The examples below demonstrate how to override the configs in a pipeline or model. + +```py +from diffusers import StableDiffusionXLInstructPix2PixPipeline + +ckpt_path = "https://huggingface.co/stabilityai/cosxl/blob/main/cosxl_edit.safetensors" +pipeline = StableDiffusionXLInstructPix2PixPipeline.from_single_file( + ckpt_path, config="diffusers/sdxl-instructpix2pix-768", is_cosxl_edit=True +) +``` + +```py +from diffusers import UNet2DConditionModel + +ckpt_path = "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors" +model = UNet2DConditionModel.from_single_file(ckpt_path, upcast_attention=True) +``` + +### Local files + +The [`~loaders.FromSingleFileMixin.from_single_file`] method attempts to configure a pipeline or model by inferring the model type from the keys in the checkpoint file. For example, any single file checkpoint based on the Stable Diffusion XL base model is configured from [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0). + +If you're working with local files, download the config files with the [`~huggingface_hub.snapshot_download`] method and the model checkpoint with [`~huggingface_hub.hf_hub_download`]. These files are downloaded to your [cache directory](https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache), but you can download them to a specific directory with the `local_dir` argument. + +```py +from huggingface_hub import hf_hub_download, snapshot_download +from diffusers import QwenImagePipeline + +my_local_checkpoint_path = hf_hub_download( + repo_id="lightx2v/Qwen-Image-Lightning", + filename="Qwen-Image-Lightning-8steps-V1.1-bf16.safetensors" +) + +my_local_config_path = snapshot_download( + repo_id="lightx2v/Qwen-Image-Lightning", + allow_patterns=["*.json", "**/*.json", "*.txt", "**/*.txt"] +) + +pipeline = QwenImagePipeline.from_single_file( + my_local_checkpoint_path, config=my_local_config_path, local_files_only=True +) +``` + +### Symlink + +If you're working with a file system that does not support symlinking, download the checkpoint file to a local directory first. Disable symlinking with `local_dir_use_symlink=False` in [`~huggingface_hub.snapshot_download`] and [`~huggingface_hub.hf_hub_download`]. + +```py +from huggingface_hub import hf_hub_download, snapshot_download + +my_local_checkpoint_path = hf_hub_download( + repo_id="lightx2v/Qwen-Image-Lightning", + filename="Qwen-Image-Lightning-8steps-V1.1-bf16.safetensors", + local_dir="my_local_checkpoints", + local_dir_use_symlinks=False, +) +print("My local checkpoint: ", my_local_checkpoint_path) + +my_local_config_path = snapshot_download( + repo_id="lightx2v/Qwen-Image-Lightning", + allow_patterns=["*.json", "**/*.json", "*.txt", "**/*.txt"], + local_dir_use_symlinks=False, +) +print("My local config: ", my_local_config_path) +``` + +Pass these paths to [`~loaders.FromSingleFileMixin.from_single_file`]. + +```py +from diffusers import QwenImagePipeline + +pipeline = QwenImagePipeline.from_single_file( + my_local_checkpoint_path, config=my_local_config_path, local_files_only=True +) +``` ## File types @@ -115,7 +221,7 @@ pipeline = DiffusionPipeline.from_single_file( ### dduf -> [~TIP] +> [!TIP] > DDUF is an experimental file type and the API may change. Refer to the DDUF [docs](https://huggingface.co/docs/hub/dduf) to learn more. DDUF is a file type designed to unify different diffusion model distribution methods and weight-saving formats. It is a standardized and flexible method to package all components of a diffusion model into a single file, providing a balance between the Diffusers and single-file formats. From 7ac5bb5bca1d05533a04b72e046f00eb8023293b Mon Sep 17 00:00:00 2001 From: stevhliu Date: Tue, 2 Sep 2025 15:52:14 -0700 Subject: [PATCH 3/8] lora metadata --- .../source/en/using-diffusers/other-formats.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/source/en/using-diffusers/other-formats.md b/docs/source/en/using-diffusers/other-formats.md index 52cf4931ee54..e0008022ff3a 100644 --- a/docs/source/en/using-diffusers/other-formats.md +++ b/docs/source/en/using-diffusers/other-formats.md @@ -203,6 +203,24 @@ pipeline = DiffusionPipeline.from_single_file( ) ``` +If you're using a checkpoint trained with a Diffusers training script, metadata such as the LoRA configuration, is automatically saved. When the file is loaded, the metadata is parsed to correctly configure the LoRA and avoid missing or incorrect LoRA configs. Inspect the metadata of a safetensors file by clicking on the safetensors logo logo next to the file on the Hub. + +Save the metadata for LoRAs that aren't trained with Diffusers with the `transformer_lora_adapter_metadata` and `text_encoder_lora_adapter_metadata` arguments in [`~loaders.FluxLoraLoaderMixin.save_lora_weights`]. This is only supported for safetensors files. + +```py +import torch +from diffusers import FluxPipeline + +pipeline = FluxPipeline.from_pretrained( + "black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16 +).to("cuda") +pipeline.load_lora_weights("linoyts/yarn_art_Flux_LoRA") +pipeline.save_lora_weights( + transformer_lora_adapter_metadata={"r": 16, "lora_alpha": 16}, + text_encoder_lora_adapter_metadata={"r": 8, "lora_alpha": 8} +) +``` + ### ckpt Older model weights are commonly saved with Python's [pickle](https://docs.python.org/3/library/pickle.html) utility in a ckpt file. From 826a71f1338b46aa14f6c98ee38278c4600372de Mon Sep 17 00:00:00 2001 From: stevhliu Date: Wed, 3 Sep 2025 12:33:43 -0700 Subject: [PATCH 4/8] feedback --- .../en/using-diffusers/other-formats.md | 82 +++++++++---------- src/diffusers/__init__.py | 14 ++++ 2 files changed, 51 insertions(+), 45 deletions(-) diff --git a/docs/source/en/using-diffusers/other-formats.md b/docs/source/en/using-diffusers/other-formats.md index e0008022ff3a..8c719c07ae99 100644 --- a/docs/source/en/using-diffusers/other-formats.md +++ b/docs/source/en/using-diffusers/other-formats.md @@ -17,7 +17,7 @@ specific language governing permissions and limitations under the License. Diffusion models are typically stored in the Diffusers format or single-file format. Model files can be stored in various file types such as safetensors, dduf, or ckpt. > [!TIP] -> Format refers to the directory structure and file refers to the file type. +> Format refers to whether the weights are stored in a directory structure and file refers to the file type. This guide will show you how to load pipelines and models from these formats and files. @@ -41,12 +41,11 @@ Use [`~loaders.FromSingleFileMixin.from_single_file`] to load a single file. ```py import torch -from diffusers import DiffusionPipeline +from diffusers import StableDiffusionXLPipeline -pipeline = DiffusionPipeline.from_single_file( +pipeline = StableDiffusionXLPipeline.from_single_file( "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors", torch_dtype=torch.float16, - device_map="cuda" ) ``` @@ -54,46 +53,41 @@ The [`~loaders.FromSingleFileMixin.from_single_file`] method also supports passi ```py import torch -from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler +from diffusers import FluxPipeline, FluxTransformer2DModel -scheduler = DPMSolverMultistepScheduler() -pipeline = DiffusionPipeline.from_single_file( - "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors", - scheduler=scheduler, - torch_dtype=torch.float16, - device_map="cuda" +transformer = FluxTransformer2DModel.from_single_file( + "https://huggingface.co/Kijai/flux-fp8/blob/main/flux1-dev-fp8.safetensors", torch_dtype=torch.bfloat16 +) +pipeline = FluxPipeline.from_single_file( + "black-forest-labs/FLUX.1-dev", + transformer=transformer, + torch_dtype=torch.bfloat16, ) ``` ### Configuration options -Models have a `config.json` file in their repositories with important attributes such as the number of layers and attention heads. The [`~loaders.FromSingleFileMixin.from_single_file`] method automatically determines the appropriate config to use from `config.json`. +Diffusers format models have a `config.json` file in their repositories with important attributes such as the number of layers and attention heads. The [`~loaders.FromSingleFileMixin.from_single_file`] method automatically determines the appropriate config to use from `config.json`. This may fail in a few rare instances though, in which case, you should use the `config` argument. -But if the models in a pipeline are different from the original implementation or if it doesn't have to necessary metadata to determine the correct config, then you need to use the `config` argument. +You should also use the `config` argument if the models in a pipeline are different from the original implementation or if it doesn't have the necessary metadata to determine the correct config. ```py -from diffusers import QwenImagePipeline +from diffusers import StableDiffusionXLPipeline -ckpt_path = "https://huggingface.co/lightx2v/Qwen-Image-Lightning/blob/main/Qwen-Image-Lightning-8steps-V1.1-bf16.safetensors" +ckpt_path = "https://huggingface.co/segmind/SSD-1B/blob/main/SSD-1B.safetensors" -pipeline = QwenImagePipeline.from_single_file( - ckpt_path, - config="lightx2v/Qwen-Image-Lightning" -) +pipeline = StableDiffusionXLPipeline.from_single_file(ckpt_path, config="segmind/SSD-1B") ``` You could also load a config file not stored on the Hub by passing a local path or URL of the config file to the `original_config` argument. ```py -from diffusers import WanPipeline +from diffusers import StableDiffusionXLPipeline -ckpt_path = "https://huggingface.co/lightx2v/Qwen-Image-Lightning/blob/main/Qwen-Image-Lightning-8steps-V1.1-bf16.safetensors" -original_config = "https://raw.githubusercontent.com/Wan-Video/Wan2.2/refs/heads/main/wan/configs/wan_ti2v_5B.py" +ckpt_path = "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors" +original_config = "https://raw.githubusercontent.com/Stability-AI/generative-models/main/configs/inference/sd_xl_base.yaml" -pipeline = WanPipeline.from_single_file( - ckpt_path, - original_config=original_config -) +pipeline = StableDiffusionXLPipeline.from_single_file(ckpt_path, original_config=original_config) ``` Diffusers attempts to infer the pipeline components based on the signature types of the pipeline class when using `original_config` with `local_files_only=True`. It won't download the config files from a Hub repository to avoid backward breaking changes when you can't connect to the internet. This method isn't as reliable as providing a path to a local model with the `config` argument and may lead to errors. You should run the pipeline with `local_files_only=False` to download the config files to the local cache to avoid errors. @@ -124,19 +118,19 @@ If you're working with local files, download the config files with the [`~huggin ```py from huggingface_hub import hf_hub_download, snapshot_download -from diffusers import QwenImagePipeline +from diffusers import StableDiffusionXLPipeline my_local_checkpoint_path = hf_hub_download( - repo_id="lightx2v/Qwen-Image-Lightning", - filename="Qwen-Image-Lightning-8steps-V1.1-bf16.safetensors" + repo_id="segmind/SSD-1B", + filename="SSD-1B.safetensors" ) my_local_config_path = snapshot_download( - repo_id="lightx2v/Qwen-Image-Lightning", + repo_id="segmind/SSD-1B", allow_patterns=["*.json", "**/*.json", "*.txt", "**/*.txt"] ) -pipeline = QwenImagePipeline.from_single_file( +pipeline = StableDiffusionXLPipeline.from_single_file( my_local_checkpoint_path, config=my_local_config_path, local_files_only=True ) ``` @@ -147,18 +141,19 @@ If you're working with a file system that does not support symlinking, download ```py from huggingface_hub import hf_hub_download, snapshot_download +from diffusers import StableDiffusionXLPipeline my_local_checkpoint_path = hf_hub_download( - repo_id="lightx2v/Qwen-Image-Lightning", - filename="Qwen-Image-Lightning-8steps-V1.1-bf16.safetensors", + repo_id="segmind/SSD-1B", + filename="SSD-1B.safetensors" local_dir="my_local_checkpoints", - local_dir_use_symlinks=False, + local_dir_use_symlinks=False ) print("My local checkpoint: ", my_local_checkpoint_path) my_local_config_path = snapshot_download( - repo_id="lightx2v/Qwen-Image-Lightning", - allow_patterns=["*.json", "**/*.json", "*.txt", "**/*.txt"], + repo_id="segmind/SSD-1B", + allow_patterns=["*.json", "**/*.json", "*.txt", "**/*.txt"] local_dir_use_symlinks=False, ) print("My local config: ", my_local_config_path) @@ -167,9 +162,7 @@ print("My local config: ", my_local_config_path) Pass these paths to [`~loaders.FromSingleFileMixin.from_single_file`]. ```py -from diffusers import QwenImagePipeline - -pipeline = QwenImagePipeline.from_single_file( +pipeline = StableDiffusionXLPipeline.from_single_file( my_local_checkpoint_path, config=my_local_config_path, local_files_only=True ) ``` @@ -182,7 +175,7 @@ Models can be stored in several file types. Safetensors is the most common file [Safetensors](https://hf.co/docs/safetensors) is a safe and fast file type for securely storing and loading tensors. It restricts the header size to limit certain types of attacks, supports lazy loading (useful for distributed setups), and generally loads faster. -Diffusers loads safetensors file by default if they are available and the Safetensors library is installed. +Diffusers loads safetensors file by default (a required dependency) if they are available and the Safetensors library is installed. Use [`~DiffusionPipeline.from_pretrained`] or [`~loaders.FromSingleFileMixin.from_single_file`] to load safetensor files. @@ -199,13 +192,12 @@ pipeline = DiffusionPipeline.from_pretrained( pipeline = DiffusionPipeline.from_single_file( "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors", torch_dtype=torch.float16, - device_map="cuda" ) ``` -If you're using a checkpoint trained with a Diffusers training script, metadata such as the LoRA configuration, is automatically saved. When the file is loaded, the metadata is parsed to correctly configure the LoRA and avoid missing or incorrect LoRA configs. Inspect the metadata of a safetensors file by clicking on the safetensors logo logo next to the file on the Hub. +If you're using a checkpoint trained with a Diffusers training script, metadata such as the LoRA configuration, is automatically saved. When the file is loaded, the metadata is parsed to correctly configure the LoRA and avoid missing or incorrect LoRA configs. Inspect the metadata of a safetensors file by clicking on the safetensors logo logo next to the file on the Hub. -Save the metadata for LoRAs that aren't trained with Diffusers with the `transformer_lora_adapter_metadata` and `text_encoder_lora_adapter_metadata` arguments in [`~loaders.FluxLoraLoaderMixin.save_lora_weights`]. This is only supported for safetensors files. +Save the metadata for LoRAs that aren't trained with Diffusers with either `transformer_lora_adapter_metadata` or `unet_lora_adapter_metadata` depending on your model. For the text encoder, use the `text_encoder_lora_adapter_metadata` and `text_encoder_2_lora_adapter_metadata` arguments in [`~loaders.FluxLoraLoaderMixin.save_lora_weights`]. This is only supported for safetensors files. ```py import torch @@ -216,8 +208,8 @@ pipeline = FluxPipeline.from_pretrained( ).to("cuda") pipeline.load_lora_weights("linoyts/yarn_art_Flux_LoRA") pipeline.save_lora_weights( - transformer_lora_adapter_metadata={"r": 16, "lora_alpha": 16}, - text_encoder_lora_adapter_metadata={"r": 8, "lora_alpha": 8} + text_encoder_lora_adapter_metadata={"r": 8, "lora_alpha": 8}, + text_encoder_2_lora_adapter_metadata={"r": 8, "lora_alpha": 8} ) ``` diff --git a/src/diffusers/__init__.py b/src/diffusers/__init__.py index 741fcd14f283..f41fb4fbd971 100644 --- a/src/diffusers/__init__.py +++ b/src/diffusers/__init__.py @@ -343,6 +343,13 @@ ] ) _import_structure["training_utils"] = ["EMAModel"] + _import_structure["image_processor"] = [ + "VaeImageProcessor", + "VaeImageProcessorLDM3D", + "PixArtImageProcessor", + "IPAdapterMaskProcessor", + ] + _import_structure["video_processor"] = ["VideoProcessor"] try: if not (is_torch_available() and is_scipy_available()): @@ -1019,6 +1026,13 @@ VQDiffusionScheduler, ) from .training_utils import EMAModel + from .image_processor import ( + VaeImageProcessor, + VaeImageProcessorLDM3D, + PixArtImageProcessor, + IPAdapterMaskProcessor, + ) + from .video_processor import VideoProcessor try: if not (is_torch_available() and is_scipy_available()): From ba45c253281d858eba463d24088dd151f925f45c Mon Sep 17 00:00:00 2001 From: stevhliu Date: Wed, 3 Sep 2025 12:45:51 -0700 Subject: [PATCH 5/8] fix --- src/diffusers/__init__.py | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/src/diffusers/__init__.py b/src/diffusers/__init__.py index f41fb4fbd971..741fcd14f283 100644 --- a/src/diffusers/__init__.py +++ b/src/diffusers/__init__.py @@ -343,13 +343,6 @@ ] ) _import_structure["training_utils"] = ["EMAModel"] - _import_structure["image_processor"] = [ - "VaeImageProcessor", - "VaeImageProcessorLDM3D", - "PixArtImageProcessor", - "IPAdapterMaskProcessor", - ] - _import_structure["video_processor"] = ["VideoProcessor"] try: if not (is_torch_available() and is_scipy_available()): @@ -1026,13 +1019,6 @@ VQDiffusionScheduler, ) from .training_utils import EMAModel - from .image_processor import ( - VaeImageProcessor, - VaeImageProcessorLDM3D, - PixArtImageProcessor, - IPAdapterMaskProcessor, - ) - from .video_processor import VideoProcessor try: if not (is_torch_available() and is_scipy_available()): From 4434750d49b0730348d203f5ecc252b70a3e2379 Mon Sep 17 00:00:00 2001 From: stevhliu Date: Tue, 9 Sep 2025 08:34:49 -0700 Subject: [PATCH 6/8] cache allocator warmup for from_single_file --- docs/source/en/using-diffusers/other-formats.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/source/en/using-diffusers/other-formats.md b/docs/source/en/using-diffusers/other-formats.md index 8c719c07ae99..37364257390d 100644 --- a/docs/source/en/using-diffusers/other-formats.md +++ b/docs/source/en/using-diffusers/other-formats.md @@ -37,7 +37,7 @@ A single-file format stores *all* the model (UNet, transformer, text encoder) we - Greater compatibility with [ComfyUI](https://github.com/comfyanonymous/ComfyUI) or [Automatic1111](https://github.com/AUTOMATIC1111/stable-diffusion-webui). - Easier to download and share a single file. -Use [`~loaders.FromSingleFileMixin.from_single_file`] to load a single file. +Use [`~loaders.FromSingleFileMixin.from_single_file`] to load a single file. Pass `"cuda"` to the `device_map` argument to pre-allocate GPU memory and reduce model loading time (refer to the [parallel loading](../using-diffusers/loading#parallel-loading) docs for more details). ```py import torch @@ -46,6 +46,7 @@ from diffusers import StableDiffusionXLPipeline pipeline = StableDiffusionXLPipeline.from_single_file( "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors", torch_dtype=torch.float16, + device_map="cuda" ) ``` @@ -62,6 +63,7 @@ pipeline = FluxPipeline.from_single_file( "black-forest-labs/FLUX.1-dev", transformer=transformer, torch_dtype=torch.bfloat16, + device_map="cuda" ) ``` From 27e92c5baddf82df37d44f34b43228808e6270b6 Mon Sep 17 00:00:00 2001 From: stevhliu Date: Mon, 22 Sep 2025 12:47:11 -0700 Subject: [PATCH 7/8] feedback --- docs/source/en/using-diffusers/other-formats.md | 17 ++--------------- 1 file changed, 2 insertions(+), 15 deletions(-) diff --git a/docs/source/en/using-diffusers/other-formats.md b/docs/source/en/using-diffusers/other-formats.md index 37364257390d..a5f396b3a011 100644 --- a/docs/source/en/using-diffusers/other-formats.md +++ b/docs/source/en/using-diffusers/other-formats.md @@ -59,7 +59,7 @@ from diffusers import FluxPipeline, FluxTransformer2DModel transformer = FluxTransformer2DModel.from_single_file( "https://huggingface.co/Kijai/flux-fp8/blob/main/flux1-dev-fp8.safetensors", torch_dtype=torch.bfloat16 ) -pipeline = FluxPipeline.from_single_file( +pipeline = FluxPipeline.from_pretrained( "black-forest-labs/FLUX.1-dev", transformer=transformer, torch_dtype=torch.bfloat16, @@ -81,17 +81,6 @@ ckpt_path = "https://huggingface.co/segmind/SSD-1B/blob/main/SSD-1B.safetensors" pipeline = StableDiffusionXLPipeline.from_single_file(ckpt_path, config="segmind/SSD-1B") ``` -You could also load a config file not stored on the Hub by passing a local path or URL of the config file to the `original_config` argument. - -```py -from diffusers import StableDiffusionXLPipeline - -ckpt_path = "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors" -original_config = "https://raw.githubusercontent.com/Stability-AI/generative-models/main/configs/inference/sd_xl_base.yaml" - -pipeline = StableDiffusionXLPipeline.from_single_file(ckpt_path, original_config=original_config) -``` - Diffusers attempts to infer the pipeline components based on the signature types of the pipeline class when using `original_config` with `local_files_only=True`. It won't download the config files from a Hub repository to avoid backward breaking changes when you can't connect to the internet. This method isn't as reliable as providing a path to a local model with the `config` argument and may lead to errors. You should run the pipeline with `local_files_only=False` to download the config files to the local cache to avoid errors. Override default configs by passing the arguments directly to [`~loaders.FromSingleFileMixin.from_single_file`]. The examples below demonstrate how to override the configs in a pipeline or model. @@ -139,7 +128,7 @@ pipeline = StableDiffusionXLPipeline.from_single_file( ### Symlink -If you're working with a file system that does not support symlinking, download the checkpoint file to a local directory first. Disable symlinking with `local_dir_use_symlink=False` in [`~huggingface_hub.snapshot_download`] and [`~huggingface_hub.hf_hub_download`]. +If you're working with a file system that does not support symlinking, download the checkpoint file to a local directory first with the `local_dir` parameter. Using the `local_dir` parameter automatically disables symlinks. ```py from huggingface_hub import hf_hub_download, snapshot_download @@ -149,14 +138,12 @@ my_local_checkpoint_path = hf_hub_download( repo_id="segmind/SSD-1B", filename="SSD-1B.safetensors" local_dir="my_local_checkpoints", - local_dir_use_symlinks=False ) print("My local checkpoint: ", my_local_checkpoint_path) my_local_config_path = snapshot_download( repo_id="segmind/SSD-1B", allow_patterns=["*.json", "**/*.json", "*.txt", "**/*.txt"] - local_dir_use_symlinks=False, ) print("My local config: ", my_local_config_path) ``` From efc17aba9f653ab78cd641c2cc97928442d7312f Mon Sep 17 00:00:00 2001 From: stevhliu Date: Wed, 24 Sep 2025 10:54:35 -0700 Subject: [PATCH 8/8] feedback --- docs/source/en/using-diffusers/other-formats.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/en/using-diffusers/other-formats.md b/docs/source/en/using-diffusers/other-formats.md index a5f396b3a011..b6e333ed7715 100644 --- a/docs/source/en/using-diffusers/other-formats.md +++ b/docs/source/en/using-diffusers/other-formats.md @@ -37,7 +37,7 @@ A single-file format stores *all* the model (UNet, transformer, text encoder) we - Greater compatibility with [ComfyUI](https://github.com/comfyanonymous/ComfyUI) or [Automatic1111](https://github.com/AUTOMATIC1111/stable-diffusion-webui). - Easier to download and share a single file. -Use [`~loaders.FromSingleFileMixin.from_single_file`] to load a single file. Pass `"cuda"` to the `device_map` argument to pre-allocate GPU memory and reduce model loading time (refer to the [parallel loading](../using-diffusers/loading#parallel-loading) docs for more details). +Use [`~loaders.FromSingleFileMixin.from_single_file`] to load a single file. ```py import torch