[Single File] Add single file support for Flux Canny, Depth and Fill

[DN6]

What does this PR do?

Fixes # (issue)
#10282

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[hlky]

Thanks @DN6! FLUX.1-Canny-dev uses the same config as FLUX.1-Depth-dev so it's also covered.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[chuck-ma]

Thanks for your quick response. Now I can create a transformer from gguf using your branch. However, it throws errors when I load loras.

adapter_id = "alimama-creative/FLUX.1-Turbo-Alpha"
pipe.transformer = self.get_transformer_dev()

pipe.load_lora_weights(
                 adapter_id,
                 adapter_name="turbo_alpha_fill",
             )

It gots errors:

File [~/autodl-tmp/diffusers/src/diffusers/loaders/lora_pipeline.py:1856](https://a155156-a316-5f561747.westc.gpuhub.com:8443/jupyter/lab/tree/autodl-tmp/~/autodl-tmp/diffusers/src/diffusers/loaders/lora_pipeline.py#line=1855), in FluxLoraLoaderMixin.load_lora_weights(self, pretrained_model_name_or_path_or_dict, adapter_name, **kwargs)
   1849 transformer_norm_state_dict = {
   1850     k: state_dict.pop(k)
   1851     for k in list(state_dict.keys())
   1852     if "transformer." in k and any(norm_key in k for norm_key in self._control_lora_supported_norm_keys)
   1853 }
   1855 transformer = getattr(self, self.transformer_name) if not hasattr(self, "transformer") else self.transformer
-> 1856 has_param_with_expanded_shape = self._maybe_expand_transformer_param_shape_or_error_(
   1857     transformer, transformer_lora_state_dict, transformer_norm_state_dict
   1858 )
   1860 if has_param_with_expanded_shape:
   1861     logger.info(
   1862         "The LoRA weights contain parameters that have different shapes that expected by the transformer. "
   1863         "As a result, the state_dict of the transformer has been expanded to match the LoRA parameter shapes. "
   1864         "To get a comprehensive list of parameter names that were modified, enable debug logging."
   1865     )

File [~/autodl-tmp/diffusers/src/diffusers/loaders/lora_pipeline.py:2357](https://a155156-a316-5f561747.westc.gpuhub.com:8443/jupyter/lab/tree/autodl-tmp/~/autodl-tmp/diffusers/src/diffusers/loaders/lora_pipeline.py#line=2356), in FluxLoraLoaderMixin._maybe_expand_transformer_param_shape_or_error_(cls, transformer, lora_state_dict, norm_state_dict, prefix)
   2354 parent_module = transformer.get_submodule(parent_module_name)
   2356 with torch.device("meta"):
-> 2357     expanded_module = torch.nn.Linear(
   2358         in_features, out_features, bias=bias, dtype=module_weight.dtype
   2359     )
   2360 # Only weights are expanded and biases are not.
   2361 new_weight = torch.zeros_like(
   2362     expanded_module.weight.data, device=module_weight.device, dtype=module_weight.dtype
   2363 )

File [~/miniconda3/lib/python3.12/site-packages/torch/nn/modules/linear.py:105](https://a155156-a316-5f561747.westc.gpuhub.com:8443/jupyter/lab/tree/autodl-tmp/~/miniconda3/lib/python3.12/site-packages/torch/nn/modules/linear.py#line=104), in Linear.__init__(self, in_features, out_features, bias, device, dtype)
    103 self.in_features = in_features
    104 self.out_features = out_features
--> 105 self.weight = Parameter(
    106     torch.empty((out_features, in_features), **factory_kwargs)
    107 )
    108 if bias:
    109     self.bias = Parameter(torch.empty(out_features, **factory_kwargs))

File [~/miniconda3/lib/python3.12/site-packages/torch/nn/parameter.py:46](https://a155156-a316-5f561747.westc.gpuhub.com:8443/jupyter/lab/tree/autodl-tmp/~/miniconda3/lib/python3.12/site-packages/torch/nn/parameter.py#line=45), in Parameter.__new__(cls, data, requires_grad)
     42     data = torch.empty(0)
     43 if type(data) is torch.Tensor or type(data) is Parameter:
     44     # For ease of BC maintenance, keep this path for standard Tensor.
     45     # Eventually (tm), we should change the behavior for standard Tensor to match.
---> 46     return torch.Tensor._make_subclass(cls, data, requires_grad)
     48 # Path for custom tensors: set a flag on the instance to indicate parameter-ness.
     49 t = data.detach().requires_grad_(requires_grad)

RuntimeError: Only Tensors of floating point and complex dtype can require gradients

Any idea?
@DN6 @hlky

[DN6]

@chuck-ma We still have to add LoRA support for quantized models.

[chuck-ma]

@chuck-ma We still have to add LoRA support for quantized models.

Understood