[Modular] Updates for Custom Pipeline Blocks #11940
Conversation
| None, | ||
| None, |
There was a problem hiding this comment.
@yiyixuxu Here for serializing PipelineBlocks, we deliberately set the library, class tuple to None. But type_hint also provides this information. Do we need to have both of these in the config then? Will modular configs ever have the class library tuple be non null values?
There was a problem hiding this comment.
I don't think we need to set them to None, they should just be the actual library/class
but, yes that's a good question, let's try to find a solution here:
we do not need these two fields technically, I left it there because that's I want to have an intuitive way to be able to tell if a components are loaded or not, in DiffusionPipeline these tuples will be null values if not loaded so I did same with modular.
maybe we can keep these 2 fields (or something else that can indicate whehter they are loaded or not) in config dict ( which gets print out when you do print(modular_pipeline) ), but remove them when saving to the modular_model_index.json?
let me know what you think!
There was a problem hiding this comment.
ohh actually, do you include the components info in the config.json for pipeline blocks too?
I was thinking maybe we don't need this, because pipeline blocks are just definations, so it does not need to include the loading related info - these are already saved in the pipeline, https://huggingface.co/YiYiXu/modular-diffdiff-0704
There was a problem hiding this comment.
updates ModularPipelineBlock serialization so that type hint, repo, subfolder, variant information gets saved when calling block.save_pretrained. Useful for when you have custom pipeline blocks that have components that use from_pretrained
so not sure if I understand here, I think the block's config.json is only used to create block, the auto_map points to the code defination is enough becausee you can define the default loading spec inside the custom code; user may want to change the loading spec at run time but they need to do that with the ModuarPipeline's config, not the blocks, no?
There was a problem hiding this comment.
Hmm so what about custom pipeline blocks that are meant to work with a particular model type? e.g Trying to create a custom block here to work with the Florence2-large model.
https://huggingface.co/diffusers-internal-dev/florence2-image-annotator/blob/main/config.json
How should one structure this? The goal is to insert a step that automatically creates a mask for an image
for inpainting based on a text prompt.
import torch
from diffusers.modular_pipelines import ModularPipelineBlocks, SequentialPipelineBlocks
from diffusers.modular_pipelines.stable_diffusion_xl import INPAINT_BLOCKS
from diffusers.utils import load_image
# fetch the Florence2 image annotator block that will create our mask
image_annotator_block = ModularPipelineBlocks.from_pretrained("diffusers-internal-dev/florence2-image-annotator", trust_remote_code=True)
my_blocks = INPAINT_BLOCKS.copy()
# insert the annotation block before the image encoding step
my_blocks.insert("image_annotator", image_annotator_block, 1)
# Create our initial set of inpainting blocks
blocks = SequentialPipelineBlocks.from_blocks_dict(my_blocks)
pipe = blocks.init_pipeline()If the component spec information like repo/subfolder for the model is not present in the config, the init_pipeline call won't work?
There was a problem hiding this comment.
so I see you already defined here https://huggingface.co/diffusers-internal-dev/florence2-image-annotator/blob/main/block.py#L17
so once you convert the blocks into pipeline, it will show up in pipeline's config like this
>>> image_annotator_block.init_pipeline()
ModularPipeline {
"_blocks_class_name": "Florence2ImageAnnotatorBlock",
"_class_name": "ModularPipeline",
"_diffusers_version": "0.35.0.dev0",
"image_annotator": [
null,
null,
{
"repo": "microsoft/Florence-2-large",
"revision": null,
"subfolder": "",
"type_hint": [
"transformers",
"AutoModelForCausalLM"
],
"variant": null
}
],
"image_annotator_processor": [
null,
null,
{
"repo": "microsoft/Florence-2-large",
"revision": null,
"subfolder": "",
"type_hint": [
"transformers",
"AutoProcessor"
],
"variant": null
}
]
}and if you run save_pretrained() on the pipeline, you will get the modular_model_index.json contains the loading specs for the cusom model
pipe = image_annotator_block.init_pipeline()
pipe.save_pretraind(...)e.g. this is what I got by just adding pipe.save_pretrained("YiYiXU/test_image_annotator", push_to_hub=True) to the code example you shared https://huggingface.co/YiYiXu/test_image_annotator
ideally, we also push the custom code in the same repo, so everything can be together with ModularPipeline.from_pretrained() (the custom blocks + loading configs -> you got a fully initialized custom pipeline)
and since everything is in the same folder, the author of the custom pipeline also have the option to fill out the repo/subfolder info in modular_model_index.json if they like (instead of defining it in the code)
There was a problem hiding this comment.
Ah I see it is expected to load from the custom code itself. Okay, so I made changes to ModularPipelineBlocks to allow running init_pipeline lmk if those look okay.
Re: Having all components + custom code in the same repo. Assuming we load two custom blocks from two different repos, each has a block.py file defining the custom code. If we serialize the Pipeline with code into one repo, how do we figure out where to put all the components? Also how do we handle components with custom code in modular_model_index.json?
There was a problem hiding this comment.
Re: (None, None, <config details>) format in modular_model_index.json and this
maybe we can keep these 2 fields (or something else that can indicate whehter they are loaded or not) in config dict ( which gets print out when you do print(modular_pipeline) ), but remove them when saving to the modular_model_index.json?
I think this could work. I can put together a quick PR to just see how it feels.
There was a problem hiding this comment.
Re: Having all components + custom code in the same repo. Assuming we load two custom blocks from two different repos, each has a block.py file defining the custom code. If we serialize the Pipeline with code into one repo, how do we figure out where to put all the components? Also how do we handle components with custom code in modular_model_index.json?
For the blocks author, they can define the default component loading specs (repo/subfolder etc) when they write the pipeline blocks code, in the expected_components field. It's optional to include the repo info, but if they did that, like you did here with the florence-image-annotator block https://huggingface.co/diffusers-internal-dev/florence2-image-annotator/blob/main/block.py#L17 - these info info will stick with the block wherever it goes forever. lol
so in your question, if you load two different custom blocks from two different repos and mix them together into one pipeline, the loading specs for these custom components from both blocks should show up as default in your final pipeline. If you save the final pipeline into a repo, it will serialize the pipeline config dict into a modular_model_index.json in that repo, which will contain the default loading spec for the custom components, but the person who put together the final pipeline can edit the modular_model_index.json if they want to use something else. The end user could use a different loading specs at run time as well
how do we handle components with custom code in modular_model_index.json?
not sure I understand the question correctly, but ModularPipeline.from_pretrained() will look for custom code first, so if you save the code in the same repo, it should work
the code is here:
https://github.com/huggingface/diffusers/blame/main/src/diffusers/modular_pipelines/modular_pipeline.py#L2053
if your pipeline uses two different blocks from two different repos, I imagine your repo also need a block.py, and in that block.py you import these custom code, so I think it would also work lol but have not tested with such cases so let me know
There was a problem hiding this comment.
and if you only want to host the block in your repo, (not the entire pipeline and do not want to save a modular_model_index.json) - defining the custom componenet in the code is enough,
I think your current code alreaedy work out of box very well: if the user just want to import that custom block, mix into their existing inpanting pipeline, they can pass the inpaiting repo in init_pipeline() - the final pipeline will contain the loading info for the custom component (since we defined a default) and all the loading info for other components (fetchd from the modular_model_index.json in that inpainting repo we passed)
e.g.
your code, only difference is I passed a "YiYiXu/modular-demo-auto" to init_pipeline() - "YiYiXu/modular-demo-auto" does not contain the custom component but we have a default defined so pipeline still have that info
import torch
from diffusers.modular_pipelines import ModularPipelineBlocks, SequentialPipelineBlocks
from diffusers.modular_pipelines.stable_diffusion_xl import INPAINT_BLOCKS
from diffusers.utils import load_image
# fetch the Florence2 image annotator block that will create our mask
from diffusers.modular_pipelines import PipelineBlock
image_annotator_block = ModularPipelineBlocks.from_pretrained("diffusers-internal-dev/florence2-image-annotator", trust_remote_code=True)
my_blocks = INPAINT_BLOCKS.copy()
# insert the annotation block before the image encoding step
my_blocks.insert("image_annotator", image_annotator_block, 1)
# Create our initial set of inpainting blocks
blocks = SequentialPipelineBlocks.from_blocks_dict(my_blocks)
pipe = blocks.init_pipeline("YiYiXu/modular-demo-auto")
print(pipe)the pipeline is ready to use and knows how to load all the components
StableDiffusionXLModularPipeline {
"_blocks_class_name": "SequentialPipelineBlocks",
"_class_name": "StableDiffusionXLModularPipeline",
"_diffusers_version": "0.35.0.dev0",
"force_zeros_for_empty_prompt": true,
"image_annotator": [
null,
null,
{
"repo": "microsoft/Florence-2-large",
"revision": null,
"subfolder": "",
"type_hint": [
"transformers",
"AutoModelForCausalLM"
],
"variant": null
}
],
"image_annotator_processor": [
null,
null,
{
"repo": "microsoft/Florence-2-large",
"revision": null,
"subfolder": "",
"type_hint": [
"transformers",
"AutoProcessor"
],
"variant": null
}
],
"requires_aesthetics_score": false,
"scheduler": [
null,
null,
{
"repo": "SG161222/RealVisXL_V4.0",
"revision": null,
"subfolder": "scheduler",
"type_hint": [
"diffusers",
"EulerDiscreteScheduler"
],
"variant": null
}
],
"text_encoder": [
null,
null,
{
"repo": "SG161222/RealVisXL_V4.0",
"revision": null,
"subfolder": "text_encoder",
"type_hint": [
"transformers",
"CLIPTextModel"
],
"variant": null
}
],
"text_encoder_2": [
null,
null,
{
"repo": "SG161222/RealVisXL_V4.0",
"revision": null,
"subfolder": "text_encoder_2",
"type_hint": [
"transformers",
"CLIPTextModelWithProjection"
],
"variant": null
}
],
"tokenizer": [
null,
null,
{
"repo": "SG161222/RealVisXL_V4.0",
"revision": null,
"subfolder": "tokenizer",
"type_hint": [
"transformers",
"CLIPTokenizer"
],
"variant": null
}
],
"tokenizer_2": [
null,
null,
{
"repo": "SG161222/RealVisXL_V4.0",
"revision": null,
"subfolder": "tokenizer_2",
"type_hint": [
"transformers",
"CLIPTokenizer"
],
"variant": null
}
],
"unet": [
null,
null,
{
"repo": "SG161222/RealVisXL_V4.0",
"revision": null,
"subfolder": "unet",
"type_hint": [
"diffusers",
"UNet2DConditionModel"
],
"variant": null
}
],
"vae": [
null,
null,
{
"repo": "SG161222/RealVisXL_V4.0",
"revision": null,
"subfolder": "vae",
"type_hint": [
"diffusers",
"AutoencoderKL"
],
"variant": null
}
]
}
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
|
||
|
|
||
| EXPECTED_PARENT_CLASSES = ["ModularPipelineBlocks"] | ||
| EXPECTED_PARENT_CLASSES = ["PipelineBlock"] |
There was a problem hiding this comment.
why it's only PipelineBlock?
There was a problem hiding this comment.
users should be able to define custom ModularPipelineBlocks too where they assmeble their own presets so that it is easier to use
in diff-diff remote code example, https://huggingface.co/YiYiXu/modular-diffdiff-0704/blob/main/config.json
There was a problem hiding this comment.
maybe support all block types?
There was a problem hiding this comment.
Ah. Yeah that make sense will update.
What does this PR do?
Updates ModularPipelineBlock serialization so that type hint, repo, subfolder, variant information gets saved when calling
block.save_pretrained. Useful for when you have custom pipeline blocks that have components that usefrom_pretrainedPass
trust_remote_codetohub_kwargsso that it gets passed through to model loading. Useful when loading custom pipeline blocks with models that also have custom code. e.g. Florence2 in transformersMakes subfolder default value an empty string in ComponentSpec because passing
subfolder=Noneto transformer Auto classes does not work. We also substituteNonefor an empty string when runningfrom_pretrainedfor diffusers models. So it should be a safe change.Updates the custom block repo creation CLI command to search for classes inheriting from
PipelineBlock. Not sure here if custom pipeline blocks need to inherit fromModularPipelineBlocksorPipelineBlock. Can revert if needed, but from my understanding it should be PipelineBlock? cc: @yiyixuxuFixes # (issue)
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.