[Modular] implement requirements validation for custom blocks

[sayakpaul]

What does this PR do?

As discussed with Dhruv.

When implementing custom blocks (which we all believe will grow big :)), it can be useful for the block author and the users to have some kind of dependencies and their versions specified. This PR implements that feature.

Sample output: https://huggingface.co/diffusers-internal-dev/gemini-prompt-expander/blob/main/modular_config.json.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

@DN6 a gentle ping :)

[sayakpaul]

Gentle ping @DN6

[yiyixuxu]

ummm, should requirements be defined at the pipeline level instead of the block level?
Blocks are meant to be reusable components that can be assembled in different ways. While individual blocks can define default loading behavior (which may have different dependency rquirement), these should be overridable at the pipeline level when blocks are assembled together in modular pipeline

the person who put together the pipeline should be responsible and have the final say over the final components, e.g requirements and modular_model_index.json etc, no?

[sayakpaul]

I think implementing it at both the block and pipeline levels could be nice. What I am thinking:

• If a pipeline doesn't define any custom requirements, then we could check if any of its blocks define them and perform validation.
• If a pipeline defines it, then the validation logic, at least, still remains the same.

WDYT about this approach? 👀

[yiyixuxu]

for blocks, maybe define requirements within the block definition similar to inputs/outputs?

class GeminiPromptExpander:
    _requirements = {
        "google-generativeai": ">=0.8.0",
        "pillow": ">=10.0.0"
    }

class FluxTransformer:
    _requirements = {
        "transformers": ">=4.44.0",
        "sentencepiece": ">=0.2.0"
    }

When we combine them:

pipe = SequentialPipelineBlocks.from_blocks_dict({
    "prompt_expander": GeminiPromptExpander, 
    "transformer": FluxTransformer
})

print(pipe._requirements)
# Output:
# {
#     "prompt_expander": {"google-generativeai": ">=0.8.0", "pillow": ">=10.0.0"},
#     "transformer": {"transformers": ">=4.44.0", "sentencepiece": ">=0.2.0"}
# }

This should be just meta data thought, it just helps generate the final requirements.txt, but the pipeline author still need to manually write the requirement file and test it

[sayakpaul]

@yiyixuxu yes that makes sense. I will apply those changes in the PR.