context is not really done but deleted

[OdedRotem314]

whenever context is passed to a block like in:
h, res_samples = downsample_block(hidden_states=h, temb=emb, context=context)

the forward function uses "del context"

so it is not really implementing this conditioning.

further more if the context is an image it should pass through an encoder before added/concat to the image embeddings.

[virginiafdez]

Dear Oded,

The forward function that uses "del context" is the DownsampleBlock one. The tensor "context" is only used with cross-attention conditioning within the CrossAttnDownsampleBlock, and is not needed for the DownsampleBlock one. Because all the DownsampleBlock types need to be compatible, context has to be passed to the forward, but it gets deleted cause it is not necessary.

When you want to use concatenation-based conditioning, often used to condition on images or derivatives, the concatenation - which happens at the channel level - the operation is done at the inferer level (or, if you do not use an inferer, manually).

The cross-attention used within this diffusion model is the simple one; thus, I do not think it is compatible with images unless you flatten them or encode them prior (but that is not supported within the DiffusionModelUNet function).

If you believe additional functionalities should be added to the generative code, please do open a feature request issue at MONAI Core.

MONAI Generative Models has been moved to MONAI Core, and no further modifications will be therefore done to this repository, as it will be soon archived.

Thanks a lot!

Virginia