The Attention Class is not working for only_cross_attention = True

[JayL0321]

diffusers/src/diffusers/models/attention_processor.py

Line 128 in 1a5797c

self.to_k = None

self.to_k and self.to_v will be none if the attention object is initialized with only_cross_attention = True

[patrickvonplaten]

Hey @JayL0321,

Could you add a reproducible code snippet that shows your failure case?

[JayL0321]

I tried the following simple test:

query_dim, cross_attention_dim, added_kv_proj_dim, only_cross_attention = 4, 8, 16, True
attn1 = Attention(query_dim, cross_attention_dim, added_kv_proj_dim=added_kv_proj_dim, only_cross_attention=only_cross_attention)

batch_size, q_len, k_len = 2, 6, 12
hidden_states = torch.randn(batch_size, q_len, query_dim)
encoder_hidden_states = torch.randn(batch_size, k_len, added_kv_proj_dim)
attn1(hidden_states, encoder_hidden_states)

It had the following error:

---> 26 key = attn.to_k(encoder_hidden_states)
     27 value = attn.to_v(encoder_hidden_states)
     29 head_dim = inner_dim [/](https://file+.vscode-resource.vscode-cdn.net/)[/](https://file+.vscode-resource.vscode-cdn.net/) attn.heads

TypeError: 'NoneType' object is not callable

[patrickvonplaten]

Hey @JayL0321,

If you pass only_cross_attention=True, you have to also pass added_kv_proj_dim to a value that is defined (not None). Otherwise there are no k v projection matrices.

[JayL0321]

Hi @patrickvonplaten , in the example I posted, I have defined the added_kw_proj_dim value.

[patrickvonplaten]

Ah yeah the correct processor needs to be passed as well:

from diffusers.models.attention_processor import Attention, AttnAddedKVProcessor2_0
import torch

query_dim, cross_attention_dim, added_kv_proj_dim, only_cross_attention = 4, 8, 16, True
attn1 = Attention(query_dim, cross_attention_dim, added_kv_proj_dim=added_kv_proj_dim, only_cross_attention=only_cross_attention, norm_num_groups=1)
attn1.set_processor(AttnAddedKVProcessor2_0())

batch_size, q_len, k_len = 2, 6, 12
hidden_states = torch.randn(batch_size, q_len, query_dim)
hidden_states = torch.randn(batch_size, q_len, query_dim)
encoder_hidden_states = torch.randn(batch_size, k_len, added_kv_proj_dim)
attn1(hidden_states, encoder_hidden_states)

The dimensions don't match for the forward, but this code works if the dims are correct

[sunvember]

pip install diffusers==0.29.2