Prompt adherence for FluxPipeline is broken

[dxqb]

Describe the bug

For prompts much shorter than the max_sequence_length (which is most prompts, because default 512), prompts are not followed because the attention calculation spends most of the attention to the padding tokens of the encoder hidden state.

Prompt: Portrait photo of an angry man
First picture: Flux pipeline with defaults
Second: No attention to padding; here, using the workaround of setting a short max_sequence_length. A generic implementation would use attention masking (and not always use the max_sequence_length but only pad up to the longest prompt in the batch)

Other model pipelines might be affected similarly.

This has been discussed here #10194 and other issues in more detail, but I decided to create a separate issue to bring to your attention that this also affects default use of the pipeline, not only finetuning.

Reproduction

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16
).to('cuda')

prompt = "Portrait photo of an angry man"
prompt_T5_token_length = 7

generator = torch.Generator(device='cuda').manual_seed(1)
images = pipe(
    prompt=prompt,
    guidance_scale=3.5,
    height=1024,
    width=1024,
    num_inference_steps=20,
    generator=generator,
).images

images[0].save(f"attention-to-padding.jpg")

generator = torch.Generator(device='cuda').manual_seed(1)
images = pipe(
    prompt=prompt,
    guidance_scale=3.5,
    height=1024,
    width=1024,
    num_inference_steps=20,
    generator=generator,
    max_sequence_length = prompt_T5_token_length,
).images

images[0].save(f"no-attention-to-padding.jpg")

Logs

System Info

certifi==2025.4.26
charset-normalizer==3.4.2
diffusers==0.33.1
filelock==3.18.0
fsspec==2025.3.2
hf-xet==1.1.0
huggingface-hub==0.31.1
idna==3.10
importlib_metadata==8.7.0
Jinja2==3.1.6
MarkupSafe==3.0.2
mpmath==1.3.0
networkx==3.4.2
numpy==2.2.5
nvidia-cublas-cu12==12.6.4.1
nvidia-cuda-cupti-cu12==12.6.80
nvidia-cuda-nvrtc-cu12==12.6.77
nvidia-cuda-runtime-cu12==12.6.77
nvidia-cudnn-cu12==9.5.1.17
nvidia-cufft-cu12==11.3.0.4
nvidia-cufile-cu12==1.11.1.6
nvidia-curand-cu12==10.3.7.77
nvidia-cusolver-cu12==11.7.1.2
nvidia-cusparse-cu12==12.5.4.2
nvidia-cusparselt-cu12==0.6.3
nvidia-nccl-cu12==2.26.2
nvidia-nvjitlink-cu12==12.6.85
nvidia-nvtx-cu12==12.6.77
packaging==25.0
pillow==11.2.1
protobuf==6.30.2
PyYAML==6.0.2
regex==2024.11.6
requests==2.32.3
safetensors==0.5.3
sentencepiece==0.2.0
sympy==1.14.0
tokenizers==0.21.1
torch==2.7.0
tqdm==4.67.1
transformers==4.51.3
triton==3.3.0
typing_extensions==4.13.2
urllib3==2.4.0
zipp==3.21.0

Who can help?

@DN6 @yiyixuxu @sayakpaul

[sayakpaul]

(and not always use the max_sequence_length but only pad up to the longest prompt in the batch)

This is doable but the example you provided fails to convince me if no attention masking (second) is necessarily following the prompt better.

[dxqb]

(and not always use the max_sequence_length but only pad up to the longest prompt in the batch)
This is doable

This alone only works with batch size 1

but the example you provided fails to convince me if no attention masking (second)

the second is with attention masking (effectively, by reducing the max_sequence_len)

is necessarily following the prompt better.

you can try other prompts using the code I've provided. Just make sure that prompt_T5_token_length matches the length of your prompt to use this workaround.

You can of course debate how strongly a model should follow the prompt, but then let me rephrase the issue:
Why should the "max_sequence_len" parameter to the pipe change what is generated?

[asomoza]

we must be careful when using sensationalist titles like "Prompt adherence for FluxPipeline is broken)", it doesn't affects us that much because we follow the original implementation, but people can wrongly assume stuff with this kind of titles.

Fist, you're using 20 steps instead of 28 which is the recommended.

Also did you use an empirical way of testing your claim? Are you completely sure that prompt adherence is broken? Using that prompt is not a real prompt adherence test, the "angry" parts is really subjective, both can be angry but in a different degree.

It only took me couple of tries to do the same you did but backwards, meaning that I get the not so angry person with the 7 token limit and the really angry person with the 512 one:

With 28 steps and a seed of 93192706:

7 token liimit	512 token limit

Also why does it change when you change the max_sequence_length, as you said, because of the padding since they're different embeddings, but I don't see that it breaks the adherence, it just slightly changes the image.

When you use the exact tokens you're not using padding which changes a lot the image, but AFAIK the model wasn't trained like this, they used padding and long prompts, that's why Flux mostly really works decent with those weird long prompts and it's really bad with short concise prompts.

512 token limit	256 token limit

I'm not saying that you're wrong but if you're going to do such a big and grandiose claim, you should at least make some proper testing and show irrefutable proof of what you're saying.

P.S.: Made some edits to make it more easy to understand and fixed some grammar errors I made.

[dxqb]

Looked into this further in the meantime and looks like I am wrong. The original inference code uses max length tokens and does no attention masking. SwarmUI (with Comfy as backend) does appear to do something similar.

I chose a clear title because there has been a stale issue since last year to support attention masking, and my point was that it also affect inference, not only finetuning - which it does. But apparently it was intended this way by the designers of the model.

Yes I have many samples that do show that attention masking can make the prompt stronger, and I have samples that show that attention masking can improve finetuning.
But there is a separate issue for this already and inference works as intended, so I am going to close this issue.

Thank you for looking into this!

[asomoza]

No problem, I also thank you for looking into this, I wasn't trying to disregard your claim, just to show us some irrefutable findings with a better prompt that's not subjective.

We always use the original implementation approach unless we made some mistake (it can happen), so I really appreciate it when people do this, but the title can really make people think that the diffusers implementation is broken.