IndexError: index 1 is out of bounds for dimension 0 with size 1

[raj7AIfrac]

While runing the Qwen edit (plus) model with test_example_qwen_image_edit_plus_fp16.yaml on multi GPU.
I am getting this error.

[rank7]: Traceback (most recent call last): [rank7]: File "<frozen runpy>", line 198, in _run_module_as_main [rank7]: File "<frozen runpy>", line 88, in _run_code [rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/main.py", line 99, in <module> [rank7]: main() [rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/main.py", line 95, in main [rank7]: trainer.fit(train_dataloader) [rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/trainer/base_trainer.py", line 648, in fit [rank7]: self.train_epoch(epoch, train_dataloader) [rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/trainer/base_trainer.py", line 522, in train_epoch [rank7]: loss = self.training_step(batch) [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^ [rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/trainer/base_trainer.py", line 461, in training_step [rank7]: return self._training_step_compute(batch) [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/trainer/base_trainer.py", line 470, in _training_step_compute [rank7]: embeddings = self.prepare_embeddings(batch, stage="fit") [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/trainer/qwen_image_edit_plus_trainer.py", line 188, in prepare_embeddings [rank7]: prompt_embeds, prompt_embeds_mask = self.encode_prompt(prompt=batch["prompt"], image=condition_images) [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/trainer/qwen_image_edit_plus_trainer.py", line 312, in encode_prompt [rank7]: prompt_embeds, prompt_embeds_mask = self._get_qwen_prompt_embeds(prompt, image, device) [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/trainer/qwen_image_edit_plus_trainer.py", line 346, in _get_qwen_prompt_embeds [rank7]: model_inputs = self.processor( [rank7]: ^^^^^^^^^^^^^^^ [rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/qwen-image-finetune/venv1/lib/python3.12/site-packages/transformers/models/qwen2_vl/processing_qwen2_vl.py", line 159, in __call__ [rank7]: num_image_tokens = image_grid_thw[index].prod() // merge_length [rank7]: ~~~~~~~~~~~~~~^^^^^^^ [rank7]: IndexError: index 1 is out of bounds for dimension 0 with size 1``

[raj7AIfrac]

[rank7]: Traceback (most recent call last):
[rank7]: File "", line 198, in _run_module_as_main
[rank7]: File "", line 88, in _run_code
[rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/main.py", line 99, in
[rank7]: main()
[rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/main.py", line 95, in main
[rank7]: trainer.fit(train_dataloader)
[rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/trainer/base_trainer.py", line 648, in fit
[rank7]: self.train_epoch(epoch, train_dataloader)
[rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/trainer/base_trainer.py", line 522, in train_epoch
[rank7]: loss = self.training_step(batch)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/trainer/base_trainer.py", line 461, in training_step
[rank7]: return self._training_step_compute(batch)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/trainer/base_trainer.py", line 470, in _training_step_compute
[rank7]: embeddings = self.prepare_embeddings(batch, stage="fit")
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/trainer/qwen_image_edit_plus_trainer.py", line 188, in prepare_embeddings
[rank7]: prompt_embeds, prompt_embeds_mask = self.encode_prompt(prompt=batch["prompt"], image=condition_images)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/trainer/qwen_image_edit_plus_trainer.py", line 312, in encode_prompt
[rank7]: prompt_embeds, prompt_embeds_mask = self._get_qwen_prompt_embeds(prompt, image, device)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/Features/qwen-image-finetune/src/qflux/trainer/qwen_image_edit_plus_trainer.py", line 346, in _get_qwen_prompt_embeds
[rank7]: model_inputs = self.processor(
[rank7]: ^^^^^^^^^^^^^^^
[rank7]: File "/weka/innmivfractalslurmh1/home/raj.yadav/Work/Task3/qwen-image-finetune/venv1/lib/python3.12/site-packages/transformers/models/qwen2_vl/processing_qwen2_vl.py", line 159, in call
[rank7]: num_image_tokens = image_grid_thw[index].prod() // merge_length
[rank7]: ~~~~~~~~~~~~~~^^^^^^^
[rank7]: IndexError: index 1 is out of bounds for dimension 0 with size 1

[Wang-Shengyuan]

Same issue.

[chenmeiya]

same +1
I solve the problem by comment out the code on line 146 in file qwen-image-finetune/src/qflux/trainer/qwen_image_edit_plus_trainer.py, and instead of "condition_images = control_image"

[XinleiNIU]

I got the same issue for multi image control with batch size >1. The issue is caused by line 146 and line 164, it removes the batch dim. The correct format should be a list of list List[List[control images]]. For single control, swap 'condition_images.append(self.process_condition_image(control_image))' to 'condition_images = control_image.' For multiple controls, replace line 164 'condition_images.append(self.process_condition_image(additional_control_image))' to 'condition_images = [[img1, img2] for img1, img2 in zip(control_image, additional_control_image)]'. You also need to modify the self._get_qwen_prompt_embeds function as well.

Inside the _get_qwen_prompt_embeds function:

`
if isinstance(image, list):
base_img_prompt = ""
## UPDATE: fix the batch size incorrect issue
# If the image is list of list.
if isinstance(image[0], list):
num_controls = len(image[0])
elif len(image) == 0:
num_controls = 0
else:
num_controls = len(image)
## UPDATE: the image should be a list of list, where len(list) = Batch, len(list[0])=num controls, enumerate the list[0] instead
# for i, _ in enumerateTensor:
for i in range(num_controls):
base_img_prompt += img_prompt_template.format(i + 1)
elif image is not None:
base_img_prompt = img_prompt_template.format(1)
else:
base_img_prompt = ""

    template = self.prompt_template_encode

    drop_idx = self.prompt_template_encode_start_idx
    txt = [template.format(base_img_prompt + e) for e in prompt]

    # 1) left padding for batched Flash-Attn
    tok = self.processor.tokenizer
    tok.padding_side = "left"
    if tok.pad_token_id is None:
        tok.pad_token = tok.eos_token
    # 2) processor call (txt: List[str], image: List[List[img]] for 2-image-per-sample)
    model_inputs = self.processor(text=txt, images=image, padding=True, return_tensors="pt").to(device)
    
    # 3) qwen vl forward
    outputs = self.text_encoder(
        input_ids=model_inputs.input_ids,
        attention_mask=model_inputs.attention_mask,
        pixel_values=model_inputs.pixel_values,
        image_grid_thw=model_inputs.image_grid_thw,
        output_hidden_states=True,
        
    )`

This modification works to me.