[GLM-Image] Add batch > 1 support and fix configuration defaults

[JaredforReal]

What does this PR do?

This PR adds full batch processing support (batch_size > 1) for the GLM-Image model, fixes padding direction for autoregressive generation, and aligns configuration defaults with the official model.

Problem

1. Batch restriction: The GlmImageProcessor explicitly rejected batch_size > 1
2. Missing config defaults: pad_token_id and eos_token_id were not set in GlmImageTextConfig, and max_position_embeddings didn't match the official config.json

Solution

1. Batch support

Processor changes (processing_glm_image.py):

• Removed the batch size restriction
• Added images_per_sample tensor to track number of grids per sample
• Added num_source_images_per_sample tensor to distinguish source images from target grids

Model changes (modeling_glm_image.py via modular_glm_image.py):

• Updated get_rope_index() to compute position IDs per-sample with batch support
• Updated _cached_decode_position_ids shape from [3, max_len] to [batch, 3, max_len]
• Updated forward() to properly handle packed image_grid_thw by splitting per sample
• Updated _expand_inputs_for_generation() and prepare_inputs_for_generation() for beam search compatibility

2. Left padding for autoregressive generation

Processor changes (processing_glm_image.py):

• Added automatic left padding when padding=True is specified
• This ensures proper attention masking for autoregressive image token generation in batch mode

3. Configuration alignment with official model

Config changes (modular_glm_image.py → configuration_glm_image.py):

• Added pad_token_id=167841 to GlmImageTextConfig
• Added eos_token_id=16385 to GlmImageTextConfig
• Updated max_position_embeddings from 32768 to 131072 to match official config.json

Tests

All 64 modeling tests pass. Added test_batch_consistency to verify:

1. Batch processing produces identical predictions (argmax) as single processing
2. Identical inputs in a batch produce identical outputs

Breaking changes

None. All new parameters are optional and backward compatible.

Checklist

• This PR fixes a bug / adds a feature without changing the model behavior
• Tests pass locally
• Documentation updated (docstrings for new parameters)
• Code follows the project style guidelines

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[Copilot]

Pull request overview

This PR adds comprehensive batch processing support (batch_size > 1) for the GLM-Image model, enabling efficient parallel image generation. Previously, the processor explicitly rejected batch sizes greater than 1. The implementation introduces two new tracking tensors (images_per_sample and num_source_images_per_sample) to manage packed image grids across batch samples, updates the RoPE position encoding computation to work per-sample, and modifies the generation utilities to correctly expand inputs for beam search.

Changes:

• Removed batch size restriction in processor and added per-sample image tracking
• Updated position ID computation to handle batches with independent per-sample caching
• Modified beam search expansion logic to correctly handle packed visual inputs across batch samples

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
tests/models/glm_image/test_modeling_glm_image.py	Adds test_batch_consistency to verify batch and single processing produce identical predictions
src/transformers/models/glm_image/processing_glm_image.py	Removes batch size restriction, adds image counting per sample, and implements per-sample prompt/grid construction
src/transformers/models/glm_image/modular_glm_image.py	Updates get_rope_index() for per-sample position IDs, modifies forward() to split packed grids, updates generation utilities for batch support
src/transformers/models/glm_image/modeling_glm_image.py	Auto-generated from modular file with matching changes for batch support
src/transformers/models/glm_image/image_processing_glm_image_fast.py	Removes min_pixels/max_pixels class attributes and simplifies initialization
src/transformers/models/glm_image/configuration_glm_image.py	Reorders tie_word_embeddings parameter and removes it from super().init() call

[Copilot]

This line assumes all samples have the same number of target grids by using all_target_grids[0].shape[0]. If different samples could have different numbers of target grids (e.g., due to different is_text_to_image settings), this would cause incorrect counting. Consider validating that all samples have the same num_target_grids, or handle varying target grid counts per sample.

Suggested change

	num_target_grids = all_target_grids[0].shape[0]
	image_inputs["images_per_sample"] = torch.tensor(
	[n + num_target_grids for n in images_per_sample], dtype=torch.long
	target_grids_per_sample = [grids.shape[0] for grids in all_target_grids]
	image_inputs["images_per_sample"] = torch.tensor(
	[n_source + n_target for n_source, n_target in zip(images_per_sample, target_grids_per_sample)],
	dtype=torch.long,

[Copilot]

The variable all_final_positions is populated in the loop but never used. Consider removing this unused variable to improve code clarity.

Suggested change

all_final_positions.append(current_pos)

[Copilot]

This line assumes all samples have the same number of target grids by using all_target_grids[0].shape[0]. If different samples could have different numbers of target grids (e.g., due to different is_text_to_image settings), this would cause incorrect counting. Consider validating that all samples have the same num_target_grids, or handle varying target grid counts per sample.

Suggested change

	num_target_grids = all_target_grids[0].shape[0]
	image_inputs["images_per_sample"] = torch.tensor(
	[n + num_target_grids for n in images_per_sample], dtype=torch.long
	)
	num_target_grids_per_sample = [g.shape[0] for g in all_target_grids]
	if len(set(num_target_grids_per_sample)) == 1:
	num_target_grids = num_target_grids_per_sample[0]
	images_per_sample_with_targets = [n + num_target_grids for n in images_per_sample]
	else:
	images_per_sample_with_targets = [n + t for n, t in zip(images_per_sample, num_target_grids_per_sample)]
	image_inputs["images_per_sample"] = torch.tensor(images_per_sample_with_targets, dtype=torch.long)

[Copilot]

When splitting pixel_values for beam search expansion, if sum(source_image_nums) == 0 (no source images), the pixel_values tensor is not handled. While the code checks if sum > 0 before processing, it doesn't explicitly handle the else case where pixel_values should remain unchanged or be set appropriately. Consider adding an explicit else clause or handling for the case where there are no source images.

Suggested change

	)
	)
	else:
	# No source images: leave pixel_values unchanged
	dict_to_expand[key] = dict_to_expand[key]

[Copilot]

The super().init() call no longer passes tie_word_embeddings. This means the parent class PreTrainedConfig won't receive this parameter. Verify that the parent class correctly handles tie_word_embeddings through **kwargs, or explicitly pass it if needed.

Suggested change

	super().__init__(ignore_keys_at_rope_validation={"mrope_section"}, **kwargs)
	super().__init__(
	tie_word_embeddings=tie_word_embeddings,
	ignore_keys_at_rope_validation={"mrope_section"},
	**kwargs,
	)

[Copilot]

The variable all_final_positions is populated in the loop but never used. Consider removing this unused variable to improve code clarity.

Suggested change

all_final_positions.append(current_pos)

[Copilot]

There's a potential issue with varying batch sizes during decode. If the batch size differs between prefill and decode stages (e.g., due to early stopping), the code attempts to index self._prefill_lens and self._cached_decode_position_ids with batch indices that may be out of bounds. Consider adding a check to ensure batch_size matches the cached dimensions.

Suggested change

	# Per-sample decode position lookup
	# _cached_decode_position_ids shape: [batch_size, 3, max_decode_len]
	# _prefill_lens shape: [batch_size]
	position_ids_list = []
	for batch_idx in range(batch_size):
	prefill_len = self._prefill_lens[batch_idx].item()
	step = cache_position[0].item() - prefill_len
	# Get position ids for this sample
	sample_pos_ids = self._cached_decode_position_ids[batch_idx, :, step : step + seq_length]
	position_ids_list.append(sample_pos_ids)
	# Stack and transpose to [3, batch_size, seq_length]
	position_ids = torch.stack(position_ids_list, dim=1)
	# Only use cached decode positions if they are compatible with the current batch size.
	use_cached_positions = (
	hasattr(self, "_cached_decode_position_ids")
	and hasattr(self, "_prefill_lens")
	and self._cached_decode_position_ids is not None
	and self._prefill_lens is not None
	and self._cached_decode_position_ids.shape[0] == batch_size
	)
	if not use_cached_positions:
	position_ids, rope_deltas = self.get_rope_index(
	input_ids,
	image_grid_thw,
	images_per_sample=images_per_sample,
	attention_mask=attention_mask_2d,
	)
	self.rope_deltas = rope_deltas
	else:
	# Per-sample decode position lookup
	# _cached_decode_position_ids shape: [batch_size, 3, max_decode_len]
	# _prefill_lens shape: [batch_size]
	position_ids_list = []
	for batch_idx in range(batch_size):
	prefill_len = self._prefill_lens[batch_idx].item()
	step = cache_position[0].item() - prefill_len
	# Get position ids for this sample
	sample_pos_ids = self._cached_decode_position_ids[batch_idx, :, step : step + seq_length]
	position_ids_list.append(sample_pos_ids)
	# Stack and transpose to [3, batch_size, seq_length]
	position_ids = torch.stack(position_ids_list, dim=1)

[Copilot]

The removal of min_pixels and max_pixels as class attributes could be a breaking change for users who directly access these attributes (e.g., processor.min_pixels). Consider documenting this change or maintaining backward compatibility by keeping them as None defaults in the class definition.

[Copilot]

The unused position_ids parameter has been removed from the function signature. However, the docstring still references this parameter in the Args section. The docstring should be updated to reflect this change.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: glm_image