Improve pre-processed image size for QwenVL

[ngxson]

Extracted the discussion from:

• llama: store mrope data in KV cell #16825 (comment)
• llama: store mrope data in KV cell #16825 (comment)
• llama: store mrope data in KV cell #16825 (comment)

In order for the bbox to be correct, I'm thinking about:

• Implement the correct max_pixels / min_pixels from the original config
• Pad right/bottom corner of the image if we require to upscale (to multiple of 2*patch_size). This will guarantee that the x/y coordinates stay unchanged. But we also need to check if the original implementation actually use this strategy or not.

CC @broadbit-hu @theo77186

[pwilkin]

I think this might be the key method from the reference implementation:

def smart_resize(
    height: int, width: int, factor: int = 28, min_pixels: int = 56 * 56, max_pixels: int = 14 * 14 * 4 * 1280
):
    """Rescales the image so that the following conditions are met:

    1. Both dimensions (height and width) are divisible by 'factor'.

    2. The total number of pixels is within the range ['min_pixels', 'max_pixels'].

    3. The aspect ratio of the image is maintained as closely as possible.

    """
    if max(height, width) / min(height, width) > 200:
        raise ValueError(
            f"absolute aspect ratio must be smaller than 200, got {max(height, width) / min(height, width)}"
        )
    h_bar = round(height / factor) * factor
    w_bar = round(width / factor) * factor
    if h_bar * w_bar > max_pixels:
        beta = math.sqrt((height * width) / max_pixels)
        h_bar = max(factor, math.floor(height / beta / factor) * factor)
        w_bar = max(factor, math.floor(width / beta / factor) * factor)
    elif h_bar * w_bar < min_pixels:
        beta = math.sqrt(min_pixels / (height * width))
        h_bar = math.ceil(height * beta / factor) * factor
        w_bar = math.ceil(width * beta / factor) * factor
    return h_bar, w_bar

[ngxson]

Yes, indeed the functions which handles the max_pixels / min_pixels is pretty much the same across multiple models: https://github.com/search?q=repo%3Ahuggingface%2Ftransformers%20max_pixels&type=code

[pwilkin]

@ngxson I checked the code for 2.5 VL and it explicitly references the 2 VL image processor.

[ngxson]

Note that it will be not just an improvement to QwenVL, but other models that use the max_pixels / min_pixels strategy will also reuse the same code from this fix