Eval bug: Qwen 3 VL provides incorrect bounding boxes on sizes that are not 1000x1000px / non-square

[ayayakirara]

Name and Version

./llama-b6887-bin-ubuntu-x64/build/bin/llama-cli: /lib64/libcurl.so.4: no version information available (required by ./llama-b6887-bin-ubuntu-x64/build/bin/llama-cli)
load_backend: loaded RPC backend from /var/home/user/LLMs/llama-b6887-bin-ubuntu-x64/build/bin/libggml-rpc.so
load_backend: loaded CPU backend from /var/home/user/LLMs/llama-b6887-bin-ubuntu-x64/build/bin/libggml-cpu-icelake.so
version: 6887 (d261223)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0 for x86_64-linux-gnu

(Using 6887 from releases)

Operating systems

Linux

GGML backends

CPU

Hardware

Ryzen 5 8640HS

Models

https://huggingface.co/unsloth/Qwen3-VL-30B-A3B-Instruct-GGUF/tree/main

unsloth-Qwen3-30B-A3B-Instruct-2507-UD-Q4_K_XL.gguf
unsloth-Qwen3-VL-30B-A3B-Instruct-mmproj-F16.gguf

Problem description & steps to reproduce

Steps to Reproduce

Run ./{{llama_path}}/llama-server -m unsloth-Qwen3-VL-30B-A3B-Instruct-UD-Q4_K_XL.gguf --mmproj unsloth-Qwen3-VL-30B-A3B-Instruct-mmproj-F16.gguf -c 32768 --jinja --no-context-shift --slot-prompt-similarity 0.0 --keep 0 --cache-ram 0 --no-perf --temp 0 -np 1

Use test image here:

Use test HTML provided here: #16207 (comment)

Expected Results

Expectation is that bounding boxes are accurate without any adjustments of the pixel coordinates. However in reality for the square image we need to resize the image to be 1.25x so that the bounding box coordinates match the bounding box output.

Actual results

After scaling by 1.25x

For the non-square image, bounding boxes have incorrect aspect ratio for bounding boxes.

Related

Probably relates to #16842

First Bad Commit

6887 and 6891 have the same observed behavior

Relevant log output

N/a

[ayayakirara]

Might also be related to observation here: #16780 (comment)

[99991]

Qwen3-VL's bounding box coordinates are relative to a 1000x100 image (Thanks to @theo77186 below for pointing this out).

I transformed the bounding box coordinates back to the original image size like this:

for bbox in bboxes:
    x0, y0, x1, y1 = bbox["bbox_2d"]

    size = 1000

    x0 = x0 / size * image.width
    y0 = y0 / size * image.height
    x1 = x1 / size * image.width
    y1 = y1 / size * image.height

Full example

import json, base64, typing, requests
from PIL import Image
from pathlib import Path
import matplotlib.pyplot as plt

def main():
    url = "https://i.sstatic.net/ya9Pq.png"
    prompt = "bounding boxes for all objects"

    # Download image if it does not exist yet
    filename = url.split("/")[-1]
    if not Path(filename).exists():
        data = requests.get(url).content
        with open(filename, "wb") as f:
            f.write(data)

    buf = ""
    for chunk in image_query(filename, prompt):
        print(chunk, flush=True, end="")
        buf += chunk

    bboxes = json.loads(buf.strip().removeprefix("```json").removesuffix("```"))

    image = Image.open(filename)

    plt.imshow(image)

    for bbox in bboxes:
        x0, y0, x1, y1 = bbox["bbox_2d"]

        size = 1000

        x0 = x0 / size * image.width
        y0 = y0 / size * image.height
        x1 = x1 / size * image.width
        y1 = y1 / size * image.height

        plt.plot([x0, x1, x1, x0, x0], [y0, y0, y1, y1, y0])

    plt.show()

def image_query(filename: str, prompt: str) -> typing.Generator[str, None, None]:
    url = "http://127.0.0.1:8080/v1/chat/completions"
    ext = {"jpg": "jpeg", "jpg": "jpg", "png": "png"}[filename.rsplit(".")[-1]]

    with open(filename, "rb") as f:
        data = base64.b64encode(f.read()).decode("ascii")

    data = {
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": [
                {"type": "image_url", "image_url": {
                    "url": f"data:image/{ext};base64," + data}
                },
                {"type": "text", "text": prompt}
            ]}
        ],
        "temperature": 0,
        "seed": 0,
        "stream": True
    }

    with requests.post(url, json=data, stream=True) as r:
        r.raise_for_status()

        chunks = (chunk.decode("utf-8") for chunk in r.iter_lines() if chunk)

        for chunk in chunks:
            if chunk == "data: [DONE]": break
            assert not chunk.startswith("event: error"), decoded_chunk

            content = json.loads(chunk.removeprefix("data: "))["choices"][0]["delta"].get("content")

            if content:
                yield content

if __name__ == "__main__":
    main()

llama.cpp invocation (version 6898) for less than 6GB VRAM

llama-server \
    --jinja \
    -ngl 999 \
    --flash-attn auto \
    --ctx-size 4096 \
    --mmproj mmproj-BF16.gguf \
    --model Qwen3-VL-4B-Instruct-Q5_K_M.gguf

Models downloaded from here: https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-GGUF/tree/main

[theo77186]

Qwen3-VL bounding box coordinates are relative (per thousand unit), see https://github.com/QwenLM/Qwen3-VL/blob/599ce6104cab6e111f40a30e786561e41b9731ba/cookbooks/2d_grounding.ipynb. So regardless of the image's original size, both x and y will always be from 0 to 1000.

Coordinate System: Qwen3-VL's default coordinate system has been changed from the absolute coordinates used in Qwen2.5-VL to relative coordinates ranging from 0 to 1000. (You don't need to calculate the resized_w) [emphasis mine]

[tarruda]

@ayayakirara In the annotate HTML, search for the bboxes.forEach block and replace it with this:

                        bboxes.forEach(bbox => {
                            const [label, x1, y1, x2, y2] = bbox;
                            const imgWidth = document.getElementById('imageWidth').value;
                            const imgHeight = document.getElementById('imageHeight').value;
                            const size = 1000;
                            const X1 = x1 / size * imgWidth;
                            const Y1 = y1 / size * imgHeight;
                            const X2 = x2 / size * imgWidth;
                            const Y2 = y2 / size * imgHeight;
                            const box = document.createElement('div');
                            box.className = 'bbox';
                            box.style.left = `${X1}px`;
                            box.style.top = `${Y1}px`;
                            box.style.width = `${X2 - X1}px`;
                            box.style.height = `${Y2 - Y1}px`;
                            // Add label to the box
                            box.title = label;
                            container.appendChild(box);
                        });

[ayayakirara]

I see that makes sense. I didn't realize it was changed for Qwen3 VL. Thanks, yall! Hopefully this helps someone else in case they also don't realize this.

[ngxson]

If someone can, please give a try with #16878

The bbox accuracy should be improved, even for big images