Visual cue anomaly

[tangrui656]

Hello, I am using your ONNX weights and the text prompts are functioning properly. However, there was an abnormal phenomenon with the box prompt. I'm not sure if it is related to my version of onnxruntime. The version of onnxruntime is 1.18.1 and the CUDA version is 11.8. There was a situation of confusion in the inference process.

Use the script you provided, but there are no abnormalities in the text prompts.

[tangrui656]

This abnormal situation has occurred on different computers. Do you have any solutions for this? Thank you.

[jamjamjon]

@tangrui656 Hello, you should upgrade your ONNX Runtime to the latest version (1.23.2) and then try running it to see the results.
I ran the Rust example on my side and didn’t encounter any issues with the model.
Are you using the Python code I provided? Which precision of the model are you using? I’ll verify it later.

[tangrui656]

Hello, thank you for your reply. I have upgraded to version 1.23.2 ONNX. The execution results on both CPU and GPU are the same - they are all chaotic. I am using the script you provided. python infer.py --image test/bus.jpg --boxes "pos:144,750,194,517" --model-dir onnxfp16 --tokenizer onnxfp16/tokenizer.json
--output output-box.png

[tangrui656]

Both FP16 and full precision were tested in terms of accuracy, but the results were different and still quite confusing. The text is free of any issues.

[jamjamjon]

@tangrui656 The error is caused by an incorrect box coordinate input.

The box format must be xywh, i.e. x_min, y_min, width, height.
Please refer to: https://github.com/jamjamjon/usls/blob/main/examples/open-set-segmentation/README.md

Your input:
pos: 144,750,194,517

Correct input:
pos: 47,399,194,517

[tangrui656]

OK, thank you. I thought you meant the center point as mentioned by xywh. Thank you for your reply. Additionally, I would like to ask if you have considered trying out Triton to deploy SAM3. This seems to be something no one has attempted yet, but the open-source aspect of it is quite innovative.

[tangrui656]

There is one more minor issue. Besides BOX, does this ONNX support providing prompts through clicks?

[jamjamjon]

@tangrui656

point prompt	desc	example
pos:x,y	Positive point (2 coords)	-p "pos:500,375"
neg:x,y	Negative point (2 coords)	-p "neg:300,400"

https://github.com/jamjamjon/usls/tree/main/examples/open-set-segmentation
https://github.com/jamjamjon/usls/tree/main/examples/image-segmentation

[tangrui656]

Hello, I would like to use Python to provide coordinates for the segmentation process. After reviewing your ONNX model, it seems that it can only accept BOX input. Could you please provide a demo that can output a prompt with point coordinates that can be processed by Python?

[jamjamjon]

@tangrui656 Sorry, I’m not able to provide the corresponding Python code at the moment. usls is primarily focused on Rust implementations.

For point handling, you can:

1. Refer to the Rust implementation in usls, if you’re familiar with Rust;
2. Refer to the implementations in transformers:

• sam3-image: https://github.com/huggingface/transformers/tree/main/src/transformers/models/sam3
• sam3-tracker: https://github.com/huggingface/transformers/tree/main/src/transformers/models/sam3_tracker

[tangrui656]

Does your open-source ONNX weights support point input? Because I noticed the dimension is 4.

[jamjamjon]

Does your open-source ONNX weights support point input? Because I noticed the dimension is 4.

The geometry-encoder model you are currently looking at is for the sam3-image task.

If you need interactive image segmentation using point or box prompts, you should use sam3-tracker instead.

You can obtain the ONNX models here:
https://github.com/jamjamjon/assets/releases/tag/sam3

All models whose names start with tracker- are for the sam3-tracker task.

Additional notes:
sam3-image and sam3-tracker are the official SAM3 implementations provided by SAM3 itself and by Transformers, respectively, for these two tasks.
• sam3-image implements opset-based segmentation with text and box inputs.
• sam3-tracker continues the visual prompt–based workflow of SAM and SAM2 (e.g., point and box prompts).

Note: usls does not export the memory bank, so video tracking is not supported.
You can refer to the sam3-video-tracker implementation in the Transformers repository for video-related functionality.