Regarding speed of the computation

[Jimmy291]

Thanks for updating the codebase of SG-Nav, I have managed to install and able to run SG-Nav. However, I only managed to clock 24 episodes in 16 hours, which I think is too slow for evaluation. May I know if this is normal?

I noticed that sometime the episode will step normally but then get stuck for a long time before resuming.

[Avalon-S]

Me too. It took 18 hours to complete only 21 episodes. The bottleneck should be the ollama api. Some responses take 7 minutes, causing the step to get stuck.

Running on RTX 3090.

[Yuxin916]

Me too. Some requests take 36/37 seconds. Not sure why.

/

[Avalon-S]

Me too. Some requests take 36/37 seconds. Not sure why.

/

Hi, @Yuxin916 . I tested other mllm and the combination of llm+mllm (the paper mentioned) based on ollama. All of them will get stuck for a few minutes, which is intolerable for testing 2k episodes.
I gave up using ollama and directly pull model from hugging face then built the flask api. The stuck did not occur again. About 170 episodes can be tested per day. If your have enough gpu memery (like 4090*2), you can try vllm. It's designed for high throughput, but will take up more memery.

[Yuxin916]

Thank you for your information! Glad to hear that. I will also give it a try to directly pull the model and inference.

[Jimmy291]

Hi @Avalon-S , have you tried using vllm?

[Avalon-S]

Hi @Avalon-S , have you tried using vllm?

I haven't tried it. From experience, it should be fine.

[Chenrf1121]

Me too. Some requests take 36/37 seconds. Not sure why.
/

Hi, @Yuxin916 . I tested other mllm and the combination of llm+mllm (the paper mentioned) based on ollama. All of them will get stuck for a few minutes, which is intolerable for testing 2k episodes. I gave up using ollama and directly pull model from hugging face then built the flask api. The stuck did not occur again. About 170 episodes can be tested per day. If your have enough gpu memery (like 4090*2), you can try vllm. It's designed for high throughput, but will take up more memery.

Would you like to share your flask server file? I tried to call llama3.2-11B locally with flask, but it always fails.

[paobuliao]

I’ve attempted inference using Qwen3-VL-8B on a dual-RTX 4090 setup, yet the computation remains extremely slow—processing just 18 episodes takes approximately 1 hour. At this rate, completing the full set of 2,195 episodes is practically feasible?