Model repetitive answering pattern and getting stuck observed

[purvanshgarg]

Hey,
I had served the model using cli script but I found the model giving repetitive answers and getting stuck in a loop and ending abruptly also for at some instances it was just repeating the prompts of the user and not giving any response. I had tried running the model in 4 bit, 8 bit and full precision mode but the issue did not resolve.
Also, I was trying to run the eval script on 6gb VRAM gpu but it did not have the 4 bit configuration and using the 4 bit configuration while loading pretrained model gave some data mismatch errors for me. Can you please share if you have some configuration of eval script that can be run using 4bit mode. I am using the pretrained weights here due to hardware limitations

[purvanshgarg]

Scripts for your reference:

cli.py

while running cli:

/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/transformers/utils/generic.py:441: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/transformers/utils/generic.py:309: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
[2025-09-03 10:51:14,778] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/deepspeed/runtime/zero/linear.py:49: FutureWarning: torch.cuda.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda') instead.
def forward(ctx, input, weight, bias=None):
/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/deepspeed/runtime/zero/linear.py:67: FutureWarning: torch.cuda.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda') instead.
def backward(ctx, grad_output):
/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/transformers/utils/generic.py:309: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
usage: cli.py [-h] [--model-path MODEL_PATH] [--model-base MODEL_BASE] --image-file IMAGE_FILE [--num-gpus NUM_GPUS] [--conv-mode CONV_MODE] [--temperature TEMPERATURE]
[--max-new-tokens MAX_NEW_TOKENS] [--load-8bit] [--load-4bit] [--debug]
cli.py: error: argument --max-new-tokens: expected one argument
(segearthr1) user@STPL:~/mllm/SegEarth-R1$ python -m segearth_r1.serve.cli --model-path /home/suhora/mllm/SegEarth-R1/SegEarth-R1-EarthReason --image-file /home/suhora/mllm/SegEarth-R1/demo/oem_koeln_50.tif --temperature 0.6 --max-new-tokens 256 --debug
/home/envs/segearthr1/lib/python3.10/site-packages/transformers/utils/generic.py:441: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/transformers/utils/generic.py:309: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
[2025-09-03 10:51:40,977] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/deepspeed/runtime/zero/linear.py:49: FutureWarning: torch.cuda.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda') instead.
def forward(ctx, input, weight, bias=None):
/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/deepspeed/runtime/zero/linear.py:67: FutureWarning: torch.cuda.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda') instead.
def backward(ctx, grad_output):
/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/transformers/utils/generic.py:309: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
loading segmentation model
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
current model is SegEarth-R1-EarthReason
/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/torch/functional.py:513: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1720538438429/work/aten/src/ATen/native/TensorShape.cpp:3609.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Mask Decoder has been trained, init directly
current seg concat mode: False, seg_norm: False, seg_proj: True, seg_fuse_score: False
[INFO] Successfully loaded model: SegEarth-R1-EarthReason
[INFO] Model architecture: segearth_r1
[INFO] Model is running on device: cuda:0
[INFO] Model precision: full precision (fp16/fp32 depending on GPU support)
Human: What do you see in the image?
Assistant: Yes, It is
The tennis court is a carefully constructed space dedicated to the game of tennis, where players aim to strike a ball over. [SEG]. over the water, the tennis court serves as a dedicated arena for enthusiasts of the sport, fostering both physical fitness and social connections. [SEG] connections.
The tennis court stands as a vital hub for the community, offering a dedicated space for engaging in physical activity and fostering social connections among people. [SEG] connections.
The tennis court is a carefully crafted space, tailored for the game of tennis, where players aim to hit a ball
with both purpose and enjoyment.
The tennis court serves as a dedicated arena for enthusiasts of the game, fostering not only physical health but also social connections among people.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image.
There is no target object in the image. ^Z

Demo Image:

oem_koeln_50.tif

eval.py

eval.sh

While running eval script gives model loading is successful, and mask decoder has been trained message but then this follows:

current seg concat mode: False, seg_norm: False, seg_proj: True, seg_fuse_score: False
Found 1135 files in /home/mllm/SegEarth-R1/data_path/rs_reason_seg/RSReasonSeg/val/images
Found 1135 files in /home/mllm/SegEarth-R1/data_path/rs_reason_seg/RSReasonSeg/val/labels
Found 1135 files in /home/mllm/SegEarth-R1/data_path/rs_reason_seg/RSReasonSeg/val/QAs
  0%|                                                                         | 0/227 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/home/mllm/SegEarth-R1/segearth_r1/eval_and_test/eval.py", line 270, in <module>
    evaluation()
  File "/home/mllm/SegEarth-R1/segearth_r1/eval_and_test/eval.py", line 220, in evaluation
    outputs = model.eval_seg(
  File "/home/mllm/SegEarth-R1/segearth_r1/model/language_model/llava_phi.py", line 1065, in eval_seg
    image_features = self.get_vision_tower_feature(images)
  File "/home/mllm/SegEarth-R1/segearth_r1/model/language_model/llava_phi.py", line 201, in get_vision_tower_feature
    features = vision_tower(images)
  File "/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/mllm/SegEarth-R1/segearth_r1/model/multimodal_encoder/swin_trans.py", line 624, in forward
    x_out, H, W, x, Wh, Ww = layer(x, Wh, Ww)
  File "/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/mllm/SegEarth-R1/segearth_r1/model/multimodal_encoder/swin_trans.py", line 394, in forward
    x = blk(x, attn_mask)
  File "/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/mllm/SegEarth-R1/segearth_r1/model/multimodal_encoder/swin_trans.py", line 232, in forward
    attn_windows = self.attn(x_windows, mask=attn_mask)  # nW*B, window_size*window_size, C
  File "/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/anaconda3/envs/segearthr1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/mllm/SegEarth-R1/segearth_r1/model/multimodal_encoder/swin_trans.py", line 129, in forward
    attn = (q @ k.transpose(-2, -1))
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 384.00 MiB. GPU 0 has a total capacity of 5.78 GiB of which 194.56 MiB is free. Including non-PyTorch memory, this process has 5.15 GiB memory in use. Of the allocated memory 4.47 GiB is allocated by PyTorch, and 596.74 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

[purvanshgarg]

was able to resolve the cuda memory error and make it feasible to run for eval script within the hardware constraints but when I ran the script I observed that the model was loosing the segmentation learning due to the quantisation (as I only made the 4 bit conversion for LLM part and not the segmentation part to avoid consistent tensor mismatch issues). Have you guys made a 4 bit config of the model that runs on low end gpus, as the pretrained model could be loaded in 4 bit mode in your script but using that lead to some type mismatch when doing tensor operations.