[BUG] 使用processor处理时position_ids作为key缺失

[EEE1even]

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

• 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

• 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

我在使用MiniCPM-V-4时出现报错，使用processor处理图文时position_ids为None，下面是我的code

from PIL import Image
import torch
from transformers import AutoModel, AutoTokenizer, AutoProcessor

model_path = 'MiniCPM-V-4'
model = AutoModel.from_pretrained(model_path, trust_remote_code=True, attn_implementation='sdpa', torch_dtype=torch.bfloat16)
model = model.eval().cuda()
device = next(model.parameters()).device
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

image = Image.open('/path/to/image.png').convert('RGB')

question = "What is the landform in the picture?"
msgs = [{'role': 'user', 'content': f'(<image>./</image>)\n{question}'}]

input_query = processor.tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
inputs = processor(
    input_query,
    image,
    padding=True,
    return_tensors="pt",
    max_inp_length=32768
).to(device)

labels = inputs['input_ids'].clone()
labels[:,:400] = -100 

out = model(**inputs ,labels=labels)
out.loss

但是我无法通过model(**inputs ,labels=labels)获取到out以及我所期望的loss,因为会出现下述报错

Traceback (most recent call last):
  File "/mnt/shared-storage-user/huangshiyang/mllm_sv/sv_attack_minicpm/data_augment/test_minicpm.py", line 82, in <module>
    out = model(**inputs ,labels=labels)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/shared-storage-user/huangshiyang/miniconda3/envs/sv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/shared-storage-user/huangshiyang/miniconda3/envs/sv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/huangshiyang/.cache/huggingface/modules/transformers_modules/85a1b575fcfeaecbe8abcecb41004dc0926c9b56/modeling_minicpmv.py", line 217, in forward
    if position_ids.dtype != torch.int64:
       ^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'dtype'

我在检查源码发现在modeling_minicpmv.py的第217行会有position_ids.dtype的判断，但是position_ids为空；我在使用官方文档提供的model.chat也是如此，processor的处理并没有生成position_ids这个key

期望行为 | Expected Behavior

我希望能通过model(**inputs,labels)正常获得loss，而非出现position_ids为None的报错

复现方法 | Steps To Reproduce

使用官方文档提供的案例或是上述提供的code就能进行复现，会发现经过processor处理后，inputs并没有position_ids这个key

运行环境 | Environment

- OS:Ubuntu22.04
- Python:3.11
- Transformers:4.55.4
- PyTorch:2.7.1
- CUDA (`python -c 'import torch; print(torch.version.cuda)'`):12.8

备注 | Anything else?

No response

[tc-mb]

https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/inference/single_image.md
您好，我看到log里面好像不是我们提供的推理示例代码，我们的推理应该是model.chat函数，可以参考这个文档或者hf仓库。
希望这可以帮到您，如果问题可以再提问。