model: nvidia/llama-nemotron-embed-vl-1b-v2 for ViDoRe

[gabrielspmoreira]

• I have filled out the ModelMeta object to the extent possible
• I have ensured that my model can be loaded using
  • mteb.get_model(model_name, revision) and
  • mteb.get_model_meta(model_name, revision)
• I have tested the implementation works on a representative set of tasks.
• The model is public, i.e., is available either as an API or the weights are publicly available to download

Close #4169

[Samoed]

Looks good. Added minor comment

[KennethEnevoldsen]

@gabrielspmoreira can you take a look at @Samoed comment. Once that is considered then we can merge this

[Samoed]

I don't think that disabling one of modalities by default is expected behavior for models

[KennethEnevoldsen]

I could see it as a routing thing. Like when there is text+images then use just the text, but this will def. punish the model on other tasks where the text doesn't provide enough information (I don't think that reflects the intended use).

[KennethEnevoldsen]

You can submit text-only as an experiment (docs)

[gabrielspmoreira]

I see @Samoed . I did that so that users can reproduce the results PR I submitted, which were run with these default values (use_text_modality = True, use_image_modality = False). The reason is that for Vidore V3, text-only modality provides slightly higher accuracy than image modality, but the embedding throughput is much higher. providing a better trade-off for this dataset. Does that make sense?

[Samoed]

Yes, that make sense, but I think model should use all modalities by default

[boliu61]

Hi @Samoed , I think each model should use their best modality.

E.g. we implemented and evaluated Qwen3-VL-Embedding in three modalities: image only, image + text, text only. The best one is image only (higher accuracy than image + text), so we reported image only results and made image only as the default (see #4198)

[Samoed]

I understand this, but this would create problems for users that would run text only tasks. Your approach is vidore focused, but I think we should use more general approach when we're adding models

[gabrielspmoreira]

@KennethEnevoldsen @Samoed we have followed your suggestions. Default modality for Nemotron Embed VL 1B is is image + text (when available). Results with text-only were submitted as an experiment