Ensure that `n_embedding_parameters` is not `None`

[ayush1298]

Originally posted by @KennethEnevoldsen in #3837

Code for calculation:

If the model can be loaded from transformers/senetence-transformers then load the model and use below code:

import numpy as np
from transformers import AutoModel

model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
emb = model[0].auto_model.get_input_embeddings()
meta.n_embedding_parameters = int(np.prod(emb.weight.shape))

If the model loaded using CrossEncoder then:

import numpy as np
from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
emb = model.model.get_input_embeddings()
n_embedding_parameters = int(np.prod(emb.weight.shape))

List of models for which n_embedding_parameters is to be evaluated is as follows:

• ApsaraStackMaaS/EvoQwen2.5-VL-Retriever-3B-v1
• ApsaraStackMaaS/EvoQwen2.5-VL-Retriever-7B-v1
• BAAI/bge-visualized-base
• BAAI/bge-visualized-m3
• ByteDance/ListConRanker
• Bytedance/Seed1.6-embedding
• GritLM/GritLM-8x7B
• NovaSearch/stella_en_400M_v5
• QuanSun/EVA02-CLIP-B-16
• QuanSun/EVA02-CLIP-L-14
• QuanSun/EVA02-CLIP-bigE-14
• QuanSun/EVA02-CLIP-bigE-14-plus
• Salesforce/SFR-Embedding-Code-2B_R
• Salesforce/blip2-opt-2.7b
• Salesforce/blip2-opt-6.7b-coco
• Snowflake/snowflake-arctic-embed-m-long
• TencentBAC/Conan-embedding-v2
• TomoroAI/tomoro-colqwen3-embed-4b
• TomoroAI/tomoro-colqwen3-embed-8b
• ai-sage/Giga-Embeddings-instruct
• baseline/bm25s
• consciousAI/cai-stellaris-text-embeddings
• deepvk/USER2-base
• deepvk/USER2-small
• facebook/SONAR
• facebook/dinov2-base
• facebook/dinov2-giant
• facebook/dinov2-large
• facebook/dinov2-small
• facebook/webssl-dino1b-full2b-224
• facebook/webssl-dino2b-full2b-224
• facebook/webssl-dino2b-heavy2b-224
• facebook/webssl-dino2b-light2b-224
• facebook/webssl-dino300m-full2b-224
• facebook/webssl-dino3b-full2b-224
• facebook/webssl-dino3b-heavy2b-224
• facebook/webssl-dino3b-light2b-224
• facebook/webssl-mae1b-full2b-224
• facebook/webssl-mae300m-full2b-224
• ibm-granite/granite-vision-3.3-2b-embedding
• intfloat/mmE5-mllama-11b-instruct
• jinaai/jina-clip-v1
• jinaai/jina-colbert-v2
• jinaai/jina-reranker-v2-base-multilingual
• jxm/cde-small-v1
• jxm/cde-small-v2
• kakaobrain/align-base
• laion/CLIP-ViT-B-16-DataComp.XL-s13B-b90K
• laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K
• laion/CLIP-ViT-B-32-laion2B-s34B-b79K
• laion/CLIP-ViT-H-14-laion2B-s32B-b79K
• laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K
• laion/CLIP-ViT-L-14-laion2B-s32B-b82K
• laion/CLIP-ViT-bigG-14-laion2B-39B-b160k
• laion/CLIP-ViT-g-14-laion2B-s34B-b88K
• malenia1/ternary-weight-embedding
• microsoft/LLM2CLIP-Openai-B-16
• microsoft/LLM2CLIP-Openai-L-14-224
• microsoft/LLM2CLIP-Openai-L-14-336
• nomic-ai/colnomic-embed-multimodal-3b
• nomic-ai/colnomic-embed-multimodal-7b
• nomic-ai/modernbert-embed-base
• nomic-ai/nomic-embed-text-v1
• nomic-ai/nomic-embed-text-v1-ablated
• nomic-ai/nomic-embed-text-v1-unsupervised
• nomic-ai/nomic-embed-text-v1.5
• nomic-ai/nomic-embed-text-v2-moe
• nomic-ai/nomic-embed-vision-v1.5
• nyu-visionx/moco-v3-vit-b
• nyu-visionx/moco-v3-vit-l
• openai/clip-vit-base-patch16
• openai/clip-vit-base-patch32
• openai/clip-vit-large-patch14
• samaya-ai/promptriever-llama2-7b-v1
• samaya-ai/promptriever-llama3.1-8b-instruct-v1
• samaya-ai/promptriever-llama3.1-8b-v1
• vidore/colSmol-256M
• vidore/colSmol-500M
• vidore/colpali-v1.1
• vidore/colpali-v1.2
• vidore/colpali-v1.3
• vidore/colqwen2-v1.0
• vidore/colqwen2.5-v0.2
• voyageai/voyage-multimodal-3

[ayush1298]

Also need to add test to prevent n_embedding_parameters=None when its get calculated for all models

[ayush1298]

@KennethEnevoldsen I have updated the list and computed parameters to most of the models.
For the models for which I was not able to calculate, below is error list that I got:

1. TIGER-Lab/VLM2Vec-LoRA, TIGER-Lab/VLM2Vec-Full: Not able to run
2. ibm-granite/granite-vision-3.3-2b-embedding: Getting AttributeError: 'LlavaNextWithCustomPacking' object has no attribute 'language_model'
3. TomoroAI/tomoro-colqwen3-embed-4b,  TomoroAI/tomoro-colqwen3-embed-8b: Getting TypeError: ColQwen3.tie_weights() got an unexpected keyword argument 'missing_keys'
4. dmedhi/PawanEmbd-68M: Getting NotImplementedError: `get_input_embeddings` not auto‑handled for PawanEmbdModel; please override in the subclass.
5. consciousAI/cai-stellaris-text-embeddings: Not able to load model, already issue opened on hf
6. codefuse-ai/C2LLM-0.5B: Getting AttributeError: 'C2LLMForEmbedding' object has no attribute 'all_tied_weights_keys'
7. sensenova/piccolo-large-zh-v2: No Open weights
8. amazon/Titan-text-embeddings-v2: Some unknown error
9. ai-sage/Giga-Embeddings-instruct: Some very long error
10. TencentBAC/Conan-embedding-v2, sensenova/piccolo-large-zh-v2: No model weights on HF
11. McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-unsup-simcse:  ImportError: cannot import name 'PEFT_TYPE_TO_PREFIX_MAPPING' from 'peft.mapping' (/usr/local/lib/python3.10/dist-packages/peft/mapping.py)
12. jxm/cde-small-v1: NotImplementedError: `get_input_embeddings` not auto‑handled for DatasetTransformer; please override in the subclass.
13. nomic-ai/nomic-embed-text-v1, nomic-ai/nomic-embed-text-v1-ablated, nomic-ai/nomic-embed-text-v1-unsupervised,  nomic-ai/nomic-embed-text-v1.5: NotImplementedError: `get_input_embeddings` not auto‑handled for NomicBertModel; please override in the subclass.
14.: NotImplementedError: `get_input_embeddings` not auto‑handled for 
15. nvidia/NV-Embed-v2, nvidia/NV-Embed-v1: Torch related issue and then https://huggingface.co/nvidia/NV-Embed-v2/discussions/43, same issue, and suggestion is not working
16. malenia1/ternary-weight-embedding: Some very large error
17. Snowflake/snowflake-arctic-embed-m-v2.0, Snowflake/snowflake-arctic-embed-m-long: Error related to xformers and torchvision
18. NovaSearch/stella_en_400M_v5,  VPLabs/SearchMap_Preview: Error related to xformers
19. Kingsoft-LLM/QZhou-Embedding-Zh: Getting ImportError: cannot import name 'SlidingWindowCache' from 'transformers.cache_utils' (/usr/local/lib/python3.10/dist-packages/transformers/cache_utils.py). This is because this is deprecated now, while on downgrading  it was giving `Qwen` related error which needs newer version.
20. infly/inf-retriever-v1: Not able to load even after using cpu. Still reach over limit
21. tencent/Youtu-Embedding: Getting ImportError: cannot import name 'LossKwargs' from 'transformers.utils' (/usr/local/lib/python3.12/dist-packages/transformers/utils/__init__.py), on upgrading this, getting error ImportError: cannot import name 'is_flash_attn_available' from 'transformers.utils' which requires downgrading transformers to 4.34.1
22. dmedhi/PawanEmbd-68M: Getting NotImplementedError: `get_input_embeddings` not auto‑handled for PawanEmbdModel; please override in the subclass.
23. ByteDance/ListConRanker: Getting AttributeError: 'ListConRankerModel' object has no attribute 'all_tied_weights_keys'
24. openai/clip-vit-base-patch16, openai/clip-vit-base-patch32, openai/clip-vit-large-patch14: Getting NotImplementedError: `get_input_embeddings` not auto‑handled for CLIPModel; please override in the subclass.

Attribute Errors are still there even after using the latest version of the transformer

Also, should we remove proprietary models/models not on hf from this list, as it will not be possible to calculate values for them? And for image models/Late interaction models(which use pylate), this method of embedding calculations won't work. So, need to identify how to do this for them.

[Samoed]

How did you run these models?

[ayush1298]

How did you run these models?

In kaggle/Colab depending on size of models.

[Samoed]

I mean what code did you use? Some of the models have strange errors

[ayush1298]

I mean what code did you use? Some of the models have strange errors

Which error particularly seems strange?

import torch
import numpy as np
from sentence_transformers import CrossEncoder, SentenceTransformer
from transformers import AutoTokenizer, AutoModel, AutoConfig, AutoModelForCausalLM, PreTrainedModel, AutoModelForZeroShotImageClassification, LlavaNextForConditionalGeneration

def get_parameters_from_transformers(model_name):
    # model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True)
    # model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
    model = LlavaNextForConditionalGeneration.from_pretrained(model_name)
    emb = model.get_input_embeddings()
    if emb is not None:
        input_embedding_params = int(np.prod(emb.weight.shape))
    
    total_params = sum(p.numel() for p in model.parameters())
    total_params = int(total_params)
    
    if input_embedding_params is not None:
        active_params = (
            total_params - input_embedding_params
        )
    else:
        active_params = total_params
    
    print(model_name, ": ", total_params, active_params, input_embedding_params)
    if "model" in locals():
        del model
    torch.cuda.empty_cache()

def get_parameters_from_sentence_transformer(model_name):
    model = SentenceTransformer(model_name, trust_remote_code=True)
    first_module = model[0] 
    input_embedding_params = None
    if hasattr(first_module, "auto_model"):
        auto_model = first_module.auto_model
        if isinstance(auto_model, PreTrainedModel):
            emb = auto_model.get_input_embeddings()
            if emb is not None and hasattr(emb, "weight"):
                weight_shape = emb.weight.shape
                input_embedding_params = int(np.prod(weight_shape))
    
    total_params = sum(p.numel() for p in model.parameters())
    total_params = int(total_params)
    
    if input_embedding_params is not None:
        active_params = (
            total_params - input_embedding_params
        )
    else:
        active_params = total_params
    print(model_name, ": ", total_params, active_params, input_embedding_params)
    if "model" in locals():
        del model
    torch.cuda.empty_cache()


def get_parameters_from_cross_encoder(model_name):
    model = CrossEncoder(model_name)
    if isinstance(model.model, PreTrainedModel):
        emb = model.model.get_input_embeddings()
        if emb is not None and hasattr(emb, "weight"):
            weight_shape = emb.weight.shape
            input_embedding_params = int(np.prod(weight_shape))
            
    print(model_name, input_embedding_params)
    if "model" in locals():
        del model
    torch.cuda.empty_cache()


# model_name = "ibm-granite/granite-vision-3.3-2b-embedding"
model_name = "sensenova/piccolo-large-zh-v2"
# get_parameters_from_transformers(model_name)
get_parameters_from_sentence_transformer(model_name)
# get_parameters_from_cross_encoder(model_name)

This was whole code, its same thing which we have integrated. I just separate it out in different function format. And mostly I need to do it manually, as many of them are using transformers (not integrated in sentence-transformers) and have specific loader classes and dependency requirements to load the model.

[Samoed]

Which error particularly seems strange?

For ColQwen models, but I see that you have different functions for different model types.

AttributeError: 'LlamaConfig' object has no attribute 'rope_theta'

For this you can downgrade transformers version

[ayush1298]

Which error particularly seems strange?

For ColQwen models, but I see that you have different functions for different model types.

Yes, mostly If model was supporting sentence-transformer then I was using it directly, but if not, then I have to check and use exact loader class which was different from AutoModel sometimes.

AttributeError: 'LlamaConfig' object has no attribute 'rope_theta'

For this you can downgrade transformers version

Tried this one for model reasonir/ReasonIR-8B. Now, got Cuda OOM

[Samoed]

Tried this one for model reasonir/ReasonIR-8B. Now, got Cuda OOM

This related to all object has no attribute 'rope_theta'

[ayush1298]

Tried this one for model reasonir/ReasonIR-8B. Now, got Cuda OOM

This related to all object has no attribute 'rope_theta'

Trying for others, mostly they are 7B or higher, so I am expecting Cuda OOM there also.