Add active parameter to `ModelMeta` #3837

ayush1298 · 2026-01-03T19:40:58Z

ayush1298 · 2026-01-03T19:41:39Z

Just added csv for now. Will remove memory usage column, add active embedding to ModelMeta of each model and then add column to LB.

Copilot

Pull request overview

This pull request adds a CSV file containing model parameter data for models in the MTEB leaderboard. The file documents total parameters, active parameters, and input embedding parameters for over 500 models. This addresses issues #3258 and #3259.

Key Changes

Addition of a CSV file documenting parameter counts (total, active, and input embedding) for 554 models
The CSV includes success/error status for each model, tracking whether parameter extraction succeeded or encountered issues

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ayush1298 · 2026-01-04T06:33:16Z

@Samoed @KennethEnevoldsen @isaac-chung
1 doubt that I have is, value of n_parameters (total number of parameters) which we have in ModelMeta, is almost equal to(though not exact) same as total_params + embedding_params counted using code.

For model: Alibaba-NLP/gte-Qwen2-7B-instruct

  ModelMeta:  7,613,000,000
  In CSV:  total_params - 7069121024, active_params - 6525621760, embedding_params - 543499264 
so, total_params + embedding_params = 7,612,620,288 which is almost equal to what we have in ModelMeta.

Also in csv these 3 parameters came from below code.

import numpy as np
from transformers import AutoModel

model_name = "google/embeddinggemma-300m"
model = AutoModel.from_pretrained(model_name)

input_params = np.prod(model.get_input_embeddings().weight.shape)
total_params = np.sum([np.prod(x.shape) for x in list(model.parameters())])
active_params = total_params - input_params

So, what exactly, I should add in ModelMeta? Should I add 2 new field in ModelMeta with name active_parameter, and embedding_parameter, and should we show both of these on LB, and remove memory_usage and number of parameters?

Samoed · 2026-01-04T07:54:10Z

Let's modify leaderboard in separate PR

1 doubt that I have is, value of n_parameters (total number of parameters) which we have in ModelMeta, is almost equal to(though not exact) same as total_params + embedding_params counted using code.

Yes, this is expected that total parameters = active parameters + embedding paramters

Should I add 2 new field in ModelMeta with name active_parameter, and embedding_parameter

Probably yes

should we show both of these on LB, and remove memory_usage and number of parameters?

Let's discuss this in issue

KennethEnevoldsen · 2026-01-04T11:54:46Z

Converting this to a draft given the current state, but great to see it!

Should we have a script to compute active parameters (e.g. the automatic metadata). Should we consider how we handle MoE? (this is where active != total - embedding)
Leaderboard: Let us deal with that in a seperate PR
embedding parameters: If we define active parameters as total - embedding (might be a good enough assumption) I would probably just add embedding parameters and let the other one be a property. We could potentially allow an overwrite using e.g. self._active_parameters

ayush1298 · 2026-01-04T12:13:18Z

Should we have a script to compute active parameters (e.g. the automatic metadata). Should we consider how we handle MoE? (this is where active != total - embedding)

Yes, I will be adding the logic to calculate automatically in from_hub method. I think MOE are case, where I was not able to calculate it. Hence I think atleast in ModelMeta we can keep n_parameters and add both active and embedding. Also, as I have already calculated these parameters for around 315 models, and around 60 models are propritory out of 498 models we have currently.

embedding parameters: If we define active parameters as total - embedding (might be a good enough assumption) I would probably just add embedding parameters and let the other one be a property. We could potentially allow an overwrite using e.g. self._active_parameters

For now, we can use csv. But, I think its better to keep both in ModelMeta as then we could also handle it for MOE case, where these property will not be useful.

ayush1298 · 2026-01-04T12:49:16Z

I have updated files with these meta and also added it to .from_hub() method. I have also added script that I have used to update ModelMeta. Will remove these script at end.

ayush1298 · 2026-01-04T15:07:55Z

Previously when I was doing make lint, it was fixing errors itself (before adding uv everywhere), but now somehow it was just giving error, but not fixing them. Why this was happening?

Samoed · 2026-01-04T15:35:12Z

Strange. I don't have this behavior

KennethEnevoldsen · 2026-01-04T15:48:51Z

Hmm do we want to use the csv? I think we should rather use the ModelMeta object - if not then I think we should use the json format similar to what we do with descriptive statistics

Samoed · 2026-01-04T15:50:39Z

I think csv just for demonstration

ayush1298 · 2026-01-05T05:23:52Z

Hmm do we want to use the csv? I think we should rather use the ModelMeta object - if not then I think we should use the json format similar to what we do with descriptive statistics

Its just for demonstration of all results I have calculated, and I had filled ModelMeta of models using that csv.
For future if someone wants to calculate or add it for models where I am not able to calculate, then can just do it in below way:

model_meta = mteb.get_model_meta("model_name")
active_parameters, embedding_parameters  = model_meta.extract_parameter_breakdown_from_hub

KennethEnevoldsen

I would convert active parameters to a property and just add embedding parameters.

mteb/models/model_implementations/andersborges.py

mteb/models/model_meta.py

ayush1298 · 2026-01-06T11:42:35Z

Converted active_parameters to property. I will check for which models n_parameter != embedding + active based on results in csv, and only for them will keep _n_active_parameters field in ModelMeta and will remove these field for rest of all models.

ayush1298 · 2026-01-07T06:51:49Z

@KennethEnevoldsen @Samoed How do I actually check for MOE, here problem is, n_parameters which we already have will not be exactly matching with embedding + active as n_parameters was reported and not excatly calculated by loading model, So, how I should decide which one is MOE, I tried keeping some threshold between difference of embedding + active value and n_parameters value, but I am not sure how correct it is to consider MOE by that way?

Samoed · 2026-01-07T11:59:34Z

n_parameters which we already have will not be exactly matching with embedding + active

Yes, but you can set _n_active_parameters and it will be returned correctly.

So, how I should decide which one is MOE, I tried keeping some threshold between difference of embedding + active value and n_parameters value, but I am not sure how correct it is to consider MOE by that way?

I'm not sure about automating it. I think model authors can submit their score by themselves. You can try to test this on nomic-ai/nomic-embed-text-v2-moe

ayush1298 · 2026-01-07T12:15:27Z

n_parameters which we already have will not be exactly matching with embedding + active

Yes, but you can set _n_active_parameters and it will be returned correctly.

But, kenneth want to have it only for MOE models in ModelMeta, and for rest of models it should not be present. So, I was thinking how to get MOE. I am okay with keeping it for all models in ModelMeta, but then what's point of keeping it as a property? though it will be used for future models where results are not reported.

So, how I should decide which one is MOE, I tried keeping some threshold between difference of embedding + active value and n_parameters value, but I am not sure how correct it is to consider MOE by that way?

I'm not sure about automating it. I think model authors can submit their score by themselves. You can try to test this on nomic-ai/nomic-embed-text-v2-moe

So, I think the best way is to keep _n_active_parameters in all ModelMeta for now. In the future, if results are not reported then property will be used in that case.

ayush1298 · 2026-01-07T12:16:33Z

1 more question in ModelMeta, it can't be named as _n_active_parameters, right as it becomes private variable, we have to name it as active_parameters or n_active_parameters?

Samoed · 2026-01-07T12:22:57Z

But, kenneth want to have it only for MOE models in ModelMeta

Yes, you shouldn't fill it for other models, I think. I don't see a problem here.

we have to name it as active_parameters or n_active_parameters?

I think you can name change _n_active_parameters to n_active_parameters and name property as active_parameters

ayush1298 · 2026-01-07T13:11:51Z

But, kenneth want to have it only for MOE models in ModelMeta

Yes, you shouldn't fill it for other models, I think. I don't see a problem here.

Yes, problem is just to identify which models are MOE.

ayush1298 · 2026-01-07T13:46:17Z

I have added _calculate_embedding_parameters to from_cross_encoder and from_senetence_tranformer. Do we have to keep calculate_embedding_parameters to access it directly for someone or not is yet to decide.
I have also updated model2vec models with n_embedding_parameters=n_parameters

As per my analysis, below are list of model2vec models, let me know if there is any model which is not, or are there any other model2vec models:

NeuML/pubmedbert-base-embeddings-100K
NeuML/pubmedbert-base-embeddings-1M
NeuML/pubmedbert-base-embeddings-2M
NeuML/pubmedbert-base-embeddings-500K
NeuML/pubmedbert-base-embeddings-8M
minishlab/M2V_base_glove
minishlab/M2V_base_glove_subword
minishlab/M2V_base_output
minishlab/M2V_multilingual_output
minishlab/potion-base-2M
minishlab/potion-base-4M
minishlab/potion-base-8M
minishlab/potion-multilingual-128M
rasgaard/m2v-dfm-large

mteb/models/model_meta.py

ayush1298 · 2026-01-08T06:09:40Z

I don't know why a typecheck error is coming; I have also added explicit conditions for that. And I have also tested code for both sentence-transformer and cross-encoder models. Also, there are some conflicts in dependencies.

Samoed · 2026-01-08T19:29:14Z

mteb/models/model_meta.py


        meta = cls._from_hub(model.model.name_or_path, revision, compute_metadata)
+        try:
+            if isinstance(model.model, PreTrainedModel):


I don't think this check is necessary because you'll get valid cross-encoder model

I though typecheck is failing because it was not able to identify this one. So, I have added explicit checks.

Samoed · 2026-01-08T19:29:32Z

mteb/models/model_meta.py

        meta = cls._from_hub(name, revision, compute_metadata)
+        try:
+            first_module = model[0]
+            if hasattr(first_module, "auto_model"):


Why do you need these checks?

Add active parameter column on LB

f2b424b

Copilot AI review requested due to automatic review settings January 3, 2026 19:40

Copilot started reviewing on behalf of ayush1298 January 3, 2026 19:41 View session

Copilot AI reviewed Jan 3, 2026

View reviewed changes

Samoed changed the title ~~Add active parameter column on LB~~ Add active parameter to ModelMeta Jan 4, 2026

KennethEnevoldsen marked this pull request as draft January 4, 2026 11:48

ayush1298 added 2 commits January 4, 2026 17:51

update ModelMeta with parameters

fa8e3cc

update ModelMeta of models

e2a296d

ayush1298 added 2 commits January 4, 2026 18:20

Delete parameter_update_results.csv

8a884ec

fix test

6aa26ff

fix tests

8081c7e

ayush1298 marked this pull request as ready for review January 5, 2026 11:50

delete script

50e5e81

KennethEnevoldsen reviewed Jan 6, 2026

View reviewed changes

ayush1298 added 2 commits January 6, 2026 16:05

rename for consistency

92bd456

convert active_parameter to property

3db3369

ayush1298 force-pushed the add_active_parameter branch from 5eda71b to 3db3369 Compare January 6, 2026 14:38

Merge branch 'main' into add_active_parameter

a2ea58d

ayush1298 added 2 commits January 7, 2026 19:03

rename and fix property

222aae5

update embedding parameters for model2vec models

bb4fa46

Samoed reviewed Jan 7, 2026

View reviewed changes

mteb/models/model_meta.py Outdated Show resolved Hide resolved

ayush1298 added 3 commits January 8, 2026 00:01

remove duplicate loading of models

a541b55

fix

359e1ef

lintter

59b5406

Samoed reviewed Jan 7, 2026

View reviewed changes

mteb/models/model_meta.py Outdated Show resolved Hide resolved

fix

51a60a5

Samoed reviewed Jan 7, 2026

View reviewed changes

mteb/models/model_meta.py Outdated Show resolved Hide resolved

ayush1298 added 3 commits January 8, 2026 00:38

remove separate method for embedding parameter calculation

3ce7779

fix embedding calculation to pass typecheck

659845b

lintter

77a0d58

Samoed reviewed Jan 8, 2026

View reviewed changes

Add active parameter to ModelMeta #3837

Are you sure you want to change the base?

Add active parameter to ModelMeta #3837

Uh oh!

Conversation

ayush1298 commented Jan 3, 2026 • edited by Samoed Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ayush1298 commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Uh oh!

ayush1298 commented Jan 4, 2026 • edited by Samoed Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Samoed commented Jan 4, 2026

Uh oh!

KennethEnevoldsen commented Jan 4, 2026

Uh oh!

ayush1298 commented Jan 4, 2026

Uh oh!

ayush1298 commented Jan 4, 2026

Uh oh!

ayush1298 commented Jan 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Samoed commented Jan 4, 2026

Uh oh!

KennethEnevoldsen commented Jan 4, 2026

Uh oh!

Samoed commented Jan 4, 2026

Uh oh!

ayush1298 commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KennethEnevoldsen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ayush1298 commented Jan 6, 2026

Uh oh!

ayush1298 commented Jan 7, 2026

Uh oh!

Samoed commented Jan 7, 2026

Uh oh!

ayush1298 commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ayush1298 commented Jan 7, 2026

Uh oh!

Samoed commented Jan 7, 2026

Uh oh!

ayush1298 commented Jan 7, 2026

Uh oh!

ayush1298 commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ayush1298 commented Jan 8, 2026

Uh oh!

Samoed Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

ayush1298 Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

Samoed Jan 8, 2026

Add active parameter to `ModelMeta` #3837

Add active parameter to `ModelMeta` #3837

ayush1298 commented Jan 3, 2026 •

edited by Samoed

Loading

ayush1298 commented Jan 3, 2026 •

edited

Loading

ayush1298 commented Jan 4, 2026 •

edited by Samoed

Loading

ayush1298 commented Jan 4, 2026 •

edited

Loading

ayush1298 commented Jan 5, 2026 •

edited

Loading

ayush1298 commented Jan 7, 2026 •

edited

Loading

ayush1298 commented Jan 7, 2026 •

edited

Loading