Skip to content

fix: Update metadata to include active number of parameter to ModelMeta #3837

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Samoed merged 32 commits into from
Jan 19, 2026
Merged

fix: Update metadata to include active number of parameter to ModelMeta #3837

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
f2b424b
Add active parameter column on LB
ayush1298 Jan 3, 2026
fa8e3cc
update ModelMeta with parameters
ayush1298 Jan 4, 2026
e2a296d
update ModelMeta of models
ayush1298 Jan 4, 2026
8a884ec
Delete parameter_update_results.csv
ayush1298 Jan 4, 2026
6aa26ff
fix test
ayush1298 Jan 4, 2026
8081c7e
fix tests
ayush1298 Jan 5, 2026
50e5e81
delete script
ayush1298 Jan 6, 2026
92bd456
rename for consistency
ayush1298 Jan 6, 2026
3db3369
convert active_parameter to property
ayush1298 Jan 6, 2026
a2ea58d
Merge branch 'main' into add_active_parameter
ayush1298 Jan 6, 2026
222aae5
rename and fix property
ayush1298 Jan 7, 2026
bb4fa46
update embedding parameters for model2vec models
ayush1298 Jan 7, 2026
a541b55
remove duplicate loading of models
ayush1298 Jan 7, 2026
359e1ef
fix
ayush1298 Jan 7, 2026
59b5406
lintter
ayush1298 Jan 7, 2026
51a60a5
fix
ayush1298 Jan 7, 2026
3ce7779
remove separate method for embedding parameter calculation
ayush1298 Jan 7, 2026
659845b
fix embedding calculation to pass typecheck
ayush1298 Jan 8, 2026
77a0d58
lintter
ayush1298 Jan 8, 2026
71ed72c
fix checking
Samoed Jan 9, 2026
d757637
Merge branch 'main' into add_active_parameter
Samoed Jan 13, 2026
9846d7c
rename active parameters
Samoed Jan 13, 2026
6f81286
upd docstring
Samoed Jan 13, 2026
ec1b1bd
fix tests
ayush1298 Jan 14, 2026
7ebdad4
remove n_active_parameters_override from ModelMeta of all models
ayush1298 Jan 15, 2026
d69dd0d
lintter
ayush1298 Jan 15, 2026
7eb555b
rename file instead of merging main
ayush1298 Jan 15, 2026
3a2e37a
Merge branch 'main' into add_active_parameter
ayush1298 Jan 15, 2026
e151a30
fix tests
ayush1298 Jan 15, 2026
797a805
correct tests
ayush1298 Jan 15, 2026
7896319
Delete model total and active parameters - model_parameters.csv
ayush1298 Jan 16, 2026
9cbabb5
Merge branch 'main' into add_active_parameter
Samoed Jan 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions mteb/models/model_implementations/align_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,7 @@ def encode(
release_date="2023-02-24",
modalities=["image", "text"],
n_parameters=176_000_000,
n_embedding_parameters=None,
memory_usage_mb=671,
max_tokens=64,
embed_dim=768,
Expand Down
1 change: 1 addition & 0 deletions mteb/models/model_implementations/amazon_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
release_date="2024-04-30",
languages=["eng-Latn"],
n_parameters=None,
n_embedding_parameters=None,
memory_usage_mb=None,
max_tokens=None,
embed_dim=None,
Expand Down
2 changes: 2 additions & 0 deletions mteb/models/model_implementations/andersborges.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
revision="cb576c78dcc1b729e4612645f61db59929d69e61",
release_date="2025-11-21",
n_parameters=48042496,
n_embedding_parameters=None,
memory_usage_mb=183,
max_tokens=np.inf,
embed_dim=256,
Expand Down Expand Up @@ -43,6 +44,7 @@
revision="cb576c78dcc1b729e4612645f61db59929d69e61",
release_date="2025-11-21",
n_parameters=48578560,
n_embedding_parameters=None,
memory_usage_mb=185,
max_tokens=np.inf,
embed_dim=256,
Expand Down
1 change: 1 addition & 0 deletions mteb/models/model_implementations/ara_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
revision="ed357f222f0b6ea6670d2c9b5a1cb93950d34200",
release_date="2024-07-28",
n_parameters=135_000_000,
n_embedding_parameters=49_152_000,
memory_usage_mb=516,
embed_dim=768,
license="apache-2.0",
Expand Down
8 changes: 8 additions & 0 deletions mteb/models/model_implementations/arctic_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@
open_weights=True,
framework=["Sentence Transformers", "PyTorch", "ONNX", "safetensors"],
n_parameters=22_600_000,
n_embedding_parameters=11_720_448,
memory_usage_mb=86,
max_tokens=512,
embed_dim=384,
Expand All @@ -173,6 +174,7 @@
open_weights=True,
framework=["Sentence Transformers", "PyTorch", "ONNX", "safetensors"],
n_parameters=32_200_000,
n_embedding_parameters=11_720_448,
memory_usage_mb=127,
max_tokens=512,
embed_dim=384,
Expand All @@ -199,6 +201,7 @@
open_weights=True,
framework=["Sentence Transformers", "PyTorch", "ONNX", "safetensors"],
n_parameters=109_000_000,
n_embedding_parameters=23_440_896,
memory_usage_mb=415,
max_tokens=512,
embed_dim=768,
Expand All @@ -225,6 +228,7 @@
open_weights=True,
framework=["Sentence Transformers", "PyTorch", "ONNX", "safetensors"],
n_parameters=137_000_000,
n_embedding_parameters=None,
memory_usage_mb=522,
max_tokens=2048,
embed_dim=768,
Expand All @@ -250,6 +254,7 @@
open_weights=True,
framework=["Sentence Transformers", "PyTorch", "ONNX", "safetensors"],
n_parameters=335_000_000,
n_embedding_parameters=31_254_528,
memory_usage_mb=1274,
max_tokens=512,
embed_dim=1024,
Expand Down Expand Up @@ -280,6 +285,7 @@
open_weights=True,
framework=["Sentence Transformers", "PyTorch", "ONNX", "safetensors", "GGUF"],
n_parameters=109_000_000,
n_embedding_parameters=23_440_896,
memory_usage_mb=415,
max_tokens=512,
embed_dim=768,
Expand All @@ -306,6 +312,7 @@
open_weights=True,
framework=["Sentence Transformers", "PyTorch", "ONNX", "safetensors"],
n_parameters=305_000_000,
n_embedding_parameters=None,
memory_usage_mb=1165,
max_tokens=8192,
embed_dim=768,
Expand All @@ -331,6 +338,7 @@
open_weights=True,
framework=["Sentence Transformers", "PyTorch", "ONNX", "safetensors"],
n_parameters=568_000_000,
n_embedding_parameters=256_002_048,
memory_usage_mb=2166,
max_tokens=8192,
embed_dim=1024,
Expand Down
1 change: 1 addition & 0 deletions mteb/models/model_implementations/b1ade_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
open_weights=True,
release_date="2025-03-10",
n_parameters=335_000_000,
n_embedding_parameters=31_254_528,
memory_usage_mb=1278,
embed_dim=1024,
license="mit",
Expand Down
4 changes: 4 additions & 0 deletions mteb/models/model_implementations/bedrock_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,7 @@ def _to_numpy(self, embedding_response) -> np.ndarray:
embed_dim=1536,
open_weights=False,
n_parameters=None,
n_embedding_parameters=None,
memory_usage_mb=None,
public_training_code=None,
public_training_data=None, # assumed
Expand Down Expand Up @@ -206,6 +207,7 @@ def _to_numpy(self, embedding_response) -> np.ndarray:
embed_dim=1024,
open_weights=False,
n_parameters=None,
n_embedding_parameters=None,
memory_usage_mb=None,
public_training_code=None,
public_training_data=None, # assumed
Expand Down Expand Up @@ -235,6 +237,7 @@ def _to_numpy(self, embedding_response) -> np.ndarray:
revision="1",
release_date="2023-11-02",
n_parameters=None,
n_embedding_parameters=None,
memory_usage_mb=None,
public_training_code=None,
public_training_data=None, # assumed
Expand Down Expand Up @@ -263,6 +266,7 @@ def _to_numpy(self, embedding_response) -> np.ndarray:
revision="1",
release_date="2023-11-02",
n_parameters=None,
n_embedding_parameters=None,
memory_usage_mb=None,
public_training_code=None,
public_training_data=None, # assumed
Expand Down
17 changes: 17 additions & 0 deletions mteb/models/model_implementations/bge_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,7 @@
revision="5c38ec7c405ec4b44b94cc5a9bb96e735b38267a",
release_date="2023-09-12", # initial commit of hf model.
n_parameters=33_400_000,
n_embedding_parameters=11_720_448,
memory_usage_mb=127,
embed_dim=512,
license="mit",
Expand Down Expand Up @@ -357,6 +358,7 @@
revision="a5beb1e3e68b9ab74eb54cfd186867f64f240e1a",
release_date="2023-09-11", # initial commit of hf model.
n_parameters=109_000_000,
n_embedding_parameters=23_440_896,
memory_usage_mb=390,
embed_dim=768,
license="mit",
Expand Down Expand Up @@ -389,6 +391,7 @@
revision="d4aa6901d3a41ba39fb536a557fa166f842b0e09",
release_date="2023-09-12", # initial commit of hf model.
n_parameters=335_000_000,
n_embedding_parameters=31_254_528,
memory_usage_mb=1242,
embed_dim=1024,
license="mit",
Expand Down Expand Up @@ -421,6 +424,7 @@
revision="1d2363c5de6ce9ba9c890c8e23a4c72dce540ca8",
release_date="2023-08-05", # initial commit of hf model.
n_parameters=33_400_000,
n_embedding_parameters=10_817_536,
memory_usage_mb=127,
embed_dim=512,
license="mit",
Expand Down Expand Up @@ -448,6 +452,7 @@
revision="0e5f83d4895db7955e4cb9ed37ab73f7ded339b6",
release_date="2023-08-05", # initial commit of hf model.
n_parameters=109_000_000,
n_embedding_parameters=16_226_304,
memory_usage_mb=390,
embed_dim=768,
license="mit",
Expand Down Expand Up @@ -475,6 +480,7 @@
revision="b5d9f5c027e87b6f0b6fa4b614f8f9cdc45ce0e8",
release_date="2023-08-02", # initial commit of hf model.
n_parameters=335_000_000,
n_embedding_parameters=21_635_072,
memory_usage_mb=1242,
embed_dim=1024,
license="mit",
Expand Down Expand Up @@ -502,6 +508,7 @@
revision="4778d71a06863076696b03fd2777eb118712cad8",
release_date="2023-08-05", # initial commit of hf model.
n_parameters=33_400_000,
n_embedding_parameters=11_720_448,
memory_usage_mb=127,
embed_dim=512,
license="mit",
Expand Down Expand Up @@ -529,6 +536,7 @@
revision="b737bf5dcc6ee8bdc530531266b4804a5d77b5d8",
release_date="2023-08-05", # initial commit of hf model.
n_parameters=109_000_000,
n_embedding_parameters=23_440_896,
memory_usage_mb=390,
embed_dim=768,
license="mit",
Expand Down Expand Up @@ -562,6 +570,7 @@
revision="abe7d9d814b775ca171121fb03f394dc42974275",
release_date="2023-08-05", # initial commit of hf model.
n_parameters=335_000_000,
n_embedding_parameters=31_254_528,
memory_usage_mb=1242,
embed_dim=1024,
license="mit",
Expand Down Expand Up @@ -590,6 +599,7 @@
revision="7999e1d3359715c523056ef9478215996d62a620",
release_date="2023-09-12", # initial commit of hf model.
n_parameters=33_400_000,
n_embedding_parameters=10_817_536,
memory_usage_mb=91,
embed_dim=512,
license="mit",
Expand All @@ -616,6 +626,7 @@
revision="f03589ceff5aac7111bd60cfc7d497ca17ecac65",
release_date="2023-09-11", # initial commit of hf model.
n_parameters=109_000_000,
n_embedding_parameters=16_226_304,
memory_usage_mb=416,
embed_dim=768,
license="mit",
Expand All @@ -642,6 +653,7 @@
revision="79e7739b6ab944e86d6171e44d24c997fc1e0116",
release_date="2023-09-12", # initial commit of hf model.
n_parameters=335_000_000,
n_embedding_parameters=21_635_072,
memory_usage_mb=1278,
embed_dim=1024,
license="mit",
Expand All @@ -665,6 +677,7 @@
revision="5617a9f61b028005a4858fdac845db406aefb181",
release_date="2024-06-28",
n_parameters=568_000_000,
n_embedding_parameters=256_002_048,
memory_usage_mb=2167,
embed_dim=1024,
license="mit",
Expand Down Expand Up @@ -761,6 +774,7 @@
revision="992e13d8984fde2c31ef8a3cb2c038aeec513b8a",
release_date="2024-07-25", # initial commit of hf model.
n_parameters=int(9.24 * 1e9),
n_embedding_parameters=917_511_168,
memory_usage_mb=35254,
embed_dim=3584, # from old C-MTEB leaderboard
license="https://ai.google.dev/gemma/terms",
Expand Down Expand Up @@ -808,6 +822,7 @@
revision="971c7e1445cc86656ca0bd85ed770b8675a40bb5",
release_date="2024-07-25", # initial commit of hf model.
n_parameters=int(7.11 * 1e9),
n_embedding_parameters=131_084_288,
memory_usage_mb=27125,
embed_dim=4096,
license="apache-2.0",
Expand Down Expand Up @@ -842,6 +857,7 @@
revision="46f03bc86361cf88102b0b517b36c8259f2946b1",
release_date="2024-01-30", # January 30, 2024 - BGE-M3 release date
n_parameters=568_000_000,
n_embedding_parameters=256_002_048,
memory_usage_mb=2167,
embed_dim=1024,
license="mit",
Expand Down Expand Up @@ -871,6 +887,7 @@
languages=None,
loader=sentence_transformers_loader,
n_parameters=567754752,
n_embedding_parameters=256_002_048,
memory_usage_mb=2166,
max_tokens=8194.0,
embed_dim=1024,
Expand Down
1 change: 1 addition & 0 deletions mteb/models/model_implementations/bica_model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
revision="31237a836e5ae908c308a256573e5f0986498574",
release_date="2025-11-14",
n_parameters=110_000_000,
n_embedding_parameters=23_440_896,
memory_usage_mb=418,
embed_dim=768,
license="mit",
Expand Down
2 changes: 2 additions & 0 deletions mteb/models/model_implementations/blip2_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@ def encode(
release_date="2024-03-22",
modalities=["image", "text"],
n_parameters=3_740_000_000,
n_embedding_parameters=None,
memory_usage_mb=14285,
max_tokens=None,
embed_dim=768,
Expand All @@ -201,6 +202,7 @@ def encode(
release_date="2024-03-31",
modalities=["image", "text"],
n_parameters=7_750_000_000,
n_embedding_parameters=None,
memory_usage_mb=29577,
max_tokens=None,
embed_dim=768,
Expand Down
8 changes: 8 additions & 0 deletions mteb/models/model_implementations/blip_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,7 @@ def encode(
release_date="2023-12-07",
modalities=["image", "text"],
n_parameters=470_000_000,
n_embedding_parameters=23_442_432,
memory_usage_mb=1792,
max_tokens=512,
embed_dim=768,
Expand Down Expand Up @@ -169,6 +170,7 @@ def encode(
release_date="2023-08-01",
modalities=["image", "text"],
n_parameters=247_000_000,
n_embedding_parameters=23_442_432,
memory_usage_mb=942,
max_tokens=512,
embed_dim=768,
Expand Down Expand Up @@ -198,6 +200,7 @@ def encode(
release_date="2023-12-07",
modalities=["image", "text"],
n_parameters=247_000_000,
n_embedding_parameters=23_442_432,
memory_usage_mb=1467,
max_tokens=512,
embed_dim=768,
Expand Down Expand Up @@ -225,6 +228,7 @@ def encode(
release_date="2023-01-22",
modalities=["image", "text"],
n_parameters=247_000_000,
n_embedding_parameters=23_442_432,
memory_usage_mb=942,
max_tokens=512,
embed_dim=768,
Expand Down Expand Up @@ -252,6 +256,7 @@ def encode(
release_date="2023-08-01",
modalities=["image", "text"],
n_parameters=247_000_000,
n_embedding_parameters=23_442_432,
memory_usage_mb=942,
max_tokens=512,
embed_dim=768,
Expand Down Expand Up @@ -279,6 +284,7 @@ def encode(
release_date="2023-08-01",
modalities=["image", "text"],
n_parameters=470_000_000,
n_embedding_parameters=23_442_432,
memory_usage_mb=1793,
max_tokens=512,
embed_dim=768,
Expand Down Expand Up @@ -307,6 +313,7 @@ def encode(
release_date="2023-08-01",
modalities=["image", "text"],
n_parameters=247_000_000,
n_embedding_parameters=23_442_432,
memory_usage_mb=942,
max_tokens=512,
embed_dim=768,
Expand Down Expand Up @@ -335,6 +342,7 @@ def encode(
release_date="2023-08-01",
modalities=["image", "text"],
n_parameters=470_000_000,
n_embedding_parameters=23_442_432,
memory_usage_mb=1793,
max_tokens=512,
embed_dim=768,
Expand Down
1 change: 1 addition & 0 deletions mteb/models/model_implementations/bm25.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@ def encode(self, texts: list[str]):
revision="0_1_10",
release_date="2024-07-10", # release of version 0.1.10
n_parameters=None,
n_embedding_parameters=None,
memory_usage_mb=None,
embed_dim=None,
license=None,
Expand Down
Loading