Add active_parameters_override for MOE models

[ayush1298]

Originally posted by @KennethEnevoldsen #3837 (comment)

Identify which models from MTEB are MOE, calculate active_parameters for them, and add them to their ModelMeta.

[KennethEnevoldsen]

I few models that I believe is MoE:

GritLM/GritLM-8x7B
nomic-ai/nomic-embed-text-v2-moe

@Samoed do you know of others?

[Samoed]

No. I think this is all we have for now

[ayush1298]

For GritLM/GritLM-8x7B:
GRITLM 8X7B is the best open generative language model on our task average, while only using 13B parameters at inference.
So, I think active_parameters = 13B here

For nomic-ai/nomic-embed-text-v2-moe ( sourec from their hf):

Total Parameters: 475M
Active Parameters During Inference: 305M
Architecture Type: Mixture of Experts (MoE)

think active_parameters = 305M here

So, should we add these directly? I was not able to load the GritLM model

[KennethEnevoldsen]

Yeah these sound great to add

[ayush1298]

Added these in PR with these 2 models. I think we can keep this issue open, if we found any other MOE in case.

[Samoed]

I don't think we need to keep it open. That is all models, and we can just fill later if we find some

[ayush1298]

okay, sure.