Scaling the benchmark

[imenelydiaker]

Think about how to make the benchmark faster:

• Subsampling some huge datasets?

Any other suggestions?

@KennethEnevoldsen @MartinBernstorff @Muennighoff feel free to edit my post and add details

[KennethEnevoldsen]

• How much time is running the task vs. downloading it? A solution might also be to stream only relevant segments in or subsample the raw HF dataset. This naturally depend on the models as well.
• Another solution for some of the datasets could be to re-use embeddings in a bootstrapping like fashion
  • e.g. for clustering:
    • embed N documents (this is the primary reduction)
      • sample K embeddings
      • do clustering evaluation on those
      • repeat however many times you want

[Arsh-10]

Think about how to make the benchmark faster:

• Subsampling some huge datasets?

Any other suggestions?

@KennethEnevoldsen @MartinBernstorff @Muennighoff feel free to edit my post and add details

Hi @imenelydiaker, I do was trying subsampling; However, I keep getting MTEB Evaluation error. Is there something I am missing, have you tried subsampling if so can you guide me a bit?

Error -> ERROR:mteb.evaluation.MTEB:Error while evaluating RedditClustering: Found input variables with inconsistent numbers of samples: [7194, 17]

Basically I was trying to only run for 17 sentences in RedditClustering dataset.

[KennethEnevoldsen]

Hi @Arsh-10 how did you do the sampling?

[Arsh-10]

I basically took fraction of the whole dataset and then ran embedding on that.
Sorry if you didn’t get my issue, issue is that if i want to evaluate using evaluate function of MTEB which gives accuracy as output. Upon doing so its giving me the above error.

So basically I want to know, Can we evaluate the accuracy (MTEB built in function from MTEB library) on fraction of dataset instead of whole?

[KennethEnevoldsen]

Currently, there is no support for that (though this discussion discusses ways we can speed up the benchmark).