Accessing the embeddings from the leaderboard

[flogrammer]

Hi all,

is there any way to access the raw embedding arrays for all models and tasks on the leaderboard? Are they stored somewhere?

Otherwise one has to download all models and taks and re-run the entire benchmark.

Thanks!

[Samoed]

No, we don't store them. You need to run benchmarks

[flogrammer]

Ok thanks for the fast reply. I tried using CachedEmbeddingWrapper for this. Is there a way to store all of the embeddings for different tasks and models in separate directories or do I have to do this separately for each model and task?

[Samoed]

I'm not sure that current CachedEmbeddingWrapper supports this, but I think we can change it to store results per-model per-task

[flogrammer]

Ok thanks. I'll have a look, maybe I can support with a PR.

[KennethEnevoldsen]

Thanks for the question here - as this is more of a discussion I will move it over :)