Results Cache invalidation

[isaac-chung]

We aim to achieve the following:

1. Add back the file writing to disk: all_results.to_disk(results_cache_path)
2. Add _load_results (without cache), which just loads the results. This function is then used in the results repo to generate the cache as well. For debugging, we add the flag --no-cache

---

Quote

The reason the cache was removed was that the cache invalidation makes leaderboard development hard (but it has been a while so I am a bit uncertain - could also just be a merge)

Example workflow:

1. Someone reports a bug (model X does not appear)
2. I run the leaderboard locally to confirm (creating a cache file)
3. I fix the bug
4. I rerun the leaderboard, which loads from cache and then hides the bug fix

Seems like we have two changes here that are meaningful: 1) faster loading using a pre-computed file, and 2) faster download by reducing the file size.

To me it seems like this could be incorporated in the current _load_results as:

# suggest that we rename to: _load_results_with_remote_cache
def _load_results(cache: ResultCache) -> BenchmarkResults:
    # check if ResultCache has a remote with a valid cached result
      # attempt to download the remote 
      # load and return
    # if not available load results and usual 

Problems I see from this are how we ensure that the loaded results are equivalent to what would be generated. An easy fix here would be to add _load_results (without cache), which just loads the results. This function is then used in the results repo to generate the cache as well. For debugging, we add the flag --no-cache.

I would ideally want these changes added to the ResultCache rather than to the leaderboard, as they are not leaderboard-specific and could benefit us elsewhere, but I am not entirely sure where to work them in. Maybe a general concern is that we could speed it up further down the chain without (reducing download size + loading time), and that would be a simpler solution that tackles the root problems (we e.g. have a PR open here: embeddings-benchmark/results#364)

I am unsure how far we can get with these. I might be leaning towards adding the above-mentioned _load_results_with_remote_cache for now - would love to hear your thoughts here as well.

Originally posted by @KennethEnevoldsen in #3795 (comment)

[isaac-chung]

Thought about this a bit more.

@Samoed @KennethEnevoldsen Let's see if the following plan makes sense:

PR 1: Add load_from_cache() method to ResultCache

Foundation refactor with disk persistence.

Files:

• mteb/cache.py - Add new method to ResultCache class:

  def load_from_cache(
      self,
      cache_path: Path | None = None,
  ) -> BenchmarkResults:
      """Load benchmark results using the best available strategy.
      
      Args:
          cache_path: Path to local JSON cache file. If None, uses default location.
      
      Strategy:
          1. If local cache exists at cache_path → load and return
          2. Try downloading pre-computed cache from 'cached-data' branch → save to cache_path and return
          3. Fallback: clone the full results repository, build from individual model files, 
             call results.to_disk(cache_path), and return.
      
      Returns:
          BenchmarkResults ready for leaderboard display
      """

• mteb/leaderboard/app.py - Simplify _load_results() to delegate to cache.load_from_cache()

PR 2: Add --rebuild flag

Adds debugging capability for leaderboard development. I opted out of "no cache" as the inversion will invite unwanted bugs and mental load.

What --rebuild does:

• Clones/pulls the full results git repository
• Builds BenchmarkResults in memory from individual model files
• Overwrites existing JSON cache via results.to_disk(cache_path)
• Bypasses any pre-computed JSON cache

Files:

• mteb/cache.py - Add rebuild: bool = False parameter to load_from_cache()
• mteb/cli/build_cli.py - Add --rebuild argument to _add_leaderboard_parser()
• mteb/leaderboard/app.py - Pass flag through

PR 3: Automatic cache staleness detection via git commit hash

Solves the core cache invalidation problem.

Behavior:

• On every load_from_cache() call, check if local cache is stale via git ls-remote (single HTTP request)
• If stale → automatically rebuild from repository
• If fresh → use cached results

Files:

• mteb/results/benchmark_results.py - Add optional fields:
  source_commit: str | None = None
  source_branch: str | None = None
• mteb/cache.py - Store commit hash and branch when building; add staleness check:

  def _is_cache_stale(self, cache_path: Path) -> bool:
      """Check if cached results are outdated compared to remote repository.
      
      Compares the git commit hash and branch stored in the cache file against
      the current HEAD of that branch on the remote (via git ls-remote).
      
      Returns:
          True if:
            - Cache file missing commit/branch metadata
            - Local repo is on a different branch than cache was built from
            - Remote branch has newer commits than cached commit
          False if cache is up-to-date
      """

Updated strategy in load_from_cache():

1. If local cache exists and _is_cache_stale() returns False → load and return
2. Otherwise → rebuild from repository and persist with new commit hash

---

[Samoed]

I like 1 and 3. For 2 maybe we can just skip loading from cache?

[isaac-chung]

Can you elaborate on your perspective?

[Samoed]

I think we can just update this

mteb/mteb/leaderboard/app.py

Line 72 in 3723d27

if not results_cache_path.exists():

to
if rebuild or not results_cache_path.exists():

[isaac-chung]

Yeah, that can surely be part of PR 2. It does already say skip over loading cache (Bypasses any pre-computed JSON cache). The descriptions above captures a fuller picture though (e.g. writing to disk as well).

[KennethEnevoldsen]

Generally 1) looks reasonable though some of the names are unclear:

load_from_cache could load anything. The behaviour that we currently wants seems very close to ResultCache.load_results. This does not download from remote though. I am not sure what the best name is, but I would probably suggest something like: ResultCache._quickload_results_from_remote. I think it is reasonably to keep it private.

I am not sure what the goal is of 2) but I suspect it is to be able to rebuilt the "quickcache" (we need a consistent name here) the same way every time. If so I would just make a function like ResultCache._to_quickcache.

[isaac-chung]

The goal of 2) is to help address the concern you brought up with cache invalidation. It rebuilds the quick cache from the cloned results repo:

• Overwrites existing JSON cache via results.to_disk(cache_path)
• Bypasses any pre-computed JSON cache

[isaac-chung]

For 1) maybe ResultCache._load_quickcache_from_remote ?

[KennethEnevoldsen]

Seems like 1) didn't work out @isaac-chung. Maybe an alternative it to completely remove it from

from mteb.leaderboard.quickcache import build_quickcache, load_quickcache

and then we can remove it completely from ResultCache? WDYT?

[isaac-chung]

No, I think your original suggestion is the most elegant (let it be part of ResultCache).

[isaac-chung]

Maybe we could use a small overview first? I feel we've been too much into the details and forget what the big picture looks like.