test: Add HF Space Dockerfile using pre-built leaderboard image

[isaac-chung]

Fixes #3821

Adds a lightweight Dockerfile for HuggingFace Space deployment that uses the pre-built ghcr.io/embeddings-benchmark/mteb/leaderboard image as base. Also adds a workflow to test the Dockerfile.

Note: The workflow has GITHUB_TOKEN so it'll be able to pull the base image. But it won't be visible from the HF Space, so the images need to be made public.

[isaac-chung]

@Samoed can you add a secret in the HF space? https://huggingface.co/docs/hub/spaces-overview#managing-secrets
I can remove the test.

[Samoed]

I don't think that token would work, because we would need to authorize somehow to pull image. I can't find documentation for that on HF

[isaac-chung]

I don't think that token would work, because we would need to authorize somehow to pull image. I can't find documentation for that on HF

AFAIU from what I tried, we need either cases to be true to pull the image:

1. Authorize images/packages to be public in this repo, OR
2. Use GITHUB_TOKEN and pull the image normally (example done in CI)

Would you mind adding the token and we can try this out?

[Samoed]

Use GITHUB_TOKEN and pull the image normally (example done in CI)

In ci you basically do docker login ..., but we can't do the same in HF, because we can't run commands before image running

[isaac-chung]

Ah yes, you're right. We'll have to wait for the packages to be made public then.

[KennethEnevoldsen]

public!

[isaac-chung]

Also https://huggingface.co/spaces/mteb/leaderboard/discussions/175

[KennethEnevoldsen]

@isaac-chung feel free to merge that - it looks good (but do it at a time where you have to time to check if the leaderboard breaks)

[isaac-chung]

@KennethEnevoldsen btw I don't have write permissions to the HF LB space, so I'm not able to merge that PR yet :(

[KennethEnevoldsen]

Oh you should have write - you should have them now

[isaac-chung]

Thanks! Confirming that I now see the merge button.

[Samoed]

Do we need this pr now?

[isaac-chung]

Yes. The point is to keep a copy of the LB dockerfile, so we won't miss updates in future.

[isaac-chung]

LB restarted and seems to be running well 🎉

[isaac-chung]

Before we merge this, I'd like to include a small CI workflow to simply run this HF space dockerfile.

[KennethEnevoldsen]

Yeah it would be great to confirm that this runs so that we see the error here before we see it on the leaderboard

[Samoed]

But don't we test it already in https://github.com/embeddings-benchmark/mteb/blob/main/.github/workflows/leaderboard_docker.yml

[Samoed]

I think we're testing basically the same image with same command in https://github.com/embeddings-benchmark/mteb/blob/main/.github/workflows/leaderboard_docker.yml

[Samoed]

I think this action won't be triggered

[isaac-chung]

Did you check the right sections? https://github.com/embeddings-benchmark/mteb/actions/runs/20725664296/job/59500466946?pr=3838

[Samoed]

I checked this. This will be triggered only on changed in dockerfile for spaces, but I'm not sure if it will be enough to test then

[Samoed]

I think it's better to move these checks to leaderboard_docker.yml to test this more frequently and make sure that our leaderboard will work

[isaac-chung]

Correct, that's what the paths section of the workflow file shows. I'm not sure there's new info here.

[isaac-chung]

Keeping these tests separate makes more sense.

[isaac-chung]

Once again, the point of this test is to track the exact same dockerfile used in the LB. We then will be able to:

1. track any code that will affect the LB in the HF Space
2. Test the exact same dockerfile used in the HF Space

Even though there's a lot of overlap, this test completes the LB test coverage.
e.g. If the images were not made public, this test would have failed.

[isaac-chung]

Let's try to keep the nit to the least if possible :) going to merge now