Skip to content

Commit 34c524b

Browse files
authored
Separate results from clip benchmark repo (#141)
* delete benchmark/ and probe_benchmark/ from repo. Will be moved to https://github.com/LAION-AI/CLIP_benchmark_results. * update README and add webdatasets.txt back * update citation
1 parent 48de38c commit 34c524b

23 files changed

+25
-9110
lines changed

README.md

Lines changed: 25 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,9 @@ The goal of this repo is to evaluate CLIP-like models on a standard set
55
of datasets on different tasks such as zero-shot classification and zero-shot
66
retrieval, and captioning.
77

8-
Below we show the average rank (1 is the best, lower is better) of different CLIP models, evaluated
9-
on different datasets.
10-
11-
![benchmark.png](benchmark.png)
12-
13-
The current detailed results of the benchmark can be seen [here](benchmark/README.md)
14-
or directly in the [notebook](benchmark/results.ipynb).
8+
- Results of OpenCLIP models on 38 datasets can be seen here: <https://github.com/mlfoundations/open_clip/blob/main/docs/openclip_results.csv>.
9+
- Results from "Reproducible scaling laws for contrastive language-image learning" [[arXiv]](https://arxiv.org/abs/2212.07143) can be seen here: <https://github.com/LAION-AI/scaling-laws-openclip>
10+
- Additional results are available in <https://github.com/LAION-AI/CLIP_benchmark_results>.
1511

1612
## Features
1713

@@ -34,7 +30,7 @@ ViT-B-32,openai
3430

3531
to get the list of datasets
3632
```
37-
wget https://raw.githubusercontent.com/LAION-AI/CLIP_benchmark/main/benchmark/webdatasets.txt
33+
wget https://raw.githubusercontent.com/LAION-AI/CLIP_benchmark/main/clip_benchmark/datasets/webdatasets.txt
3834
```
3935

4036
Then to run
@@ -411,3 +407,24 @@ python setup.py install
411407
- Thanks to [@teasgen](https://github.com/teasgen) for support of validation set and tuning linear probing similar to OpenAI's CLIP.
412408
- Thanks to [@visheratin](https://github.com/visheratin) for multilingual retrieval datasets support from <https://arxiv.org/abs/2309.01859>.
413409
- This package was created with [Cookiecutter](https://github.com/audreyr/cookiecutter) and the [audreyr/cookiecutter-pypackage](https://github.com/audreyr/cookiecutter-pypackage) project template. Thanks to the author.
410+
411+
## Citation
412+
413+
```
414+
@software{cherti_2025_15403103,
415+
author = {Cherti, Mehdi and
416+
Beaumont, Romain},
417+
title = {CLIP benchmark},
418+
month = may,
419+
year = 2025,
420+
publisher = {Zenodo},
421+
doi = {10.5281/zenodo.15403103},
422+
url = {https://doi.org/10.5281/zenodo.15403103},
423+
swhid = {swh:1:dir:8cf49a5dd06f59224844a1e767337a1d14ee56c2
424+
;origin=https://doi.org/10.5281/zenodo.15403102;vi
425+
sit=swh:1:snp:dd153b26f702d614346bf814f723d59fef3d
426+
77a2;anchor=swh:1:rel:cff2aeb98f42583b44fdab5374e9
427+
fa71793f2cff;path=CLIP\_benchmark-main
428+
},
429+
}
430+
```

benchmark/README.md

Lines changed: 0 additions & 56 deletions
This file was deleted.

0 commit comments

Comments
 (0)