Search: New CLIP models and comparison of their effectiveness

[DevServs]

Thanks to @mertalev for his work and the new tables with the performance and efficiency of search models.

My manual tests do not confirm the results shown in this tables. In my opinion, the XLM-Roberta-Large-ViT-H-14__frozen_laion5b_s13b_b90k model copes better with Russian language search than ViT-SO400M-16-SigLIP2-384__webli, which is the best in the table.

Who tested the new models in other languages? What can you say?

@mertalev, could you conduct tests on some other data set?

[mertalev]

The models were tested using CLIP-Benchmark on Crossmodal-3600. It's possible that the ranking would be different on another dataset, but I don't really have the time to look into it right now. What have you done to compare the two?

[DevServs]

I compared manually using made-up queries on my photos. In some cases I saw improvement, but in most cases the results were disappointing.

So at this point I still can't be sure that I'm using the best solution.

Thank you for your work!

[mertalev]

To be clear, are the queries the same as what you were searching for on the old model? It's hard to compare different queries against each other.

[DevServs]

Yes, the requests are completely identical, since I specifically compared before and after changing the model, taking screenshots.

[mertalev]

Hmm, that's interesting. Testing multiple datasets is probably a good idea to get a more representative result.

[mertalev]

I tested on XTD-10 as well and ViT-SO400M-16-SigLIP2-384__webli is still top for Russian.

[DrSpaldo]

I am not sure if anyone has suggestions for the english models. I just tried ViT-B-16-SigLIP2__webli because on the table it looked high. But when you search it is providing really poor results. I wish I did better testing before I switched over from the previous one!

I am going to try ViT-H-14-378-quickgelu__dfn5b next

[SpookyUSAF]

I can't find in the documentation what the default model is. I changed it so long ago that I don't even know what the default was. Does anyone have an opinion on the best overall English language CLIP model? I am running a DS920+ with HW acceleration working and 20 GB of memory. I am currently using ViT-B-16-SigLIP2__webli after using ViT-B-16-SigLIP__webli for a long time. Now with 20 GB of memory I COULD use ViT-SO400M-16-SigLIP2-384__webli but the "waiting" number drops by 1 every five or six seconds (concurrency of 4) compared to ViT-B-16-SigLIP2__webli which drops by ~9-12 every 5 seconds.

ViT-SO400M-16-SigLIP2-384__webli - Uses ~7 GB memory during smart search job
ViT-B-16-SigLIP2__webli - Uses ~1.45 GB memory during smart search job.

I'm not concerned with the memory usage as I have plenty.

Looking for some recommendations from actual users (English only) on what provides best results for them.