Guide: Getting started with choosing a Machine Learning CLIP Model for Smart Search

[radh21301]

I am an absolute newbie when it comes to ML, so like many others, I was lost on how to choose a CLIP model. For over a year, I just stuck with the default, mostly because I didn't know what else to choose and because it worked pretty well. But after the recent release of some new models and the post about those being better, I was curious, like many others, about how to proceed. I realized a lot of people were asking questions on the discord channel, and the dev (mertalev) gave some really helpful advice and information there. Since we don't have an official guide yet, I thought I will curate some of their responses here.

Note: All this information is mostly just a copy/paste or a rephrased version of what I read on Discord. I could be wrong. But hopefully, it helps someone.

Performance Metrics of Different Models 📈
I got these links from a PR (#11468). It has the performance metrics of many models. They should have most of the models supported by immich (https://huggingface.co/immich-app).
Monolingual models metrics: https://github.com/mlfoundations/open_clip/blob/main/docs/openclip_retrieval_results.csv
Multilingual models metrics: https://github.com/mlfoundations/open_clip/blob/main/docs/openclip_multilingual_retrieval_results.csv

Easiest way to choose a model
The easiest way to determine what model to choose is to look at the attached interactive plots. I believe these plots were generated using information from the links listed above. These plots are in the .zip that is attached. There are three .html files that Mertalev shared (GitHub would not let me attach a .html file. So I had to .zip). Download them and open them on any browser. You might need a computer to open it.
clip_models_efficiency_plots.zip

The following are the specs of some of the popular models in tabular form:

Bubble Size -> RAM 💾
The bigger the bubble, the more RAM it's going to consume. Note: The charts don't factor in concurrency (running multiple things at once). At the default setting of 2, you might see a tiny bump in RAM use (like 10-20 MB). Crank it up higher, and you'll notice more of a difference.
Pro tip: Hover over a bubble to get more information about that model, including the RAM.

MACs (x-axis) -> Model Speed 🛩️
If you know FLOPs, MACs is a similar metric. It’s basically how much computation the model needs to do, which relates to how much time the device needs to spend to do them. More MACs = more time your device spends thinking. On a powerful GPU, speed might not be a big deal.

• Smaller MACs = Faster model
• Larger MACs = Slower model

Quality of the search (y - axis)
Higher quality = Better search results. As simple as that. I don't know how quality is compared when the different models use different datasets. But it does give a very close idea of how the model performs.

Efficiency
"Efficiency" is related to both the x-axis (MACs) and y-axis (Quality). Models that require less number-crunching to get the same quality are more efficient.

For example: The model 'ViT-B-16-SigLIP-256__webli' has MACs = 29.45 Billion, and quality = 0.767. And the model 'ViT-H-14-378-quickgelu__dfn5b' has a staggering MACs = 542.15 Billion, and quality = 0.828. Now, certainly the bigger model will give better results. But is that increase of (~ 0.06 quality (7%)) worth the extra time to process things, you need to decide.

Remember, these charts just give you a general vibe. Don't stress too much about picking the "perfect" model based on them.

Some Questions You Might Have:
What Does "Slow" Mean? 🐌
When we're talking "slow," we're looking at two major things:

• Initial processing time (running the smart search job)
• Response time when you're actually searching for stuff
  • Time to load model into memory:
    • By default, Immich loads your model into memory when you make your first search after starting up Immich. And this might take a few seconds if the model is large. And after a default duration of 300s, the model is unloaded to reduce memory usage. TIP: If you want to keep the model constantly in memory, set the environment variable MACHINE_LEARNING_MODEL_TTL=0 for the machine learning container (in your docker compose).
    • TIP: To mitigate this initial loading time, set the environment variable MACHINE_LEARNING_PRELOAD__CLIP=<model-name> for the machine learning container. This will make the service load the specified model at startup instead of in the first request. For preloaded models, once it is loaded in memory, it will stay in memory until Immich shuts down. This means it will constantly use that RAM. Preloading the model also ensures that Immich does not unload the model after a certain duration of inactivity. If you have enough RAM, I recommend you preload your models. It has a noticeable effect on the search experience.
  • Time to search for things: This depends on the model you choose. Lower MACs = Faster.

Both 'Initial processing time' and 'Response time when a search is done' can be affected by the model choice. If you are like me, you probably hoped it would only affect the initial processing time. I don't care how long the initial time is, but it does become a bother when every search takes longer to show results. So, pick your poison. But hey, "slow" is relative, right? What is slow to you, might be fast for me!

Time Estimates? 🕰️
Here's the deal: Giving you exact times is impossible. It all depends on your hardware.

What is this 'webli', 'DFN-5b' etc?
Not all models are trained on the same data. It's like they've all read different books:

• Models ending with "webli" = WebLI dataset
• DFN5b models = DFN-5b dataset
  And so on...

How to set the model in Immich
Say you looked at the plots and have a model you want to try. Go to the huggingface page for Immich. Ctrl+F and find that model. Click on it and copy the exact name (as in the figure). Then paste that into the settings page under CLIP models (https://immich.app/docs/features/smart-search)

TIP - Both of these conventions are acceptable:

• immich-app/ViT-H-14-378-quickgelu__dfn5b
• ViT-H-14-378-quickgelu__dfn5b

Some general tips:

• Check your setup: Got enough RAM and a beefy machine? You're golden to try anything!
• Start big: Go for the model with the best efficiency/average score -> Keeping in mind the capability of your hardware.
• Test drive: Run the smart search job and then run some searches to see how it feels.
• Adjust as needed: If it's too slow, step down to a smaller model.
  FYI: This does mean, rerunning the smart search jobs, waiting for a while for all that is done. I suggest you take backups of your postgres database after every trial. So, once you are done with your trials, and know what you want, you can just switch to the database backup.

So, there you have it! Happy model hunting! 🎉

---

Some stats from my setup (Do not use this as a reference). Just to give you an idea:

You can do something similar with your trials to help you decide on one.

Hardware	Model	Macs	Quality	Assets	Duration (Initial Processing)	Duration Feel (Searching). Cant really measure this :)
Ryzen 2600, Nvidia P600, Free RAM=32 GB	ViT-B-32__laion2b_e16	7.47 B	0.71	80,000	80 mins	less than a second
	ViT-B-16-SigLIP-384__webli	63.64 B	0.79	80,000	270 mins	slightly close to a second. Little more than ViT-B-32__laion2b_e16. But the difference was in milli seconds. Not much of a noticeable one. So, I stuck with this

[mertalev]

Thanks for the write-up! Most of this could be added straight to the docs.

Small correction: "efficiency" is related to both the x-axis (MACs) and y-axis (quality). Models that require less number-crunching to get the same quality are more efficient.

A general tip to mitigate the search time for larger models is to set the env MACHINE_LEARNING_PRELOAD__CLIP=<model-name> for the machine learning service. This will make the service load the specified model at startup instead of in the first request. But of course, this means it will be constantly using that RAM.

[radh21301]

Awesome. Updated the post.

[radh21301]

Hey. I tried running the biggest model (4 gig) on a gpu with 2 gig VRAM :), and it failed - and I saw a github discussion that explained the cause. For some reason I kept thinking it was getting loaded onto the regular RAM and not the VRAM. Is the following statement correct.

Bubble Size -> Memory

• When running machine learning only on the CPU -> the 'Memory' is basically RAM
• When using hardware accelerated machine learning -> the 'Memory' is the VRAM. So limit yourself to what your GPU is capable of.

[mertalev]

Yes, if it's a discrete GPU. Integrated GPUs do use normal RAM (mainly applicable to OpenVINO).

[MiguelNdeCarvalho]

I have an Intel ARC A380 that have 6GB of VRAM, but when I try to use the ViT-H-14-378-quickgelu__dfn5b model it appears to not fit my GPU and defaults to CPU. I took a look the model should consume ~4.8GB and I'm running 2 cuncurrent jobs. Do you guys know anything about this? Thanks!

[ammgws]

@MiguelNdeCarvalho Did you end up being able to use the larger model?

[raisinbear]

Thanks for the writeup! Exactly what I was looking for. FWIW, I have Immich running on a low power RK3399 machine. So I didn't want to slow things down any furhter. I know, I could potentially use rockchip hardware acceleration, but it's by far not as straightforward as I'd like and I'm not at all in the mood for experimenting on this system right now, as it's running a bunch of other services, too. (Also, would HWA affect search speed or just inital processing?)

So I started the test drive, coming from ViT-B-32__openai and going with the obviously next better ViT-B-16-SigLIP__webli. This post includes a table showing the new model to increase MAC by over a factor of 3 vs. the old one. Indeed, initial processing took much longer. By how much, I cannot say, I'd estimate more than twice. Search, however, is not noticeably slower. It was slightly over a second with ViT-B-32__openai and if at all increased in the ms range with ViT-B-16-SigLIP__webli.
Just wanted to leave that here, in case that helps anyone.

[radh21301]

That is actually not too bad. 1 second is very reasonable. I also added some stats from my trials to the original post.
I went from 7.47 B MACs to 63.64 B. And my search time did not increase too much. Maybe a few milliseconds, which is totally fine.

[sveken]

Thanks for this.
I picked ViT-SO400M-14-SigLIP-384__webli and wondered if i was reading the graph right that this was one of the better ones (on pure quality). Though the results where mixed depending on the subject. Seems to be very good on objects but not very good at animals.

Also i did not realize the model was required for individual searches, Good to know.

Also does anyone have any insights in how much better ViT-H-14-378-quickgelu__dfn5b responds over the default in the real world. I know numbers wise its a small jump but do other people find it noticable?

[mertalev]

Just to check, but did you use any uppercase letters for the animal searches? If so, it's related to #11865.

The difference between ViT-B-32__openai and ViT-H-14-378-quickgelu__dfn5b will depend on the query. They're both great at common things like "cat". But for queries involving a less common subject, describing a scene, or searching for text in an image, the latter is much more likely to find exactly what you want. The ranking is also better in general, so the most relevant results are usually at the very top and less scattered.

I don't have experience with ViT-SO400M-14-SigLIP-384__webli, but it should be similar to ViT-H-14-378-quickgelu__dfn5b on paper, maybe with certain domains it's better or worse at.

[sveken]

Thanks, that linked issue is what i was experiencing.

[radh21301]

Hey Mertalev/internet stranger, if the content in this post looks good to you, would it be worthwhile to add it to the community guides?

https://immich.app/docs/community-guides

I would appreciate if someone would be kind enough to add it on my behalf. I know nothing of software and never used git..ever - so it is a bit daunting to try myself.

[mertalev]

I think it'd be better to add it to the Smart Search documentation. I can make a PR for that.

[radh21301]

That sounds great. Thank you!

[andyzukunft]

Hey all,

thanks for the data provided. I was looking for a good model for German and English usage with "medium" or less hardware requirements.

Based on the benchmarks my current choice is ViT-L-14-quickgelu which scores around 0.80 for both languages.

ViT-L-14-quickgelu stats
MAC: ~90
RAM: ~1900 MB
English score: 0.79
French score: 0.82
German score: 0.79

[kimdre]

Thank you for your insights! I choose the immich-app/ViT-L-14-quickgelu__dfn2b model and I like the results very much.

[CiNcH83]

I am going with the ViT-B-16-SigLIP__webli for now. It's good on RAM and performance with good score for English and German. Also delivers good results with my library.

Using it on a Raspberry Pi 5 with 8 GB.

[Qhilm]

I didn't see this mentioned here, hence adding the comment: use remote machine learning if you have a laptop which is more powerful than the machine running immich.

My MacBook Pro M1 is 35 times faster than my Synology DS918+ for example. I can run a new large model on my entire collection of 50k+ pictures in two days vs. more than two months for the Synology NAS. And that's without hardware acceleration / GPU.

I still wish we are a proper OCR job in Immich. CLIP models are just not conceived for recognising text and hence are really bad at it. Ideally, additionally to the OCR job, a way to exclude all text from CLIP, because it just confuses the model a lot I noticed. Pictures with text are often returned for unrelated searches.

[Qhilm]

I think it's not niche at all (lots of people run immich on rasberry pis and similar, this is why the remote ML feature was added) and I think it would be awesome if the remote ML feature could be extended to automatically switch between two destination URLs for ML, each with different job concurrency settings.

[slamp]

And it's the reason I choose Immich: use a remote gaming PC for ML, I discovered this feature in this video: https://www.youtube.com/watch?v=QHWNu_in0Zc

[mertalev]

I think it'd be easy to change the URL setting to accept a list and try the URLs in order. Having separate job concurrency would be a lot harder through.

[tbelway]

I think it'd be easy to change the URL setting to accept a list and try the URLs in order. Having separate job concurrency would be a lot harder through.

Separate job concurrency could be managed through queuing. If redis/valkey is retained as the queuing management, or if it's switched to postgres, both support different mechanics to do so.

[Untimed4598]

Thank you for the procedure, it would be super great (if a bit niche) that immich would be able to do this automatically (if powerfull server is up > it uses it and it's fast, if it isn't > it does it locally).

Heads up, v1.122.0 has answered your prayers!

[wnhre2ur8cxx8]

Are there maybe similar interactive plots available for the latest models that can be included since 1.130? 🥺

[Qhilm]

Interactive plots, I don't know, but the release notes link to a nice table.

At least for me, the benefits are quite limited, I'm not sure I want to run the smart search again. I would be gaining less than 1% on the recall metric.

[mertalev]

It isn't super straightforward to add non-Markdown elements in the docs, so I stuck to tables that can be displayed easily.

[akostadinov]

Hi,

I have a few questions to make the choice more efficient.

• How is it possible to run model benchmarks locally?
  I'm mostly interested in running performance benchmarks because I trust the other metrics will be the same regardless. But I find it useful to know how models perform on the specific hardware I have (which is intel 6th gen /skylake/ presently).
• How does the multi-language support work?
  I'm interested in a language that is not in the list in the table (Bulgarian). So will multi-lang models work with random non-english models or they only support the listed languages? I wonder if it would be useful in the long term to add some translation capability or external integration, so that people can use a strong English language model but local language queries will be translated before running the search.
• Is it possible to switch to FP16 or other more efficient number format?
  I read somewhere that I can't find now, that FP32 is used by default in immich. But most GPUs and accelerators support FP16 or lower bits computation with twice and 4 times the performance of FP32. So I wonder if it is possible to change that and I'm wondering whether there would be significant change of accuracy for the models.
• Is the memory usage listed for the models per-job or it is more or less a constant for the machine regardless of concurrent operations?

[akostadinov]

My observation with OpenVINO on the skylake iGPU is that the model doesn't take up the listed amount of memory multiplied by the concurrent jobs but it is a total. Because I decided to run ViT-B-16-SigLIP2__webli and total server memory usage is 6.5GB while running multiple other services as well. With ViT-B-16-SigLIP__webli it was 5.5GB so mem usage is even lower than I expected.

wrt performance between these two models, the performance difference is negligible as far as I can tell from eyeballing how quickly the queue is being processed which is in line with the performance number in the documentation.

Also power consumption of the whole server seems the same with 5 concurrent smart search jobs. Basically a total of 52watts initially but no it jumped towards 56 watts perhaps because fans need to work harder.

So far so good, I'm amazed what a good job an old intel cpu can do. Shame newer ones are not so efficient comparative to AMD.

Still would be useful to know if FP16 can be used as it should be twice as fast on this hardware. And about multi-lang support.

Update: reducing smart search concurrent jobs from 5 to 3 seems to retain the same performance so I'm keeping it at 3 to reduce scheduling overhead.

[mertalev]

How is it possible to run model benchmarks locally?
I'm mostly interested in running performance benchmarks because I trust the other metrics will be the same regardless. But I find it useful to know how models perform on the specific hardware I have (which is intel 6th gen /skylake/ presently).

To get similar data as in the tables (but obviously different speed), you can use the immich_model_exporter's profile command (you need to export the model first and point the profile command to that directory). But there are no formal instructions on this yet.

You can also do load testing with Locust to see how many requests per second the ML server can process for a model. Since you're only interested in the CLIP models, you can comment out the RecognitionFormDataLoadTest section in the locustfile.py file so it only does requests for the CLIP model.

How does the multi-language support work?

The languages listed are the languages that the evaluation benchmark covers. The models that perform well across languages should work well for Bulgarian as well. I believe Croatian is similar (correct me if I'm wrong), so the ranking for that should mostly apply to Bulgarian as well.

Is it possible to switch to FP16 or other more efficient number format?

Only for ARMNN and RKNPU at the moment. FP16 for OpenVINO would be trivial to add as well. There's no runtime conversion setting for CUDA or ROCm in ONNX Runtime, so it's a little bit more work for those devices, but still definitely worth it. Quantization is where things get murkier because the accuracy loss is more significant (it has more negative impact on smaller models compared to LLMs).

Is the memory usage listed for the models per-job or it is more or less a constant for the machine regardless of concurrent operations?

Concurrency isn't accounted for in the numbers. Memory usage will increase with higher concurrency, maybe 20MB per concurrency? VRAM usage will increase by a smaller proportion than RAM because image processing is done on CPU while the smaller, processed image is sent to the GPU.

[mertalev]

My observation with OpenVINO on the skylake iGPU is that the model doesn't take up the listed amount of memory multiplied by the concurrent jobs but it is a total. Because I decided to run ViT-B-16-SigLIP2__webli and total server memory usage is 6.5GB while running multiple other services as well. With ViT-B-16-SigLIP__webli it was 5.5GB so mem usage is even lower than I expected.

The memory usage is the combined usage of the textual model (used during searching) and the visual model (used during smart search jobs) to give a better idea of peak usage. Also, it's worth noting that OpenVINO in particular has different memory usage than other backends. It uses much more RAM the first load you load a model, after which it loads a cached model more quickly with more comparable RAM usage.

[akostadinov]

Thanks a lot for the answers! They are very useful!

I think I'm not going to run the benchmark, from what I observe the listed numbers are comparatively correct on my hardware.

wrt language, it seems like comparison with similar type of languages might vary significantly. Bulgarian is a Slavic language so has similarities with Croatian, Serbian, Ukrainian and Russian. My guess would be 20-30% similarities.

I see top model being different for the different ones and the accuracy being different. Especially I'm surprised to see a huge difference in Russian where ViT-B-16-SigLIP2__webli is like reasonable while for the others, it's very bad.

Anyway, so far I'm good with English. But I wonder if there is a reasonable OSS translator project where queries can be translated to a single language to simplify smart search. For example all queries might be translated do Croatian because its top score is 87.5% which beats everything else 🤣

[mertalev]

I see top model being different for the different ones and the accuracy being different. Especially I'm surprised to see a huge difference in Russian where ViT-B-16-SigLIP2__webli is like reasonable while for the others, it's very bad.

A general issue is that the models naturally see more content of major languages than less common languages, so they tend to prioritize learning major languages first and can get confused with other similar languages. Smaller models have less capacity to learn different languages, so they tend to struggle with this more. On the other hand, the NLLB models are the most consistent performers because they leave it to you to tell them what language the input is instead of guessing.

Also, some languages are tested on multiple datasets and others on only one. I think the difficulty varies by language even within a dataset, so it's probably more that Croatian had an easier test.