CLIP-based semantic image/text search

[mvcisback]

What is the problem you are trying to solve with this feature?

Sometimes I may look for an item under a name that doesn't match the current title. This could happen because I change the way I think about it, or because its a different user.

This happens both in retrieval (where was that thing?) as well as storage (where is this supposed to go?)

What is the solution you are proposing?

It would be nice to be able to use a CLIP embedding (https://github.com/mlfoundations/open_clip) of both the textual description/title and the asset images.

That way one could find a record by taking a picture or find images that match a text description.

The key primitive necessary is a vector database (with plugins to common ones like sqlite already existing or a special vector database such as chromadb)

What alternatives have you considered?

Tagging although this is fairly tedious.

Additional context

No response

Contributions

• I have searched through existing ideas in the discussions to check if this is a duplicate