You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: learn/ai_powered_search/image_search_with_multimodal_embeddings.mdx
+11-6Lines changed: 11 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -32,11 +32,11 @@ Much like other embedders, multimodal embedders must set their `source` to `rest
32
32
33
33
All multimodal embedders must contain an `indexingFragments` field and a `searchFragments` field. Fragments are sets of embeddings built out of specific parts of document data.
34
34
35
-
Fragments must follow the structure defined by your chosen model and provider.
35
+
Fragments must follow the structure defined by the REST API of your chosen provider.
36
36
37
37
### `indexingFragments`
38
38
39
-
Use `indexingFragments` to tell Meilisearch what data to use when generating document embeddings.
39
+
Use `indexingFragments` to tell Meilisearch how to send document data to the provider's API when generating document embeddings.
40
40
41
41
For example, when using VoyageAI's multimodal model, an indexing fragment might look like this:
42
42
@@ -67,11 +67,11 @@ For example, when using VoyageAI's multimodal model, an indexing fragment might
67
67
68
68
The example above requests Meilisearch to create two sets of embeddings during indexing: one for the textual description of an image, and another for the actual image.
69
69
70
-
Each fragment must have one field with a Liquid template where you interpolate document data present in `doc`. In `IMAGE_FRAGMENT_NAME`, that's `image_url` which outputs the plain URL string in the document field `poster_url`. In `TEXT_FRAGMENT_NAME`, `text` contains a longer string contextualizing two document fields, `title` and `description`.
70
+
Any JSON string value appearing in a fragment is handled as a Liquid template, where you interpolate document data present in `doc`. In `IMAGE_FRAGMENT_NAME`, that's `image_url` which outputs the plain URL string in the document field `poster_url`. In `TEXT_FRAGMENT_NAME`, `text` contains a longer string contextualizing two document fields, `title` and `description`.
71
71
72
72
### `searchFragments`
73
73
74
-
Use `searchFragments` to tell Meilisearch what data to use when converting user queries into embeddings:
74
+
Use `searchFragments` to tell Meilisearch how to send search query data to the chosen provider's REST API when converting them into embeddings:
75
75
76
76
```json
77
77
"searchFragments": {
@@ -98,10 +98,15 @@ Use `searchFragments` to tell Meilisearch what data to use when converting user
98
98
}
99
99
```
100
100
101
-
This configuration tells Meilisearch how to submit query data for vectorization.
101
+
In this example, two modes of search are configured:
102
+
103
+
1. A textual search based on the `q` parameter, which will be embedded as text
104
+
2. An image search based on [data url](https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Schemes/data) rebuilt from the `image.mime` and `image.data` field in the `media` field of the query
102
105
103
106
Search fragments have access to data present in the query parameters `media` and `q`.
104
107
108
+
Each semantic search query for this embedder should match exactly one search fragment of this embedder, so the fragments should each have at least one disambiguating field
109
+
105
110
### Complete embedder configuration
106
111
107
112
Your embedder should look similar to this example with all fragments and embedding provider data:
@@ -218,7 +223,7 @@ In most cases you will need a GUI interface that allows users to submit their im
218
223
219
224
With multimodal embedders you can:
220
225
221
-
1. Configure Meilisearch to vectorize both images and queries
226
+
1. Configure Meilisearch to embed both images and queries
0 commit comments