Skip to content

Commit bb95398

Browse files
guimachiavellidureuill
guimachiavelli
and
dureuill
authored
v1.13: Add remote federated search documentation (#3144)
--------- Co-authored-by: Louis Dureuil <[email protected]>
1 parent be78a40 commit bb95398

File tree

10 files changed

+487
-37
lines changed

10 files changed

+487
-37
lines changed

‎.code-samples.meilisearch.yaml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1444,3 +1444,46 @@ experimental_post_logs_stream_1: |-
14441444
experimental_delete_logs_stream_1: |-
14451445
curl \
14461446
-X DELETE MEILISEARCH_URL/logs/stream
1447+
multi_search_remote_federated_1: |-
1448+
curl \
1449+
-X POST 'MEILISEARCH_URL/multi-search' \
1450+
-H 'Content-Type: application/json' \
1451+
--data-binary '{
1452+
"federation": {},
1453+
"queries": [
1454+
{
1455+
"indexUid": "movies",
1456+
"q": "batman",
1457+
"federationOptions": {
1458+
"remote": "ms-00"
1459+
}
1460+
},
1461+
{
1462+
"indexUid": "movies",
1463+
"q": "batman",
1464+
"federationOptions": {
1465+
"remote": "ms-01"
1466+
}
1467+
}
1468+
]
1469+
}'
1470+
get_network_1: |-
1471+
curl \
1472+
-X GET 'MEILISEARCH_URL/network'
1473+
update_network_1: |-
1474+
curl \
1475+
-X PATCH 'MEILISEARCH_URL/network' \
1476+
-H 'Content-Type: application/json' \
1477+
--data-binary '{
1478+
"self": "ms-00",
1479+
"remotes": {
1480+
"ms-00": {
1481+
"url": "http://INSTANCE_URL",
1482+
"searchApiKey": "INSTANCE_API_KEY"
1483+
},
1484+
"ms-01": {
1485+
"url": "http://ANOTHER_INSTANCE_URL",
1486+
"searchApiKey": "ANOTHER_INSTANCE_API_KEY"
1487+
}
1488+
}
1489+
}'

‎config/sidebar-learn.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -267,6 +267,11 @@
267267
"label": "Using multi-search to perform a federated search",
268268
"slug": "performing_federated_search"
269269
},
270+
{
271+
"source": "learn/multi_search/implement_sharding.mdx",
272+
"label": "Implement sharding with remote federated search",
273+
"slug": "implement_sharding"
274+
},
270275
{
271276
"source": "learn/multi_search/multi_search_vs_federated_search.mdx",
272277
"label": "Differences between multi-search and federated search",

‎config/sidebar-reference.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,11 @@
2828
"label": "Multi-search",
2929
"slug": "multi_search"
3030
},
31+
{
32+
"source": "reference/api/network.mdx",
33+
"label": "Network",
34+
"slug": "network"
35+
},
3136
{
3237
"source": "reference/api/similar.mdx",
3338
"label": "Similar documents",

‎learn/multi_search/implement_sharding.mdx‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
---
2+
title: Implement sharding with remote federated search — Meilisearch documentation
3+
description: This guide walks you through implementing a sharding strategy by activating the `/network` route, configuring the network object, and performing remote federated searches.
4+
---
5+
6+
# Implement sharding with remote federated search <NoticeTag type="experimental" label="experimental" />
7+
8+
Sharding is the process of splitting an index containing many documents into multiple smaller indexes, often called shards. This horizontal scaling technique is useful when handling large databases. In Meilisearch, the best way to implement a sharding strategy is to use remote federated search.
9+
10+
This guide walks you through activating the `/network` route, configuring the network object, and performing remote federated searches.
11+
12+
<Capsule intent="tip" title="Configuring multiple instances">
13+
To minimize issues and limit unexpected behavior, instance, network, and index configuration should be identical for all shards. This guide describes the individual steps you must take on a single instance and assumes you will replicate them across all instances.
14+
</Capsule>
15+
16+
## Prerequisites
17+
18+
- Multiple Meilisearch projects (instances) running Meilisearch >=v1.13
19+
20+
## Activate the `/network` endpoint
21+
22+
### Meilisearch Cloud
23+
24+
If you are using Meilisearch Cloud, contact support to enable this feature in your projects.
25+
26+
### Self-hosting
27+
28+
Use the `/experimental-features` route to enable `network`:
29+
30+
```sh
31+
curl \
32+
-X PATCH 'MEILISEARCH_URL/experimental-features/' \
33+
-H 'Content-Type: application/json' \
34+
--data-binary '{
35+
"network": true
36+
}'
37+
```
38+
39+
Meilisearch should respond immediately, confirming the route is now accessible. Repeat this process for all instances.
40+
41+
## Configuring the network object
42+
43+
Next, you must configure the network object. It consists of the following fields:
44+
45+
- `remotes`: defines a list with the required information to access each remote instance
46+
- `self`: specifies which of the configured `remotes` corresponds to the current instance
47+
48+
### Setting up the list of remotes
49+
50+
Use the `/network` route to configure the `remotes` field of the network object. `remotes` should be an object containing one or more objects. Each one of the nested objects should consist of the name of each instance, associated with its URL and an API key with search permission:
51+
52+
```sh
53+
curl \
54+
-X PATCH 'MEILISEARCH_URL/network' \
55+
-H 'Content-Type: application/json' \
56+
--data-binary '{
57+
"remotes": {
58+
"REMOTE_NAME_1": {
59+
"url": "INSTANCE_URL_1",
60+
"searchApiKey": "SEARCH_API_KEY_1"
61+
},
62+
"REMOTE_NAME_2": {
63+
"url": "INSTANCE_URL_2",
64+
"searchApiKey": "SEARCH_API_KEY_2"
65+
},
66+
"REMOTE_NAME_3": {
67+
"url": "INSTANCE_URL_3",
68+
"searchApiKey": "SEARCH_API_KEY_3"
69+
},
70+
…
71+
}
72+
}'
73+
```
74+
75+
Configure the entire set of remote instances in your sharded database, making sure to send the same remotes to each instance.
76+
77+
### Specify the name of the current instance
78+
79+
Now all instances share the same list of remotes, set the `self` field to specify which of the remotes corresponds to the current instance:
80+
81+
```sh
82+
curl \
83+
-X PATCH 'MEILISEARCH_URL/network' \
84+
-H 'Content-Type: application/json' \
85+
--data-binary '{
86+
"self": "REMOTE_NAME_1"
87+
}'
88+
```
89+
90+
Meilisearch processes searches on the remote that corresponds to `self` locally instead of making a remote request.
91+
92+
### Adding or removing an instance
93+
94+
Changing the topology of the network involves moving some documents from an instance to another, depending on your hashing scheme.
95+
96+
As Meilisearch does not provide atomicity across multiple instances, you will need to either:
97+
98+
1. accept search downtime while migrating documents
99+
2. accept some documents will not appear in search results during the migration
100+
3. accept some duplicate documents may appear in search results during the migration
101+
102+
#### Reducing downtime
103+
104+
If your disk space allows, you can reduce the downtime by applying the following algorithm:
105+
106+
1. Create a new temporary index in each remote instance
107+
2. Compute the new instance for each document
108+
3. Send the documents to the temporary index of their new instance
109+
4. Once Meilisearch has copied all documents to their instance of destination, swap the new index with the previously used index
110+
5. Delete the temporary index after the swap
111+
6. Update network configuration and search queries across all instances
112+
113+
## Create indexes and add documents
114+
115+
Create the same empty indexes with the same settings on all instances. Keeping the settings and indexes in sync is important to avoid errors and unexpected behavior, though not strictly required.
116+
117+
Distribute your documents across all instances. Do not send the same document to multiple instances as this may lead to duplicate search results. Similarly, you should ensure all future versions of a document are sent to the same instance. Meilisearch recommends you hash their primary key using [rendezvous hashing](https://en.wikipedia.org/wiki/Rendezvous_hashing).
118+
119+
### Updating index settings
120+
121+
Changing settings in a sharded database is not fundamentally different from changing settings on a single Meilisearch instance. If the update enables a feature, such as setting filterable attributes, wait until all changes have been processed before using the `filter` search parameter in a query. Likewise, if an update disables a feature, first remove it from your search requests, then update your settings.
122+
123+
## Perform a search
124+
125+
Send your federated search request containing one query per instance:
126+
127+
<CodeSamples id="multi_search_remote_federated_1" />
128+
129+
If all instances share the same network configuration, you can send the search request to any instance. Having `"remote": "ms-00"` appear in the list of queries on the instance of that name will not cause an actual proxy search thanks to `network.self`.

‎learn/resources/experimental_features_overview.mdx‎

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -42,13 +42,15 @@ Activating or deactivating experimental features this way does not require you t
4242

4343
## Current experimental features
4444

45-
| Name | Description | How to configure |
46-
|----------------------------------------------------------------------------------|------------------------------------------------------------|---------------------------------------------------|
47-
| [Limit task batch size](/learn/self_hosted/configure_meilisearch_at_launch) | Limits number of tasks processed in a single batch | At launch with a CLI flag or environment variable |
48-
| [Log customization](/reference/api/logs) | Customize log output and set up log streams | At launch with a CLI flag or environment variable, during runtime with the API route |
49-
| [Metrics API](/reference/api/metrics) | Exposes Prometheus-compatible analytics data | At launch with a CLI flag or environment variable, during runtime with the API route |
50-
| [Reduce indexing memory usage](/learn/self_hosted/configure_meilisearch_at_launch) | Optimizes indexing performance | At launch with a CLI flag or environment variable |
51-
| [Replication parameters](/learn/self_hosted/configure_meilisearch_at_launch) | Alters task processing for clustering compatibility | At launch with a CLI flag or environment variable |
52-
| [Search queue size](/learn/self_hosted/configure_meilisearch_at_launch) | Configure maximum number of concurrent search requests | At launch with a CLI flag or environment variable |
53-
| [`CONTAINS` filter operator](/learn/filtering_and_sorting/filter_expression_reference#contains) | Enables usage of `CONTAINS` with the `filter` search parameter | During runtime with the API route |
54-
| [Edit documents with function](/reference/api/documents#update-documents-with-function) | Use a RHAI function to edit documents directly in the Meilisearch database | During runtime with the API route |
45+
| Name | Description | How to configure |
46+
| ----------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------- |
47+
| [Limit task batch size](/learn/self_hosted/configure_meilisearch_at_launch) | Limits number of tasks processed in a single batch | CLI flag or environment variable |
48+
| [Log customization](/reference/api/logs) | Customize log output and set up log streams | CLI flag or environment variable, API route |
49+
| [Metrics API](/reference/api/metrics) | Exposes Prometheus-compatible analytics data | CLI flag or environment variable, API route |
50+
| [Reduce indexing memory usage](/learn/self_hosted/configure_meilisearch_at_launch) | Optimizes indexing performance | CLI flag or environment variable |
51+
| [Replication parameters](/learn/self_hosted/configure_meilisearch_at_launch) | Alters task processing for clustering compatibility | CLI flag or environment variable |
52+
| [Search queue size](/learn/self_hosted/configure_meilisearch_at_launch) | Configure maximum number of concurrent search requests | CLI flag or environment variable |
53+
| [`CONTAINS` filter operator](/learn/filtering_and_sorting/filter_expression_reference#contains) | Enables usage of `CONTAINS` with the `filter` search parameter | API route |
54+
| [Edit documents with function](/reference/api/documents#update-documents-with-function) | Use a RHAI function to edit documents directly in the Meilisearch database | API route |
55+
| [`/network` route](/reference/api/network) | Enable `/network` route | API route |
56+
| [Dumpless upgrade](/learn/self_hosted/configure_meilisearch_at_launch#dumpless-upgrade) | Upgrade Meilisearch without generating a dump | API route |

‎reference/api/experimental_features.mdx‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,16 +20,18 @@ The experimental API route is not compatible with all experimental features. Con
2020
"metrics": false,
2121
"logsRoute": true,
2222
"containsFilter": false,
23-
"editDocumentsByFunction": false
23+
"editDocumentsByFunction": false,
24+
"network": false
2425
}
2526
```
2627

27-
| Name | Type | Description |
28+
| Name | Type | Description |
2829
| :---------------------------- | :------ | :--------------------------------------------- |
2930
| **`metrics`** | Boolean | `true` if feature is active, `false` otherwise |
3031
| **`logsRoute`** | Boolean | `true` if feature is active, `false` otherwise |
3132
| **`containsFilter`** | Boolean | `true` if feature is active, `false` otherwise |
3233
| **`editDocumentsByFunction`** | Boolean | `true` if feature is active, `false` otherwise |
34+
| **`network`** | Boolean | `true` if feature is active, `false` otherwise |
3335

3436
## Get all experimental features
3537

@@ -48,7 +50,8 @@ Get a list of all experimental features that can be activated via the `/experime
4850
"metrics": false,
4951
"logsRoute": true,
5052
"containsFilter": false,
51-
"editDocumentsByFunction": false
53+
"editDocumentsByFunction": false,
54+
"network": false
5255
}
5356
```
5457

@@ -75,6 +78,7 @@ Setting a field to `null` leaves its value unchanged.
7578
"metrics": false,
7679
"logsRoute": true,
7780
"containsFilter": false,
78-
"editDocumentsByFunction": false
81+
"editDocumentsByFunction": false,
82+
"network": false
7983
}
8084
```

‎reference/api/keys.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,8 @@ For security reasons, we do not recommend creating keys that can perform all act
101101
| **`keys.create`** | Provides access to the [create key](#create-a-key) endpoint |
102102
| **`keys.update`** | Provides access to the [update key](#update-a-key) endpoint |
103103
| **`keys.delete`** | Provides access to the [delete key](#delete-a-key) endpoint |
104+
| **`network.get`** | Provides access to the [get the network object](/reference/api/network#get-the-network-object) endpoint |
105+
| **`network.update`** | Provides access to the [update the network object](/reference/api/network#update-the-network-object) endpoint |
104106

105107
### `indexes`
106108

0 commit comments

Comments
 (0)