You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: learn/multi_search/implement_sharding.mdx
+39-12Lines changed: 39 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,6 +11,10 @@ Sharding is the process of splitting an index containing many documents into mul
11
11
12
12
This guide walks you through activating the `/network` route, configuring the network object, and performing remote federated searches.
13
13
14
+
<NoticeTaglabel="Enterprise Edition">
15
+
Sharding is an Enterprise Edition feature. You are free to use for evaluation purposes. Please [reach out to us](mailto:[email protected]) before using it in production.
16
+
</NoticeTag>
17
+
14
18
<Tip>
15
19
## Configuring multiple instances
16
20
@@ -19,7 +23,7 @@ To minimize issues and limit unexpected behavior, instance, network, and index c
@@ -48,6 +52,7 @@ Next, you must configure the network object. It consists of the following fields
48
52
49
53
-`remotes`: defines a list with the required information to access each remote instance
50
54
-`self`: specifies which of the configured `remotes` corresponds to the current instance
55
+
-`sharding`: whether to use sharding.
51
56
52
57
### Setting up the list of remotes
53
58
@@ -93,32 +98,54 @@ curl \
93
98
94
99
Meilisearch processes searches on the remote that corresponds to `self` locally instead of making a remote request.
95
100
101
+
### Enabling sharding
102
+
103
+
Finally enable the automatic sharding of documents by Meilisearch on all instances:
104
+
105
+
```sh
106
+
curl \
107
+
-X PATCH 'MEILISEARCH_URL/network' \
108
+
-H 'Content-Type: application/json' \
109
+
--data-binary '{
110
+
"sharding": true
111
+
}'
112
+
```
113
+
96
114
### Adding or removing an instance
97
115
98
116
Changing the topology of the network involves moving some documents from an instance to another, depending on your hashing scheme.
99
117
100
118
As Meilisearch does not provide atomicity across multiple instances, you will need to either:
101
119
102
-
1. accept search downtime while migrating documents
103
-
2. accept some documents will not appear in search results during the migration
120
+
1. accept search downtime while migrating documents
121
+
2. accept some documents will not appear in search results during the migration
104
122
3. accept some duplicate documents may appear in search results during the migration
105
123
106
124
#### Reducing downtime
107
125
108
126
If your disk space allows, you can reduce the downtime by applying the following algorithm:
109
127
110
-
1. Create a new temporary index in each remote instance
111
-
2. Compute the new instance for each document
112
-
3. Send the documents to the temporary index of their new instance
113
-
4. Once Meilisearch has copied all documents to their instance of destination, swap the new index with the previously used index
114
-
5. Delete the temporary index after the swap
115
-
6. Update network configuration and search queries across all instances
128
+
1. Create a new temporary index in each remote instance
129
+
2. Compute the new instance for each document
130
+
3. Send the documents to the temporary index of their new instance
131
+
4. Once Meilisearch has copied all documents to their instance of destination, swap the new index with the previously used index
132
+
5. Delete the temporary index after the swap
133
+
6. Update network configuration and search queries across all instances
134
+
135
+
## Create indexes
136
+
137
+
Create the same empty indexes with the same settings on all instances.
138
+
Keeping the settings and indexes in sync is important to avoid errors and unexpected behavior, though not strictly required.
139
+
140
+
## Add documents
141
+
142
+
Send documents to any instance to have them replicated to the other instances.
116
143
117
-
## Create indexes and add documents
144
+
Each instance will index the documents they are responsible of and ignore the others.
118
145
119
-
Create the same empty indexes with the same settings on all instances. Keeping the settings and indexes in sync is important to avoid errors and unexpected behavior, though not strictly required.
146
+
You *may* send send the same document to multiple instances, the task will be replicated to all instances, and only the instance responsible for the document will index it.
120
147
121
-
Distribute your documents across all instances. Do not send the same document to multiple instances as this may lead to duplicate search results. Similarly, you should ensure all future versions of a document are sent to the same instance. Meilisearch recommends you hash their primary key using [rendezvous hashing](https://en.wikipedia.org/wiki/Rendezvous_hashing).
148
+
Similarly, you may send any future versions of any document to any instance and only the correct instance will process that document.
0 commit comments