Skip to content

Commit 7bc3784

Browse files
natasha-moore-elasticnatasha-moore-elastic
committed
Anomaly detection results migration
1 parent 62af413 commit 7bc3784

File tree

1 file changed

+252
-1
lines changed

1 file changed

+252
-1
lines changed

‎deploy-manage/upgrade/prepare-to-upgrade.md‎

Lines changed: 252 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,255 @@
44

55
% Scope notes: Prerequisites and requirements
66

7-
⚠️ **This page is a work in progress.** ⚠️
7+
⚠️ **This page is a work in progress.** ⚠️
8+
9+
10+
## Anomaly detection results migration
11+
12+
The {{anomaly-detect}} result indices `.ml-anomalies-*` created in {{es}} 7.x must be either reindexed, marked read-only, or deleted before upgrading to 9.x.
13+
14+
**Reindexing**: While {{anomaly-detect}} results are being reindexed, jobs continue to run and process new data. However, you cannot completely delete an {{anomaly-job}} that stores results in this index until the reindexing is complete.
15+
16+
**Marking indices as read-only**: This is useful for large indexes that contain the results of only one or a few {{anomaly-jobs}}. If you delete these jobs later, you will not be able to create a new job with the same name.
17+
18+
**Deleting**: Delete jobs that are no longer needed in the {{ml-app}} app in {{kib}}. The result index is deleted when all jobs that store results in it have been deleted.
19+
20+
:::{dropdown} Which indices require attention?
21+
To identify indices that require action, use the [Deprecation info API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-migration-deprecations-1):
22+
23+
```
24+
GET /.ml-anomalies-*/_migration/deprecations
25+
```
26+
27+
The response contains the list of critical deprecation warnings in the `index_settings` section:
28+
29+
```json
30+
"index_settings": {
31+
".ml-anomalies-shared": [
32+
{
33+
"level": "critical",
34+
"message": "Index created before 8.0",
35+
"url": "https://ela.st/es-deprecation-8-reindex",
36+
"details": "This index was created with version 7.8.23 and is not compatible with 9.0. Reindex or remove the index before upgrading.",
37+
"resolve_during_rolling_upgrade": false
38+
}
39+
]
40+
}
41+
```
42+
:::
43+
44+
:::{dropdown} Reindexing anomaly result indices
45+
For an index with less than 10GB that contains results from multiple jobs that are still required, we recommend reindexing into a new format using UI. You can use the [Get index information API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-indices-1) to obtain the size of an index:
46+
47+
```
48+
GET _cat/indices/.ml-anomalies-custom-example?v&h=index,store.size
49+
```
50+
51+
The reindexing can be initiated in the {{kib}} Upgrade Assistant.
52+
53+
If an index size is greater than 10 GB, it is recommended to use the Reindex API. Reindexing consists of the following steps:
54+
55+
1. Set the original index to read-only.
56+
57+
```
58+
PUT .ml-anomalies-custom-example/_block/read_only
59+
```
60+
61+
2. Create a new index from the legacy index.
62+
63+
```
64+
POST _create_from/.ml-anomalies-custom-example/.reindexed-v9-ml-anomalies-custom-example
65+
```
66+
67+
3. Reindex documents. To accelerate the reindexing process, it is recommended that the number of replicas be set to `0` before the reindexing and then set back to the original number once it is completed.
68+
69+
1. Get the number of replicas.
70+
71+
```
72+
GET /.reindexed-v9-ml-anomalies-custom-example/_settings
73+
```
74+
75+
Note the number of replicas in the response. For example:
76+
77+
```json
78+
{
79+
".reindexed-v9-ml-anomalies-custom-example": {
80+
"settings": {
81+
"index": {
82+
"number_of_replicas": "1",
83+
"number_of_shards": "1"
84+
}
85+
}
86+
}
87+
}
88+
```
89+
90+
2. Set the number of replicas to `0.`
91+
92+
```json
93+
PUT /.reindexed-v9-ml-anomalies-custom-example/_settings
94+
{
95+
"index": {
96+
"number_of_replicas": 0
97+
}
98+
}
99+
```
100+
101+
3. Start the reindexing process in asynchronous mode.
102+
103+
```json
104+
POST _reindex?wait_for_completion=false
105+
{
106+
"source": {
107+
"index": ".ml-anomalies-custom-example"
108+
},
109+
"dest": {
110+
"index": ".reindexed-v9-ml-anomalies-custom-example"
111+
}
112+
}
113+
```
114+
115+
The response will contain a `task_id`. You can check when the task is completed using the following command:
116+
117+
```
118+
GET _tasks/<task_id>
119+
```
120+
121+
4. Set the number of replicas to the original number when the reindexing is finished.
122+
123+
```json
124+
PUT /.reindexed-v9-ml-anomalies-custom-example/_settings
125+
{
126+
"index": {
127+
"number_of_replicas": "<original_number_of_replicas>"
128+
}
129+
}
130+
```
131+
132+
4. Get the aliases the original index is pointing to.
133+
134+
```
135+
GET .ml-anomalies-custom-example/_alias
136+
```
137+
138+
The response may contain multiple aliases if the results of multiple jobs are stored in the same index.
139+
140+
```json
141+
{
142+
".ml-anomalies-custom-example": {
143+
"aliases": {
144+
".ml-anomalies-example1": {
145+
"filter": {
146+
"term": {
147+
"job_id": {
148+
"value": "example1"
149+
}
150+
}
151+
},
152+
"is_hidden": true
153+
},
154+
".ml-anomalies-example2": {
155+
"filter": {
156+
"term": {
157+
"job_id": {
158+
"value": "example2"
159+
}
160+
}
161+
},
162+
"is_hidden": true
163+
}
164+
}
165+
}
166+
}
167+
```
168+
169+
5. Now you can reassign the aliases to the new index and delete the original index in one step. Note that when adding the new index to the aliases, you must use the same `filter` and `is_hidden` parameters as for the original index.
170+
171+
```json
172+
POST _aliases
173+
{
174+
"actions": [
175+
{
176+
"add": {
177+
"index": ".reindexed-v9-ml-anomalies-custom-example",
178+
"alias": ".ml-anomalies-example1",
179+
"filter": {
180+
"term": {
181+
"job_id": {
182+
"value": "example1"
183+
}
184+
}
185+
},
186+
"is_hidden": true
187+
}
188+
},
189+
{
190+
"add": {
191+
"index": ".reindexed-v9-ml-anomalies-custom-example",
192+
"alias": ".ml-anomalies-example2",
193+
"filter": {
194+
"term": {
195+
"job_id": {
196+
"value": "example2"
197+
}
198+
}
199+
},
200+
"is_hidden": true
201+
}
202+
},
203+
{
204+
"remove": {
205+
"index": ".ml-anomalies-custom-example",
206+
"aliases": ".ml-anomalies-*"
207+
}
208+
},
209+
{
210+
"remove_index": {
211+
"index": ".ml-anomalies-custom-example"
212+
}
213+
},
214+
{
215+
"add": {
216+
"index": ".reindexed-v9-ml-anomalies-custom-example",
217+
"alias": ".ml-anomalies-custom-example",
218+
"is_hidden": true
219+
}
220+
}
221+
]
222+
}
223+
```
224+
:::
225+
226+
227+
:::{dropdown} Marking anomaly result indices as read-only
228+
Legacy indices created in {{es}} 7.x can be made read-only and supported in {{es}} 9.x. Making an index with a large amount of historical results read-only allows for a quick migration to the next major release, since you don’t have to wait for the data to be reindexed into the new format. However, it has the limitation that even after deleting an {{anomaly-job}}, the historical results associated with this job are not completely deleted. Therefore, the system will prevent you from creating a new job with the same name.
229+
230+
To set the index as read-only, add the write block to the index:
231+
232+
```
233+
PUT .ml-anomalies-custom-example/_block/write
234+
```
235+
236+
Indices created in {{es}} 7.x that have a write block will not raise a critical deprecation warning.
237+
:::
238+
239+
:::{dropdown} Deleting anomaly result indices
240+
If an index contains results of the jobs that are no longer required. To list all jobs that stored results in an index, use the terms aggregation:
241+
242+
```json
243+
GET .ml-anomalies-custom-example/_search
244+
{
245+
"size": 0,
246+
"aggs": {
247+
"job_ids": {
248+
"terms": {
249+
"field": "job_id",
250+
"size": 100
251+
}
252+
}
253+
}
254+
}
255+
```
256+
257+
The jobs can be deleted in the UI. After the last job is deleted, the index will be deleted as well.
258+
:::

0 commit comments

Comments
 (0)