Skip to content

Commit a535da3

Browse files
szabosteveszabosteve
committed
[DOCS] Adds anomaly detection info to migration guide.
1 parent b94a20e commit a535da3

File tree

1 file changed

+318
-0
lines changed

1 file changed

+318
-0
lines changed

‎docs/reference/migration/migrate_9_0.asciidoc‎

Lines changed: 318 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,324 @@ Lucene 10 ships with an updated Korean dictionary (mecab-ko-dic-2.1.1). For det
7373
The change is small and should generally provide better analysis results. Existing indices for full-text use cases should be reindexed though.
7474
====
7575

76+
[discrete]
77+
[[breaking_90_anomaly_detection_results]]
78+
==== Anomaly detection results
79+
80+
[[prepare_anomaly_detection_result_indices_for_upgrade]]
81+
.Preparing anomaly detection result indices for upgrade
82+
[%collapsible]
83+
====
84+
The {anomaly-detect} result indices `.ml-anomalies-*` created in {es} 7.x must be either reindexed, marked read-only, or deleted before upgrading to 9.x.
85+
86+
To identify indices that require action, use the <<migration-api-deprecation,Deprecation info API>>:
87+
88+
[source,console]
89+
------------------------------------------------------------
90+
GET /.ml-anomalies-*/_migration/deprecations
91+
------------------------------------------------------------
92+
// TEST[skip:TBD]
93+
94+
The response contains the list of critical deprecation warnings in the `index_settings` section:
95+
96+
[source,console-result]
97+
------------------------------------------------------------
98+
...
99+
"index_settings" : {
100+
".ml-anomalies-shared" : [
101+
{
102+
"level" : "critical",
103+
"message" : "Index created before 8.0",
104+
"url" : "https://ela.st/es-deprecation-8-reindex",
105+
"details" : "This index was created with version 7.8.23 and is not compatible with 9.0. Reindex or remove the index before upgrading.",
106+
"resolve_during_rolling_upgrade" : false
107+
}
108+
]
109+
},
110+
...
111+
------------------------------------------------------------
112+
// NOTCONSOLE
113+
114+
**Reindexing**: While anomaly detection results are being reindexed, jobs continue to run and process new data.
115+
However, you cannot completely delete an {anomaly-job} that stores results in this index until the reindexing is complete.
116+
117+
**Marking indices as read-only**: This is useful for large indexes that contain the results of only one or a few {anomaly-jobs}.
118+
If you delete these jobs later, you will not be able to create a new job with the same name.
119+
120+
**Deleting**: Delete jobs that are no longer needed in the {ml-app} in {kib}.
121+
The result index is deleted when all jobs that store results in it have been deleted.
122+
====
123+
124+
[[reindex_anomaly_result_index_read_only]]
125+
.Reindexing anomaly result indices
126+
[%collapsible]
127+
====
128+
For an index with less than 10GB that contains results from multiple jobs that are still required, we recommend reindexing into a new format using UI.
129+
You can use the <<cat-indices>> to obtain the size of an index:
130+
131+
[source,console]
132+
------------------------------------------------------------
133+
GET _cat/indices/.ml-anomalies-custom-example?v&h=index,store.size
134+
------------------------------------------------------------
135+
// TEST[skip:TBD]
136+
137+
The reindexing can be initiated in the Kibana Upgrade Assistant.
138+
139+
If an index size is greater than 10 GB it is recommended to use the Reindex API.
140+
Reindexing consists of the following steps:
141+
142+
. Set the original index to read-only.
143+
+
144+
--
145+
[source,console]
146+
------------------------------------------------------------
147+
PUT .ml-anomalies-custom-example/_block/read_only
148+
------------------------------------------------------------
149+
// TEST[skip:TBD]
150+
--
151+
152+
. Create a new index from the legacy index.
153+
+
154+
--
155+
[source,console]
156+
------------------------------------------------------------
157+
POST _create_from/.ml-anomalies-custom-example/.reindexed-v9-ml-anomalies-custom-example
158+
------------------------------------------------------------
159+
// TEST[skip:TBD]
160+
--
161+
162+
. Reindex documents.
163+
To accelerate the reindexing process, it is recommended that the number of replicas be set to `0` before the reindexing and then set back to the original number once it is completed.
164+
.. Get the number of replicas.
165+
+
166+
--
167+
[source,console]
168+
------------------------------------------------------------
169+
GET /.reindexed-v9-ml-anomalies-custom-example/_settings
170+
------------------------------------------------------------
171+
// TEST[skip:TBD]
172+
Note the number of replicas in the response. For example:
173+
[source,console-result]
174+
------------------------------------------------------------
175+
{
176+
".reindexed-v9-ml-anomalies-custom-example": {
177+
"settings": {
178+
"index": {
179+
"number_of_replicas": "1",
180+
"number_of_shards": "1"
181+
}
182+
}
183+
}
184+
}
185+
------------------------------------------------------------
186+
// NOTCONSOLE
187+
--
188+
.. Set the number of replicas to `0`.
189+
+
190+
--
191+
[source,console]
192+
------------------------------------------------------------
193+
PUT /.reindexed-v9-ml-anomalies-custom-example/_settings
194+
{
195+
"index": {
196+
"number_of_replicas": 0
197+
}
198+
}
199+
------------------------------------------------------------
200+
// TEST[skip:TBD]
201+
--
202+
.. Start the reindexing process in asynchronous mode.
203+
+
204+
--
205+
[source,console]
206+
------------------------------------------------------------
207+
POST _reindex?wait_for_completion=false
208+
{
209+
"source": {
210+
"index": ".ml-anomalies-custom-example"
211+
},
212+
"dest": {
213+
"index": ".reindexed-v9-ml-anomalies-custom-example"
214+
}
215+
}
216+
------------------------------------------------------------
217+
// TEST[skip:TBD]
218+
The response will contain a task_id. You can check when the task is completed using the following command:
219+
[source,console]
220+
------------------------------------------------------------
221+
GET _tasks/<task_id>
222+
------------------------------------------------------------
223+
// TEST[skip:TBD]
224+
--
225+
.. Set the number of replicas to the original number when the reindexing is finished.
226+
+
227+
--
228+
[source,console]
229+
------------------------------------------------------------
230+
PUT /.reindexed-v9-ml-anomalies-custom-example/_settings
231+
{
232+
"index": {
233+
"number_of_replicas": <original_number_of_replicas>
234+
}
235+
}
236+
------------------------------------------------------------
237+
// TEST[skip:TBD]
238+
--
239+
240+
. Get the aliases the original index is pointing to.
241+
+
242+
--
243+
[source,console]
244+
------------------------------------------------------------
245+
GET .ml-anomalies-custom-example/_alias
246+
------------------------------------------------------------
247+
// TEST[skip:TBD]
248+
249+
The response may contain multiple aliases if the results of multiple jobs are stored in the same index.
250+
251+
[source,console-result]
252+
------------------------------------------------------------
253+
{
254+
".ml-anomalies-custom-example": {
255+
"aliases": {
256+
".ml-anomalies-example1": {
257+
"filter": {
258+
"term": {
259+
"job_id": {
260+
"value": "example1"
261+
}
262+
}
263+
},
264+
"is_hidden": true
265+
},
266+
".ml-anomalies-example2": {
267+
"filter": {
268+
"term": {
269+
"job_id": {
270+
"value": "example2"
271+
}
272+
}
273+
},
274+
"is_hidden": true
275+
}
276+
}
277+
}
278+
}
279+
------------------------------------------------------------
280+
// NOTCONSOLE
281+
--
282+
283+
. Now you can reassign the aliases to the new index and delete the original index in one step.
284+
Note that when adding the new index to the aliases, you must use the same filter and is_hidden parameters as for the original index.
285+
+
286+
--
287+
[source,console]
288+
------------------------------------------------------------
289+
POST _aliases
290+
{
291+
"actions": [
292+
{
293+
"add": {
294+
"index": ".reindexed-v9-ml-anomalies-custom-example",
295+
"alias": ".ml-anomalies-example1",
296+
"filter": {
297+
"term": {
298+
"job_id": {
299+
"value": "example1"
300+
}
301+
}
302+
},
303+
"is_hidden": true
304+
}
305+
},
306+
{
307+
"add": {
308+
"index": ".reindexed-v9-ml-anomalies-custom-example",
309+
"alias": ".ml-anomalies-example2",
310+
"filter": {
311+
"term": {
312+
"job_id": {
313+
"value": "example2"
314+
}
315+
}
316+
},
317+
"is_hidden": true
318+
}
319+
},
320+
{
321+
"remove": {
322+
"index": ".ml-anomalies-custom-example",
323+
"aliases": ".ml-anomalies-*"
324+
}
325+
},
326+
{
327+
"remove_index": {
328+
"index": ".ml-anomalies-custom-example"
329+
}
330+
},
331+
{
332+
"add": {
333+
"index": ".reindexed-v9-ml-anomalies-custom-example",
334+
"alias": ".ml-anomalies-custom-example",
335+
"is_hidden": true
336+
}
337+
}
338+
]
339+
}
340+
------------------------------------------------------------
341+
// TEST[skip:TBD]
342+
--
343+
====
344+
345+
[[delete_anomaly_result_index]]
346+
.Deleting anomaly result indices
347+
[%collapsible]
348+
====
349+
If an index contains results of the jobs that are no longer required.
350+
To list all jobs that stored results in an index, use the terms aggregation:
351+
352+
[source,console]
353+
------------------------------------------------------------
354+
GET .ml-anomalies-custom-example/_search
355+
{
356+
"size": 0,
357+
"aggs": {
358+
"job_ids": {
359+
"terms": {
360+
"field": "job_id",
361+
"size": 100
362+
}
363+
}
364+
}
365+
}
366+
------------------------------------------------------------
367+
// TEST[skip:TBD]
368+
369+
The jobs can be deleted in the UI.
370+
After the last job is deleted, the index will be deleted as well.
371+
====
372+
373+
[[mark_anomaly_result_index_read_only]]
374+
.Marking anomaly result indices as read-only
375+
[%collapsible]
376+
====
377+
Legacy indexes created in {es} 7.x can be made read-only and supported in {es} 9.x.
378+
Making an index with a large amount of historical results read-only allows for a quick migration to the next major release, since you don't have to wait for the data to be reindexed into the new format.
379+
However, it has the limitation that even after deleting an {anomaly-job}, the historical results associated with this job are not completely deleted.
380+
Therefore, the system will prevent you from creating a new job with the same name.
381+
382+
To set the index as read-only, add the `write` block to the index:
383+
384+
[source,console]
385+
------------------------------------------------------------
386+
PUT .ml-anomalies-custom-example/_block/write
387+
------------------------------------------------------------
388+
// TEST[skip:TBD]
389+
390+
Indices created in {es} 7.x that have a `write` block will not raise a critical deprecation warning.
391+
====
392+
393+
76394
[discrete]
77395
[[breaking_90_cluster_and_node_setting_changes]]
78396
==== Cluster and node setting changes

0 commit comments

Comments
 (0)