Skip to content

Commit b925b0c

Browse files
szabostevevaleriy42
szabosteve
and
valeriy42
authored
[DOCS] Adds anomaly detection info to migration guide (#121015)
Co-authored-by: Valeriy Khakhutskyy <[email protected]>
1 parent b2cc9d9 commit b925b0c

File tree

1 file changed

+318
-0
lines changed

1 file changed

+318
-0
lines changed

‎docs/reference/migration/migrate_9_0.asciidoc‎

Lines changed: 318 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ Lucene 10 ships with an updated Korean dictionary (mecab-ko-dic-2.1.1). For det
7373
The change is small and should generally provide better analysis results. Existing indices for full-text use cases should be reindexed though.
7474
====
7575

76+
7677
[discrete]
7778
[[breaking_90_cluster_and_node_setting_changes]]
7879
==== Cluster and node setting changes
@@ -318,3 +319,320 @@ The `elser` service of the inference API will be removed in an upcoming release.
318319
In the current version there is no impact. In a future version, users of the `elser` service will no longer be able to use it, and will be required to use the `elasticsearch` service to access elser through the inference API.
319320
====
320321

322+
[discrete]
323+
[[breaking_90_anomaly_detection_results]]
324+
=== Anomaly detection results migration
325+
326+
The {anomaly-detect} result indices `.ml-anomalies-*` created in {es} 7.x must be either reindexed, marked read-only, or deleted before upgrading to 9.x.
327+
328+
**Reindexing**: While anomaly detection results are being reindexed, jobs continue to run and process new data.
329+
However, you cannot completely delete an {anomaly-job} that stores results in this index until the reindexing is complete.
330+
331+
**Marking indices as read-only**: This is useful for large indexes that contain the results of only one or a few {anomaly-jobs}.
332+
If you delete these jobs later, you will not be able to create a new job with the same name.
333+
334+
**Deleting**: Delete jobs that are no longer needed in the {ml-app} in {kib}.
335+
The result index is deleted when all jobs that store results in it have been deleted.
336+
337+
[[which_indices_require_attention]]
338+
.Which indices require attention?
339+
[%collapsible]
340+
====
341+
342+
To identify indices that require action, use the <<migration-api-deprecation,Deprecation info API>>:
343+
344+
[source,console]
345+
------------------------------------------------------------
346+
GET /.ml-anomalies-*/_migration/deprecations
347+
------------------------------------------------------------
348+
// TEST[skip:TBD]
349+
350+
The response contains the list of critical deprecation warnings in the `index_settings` section:
351+
352+
[source,console-result]
353+
------------------------------------------------------------
354+
"index_settings": {
355+
".ml-anomalies-shared": [
356+
{
357+
"level": "critical",
358+
"message": "Index created before 8.0",
359+
"url": "https://ela.st/es-deprecation-8-reindex",
360+
"details": "This index was created with version 7.8.23 and is not compatible with 9.0. Reindex or remove the index before upgrading.",
361+
"resolve_during_rolling_upgrade": false
362+
}
363+
]
364+
}
365+
------------------------------------------------------------
366+
// NOTCONSOLE
367+
368+
369+
====
370+
371+
[[reindex_anomaly_result_index]]
372+
.Reindexing anomaly result indices
373+
[%collapsible]
374+
====
375+
For an index with less than 10GB that contains results from multiple jobs that are still required, we recommend reindexing into a new format using UI.
376+
You can use the <<cat-indices>> to obtain the size of an index:
377+
378+
[source,console]
379+
------------------------------------------------------------
380+
GET _cat/indices/.ml-anomalies-custom-example?v&h=index,store.size
381+
------------------------------------------------------------
382+
// TEST[skip:TBD]
383+
384+
The reindexing can be initiated in the Kibana Upgrade Assistant.
385+
386+
If an index size is greater than 10 GB it is recommended to use the Reindex API.
387+
Reindexing consists of the following steps:
388+
389+
. Set the original index to read-only.
390+
+
391+
--
392+
[source,console]
393+
------------------------------------------------------------
394+
PUT .ml-anomalies-custom-example/_block/read_only
395+
------------------------------------------------------------
396+
// TEST[skip:TBD]
397+
--
398+
399+
. Create a new index from the legacy index.
400+
+
401+
--
402+
[source,console]
403+
------------------------------------------------------------
404+
POST _create_from/.ml-anomalies-custom-example/.reindexed-v9-ml-anomalies-custom-example
405+
------------------------------------------------------------
406+
// TEST[skip:TBD]
407+
--
408+
409+
. Reindex documents.
410+
To accelerate the reindexing process, it is recommended that the number of replicas be set to `0` before the reindexing and then set back to the original number once it is completed.
411+
.. Get the number of replicas.
412+
+
413+
--
414+
[source,console]
415+
------------------------------------------------------------
416+
GET /.reindexed-v9-ml-anomalies-custom-example/_settings
417+
------------------------------------------------------------
418+
// TEST[skip:TBD]
419+
Note the number of replicas in the response. For example:
420+
[source,console-result]
421+
------------------------------------------------------------
422+
{
423+
".reindexed-v9-ml-anomalies-custom-example": {
424+
"settings": {
425+
"index": {
426+
"number_of_replicas": "1",
427+
"number_of_shards": "1"
428+
}
429+
}
430+
}
431+
}
432+
------------------------------------------------------------
433+
// NOTCONSOLE
434+
--
435+
.. Set the number of replicas to `0`.
436+
+
437+
--
438+
[source,console]
439+
------------------------------------------------------------
440+
PUT /.reindexed-v9-ml-anomalies-custom-example/_settings
441+
{
442+
"index": {
443+
"number_of_replicas": 0
444+
}
445+
}
446+
------------------------------------------------------------
447+
// TEST[skip:TBD]
448+
--
449+
.. Start the reindexing process in asynchronous mode.
450+
+
451+
--
452+
[source,console]
453+
------------------------------------------------------------
454+
POST _reindex?wait_for_completion=false
455+
{
456+
"source": {
457+
"index": ".ml-anomalies-custom-example"
458+
},
459+
"dest": {
460+
"index": ".reindexed-v9-ml-anomalies-custom-example"
461+
}
462+
}
463+
------------------------------------------------------------
464+
// TEST[skip:TBD]
465+
The response will contain a task_id. You can check when the task is completed using the following command:
466+
[source,console]
467+
------------------------------------------------------------
468+
GET _tasks/<task_id>
469+
------------------------------------------------------------
470+
// TEST[skip:TBD]
471+
--
472+
.. Set the number of replicas to the original number when the reindexing is finished.
473+
+
474+
--
475+
[source,console]
476+
------------------------------------------------------------
477+
PUT /.reindexed-v9-ml-anomalies-custom-example/_settings
478+
{
479+
"index": {
480+
"number_of_replicas": "<original_number_of_replicas>"
481+
}
482+
}
483+
------------------------------------------------------------
484+
// TEST[skip:TBD]
485+
--
486+
487+
. Get the aliases the original index is pointing to.
488+
+
489+
--
490+
[source,console]
491+
------------------------------------------------------------
492+
GET .ml-anomalies-custom-example/_alias
493+
------------------------------------------------------------
494+
// TEST[skip:TBD]
495+
496+
The response may contain multiple aliases if the results of multiple jobs are stored in the same index.
497+
498+
[source,console-result]
499+
------------------------------------------------------------
500+
{
501+
".ml-anomalies-custom-example": {
502+
"aliases": {
503+
".ml-anomalies-example1": {
504+
"filter": {
505+
"term": {
506+
"job_id": {
507+
"value": "example1"
508+
}
509+
}
510+
},
511+
"is_hidden": true
512+
},
513+
".ml-anomalies-example2": {
514+
"filter": {
515+
"term": {
516+
"job_id": {
517+
"value": "example2"
518+
}
519+
}
520+
},
521+
"is_hidden": true
522+
}
523+
}
524+
}
525+
}
526+
------------------------------------------------------------
527+
// NOTCONSOLE
528+
--
529+
530+
. Now you can reassign the aliases to the new index and delete the original index in one step.
531+
Note that when adding the new index to the aliases, you must use the same filter and is_hidden parameters as for the original index.
532+
+
533+
--
534+
[source,console]
535+
------------------------------------------------------------
536+
POST _aliases
537+
{
538+
"actions": [
539+
{
540+
"add": {
541+
"index": ".reindexed-v9-ml-anomalies-custom-example",
542+
"alias": ".ml-anomalies-example1",
543+
"filter": {
544+
"term": {
545+
"job_id": {
546+
"value": "example1"
547+
}
548+
}
549+
},
550+
"is_hidden": true
551+
}
552+
},
553+
{
554+
"add": {
555+
"index": ".reindexed-v9-ml-anomalies-custom-example",
556+
"alias": ".ml-anomalies-example2",
557+
"filter": {
558+
"term": {
559+
"job_id": {
560+
"value": "example2"
561+
}
562+
}
563+
},
564+
"is_hidden": true
565+
}
566+
},
567+
{
568+
"remove": {
569+
"index": ".ml-anomalies-custom-example",
570+
"aliases": ".ml-anomalies-*"
571+
}
572+
},
573+
{
574+
"remove_index": {
575+
"index": ".ml-anomalies-custom-example"
576+
}
577+
},
578+
{
579+
"add": {
580+
"index": ".reindexed-v9-ml-anomalies-custom-example",
581+
"alias": ".ml-anomalies-custom-example",
582+
"is_hidden": true
583+
}
584+
}
585+
]
586+
}
587+
------------------------------------------------------------
588+
// TEST[skip:TBD]
589+
--
590+
====
591+
592+
[[mark_anomaly_result_index_read_only]]
593+
.Marking anomaly result indices as read-only
594+
[%collapsible]
595+
====
596+
Legacy indexes created in {es} 7.x can be made read-only and supported in {es} 9.x.
597+
Making an index with a large amount of historical results read-only allows for a quick migration to the next major release, since you don't have to wait for the data to be reindexed into the new format.
598+
However, it has the limitation that even after deleting an {anomaly-job}, the historical results associated with this job are not completely deleted.
599+
Therefore, the system will prevent you from creating a new job with the same name.
600+
601+
To set the index as read-only, add the `write` block to the index:
602+
603+
[source,console]
604+
------------------------------------------------------------
605+
PUT .ml-anomalies-custom-example/_block/write
606+
------------------------------------------------------------
607+
// TEST[skip:TBD]
608+
609+
Indices created in {es} 7.x that have a `write` block will not raise a critical deprecation warning.
610+
====
611+
612+
[[delete_anomaly_result_index]]
613+
.Deleting anomaly result indices
614+
[%collapsible]
615+
====
616+
If an index contains results of the jobs that are no longer required.
617+
To list all jobs that stored results in an index, use the terms aggregation:
618+
619+
[source,console]
620+
------------------------------------------------------------
621+
GET .ml-anomalies-custom-example/_search
622+
{
623+
"size": 0,
624+
"aggs": {
625+
"job_ids": {
626+
"terms": {
627+
"field": "job_id",
628+
"size": 100
629+
}
630+
}
631+
}
632+
}
633+
------------------------------------------------------------
634+
// TEST[skip:TBD]
635+
636+
The jobs can be deleted in the UI.
637+
After the last job is deleted, the index will be deleted as well.
638+
====

0 commit comments

Comments
 (0)