[ML] Roll-over .ml-anomalies-* indices when they are at 50GB

[valeriy42]

When customers store anomaly detection results in a shared results index for a long time, the index can grow to a considerable size of 100+GB. This leads to multiple problems:

• Old data can only be deleted using delete-by-query instead of removing the complete outdated index.
• Maintenance tasks timeout, deleting outdated data by query
• Reindexing for a major upgrade can take a very long time

To remedy the situation, we need to roll over the alias on 50GB, so that we end up with a sequence of small indices instead of a single big index.

There are two possible alternatives worth investigating:

• A new maintenance task
• ILM policy similar to the ml state index (see my comment bellow).

Acceptance criteria:

Scenario: Shared results index with multiple jobs rolls over

GIVER: Several anomaly detection jobs run and configured to store results into the default alias .ml-anomalies-shared
WHEN: The backing index reaches the size of 50 GB
THEN: Alias rolls over to a new backing index
All jobs successfully continue to work and write new results
Old results can be renormalized even if they are stored in the old backing index
The single metric viewer and Anomaly Explorer show results for the complete running period.

Scenario: Custom results index with a single job rolls over

GIVER: A single anomaly detection job runs and is configured to store results into the custom index alias .ml-anomalies-custom-job-id
WHEN: The backing index reaches the size of 50 GB
THEN: Alias rolls over to a new backing index
The job continues to work successfully and write new results
Old results can be renormalized even if they are stored in the old backing index
The single metric viewer and Anomaly Explorer show results for the complete running period.

Scenario: Custom results index with multiple jobs pointing to it rolls over

GIVER: Multiple anomaly detection jobs run and are configured to store results into the custom index alias .ml-anomalies-custom-job-id
WHEN: The backing index reaches the size of 50 GB
THEN: Alias rolls over to a new backing index
The jobs successfully continue to work and write new results
Old results can be renormalized even if they are stored in the old backing index
Single metric viewer and Anomaly Explorer show results for the complete running period.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[valeriy42]

Can we job writing to an index with a high number, e.g. .ml-anomalies-shared-000100 and let ILM rollover?

Yes. However, ILM settings have to be integrated into ML index creation. They cannot be applied using index templates.

Steps to reproduce:

1. Create a new ILM policy:

PUT _ilm/policy/ml-anomalies-100docs
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_docs": 100
          }
        }
      }
    }
  }
}

2. Create an anomaly detection job with a custom result index

PUT _ml/anomaly_detectors/nginx
{
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [
      {
        "function": "count",
        "detector_description": "count"
      },
      {
        "function": "mean",
        "field_name": "nginx.access.body_sent.bytes",
        "detector_description": "mean(\"nginx.access.body_sent.bytes\")"
      }
    ],
    "influencers": [
      "nginx.access.geoip.city_name",
      "nginx.access.user_agent.build"
    ],
    "model_prune_window": "30d"
  },
  "analysis_limits": {
    "model_memory_limit": "110MB",
    "categorization_examples_limit": 4
  },
  "data_description": {
    "time_field": "@timestamp",
    "time_format": "epoch_ms"
  },
  "model_snapshot_retention_days": 10,
  "daily_model_snapshot_retention_after_days": 1,
  "results_index_name": "nginx-000010",
  "datafeed_config": {
    "datafeed_id": "datafeed-nginx",
    "query": {
      "bool": {
        "must": [
          {
            "match_all": {}
          }
        ]
      }
    },
    "indices": [
      "filebeat-nginx-elasticco-full"
    ]
  }
}

3. Open the job, this will create a new index .ml-anomalies-custom-nginx-000010

POST _ml/anomaly_detectors/nginx/_open
Add ILM policy to the new index.
PUT .ml-anomalies-custom-nginx-000010/_settings
{
    "settings": {
      "index.lifecycle.name": "ml-anomalies-100docs",
      "index.lifecycle.rollover_alias": ".ml-anomalies-.write-nginx"
    }
}

4. Add reading alias

POST /_aliases?pretty=true
{
  "actions": [
    {
      "add": {
        "indices": ".ml-anomalies-custom-nginx-*", 
        "alias": ".ml-anomalies-nginx"
      }
    }
  ]
}

5. (Optionally) update the ILM poll setting to see results faster

PUT _cluster/settings
{
  "persistent": {
    "indices.lifecycle.poll_interval": "1s"
  }
}

7. Start the datafeed:

POST _ml/datafeeds/datafeed-nginx/_start

This will fill the index .ml-anomalies-custom-nginx-000010 and then roll over to .ml-anomalies-custom-nginx-000011.

Notes:

• We would need to update index settings in Java, similar to Implement ILM policy for .ml-state* indices #52356.
• In practice, we can continue using ml-size-based-ilm-policy which rolls over after 50GB.

[edsavage]

Fixed by #136065