[Upgrade Assistant] Add resolutions for new `templates` and `ilm_policies` warnings

[gmarouli]

Follow up from #208496.

In elastic/elasticsearch#120505 ES introduced new warnings, in this issue we list the resolution paths for these.

Configuring source mode in mappings

This is a deprecation warning in the templates section and it refers to a component template. Resolution path requires to change the component template:

# Component template sample
{
  "template": {
      "mappings": {
           "_source": {
                "mode": "stored"
           }
      }
   }
}
# Updated component template
{
  "template": {
      "settings": {
           "index.mapping.source.mode": "stored"
      }
   }
}

See https://www.elastic.co/guide/en/elasticsearch/reference/current/migrating-8.17.html#deprecations_817_mapping.

Tiers via filtered allocation

This is a warning that affects individual indices, index templates, component templates and ILM policies. Here we can choose between two resolution paths:

• We can use the migration API and deal with this holistically; So just use POST /_ilm/migrate_to_data_tiers, it should work in most cases and I think it's worth a try.

• or if that's too"risky", we can address each index/template/policy individually. I can elaborate more depending on how we want to proceed.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[rudolf]

Scope for 8.18: show ES deprecation message (which includes doc link) to help users manually resolve these.

In 8.19 we can consider having a button for automatically resolving the template deprecation

[rudolf]

@gmarouli Can you clarify under which deprecation sections "Tiers via filtered allocation" would show up and how we could reproduce these deprecations?

[afharo]

Scope for 8.18: show ES deprecation message (which includes doc link) to help users manually resolve these.

I tested this on 8.18, and noticed that the critical deprecation message is shown:

No automatic corrective action on 8.18, though.

@gmarouli, I noticed that the provided link points to elastic/elasticsearch#117172. That PR doesn't provide clear guidance about how to fix the issue. Can we update the link provided by ES (ideally to a documentation page explaining how to fix the issue)?

[gmarouli]

@kkrik-es do you have any docs for this? I am asking since they were not included in the initial PR.

[gmarouli]

Reproduction path for tiers via filtered allocation.

PUT _ilm/policy/my-policy
{
  "policy": {
    "phases": {
      "warm": {
        "actions": {
          "allocate" : {
            "require" : {
              "data": "warm"
            }
          }
        }
      }
    }
  }
}

PUT _component_template/my-cold-template
{
  "template": { 
    "settings": {
      "index.routing.allocation.include.data": "hot"
    }
  }
}

PUT _index_template/my-index-template
{
  "index_patterns": [
    "my-*"
  ],
  "composed_of": ["my-cold-template"],
  "template": {
    "settings": {
      "index.routing.allocation.include.data": "hot",
      "index.lifecycle.rollover_alias": "my-index",
      "index.lifecycle.name": "my-index"
    }
  }
}

PUT my-index-000001
{
    "aliases": {
      "my-index": {}
    }
}

# you can check that the deprecations look as expected:
GET _migration/deprecations
#! this request accesses system indices: [.apm-agent-configuration, .apm-custom-link, .kibana_8.18.0_001, .kibana_alerting_cases_8.18.0_001, .kibana_analytics_8.18.0_001, .kibana_entities-definitions-1, .kibana_ingest_8.18.0_001, .kibana_security_session_1, .kibana_security_solution_8.18.0_001, .kibana_task_manager_8.18.0_001, .kibana_usage_counters_8.18.0_001, .security-7, .security-profile-8, .security-tokens-7], but in a future major version, direct access to system indices will be prevented by default
{
  "cluster_settings": [],
  "node_settings": [],
  "index_settings": {
    "my-index-000001": [
      {
        "level": "warning",
        "message": "index [my-index-000001] is configuring tiers via filtered allocation which is not recommended.",
        "url": "https://ela.st/migrate-to-tiers",
        "details": "One or more of your indices is configured with 'index.routing.allocation.*.data' settings. This is typically used to create a hot/warm or tiered architecture, based on legacy guidelines. Data tiers are a recommended replacement for tiered architecture clusters.",
        "resolve_during_rolling_upgrade": false,
        "_meta": {
          "actions": [
            {
              "action_type": "remove_settings",
              "objects": [
                "index.routing.allocation.include.data"
              ]
            }
          ]
        }
      }
    ]
  },
  "ilm_policies": {
    "my-policy": [
      {
        "level": "warning",
        "message": "Configuring tiers via filtered allocation is not recommended.",
        "url": "https://ela.st/migrate-to-tiers",
        "details": "One or more of your ILM policies is configuring tiers via the 'data' node attribute. This is typically used to create a hot/warm or tiered architecture, based on legacy guidelines. Data tiers are a recommended replacement for tiered architecture clusters.",
        "resolve_during_rolling_upgrade": false
      }
    ]
  },
  "templates": {
    "my-index-template": [
      {
        "level": "warning",
        "message": "Configuring tiers via filtered allocation is not recommended.",
        "url": "https://ela.st/migrate-to-tiers",
        "details": "One or more of your index templates is configured with 'index.routing.allocation.*.data' settings. This is typically used to create a hot/warm or tiered architecture, based on legacy guidelines. Data tiers are a recommended replacement for tiered architecture clusters.",
        "resolve_during_rolling_upgrade": false,
        "_meta": {
          "actions": [
            {
              "action_type": "remove_settings",
              "objects": [
                "index.routing.allocation.include.data"
              ]
            }
          ]
        }
      }
    ],
    "my-cold-template": [
      {
        "level": "warning",
        "message": "Configuring tiers via filtered allocation is not recommended.",
        "url": "https://ela.st/migrate-to-tiers",
        "details": "One or more of your component templates is configured with 'index.routing.allocation.*.data' settings. This is typically used to create a hot/warm or tiered architecture, based on legacy guidelines. Data tiers are a recommended replacement for tiered architecture clusters.",
        "resolve_during_rolling_upgrade": false,
        "_meta": {
          "actions": [
            {
              "action_type": "remove_settings",
              "objects": [
                "index.routing.allocation.include.data"
              ]
            }
          ]
        }
      }
    ]
  },
  "ml_settings": [],
  "data_streams": {}
}

Mitigation
I think the easiest mitigation would be to run the migration API. So this is what I did in my deployment:

# I run the dry run to see if it will impact what I expect to impact
POST /_ilm/migrate_to_data_tiers?dry_run 

# then stop ILM
POST _ilm/stop

# Run the migration
POST /_ilm/migrate_to_data_tiers

# check that everything is resolved
GET _migration/deprecations

# Start ILM
POST _ilm/start

Do you think this is ok, or do you think it's better to work on a warning by warning basis?

[rudolf]

@gmarouli Thanks for the additional details! The details and the mitigation step in the doclink seems sufficient to me. Unless we can think of a good reason users would prefer to do this one warning at a time doing it all at once seems preferable.

@kkrik-es
Upgrade assistant just show what the Elasticsearch deprecation returns. So we need to make sure the details copy (currently undefined for source mode settings) is clear and actionable and that the doclink goes to useful docs.

[kkrik-es]

@martijnvg we don't have a doc page describing this migration, right?

[kkrik-es]

@gmarouli I think the warning message was dropped from the details in your refactoring:

"Configuring source mode in mappings is deprecated and will be removed in future versions. Use [index.mapping.source.mode] index setting instead."

Do you want to send a quick pr for that? I can do it too.

I do agree that it'd be better to have a migration page too, with a simple example.

[jloleysens]

Once we have a more detailed message from ES with basic support for showing these deprecations (PR) Kibana should be ready for 8.18.0