(Doc+) Flush out Data Tiers (#107981) (#111074)

I highly value the content on this [Data Tiers](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-tiers.html) page. Thanks for writing it! In my experience, some users may become slightly confused by its golden nuggets due to its brevity. This PR attempts to flush out common questions while remaining concise. 

The main changes are in the first and second-to-last sections; however, I do attempt some heading restructuring to make the TOC idea-groupings more clear for easier scan-throughs. 

The specific clarifications I'd like to push in order of appearance:

- There's content tier (for "data category" > "content" as we've dubbed it on the higher page) and the data temperature tiers (for time series). That the temperature tiers group together is technically not stated so users end up asking about when they'd go hot>warm vs content>warm, etc. I suspect this confusion is only because users come straight to this page instead of starting at the hierarchy-parent page so have linked up. 
- (Main) Frozen being accessed/searched "rarely" should imply, well rarely. I wrote 1% in the PR `[TIP]` guideline section as a discussion starting point. Frequently we see users not understanding either that they actually have been or that they shouldn't have ≥25% of all searches hitting frozen tier. This comes up because of architecture bugs (e.g. frozen indices with future timestamps) but also just happenstance (e.g. 01605242 where of searches they hit majority hot, ~5% cold, but then again hit 75% frozen).
- There's a slew of "how do I check that?", "how do I change that (at creation/later)?", "what if I set it null?" questions we get about `_tier_preference` so just extended the existing section already about it. 

---------

Co-authored-by: shainaraskas <[email protected]>
Co-authored-by: Lee Hinman <[email protected]>

Author: 3 people

docs/reference/datatiers.asciidoc

@@ -2,43 +2,65 @@
 [[data-tiers]]
 == Data tiers
 
-A _data tier_ is a collection of nodes with the same data role that
-typically share the same hardware profile:
+A _data tier_ is a collection of <<modules-node,nodes>> within a cluster that share the same 
+<<node-roles,data node role>>, and a hardware profile that's appropriately sized for the role. Elastic recommends that nodes in the same tier share the same 
+hardware profile to avoid <<hotspotting,hot spotting>>. 
 
-* <<content-tier, Content tier>> nodes handle the indexing and query load for content such as a product catalog.
-* <<hot-tier, Hot tier>> nodes handle the indexing load for time series data such as logs or metrics
-and hold your most recent, most-frequently-accessed data.
-* <<warm-tier, Warm tier>> nodes hold time series data that is accessed less-frequently
+The data tiers that you use, and the way that you use them, depends on the data's <<data-management,category>>.
+
+The following data tiers are can be used with each data category:
+
+Content data:
+
+* <<content-tier,Content tier>> nodes handle the indexing and query load for non-timeseries 
+indices, such as a product catalog.
+
+Time series data:
+
+* <<hot-tier,Hot tier>> nodes handle the indexing load for time series data, 
+such as logs or metrics. They hold your most recent, most-frequently-accessed data.
+* <<warm-tier,Warm tier>> nodes hold time series data that is accessed less-frequently
 and rarely needs to be updated.
 * <<cold-tier,Cold tier>> nodes hold time series data that is accessed
 infrequently and not normally updated. To save space, you can keep
 <<fully-mounted,fully mounted indices>> of
 <<ilm-searchable-snapshot,{search-snaps}>> on the cold tier. These fully mounted
 indices eliminate the need for replicas, reducing required disk space by
 approximately 50% compared to the regular indices.
-* <<frozen-tier, Frozen tier>> nodes hold time series data that is accessed 
+* <<frozen-tier,Frozen tier>> nodes hold time series data that is accessed 
 rarely and never updated. The frozen tier stores <<partially-mounted,partially
 mounted indices>> of <<ilm-searchable-snapshot,{search-snaps}>> exclusively.
 This extends the storage capacity even further — by up to 20 times compared to
 the warm tier. 
 
-TIP: The performance of an {es} node is often limited by the performance of the underlying storage. 
+TIP: The performance of an {es} node is often limited by the performance of the underlying storage and hardware profile. 
+For example hardware profiles, refer to Elastic Cloud's {cloud}/ec-reference-hardware.html[instance configurations]. 
 Review our recommendations for optimizing your storage for <<indexing-use-faster-hardware,indexing>> and <<search-use-faster-hardware,search>>.
 
 IMPORTANT: {es} generally expects nodes within a data tier to share the same 
 hardware profile. Variations not following this recommendation should be 
 carefully architected to avoid <<hotspotting,hot spotting>>.
 
-When you index documents directly to a specific index, they remain on content tier nodes indefinitely.
+The way data tiers are used often depends on the data's category:
+
+- Content data remains on the <<content-tier,content tier>> for its entire
+data lifecycle. 
 
-When you index documents to a data stream, they initially reside on hot tier nodes.
-You can configure <<index-lifecycle-management, {ilm}>> ({ilm-init}) policies
-to automatically transition your time series data through the hot, warm, and cold tiers
-according to your performance, resiliency and data retention requirements.
+- Time series data may progress through the 
+descending temperature data tiers (hot, warm, cold, and frozen) according to your 
+performance, resiliency, and data retention requirements. 
++ 
+You can automate these lifecycle transitions using the <<data-streams,data stream lifecycle>>, or custom <<index-lifecycle-management,{ilm}>>. 
+
+[discrete]
+[[available-tier]]
+=== Available data tiers
+
+Learn more about each data tier, including when and how it should be used.
 
 [discrete]
 [[content-tier]]
-=== Content tier
+==== Content tier
 
 // tag::content-tier[]
 Data stored in the content tier is generally a collection of items such as a product catalog or article archive.
@@ -53,13 +75,14 @@ While they are also responsible for indexing, content data is generally not inge
 as time series data such as logs and metrics. From a resiliency perspective the indices in this
 tier should be configured to use one or more replicas.
 
-The content tier is required. System indices and other indices that aren't part
-of a data stream are automatically allocated to the content tier.
+The content tier is required and is often deployed within the same node 
+grouping as the hot tier. System indices and other indices that aren't part
+of a data stream are automatically allocated to the content tier. 
 // end::content-tier[]
 
 [discrete]
 [[hot-tier]]
-=== Hot tier
+==== Hot tier
 
 // tag::hot-tier[]
 The hot tier is the {es} entry point for time series data and holds your most-recent,
@@ -74,7 +97,7 @@ data stream>> are automatically allocated to the hot tier.
 
 [discrete]
 [[warm-tier]]
-=== Warm tier
+==== Warm tier
 
 // tag::warm-tier[]
 Time series data can move to the warm tier once it is being queried less frequently
@@ -87,7 +110,7 @@ For resiliency, indices in the warm tier should be configured to use one or more
 
 [discrete]
 [[cold-tier]]
-=== Cold tier
+==== Cold tier
 
 // tag::cold-tier[]
 When you no longer need to search time series data regularly, it can move from
@@ -109,7 +132,7 @@ but doesn't reduce required disk space compared to the warm tier.
 
 [discrete]
 [[frozen-tier]]
-=== Frozen tier
+==== Frozen tier
 
 // tag::frozen-tier[]
 Once data is no longer being queried, or being queried rarely, it may move from
@@ -123,9 +146,15 @@ sometimes fetch frozen data from the snapshot repository, searches on the frozen
 tier are typically slower than on the cold tier.
 // end::frozen-tier[]
 
+[discrete]
+[[configure-data-tiers]]
+=== Configure data tiers
+
+Follow the instructions for your deployment type to configure data tiers.
+
 [discrete]
 [[configure-data-tiers-cloud]]
-=== Configure data tiers on {ess} or {ece}
+==== {ess} or {ece}
 
 The default configuration for an {ecloud} deployment includes a shared tier for
 hot and content data. This tier is required and can't be removed.
@@ -159,7 +188,7 @@ tier].
 
 [discrete]
 [[configure-data-tiers-on-premise]]
-=== Configure data tiers for self-managed deployments
+==== Self-managed deployments
 
 For self-managed deployments, each node's <<data-node,data role>> is configured
 in `elasticsearch.yml`. For example, the highest-performance nodes in a cluster
@@ -177,25 +206,59 @@ tier.
 [[data-tier-allocation]]
 === Data tier index allocation
 
-When you create an index, by default {es} sets
-<<tier-preference-allocation-filter, `index.routing.allocation.include._tier_preference`>>
+The <<tier-preference-allocation-filter, `index.routing.allocation.include._tier_preference`>> setting determines which tier the index should be allocated to.
+
+When you create an index, by default {es} sets the `_tier_preference`
 to `data_content` to automatically allocate the index shards to the content tier.
 
 When {es} creates an index as part of a <<data-streams, data stream>>,
-by default {es} sets
-<<tier-preference-allocation-filter, `index.routing.allocation.include._tier_preference`>>
+by default {es} sets the `_tier_preference`
 to `data_hot` to automatically allocate the index shards to the hot tier.
 
-You can explicitly set `index.routing.allocation.include._tier_preference`
-to opt out of the default tier-based allocation.
+At the time of index creation, you can override the default setting by explicitly setting 
+the preferred value in one of two ways:
+
+- Using an <<index-templates,index template>>. Refer to <<getting-started-index-lifecycle-management,Automate rollover with ILM>> for details.
+- Within the <<indices-create-index,create index>> request body. 
+
+You can override this 
+setting after index creation by <<indices-update-settings,updating the index setting>> to the preferred 
+value. 
+
+This setting also accepts multiple tiers in order of preference. This prevents indices from remaining unallocated if no nodes are available in the preferred tier. For example, when {ilm} migrates an index to the cold phase, it sets the index `_tier_preference` to `data_cold,data_warm,data_hot`.
+
+To remove the data tier preference 
+setting, set the `_tier_preference` value to `null`. This allows the index to allocate to any data node within the cluster. Setting the `_tier_preference` to `null` does not restore the default value. Note that, in the case of managed indices, a <<ilm-migrate,migrate>> action might apply a new value in its place. 
+
+[discrete]
+[[data-tier-allocation-value]]
+==== Determine the current data tier preference
+
+You can check an existing index's data tier preference by <<indices-get-settings,polling its 
+settings>> for `index.routing.allocation.include._tier_preference`:
+
+[source,console]
+--------------------------------------------------
+GET /my-index-000001/_settings?filter_path=*.settings.index.routing.allocation.include._tier_preference
+--------------------------------------------------
+// TEST[setup:my_index]
+
+[discrete]
+[[data-tier-allocation-troubleshooting]]
+==== Troubleshooting
+
+The `_tier_preference` setting might conflict with other allocation settings. This conflict might prevent the shard from allocating. A conflict might occur when a cluster has not yet been completely <<troubleshoot-migrate-to-tiers,migrated 
+to data tiers>>. 
+
+This setting will not unallocate a currently allocated shard, but might prevent it from migrating from its current location to its designated data tier. To troubleshoot, call the <<cluster-allocation-explain,cluster allocation explain API>> and specify the suspected problematic shard.
 
 [discrete]
 [[data-tier-migration]]
-=== Automatic data tier migration
+==== Automatic data tier migration
 
 {ilm-init} automatically transitions managed
 indices through the available data tiers using the <<ilm-migrate, migrate>> action.
 By default, this action is automatically injected in every phase.
-You can explicitly specify the migrate action with `"enabled": false` to disable automatic migration,
+You can explicitly specify the migrate action with `"enabled": false` to <<ilm-disable-migrate-ex,disable automatic migration>>,
 for example, if you're using the <<ilm-allocate, allocate action>> to manually
 specify allocation rules.