upgrade docs signifficant refinement (#2296)

This PR include multiple changes and improvements in the upgrade docs:

- Upgrade orchestrators moved downward in the TOC

- Landing page
([upgrade.md](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/2296/deploy-manage/upgrade)):
Improved introduction, added an overview upgrade section as a map for
the entire upgrade documentation, and upgrade-path to highlight that
important information.

- [Plan
upgrade](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/2296/deploy-manage/upgrade/plan-upgrade):
Scope clarified, the page is about general recommendations for the
upgrade and compatibility verifications in order to create an upgrade
plan. Added an `upgrade order` section, and a final `example`.

- [Prepare to
upgade](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/2296/deploy-manage/upgrade/prepare-to-upgrade):
We had lost part of the content (minor upgrades preparation steps)
during the docs migration, and now that we have multiple 9.x versions
that part feels needed. I've also re-worked that page and put all common
steps (for major or minor upgrades) together, as the original asciidoc
document
(https://www.elastic.co/guide/en/elastic-stack/8.18/upgrading-elastic-stack.html)
also felt repetitive and a bit duplicate.

- [Upgrade your deployment or
cluster](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/2296/deploy-manage/upgrade/deployment-or-cluster):
changed a bit the intro and structure, plus added `out of order`
releases section, related and linked from the upgrade-paths of the
landing page.

- [Upgrade Elasticsearch self-managed (rolling upgrade
instructions)](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/2296/deploy-manage/upgrade/deployment-or-cluster/elasticsearch):
Just minor changes adding some headings for better visibility.

I've tried to also position properly the documents in the flow, with
"next steps" sections and proper introductions.

Closes: #1608

Part of the work of #1685

---------

Co-authored-by: David Kilfoyle <[email protected]>
Co-authored-by: shainaraskas <[email protected]>

Author: 3 people

deploy-manage/_snippets/serverless-upgrade.md

@@ -0,0 +1,5 @@
+::::{note}
+With [{{serverless-full}}](/deploy-manage/deploy/elastic-cloud/serverless.md), upgrades of the Elastic-managed infrastructure and project components are fully handled by Elastic. Users automatically receive the latest features and improvements, with no need to plan or perform project-level upgrades.
+
+Keep in mind that the [ingest components](/deploy-manage/upgrade/ingest-components.md) or clients you manage externally, such as {{agent}}, {{beats}}, {{ls}}, or custom applications, still need to be updated manually.
+::::

deploy-manage/deploy/cloud-on-k8s.md

@@ -91,7 +91,7 @@ ECK should work with all conformant **installers** listed in these [FAQs](https:
 
 Alpha, beta, and stable API versions follow the same [conventions used by Kubernetes](https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-versioning).
 
-### {{stack}} compatibility
+### {{stack}} compatibility [stack-compatibility]
 
 ECK is compatible with the following {{stack}} applications:
 

deploy-manage/toc.yml

@@ -797,12 +797,6 @@ toc:
   - file: upgrade.md
     children:
       - file: upgrade/plan-upgrade.md
-      - file: upgrade/orchestrator.md
-        children:
-          - file: upgrade/orchestrator/upgrade-cloud-enterprise.md
-            children:
-              - file: upgrade/orchestrator/re-running-the-ece-upgrade.md
-          - file: upgrade/orchestrator/upgrade-cloud-on-k8s.md
       - file: upgrade/prepare-to-upgrade.md
         children:
           - file: upgrade/prepare-to-upgrade/upgrade-assistant.md      
@@ -823,6 +817,12 @@ toc:
                   - file: upgrade/deployment-or-cluster/kibana-roll-back.md
           - file: upgrade/deployment-or-cluster/enterprise-search.md   
       - file: upgrade/ingest-components.md        
+      - file: upgrade/orchestrator.md
+        children:
+          - file: upgrade/orchestrator/upgrade-cloud-enterprise.md
+            children:
+              - file: upgrade/orchestrator/re-running-the-ece-upgrade.md
+          - file: upgrade/orchestrator/upgrade-cloud-on-k8s.md
   - file: uninstall.md
     children:
       - file: uninstall/uninstall-elastic-cloud-enterprise.md

deploy-manage/upgrade.md

@@ -1,14 +1,99 @@
+---
+applies_to:
+  stack:
+  deployment:
+    eck:
+    ess:
+    ece:
+    self:
+products:
+  - id: kibana
+  - id: cloud-enterprise
+  - id: cloud-hosted
+  - id: cloud-kubernetes
+  - id: elasticsearch
+---
+
 # Upgrade
 
-Upgrading to the latest version provides access to the newest Elastic features, enhancements, performance improvements, and bug fixes. These updates reduce costs, speed up threat response, and improve investigative and analytical data tools. 
+Upgrading to the latest version provides access to the newest Elastic features, security patches, performance improvements, and bug fixes. These updates reduce costs, speed up threat response, and improve investigative and analytical data tools.
 
-:::{important}
-In {{stack}} 9.0.0 and beyond, Enterprise Search is unavailable. For more information, refer to [Migrating to 9.x from Enterprise Search 8.x versions](https://www.elastic.co/guide/en/enterprise-search/8.18/upgrading-to-9-x.html).
+As Elastic releases new versions, older versions of Elastic products reach their end of life on a set schedule. To keep your deployment supported, stay up to date. For more information, refer to [Product End of Life Dates](https://www.elastic.co/support/eol).
+
+:::{include} /deploy-manage/_snippets/serverless-upgrade.md
 :::
 
-When Elastic releases new versions, older versions reach their end of life on a set schedule. To keep your deployment supported, stay up to date. For more information, refer to [Product End of Life Dates](https://www.elastic.co/support/eol).
+The upgrade procedure depends on how your deployment is managed. If you're using {{ech}} or {{ece}}, upgrades can often be performed with a single click in the {{ecloud}} UI. For self-managed deployments, upgrades must be carried out manually using a rolling upgrade process, upgrading the nodes one by one to minimize downtime and ensure cluster stability.
+
+This section provides guidance to help you plan and safely perform upgrades of your {{stack}} components, with a primary focus on {{es}} and {{kib}} as the core of the stack. It also covers upgrades for orchestration platforms like {{ece}} and {{eck}}, as well as related components such as APM, {{beats}}, {{agent}}, and {{ls}}.
 
-:::{note}
-Upgrading from a release candidate build, such as 9.0.0-rc1, is unsupported. Use pre-releases only for testing in a temporary environment.
+:::{important}
+In {{stack}} 9.0.0 and beyond, Enterprise Search is unavailable. For more information, refer to [Migrating to 9.x from Enterprise Search 8.x versions](https://www.elastic.co/guide/en/enterprise-search/8.19/upgrading-to-9-x.html).
 :::
 
+## Upgrade overview
+
+Upgrading your Elastic cluster or deployment involves several stages, including planning, preparation, and execution. This section guides you through the full upgrade process:
+
+- [Plan your upgrade](./upgrade/plan-upgrade.md): Review compatibility, define your upgrade path and order, and understand important pre-upgrade considerations.
+
+- [Prepare to upgrade](./upgrade/prepare-to-upgrade.md): Follow detailed preparation steps for major, minor, and patch upgrades. Identify breaking changes, run the Upgrade Assistant (for major upgrades), and verify readiness.
+
+- [Upgrade your deployment or cluster](./upgrade/deployment-or-cluster.md): Step-by-step instructions for performing the upgrade, organized by deployment type:
+
+    - [Upgrade deployments on {{ech}}](/deploy-manage/upgrade/deployment-or-cluster/upgrade-on-ech.md)
+    - [Upgrade deployments on {{ece}}](/deploy-manage/upgrade/deployment-or-cluster/upgrade-on-ece.md)
+    - [Upgrade deployments on {{eck}}](/deploy-manage/upgrade/deployment-or-cluster/upgrade-on-eck.md)
+    - [Upgrade self-managed clusters](/deploy-manage/upgrade/deployment-or-cluster/self-managed.md)
+
+- [Upgrade ingest components](./upgrade/ingest-components.md): Covers supporting components such as {{beats}}, {{agent}}, {{ls}}, and APM Server.
+
+Additionally, if you're using a self-managed orchestration platform such as {{ece}} or {{eck}}, refer to [Upgrade your ECE or ECK orchestrator](/deploy-manage/upgrade/orchestrator.md) to keep the orchestrator up to date.
+
+## Availability during upgrades
+
+{{es}} supports rolling upgrades, where nodes are upgraded one at a time to minimize downtime and ensure cluster stability. If your deployment or cluster follows high availability best practices, such as using replicated indices and distributing nodes across multiple availability zones, it should remain available throughout the entire upgrade process.
+
+{{kib}} upgrades, however, require downtime. All running instances must be shut down before starting the upgrade.
+
+::::{important}
+Ensure your deployment is properly sized and configured for high availability to reduce the risk of service interruption during upgrades.
+
+If you want to learn more about scaling, high availability, and performance, refer to:
+* [Run {{es}} in production](/deploy-manage/production-guidance/elasticsearch-in-production-environments.md)
+* [Run {{kib}} in production](/deploy-manage/production-guidance/kibana-in-production-environments.md)
+* [{{ech}} > Plan for production](/deploy-manage/deploy/elastic-cloud/elastic-cloud-hosted-planning.md)
+::::
+
+You’ll find more information about availability considerations for specific deployment types in the related upgrade guides within this section.
+
+## Upgrade paths [upgrade-paths]
+
+You can upgrade to a higher version if the target version was released *after* your current version. Upgrades to versions released *before* your current version are not supported, even if the version number is higher. Refer to [out-of-order releases](/deploy-manage/upgrade/deployment-or-cluster.md#out-of-order-releases) for more information.
+
+For example:  
+- ✅ Upgrade allowed: From 9.0.4 to 9.1.0 (9.1.0 released *after* 9.0.4)
+- ❌ Not allowed: From 9.0.5 to 9.1.0 (9.1.0 released *before* 9.0.5) → wait for 9.1.1 to be released
+<!--
+Uncomment this example when 9.1.1 is released.
+- ✅ Upgrade allowed: From 9.0.5 to 9.1.1 (9.1.1 released *after* 9.0.5)
+-->
+
+### Upgrade paths from 8.x [upgrade-paths-8.x]
+
+To perform a major upgrade from 8.x, the required starting version depends on the target release:
+
+- To upgrade to the **9.0.x series**, you must be on **8.18.x**.
+- To upgrade to **9.1.0 or later**, you must be on **8.19.x**, which is the latest minor release of the 8.x series.
+
+::::{note}
+While 8.19 is the final minor release in the 8.x series, 8.18 was released at the same time as 9.0, enabling a supported upgrade path between the 8.18.x and 9.0.x series. This compatibility also applies to other features and clients.
+::::
+
+If you're upgrading to the current {{version.stack}} release from an earlier 8.x version, first upgrade to the latest available 8.19 release.
+
+#### Ingest tools and clients considerations
+
+For flexible upgrade scheduling, 8.19 {{agent}}, {{beats}}, and {{ls}} are compatible with all 9.x versions of {{es}}.
+
+By default, 8.x {{es}} clients are compatible with 9.x and use [REST API compatibility](elasticsearch://reference/elasticsearch/rest-apis/compatibility.md) to maintain compatibility with the 9.x cluster.

deploy-manage/upgrade/deployment-or-cluster.md

@@ -28,17 +28,38 @@ products:
 
 # Upgrade your deployment or cluster [upgrade-deployment-cluster]
 
-When upgrading an existing cluster, you perform a minor or major upgrade. For example, a minor upgrade takes you from version 9.0.0 to 9.1.0, while a major upgrade takes you from version 8.0.0 to 9.0.0.
+This section contains the upgrade instructions for {{es}} clusters and {{kib}} instances. Upgrade procedures depend on whether you installed Elastic components using Elastic-managed or self-managed infrastructure.
 
-Upgrade procedures depend on whether you installed Elastic components using Elastic-managed or self-managed infrastructure.
+## Prerequisites
+
+Before proceeding with the upgrade, review the [Plan your upgrade](/deploy-manage/upgrade/plan-upgrade.md) guidance to understand compatibility and timing considerations, and follow the steps in [Prepare to upgrade](/deploy-manage/upgrade/prepare-to-upgrade.md) to get your environment ready for the upgrade.
+
+## Out-of-order releases [out-of-order-releases]
+
+Elastic maintains several minor versions of {{es}} at the same time. This means releases do not always happen in order of their version numbers. You can only upgrade to {{version.stack}} if the version you are currently running meets both of these conditions:
+
+* Has a lower version number than {{version.stack}}
+* Has an earlier release date than {{version.stack}}
+
+If you are currently running a version with a lower version number but a later release date than {{version.stack}}, wait for a newer release before upgrading.
+
+Additionally, upgrading from a release candidate build, such as 9.0.0-rc1, is unsupported. Use pre-releases only for testing in a temporary environment.
+
+## Upgrade methods
 
 If you’re using Elastic-managed infrastructure, use the following options:
 
 * [Upgrade on {{ech}}](/deploy-manage/upgrade/deployment-or-cluster/upgrade-on-ech.md)
-* Upgrade on [{{serverless-full}}](/deploy-manage/deploy/elastic-cloud/serverless.md), which is automatically performed by Elastic and requires no user management
 
 If you’re using self-managed infrastructure - either on-prem or public cloud - use the following options:
 
 * [Upgrade the {{stack}} on a self-managed cluster](/deploy-manage/upgrade/deployment-or-cluster/self-managed.md)
-* [Upgrade on {{ece}} (ECE)](/deploy-manage/upgrade/deployment-or-cluster/upgrade-on-ece.md)
-* [Upgrade on {{eck}} (ECK)](/deploy-manage/upgrade/deployment-or-cluster/upgrade-on-eck.md)
+* [Upgrade your deployment on {{ece}} (ECE)](/deploy-manage/upgrade/deployment-or-cluster/upgrade-on-ece.md)
+* [Upgrade your deployment on {{eck}} (ECK)](/deploy-manage/upgrade/deployment-or-cluster/upgrade-on-eck.md)
+
+:::{include} /deploy-manage/_snippets/serverless-upgrade.md
+:::
+
+## Next steps
+
+Once you've successfully upgraded your deployment, you can [upgrade your ingest components](./ingest-components.md), such as {{ls}}, {{agent}}, or {{beats}}.

deploy-manage/upgrade/deployment-or-cluster/archived-settings.md

@@ -1,6 +1,13 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/archived-settings.html
+applies_to:
+  stack:
+  deployment:
+    eck:
+    ess:
+    ece:
+    self:
 products:
   - id: elasticsearch
 ---

deploy-manage/upgrade/deployment-or-cluster/elasticsearch.md

@@ -3,20 +3,39 @@ applies_to:
   stack:
   deployment:
     self: all
+products:
+  - id: elasticsearch
 ---
 
 # Upgrade {{es}} [upgrading-elasticsearch]
 
 An {{es}} cluster can be upgraded one node at a time so upgrading does not interrupt service. Running multiple versions of {{es}} in the same cluster beyond the duration of an upgrade is not supported, as shards cannot be replicated from upgraded nodes to nodes running the older version.
 
-Before you start, [take the upgrade preparation steps](/deploy-manage/upgrade/prepare-to-upgrade.md). When performing a [rolling upgrade](#rolling-upgrades):
+:::{note}
+Upgrading from a release candidate build, such as 9.0.0-rc1, is unsupported. Use pre-releases only for testing in a temporary environment.
+:::
 
-1. Upgrade the data nodes first, tier-by-tier, starting with the frozen tier, then the cold tier, then the warm tier, then the hot tier, and finally any other data nodes which are not in a tier. Complete the upgrade for all nodes in each data tier before moving to the next. This ensures {{ilm-init}} can continue to move data through the tiers during the upgrade. You can get the list of nodes in a specific tier with a `GET /_nodes` request, for example: `GET /_nodes/data_frozen:true/_none`.
+Before you start the rolling upgrade procedure, [plan your upgrade](/deploy-manage/upgrade/plan-upgrade.md) and [take the upgrade preparation steps](/deploy-manage/upgrade/prepare-to-upgrade.md).
+
+## {{es}} nodes upgrade order
+
+When performing a [rolling upgrade](#rolling-upgrades):
+
+1. Upgrade the data nodes first, tier-by-tier, in the following order: 
+    1. The frozen tier
+    2. The cold tier
+    3. The warm tier
+    4. The hot tier
+    5. Any other data nodes which are not in a tier.
+    
+    Complete the upgrade for all nodes in each data tier before moving to the next. This ensures {{ilm-init}} can continue to move data through the tiers during the upgrade. You can get the list of nodes in a specific tier with a `GET /_nodes` request, for example: `GET /_nodes/data_frozen:true/_none`.
 2. Upgrade all remaining nodes that are neither master-eligible nor data nodes. This includes dedicated ML nodes, dedicated ingest nodes, and dedicated coordinating nodes.
 3. Upgrade the master-eligible nodes last. You can retrieve a list of these nodes with `GET /_nodes/master:true/_none`.
 
 This order ensures that all nodes can join the cluster during the upgrade. Upgraded nodes can join a cluster with an older master, but older nodes cannot always join a cluster with a upgraded master.
 
+## Upgrade process
+
 To upgrade a cluster:
 
 1. **Disable shard allocation**.
@@ -45,10 +64,9 @@ To upgrade a cluster:
     It is possible to leave your {{ml}} jobs running during the upgrade, but it puts increased load on the cluster. When you shut down a {{ml}} node, its jobs automatically move to another node and restore the model states.
 
     ::::{note}
-    Any {{ml}} indices created before 8.x must be reindexed before upgrading, which you can initiate from the **Upgrade Assistant** in 8.18. For more information, refer to [Anomaly detection results migration]
+    Any {{ml}} indices created before 8.x must be reindexed before upgrading, which you can initiate from the **Upgrade Assistant** in 8.19. For more information, refer to [Anomaly detection results migration]
     ::::
 
-
     * Temporarily halt the tasks associated with your {{ml}} jobs and {{dfeeds}} and prevent new jobs from opening by using the [set upgrade mode API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-set-upgrade-mode):
 
         ```console
@@ -187,7 +205,7 @@ To upgrade a cluster:
 
 
 
-## Rolling upgrades [rolling-upgrades]
+## Rolling upgrades considerations [rolling-upgrades]
 
 During a rolling upgrade, the cluster continues to operate normally. However, any new functionality is disabled or operates in a backward compatible mode until all nodes in the cluster are upgraded. New functionality becomes operational once the upgrade is complete and all nodes are running the new version. Once that has happened, there’s no way to return to operating in a backward compatible mode. Nodes running the previous version will not be allowed to join the fully-updated cluster.
 

deploy-manage/upgrade/deployment-or-cluster/enterprise-search.md

@@ -12,4 +12,4 @@ applies_to:
 
 In {{stack}} 9.0.0 and beyond, Enterprise Search is unavailable. We recommend migrating to {{es}} for improved performance, scalability, and flexibility. For information about what {{es}} features you can use, check out this [blog post](https://www.elastic.co/blog/app-search-to-elasticsearch). 
 
-For complete information about resources and initiatives related to App Search migrations, future enhancements, and decommissioning, refer to [Migrating to 9.x from Enterprise Search 8.x versions](https://www.elastic.co/guide/en/enterprise-search/8.18/upgrading-to-9-x.html).
+For complete information about resources and initiatives related to App Search migrations, future enhancements, and decommissioning, refer to [Migrating to 9.x from Enterprise Search 8.x versions](https://www.elastic.co/guide/en/enterprise-search/8.19/upgrading-to-9-x.html).

deploy-manage/upgrade/deployment-or-cluster/kibana-roll-back.md

@@ -2,6 +2,9 @@
 navigation_title: Roll back to a previous version
 mapped_pages:
   - https://www.elastic.co/guide/en/kibana/current/upgrade-migrations-rolling-back.html
+applies_to:
+  deployment:
+    self: all
 products:
   - id: kibana
 ---