Merge branch 'main' into lucene_snapshot

Author: ChrisHegarty

docs/changelog/118599.yaml

@@ -0,0 +1,5 @@
+pr: 118599
+summary: Archive-Index upgrade compatibility
+area: Search
+type: enhancement
+issues: []

docs/changelog/118959.yaml

@@ -0,0 +1,5 @@
+pr: 118959
+summary: Allow kibana_system user to manage .reindexed-v8-internal.alerts indices
+area: Authorization
+type: enhancement
+issues: []

docs/reference/data-management.asciidoc

@@ -6,29 +6,26 @@
 --
 The data you store in {es} generally falls into one of two categories:
 
-* Content: a collection of items you want to search, such as a catalog of products
-* Time series data: a stream of continuously-generated timestamped data, such as log entries
-
-Content might be frequently updated,
+* *Content*: a collection of items you want to search, such as a catalog of products
+* *Time series data*: a stream of continuously-generated timestamped data, such as log entries
+*Content* might be frequently updated,
 but the value of the content remains relatively constant over time.
 You want to be able to retrieve items quickly regardless of how old they are.
 
-Time series data keeps accumulating over time, so you need strategies for
+*Time series data* keeps accumulating over time, so you need strategies for
 balancing the value of the data against the cost of storing it.
 As it ages, it tends to become less important and less-frequently accessed,
 so you can move it to less expensive, less performant hardware.
 For your oldest data, what matters is that you have access to the data.
 It's ok if queries take longer to complete.
 
-To help you manage your data, {es} offers you:
-
-* <<index-lifecycle-management, {ilm-cap}>> ({ilm-init}) to manage both indices and data streams and it is fully customisable, and
-* <<data-stream-lifecycle, Data stream lifecycle>> which is the built-in lifecycle of data streams and addresses the most
-common lifecycle management needs.
+To help you manage your data, {es} offers you the following options:
 
-preview::["The built-in data stream lifecycle is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but this feature is not subject to the support SLA of official GA features."]
+* <<index-lifecycle-management, {ilm-cap}>>
+* <<data-stream-lifecycle, Data stream lifecycle>>
+* {curator-ref-current}/about.html[Elastic Curator]
 
-**{ilm-init}** can be used to manage both indices and data streams and it allows you to:
+**{ilm-init}** can be used to manage both indices and data streams. It allows you to do the following:
 
 * Define the retention period of your data. The retention period is the minimum time your data will be stored in {es}.
 Data older than this period can be deleted by {es}.
@@ -38,12 +35,24 @@ Data older than this period can be deleted by {es}.
 for your older indices while reducing operating costs and maintaining search performance.
 * Perform <<async-search-intro, asynchronous searches>> of data stored on less-performant hardware.
 
-**Data stream lifecycle** is less feature rich but is focused on simplicity, so it allows you to easily:
+**Data stream lifecycle** is less feature rich but is focused on simplicity. It allows you to do the following:
 
 * Define the retention period of your data. The retention period is the minimum time your data will be stored in {es}.
 Data older than this period can be deleted by {es} at a later time.
-* Improve the performance of your data stream by performing background operations that will optimise the way your data
-stream is stored.
+* Improve the performance of your data stream by performing background operations that will optimise the way your data stream is stored.
+
+**Elastic Curator** is a tool that allows you to manage your indices and snapshots using user-defined filters and predefined actions. If ILM provides the functionality to manage your index lifecycle, and you have at least a Basic license, consider using ILM in place of Curator. Many stack components make use of ILM by default. {curator-ref-current}/ilm.html[Learn more].
+
+NOTE: <<xpack-rollup,Data rollup>> is a deprecated Elasticsearch feature that allows you to manage the amount of data that is stored in your cluster, similar to the downsampling functionality of {ilm-init} and data stream lifecycle. This feature should not be used for new deployments.
+
+[TIP]
+====
+{ilm-init} is not available on {es-serverless}.
+
+In an {ecloud} or self-managed environment, ILM lets you automatically transition indices through data tiers according to your performance needs and retention requirements. This allows you to balance hardware costs with performance. {es-serverless} eliminates this complexity by optimizing your cluster performance for you.
+
+Data stream lifecycle is an optimized lifecycle tool that lets you focus on the most common lifecycle management needs, without unnecessary hardware-centric concepts like data tiers.
+====
 --
 
 include::ilm/index.asciidoc[]

docs/reference/data-store-architecture.asciidoc

@@ -0,0 +1,18 @@
+= Data store architecture
+
+[partintro]
+--
+
+{es} is a distributed document store. Instead of storing information as rows of columnar data, {es} stores complex data structures that have been serialized as JSON documents. When you have multiple {es} nodes in a cluster, stored documents are distributed across the cluster and can be accessed immediately
+from any node.
+
+The topics in this section provides information about the architecture of {es} and how it stores and retrieves data: 
+
+* <<nodes-shards,Nodes and shards>>: Learn about the basic building blocks of an {es} cluster, including nodes, shards, primaries, and replicas.
+* <<docs-replication,Reading and writing documents>>: Learn how {es} replicates read and write operations across shards and shard copies.
+* <<shard-allocation-relocation-recovery,Shard allocation, relocation, and recovery>>: Learn how {es} allocates and balances shards across nodes.
+--
+
+include::nodes-shards.asciidoc[]
+include::docs/data-replication.asciidoc[leveloffset=-1]
+include::modules/shard-ops.asciidoc[]

docs/reference/docs.asciidoc

@@ -7,9 +7,7 @@
 For the most up-to-date API details, refer to {api-es}/group/endpoint-document[Document APIs].
 --
 
-This section starts with a short introduction to {es}'s <<docs-replication,data
-replication model>>, followed by a detailed description of the following CRUD
-APIs:
+This section describes the following CRUD APIs:
 
 .Single document APIs
 * <<docs-index_>>
@@ -24,8 +22,6 @@ APIs:
 * <<docs-update-by-query>>
 * <<docs-reindex>>
 
-include::docs/data-replication.asciidoc[]
-
 include::docs/index_.asciidoc[]
 
 include::docs/get.asciidoc[]

docs/reference/docs/data-replication.asciidoc

@@ -1,6 +1,6 @@
 
 [[docs-replication]]
-=== Reading and Writing documents
+=== Reading and writing documents
 
 [discrete]
 ==== Introduction

docs/reference/high-availability.asciidoc

@@ -3,28 +3,28 @@
 
 [partintro]
 --
-Your data is important to you. Keeping it safe and available is important
-to {es}. Sometimes your cluster may experience hardware failure or a power
-loss. To help you plan for this, {es} offers a number of features
-to achieve high availability despite failures.
+Your data is important to you. Keeping it safe and available is important to Elastic. Sometimes your cluster may experience hardware failure or a power loss. To help you plan for this, {es} offers a number of features to achieve high availability despite failures. Depending on your deployment type, you might need to provision servers in different zones or configure external repositories to meet your organization's availability needs.
 
-* With proper planning, a cluster can be
-  <<high-availability-cluster-design,designed for resilience>> to many of the
-  things that commonly go wrong, from the loss of a single node or network
-  connection right up to a zone-wide outage such as power loss.
+* *<<high-availability-cluster-design,Design for resilience>>* 
++
+Distributed systems like Elasticsearch are designed to keep working even if some of their components have failed. An Elasticsearch cluster can continue operating normally if some of its nodes are unavailable or disconnected, as long as there are enough well-connected nodes to take over the unavailable node's responsibilities.
++
+If you're designing a smaller cluster, you might focus on making your cluster resilient to single-node failures. Designers of larger clusters must also consider cases where multiple nodes fail at the same time.
+// need to improve connections to ECE, EC hosted, ECK pod/zone docs in the child topics
 
-* You can use <<xpack-ccr,{ccr}>> to replicate data to a remote _follower_
-  cluster which may be in a different data centre or even on a different
-  continent from the leader cluster. The follower cluster acts as a hot
-  standby, ready for you to fail over in the event of a disaster so severe that
-  the leader cluster fails. The follower cluster can also act as a geo-replica
-  to serve searches from nearby clients.
+* *<<xpack-ccr,Cross-cluster replication>>*
++
+To effectively distribute read and write operations across nodes, the nodes in a cluster need good, reliable connections to each other. To provide better connections, you typically co-locate the nodes in the same data center or nearby data centers.
++
+Co-locating nodes in a single location exposes you to the risk of a single outage taking your entire cluster offline. To maintain high availability, you can prepare a second cluster that can take over in case of disaster by implementing {ccr} (CCR).
++
+CCR provides a way to automatically synchronize indices from a leader cluster to a follower cluster. This cluster could be in a different data center or even a different content from the leader cluster. If the primary cluster fails, the secondary cluster can take over.
++
+TIP: You can also use CCR to create secondary clusters to serve read requests in geo-proximity to your users.
 
-* The last line of defence against data loss is to take
-  <<snapshots-take-snapshot,regular snapshots>> of your cluster so that you can
-  restore a completely fresh copy of it elsewhere if needed.
+* *<<snapshot-restore,Snapshots>>* 
++
+Take snapshots of your cluster that can be restored in case of failure.
 --
 
 include::high-availability/cluster-design.asciidoc[]
-
-include::ccr/index.asciidoc[]

docs/reference/index.asciidoc

@@ -76,8 +76,12 @@ include::autoscaling/index.asciidoc[]
 
 include::snapshot-restore/index.asciidoc[]
 
+include::ccr/index.asciidoc[leveloffset=-1]
+
 // reference
 
+include::data-store-architecture.asciidoc[]
+
 include::rest-api/index.asciidoc[]
 
 include::commands/index.asciidoc[]

docs/reference/intro.asciidoc

@@ -397,51 +397,18 @@ geographic location of your users and your resources.
 [[use-multiple-nodes-shards]]
 ==== Use multiple nodes and shards
 
-[NOTE]
-====
-Nodes and shards are what make {es} distributed and scalable.
+When you move to production, you need to introduce multiple nodes and shards to your cluster. Nodes and shards are what make {es} distributed and scalable. The size and number of these nodes and shards depends on your data, your use case, and your budget.
 
-These concepts aren’t essential if you’re just getting started. How you <<elasticsearch-intro-deploy,deploy {es}>> in production determines what you need to know:
+These concepts aren't essential if you're just getting started. How you <<elasticsearch-intro-deploy,deploy {es}>> in production determines what you need to know:
 
 * *Self-managed {es}*: You are responsible for setting up and managing nodes, clusters, shards, and replicas. This includes
 managing the underlying infrastructure, scaling, and ensuring high availability through failover and backup strategies.
 * *Elastic Cloud*: Elastic can autoscale resources in response to workload changes. Choose from different deployment types
 to apply sensible defaults for your use case. A basic understanding of nodes, shards, and replicas is still important.
-* *Elastic Cloud Serverless*: You don’t need to worry about nodes, shards, or replicas. These resources are 100% automated
+* *Elastic Cloud Serverless*: You don't need to worry about nodes, shards, or replicas. These resources are 100% automated
 on the serverless platform, which is designed to scale with your workload.
-====
-
-You can add servers (_nodes_) to a cluster to increase capacity, and {es} automatically distributes your data and query load
-across all of the available nodes.
-
-Elastic is able to distribute your data across nodes by subdividing an index into _shards_. Each index in {es} is a grouping
-of one or more physical shards, where each shard is a self-contained Lucene index containing a subset of the documents in
-the index. By distributing the documents in an index across multiple shards, and distributing those shards across multiple
-nodes, {es} increases indexing and query capacity.
-
-There are two types of shards: _primaries_ and _replicas_. Each document in an index belongs to one primary shard. A replica
-shard is a copy of a primary shard. Replicas maintain redundant copies of your data across the nodes in your cluster. 
-This protects against hardware failure and increases capacity to serve read requests like searching or retrieving a document.
-
-[TIP]
-====
-The number of primary shards in an index is fixed at the time that an index is created, but the number of replica shards can
-be changed at any time, without interrupting indexing or query operations.
-====
-
-Shard copies in your cluster are automatically balanced across nodes to provide scale and high availability. All nodes are
-aware of all the other nodes in the cluster and can forward client requests to the appropriate node. This allows {es}
-to distribute indexing and query load across the cluster.
-
-If you’re exploring {es} for the first time or working in a development environment, then you can use a cluster with a single node and create indices
-with only one shard. However, in a production environment, you should build a cluster with multiple nodes and indices
-with multiple shards to increase performance and resilience.
-
-// TODO - diagram
 
-To learn about optimizing the number and size of shards in your cluster, refer to <<size-your-shards,Size your shards>>. 
-To learn about how read and write operations are replicated across shards and shard copies, refer to <<docs-replication,Reading and writing documents>>.
-To adjust how shards are allocated and balanced across nodes, refer to <<shard-allocation-relocation-recovery,Shard allocation, relocation, and recovery>>.
+Learn more about <<nodes-shards,nodes and shards>>.
 
 [discrete]
 [[ccr-disaster-recovery-geo-proximity]]

docs/reference/modules/shard-ops.asciidoc

@@ -1,5 +1,5 @@
 [[shard-allocation-relocation-recovery]]
-=== Shard allocation, relocation, and recovery
+== Shard allocation, relocation, and recovery
 
 Each <<documents-indices,index>> in Elasticsearch is divided into one or more <<scalability,shards>>. 
 Each document in an index belongs to a single shard.
@@ -12,22 +12,25 @@ Over the course of normal operation, Elasticsearch allocates shard copies to nod
 
 TIP: To learn about optimizing the number and size of shards in your cluster, refer to <<size-your-shards,Size your shards>>. To learn about how read and write operations are replicated across shards and shard copies, refer to <<docs-replication,Reading and writing documents>>.
 
+[discrete]
 [[shard-allocation]]
-==== Shard allocation
+=== Shard allocation
 
 include::{es-ref-dir}/modules/shard-allocation-desc.asciidoc[]
 
 By default, the primary and replica shard copies for an index can be allocated to any node in the cluster, and may be relocated to rebalance the cluster. 
 
-===== Adjust shard allocation settings
+[discrete]
+==== Adjust shard allocation settings
 
 You can control how shard copies are allocated using the following settings:
 
 - <<modules-cluster,Cluster-level shard allocation settings>>: Use these settings to control how shard copies are allocated and balanced across the entire cluster. For example, you might want to allocate nodes availability zones, or prevent certain nodes from being used so you can perform maintenance.
 
 - <<index-modules-allocation,Index-level shard allocation settings>>: Use these settings to control how the shard copies for a specific index are allocated. For example, you might want to allocate an index to a node in a specific data tier, or to an node with specific attributes.
 
-===== Monitor shard allocation
+[discrete]
+==== Monitor shard allocation
 
 If a shard copy is unassigned, it means that the shard copy is not allocated to any node in the cluster. This can happen if there are not enough nodes in the cluster to allocate the shard copy, or if the shard copy can't be allocated to any node that satisfies the shard allocation filtering rules. When a shard copy is unassigned, your cluster is considered unhealthy and returns a yellow or red cluster health status.
 
@@ -39,12 +42,14 @@ You can use the following APIs to monitor shard allocation:
 
 <<red-yellow-cluster-status,Learn more about troubleshooting unassigned shard copies and recovering your cluster health>>.
 
+[discrete]
 [[shard-recovery]]
-==== Shard recovery
+=== Shard recovery
 
 include::{es-ref-dir}/modules/shard-recovery-desc.asciidoc[]
 
-===== Adjust shard recovery settings
+[discrete]
+==== Adjust shard recovery settings
 
 To control how shards are recovered, for example the resources that can be used by recovery operations, and which indices should be prioritized for recovery, you can adjust the following settings: 
 
@@ -54,21 +59,24 @@ To control how shards are recovered, for example the resources that can be used
 
 Shard recovery operations also respect general shard allocation settings. 
 
-===== Monitor shard recovery
+[discrete]
+==== Monitor shard recovery
 
 You can use the following APIs to monitor shard allocation:
 
  - View a list of in-progress and completed recoveries using the <<cat-recovery,cat recovery API>>
  - View detailed information about a specific recovery using the <<indices-recovery,index recovery API>>
 
+[discrete]
 [[shard-relocation]]
-==== Shard relocation
+=== Shard relocation
 
 Shard relocation is the process of moving shard copies from one node to another. This can happen when a node joins or leaves the cluster, or when the cluster is rebalancing.
 
 When a shard copy is relocated, it is created as a new shard copy on the target node. When the shard copy is fully allocated and recovered, the old shard copy is deleted. If the shard copy being relocated is a primary, then the new shard copy is marked as primary before the old shard copy is deleted.
 
-===== Adjust shard relocation settings
+[discrete]
+==== Adjust shard relocation settings
 
 You can control how and when shard copies are relocated. For example, you can adjust the rebalancing settings that control when shard copies are relocated to balance the cluster, or the high watermark for disk-based shard allocation that can trigger relocation. These settings are part of the <<modules-cluster,cluster-level shard allocation settings>>.