101 prod exploration

Author: shainaraskas

docs/reference/data-management.asciidoc

@@ -4,31 +4,10 @@
 
 [partintro]
 --
-The data you store in {es} generally falls into one of two categories:
 
-* Content: a collection of items you want to search, such as a catalog of products
-* Time series data: a stream of continuously-generated timestamped data, such as log entries
+include::{es-ref-dir}/lifecycle-options.asciidoc[]
 
-Content might be frequently updated,
-but the value of the content remains relatively constant over time.
-You want to be able to retrieve items quickly regardless of how old they are.
-
-Time series data keeps accumulating over time, so you need strategies for
-balancing the value of the data against the cost of storing it.
-As it ages, it tends to become less important and less-frequently accessed,
-so you can move it to less expensive, less performant hardware.
-For your oldest data, what matters is that you have access to the data.
-It's ok if queries take longer to complete.
-
-To help you manage your data, {es} offers you:
-
-* <<index-lifecycle-management, {ilm-cap}>> ({ilm-init}) to manage both indices and data streams and it is fully customisable, and
-* <<data-stream-lifecycle, Data stream lifecycle>> which is the built-in lifecycle of data streams and addresses the most
-common lifecycle management needs.
-
-preview::["The built-in data stream lifecycle is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but this feature is not subject to the support SLA of official GA features."]
-
-**{ilm-init}** can be used to manage both indices and data streams and it allows you to:
+**{ilm-init}** can be used to manage both indices and data streams. It allows you to do the following:
 
 * Define the retention period of your data. The retention period is the minimum time your data will be stored in {es}.
 Data older than this period can be deleted by {es}.
@@ -38,12 +17,24 @@ Data older than this period can be deleted by {es}.
 for your older indices while reducing operating costs and maintaining search performance.
 * Perform <<async-search-intro, asynchronous searches>> of data stored on less-performant hardware.
 
-**Data stream lifecycle** is less feature rich but is focused on simplicity, so it allows you to easily:
+**Data stream lifecycle** is less feature rich but is focused on simplicity. It allows you to do the following:
 
 * Define the retention period of your data. The retention period is the minimum time your data will be stored in {es}.
 Data older than this period can be deleted by {es} at a later time.
-* Improve the performance of your data stream by performing background operations that will optimise the way your data
-stream is stored.
+* Improve the performance of your data stream by performing background operations that will optimize the way your data stream is stored.
+
+**Elastic Curator** is a tool that allows you to manage your indices and snapshots using user-defined filters and predefined actions. If ILM provides the functionality to manage your index lifecycle, and you have at least a Basic license, consider using ILM in place of Curator. Many stack components make use of ILM by default. {curator-ref-current}/ilm.html[Learn more].
+
+NOTE: <<xpack-rollup,data rollup>> is a deprecated Elasticsearch feature that allows you to manage the amount of data that is stored in your cluster, similar to the downsampling functionality of {ilm-init} and data stream lifecycle. This feature should not be used for new deployments.
+
+[TIP]
+====
+{ilm-init} is not available on {es-serverless}.
+
+In an {cloud} or self-managed environment, ILM lets you automatically transition indices through data tiers according to your performance needs and retention requirements. This allows you to balance hardware costs with performance. {es-serverless} eliminates this complexity by optimizing your cluster performance for you.
+
+Data stream lifecycle is an optimized lifecycle tool that lets you focus on the most common lifecycle management needs, without unnecessary hardware-centric concepts like data tiers.
+====
 --
 
 include::ilm/index.asciidoc[]

docs/reference/data-store-architecture.asciidoc

@@ -0,0 +1,17 @@
+= Data store architecture
+
+[partintro]
+--
+
+{es} is a distributed document store. Instead of storing information as rows of columnar data, {es} stores complex data structures that have been serialized as JSON documents. When you have multiple {es} nodes in a cluster, stored documents are distributed across the cluster and can be accessed immediately
+from any node.
+
+The topics in this section provides information about the architecture of {es} and how it stores and retrieves data: 
+
+<<nodes-shards,Nodes and shards>>: Learn about the basic building blocks of an {es} cluster, including nodes, shards, primaries, and replicas.
+<<docs-replication,Reading and writing documents>>: Learn how {es} replicates read and write operations across shards and shard copies.
+--
+
+include::nodes-shards.asciidoc[]
+include::docs/data-replication.asciidoc[leveloffset=-1]
+include::modules/shard-ops.asciidoc[]

docs/reference/docs.asciidoc

@@ -1,9 +1,7 @@
 [[docs]]
 == Document APIs
 
-This section starts with a short introduction to {es}'s <<docs-replication,data
-replication model>>, followed by a detailed description of the following CRUD
-APIs:
+This section describes the following CRUD APIs:
 
 .Single document APIs
 * <<docs-index_>>
@@ -18,8 +16,6 @@ APIs:
 * <<docs-update-by-query>>
 * <<docs-reindex>>
 
-include::docs/data-replication.asciidoc[]
-
 include::docs/index_.asciidoc[]
 
 include::docs/get.asciidoc[]

docs/reference/docs/data-replication.asciidoc

@@ -1,6 +1,6 @@
 
 [[docs-replication]]
-=== Reading and Writing documents
+=== Reading and writing documents
 
 [discrete]
 ==== Introduction

docs/reference/high-availability.asciidoc

@@ -26,5 +26,3 @@ to achieve high availability despite failures.
 --
 
 include::high-availability/cluster-design.asciidoc[]
-
-include::ccr/index.asciidoc[]

docs/reference/index.asciidoc

@@ -60,24 +60,30 @@ include::geospatial-analysis.asciidoc[]
 
 include::watcher/index.asciidoc[]
 
-// cluster management
-
-include::monitoring/index.asciidoc[]
+// production tasks
 
-include::security/index.asciidoc[]
+include::production.asciidoc[]
 
-// production tasks
+include::how-to.asciidoc[]
 
 include::high-availability.asciidoc[]
 
-include::how-to.asciidoc[]
+include::snapshot-restore/index.asciidoc[]
+
+include::ccr/index.asciidoc[leveloffset=-1]
 
 include::autoscaling/index.asciidoc[]
 
-include::snapshot-restore/index.asciidoc[]
+// cluster management
+
+include::security/index.asciidoc[]
+
+include::monitoring/index.asciidoc[]
 
 // reference
 
+include::data-store-architecture.asciidoc[]
+
 include::rest-api/index.asciidoc[]
 
 include::commands/index.asciidoc[]

docs/reference/intro.asciidoc

@@ -375,121 +375,3 @@ Does not yet support full-text search.
 | N/A
 
 |===
-
-// New html page
-[[scalability]]
-=== Get ready for production
-
-Many teams rely on {es} to run their key services. To keep these services running, you can design your {es} deployment
-to keep {es} available, even in case of large-scale outages. To keep it running fast, you also can design your
-deployment to be responsive to production workloads.
-
-{es} is built to be always available and to scale with your needs. It does this using a distributed architecture.
-By distributing your cluster, you can keep Elastic online and responsive to requests.
-
-In case of failure, {es} offers tools for cross-cluster replication and cluster snapshots that can
-help you fall back or recover quickly. You can also use cross-cluster replication to serve requests based on the
-geographic location of your users and your resources.
-
-{es} also offers security and monitoring tools to help you keep your cluster highly available.
-
-[discrete]
-[[use-multiple-nodes-shards]]
-==== Use multiple nodes and shards
-
-[NOTE]
-====
-Nodes and shards are what make {es} distributed and scalable.
-
-These concepts aren’t essential if you’re just getting started. How you <<elasticsearch-intro-deploy,deploy {es}>> in production determines what you need to know:
-
-* *Self-managed {es}*: You are responsible for setting up and managing nodes, clusters, shards, and replicas. This includes
-managing the underlying infrastructure, scaling, and ensuring high availability through failover and backup strategies.
-* *Elastic Cloud*: Elastic can autoscale resources in response to workload changes. Choose from different deployment types
-to apply sensible defaults for your use case. A basic understanding of nodes, shards, and replicas is still important.
-* *Elastic Cloud Serverless*: You don’t need to worry about nodes, shards, or replicas. These resources are 100% automated
-on the serverless platform, which is designed to scale with your workload.
-====
-
-You can add servers (_nodes_) to a cluster to increase capacity, and {es} automatically distributes your data and query load
-across all of the available nodes.
-
-Elastic is able to distribute your data across nodes by subdividing an index into _shards_. Each index in {es} is a grouping
-of one or more physical shards, where each shard is a self-contained Lucene index containing a subset of the documents in
-the index. By distributing the documents in an index across multiple shards, and distributing those shards across multiple
-nodes, {es} increases indexing and query capacity.
-
-There are two types of shards: _primaries_ and _replicas_. Each document in an index belongs to one primary shard. A replica
-shard is a copy of a primary shard. Replicas maintain redundant copies of your data across the nodes in your cluster. 
-This protects against hardware failure and increases capacity to serve read requests like searching or retrieving a document.
-
-[TIP]
-====
-The number of primary shards in an index is fixed at the time that an index is created, but the number of replica shards can
-be changed at any time, without interrupting indexing or query operations.
-====
-
-Shard copies in your cluster are automatically balanced across nodes to provide scale and high availability. All nodes are
-aware of all the other nodes in the cluster and can forward client requests to the appropriate node. This allows {es}
-to distribute indexing and query load across the cluster.
-
-If you’re exploring {es} for the first time or working in a development environment, then you can use a cluster with a single node and create indices
-with only one shard. However, in a production environment, you should build a cluster with multiple nodes and indices
-with multiple shards to increase performance and resilience.
-
-// TODO - diagram
-
-To learn about optimizing the number and size of shards in your cluster, refer to <<size-your-shards,Size your shards>>. 
-To learn about how read and write operations are replicated across shards and shard copies, refer to <<docs-replication,Reading and writing documents>>.
-To adjust how shards are allocated and balanced across nodes, refer to <<shard-allocation-relocation-recovery,Shard allocation, relocation, and recovery>>.
-
-[discrete]
-[[ccr-disaster-recovery-geo-proximity]]
-==== CCR for disaster recovery and geo-proximity
-
-To effectively distribute read and write operations across nodes, the nodes in a cluster need good, reliable connections
-to each other. To provide better connections, you typically co-locate the nodes in the same data center or nearby data centers.
-
-Co-locating nodes in a single location exposes you to the risk of a single outage taking your entire cluster offline. To
-maintain high availability, you can prepare a second cluster that can take over in case of disaster by implementing
-cross-cluster replication (CCR).
-
-CCR provides a way to automatically synchronize indices from your primary cluster to a secondary remote cluster that
-can serve as a hot backup. If the primary cluster fails, the secondary cluster can take over.
-
-You can also use CCR to create secondary clusters to serve read requests in geo-proximity to your users.
-
-Learn more about <<xpack-ccr,cross-cluster replication>> and about <<high-availability-cluster-design,designing for resilience>>.
-
-[TIP]
-====
-You can also take <<snapshot-restore,snapshots>> of your cluster that can be restored in case of failure.
-====
-
-[discrete]
-[[security-and-monitoring]]
-==== Security and monitoring
-
-As with any enterprise system, you need tools to secure, manage, and monitor your {es} clusters. Security,
-monitoring, and administrative features that are integrated into {es} enable you to use {kibana-ref}/introduction.html[Kibana] as a
-control center for managing a cluster.
-
-<<secure-cluster,Learn about securing an {es} cluster>>.
-
-<<monitor-elasticsearch-cluster,Learn about monitoring your cluster>>.
-
-[discrete]
-[[cluster-design]]
-==== Cluster design
-
-{es} offers many options that allow you to configure your cluster to meet your organization’s goals, requirements,
-and restrictions. You can review the following guides to learn how to tune your cluster to meet your needs:
-
-* <<high-availability-cluster-design,Designing for resilience>>
-* <<tune-for-indexing-speed,Tune for indexing speed>>
-* <<tune-for-search-speed,Tune for search speed>>
-* <<tune-for-disk-usage,Tune for disk usage>>
-* <<use-elasticsearch-for-time-series-data,Tune for time series data>>
-
-Many {es} options come with different performance considerations and trade-offs. The best way to determine the
-optimal configuration for your use case is through https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing[testing with your own data and queries].

docs/reference/lifecycle-options.asciidoc

@@ -0,0 +1,21 @@
+The data you store in {es} generally falls into one of two categories:
+
+* *Content*: a collection of items you want to search, such as a catalog of products
+* *Time series data*: a stream of continuously-generated timestamped data, such as log entries
+
+*Content* might be frequently updated,
+but the value of the content remains relatively constant over time.
+You want to be able to retrieve items quickly regardless of how old they are.
+
+*Time series data* keeps accumulating over time, so you need strategies for
+balancing the value of the data against the cost of storing it.
+As it ages, it tends to become less important and less-frequently accessed,
+so you can move it to less expensive, less performant hardware.
+For your oldest data, what matters is that you have access to the data.
+It's ok if queries take longer to complete.
+
+To help you manage your data, {es} offers you the following options:
+
+* <<index-lifecycle-management, {ilm-cap}>>
+* <<data-stream-lifecycle, Data stream lifecycle>>
+* {curator-ref-current}/about.html[Elastic Curator]

docs/reference/modules/shard-ops.asciidoc

@@ -1,5 +1,5 @@
 [[shard-allocation-relocation-recovery]]
-=== Shard allocation, relocation, and recovery
+== Shard allocation, relocation, and recovery
 
 Each <<documents-indices,index>> in Elasticsearch is divided into one or more <<scalability,shards>>. 
 Each document in an index belongs to a single shard.
@@ -12,22 +12,25 @@ Over the course of normal operation, Elasticsearch allocates shard copies to nod
 
 TIP: To learn about optimizing the number and size of shards in your cluster, refer to <<size-your-shards,Size your shards>>. To learn about how read and write operations are replicated across shards and shard copies, refer to <<docs-replication,Reading and writing documents>>.
 
+[discrete]
 [[shard-allocation]]
-==== Shard allocation
+=== Shard allocation
 
 include::{es-ref-dir}/modules/shard-allocation-desc.asciidoc[]
 
 By default, the primary and replica shard copies for an index can be allocated to any node in the cluster, and may be relocated to rebalance the cluster. 
 
-===== Adjust shard allocation settings
+[discrete]
+==== Adjust shard allocation settings
 
 You can control how shard copies are allocated using the following settings:
 
 - <<modules-cluster,Cluster-level shard allocation settings>>: Use these settings to control how shard copies are allocated and balanced across the entire cluster. For example, you might want to allocate nodes availability zones, or prevent certain nodes from being used so you can perform maintenance.
 
 - <<index-modules-allocation,Index-level shard allocation settings>>: Use these settings to control how the shard copies for a specific index are allocated. For example, you might want to allocate an index to a node in a specific data tier, or to an node with specific attributes.
 
-===== Monitor shard allocation
+[discrete]
+==== Monitor shard allocation
 
 If a shard copy is unassigned, it means that the shard copy is not allocated to any node in the cluster. This can happen if there are not enough nodes in the cluster to allocate the shard copy, or if the shard copy can't be allocated to any node that satisfies the shard allocation filtering rules. When a shard copy is unassigned, your cluster is considered unhealthy and returns a yellow or red cluster health status.
 
@@ -39,12 +42,14 @@ You can use the following APIs to monitor shard allocation:
 
 <<red-yellow-cluster-status,Learn more about troubleshooting unassigned shard copies and recovering your cluster health>>.
 
+[discrete]
 [[shard-recovery]]
-==== Shard recovery
+=== Shard recovery
 
 include::{es-ref-dir}/modules/shard-recovery-desc.asciidoc[]
 
-===== Adjust shard recovery settings
+[discrete]
+==== Adjust shard recovery settings
 
 To control how shards are recovered, for example the resources that can be used by recovery operations, and which indices should be prioritized for recovery, you can adjust the following settings: 
 
@@ -54,21 +59,24 @@ To control how shards are recovered, for example the resources that can be used
 
 Shard recovery operations also respect general shard allocation settings. 
 
-===== Monitor shard recovery
+[discrete]
+==== Monitor shard recovery
 
 You can use the following APIs to monitor shard allocation:
 
  - View a list of in-progress and completed recoveries using the <<cat-recovery,cat recovery API>>
  - View detailed information about a specific recovery using the <<indices-recovery,index recovery API>>
 
+[discrete]
 [[shard-relocation]]
-==== Shard relocation
+=== Shard relocation
 
 Shard relocation is the process of moving shard copies from one node to another. This can happen when a node joins or leaves the cluster, or when the cluster is rebalancing.
 
 When a shard copy is relocated, it is created as a new shard copy on the target node. When the shard copy is fully allocated and recovered, the old shard copy is deleted. If the shard copy being relocated is a primary, then the new shard copy is marked as primary before the old shard copy is deleted.
 
-===== Adjust shard relocation settings
+[discrete]
+==== Adjust shard relocation settings
 
 You can control how and when shard copies are relocated. For example, you can adjust the rebalancing settings that control when shard copies are relocated to balance the cluster, or the high watermark for disk-based shard allocation that can trigger relocation. These settings are part of the <<modules-cluster,cluster-level shard allocation settings>>.