Merge branch 'main' into explore-analyze-missing-pages

Author: szabosteve

deploy-manage/reference-architectures.md

@@ -19,7 +19,7 @@ These architectures are specific to deploying Elastic on {{ech}}, {{eck}}, {{ece
 ::::
 
 
-These reference architectures are recommendations and should be adapted to fit your specific environment and needs. Each solution can vary based on the unique requirements and conditions of your deployment. In these architectures we discuss about how to deploy cluster components. For information about designing ingest architectures to feed content into your cluster, refer to [Ingest architectures](../manage-data/ingest/ingest-reference-architectures/use-case-arch.md)
+These reference architectures are recommendations and should be adapted to fit your specific environment and needs. Each solution can vary based on the unique requirements and conditions of your deployment. In these architectures we discuss about how to deploy cluster components. For information about designing ingest architectures to feed content into your cluster, refer to [Ingest architectures](../manage-data/ingest/ingest-reference-architectures.md).
 
 ## Architectures [reference-architectures-time-series] 
 

deploy-manage/reference-architectures/hotfrozen-high-availability.md

@@ -12,7 +12,7 @@ applies:
 
 The Hot/Frozen High Availability architecture is cost optimized for large time-series datasets. In this architecture, the hot tier is primarily used for indexing, searching, and continuity for automated processes. [Searchable snapshots](https://www.elastic.co/guide/en/elasticsearch/reference/current/searchable-snapshots.html) are taken from hot into a repository, such as a cloud object store or an on-premises shared filesystem, and then cached to any desired volume on the local disks of the frozen tier. Data in the repository is indexed for fast retrieval and accessed on-demand from the frozen nodes. Index and snapshot lifecycle management are used to automate this process.
 
-This architecture is ideal for time-series use cases, such as Observability or Security, that do not require updating. All the necessary components of the {{stack}} are included. This is not intended for sizing workloads, but rather as a basis to ensure that your cluster is ready to handle any desired workload with resiliency. A very high level representation of data flow is included, and for more detail around ingest architecture see our [ingest architectures](../../manage-data/ingest/ingest-reference-architectures/use-case-arch.md) documentation.
+This architecture is ideal for time-series use cases, such as Observability or Security, that do not require updating. All the necessary components of the {{stack}} are included. This is not intended for sizing workloads, but rather as a basis to ensure that your cluster is ready to handle any desired workload with resiliency. A very high level representation of data flow is included, and for more detail around ingest architecture see our [ingest architectures](../../manage-data/ingest/ingest-reference-architectures.md) documentation.
 
 
 ## Hot/Frozen use case [hot-frozen-use-case]

manage-data/index.md

@@ -1,5 +1,35 @@
 # Manage data
 
-% What needs to be done: Write from scratch
+Whether you're looking to build a fast and relevant full-text search solution, monitor business-critical applications and infrastructure, monitor endpoint security data, or one of the [many other use cases Elastic supports](/get-started/index.md#elasticsearch-intro-use-cases), you'll need to understand how to ingest and manage data stored in {{es}}.
 
-% GitHub issue: docs-projects#323
+## Learn how data is stored
+
+% Topic: Learning about Elastic data store primitives
+
+The fundamental unit of storage in {{es}}, the index, is a collection of documents uniquely identified by a name or an alias. These documents go through a process called mapping, which defines how a document and the fields it contains are stored and indexed, and a process called text analysis in which unstructured text is converted into a structured format that’s optimized for search.
+
+**Learn more in [The Elasticsearch data store](/manage-data/data-store.md)**.
+
+## Get data into {{es}}
+
+% Topic: Evaluating and implementing ingestion and data enrichment technologies
+
+Before you can start searching, visualizing, and pulling actionable insights from Elastic, you have to get your data into {{es}}.  Elastic offers a wide range of tools and methods for getting data into {{es}}. The best approach will depend on the kind of data you're ingesting and your specific use case.
+
+**Learn more in [Ingestion](/manage-data/ingest.md).**
+
+## Manage data over time
+
+% Topic: Managing your data volume (lifecycle)
+
+After you've added data to {{es}}, you'll need to manage it over time. For example, you may specify that data be deleted after a retention period or store data in multiple tiers with different performance characteristics.
+
+Strategies for managing data depend on the type of data and how it's being used. For example, with a collection of items you want to search, like a catalog of products, the value of the content remains relatively constant over time so you want to be able to retrieve items quickly regardless of how old they are. Whereas with a stream of continuously-generated timestamped data, such as log entries, the data keeps accumulating over time, so you need strategies for balancing the value of the data against the cost of storing it.
+
+**Learn more in [Data lifecycle](/manage-data/lifecycle.md).**
+
+## Migrate data between Elasticsearch clusters
+
+If you move to new infrastructure, there are several options for moving existing data between {{es}} clusters.
+
+**Learn more in [Migrate your {{es}} data](/manage-data/migrate.md).**

manage-data/ingest/ingest-reference-architectures.md

@@ -1,8 +1,29 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/ingest/current/index.html
+  - https://www.elastic.co/guide/en/ingest/current/use-case-arch.html
 ---
 
-# Ingest reference architectures
+# Ingest architectures [ingest-architectures]
+
+We offer a variety of ingest architectures to serve a wide range of use cases and network configurations.
+
+To ingest data into {{es}}, use the *simplest option that meets your needs* and satisfies your use case. For many users and use cases, the simplest approach is ingesting data with {{agent}} and sending it to {{es}}. {{agent}} and [{{agent}} integrations](https://www.elastic.co/integrations/) are available for many popular platforms and services, and are a good place to start.
+
+::::{tip}
+You can host {{es}} on your own hardware or send your data to {{es}} on {{ecloud}}. For most users, {{agent}} writing directly to {{es}} on {{ecloud}} provides the easiest and fastest time to value. {{ess-leadin-short}}
+::::
+
+**Decision tree**
+
+[Data ingestion](../ingest.md)
+
+| **Ingest architecture** | **Use when** |
+| --- | --- |
+| [*{{agent}} to Elasticsearch*](./ingest-reference-architectures/agent-to-es.md)<br><br>![Image showing {{agent}} collecting data and sending to {{es}}](../../images/ingest-ea-es.png "") | An [{{agent}} integration](https://docs.elastic.co/en/integrations) is available for your data source:<br><br>* Software components with [{{agent}} installed](./ingest-reference-architectures/agent-installed.md)<br>* Software components using [APIs for data collection](./ingest-reference-architectures/agent-apis.md)<br> |
+| [*{{agent}} to {{ls}} to Elasticsearch*](./ingest-reference-architectures/agent-ls.md)<br><br>![Image showing {{agent}} to {{ls}} to {{es}}](../../images/ingest-ea-ls-es.png "") | You need additional capabilities offered by {{ls}}:<br><br>* [**enrichment**](./ingest-reference-architectures/ls-enrich.md) between {{agent}} and {{es}}<br>* [**persistent queue (PQ) buffering**](./ingest-reference-architectures/lspq.md) to accommodate network issues and downstream unavailability<br>* [**proxying**](./ingest-reference-architectures/ls-networkbridge.md) in cases where {{agent}}s have network restrictions for connecting outside of the {{agent}} network<br>* data needs to be [**routed to multiple**](./ingest-reference-architectures/ls-multi.md) {{es}} clusters and other destinations depending on the content<br> |
+| [*{{agent}} to proxy to Elasticsearch*](./ingest-reference-architectures/agent-proxy.md)<br><br>![Image showing connections between {{agent}} and {{es}} using a proxy](../../images/ingest-ea-proxy-es.png "") | Agents have [network restrictions](./ingest-reference-architectures/agent-proxy.md) that prevent connecting outside of the {{agent}} network Note that [{{ls}} as proxy](./ingest-reference-architectures/ls-networkbridge.md) is one option.<br> |
+| [*{{agent}} to {{es}} with Kafka as middleware message queue*](./ingest-reference-architectures/agent-kafka-es.md)<br><br>![Image showing {{agent}} collecting data and using Kafka as a message queue enroute to {{es}}](../../images/ingest-ea-kafka.png "") | Kafka is your [middleware message queue](./ingest-reference-architectures/agent-kafka-es.md):<br><br>* [Kafka ES sink connector](./ingest-reference-architectures/agent-kafka-essink.md) to write from Kafka to {{es}}<br>* [{{ls}} to read from Kafka and route to {{es}}](./ingest-reference-architectures/agent-kafka-ls.md)<br> |
+| [*{{ls}} to Elasticsearch*](./ingest-reference-architectures/ls-for-input.md)<br><br>![Image showing {{ls}} collecting data and sending to {{es}}](../../images/ingest-ls-es.png "") | You need to collect data from a source that {{agent}} can’t read (such as databases, AWS Kinesis). Check out the [{{ls}} input plugins](https://www.elastic.co/guide/en/logstash/current/input-plugins.html).<br> |
+| [*Elastic air-gapped architectures*](./ingest-reference-architectures/airgapped-env.md)<br><br>![Image showing {{stack}} in an air-gapped environment](../../images/ingest-ea-airgapped.png "") | You want to deploy {{agent}} and {{stack}} in an air-gapped environment (no access to outside networks)<br> |
 
-Just a placeholder for a top index page.

manage-data/ingest/ingest-reference-architectures/use-case-arch.md

@@ -70,7 +70,6 @@ toc:
           - file: ingest/ingesting-data-from-applications/ingest-logs-from-nodejs-web-application-using-filebeat.md
       - file: ingest/ingest-reference-architectures.md
         children:
-          - file: ingest/ingest-reference-architectures/use-case-arch.md
           - file: ingest/ingest-reference-architectures/agent-to-es.md
             children:
               - file: ingest/ingest-reference-architectures/agent-installed.md