Merge branch 'main' into mw-es-ts-structure

Author: marciw

deploy-manage/reference-architectures.md

@@ -1,6 +1,11 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/reference-architectures/current/reference-architectures-overview.html
+applies:
+  stack: all
+  hosted: all
+  ece: all
+  eck: all
 ---
 
 # Reference architectures [reference-architectures-overview]
@@ -9,19 +14,20 @@ Elasticsearch reference architectures are blueprints for deploying Elasticsearch
 
 These architectures are designed by architects and engineers to provide standardized, proven solutions that help you to follow best practices when deploying {{es}}.
 
-::::{tip}
-These architectures are specific to running your deployment on-premises or on cloud. If you are using Elastic serverless your {{es}} clusters are autoscaled and fully managed by Elastic. For all the deployment options, refer to [Run Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-intro-deploy.html).
+::::{tip} 
+These architectures are specific to deploying Elastic on {{ech}}, {{eck}}, {{ece}}, or deploying a self-managed instance. If you are using {{serverless-full}}, your {{es}} clusters are autoscaled and fully managed by Elastic. To learn about all of the deployment options, refer to the [Deploy and manage overview](/deploy-manage/index.md).
 ::::
 
 
-These reference architectures are recommendations and should be adapted to fit your specific environment and needs. Each solution can vary based on the unique requirements and conditions of your deployment. In these architectures we discuss about how to deploy cluster components. For information about designing ingest architectures to feed content into your cluster, refer to [Ingest architectures](https://www.elastic.co/guide/en/ingest/current/use-case-arch.html)
+These reference architectures are recommendations and should be adapted to fit your specific environment and needs. Each solution can vary based on the unique requirements and conditions of your deployment. In these architectures we discuss about how to deploy cluster components. For information about designing ingest architectures to feed content into your cluster, refer to [Ingest architectures](../manage-data/ingest/ingest-reference-architectures/use-case-arch.md)
 
-
-## Architectures [reference-architectures-time-series-2]
+## Architectures [reference-architectures-time-series] 
 
 |     |     |
 | --- | --- |
 | **Architecture** | **When to use** |
-| [*Hot/Frozen - High Availability*](https://www.elastic.co/guide/en/reference-architectures/current/hot-frozen-architecture.html)<br>A high availability architecture that is cost optimized for large time-series datasets. | * Have a requirement for cost effective long term data storage (many months or years).<br>* Provide insights and alerts using logs, metrics, traces, or various event types to ensure optimal performance and quick issue resolution for applications.<br>* Apply Machine Learning and Search AI to assist in dealing with the large amount of data.<br>* Deploy an architecture model that allows for maximum flexibility between storage cost and performance.<br> |
+| [*Hot/Frozen - High Availability*](/deploy-manage/reference-architectures/hotfrozen-high-availability.md)<br>A high availability architecture that is cost optimized for large time-series datasets. | * Have a requirement for cost effective long term data storage (many months or years).<br>* Provide insights and alerts using logs, metrics, traces, or various event types to ensure optimal performance and quick issue resolution for applications.<br>* Apply Machine Learning and Search AI to assist in dealing with the large amount of data.<br>* Deploy an architecture model that allows for maximum flexibility between storage cost and performance.<br> |
 | Additional architectures are on the way.<br>Stay tuned for updates. |  |
 
+
+

deploy-manage/reference-architectures/hotfrozen-high-availability.md

@@ -1,6 +1,11 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/reference-architectures/current/hot-frozen-architecture.html
+applies:
+  stack: all
+  hosted: all
+  ece: all
+  eck: all
 ---
 
 # Hot/Frozen - High Availability [hot-frozen-architecture]
@@ -50,15 +55,15 @@ Machine learning nodes are optional but highly recommended for large scale time
 
 ## Recommended hardware specifications [hot-frozen-hardware]
 
-With {{ecloud}} you can deploy clusters in AWS, Azure, and Google Cloud.  Available hardware types and configurations vary across all three cloud providers but each provides instance types that meet our recommendations for the node types used in this architecture. For more details on these instance types, see our documentation on {{ecloud}} hardware for [AWS](https://www.elastic.co/guide/en/cloud/current/ec-default-aws-configurations.html), [Azure](https://www.elastic.co/guide/en/cloud/current/ec-default-azure-configurations.html), and [GCP](https://www.elastic.co/guide/en/cloud/current/ec-default-gcp-configurations.html). The **Physical** column below is guidance, based on the cloud node types, when self-deploying {{es}} in your own data center.
+With {{ech}}, you can deploy clusters in AWS, Azure, and Google Cloud. Available hardware types and configurations vary across all three cloud providers but each provides instance types that meet our recommendations for the node types used in this architecture. For more details on these instance types, see our documentation on {{ech}} hardware for [AWS](https://www.elastic.co/guide/en/cloud/current/ec-default-aws-configurations.html), [Azure](https://www.elastic.co/guide/en/cloud/current/ec-default-azure-configurations.html), and [GCP](https://www.elastic.co/guide/en/cloud/current/ec-default-gcp-configurations.html). The **Physical** column below is guidance, based on the cloud node types, when self-deploying {{es}} in your own data center.
 
-In the links provided above, Elastic has performance tested hardware for each of the cloud providers to find the optimal hardware for each node type. We use ratios to represent the best mix of CPU, RAM, and disk for each type.   In some cases the CPU to RAM ratio is key, in others the disk to memory ratio and type of disk is critical. Significantly deviating from these ratios may seem like a way to save on hardware costs, but may result in an {{es}} cluster that does not scale and perform well.
+In the links provided above, Elastic has performance tested hardware for each of the cloud providers to find the optimal hardware for each node type. We use ratios to represent the best mix of CPU, RAM, and disk for each type. In some cases the CPU to RAM ratio is key, in others the disk to memory ratio and type of disk is critical. Significantly deviating from these ratios may seem like a way to save on hardware costs, but may result in an {{es}} cluster that does not scale and perform well.
 
 This table shows our specific recommendations for nodes in a Hot/Frozen architecture.
 
 |     |     |     |     |     |
 | --- | --- | --- | --- | --- |
-| **Type** | **AWS*** | ***Azure*** | ***GCP** | **Physical** |
+| **Type** | **AWS** | **Azure** | **GCP** | **Physical** |
 | ![Hot data node](../../images/reference-architectures-hot.png "") | c6gd | f32sv2 | N2 | 16-32 vCPU<br>64 GB RAM<br>2-6 TB NVMe SSD |
 | ![Frozen data node](../../images/reference-architectures-frozen.png "") | i3en | e8dsv4 | N2 | 8 vCPU<br>64 GB RAM<br>6-20+ TB NVMe SSD<br>Depending on days cached |
 | ![Machine learning node](../../images/reference-architectures-machine-learning.png "") | m6gd | f16sv2 | N2 | 16 vCPU<br>64 GB RAM<br>256 GB SSD |
@@ -82,7 +87,7 @@ This table shows our specific recommendations for nodes in a Hot/Frozen architec
 
 **Snapshots:**
 
-* If auditable or business critical events are being logged, a backup is necessary.  The choice to back up data will depend on each individual business’s needs and requirements. Refer to our [snapshot repository](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html) documentation to learn more.
+* If auditable or business critical events are being logged, a backup is necessary. The choice to back up data will depend on each individual business’s needs and requirements. Refer to our [snapshot repository](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html) documentation to learn more.
 * To automate snapshots and attach to Index lifecycle management policies, refer to [SLM (Snapshot lifecycle management)](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-take-snapshot.html#automate-snapshots-slm).
 
 **Kibana:**

explore-analyze/machine-learning/nlp/ml-nlp-ner-example.md

@@ -113,7 +113,7 @@ Using the example text "Elastic is headquartered in Mountain View, California.",
 
 ## Add the NER model to an {{infer}} ingest pipeline [ex-ner-ingest]
 
-You can perform bulk {{infer}} on documents as they are ingested by using an [{{infer}} processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/inference-processor.html) in your ingest pipeline. The novel *Les Misérables* by Victor Hugo is used as an example for {{infer}} in the following example. [Download](https://github.com/elastic/stack-docs/blob/8.5/docs/en/stack/ml/nlp/data/les-miserables-nd.json) the novel text split by paragraph as a JSON file, then upload it by using the [Data Visualizer](../../../manage-data/ingest.md#upload-data-kibana). Give the new index the name `les-miserables` when uploading the file.
+You can perform bulk {{infer}} on documents as they are ingested by using an [{{infer}} processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/inference-processor.html) in your ingest pipeline. The novel *Les Misérables* by Victor Hugo is used as an example for {{infer}} in the following example. [Download](https://github.com/elastic/stack-docs/blob/8.5/docs/en/stack/ml/nlp/data/les-miserables-nd.json) the novel text split by paragraph as a JSON file, then upload it by using the [Data Visualizer](../../../manage-data/ingest/tools/upload-data-files.md). Give the new index the name `les-miserables` when uploading the file.
 
 Now create an ingest pipeline either in the [Stack management UI](ml-nlp-inference.md#ml-nlp-inference-processor) or by using the API:
 

explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md

@@ -103,7 +103,7 @@ In this step, you load the data that you later use in an ingest pipeline to get
 
 The data set `msmarco-passagetest2019-top1000` is a subset of the MS MARCO Passage Ranking data set used in the testing stage of the 2019 TREC Deep Learning Track. It contains 200 queries and for each query a list of relevant text passages extracted by a simple information retrieval (IR) system. From that data set, all unique passages with their IDs have been extracted and put into a [tsv file](https://github.com/elastic/stack-docs/blob/8.5/docs/en/stack/ml/nlp/data/msmarco-passagetest2019-unique.tsv), totaling 182469 passages. In the following, this file is used as the example data set.
 
-Upload the file by using the [Data Visualizer](../../../manage-data/ingest.md#upload-data-kibana). Name the first column `id` and the second one `text`. The index name is `collection`. After the upload is done, you can see an index named `collection` with 182469 documents.
+Upload the file by using the [Data Visualizer](../../../manage-data/ingest/tools/upload-data-files.md). Name the first column `id` and the second one `text`. The index name is `collection`. After the upload is done, you can see an index named `collection` with 182469 documents.
 
 :::{image} ../../../images/machine-learning-ml-nlp-text-emb-data.png
 :alt: Importing the data

manage-data/ingest.md

@@ -10,29 +10,41 @@ mapped_urls:
 
 # Ingestion
 
-% What needs to be done: Finish draft
+Bring your data! Whether you call it *adding*, *indexing*, or *ingesting* data, you have to get the data into {{es}} before you can search it, visualize it, and use it for insights.
 
-% GitHub issue: docs-projects#326
+Our ingest tools are flexible, and support a wide range of scenarios. We can help you with everything from popular and straightforward use cases, all the way to advanced use cases that require additional processing in order to modify or reshape your data before it goes to {{es}}.
 
-% Scope notes: Brief introduction on use cases  Importance of data ingestion theory / process  how to frame these products as living independently from ES?  Link to reference architectures
+You can ingest:
 
-% Use migrated content from existing pages that map to this page:
+* **General content** (data without timestamps), such as HTML pages, catalogs, and files
+* **Time series (timestamped) data**, such as logs, metrics, and traces for Elastic Security, Observability, Search solutions, or for your own custom solutions
 
-% - [ ] ./raw-migrated-files/cloud/cloud/ec-cloud-ingest-data.md
-%      Notes: Use draft Overview from Karen's PR
-% - [ ] ./raw-migrated-files/kibana/kibana/connect-to-elasticsearch.md
-%      Notes: Other existing pages might be used in the "Plan" section
-% - [ ] ./raw-migrated-files/docs-content/serverless/elasticsearch-ingest-your-data.md
-% - [ ] https://www.elastic.co/customer-success/data-ingestion
-% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/es-ingestion-overview.md
-% - [ ] ./raw-migrated-files/ingest-docs/ingest-overview/ingest-intro.md
 
-% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc):
+## Ingesting general content [ingest-general]
 
-$$$upload-data-kibana$$$
+Elastic offer tools designed to ingest specific types of general content. The content type determines the best ingest option.
 
-$$$_add_sample_data$$$
+* To index **documents** directly into {{es}}, use the {{es}} [document APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs.html).
+* To send **application data** directly to {{es}}, use an [{{es}} language client](https://www.elastic.co/guide/en/elasticsearch/client/index.html).
+* To index **web page content**, use the Elastic [web crawler](https://www.elastic.co/web-crawler).
+* To sync **data from third-party sources**, use [connectors](https://www.elastic.co/guide/en/elasticsearch/reference/current/es-connectors.html). A connector syncs content from an original data source to an {{es}} index. Using connectors you can create *searchable*, read-only replicas of your data sources.
+* To index **single files** for testing in a non-production environment, use the {{kib}} [file uploader](ingest/tools/upload-data-files.md).
 
-$$$ec-ingest-methods$$$
+If you would like to try things out before you add your own data, try using our [sample data](ingest/sample-data.md).
 
-$$$ec-data-ingest-pipeline$$$
+
+## Ingesting time series data [ingest-time-series]
+
+::::{admonition} What’s the best approach for ingesting time series data?
+The best approach for ingesting data is the *simplest option* that *meets your needs* and *satisfies your use case*.
+
+In most cases, the *simplest option* for ingesting time series data is using {{agent}} paired with an Elastic integration.
+
+* Install [Elastic Agent](https://www.elastic.co/guide/en/fleet/current) on the computer(s) from which you want to collect data.
+* Add the [Elastic integration](https://docs.elastic.co/en/integrations) for the data source to your deployment.
+
+Integrations are available for many popular platforms and services, and are a good place to start for ingesting data into Elastic solutions—​Observability, Security, and Search—​or your own search application.
+
+Check out the [Integration quick reference](https://docs.elastic.co/en/integrations/all_integrations) to search for available integrations. If you don’t find an integration for your data source or if you need additional processing to extend the integration, we still have you covered. Refer to [Transform and enrich data](ingest/transform-enrich.md) to learn more.
+
+::::

manage-data/ingest/ingest-reference-architectures/use-case-arch.md

@@ -16,7 +16,7 @@ You can host {{es}} on your own hardware or send your data to {{es}} on {{ecloud
 
 **Decision tree**
 
-[Data ingestion pipeline with decision tree](https://www.elastic.co/guide/en/cloud/current/ec-cloud-ingest-data.html#ec-data-ingest-pipeline)
+[Data ingestion](../../ingest.md)
 
 | **Ingest architecture** | **Use when** |
 | --- | --- |

manage-data/ingest/ingesting-data-from-applications/ingest-logs-from-nodejs-web-application-using-filebeat.md

@@ -567,5 +567,5 @@ You can add titles to the visualizations, resize and position them as you like,
 
 2. As your final step, remember to stop Filebeat, the Node.js web server, and the client. Enter *CTRL + C* in the terminal window for each application to stop them.
 
-You now know how to monitor log files from a Node.js web application, deliver the log event data securely into an {{ech}} or {{ece}} deployment, and then visualize the results in Kibana in real time. Consult the [Filebeat documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html) to learn more about the ingestion and processing options available for your data. You can also explore our [documentation](../../../manage-data/ingest.md#ec-ingest-methods) to learn all about ingesting data.
+You now know how to monitor log files from a Node.js web application, deliver the log event data securely into an {{ech}} or {{ece}} deployment, and then visualize the results in Kibana in real time. Consult the [Filebeat documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html) to learn more about the ingestion and processing options available for your data. You can also explore our [documentation](../../../manage-data/ingest.md) to learn all about ingesting data.
 

manage-data/ingest/ingesting-data-from-applications/ingest-logs-from-python-application-using-filebeat.md

@@ -446,5 +446,5 @@ You can add titles to the visualizations, resize and position them as you like,
 
 2. As your final step, remember to stop Filebeat and the Python script. Enter *CTRL + C* in both your Filebeat terminal and in your `elvis.py` terminal.
 
-You now know how to monitor log files from a Python application, deliver the log event data securely into an {{ech}} or {{ece}} deployment, and then visualize the results in Kibana in real time. Consult the [Filebeat documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html) to learn more about the ingestion and processing options available for your data. You can also explore our [documentation](../../../manage-data/ingest.md#ec-ingest-methods) to learn all about all about ingesting data.
+You now know how to monitor log files from a Python application, deliver the log event data securely into an {{ech}} or {{ece}} deployment, and then visualize the results in Kibana in real time. Consult the [Filebeat documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html) to learn more about the ingestion and processing options available for your data. You can also explore our [documentation](../../../manage-data/ingest.md) to learn all about all about ingesting data.
 

manage-data/ingest/ingesting-timeseries-data.md

@@ -3,7 +3,7 @@ mapped_pages:
   - https://www.elastic.co/guide/en/ingest-overview/current/ingest-tools.html
 ---
 
-# Ingesting timeseries data [ingest-tools]
+# Ingesting time series data [ingest-tools]
 
 Elastic and others offer tools to help you get your data from the original data source into {{es}}. Some tools are designed for particular data sources, and others are multi-purpose.
 

manage-data/ingest/tools/upload-data-files.md

@@ -4,11 +4,16 @@ mapped_urls:
   - https://www.elastic.co/guide/en/kibana/current/connect-to-elasticsearch.html#upload-data-kibana
 ---
 
-# Upload data files
+# Upload data files [upload-data-kibana]
 
 % What needs to be done: Align serverless/stateful
 
 % Use migrated content from existing pages that map to this page:
 
 % - [ ] ./raw-migrated-files/docs-content/serverless/elasticsearch-ingest-data-file-upload.md
-% - [ ] ./raw-migrated-files/kibana/kibana/connect-to-elasticsearch.md
+% - [ ] ./raw-migrated-files/kibana/kibana/connect-to-elasticsearch.md
+
+
+
+% Note from David: I've removed the ID $$$upload-data-kibana$$$ from manage-data/ingest.md as those links should instead point to this page. So, please ensure that the following ID is included on this page. I've added it beside the title.
+