diff --git a/deploy-manage/images/cloud-autoops-how-to-access.png b/deploy-manage/images/cloud-autoops-how-to-access.png deleted file mode 100644 index de58f68e86..0000000000 Binary files a/deploy-manage/images/cloud-autoops-how-to-access.png and /dev/null differ diff --git a/deploy-manage/images/search-ai-lake-breakdown-table.png b/deploy-manage/images/search-ai-lake-breakdown-table.png new file mode 100644 index 0000000000..f130316e9e Binary files /dev/null and b/deploy-manage/images/search-ai-lake-breakdown-table.png differ diff --git a/deploy-manage/images/search-ai-lake-project-level-features.png b/deploy-manage/images/search-ai-lake-project-level-features.png new file mode 100644 index 0000000000..24792e4e34 Binary files /dev/null and b/deploy-manage/images/search-ai-lake-project-level-features.png differ diff --git a/deploy-manage/images/search-tier-breakdown-table.png b/deploy-manage/images/search-tier-breakdown-table.png new file mode 100644 index 0000000000..b9d07931d4 Binary files /dev/null and b/deploy-manage/images/search-tier-breakdown-table.png differ diff --git a/deploy-manage/images/search-tier-project-level-features.png b/deploy-manage/images/search-tier-project-level-features.png new file mode 100644 index 0000000000..83b0efd576 Binary files /dev/null and b/deploy-manage/images/search-tier-project-level-features.png differ diff --git a/deploy-manage/monitor.md b/deploy-manage/monitor.md index b28c058a17..e33059a02c 100644 --- a/deploy-manage/monitor.md +++ b/deploy-manage/monitor.md @@ -3,6 +3,7 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/monitor-elasticsearch-cluster.html - https://www.elastic.co/guide/en/cloud/current/ec-monitoring.html applies_to: + serverless: deployment: ess: all ece: all @@ -36,6 +37,7 @@ The following sections provide more details. ### AutoOps (recommended) ```{applies_to} +serverless: deployment: ess: self: @@ -45,7 +47,7 @@ deployment: AutoOps diagnoses issues in {{es}} by analyzing hundreds of metrics, providing root-cause analysis and accurate resolution paths. With AutoOps, customers can prevent and resolve issues, cut down administration time, and optimize resource utilization. -In the [regions](/deploy-manage/monitor/autoops/ec-autoops-regions.md) where it has been rolled out, AutoOps is automatically available in [{{ech}} deployments](/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md), and can be set up for [ECE, ECK, and self-managed clusters](/deploy-manage/monitor/autoops/cc-autoops-as-cloud-connected.md). +In the [regions](/deploy-manage/monitor/autoops/ec-autoops-regions.md) where it has been rolled out, AutoOps is automatically available in [{{ech}} deployments](/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md) and [{{serverless-full}} projects](/deploy-manage/monitor/autoops/autoops-for-serverless.md), and can be set up for [ECE, ECK, and self-managed clusters](/deploy-manage/monitor/autoops/cc-autoops-as-cloud-connected.md). ### Stack monitoring diff --git a/deploy-manage/monitor/autoops.md b/deploy-manage/monitor/autoops.md index c295fe0ac9..b7d8a7b343 100644 --- a/deploy-manage/monitor/autoops.md +++ b/deploy-manage/monitor/autoops.md @@ -2,8 +2,8 @@ mapped_pages: - https://www.elastic.co/guide/en/cloud/current/ec-autoops.html applies_to: - deployment: serverless: + deployment: ess: all self: ece: @@ -48,7 +48,7 @@ AutoOps diagnoses issues in {{es}} by analyzing hundreds of metrics, providing r ## AutoOps availability -In the [regions](/deploy-manage/monitor/autoops/ec-autoops-regions.md) where it has been rolled out, AutoOps is automatically available in [{{ech}} deployments](/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md) and can be set up for [ECE, ECK, and self-managed clusters](/deploy-manage/monitor/autoops/cc-autoops-as-cloud-connected.md). +In the [regions](/deploy-manage/monitor/autoops/ec-autoops-regions.md) where it has been rolled out, AutoOps is automatically available in [{{ech}} deployments](/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md) and [{{serverless-full}} projects](/deploy-manage/monitor/autoops/autoops-for-serverless.md), and can be set up for [ECE, ECK, and self-managed clusters](/deploy-manage/monitor/autoops/cc-autoops-as-cloud-connected.md). AutoOps is currently not available for air-gapped environments since it is a cloud service and you need an internet connection to send metrics to {{ecloud}}. However, we plan to offer a locally deployable version in the future. @@ -65,7 +65,8 @@ AutoOps currently monitors only {{es}}, not the entire {{stack}}. Any deployment In this section, you'll find the following information: -* How to [access AutoOps in your {{ech}} deployments](/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md). +* How to [use AutoOps in your {{ech}} deployments](/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md). +* How to [use AutoOps in your {{serverless-full}} projects](/deploy-manage/monitor/autoops/autoops-for-serverless.md). * How to [connect your ECE, ECK, or self-managed clusters](/deploy-manage/monitor/autoops/cc-autoops-as-cloud-connected.md) to AutoOps. * [Regions](/deploy-manage/monitor/autoops/ec-autoops-regions.md) where AutoOps is available. * What [events](/deploy-manage/monitor/autoops/ec-autoops-events.md) are and how you can configure [event settings](/deploy-manage/monitor/autoops/ec-autoops-event-settings.md) and [notifications](/deploy-manage/monitor/autoops/ec-autoops-notifications-settings.md). diff --git a/deploy-manage/monitor/autoops/access-autoops-for-serverless.md b/deploy-manage/monitor/autoops/access-autoops-for-serverless.md new file mode 100644 index 0000000000..cdee9f5f5f --- /dev/null +++ b/deploy-manage/monitor/autoops/access-autoops-for-serverless.md @@ -0,0 +1,18 @@ +--- +applies_to: + serverless: +navigation_title: Access AutoOps in your project +--- + +# How to access AutoOps in your serverless project + +To access AutoOps in your serverless project, follow these steps: + +1. Log in to your {{ecloud}} account. +2. Locate the serverless project you want to work on and select **Manage**. +3. On the project overview page, in the **Usage and Performance** section, select **View in AutoOps**. + +:::{note} +AutoOps for {{serverless-full}} is only available in supported [regions](ec-autoops-regions.md#autoops-for-serverless-full-regions), with the exception of the **Search AI Lake** view, which is available in all CSP regions across AWS, Azure, and GCP. +::: + diff --git a/deploy-manage/monitor/autoops/autoops-for-serverless.md b/deploy-manage/monitor/autoops/autoops-for-serverless.md new file mode 100644 index 0000000000..9e58d66924 --- /dev/null +++ b/deploy-manage/monitor/autoops/autoops-for-serverless.md @@ -0,0 +1,83 @@ +--- +applies_to: + serverless: +navigation_title: For {{serverless-full}} +--- + +# AutoOps for {{serverless-full}} + +For [{{serverless-full}}](/deploy-manage/deploy/elastic-cloud/serverless.md) projects, AutoOps is set up and enabled automatically in all supported [regions](ec-autoops-regions.md#autoops-for-serverless-full-regions). More regions are coming soon. + +## How AutoOps monitors your serverless project + +In your {{serverless-full}} project, Elastic ensures that the appropriate resources are adequately provisioned and autoscaled so that your workloads run efficiently. This is why {{serverless-full}} is billed based on the effective usage of compute and storage resources. + +:::{note} +For more information about how {{serverless-full}} is priced and packaged, refer to the following pages: +* [{{serverless-full}} pricing page](https://www.elastic.co/pricing/serverless-search) +* [{{serverless-full}} pricing and packaging blog post](https://www.elastic.co/blog/elastic-cloud-serverless-pricing-packaging) +::: + +Since your monthly serverless bill is directly related to how many resources have been consumed, it's important for you to understand why your consumption fluctuates and how past usage was influenced by your project's performance. This information lets you adapt your workloads accordingly and have better control over your future bills. + +This is where AutoOps comes in. With AutoOps for serverless, you can: + +* understand and monitor your usage patterns through project-level and index-level performance metrics. +* access several curated dashboards to look at your project from all the different angles. +* have full visibility into the main serverless billing dimensions. + +## AutoOps for serverless billing dimensions + +AutoOps for serverless focuses on different [billing dimensions](/deploy-manage/cloud-organization/billing/serverless-project-billing-dimensions.md) related to compute and storage, which are explained in the following subsections. + +### Compute billing dimensions +On [{{es-serverless}} projects](/deploy-manage/cloud-organization/billing/elasticsearch-billing-dimensions.md), the main compute-related billing dimension is called a **Virtual Compute Unit (VCU)**. 1 VCU contains 1GB of RAM and the corresponding vCPU and local storage for caching. + +There are three main types of VCUs: +* **Search VCUs** powering the search tier, which handles all search operations. +* **Indexing VCUs** powering the indexing tier, which handles all data indexing operations. +* **Machine learning VCUs** powering the machine learning tier, which handles all ML-related operations such as inference, anomaly detection, data frame analytics, transforms, and more. + +VCUs materialize the load that each of the above tiers has to sustain to respond to your search, indexing, and machine learning needs respectively. As the load of a given tier fluctuates above or below some pre-defined thresholds, the tier will autoscale accordingly to accommodate that load. + +:::{note} +For more information about how autoscaling works in serverless, refer to the following blog posts: +* [Search tier autoscaling](https://www.elastic.co/search-labs/blog/elasticsearch-serverless-tier-autoscaling) +* [Ingest autoscaling](https://www.elastic.co/search-labs/blog/elasticsearch-ingest-autoscaling) +::: + +:::{admonition} Example: How search VCU billing is calculated +Let's say your constant search workload requires 4GB of RAM, which means your search VCU usage for one day will be 4 search VCUs/hour * 24 hours = 96 VCUs. + +Given that 1 search VCU = [$0.09/hour](https://www.elastic.co/pricing/serverless-search), this translates to $8.64 for that day. +::: + +### Storage billing dimensions + +On [Observability](/deploy-manage/cloud-organization/billing/elastic-observability-billing-dimensions.md) and [Security](/deploy-manage/cloud-organization/billing/security-billing-dimensions.md) Serverless projects, one storage-related billing dimension is called the **Ingest rate**, which represents the volume of data (in GB) ingested per unit of time. + +On all [{{es}}](/deploy-manage/cloud-organization/billing/elasticsearch-billing-dimensions.md), [Observability](/deploy-manage/cloud-organization/billing/elastic-observability-billing-dimensions.md), and [Security](/deploy-manage/cloud-organization/billing/security-billing-dimensions.md) Serverless projects, the main storage-related billing dimension is called **Storage retained** or **Retention**, and it represents the total volume of data (in GB prorated over a month) retained in your project. + +:::{admonition} Example: How ingest rate and storage retained billing is calculated +Let’s say you ingest 1TB of data into your Observability project. + +* **Ingest rate**: Given that 1GB ingested per hour = [$0.105](https://www.elastic.co/pricing/serverless-observability), your ingest rate cost will be $107.2. +* **Retention**: Given that 1GB retained per hour = [$0.018](https://www.elastic.co/pricing/serverless-observability) and assuming it took one hour to ingest 1TB of data, that 1TB will be billed 1.42GB for that slice of one hour (1TB/720 hours per month), which translates to $0.025. Each subsequent hour in that month will cost the same. +::: + +## Coming soon + +The following features are coming soon to AutoOps for Serverless: + +* An **Indexing tier** view, which will show you how indexing performance influences your use of ingest VCUs. +* A **Machine learning tier** view, which will provide insight into your machine learning jobs and inference performance, as well as token usage. +* Visibility into other billing dimensions such as data transfer out of {{ecloud}} and the various Observability and Security add-ons. + +## Section overview + +In this section, you'll find the following information: + +* How to [access AutoOps in your serverless project](access-autoops-for-serverless.md). +* How to use the [Search tier view](search-tier-view-autoops-serverless.md) to see the impact of search performance on your use of search VCUs. +* How to use the [Search AI Lake view](search-ai-lake-view-autoops-serverless.md) to drill down into your storage-related usage. + diff --git a/deploy-manage/monitor/autoops/cc-autoops-as-cloud-connected.md b/deploy-manage/monitor/autoops/cc-autoops-as-cloud-connected.md index 2773f3291f..f2ab1b6897 100644 --- a/deploy-manage/monitor/autoops/cc-autoops-as-cloud-connected.md +++ b/deploy-manage/monitor/autoops/cc-autoops-as-cloud-connected.md @@ -12,7 +12,7 @@ products: # AutoOps for self-managed clusters -Use [AutoOps](/deploy-manage/monitor/autoops.md) with your ECE ({{ece}}), ECK ({{eck}}), or self-managed clusters through [Cloud Connect](/deploy-manage/cloud-connect.md). +For ECE ({{ece}}), ECK ({{eck}}), and self-managed clusters, AutoOps can be set up in all supported [regions](ec-autoops-regions.md#autoops-for-self-managed-clusters-regions) through [Cloud Connect](/deploy-manage/cloud-connect.md). More regions are coming soon. Cloud Connect enables users of ECE, ECK, and self-managed clusters to use {{ecloud}} services. This means you can take advantage of the simplified cluster monitoring, real-time issue detection, and performance recommendations of AutoOps without having to run and manage the underlying infrastructure. diff --git a/deploy-manage/monitor/autoops/ec-autoops-faq.md b/deploy-manage/monitor/autoops/ec-autoops-faq.md index 786407247a..5415513754 100644 --- a/deploy-manage/monitor/autoops/ec-autoops-faq.md +++ b/deploy-manage/monitor/autoops/ec-autoops-faq.md @@ -23,7 +23,7 @@ $$$faq-what-is-autoops$$$What does AutoOps do? : AutoOps for {{es}} significantly simplifies cluster management with performance recommendations, resource utilization and cost insights, real-time issue detection and resolution paths. By analyzing hundreds of {{es}} metrics, your configuration, and usage patterns, AutoOps recommends operational and monitoring insights that deliver savings in administration time and hardware costs. $$$faq-autoops-deployment-types$$$Is AutoOps available in all types of deployments? -: In the [regions](ec-autoops-regions.md) where it has been rolled out, AutoOps is automatically available in [{{ech}} deployments](/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md) and can be set up for [ECE, ECK, and self-managed clusters](/deploy-manage/monitor/autoops/cc-autoops-as-cloud-connected.md). +: In the [regions](ec-autoops-regions.md) where it has been rolled out, AutoOps is automatically available in [{{ech}} deployments](/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md) and [{{serverless-full}} projects](/deploy-manage/monitor/autoops/autoops-for-serverless.md), and can be set up for [ECE, ECK, and self-managed clusters](/deploy-manage/monitor/autoops/cc-autoops-as-cloud-connected.md). $$$faq-autoops-regions$$$Why can't I see AutoOps in some deployments? : AutoOps is rolling out in phases across CSPs and [regions](ec-autoops-regions.md), so you may not see it if your deployment is in a region where AutoOps is not available yet. diff --git a/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md b/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md index 5267446c6d..9effbcf3ed 100644 --- a/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md +++ b/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md @@ -11,7 +11,7 @@ navigation_title: For {{ech}} # AutoOps for {{ech}} -For {{ech}} deployments, AutoOps is set up and enabled automatically in all supported [regions](ec-autoops-regions.md#autoops-for-ech-regions). More regions are coming soon. +For [{{ech}}](/deploy-manage/deploy/elastic-cloud/cloud-hosted.md) deployments, AutoOps is set up and enabled automatically in all supported [regions](ec-autoops-regions.md#autoops-for-ech-regions). More regions are coming soon. ## How to access AutoOps [ec-autoops-how-to-access] @@ -20,7 +20,3 @@ To access AutoOps from your {{ecloud}} console, follow these steps: 1. Log in to your {{ech}} account. 2. Locate the deployment you want to work on and select **Manage**. 4. On the deployment overview page, select **Open AutoOps**. - -:::{image} /deploy-manage/images/cloud-autoops-how-to-access.png -:alt: How to access AutoOps -::: diff --git a/deploy-manage/monitor/autoops/ec-autoops-regions.md b/deploy-manage/monitor/autoops/ec-autoops-regions.md index 019d02123a..5a6315b8a2 100644 --- a/deploy-manage/monitor/autoops/ec-autoops-regions.md +++ b/deploy-manage/monitor/autoops/ec-autoops-regions.md @@ -25,7 +25,7 @@ AutoOps is currently not available in any region for GovCloud customers. ## AutoOps for {{ECH}} regions -AutoOps for {{ECH}} is currently available in the following regions for AWS: +AutoOps for {{ECH}} is set up and enabled automatically in the following regions for AWS: | Region | Name | | --- | --- | --- | --- | @@ -53,18 +53,11 @@ AutoOps for {{ECH}} is currently available in the following regions for AWS: Regions for Azure and GCP are coming soon. -## AutoOps for self-managed clusters regions - -You can also use AutoOps with your ECE ({{ece}}), ECK ({{eck}}), or self-managed clusters through [Cloud Connect](/deploy-manage/cloud-connect.md). - -This service is currently available in the following regions for AWS: - -:::{include} ../_snippets/autoops-cc-regions.md -::: +Learn how to [access](/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md) AutoOps in your {{ECH}} deployment. ## AutoOps for {{serverless-full}} regions -AutoOps for serverless projects is currently available in the following regions for AWS: +AutoOps for Serverless is set up and enabled automatically in the following regions for AWS: | Region | Name | | --- | --- | --- | --- | @@ -73,4 +66,17 @@ AutoOps for serverless projects is currently available in the following regions | ap-southeast-1 | Asia Pacific (Singapore) | | us-west-2 | US West (Oregon) | -The only exception is the **Search AI Lake** view, which is available in all CSP regions across AWS, Azure and GCP. +The only exception is the **Search AI Lake** view, which is available in all CSP regions across AWS, Azure, and GCP. + +Learn how to [access](/deploy-manage/monitor/autoops/access-autoops-for-serverless.md) AutoOps in your serverless project. + +## AutoOps for self-managed clusters regions + +You can also use AutoOps with your ECE ({{ece}}), ECK ({{eck}}), or self-managed clusters through [Cloud Connect](/deploy-manage/cloud-connect.md). + +This service is currently available in the following regions for AWS: + +:::{include} ../_snippets/autoops-cc-regions.md +::: + +Learn how to [set up](/deploy-manage/monitor/autoops/cc-connect-self-managed-to-autoops.md) AutoOps in your ECE, ECK, or self-managed cluster. diff --git a/deploy-manage/monitor/autoops/search-ai-lake-view-autoops-serverless.md b/deploy-manage/monitor/autoops/search-ai-lake-view-autoops-serverless.md new file mode 100644 index 0000000000..03944ef55a --- /dev/null +++ b/deploy-manage/monitor/autoops/search-ai-lake-view-autoops-serverless.md @@ -0,0 +1,62 @@ +--- +applies_to: + serverless: +navigation_title: Search AI Lake view +--- + +# Search AI Lake view in AutoOps for Serverless + +The Search AI Lake view in AutoOps for Serverless provides visibility into the key [storage billing dimensions](/deploy-manage/monitor/autoops/autoops-for-serverless.md#storage-billing-dimensions) that drive the costs of your serverless projects. This view helps you understand how ingest and storage activities contribute to your storage usage costs with both high-level summaries and detailed index-level and data stream-levels breakdowns. + +The Search AI Lake view is available in all regions across AWS, Azure, and GCP. + +To get to this view, [access AutoOps](/deploy-manage/monitor/autoops/access-autoops-for-serverless.md) in your project and then select **Search AI Lake** from the navigation menu. + +## Project-level insights + +{applies_to}`observability:` {applies_to}`security:` On the **Search AI Lake** page, the top half of the screen offers project-level insights into the ingest rate and storage retained usage metrics over a selected time period. + +:::{image} /deploy-manage/images/search-ai-lake-project-level-features.png +:screenshot: +:alt: Screenshot showing the features in the top half of the Search AI Lake page +::: + +Use the following features to explore this view: +* Use the built-in **project picker** to switch between projects. This allows you to make quick context changes without needing to navigate back to your {{ecloud}} home page to select a different project. +* Select **custom time windows** let to explore usage data ranging from the last 3 hours to the last 10 days. The charted data is bucketed per day except when you select a period of up to 24 hours, when it is bucketed per hour. + +{applies_to}`elasticsearch:` {{es-serverless}} projects offer the same experience, except that unlike Observability and Security Serverless projects, they only focus on storage retained and not on ingest rate usage metrics. + +## Index and data-stream level insights + +{applies_to}`observability:` {applies_to}`security:` The bottom half of the screen offers a more granular breakdown table of index-level and data stream-level insights into ingest rate and storage retained metrics. + +:::{image} /deploy-manage/images/search-ai-lake-breakdown-table.png +:screenshot: +:alt: Screenshot showing an expanded row in the table in the bottom half of the Search AI Lake page +::: + +Each row of the table represents a single index or data stream and provides the following information: +* the **aggregated ingest rate** for the selected time period +* the **latest recorded storage retained value** during that period +* the timestamp of the **latest update** for these usage metrics + +For historical analysis, you can also expand each row to reveal usage trends over time, helping you detect patterns or anomalies in data growth or ingest activity. + +Also, this table is interactive and can be: + +* filtered by index or data stream name. +* sorted by name, ingest rate, storage retained, or latest update time. +* paginated to handle large sets of indices or data streams. + +{applies_to}`elasticsearch:` {{es-serverless}} projects offer the same experience, except that unlike Observability and Security Serverless projects, they only focus on storage retained and not on ingest rate usage metrics. + +## Factors affecting storage consumption + +The Search AI Lake view shows you how your project's ingest rate and storage retention changes over time. This section explains what might be causing these changes so you can make adjustments to manage your consumption. + +The main factor that influences storage consumption is the data retention duration set on your data streams. A longer retention period means more storage space needs to be allocated to accommodate that retention. + +A long data retention duration combined with a high ingest rate will consume even more storage. Let's say your project is ingesting a significantly large amount of data in a certain time period, causing your ingest rate to increase. If your data retention duration is not adjusted accordingly, you will require even more storage to store the additional data. + +This is why you should adjust your data retention duration to fit your requirements so that you can make the most effective use of your storage. \ No newline at end of file diff --git a/deploy-manage/monitor/autoops/search-tier-view-autoops-serverless.md b/deploy-manage/monitor/autoops/search-tier-view-autoops-serverless.md new file mode 100644 index 0000000000..d0e8cd9785 --- /dev/null +++ b/deploy-manage/monitor/autoops/search-tier-view-autoops-serverless.md @@ -0,0 +1,96 @@ +--- +applies_to: + serverless: +navigation_title: Search Tier view +--- + +# Search Tier view in AutoOps for Serverless + +The Search Tier view in AutoOps for Serverless provides visibility into the consumption of search VCUs, which are a type of [compute billing dimension](/deploy-manage/monitor/autoops/autoops-for-serverless.md#compute-billing-dimensions). This view helps you understand how search activities and performance contribute to your search VCU consumption and as a result, your project's bill. + +This view provides both high-level project summaries and detailed index-level and data stream-level breakdowns. + +To get to the Search Tier view, [access AutoOps](/deploy-manage/monitor/autoops/access-autoops-for-serverless.md) in your project and then select **Search Tier** from the navigation menu. + +## Project-level insights + +On the **Search Tier** page, the top half of the screen offers general insights at the project level. + +:::{image} /deploy-manage/images/search-tier-project-level-features.png +:screenshot: +:alt: Screenshot showing the features in the top half of the Search Tier page +::: + +Use the following features to explore this view: +* Use the built-in **project picker** to switch between projects. This allows you to make quick context changes without needing to navigate back to your {{ecloud}} home page to select a different project. +* Select **custom time windows** let to explore usage and performance data ranging from the last 3 hours to the last 10 days. The charted data is bucketed per day except when you select a period of up to 24 hours, when it is bucketed per hour. +* Explore different **visualizations** presenting the trend of search VCU usage over time and how it compares to the performance of the search tier in terms of search rate and latency. +* View the **annotations** overlaying the search VCUs usage chart to understand when the search power and boost window changed during the selected time period and how that might have affected the autoscaling of your project (and consequently your VCU consumption). +* Gain insights from the **performance charts** depicting search rate and search latency trends to understand why your VCU consumption might fluctuate over time. + +## Index and data stream-level insights + +The bottom half of the screen offers a more granular breakdown table of index-level and data stream-level insights into search performance. + +:::{image} /deploy-manage/images/search-tier-breakdown-table.png +:screenshot: +:alt: Screenshot showing an expanded row in the data set table in the bottom half of the Search Tier page +::: + +Each row of the table represents a single index or data stream and provides the following information: +* The **number of documents** in the index or data stream. +* The latest **search rate** in the selected time period. +* The latest **search latency** in the selected time period. +* The timestamp of the **last search** on the index or data stream. + +Using this table, you can detect which of your indices or data streams are currently being searched and at what rate and latency. This helps you identify which indices are suffering from search pressure, so that you can deduce where that load is coming from manage it to optimize your search VCU consumption. + +For historical analysis, you can also expand each row to reveal performance trends over time, helping you detect patterns or anomalies in search performance for each index and data stream individually. + +Also, this table is interactive and can be: + +* filtered by index or data stream name. +* sorted by index or data stream name, documents count, search rate, search latency, or last searched time. +* paginated to handle large sets of indices or data streams. + +## Factors affecting search VCU consumption +The Search Tier view shows you how many search VCUs are consumed in your project and how your usage changes over time. This section explains the possible factors behind these changes so you can adjust them to manage your consumption. + +The consumption of search VCUs is directly related to autoscaling. When your project is upscaled, more VCUs are consumed, and when your project is downscaled, fewer VCUs are consumed. + +The following factors may cause upscaling or downscaling and consequently an increase or decrease in the number of search VCUs consumed: + +### Search rate +A higher search rate will lead to a larger search load, which means the project will be upscaled and more search VCUs will be consumed. Similarly, a smaller search load means fewer search VCUs being consumed. + +The search rate on your project can increase for many different reasons, such as when more clients start issuing search requests at the same time, or when a complex dashboard with many visualizations is configured with a low auto-refresh rate. + +When that happens, the search tier will try to respond to all requests as quickly as possible, but might not be able to serve them all with the currently allocated compute power. As a result, search requests will start backing up in the queue and the search latency will start rising. The search load will eventually reach a point that will trigger upscaling of the search tier, leading to higher consumption of search VCUs. + +### Search latency + +Alternatively, the search rate on your project may remain steady, but the search latency may increase because some computationally heavy search queries have been executing for several minutes, preventing the search tier from serving the newer search queries. + +This could be caused by a number of reasons: + +* A user may be sending complex full-text queries including regular expressions or leading wildcards +* A dashboard may be issuing search queries running on a very large time frame including non-search-ready data +* Index mappings may be inefficient or they may be defining too many fields, causing higher memory consumption + +As a result, the search tier gets slowly saturated and the new search queries get queued up waiting for the long-running ones to terminate. This increase in search latency can trigger upscaling and in turn increase your search VCU consumption. Low search latency means decreased search VCU consumption. + +:::{admonition} Coming soon to AutoOps +We plan to display long-running search queries in the Search Tier view so that you can learn which queries are causing increased search latency and improve their performance. +::: + +### Project settings +Increasing the search power, boost window, or retention period in your project will cause upscaling, which consumes more search VCUs. Decreasing these settings will lead to lower consumption of VCUs. + +:::{note} +The search boost window and retention period settings are only applicable to time series data. +::: + +### Data ingestion rate +If your project settings are constant but your project is ingesting and retaining more data over time, there will be more data that needs to be search-ready or "boosted". + +This will cause upscaling, leading to higher consumption of VCUs. In the same way, less data ingestion and retention will cause downscaling and so fewer search VCUs will be consumed. diff --git a/deploy-manage/monitor/stack-monitoring.md b/deploy-manage/monitor/stack-monitoring.md index 1f21151c16..db3b3ea957 100644 --- a/deploy-manage/monitor/stack-monitoring.md +++ b/deploy-manage/monitor/stack-monitoring.md @@ -20,7 +20,7 @@ products: :::{admonition} Simplify monitoring with AutoOps Use [AutoOps](/deploy-manage/monitor/autoops.md) in your {{ech}}, ECE, ECK, or self-managed deployments. -AutoOps is a monitoring tool that simplifies cluster management through performance recommendations, resource utilization visibility, and real-time issue detection with resolution paths. In the [regions](/deploy-manage/monitor/autoops/ec-autoops-regions.md) where it has been rolled out, AutoOps is automatically available in [{{ech}} deployments](/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md) and can be set up for [ECE, ECK, and self-managed clusters](/deploy-manage/monitor/autoops/cc-autoops-as-cloud-connected.md). +AutoOps is a monitoring tool that simplifies cluster management through performance recommendations, resource utilization visibility, and real-time issue detection with resolution paths. In the [regions](/deploy-manage/monitor/autoops/ec-autoops-regions.md) where it has been rolled out, AutoOps is automatically available in [{{ech}} deployments](/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md) and [{{serverless-full}} projects](/deploy-manage/monitor/autoops/autoops-for-serverless.md), and can be set up for [ECE, ECK, and self-managed clusters](/deploy-manage/monitor/autoops/cc-autoops-as-cloud-connected.md). To help you make your decision, refer to [](/deploy-manage/monitor/autoops-vs-stack-monitoring.md). ::: diff --git a/deploy-manage/toc.yml b/deploy-manage/toc.yml index fd26f6fad2..383ca74071 100644 --- a/deploy-manage/toc.yml +++ b/deploy-manage/toc.yml @@ -662,6 +662,11 @@ toc: - file: monitor/autoops.md children: - file: monitor/autoops/ec-autoops-how-to-access.md + - file: monitor/autoops/autoops-for-serverless.md + children: + - file: monitor/autoops/access-autoops-for-serverless.md + - file: monitor/autoops/search-tier-view-autoops-serverless.md + - file: monitor/autoops/search-ai-lake-view-autoops-serverless.md - file: monitor/autoops/cc-autoops-as-cloud-connected.md children: - file: monitor/autoops/cc-connect-self-managed-to-autoops.md