diff --git a/raw-migrated-files/docs-content/serverless/collect-data-with-aws-firehose.md b/raw-migrated-files/docs-content/serverless/collect-data-with-aws-firehose.md deleted file mode 100644 index 33313bb360..0000000000 --- a/raw-migrated-files/docs-content/serverless/collect-data-with-aws-firehose.md +++ /dev/null @@ -1,139 +0,0 @@ -# Quickstart: Collect data with AWS Firehose [collect-data-with-aws-firehose] - -::::{warning} -This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. -:::: - - -In this quickstart guide, you’ll learn how to use AWS Firehose to send logs and metrics to Elastic. - -The AWS Firehose streams are created using a CloudFormation template, which can collect all available CloudWatch logs and metrics for your AWS account. - -This approach requires minimal configuration as the CloudFormation template creates a Firehose stream, enables CloudWatch metrics collection across all namespaces, and sets up an account-level subscription filter for CloudWatch log groups to send logs to Elastic via Firehose. You can use an AWS CLI command or upload the template to the AWS CloudFormation portal to customize the following parameter values: - -::::{dropdown} Required Input Parameters -* `ElasticEndpointURL`: Elastic endpoint URL. -* `ElasticAPIKey`: Elastic API Key. - -:::: - - -::::{dropdown} Optional Input Parameters -* `HttpBufferInterval`: The Kinesis Firehose HTTP buffer interval, in seconds. Default is `60`. -* `HttpBufferSize`: The Kinesis Firehose HTTP buffer size, in MiB. Default is `1`. -* `S3BackupMode`: Source record backup in Amazon S3, failed data only or all data. Default is `FailedDataOnly`. -* `S3BufferInterval`: The Kinesis Firehose S3 buffer interval, in seconds. Default is `300`. -* `S3BufferSize`: The Kinesis Firehose S3 buffer size, in MiB. Default is `5`. -* `S3BackupBucketARN`: By default, an S3 bucket for backup will be created. You can override this behaviour by providing an ARN of an existing S3 bucket that ensures the data can be recovered if record processing transformation does not produce the desired results. -* `Attributes`: List of attribute name-value pairs for HTTP endpoint separated by commas. For example "name1=value1,name2=value2". - -:::: - - -::::{dropdown} Optional Input Parameters Specific for Metrics -* `EnableCloudWatchMetrics`: Enable CloudWatch Metrics collection. Default is `true`. When CloudWatch metrics collection is enabled, by default a metric stream will be created with metrics from all namespaces. -* `FirehoseStreamNameForMetrics`: Name for Amazon Data Firehose Stream for collecting CloudWatch metrics. Default is `elastic-firehose-metrics`. -* `IncludeOrExclude`: Select the metrics you want to stream. You can include or exclude specific namespaces and metrics. If no filter namespace is given, then default to all namespaces. Default is `Include`. -* `MetricNameFilters`: Comma-delimited list of namespace-metric names pairs to use for filtering metrics from the stream. If no metric name filter is given, then default to all namespaces and all metrics. For example "AWS/EC2:CPUUtilization|NetworkIn|NetworkOut,AWS/RDS,AWS/S3:AllRequests". -* `IncludeLinkedAccountsMetrics`: If you are creating a metric stream in a monitoring account, specify `true` to include metrics from source accounts that are linked to this monitoring account, in the metric stream. Default is `false`. -* `Tags`: Comma-delimited list of tags to apply to the metric stream. For example "org:eng,project:firehose". - -:::: - - -::::{dropdown} Optional Input Parameters Specific for Logs -* `EnableCloudWatchLogs`: Enable CloudWatch Logs collection. Default is `true`. When CloudWatch logs collection is enabled, an account-level subscription filter policy is created for all CloudWatch log groups (except the log groups created for Firehose logs). -* `FirehoseStreamNameForLogs`: Name for Amazon Data Firehose Stream for collecting CloudWatch logs. Default is `elastic-firehose-logs`. - -:::: - - -::::{important} -Some AWS services need additional manual configuration to properly ingest logs and metrics. For more information, check the [AWS integration](https://www.elastic.co/docs/current/integrations/aws) documentation. -:::: - - -Data collection with AWS Firehose is supported on Amazon Web Services. - - -## Prerequisites [_prerequisites_3] - -* An {{obs-serverless}} project. To learn more, refer to [Create an Observability project](../../../solutions/observability/get-started/create-an-observability-project.md). -* A user with the **Admin** role or higher—required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). -* An active AWS account and the necessary permissions to create delivery streams. - -::::{note} -The default CloudFormation stack is created in the AWS region selected for the user’s account. This region can be modified either through the AWS Console interface or by specifying a `--region` parameter in the AWS CLI command when creating the stack. -:::: - - - -## Limitations [_limitations_2] - -The AWS Firehose receiver has the following limitations: - -* It does not support AWS PrivateLink. -* The CloudFormation template detects and ingests logs and metrics within a single AWS region only. - -The following table shows the type of data ingested by the supported AWS services: - -| AWS Service | Data type | -| --- | --- | -| VPC Flow Logs | Logs | -| API Gateway | Logs, Metrics | -| CloudTrail | Logs | -| Network Firewall | Logs, Metrics | -| Route53 | Logs | -| WAF | Logs | -| DynamoDB | Metrics | -| EBS | Metrics | -| EC2 | Metrics | -| ECS | Metrics | -| ELB | Metrics | -| EMR | Metrics | -| MSK | Metrics | -| Kinesis Data Stream | Metrics | -| Lambda | Metrics | -| NAT Gateway | Metrics | -| RDS | Metrics | -| S3 | Metrics | -| SNS | Metrics | -| SQS | Metrics | -| Transit Gateway | Metrics | -| AWS Usage | Metrics | -| VPN | Metrics | -| Uncategorized Firehose Logs | Logs | - - -## Collect your data [_collect_your_data_3] - -1. [Create a new {{obs-serverless}} project](../../../solutions/observability/get-started/create-an-observability-project.md), or open an existing one. -2. In your {{obs-serverless}} project, go to **Add Data**. -3. Under **What do you want to monitor?** select **Cloud***, ***AWS**, and then select **AWS Firehose**. - - :::{image} ../../../images/serverless-quickstart-aws-firehose-entry-point.png - :alt: AWS Firehose entry point - :class: screenshot - ::: - -4. Click **Create Firehose Stream in AWS** to create a CloudFormation stack from the CloudFormation template. -5. Go back to the **Add Observability Data** page. - - -## Visualize your data [_visualize_your_data_2] - -After installation is complete and all relevant data is flowing into Elastic, the **Visualize your data** section allows you to access the different dashboards for the various services. - -:::{image} ../../../images/serverless-quickstart-aws-firehose-dashboards.png -:alt: AWS Firehose dashboards -:class: screenshot -::: - -Here is an example of the VPC Flow logs dashboard: - -:::{image} ../../../images/serverless-quickstart-aws-firehose-vpc-flow.png -:alt: AWS Firehose VPC flow -:class: screenshot -::: - -Refer to [Observability overview](../../../solutions/observability/get-started/what-is-elastic-observability.md) for a description of other useful features. diff --git a/raw-migrated-files/docs-content/serverless/monitor-k8s-otel-edot.md b/raw-migrated-files/docs-content/serverless/monitor-k8s-otel-edot.md deleted file mode 100644 index 802691b44b..0000000000 --- a/raw-migrated-files/docs-content/serverless/monitor-k8s-otel-edot.md +++ /dev/null @@ -1,73 +0,0 @@ -# Quickstart: Unified Kubernetes Observability with Elastic Distributions of OpenTelemetry (EDOT) [monitor-k8s-otel-edot] - -::::{warning} -This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. -:::: - - -In this quickstart guide, you’ll learn how to send Kubernetes logs, metrics, and application traces to Elasticsearch, using the [OpenTelemetry Operator](https://github.com/open-telemetry/opentelemetry-operator/) to orchestrate [Elastic Distributions of OpenTelemetry](https://github.com/elastic/opentelemetry/tree/main) (EDOT) Collectors and SDK instances. - -All the components will be deployed through the [opentelemetry-kube-stack](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-kube-stack) helm chart. They include: - -* [OpenTelemetry Operator](https://github.com/open-telemetry/opentelemetry-operator/). -* `DaemonSet` EDOT Collector configured for node level metrics. -* `Deployment` EDOT Collector configured for cluster level metrics. -* `Instrumentation` object for applications [auto-instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/). - -For a more detailed description of the components and advanced configuration, refer to the [elastic/opentelemetry](https://github.com/elastic/opentelemetry/blob/main/docs/kubernetes/operator/README.md) GitHub repository. - - -## Prerequisites [_prerequisites_2] - -* An {{obs-serverless}} project. To learn more, refer to [Create an Observability project](../../../solutions/observability/get-started/create-an-observability-project.md). -* A running Kubernetes cluster (v1.23 or newer). -* [Kubectl](https://kubernetes.io/docs/reference/kubectl/). -* [Helm](https://helm.sh/docs/intro/install/). -* (optional) [Cert-manager](https://cert-manager.io/docs/installation/), if you opt for automatic generation and renewal of TLS certificates. - - -## Collect your data [_collect_your_data_2] - -1. [Create a new {{obs-serverless}} project](../../../solutions/observability/get-started/create-an-observability-project.md), or open an existing one. -2. In your {{obs-serverless}} project, go to **Add Data**. -3. Under **What do you want to monitor?** select **Kubernetes**, and then select **OpenTelemetry: Full Observability**. - - :::{image} ../../../images/serverless-quickstart-k8s-otel-entry-point.png - :alt: Kubernetes-OTel entry point - :class: screenshot - ::: - -4. Follow the on-screen instructions to install all needed components. - - ::::{note} - The default installation deploys the OpenTelemetry Operator with a self-signed TLS certificate valid for 365 days. This certificate **won’t be renewed** unless the Helm Chart release is manually updated. Refer to the [cert-manager integrated installation](https://github.com/elastic/opentelemetry/blob/main/docs/kubernetes/operator/README.md#cert-manager) guide to enable automatic certificate generation and renewal using [cert-manager](https://cert-manager.io/docs/installation/). - - :::: - - - Deploy the OpenTelemetry Operator and EDOT Collectors using the kube-stack Helm chart with the provided `values.yaml` file. You will run a few commands to: - - * Add the helm chart repository needed for the installation. - * Create a namespace. - * Create a secret with an API Key and the {{es}} endpoint to be used by the collectors. - * Install the `opentelemetry-kube-stack` helm chart with the provided `values.yaml`. - * Optionally, for instrumenting applications, apply the corresponding `annotations` as shown in {{kib}}. - - - -## Visualize your data [_visualize_your_data] - -After installation is complete and all relevant data is flowing into Elastic, the **Visualize your data** section provides a link to the **[OTEL][Metrics Kubernetes]Cluster Overview** dashboard used to monitor the health of the cluster. - -:::{image} ../../../images/serverless-quickstart-k8s-otel-dashboard.png -:alt: Kubernetes overview dashboard -:class: screenshot -::: - - -## Troubleshooting and more [_troubleshooting_and_more] - -* To troubleshoot deployment and installation, refer to [installation verification](https://github.com/elastic/opentelemetry/tree/main/docs/kubernetes/operator#installation-verification). -* For application instrumentation details, refer to [Instrumenting applications with EDOT SDKs on Kubernetes](https://github.com/elastic/opentelemetry/blob/main/docs/kubernetes/operator/instrumenting-applications.md). -* To customize the configuration, refer to [custom configuration](https://github.com/elastic/opentelemetry/tree/main/docs/kubernetes/operator#custom-configuration). -* Refer to [Observability overview](../../../solutions/observability/get-started/what-is-elastic-observability.md) for a description of other useful features. diff --git a/raw-migrated-files/docs-content/serverless/observability-quickstarts-k8s-logs-metrics.md b/raw-migrated-files/docs-content/serverless/observability-quickstarts-k8s-logs-metrics.md deleted file mode 100644 index 2fdb76655b..0000000000 --- a/raw-migrated-files/docs-content/serverless/observability-quickstarts-k8s-logs-metrics.md +++ /dev/null @@ -1,48 +0,0 @@ -# Quickstart: Monitor your Kubernetes cluster with Elastic Agent [observability-quickstarts-k8s-logs-metrics] - -In this quickstart guide, you’ll learn how to create the Kubernetes resources that are required to monitor your cluster infrastructure. - -This new approach requires minimal configuration and provides you with an easy setup to monitor your infrastructure. You no longer need to download, install, or configure the Elastic Agent, everything happens automatically when you run the kubectl command. - -The kubectl command installs the standalone Elastic Agent in your Kubernetes cluster, downloads all the Kubernetes resources needed to collect metrics from the cluster, and sends it to Elastic. - - -## Prerequisites [observability-quickstarts-k8s-logs-metrics-prerequisites] - -* An {{obs-serverless}} project. To learn more, refer to [Create an Observability project](../../../solutions/observability/get-started/create-an-observability-project.md). -* A user with the **Admin** role or higher—required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). -* A running Kubernetes cluster. -* [Kubectl](https://kubernetes.io/docs/reference/kubectl/). - - -## Collect your data [observability-quickstarts-k8s-logs-metrics-collect-your-data] - -1. [Create a new {{obs-serverless}} project](../../../solutions/observability/get-started/create-an-observability-project.md), or open an existing one. -2. In your {{obs-serverless}} project, go to **Add Data**. -3. Under **What do you want to monitor?** select **Kubernetes**, and then select **Elastic Agent: Logs & Metrics**. - - :::{image} ../../../images/serverless-quickstart-k8s-entry-point.png - :alt: Kubernetes entry point - :class: screenshot - ::: - -4. To install the Elastic Agent on your host, copy and run the install command. - - You will use the kubectl command to download a manifest file, inject user’s API key generated by Kibana, and create the Kubernetes resources. - -5. Go back to the **Add Observability Data** page. There might be a slight delay before data is ingested. When ready, you will see the message **We are monitoring your cluster**. -6. Click **Explore Kubernetes cluster** to navigate to dashboards and explore your data. - - -## Visualize your data [observability-quickstarts-k8s-logs-metrics-visualize-your-data] - -After installation is complete and all relevant data is flowing into Elastic, the **Visualize your data** section allows you to access the Kubernetes Cluster Overview dashboard that can be used to monitor the health of the cluster. - -:::{image} ../../../images/serverless-quickstart-k8s-overview.png -:alt: Kubernetes overview dashboard -:class: screenshot -::: - -Furthermore, you can access other useful prebuilt dashboards for monitoring Kubernetes resources, for example running pods per namespace, as well as the resources they consume, like CPU and memory. - -Refer to [Observability overview](../../../solutions/observability/get-started/what-is-elastic-observability.md) for a description of other useful features. diff --git a/raw-migrated-files/docs-content/serverless/observability-quickstarts-monitor-hosts-with-elastic-agent.md b/raw-migrated-files/docs-content/serverless/observability-quickstarts-monitor-hosts-with-elastic-agent.md deleted file mode 100644 index c180b5ebcb..0000000000 --- a/raw-migrated-files/docs-content/serverless/observability-quickstarts-monitor-hosts-with-elastic-agent.md +++ /dev/null @@ -1,105 +0,0 @@ -# Quickstart: Monitor hosts with {{agent}} [observability-quickstarts-monitor-hosts-with-elastic-agent] - -In this quickstart guide, you’ll learn how to scan your host to detect and collect logs and metrics, then navigate to dashboards to further analyze and explore your observability data. You’ll also learn how to get value out of your observability data. - -To scan your host, you’ll run an auto-detection script that downloads and installs {{agent}}, which is used to collect observability data from the host and send it to Elastic. - -The script also generates an {{agent}} configuration file that you can use with your existing Infrastructure-as-Code tooling. - - -## Prerequisites [observability-quickstarts-monitor-hosts-with-elastic-agent-prerequisites] - -* An {{obs-serverless}} project. To learn more, refer to [Create an Observability project](../../../solutions/observability/get-started/create-an-observability-project.md). -* A user with the **Admin** role or higher—required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). -* Root privileges on the host—required to run the auto-detection script used in this quickstart. - - -## Limitations [observability-quickstarts-monitor-hosts-with-elastic-agent-limitations] - -* The auto-detection script works on Linux and MacOS only. Support for the `lsof` command is also required if you want to detect custom log files. -* If you’ve installed Apache or Nginx in a non-standard location, you’ll need to specify log file paths manually when you run the scan. -* Because Docker Desktop runs in a VM, its logs are not auto-detected. - - -## Collect your data [observability-quickstarts-monitor-hosts-with-elastic-agent-collect-your-data] - -1. [Create a new {{obs-serverless}} project](../../../solutions/observability/get-started/create-an-observability-project.md), or open an existing one. -2. In your {{obs-serverless}} project, go to **Add Data**. -3. Under **What do you want to monitor?** select **Host**, and then select **Elastic Agent: Logs & Metrics**. - - :::{image} ../../../images/serverless-quickstart-monitor-hosts-entry-point.png - :alt: Host monitoring entry point - :class: screenshot - ::: - -4. Copy the install command. - - You’ll run this command to download the auto-detection script, scan your system for observability data, and install {{agent}}. - -5. Open a terminal on the host you want to scan, and run the command. -6. Review the list of log files: - - * Enter `Y` to ingest all the log files listed. - * Enter `n` to either exclude log files or specify additional log paths. Enter `Y` to confirm your selections. - - -When the script is done, you’ll see a message like "{{agent}} is configured and running." - -There might be a slight delay before logs and other data are ingested. - -::::{admonition} Need to scan your host again? -:class: note - -The auto-detection script (`auto_detect.sh`) is downloaded to the directory where you ran the installation command. You can re-run the script on the same host to detect additional logs. The script will scan the host and reconfigure {{agent}} with any additional logs that are found. If the script misses any custom logs, you can add them manually by entering `n` after the script has finished scanning the host. - -:::: - - - -## Visualize your data [observability-quickstarts-monitor-hosts-with-elastic-agent-visualize-your-data] - -After installation is complete and all relevant data is flowing into Elastic, the **Visualize your data** section will show links to assets you can use to analyze your data. Depending on what type of observability data was collected, the page may link to the following integration assets: - -| Integration asset | Description | -| --- | --- | -| **Apache** | Prebuilt dashboard for monitoring Apache HTTP server health using error and access log data. | -| **Custom .log files** | Logs Explorer for analyzing custom logs. | -| **Docker** | Prebuilt dashboard for monitoring the status and health of Docker containers. | -| **MySQL** | Prebuilt dashboard for monitoring MySQl server health using error and access log data. | -| **Nginx** | Prebuilt dashboard for monitoring Nginx server health using error and access log data. | -| **System** | Prebuilt dashboard for monitoring host status and health using system metrics. | -| **Other prebuilt dashboards** | Prebuilt dashboards are also available for systems and services not described here,including PostgreSQL, Redis, HAProxy, Kafka, RabbitMQ, Prometheus, Apache Tomcat, and MongoDB. | - -For example, you can navigate the **Host overview** dashboard to explore detailed metrics about system usage and throughput. Metrics that indicate a possible problem are highlighted in red. - -:::{image} ../../../images/serverless-quickstart-host-overview.png -:alt: Host overview dashboard -:class: screenshot -::: - - -## Get value out of your data [observability-quickstarts-monitor-hosts-with-elastic-agent-get-value-out-of-your-data] - -After using the dashboards to examine your data and confirm you’ve ingested all the host logs and metrics you want to monitor, you can use {{obs-serverless}} to gain deeper insight into your data. - -For host monitoring, the following capabilities and features are recommended: - -* In the [Infrastructure UI](../../../solutions/observability/infra-and-hosts/analyze-infrastructure-host-metrics.md), analyze and compare data collected from your hosts. You can also: - - * [Detect anomalies](../../../solutions/observability/infra-and-hosts/detect-metric-anomalies.md) for memory usage and network traffic on hosts. - * [Create alerts](../../../solutions/observability/incident-management/alerting.md) that notify you when an anomaly is detected or a metric exceeds a given value. - -* In the [Logs Explorer](../../../solutions/observability/logs/logs-explorer.md), search and filter your log data, get information about the structure of log fields, and display your findings in a visualization. You can also: - - * [Monitor log data set quality](../../../solutions/observability/data-set-quality-monitoring.md) to find degraded documents. - * [Run a pattern analysis](../../../solutions/observability/logs/run-pattern-analysis-on-log-data.md) to find patterns in unstructured log messages. - * [Create alerts](../../../solutions/observability/incident-management/alerting.md) that notify you when an Observability data type reaches or exceeds a given value. - -* Use [machine learning and AIOps features](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md) to apply predictive analytics and machine learning to your data: - - * [Detect anomalies](../../../explore-analyze/machine-learning/anomaly-detection.md) by comparing real-time and historical data from different sources to look for unusual, problematic patterns. - * [Analyze log spikes and drops](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#log-rate-analysis). - * [Detect change points](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#change-point-detection) in your time series data. - - -Refer to [Observability overview](../../../solutions/observability/get-started/what-is-elastic-observability.md) for a description of other useful features. diff --git a/raw-migrated-files/docs-content/serverless/observability-serverless-observability-overview.md b/raw-migrated-files/docs-content/serverless/observability-serverless-observability-overview.md deleted file mode 100644 index 698b552175..0000000000 --- a/raw-migrated-files/docs-content/serverless/observability-serverless-observability-overview.md +++ /dev/null @@ -1,129 +0,0 @@ -# Observability overview [observability-serverless-observability-overview] - -{{obs-serverless}} provides granular insights and context into the behavior of applications running in your environments. It’s an important part of any system that you build and want to monitor. Being able to detect and fix root cause events quickly within an observable system is a minimum requirement for any analyst. - -{{obs-serverless}} provides a single stack to unify your logs, metrics, and application traces. Ingest your data directly to your Observability project, where you can further process and enhance the data, before visualizing it and adding alerts. - -:::{image} ../../../images/serverless-serverless-capabilities.svg -:alt: {{obs-serverless}} overview diagram -::: - - -## Log monitoring [apm-overview] - -Analyze log data from your hosts, services, Kubernetes, Apache, and many more. - -In **Logs Explorer** (powered by Discover), you can quickly search and filter your log data, get information about the structure of the fields, and display your findings in a visualization. - -:::{image} ../../../images/serverless-log-explorer-overview.png -:alt: Logs Explorer showing log events -:class: screenshot -::: - -[Learn more about log monitoring →](../../../solutions/observability/logs.md) - - -## Application performance monitoring (APM) [observability-serverless-observability-overview-application-performance-monitoring-apm] - -Instrument your code and collect performance data and errors at runtime by installing APM agents like Java, Go, .NET, and many more. Then use {{obs-serverless}} to monitor your software services and applications in real time: - -* Visualize detailed performance information on your services. -* Identify and analyze errors. -* Monitor host-level and APM agent-specific metrics like JVM and Go runtime metrics. - -The **Service** inventory provides a quick, high-level overview of the health and general performance of all instrumented services. - -:::{image} ../../../images/serverless-services-inventory.png -:alt: Service inventory showing health and performance of instrumented services -:class: screenshot -::: - -[Learn more about Application performance monitoring (APM) →](../../../solutions/observability/apps/application-performance-monitoring-apm.md) - - -## Infrastructure monitoring [metrics-overview] - -Monitor system and service metrics from your servers, Docker, Kubernetes, Prometheus, and other services and applications. - -The **Infrastructure** UI provides a couple ways to view and analyze metrics across your infrastructure: - -The **Infrastructure inventory** page provides a view of your infrastructure grouped by resource type. - -:::{image} ../../../images/serverless-metrics-app.png -:alt: {{infrastructure-app}} in {kib} -:class: screenshot -::: - -The **Hosts** page provides a dashboard-like view of your infrastructure and is backed by an easy-to-use interface called Lens. - -:::{image} ../../../images/serverless-hosts.png -:alt: Screenshot of the Hosts page -:class: screenshot -::: - -From either page, you can view health and performance metrics to get visibility into the overall health of your infrastructure. You can also drill down into details about a specific host, including performance metrics, host metadata, running processes, and logs. - -[Learn more about infrastructure monitoring → ](../../../solutions/observability/infra-and-hosts/analyze-infrastructure-host-metrics.md) - - -## Synthetic monitoring [observability-serverless-observability-overview-synthetic-monitoring] - -Simulate actions and requests that an end user would perform on your site at predefined intervals and in a controlled environment. The end result is rich, consistent, and repeatable data that you can trend and alert on. - -For more information, see [Synthetic monitoring](../../../solutions/observability/apps/synthetic-monitoring.md). - - -## Alerting [observability-serverless-observability-overview-alerting] - -Stay aware of potential issues in your environments with {{obs-serverless}}’s alerting and actions feature that integrates with log monitoring and APM. It provides a set of built-in actions and specific threshold rules and enables central management of all rules. - -On the **Alerts** page, the **Alerts** table provides a snapshot of alerts occurring within the specified time frame. The table includes the alert status, when it was last updated, the reason for the alert, and more. - -:::{image} ../../../images/serverless-observability-alerts-overview.png -:alt: Summary of Alerts on the {{obs-serverless}} overview page -:class: screenshot -::: - -[Learn more about alerting → ](../../../solutions/observability/incident-management/alerting.md) - - -## Service-level objectives (SLOs) [observability-serverless-observability-overview-service-level-objectives-slos] - -Set clear, measurable targets for your service performance, based on factors like availability, response times, error rates, and other key metrics. Then monitor and track your SLOs in real time, using detailed dashboards and alerts that help you quickly identify and troubleshoot issues. - -From the SLO overview list, you can see all of your SLOs and a quick summary of what’s happening in each one: - -:::{image} ../../../images/serverless-slo-dashboard.png -:alt: Dashboard showing list of SLOs -:class: screenshot -::: - -[Learn more about SLOs → ](../../../solutions/observability/incident-management/service-level-objectives-slos.md) - - -## Cases [observability-serverless-observability-overview-cases] - -Collect and share information about observability issues by creating cases. Cases allow you to track key investigation details, add assignees and tags to your cases, set their severity and status, and add alerts, comments, and visualizations. You can also send cases to third-party systems, such as ServiceNow and Jira. - -:::{image} ../../../images/serverless-cases.png -:alt: Screenshot showing list of cases -:class: screenshot -::: - -[Learn more about cases → ](../../../solutions/observability/incident-management/cases.md) - - -## Machine learning and AIOps [observability-serverless-observability-overview-aiops] - -Reduce the time and effort required to detect, understand, investigate, and resolve incidents at scale by leveraging predictive analytics and machine learning: - -* Detect anomalies by comparing real-time and historical data from different sources to look for unusual, problematic patterns. -* Find and investigate the causes of unusual spikes or drops in log rates. -* Detect distribution changes, trend changes, and other statistically significant change points in a metric of your time series data. - -:::{image} ../../../images/serverless-log-rate-analysis.png -:alt: Log rate analysis page showing log rate spike -:class: screenshot -::: - -[Learn more about machine learning and AIOps →](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md) diff --git a/raw-migrated-files/docs-content/serverless/quickstart-monitor-hosts-with-otel.md b/raw-migrated-files/docs-content/serverless/quickstart-monitor-hosts-with-otel.md deleted file mode 100644 index e9112b0e79..0000000000 --- a/raw-migrated-files/docs-content/serverless/quickstart-monitor-hosts-with-otel.md +++ /dev/null @@ -1,76 +0,0 @@ -# Quickstart: Monitor hosts with OpenTelemetry [quickstart-monitor-hosts-with-otel] - -::::{warning} -This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. -:::: - - -In this quickstart guide, you’ll learn how to monitor your hosts using the Elastic Distribution of OpenTelemetry (EDOT) Collector. You’ll also learn how to use {{observability}} features to gain deeper insight into your observability data after collecting it. - - -## Prerequisites [_prerequisites] - -* An {{observability}} project. To learn more, refer to [Create an Observability project](../../../solutions/observability/get-started/create-an-observability-project.md). -* This quickstart is only available for Linux and MacOS systems. -* A user with the **Admin** role or higher—required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). -* Root privileges on the host—required to run the OpenTelemetry collector because of these components: - - * `hostmetrics` receiver to read all system metrics (all processes, memory, etc.). - * `filelog` to allow the collector to read any user or application log files. - - - -## Limitations [_limitations] - -Refer to [Elastic OpenTelemetry Collector limitations](https://github.com/elastic/opentelemetry/blob/main/docs/collector-limitations.md) for known limitations when using the EDOT Collector. - - -## Collect your data [_collect_your_data] - -Follow these steps to collect logs and metrics using the EDOT Collector: - -1. [Create a new {{obs-serverless}} project](../../../solutions/observability/get-started/create-an-observability-project.md), or open an existing one. -2. To open the quickstart, go to **Add Data**. -3. Select **Collect and analyze logs**, and then select **OpenTelemetry**. -4. Under **What do you want to monitor?** select **Host**, and then select **Elastic Agent: Logs & Metrics**. - - :::{image} ../../../images/serverless-quickstart-monitor-hosts-otel-entry-point.png - :alt: Host monitoring entry point - :class: screenshot - ::: - -5. Select the appropriate platform, and complete the following: -6. For **MacOS and Linux**, copy the command, open a terminal on your host, and run the command to download and configure the OpenTelemetry collector. -7. For **Kubernetes**, download the manifest. -8. Copy the command under Step 2: -9. For **MacOS and Linux**, run the command in your terminal to start the EDOT Collector. -10. For **Kubernetes**, run the command from the directory where you downloaded the manifest to install the EDOT Collector on every node of your cluster. - -Logs are collected from setup onward, so you won’t see logs that occurred before starting the EDOT Collector. The default log path is `/var/log/*`. To update the path, modify `otel.yml`. - -Under **Visualize your data**, you’ll see links to **Logs Explorer** to view your logs and **Hosts** to view your host metrics. - - -## Get value out of your data [_get_value_out_of_your_data] - -After using the Hosts page and Discover to confirm you’ve ingested all the host logs and metrics you want to monitor, use Elastic {{observability}} to gain deeper insight into your host data with the following capabilities and features: - -* In the [Infrastructure UI](../../../solutions/observability/infra-and-hosts/analyze-infrastructure-host-metrics.md), analyze and compare data collected from your hosts. You can also: - - * [Detect anomalies](../../../solutions/observability/infra-and-hosts/detect-metric-anomalies.md) for memory usage and network traffic on hosts. - * [Create alerts](../../../solutions/observability/incident-management/create-manage-rules.md) that notify you when an anomaly is detected or a metric exceeds a given value. - -* In the [Logs Explorer](../../../solutions/observability/logs/logs-explorer.md), search and filter your log data, get information about the structure of log fields, and display your findings in a visualization. You can also: - - * [Monitor log data set quality](../../../solutions/observability/data-set-quality-monitoring.md) to find degraded documents. - * [Run a pattern analysis](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#log-pattern-analysis) to find patterns in unstructured log messages. - * [Create alerts](../../../solutions/observability/incident-management/create-manage-rules.md) that notify you when an Observability data type reaches or exceeds a given value. - -* Use [machine learning](../../../explore-analyze/machine-learning/machine-learning-in-kibana.md) to apply predictive analytics to your data: - - * [Detect anomalies](../../../explore-analyze/machine-learning/anomaly-detection.md) by comparing real-time and historical data from different sources to look for unusual, problematic patterns. - * [Analyze log spikes and drops](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#log-rate-analysis). - * [Detect change points](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#change-point-detection) in your time series data. - - -Refer to the [Elastic Observability](../../../solutions/observability.md) for a description of other useful features. diff --git a/raw-migrated-files/observability-docs/observability/collect-data-with-aws-firehose.md b/raw-migrated-files/observability-docs/observability/collect-data-with-aws-firehose.md deleted file mode 100644 index e230bb4daa..0000000000 --- a/raw-migrated-files/observability-docs/observability/collect-data-with-aws-firehose.md +++ /dev/null @@ -1,147 +0,0 @@ -# Quickstart: Collect data with AWS Firehose [collect-data-with-aws-firehose] - -::::{warning} -This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. -:::: - - -In this quickstart guide, you’ll learn how to use AWS Firehose to send logs and metrics to Elastic. - -The AWS Firehose streams are created using a CloudFormation template, which can collect all available CloudWatch logs and metrics for your AWS account. - -This approach requires minimal configuration as the CloudFormation template creates a Firehose stream, enables CloudWatch metrics collection across all namespaces, and sets up an account-level subscription filter for CloudWatch log groups to send logs to Elastic via Firehose. You can use an AWS CLI command or upload the template to the AWS CloudFormation portal to customize the following parameter values: - -::::{dropdown} Required Input Parameters -* `ElasticEndpointURL`: Elastic endpoint URL. -* `ElasticAPIKey`: Elastic API Key. - -:::: - - -::::{dropdown} Optional Input Parameters -* `HttpBufferInterval`: The Kinesis Firehose HTTP buffer interval, in seconds. Default is `60`. -* `HttpBufferSize`: The Kinesis Firehose HTTP buffer size, in MiB. Default is `1`. -* `S3BackupMode`: Source record backup in Amazon S3, failed data only or all data. Default is `FailedDataOnly`. -* `S3BufferInterval`: The Kinesis Firehose S3 buffer interval, in seconds. Default is `300`. -* `S3BufferSize`: The Kinesis Firehose S3 buffer size, in MiB. Default is `5`. -* `S3BackupBucketARN`: By default, an S3 bucket for backup will be created. You can override this behaviour by providing an ARN of an existing S3 bucket that ensures the data can be recovered if record processing transformation does not produce the desired results. -* `Attributes`: List of attribute name-value pairs for HTTP endpoint separated by commas. For example "name1=value1,name2=value2". - -:::: - - -::::{dropdown} Optional Input Parameters Specific for Metrics -* `EnableCloudWatchMetrics`: Enable CloudWatch Metrics collection. Default is `true`. When CloudWatch metrics collection is enabled, by default a metric stream will be created with metrics from all namespaces. -* `FirehoseStreamNameForMetrics`: Name for Amazon Data Firehose Stream for collecting CloudWatch metrics. Default is `elastic-firehose-metrics`. -* `IncludeOrExclude`: Select the metrics you want to stream. You can include or exclude specific namespaces and metrics. If no filter namespace is given, then default to all namespaces. Default is `Include`. -* `MetricNameFilters`: Comma-delimited list of namespace-metric names pairs to use for filtering metrics from the stream. If no metric name filter is given, then default to all namespaces and all metrics. For example "AWS/EC2:CPUUtilization|NetworkIn|NetworkOut,AWS/RDS,AWS/S3:AllRequests". -* `IncludeLinkedAccountsMetrics`: If you are creating a metric stream in a monitoring account, specify `true` to include metrics from source accounts that are linked to this monitoring account, in the metric stream. Default is `false`. -* `Tags`: Comma-delimited list of tags to apply to the metric stream. For example "org:eng,project:firehose". - -:::: - - -::::{dropdown} Optional Input Parameters Specific for Logs -* `EnableCloudWatchLogs`: Enable CloudWatch Logs collection. Default is `true`. When CloudWatch logs collection is enabled, an account-level subscription filter policy is created for all CloudWatch log groups (except the log groups created for Firehose logs). -* `FirehoseStreamNameForLogs`: Name for Amazon Data Firehose Stream for collecting CloudWatch logs. Default is `elastic-firehose-logs`. - -:::: - - -::::{important} -Some AWS services need additional manual configuration to properly ingest logs and metrics. For more information, check the [AWS integration](https://www.elastic.co/docs/current/integrations/aws) documentation. -:::: - - -Data collection with AWS Firehose is supported on ESS deployments in AWS, Azure and GCP. - - -## Prerequisites [_prerequisites_5] - -* A deployment using our hosted {{ess}} on [{{ecloud}}](https://cloud.elastic.co/registration?page=docs&placement=docs-body). The deployment includes an {{es}} cluster for storing and searching your data, and {{kib}} for visualizing and managing your data. -* A user with the `superuser` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) or the privileges required to onboard data. - - ::::{dropdown} Expand to view required privileges - * [**Cluster**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster): `['monitor', 'manage_own_api_key']` - * [**Index**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices): `{ names: ['logs-*-*', 'metrics-*-*'], privileges: ['auto_configure', 'create_doc'] }` - * [**Kibana**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges.md): `{ spaces: ['*'], feature: { fleet: ['all'], fleetv2: ['all'] } }` - - :::: - -* An active AWS account and the necessary permissions to create delivery streams. - -::::{note} -The default CloudFormation stack is created in the AWS region selected for the user’s account. This region can be modified either through the AWS Console interface or by specifying a `--region` parameter in the AWS CLI command when creating the stack. -:::: - - - -## Limitations [_limitations_3] - -The AWS Firehose receiver has the following limitations: - -* It does not support AWS PrivateLink. -* It is not available for on-premise Elastic Stack deployments. -* The CloudFormation template detects and ingests logs and metrics within a single AWS region only. - -The following table shows the type of data ingested by the supported AWS services: - -| AWS Service | Data type | -| --- | --- | -| VPC Flow Logs | Logs | -| API Gateway | Logs, Metrics | -| CloudTrail | Logs | -| Network Firewall | Logs, Metrics | -| Route53 | Logs | -| WAF | Logs | -| DynamoDB | Metrics | -| EBS | Metrics | -| EC2 | Metrics | -| ECS | Metrics | -| ELB | Metrics | -| EMR | Metrics | -| MSK | Metrics | -| Kinesis Data Stream | Metrics | -| Lambda | Metrics | -| NAT Gateway | Metrics | -| RDS | Metrics | -| S3 | Metrics | -| SNS | Metrics | -| SQS | Metrics | -| Transit Gateway | Metrics | -| AWS Usage | Metrics | -| VPN | Metrics | -| Uncategorized Firehose Logs | Logs | - - -## Collect your data [_collect_your_data_5] - -1. In {{kib}}, go to the **Observability** UI and click **Add Data**. -2. Under **What do you want to monitor?** select **Cloud***, ***AWS**, and then select **AWS Firehose**. - - :::{image} ../../../images/observability-quickstart-aws-firehose-entry-point.png - :alt: AWS Firehose entry point - :class: screenshot - ::: - -3. Click **Create Firehose Stream in AWS** to create a CloudFormation stack from the CloudFormation template. -4. Go back to the **Add Observability Data** page. - - -## Visualize your data [_visualize_your_data_4] - -After installation is complete and all relevant data is flowing into Elastic, the **Visualize your data** section allows you to access the different dashboards for the various services. - -:::{image} ../../../images/observability-quickstart-aws-firehose-dashboards.png -:alt: AWS Firehose dashboards -:class: screenshot -::: - -Here is an example of the VPC Flow logs dashboard: - -:::{image} ../../../images/observability-quickstart-aws-firehose-vpc-flow.png -:alt: AWS Firehose VPC flow -:class: screenshot -::: - -Refer to [What is Elastic {{observability}}?](../../../solutions/observability/get-started/what-is-elastic-observability.md) for a description of other useful features. diff --git a/raw-migrated-files/observability-docs/observability/monitor-k8s-logs-metrics-with-elastic-agent.md b/raw-migrated-files/observability-docs/observability/monitor-k8s-logs-metrics-with-elastic-agent.md deleted file mode 100644 index da0f302169..0000000000 --- a/raw-migrated-files/observability-docs/observability/monitor-k8s-logs-metrics-with-elastic-agent.md +++ /dev/null @@ -1,58 +0,0 @@ -# Quickstart: Monitor your Kubernetes cluster with {{agent}} [monitor-k8s-logs-metrics-with-elastic-agent] - -In this quickstart guide, you’ll learn how to create the Kubernetes resources that are required to monitor your cluster infrastructure. - -This new approach requires minimal configuration and provides you with an easy setup to monitor your infrastructure. You no longer need to download, install, or configure the Elastic Agent, everything happens automatically when you run the kubectl command. - -The kubectl command installs the standalone Elastic Agent in your Kubernetes cluster, downloads all the Kubernetes resources needed to collect metrics from the cluster, and sends it to Elastic. - - -## Prerequisites [_prerequisites_2] - -* An {{es}} cluster for storing and searching your data, and {{kib}} for visualizing and managing your data. This quickstart is available for all Elastic deployment models. To get started quickly, try out our hosted {{ess}} on [{{ecloud}}](https://cloud.elastic.co/registration?page=docs&placement=docs-body). -* A user with the `superuser` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) or the privileges required to onboard data. - - ::::{dropdown} Expand to view required privileges - * [**Cluster**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster): `['monitor', 'manage_own_api_key']` - * [**Index**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices): `{ names: ['logs-*-*', 'metrics-*-*'], privileges: ['auto_configure', 'create_doc'] }` - * [**Kibana**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges.md): `{ spaces: ['*'], feature: { fleet: ['all'], fleetv2: ['all'] } }` - - :::: - -* A running Kubernetes cluster. -* [Kubectl](https://kubernetes.io/docs/reference/kubectl/). - - -## Collect your data [_collect_your_data_2] - -1. In {{kib}}, go to the **Observability** UI and click **Add Data**. -2. Under **What do you want to monitor?** select **Kubernetes**, and then select **Elastic Agent: Logs & Metrics**. - - :::{image} ../../../images/observability-quickstart-k8s-entry-point.png - :alt: Kubernetes entry point - :class: screenshot - ::: - -3. To install the Elastic Agent on your host, copy and run the install command. - - You will use the kubectl command to download a manifest file, inject user’s API key generated by {{kib}}, and create the Kubernetes resources. - -4. Go back to the **Add Observability Data** page. - - There might be a slight delay before data is ingested. When ready, you will see the message **We are monitoring your cluster**. - -5. Click **Explore Kubernetes cluster** to navigate to dashboards and explore your data. - - -## Visualize your data [_visualize_your_data_2] - -After installation is complete and all relevant data is flowing into Elastic, the **Visualize your data** section allows you to access the Kubernetes Cluster Overview dashboard that can be used to monitor the health of the cluster. - -:::{image} ../../../images/observability-quickstart-k8s-overview.png -:alt: Kubernetes overview dashboard -:class: screenshot -::: - -Furthermore, you can access other useful prebuilt dashboards for monitoring Kubernetes resources, for example running pods per namespace, as well as the resources they consume, like CPU and memory. - -Refer to [What is Elastic {{observability}}?](../../../solutions/observability/get-started/what-is-elastic-observability.md) for a description of other useful features. diff --git a/raw-migrated-files/observability-docs/observability/monitor-k8s-otel-edot.md b/raw-migrated-files/observability-docs/observability/monitor-k8s-otel-edot.md deleted file mode 100644 index fed03c69f6..0000000000 --- a/raw-migrated-files/observability-docs/observability/monitor-k8s-otel-edot.md +++ /dev/null @@ -1,72 +0,0 @@ -# Quickstart: Unified Kubernetes Observability with Elastic Distributions of OpenTelemetry (EDOT) [monitor-k8s-otel-edot] - -::::{warning} -This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. -:::: - - -In this quickstart guide, you’ll learn how to send Kubernetes logs, metrics, and application traces to Elasticsearch, using the [OpenTelemetry Operator](https://github.com/open-telemetry/opentelemetry-operator/) to orchestrate [Elastic Distributions of OpenTelemetry](https://github.com/elastic/opentelemetry/tree/main) (EDOT) Collectors and SDK instances. - -All the components will be deployed through the [opentelemetry-kube-stack](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-kube-stack) helm chart. They include: - -* [OpenTelemetry Operator](https://github.com/open-telemetry/opentelemetry-operator/). -* `DaemonSet` EDOT Collector configured for node level metrics. -* `Deployment` EDOT Collector configured for cluster level metrics. -* `Instrumentation` object for applications [auto-instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/). - -For a more detailed description of the components and advanced configuration, refer to the [elastic/opentelemetry](https://github.com/elastic/opentelemetry/blob/main/docs/kubernetes/operator/README.md) GitHub repository. - - -## Prerequisites [_prerequisites_4] - -* An {{es}} cluster for storing and searching your data, and {{kib}} for visualizing and managing your data. This quickstart is available for all Elastic deployment models. To get started quickly, try out our hosted {{ess}} on [{{ecloud}}](https://cloud.elastic.co/registration?page=docs&placement=docs-body). -* A running Kubernetes cluster (v1.23 or newer). -* [Kubectl](https://kubernetes.io/docs/reference/kubectl/). -* [Helm](https://helm.sh/docs/intro/install/). -* (optional) [Cert-manager](https://cert-manager.io/docs/installation/), if you opt for automatic generation and renewal of TLS certificates. - - -## Collect your data [_collect_your_data_4] - -1. In {{kib}}, go to the **Observability** UI and click **Add Data**. -2. Under **What do you want to monitor?** select **Kubernetes**, and then select **OpenTelemetry: Full Observability**. - - :::{image} ../../../images/observability-quickstart-k8s-otel-entry-point.png - :alt: Kubernetes-OTel entry point - :class: screenshot - ::: - -3. Follow the on-screen instructions to install all needed components. - - ::::{note} - The default installation deploys the OpenTelemetry Operator with a self-signed TLS certificate valid for 365 days. This certificate **won’t be renewed** unless the Helm Chart release is manually updated. Refer to the [cert-manager integrated installation](https://github.com/elastic/opentelemetry/blob/main/docs/kubernetes/operator/README.md#cert-manager) guide to enable automatic certificate generation and renewal using [cert-manager](https://cert-manager.io/docs/installation/). - - :::: - - - Deploy the OpenTelemetry Operator and EDOT Collectors using the kube-stack Helm chart with the provided `values.yaml` file. You will run a few commands to: - - * Add the helm chart repository needed for the installation. - * Create a namespace. - * Create a secret with an API Key and the {{es}} endpoint to be used by the collectors. - * Install the `opentelemetry-kube-stack` helm chart with the provided `values.yaml`. - * Optionally, for instrumenting applications, apply the corresponding `annotations` as shown in {{kib}}. - - - -## Visualize your data [_visualize_your_data_3] - -After installation is complete and all relevant data is flowing into Elastic, the **Visualize your data** section provides a link to the **[OTEL][Metrics Kubernetes]Cluster Overview** dashboard used to monitor the health of the cluster. - -:::{image} ../../../images/observability-quickstart-k8s-otel-dashboard.png -:alt: Kubernetes overview dashboard -:class: screenshot -::: - - -## Troubleshooting and more [_troubleshooting_and_more] - -* To troubleshoot deployment and installation, refer to [installation verification](https://github.com/elastic/opentelemetry/tree/main/docs/kubernetes/operator#installation-verification). -* For application instrumentation details, refer to [Instrumenting applications with EDOT SDKs on Kubernetes](https://github.com/elastic/opentelemetry/blob/main/docs/kubernetes/operator/instrumenting-applications.md). -* To customize the configuration, refer to [custom configuration](https://github.com/elastic/opentelemetry/tree/main/docs/kubernetes/operator#custom-configuration). -* Refer to [What is Elastic {{observability}}?](../../../solutions/observability/get-started/what-is-elastic-observability.md) for a description of other useful features. diff --git a/raw-migrated-files/observability-docs/observability/observability-introduction.md b/raw-migrated-files/observability-docs/observability/observability-introduction.md deleted file mode 100644 index 4f893f239b..0000000000 --- a/raw-migrated-files/observability-docs/observability/observability-introduction.md +++ /dev/null @@ -1,103 +0,0 @@ -# What is Elastic {{observability}}? [observability-introduction] - -{{observability}} provides granular insights and context into the behavior of applications running in your environments. It’s an important part of any system that you build and want to monitor. Being able to detect and fix root cause events quickly within an observable system is a minimum requirement for any analyst. - -[Elastic {{observability}}](https://www.elastic.co/observability) provides a single stack to unify your logs, infrastructure metrics, application traces, user experience data, synthetics, and universal profiling. Ingest your data directly to {{es}}, where you can further process and enhance the data, before visualizing it and adding alerts in {{kib}}. - -:::{image} ../../../images/observability-what-is-observability.svg -:alt: Elastic {{observability}} overview diagram -::: - - -## Application performance monitoring (APM) [apm-overview] - -Instrument your code and collect performance data and errors at runtime by installing APM agents like Java, Go, .NET, and many more. - -On the {{observability}} **Overview** page, the **Services** chart shows the total number of services running within your environment and the total number of transactions per minute that were captured by the Elastic APM agent instrumenting those services. - -:::{image} ../../../images/observability-apm.png -:alt: Summary of Services on the {{observability}} overview page -:class: screenshot -::: - -You can then drill down into the Applications UI by clicking **Show service inventory** to quickly find the APM traces for underlying services. - -For more information, see [Application performance monitoring (APM)](../../../solutions/observability/apps/application-performance-monitoring-apm.md). - - -## Infrastructure monitoring [metrics-overview] - -Monitor system and service metrics from your servers, Docker, Kubernetes, Prometheus, and other services and applications. - -On the {{observability}} **Overview** page, the **Hosts** table shows your top hosts with the most significant resource footprints. These metrics help you evaluate host efficiency and determine if resource consumption is impacting end users. - -:::{image} ../../../images/observability-metrics-summary.png -:alt: Summary of Hosts on the {{observability}} overview page -:class: screenshot -::: - -You can then drill down into the {{infrastructure-app}} by clicking **Show inventory**. Here you can monitor and filter your data by hosts, pods, containers,or EC2 instances and create custom groupings such as availability zones or namespaces. - -For more information, see [Infrastructure Monitoring](https://www.elastic.co/guide/en/observability/current/monitor-infrastructure-and-hosts.html). - - -## Real user monitoring (RUM) [user-experience-overview] - -Quantify and analyze the perceived performance of your web application with {{user-experience}} data, powered by the APM RUM agent. Unlike testing environments, {{user-experience}} data reflects real-world user experiences. - -On the {{observability}} **Overview** page, the **{{user-experience}}** chart provides a snapshot of core web vitals for the service with the most traffic. - -:::{image} ../../../images/observability-obs-overview-ue.png -:alt: Summary of {{user-experience}} metrics on the {{observability}} overview page -:class: screenshot -::: - -You can then drill down into the {{user-experience}} dashboard by clicking **Show dashboard** too see data by URL, operating system, browser, and location. - -For more information, see [{{user-experience}}](../../../solutions/observability/apps/real-user-monitoring-user-experience.md). - - -## Log monitoring [logs-overview] - -Analyze log data from your hosts, services, Kubernetes, Apache, and many more. - -On the {{observability}} **Overview** page, the **Log Events** chart helps you detect and inspect possible log anomalies across each of your ingested log sources to determine if the log rate is outside of your expected bounds. - -:::{image} ../../../images/observability-log-rate.png -:alt: Summary of Log Events on the {{observability}} overview page -:class: screenshot -::: - -You can then drill down into the {{logs-app}} by clicking **Show log stream** to view a live stream of your logs, and the filter, pin, or highlight the data you need. - -For more information, see [Log monitoring](../../../solutions/observability/logs/explore-logs.md). - - -## Synthetic monitoring [synthetic-monitoring-overview] - -Simulate actions and requests that an end user would perform on your site at predefined intervals and in a controlled environment. The end result is rich, consistent, and repeatable data that you can trend and alert on. - -For more information, see [Synthetic monitoring](../../../solutions/observability/apps/synthetic-monitoring.md). - - -## Universal Profiling [universal-profiling-overview] - -Build stack traces to get visibility into your system without application source code changes or instrumentation. Use flamegraphs to explore system performance and identify the most expensive lines of code, increase CPU resource efficiency, debug performance regressions, and reduce cloud spend. - -For more information, see [Universal Profiling](../../../solutions/observability/infra-and-hosts/universal-profiling.md). - - -## Alerting [alerts-overview] - -Stay aware of potential issues in your environments with {{kib}}’s alerting and actions feature that integrates with the {{logs-app}}, {{infrastructure-app}}, and Applications UI. It provides a set of built-in actions and specific threshold rules and enables central management of all rules from {{kib}} Management. - -On the {{observability}} **Overview** page, the **Alerts** table provides a snapshot of alerts occurring within the specified time frame. The table includes the alert status, when it was last updated, the reason for the alert, and more. - -:::{image} ../../../images/observability-alerts-overview.png -:alt: Summary of Alerts on the {{observability}} overview page -:class: screenshot -::: - -You can then see more details on these alerts by clicking **Show alerts**. - -For more information, see [Alerting](../../../solutions/observability/incident-management/alerting.md). diff --git a/raw-migrated-files/observability-docs/observability/quickstart-monitor-hosts-with-elastic-agent.md b/raw-migrated-files/observability-docs/observability/quickstart-monitor-hosts-with-elastic-agent.md deleted file mode 100644 index aae8ecc571..0000000000 --- a/raw-migrated-files/observability-docs/observability/quickstart-monitor-hosts-with-elastic-agent.md +++ /dev/null @@ -1,112 +0,0 @@ -# Quickstart: Monitor hosts with {{agent}} [quickstart-monitor-hosts-with-elastic-agent] - -In this quickstart guide, you’ll learn how to scan your host to detect and collect logs and metrics, then navigate to dashboards to further analyze and explore your observability data. You’ll also learn how to get value out of your observability data. - -To scan your host, you’ll run an auto-detection script that downloads and installs {{agent}}, which is used to collect observability data from the host and send it to Elastic. - -The script also generates an {{agent}} configuration file that you can use with your existing Infrastructure-as-Code tooling. - - -## Prerequisites [_prerequisites] - -* An {{es}} cluster for storing and searching your data, and {{kib}} for visualizing and managing your data. This quickstart is available for all Elastic deployment models. To get started quickly, try out our hosted {{ess}} on [{{ecloud}}](https://cloud.elastic.co/registration?page=docs&placement=docs-body). -* A user with the `superuser` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) or the privileges required to onboard data. - - ::::{dropdown} Expand to view required privileges - * [**Cluster**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster): `['monitor', 'manage_own_api_key']` - * [**Index**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices): `{ names: ['logs-*-*', 'metrics-*-*'], privileges: ['auto_configure', 'create_doc'] }` - * [**Kibana**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges.md): `{ spaces: ['*'], feature: { fleet: ['all'], fleetv2: ['all'] } }` - - :::: - -* Root privileges on the host—required to run the auto-detection script used in this quickstart. - - -## Limitations [_limitations] - -* The auto-detection script works on Linux and MacOS only. Support for the `lsof` command is also required if you want to detect custom log files. -* If you’ve installed Apache or Nginx in a non-standard location, you’ll need to specify log file paths manually when you run the scan. -* Because Docker Desktop runs in a VM, its logs are not auto-detected. - - -## Collect your data [_collect_your_data] - -1. In {{kib}}, go to the **Observability** UI and click **Add Data**. -2. Under **What do you want to monitor?** select **Host**, and then select **Elastic Agent: Logs & Metrics**. - - :::{image} ../../../images/observability-quickstart-monitor-hosts-entry-point.png - :alt: Host monitoring entry point - :class: screenshot - ::: - -3. Copy the install command. - - You’ll run this command to download the auto-detection script, scan your system for observability data, and install {{agent}}. - -4. Open a terminal on the host you want to scan, and run the command. -5. Review the list of log files: - - * Enter `Y` to ingest all the log files listed. - * Enter `n` to either exclude log files or specify additional log paths. Enter `Y` to confirm your selections. - - -When the script is done, you’ll see a message like "{{agent}} is configured and running." - -There might be a slight delay before logs and other data are ingested. - -::::{admonition} -**Need to scan your host again?** - -The auto-detection script (`auto_detect.sh`) is downloaded to the directory where you ran the installation command. You can re-run the script on the same host to detect additional logs. The script will scan the host and reconfigure {{agent}} with any additional logs that are found. If the script misses any custom logs, you can add them manually by entering `n` after the script has finished scanning the host. - -:::: - - - -## Visualize your data [_visualize_your_data] - -After installation is complete and all relevant data is flowing into Elastic, the **Visualize your data** section will show links to assets you can use to analyze your data. Depending on what type of observability data was collected, the page may link to the following integration assets: - -| Integration asset | Description | -| --- | --- | -| **Apache** | Prebuilt dashboard for monitoring Apache HTTP server health using error and access log data. | -| **Custom .log files** | Logs Explorer for analyzing custom logs. | -| **Docker** | Prebuilt dashboard for monitoring the status and health of Docker containers. | -| **MySQL** | Prebuilt dashboard for monitoring MySQl server health using error and access log data. | -| **Nginx** | Prebuilt dashboard for monitoring Nginx server health using error and access log data. | -| **System** | Prebuilt dashboard for monitoring host status and health using system metrics. | -| **Other prebuilt dashboards** | Prebuilt dashboards are also available for systems and services not described here,including PostgreSQL, Redis, HAProxy, Kafka, RabbitMQ, Prometheus, Apache Tomcat, and MongoDB. | - -For example, you can navigate the **Host overview** dashboard to explore detailed metrics about system usage and throughput. Metrics that indicate a possible problem are highlighted in red. - -:::{image} ../../../images/observability-quickstart-host-overview.png -:alt: Host overview dashboard -:class: screenshot -::: - - -## Get value out of your data [_get_value_out_of_your_data] - -After using the dashboards to examine your data and confirm you’ve ingested all the host logs and metrics you want to monitor, you can use Elastic {{observability}} to gain deeper insight into your data. - -For host monitoring, the following capabilities and features are recommended: - -* In the [Infrastructure UI](https://www.elastic.co/guide/en/observability/current/monitor-infrastructure-and-hosts.html), analyze and compare data collected from your hosts. You can also: - - * [Detect anomalies](../../../solutions/observability/infra-and-hosts/detect-metric-anomalies.md) for memory usage and network traffic on hosts. - * [Create alerts](../../../solutions/observability/incident-management/alerting.md) that notify you when an anomaly is detected or a metric exceeds a given value. - -* In the [Logs Explorer](../../../solutions/observability/logs/logs-explorer.md), search and filter your log data, get information about the structure of log fields, and display your findings in a visualization. You can also: - - * [Monitor log data set quality](../../../solutions/observability/data-set-quality-monitoring.md) to find degraded documents. - * [Run a pattern analysis](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#log-pattern-analysis) to find patterns in unstructured log messages. - * [Create alerts](../../../solutions/observability/incident-management/alerting.md) that notify you when an Observability data type reaches or exceeds a given value. - -* Use [machine learning](../../../explore-analyze/machine-learning/machine-learning-in-kibana.md) to apply predictive analytics to your data: - - * [Detect anomalies](../../../explore-analyze/machine-learning/anomaly-detection.md) by comparing real-time and historical data from different sources to look for unusual, problematic patterns. - * [Analyze log spikes and drops](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#log-rate-analysis). - * [Detect change points](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#change-point-detection) in your time series data. - - -Refer to the [What is Elastic {{observability}}?](../../../solutions/observability/get-started/what-is-elastic-observability.md) for a description of other useful features. diff --git a/raw-migrated-files/observability-docs/observability/quickstart-monitor-hosts-with-otel.md b/raw-migrated-files/observability-docs/observability/quickstart-monitor-hosts-with-otel.md deleted file mode 100644 index 487869560b..0000000000 --- a/raw-migrated-files/observability-docs/observability/quickstart-monitor-hosts-with-otel.md +++ /dev/null @@ -1,81 +0,0 @@ -# Quickstart: Monitor hosts with OpenTelemetry [quickstart-monitor-hosts-with-otel] - -::::{warning} -This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. -:::: - - -In this quickstart guide, you’ll learn how to monitor your hosts using the Elastic Distribution of OpenTelemetry (EDOT) Collector. You’ll also learn how to use {{observability}} features to gain deeper insight into your observability data after collecting it. - - -## Prerequisites [_prerequisites_3] - -* An {{es}} cluster for storing and searching your data, and {{kib}} for visualizing and managing your data. This quickstart is available for all Elastic deployment models. The quickest way to get started with this quickstart is using a trial project on [Elastic serverless](https://docs.elastic.co/serverless/quickstart-monitor-hosts-with-otel.html). -* This quickstart is only available for Linux and MacOS systems. -* A user with the **Admin** role or higher—required to onboard system logs and metrics. To learn more, refer to [User roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md). -* Root privileges on the host—required to run the OpenTelemetry collector because of these components: - - * `hostmetrics` receiver to read all system metrics (all processes, memory, etc.). - * `filelog` to allow the collector to read any user or application log files. - - - -## Limitations [_limitations_2] - -Refer to [Elastic OpenTelemetry Collector limitations](https://github.com/elastic/opentelemetry/blob/main/docs/collector-limitations.md) for known limitations when using the EDOT Collector. - - -## Collect your data [_collect_your_data_3] - -Follow these steps to collect logs and metrics using the EDOT Collector: - -1. In {{kib}}, go to the **Observability** UI and click **Add Data**. -2. Under **What do you want to monitor?** select **Host**, and then select **OpenTelemetry: Logs & Metrics**. - - :::{image} ../../../images/observability-quickstart-monitor-hosts-otel-entry-point.png - :alt: Host monitoring entry point - :class: screenshot - ::: - -3. Select the appropriate platform. -4. Copy the command under step 1, open a terminal on your host, and run the command. - - This command downloads the {{agent}} package, extracts it in a EDOT directory. For example, `elastic-distro-8.16.0-linux-x86_64`. It also adds a sample `otel.yml` configuration file to the directory and updates the storage directory, Elastic endpoint, and API key in the file. - - The default log path is `/var/log/*.log`. To update the path, modify the `otel.yml` in the EDOT directory. - - Find additional sample `otel.yml` configuration files in the EDOT directory in the `otel_samples` folder. - -5. Copy the command under Step 2 and run it in your terminal to start the EDOT Collector. - -::::{note} -Logs are collected from setup onward, so you won’t see logs that occurred before starting the EDOT Collector. -:::: - - -Under **Visualize your data**, you’ll see links to **Logs Explorer** to view your logs and **Hosts** to view your host metrics. - - -## Gain deeper insight into your host data [_gain_deeper_insight_into_your_host_data] - -After using the Hosts page and Discover to confirm you’ve ingested all the host logs and metrics you want to monitor, use Elastic {{observability}} to gain deeper insight into your host data with the following capabilities and features: - -* In the [Infrastructure UI](../../../solutions/observability/infra-and-hosts/analyze-infrastructure-host-metrics.md), analyze and compare data collected from your hosts. You can also: - - * [Detect anomalies](../../../solutions/observability/infra-and-hosts/detect-metric-anomalies.md) for memory usage and network traffic on hosts. - * [Create alerts](../../../solutions/observability/incident-management/alerting.md) that notify you when an anomaly is detected or a metric exceeds a given value. - -* In the [Logs Explorer](../../../solutions/observability/logs/logs-explorer.md), search and filter your log data, get information about the structure of log fields, and display your findings in a visualization. You can also: - - * [Monitor log data set quality](../../../solutions/observability/data-set-quality-monitoring.md) to find degraded documents. - * [Run a pattern analysis](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#log-pattern-analysis) to find patterns in unstructured log messages. - * [Create alerts](../../../solutions/observability/incident-management/alerting.md) that notify you when an Observability data type reaches or exceeds a given value. - -* Use [machine learning](../../../explore-analyze/machine-learning/machine-learning-in-kibana.md) to apply predictive analytics to your data: - - * [Detect anomalies](../../../explore-analyze/machine-learning/anomaly-detection.md) by comparing real-time and historical data from different sources to look for unusual, problematic patterns. - * [Analyze log spikes and drops](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#log-rate-analysis). - * [Detect change points](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#change-point-detection) in your time series data. - - -Refer to the [What is Elastic {{observability}}?](../../../solutions/observability/get-started/what-is-elastic-observability.md) for a description of other useful features. diff --git a/raw-migrated-files/toc.yml b/raw-migrated-files/toc.yml index 15e9580d7c..f642eae757 100644 --- a/raw-migrated-files/toc.yml +++ b/raw-migrated-files/toc.yml @@ -196,7 +196,6 @@ toc: - file: docs-content/serverless/ai-assistant-knowledge-base.md - file: docs-content/serverless/application-and-service-monitoring.md - file: docs-content/serverless/attack-discovery.md - - file: docs-content/serverless/collect-data-with-aws-firehose.md - file: docs-content/serverless/connect-to-byo-llm.md - file: docs-content/serverless/cspm-required-permissions.md - file: docs-content/serverless/detections-logsdb-index-mode-impact.md @@ -220,7 +219,6 @@ toc: - file: docs-content/serverless/ingest-third-party-cloud-security-data.md - file: docs-content/serverless/ingest-wiz-data.md - file: docs-content/serverless/intro.md - - file: docs-content/serverless/monitor-k8s-otel-edot.md - file: docs-content/serverless/observability-add-logs-service-name.md - file: docs-content/serverless/observability-aggregationOptions.md - file: docs-content/serverless/observability-ai-assistant.md @@ -257,11 +255,8 @@ toc: - file: docs-content/serverless/observability-monitor-status-alert.md - file: docs-content/serverless/observability-parse-log-data.md - file: docs-content/serverless/observability-plaintext-application-logs.md - - file: docs-content/serverless/observability-quickstarts-k8s-logs-metrics.md - - file: docs-content/serverless/observability-quickstarts-monitor-hosts-with-elastic-agent.md - file: docs-content/serverless/observability-rateAggregation.md - file: docs-content/serverless/observability-send-application-logs.md - - file: docs-content/serverless/observability-serverless-observability-overview.md - file: docs-content/serverless/observability-slos.md - file: docs-content/serverless/observability-stream-log-files.md - file: docs-content/serverless/observability-triage-slo-burn-rate-breaches.md @@ -272,7 +267,6 @@ toc: - file: docs-content/serverless/project-setting-data.md - file: docs-content/serverless/project-settings-alerts.md - file: docs-content/serverless/project-settings-content.md - - file: docs-content/serverless/quickstart-monitor-hosts-with-otel.md - file: docs-content/serverless/security-about-rules.md - file: docs-content/serverless/security-add-manage-notes.md - file: docs-content/serverless/security-advanced-settings.md @@ -507,7 +501,6 @@ toc: - file: observability-docs/observability/apm-traces.md - file: observability-docs/observability/application-and-service-monitoring.md - file: observability-docs/observability/application-logs.md - - file: observability-docs/observability/collect-data-with-aws-firehose.md - file: observability-docs/observability/configure-settings.md - file: observability-docs/observability/create-alerts-rules.md - file: observability-docs/observability/create-alerts.md @@ -532,14 +525,9 @@ toc: - file: observability-docs/observability/manage-cases.md - file: observability-docs/observability/monitor-datasets.md - file: observability-docs/observability/monitor-infrastructure-and-hosts.md - - file: observability-docs/observability/monitor-k8s-logs-metrics-with-elastic-agent.md - - file: observability-docs/observability/monitor-k8s-otel-edot.md - file: observability-docs/observability/monitor-status-alert.md - file: observability-docs/observability/obs-ai-assistant.md - file: observability-docs/observability/observability-get-started.md - - file: observability-docs/observability/observability-introduction.md - - file: observability-docs/observability/quickstart-monitor-hosts-with-elastic-agent.md - - file: observability-docs/observability/quickstart-monitor-hosts-with-otel.md - file: observability-docs/observability/rate-aggregation.md - file: observability-docs/observability/slo-burn-rate-alert.md - file: observability-docs/observability/slo-create.md diff --git a/solutions/observability/get-started/create-an-observability-project.md b/solutions/observability/get-started/create-an-observability-project.md index 0203e564e1..d9bdf07426 100644 --- a/solutions/observability/get-started/create-an-observability-project.md +++ b/solutions/observability/get-started/create-an-observability-project.md @@ -4,7 +4,7 @@ mapped_pages: - https://www.elastic.co/guide/en/serverless/current/observability-create-an-observability-project.html --- - +% Serverless only # Create an observability project [observability-create-an-observability-project] diff --git a/solutions/observability/get-started/quickstart-collect-data-with-aws-firehose.md b/solutions/observability/get-started/quickstart-collect-data-with-aws-firehose.md index 270335b576..8390984bb2 100644 --- a/solutions/observability/get-started/quickstart-collect-data-with-aws-firehose.md +++ b/solutions/observability/get-started/quickstart-collect-data-with-aws-firehose.md @@ -4,11 +4,196 @@ mapped_urls: - https://www.elastic.co/guide/en/serverless/current/collect-data-with-aws-firehose.html --- -# Quickstart: Collect data with AWS Firehose +# Quickstart: Collect data with AWS Firehose [collect-data-with-aws-firehose] -% What needs to be done: Align serverless/stateful +::::{warning} +This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. +:::: -% Use migrated content from existing pages that map to this page: -% - [ ] ./raw-migrated-files/observability-docs/observability/collect-data-with-aws-firehose.md -% - [ ] ./raw-migrated-files/docs-content/serverless/collect-data-with-aws-firehose.md \ No newline at end of file +In this quickstart guide, you’ll learn how to use AWS Firehose to send logs and metrics to Elastic. + +The AWS Firehose streams are created using a CloudFormation template, which can collect all available CloudWatch logs and metrics for your AWS account. + +This approach requires minimal configuration as the CloudFormation template creates a Firehose stream, enables CloudWatch metrics collection across all namespaces, and sets up an account-level subscription filter for CloudWatch log groups to send logs to Elastic via Firehose. You can use an AWS CLI command or upload the template to the AWS CloudFormation portal to customize the following parameter values: + +::::{dropdown} Required Input Parameters +* `ElasticEndpointURL`: Elastic endpoint URL. +* `ElasticAPIKey`: Elastic API Key. + +:::: + + +::::{dropdown} Optional Input Parameters +* `HttpBufferInterval`: The Kinesis Firehose HTTP buffer interval, in seconds. Default is `60`. +* `HttpBufferSize`: The Kinesis Firehose HTTP buffer size, in MiB. Default is `1`. +* `S3BackupMode`: Source record backup in Amazon S3, failed data only or all data. Default is `FailedDataOnly`. +* `S3BufferInterval`: The Kinesis Firehose S3 buffer interval, in seconds. Default is `300`. +* `S3BufferSize`: The Kinesis Firehose S3 buffer size, in MiB. Default is `5`. +* `S3BackupBucketARN`: By default, an S3 bucket for backup will be created. You can override this behaviour by providing an ARN of an existing S3 bucket that ensures the data can be recovered if record processing transformation does not produce the desired results. +* `Attributes`: List of attribute name-value pairs for HTTP endpoint separated by commas. For example "name1=value1,name2=value2". + +:::: + + +::::{dropdown} Optional Input Parameters Specific for Metrics +* `EnableCloudWatchMetrics`: Enable CloudWatch Metrics collection. Default is `true`. When CloudWatch metrics collection is enabled, by default a metric stream will be created with metrics from all namespaces. +* `FirehoseStreamNameForMetrics`: Name for Amazon Data Firehose Stream for collecting CloudWatch metrics. Default is `elastic-firehose-metrics`. +* `IncludeOrExclude`: Select the metrics you want to stream. You can include or exclude specific namespaces and metrics. If no filter namespace is given, then default to all namespaces. Default is `Include`. +* `MetricNameFilters`: Comma-delimited list of namespace-metric names pairs to use for filtering metrics from the stream. If no metric name filter is given, then default to all namespaces and all metrics. For example "AWS/EC2:CPUUtilization|NetworkIn|NetworkOut,AWS/RDS,AWS/S3:AllRequests". +* `IncludeLinkedAccountsMetrics`: If you are creating a metric stream in a monitoring account, specify `true` to include metrics from source accounts that are linked to this monitoring account, in the metric stream. Default is `false`. +* `Tags`: Comma-delimited list of tags to apply to the metric stream. For example "org:eng,project:firehose". + +:::: + + +::::{dropdown} Optional Input Parameters Specific for Logs +* `EnableCloudWatchLogs`: Enable CloudWatch Logs collection. Default is `true`. When CloudWatch logs collection is enabled, an account-level subscription filter policy is created for all CloudWatch log groups (except the log groups created for Firehose logs). +* `FirehoseStreamNameForLogs`: Name for Amazon Data Firehose Stream for collecting CloudWatch logs. Default is `elastic-firehose-logs`. + +:::: + + +::::{important} +Some AWS services need additional manual configuration to properly ingest logs and metrics. For more information, check the [AWS integration](https://www.elastic.co/docs/current/integrations/aws) documentation. +:::: + + +Data collection with AWS Firehose is supported on ESS deployments in AWS, Azure and GCP. + + +## Prerequisites [_prerequisites_5] + +::::{tab-set} +:group: stack-serverless + +:::{tab-item} Elastic Stack v9 +:sync: stack + +* A deployment using our hosted {{ess}} on [{{ecloud}}](https://cloud.elastic.co/registration?page=docs&placement=docs-body). The deployment includes an {{es}} cluster for storing and searching your data, and {{kib}} for visualizing and managing your data. +* A user with the `superuser` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) or the privileges required to onboard data. + + ::::{dropdown} Expand to view required privileges + * [**Cluster**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster): `['monitor', 'manage_own_api_key']` + * [**Index**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices): `{ names: ['logs-*-*', 'metrics-*-*'], privileges: ['auto_configure', 'create_doc'] }` + * [**Kibana**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges.md): `{ spaces: ['*'], feature: { fleet: ['all'], fleetv2: ['all'] } }` + + :::: + +* An active AWS account and the necessary permissions to create delivery streams. + +::::{note} +The default CloudFormation stack is created in the AWS region selected for the user’s account. This region can be modified either through the AWS Console interface or by specifying a `--region` parameter in the AWS CLI command when creating the stack. +:::: + +::: + +:::{tab-item} Serverless +:sync: serverless + +* An {{obs-serverless}} project. To learn more, refer to [Create an Observability project](../../../solutions/observability/get-started/create-an-observability-project.md). +* A user with the **Admin** role or higher—required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). +* An active AWS account and the necessary permissions to create delivery streams. + +::: + +:::: + + + +## Limitations [_limitations_3] + +The AWS Firehose receiver has the following limitations: + +* It does not support AWS PrivateLink. +* It is not available for on-premise Elastic Stack deployments. +* The CloudFormation template detects and ingests logs and metrics within a single AWS region only. + +The following table shows the type of data ingested by the supported AWS services: + +| AWS Service | Data type | +| --- | --- | +| VPC Flow Logs | Logs | +| API Gateway | Logs, Metrics | +| CloudTrail | Logs | +| Network Firewall | Logs, Metrics | +| Route53 | Logs | +| WAF | Logs | +| DynamoDB | Metrics | +| EBS | Metrics | +| EC2 | Metrics | +| ECS | Metrics | +| ELB | Metrics | +| EMR | Metrics | +| MSK | Metrics | +| Kinesis Data Stream | Metrics | +| Lambda | Metrics | +| NAT Gateway | Metrics | +| RDS | Metrics | +| S3 | Metrics | +| SNS | Metrics | +| SQS | Metrics | +| Transit Gateway | Metrics | +| AWS Usage | Metrics | +| VPN | Metrics | +| Uncategorized Firehose Logs | Logs | + + +## Collect your data [_collect_your_data_5] + +::::{tab-set} +:group: stack-serverless + +:::{tab-item} Elastic Stack v9 +:sync: stack + +1. In {{kib}}, go to the **Observability** UI and click **Add Data**. +2. Under **What do you want to monitor?** select **Cloud***, ***AWS**, and then select **AWS Firehose**. + + :::{image} ../../../images/observability-quickstart-aws-firehose-entry-point.png + :alt: AWS Firehose entry point + :class: screenshot + ::: + +3. Click **Create Firehose Stream in AWS** to create a CloudFormation stack from the CloudFormation template. +4. Go back to the **Add Observability Data** page. + + +::: + +:::{tab-item} Serverless +:sync: serverless + +1. [Create a new {{obs-serverless}} project](../../../solutions/observability/get-started/create-an-observability-project.md), or open an existing one. +2. In your {{obs-serverless}} project, go to **Add Data**. +3. Under **What do you want to monitor?** select **Cloud***, ***AWS**, and then select **AWS Firehose**. + + :::{image} ../../../images/serverless-quickstart-aws-firehose-entry-point.png + :alt: AWS Firehose entry point + :class: screenshot + ::: + +4. Click **Create Firehose Stream in AWS** to create a CloudFormation stack from the CloudFormation template. +5. Go back to the **Add Observability Data** page. + +::: + +:::: + +## Visualize your data [_visualize_your_data_4] + +After installation is complete and all relevant data is flowing into Elastic, the **Visualize your data** section allows you to access the different dashboards for the various services. + +:::{image} ../../../images/observability-quickstart-aws-firehose-dashboards.png +:alt: AWS Firehose dashboards +:class: screenshot +::: + +Here is an example of the VPC Flow logs dashboard: + +:::{image} ../../../images/observability-quickstart-aws-firehose-vpc-flow.png +:alt: AWS Firehose VPC flow +:class: screenshot +::: + +Refer to [What is Elastic {{observability}}?](../../../solutions/observability/get-started/what-is-elastic-observability.md) for a description of other useful features. \ No newline at end of file diff --git a/solutions/observability/get-started/quickstart-monitor-hosts-with-elastic-agent.md b/solutions/observability/get-started/quickstart-monitor-hosts-with-elastic-agent.md index 6f38ade1c9..8a37edd80f 100644 --- a/solutions/observability/get-started/quickstart-monitor-hosts-with-elastic-agent.md +++ b/solutions/observability/get-started/quickstart-monitor-hosts-with-elastic-agent.md @@ -4,11 +4,168 @@ mapped_urls: - https://www.elastic.co/guide/en/serverless/current/observability-quickstarts-monitor-hosts-with-elastic-agent.html --- -# Quickstart: Monitor hosts with Elastic Agent +# Quickstart: Monitor hosts with {{agent}} [quickstart-monitor-hosts-with-elastic-agent] -% What needs to be done: Align serverless/stateful +In this quickstart guide, you’ll learn how to scan your host to detect and collect logs and metrics, then navigate to dashboards to further analyze and explore your observability data. You’ll also learn how to get value out of your observability data. -% Use migrated content from existing pages that map to this page: +To scan your host, you’ll run an auto-detection script that downloads and installs {{agent}}, which is used to collect observability data from the host and send it to Elastic. -% - [ ] ./raw-migrated-files/observability-docs/observability/quickstart-monitor-hosts-with-elastic-agent.md -% - [ ] ./raw-migrated-files/docs-content/serverless/observability-quickstarts-monitor-hosts-with-elastic-agent.md \ No newline at end of file +The script also generates an {{agent}} configuration file that you can use with your existing Infrastructure-as-Code tooling. + + +## Prerequisites [_prerequisites] + +::::{tab-set} +:group: stack-serverless + +:::{tab-item} Elastic Stack v9 +:sync: stack + +* An {{es}} cluster for storing and searching your data, and {{kib}} for visualizing and managing your data. This quickstart is available for all Elastic deployment models. To get started quickly, try out our hosted {{ess}} on [{{ecloud}}](https://cloud.elastic.co/registration?page=docs&placement=docs-body). +* A user with the `superuser` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) or the privileges required to onboard data. + + ::::{dropdown} Expand to view required privileges + * [**Cluster**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster): `['monitor', 'manage_own_api_key']` + * [**Index**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices): `{ names: ['logs-*-*', 'metrics-*-*'], privileges: ['auto_configure', 'create_doc'] }` + * [**Kibana**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges.md): `{ spaces: ['*'], feature: { fleet: ['all'], fleetv2: ['all'] } }` + + :::: + +* Root privileges on the host—required to run the auto-detection script used in this quickstart. + +::: + +:::{tab-item} Serverless +:sync: serverless + +* An {{obs-serverless}} project. To learn more, refer to [Create an Observability project](../../../solutions/observability/get-started/create-an-observability-project.md). +* A user with the **Admin** role or higher—required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). +* Root privileges on the host—required to run the auto-detection script used in this quickstart. + +::: + +:::: + + +## Limitations [_limitations] + +* The auto-detection script works on Linux and MacOS only. Support for the `lsof` command is also required if you want to detect custom log files. +* If you’ve installed Apache or Nginx in a non-standard location, you’ll need to specify log file paths manually when you run the scan. +* Because Docker Desktop runs in a VM, its logs are not auto-detected. + + +## Collect your data [_collect_your_data] + +::::{tab-set} +:group: stack-serverless + +:::{tab-item} Elastic Stack v9 +:sync: stack + +1. In {{kib}}, go to the **Observability** UI and click **Add Data**. +2. Under **What do you want to monitor?** select **Host**, and then select **Elastic Agent: Logs & Metrics**. + + :::{image} ../../../images/observability-quickstart-monitor-hosts-entry-point.png + :alt: Host monitoring entry point + :class: screenshot + ::: + +3. Copy the install command. + + You’ll run this command to download the auto-detection script, scan your system for observability data, and install {{agent}}. + +4. Open a terminal on the host you want to scan, and run the command. +5. Review the list of log files: + + * Enter `Y` to ingest all the log files listed. + * Enter `n` to either exclude log files or specify additional log paths. Enter `Y` to confirm your selections. + +::: + +:::{tab-item} Serverless +:sync: serverless + +1. [Create a new {{obs-serverless}} project](../../../solutions/observability/get-started/create-an-observability-project.md), or open an existing one. +2. In your {{obs-serverless}} project, go to **Add Data**. +3. Under **What do you want to monitor?** select **Host**, and then select **Elastic Agent: Logs & Metrics**. + + :::{image} ../../../images/serverless-quickstart-monitor-hosts-entry-point.png + :alt: Host monitoring entry point + :class: screenshot + ::: + +4. Copy the install command. + + You’ll run this command to download the auto-detection script, scan your system for observability data, and install {{agent}}. + +5. Open a terminal on the host you want to scan, and run the command. +6. Review the list of log files: + + * Enter `Y` to ingest all the log files listed. + * Enter `n` to either exclude log files or specify additional log paths. Enter `Y` to confirm your selections. + +::: + +:::: + + +When the script is done, you’ll see a message like "{{agent}} is configured and running." + +There might be a slight delay before logs and other data are ingested. + +::::{admonition} Need to scan your host again? +:class: note + +The auto-detection script (`auto_detect.sh`) is downloaded to the directory where you ran the installation command. You can re-run the script on the same host to detect additional logs. The script will scan the host and reconfigure {{agent}} with any additional logs that are found. If the script misses any custom logs, you can add them manually by entering `n` after the script has finished scanning the host. + +:::: + + + +## Visualize your data [_visualize_your_data] + +After installation is complete and all relevant data is flowing into Elastic, the **Visualize your data** section will show links to assets you can use to analyze your data. Depending on what type of observability data was collected, the page may link to the following integration assets: + +| Integration asset | Description | +| --- | --- | +| **Apache** | Prebuilt dashboard for monitoring Apache HTTP server health using error and access log data. | +| **Custom .log files** | Logs Explorer for analyzing custom logs. | +| **Docker** | Prebuilt dashboard for monitoring the status and health of Docker containers. | +| **MySQL** | Prebuilt dashboard for monitoring MySQl server health using error and access log data. | +| **Nginx** | Prebuilt dashboard for monitoring Nginx server health using error and access log data. | +| **System** | Prebuilt dashboard for monitoring host status and health using system metrics. | +| **Other prebuilt dashboards** | Prebuilt dashboards are also available for systems and services not described here,including PostgreSQL, Redis, HAProxy, Kafka, RabbitMQ, Prometheus, Apache Tomcat, and MongoDB. | + +For example, you can navigate the **Host overview** dashboard to explore detailed metrics about system usage and throughput. Metrics that indicate a possible problem are highlighted in red. + +:::{image} ../../../images/observability-quickstart-host-overview.png +:alt: Host overview dashboard +:class: screenshot +::: + + +## Get value out of your data [_get_value_out_of_your_data] + +After using the dashboards to examine your data and confirm you’ve ingested all the host logs and metrics you want to monitor, you can use Elastic {{observability}} to gain deeper insight into your data. + +For host monitoring, the following capabilities and features are recommended: + +* In the [Infrastructure UI](../../../solutions/observability/infra-and-hosts/analyze-infrastructure-host-metrics.md), analyze and compare data collected from your hosts. You can also: + + * [Detect anomalies](../../../solutions/observability/infra-and-hosts/detect-metric-anomalies.md) for memory usage and network traffic on hosts. + * [Create alerts](../../../solutions/observability/incident-management/alerting.md) that notify you when an anomaly is detected or a metric exceeds a given value. + +* In the [Logs Explorer](../../../solutions/observability/logs/logs-explorer.md), search and filter your log data, get information about the structure of log fields, and display your findings in a visualization. You can also: + + * [Monitor log data set quality](../../../solutions/observability/data-set-quality-monitoring.md) to find degraded documents. + * [Run a pattern analysis](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#log-pattern-analysis) to find patterns in unstructured log messages. + * [Create alerts](../../../solutions/observability/incident-management/alerting.md) that notify you when an Observability data type reaches or exceeds a given value. + +* Use [machine learning](../../../explore-analyze/machine-learning/machine-learning-in-kibana.md) to apply predictive analytics to your data: + + * [Detect anomalies](../../../explore-analyze/machine-learning/anomaly-detection.md) by comparing real-time and historical data from different sources to look for unusual, problematic patterns. + * [Analyze log spikes and drops](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#log-rate-analysis). + * [Detect change points](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#change-point-detection) in your time series data. + + +Refer to the [OBservability overview](../../../solutions/observability/get-started/what-is-elastic-observability.md) for a description of other useful features. \ No newline at end of file diff --git a/solutions/observability/get-started/quickstart-monitor-hosts-with-opentelemetry.md b/solutions/observability/get-started/quickstart-monitor-hosts-with-opentelemetry.md index d37147987e..61c133bc71 100644 --- a/solutions/observability/get-started/quickstart-monitor-hosts-with-opentelemetry.md +++ b/solutions/observability/get-started/quickstart-monitor-hosts-with-opentelemetry.md @@ -4,11 +4,140 @@ mapped_urls: - https://www.elastic.co/guide/en/serverless/current/quickstart-monitor-hosts-with-otel.html --- -# Quickstart: Monitor hosts with OpenTelemetry +# Quickstart: Monitor hosts with OpenTelemetry [quickstart-monitor-hosts-with-otel] -% What needs to be done: Align serverless/stateful +::::{warning} +This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. +:::: -% Use migrated content from existing pages that map to this page: -% - [ ] ./raw-migrated-files/observability-docs/observability/quickstart-monitor-hosts-with-otel.md -% - [ ] ./raw-migrated-files/docs-content/serverless/quickstart-monitor-hosts-with-otel.md \ No newline at end of file +In this quickstart guide, you’ll learn how to monitor your hosts using the Elastic Distribution of OpenTelemetry (EDOT) Collector. You’ll also learn how to use {{observability}} features to gain deeper insight into your observability data after collecting it. + + +## Prerequisites [_prerequisites] + +::::{tab-set} +:group: stack-serverless + +:::{tab-item} Elastic Stack v9 +:sync: stack + +* An {{es}} cluster for storing and searching your data, and {{kib}} for visualizing and managing your data. This quickstart is available for all Elastic deployment models. The quickest way to get started with this quickstart is using a trial project on [Elastic serverless](https://docs.elastic.co/serverless/quickstart-monitor-hosts-with-otel.html). +* This quickstart is only available for Linux and MacOS systems. +* A user with the **Admin** role or higher—required to onboard system logs and metrics. To learn more, refer to [User roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md). +* Root privileges on the host—required to run the OpenTelemetry collector because of these components: + + * `hostmetrics` receiver to read all system metrics (all processes, memory, etc.). + * `filelog` to allow the collector to read any user or application log files. + +::: + +:::{tab-item} Serverless +:sync: serverless + +* An {{observability}} project. To learn more, refer to [Create an Observability project](../../../solutions/observability/get-started/create-an-observability-project.md). +* This quickstart is only available for Linux and MacOS systems. +* A user with the **Admin** role or higher—required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). +* Root privileges on the host—required to run the OpenTelemetry collector because of these components: + + * `hostmetrics` receiver to read all system metrics (all processes, memory, etc.). + * `filelog` to allow the collector to read any user or application log files. + +::: + +:::: + + +## Limitations [_limitations] + +Refer to [Elastic OpenTelemetry Collector limitations](https://github.com/elastic/opentelemetry/blob/main/docs/collector-limitations.md) for known limitations when using the EDOT Collector. + + +## Collect your data [_collect_your_data] + +Follow these steps to collect logs and metrics using the EDOT Collector: + +::::{tab-set} +:group: stack-serverless + +:::{tab-item} Elastic Stack v9 +:sync: stack + +1. In {{kib}}, go to the **Observability** UI and click **Add Data**. +2. Under **What do you want to monitor?** select **Host**, and then select **OpenTelemetry: Logs & Metrics**. + + :::{image} ../../../images/observability-quickstart-monitor-hosts-otel-entry-point.png + :alt: Host monitoring entry point + :class: screenshot + ::: + +3. Select the appropriate platform. +4. Copy the command under step 1, open a terminal on your host, and run the command. + + This command downloads the {{agent}} package, extracts it in a EDOT directory. For example, `elastic-distro-8.16.0-linux-x86_64`. It also adds a sample `otel.yml` configuration file to the directory and updates the storage directory, Elastic endpoint, and API key in the file. + + The default log path is `/var/log/*.log`. To update the path, modify the `otel.yml` in the EDOT directory. + + Find additional sample `otel.yml` configuration files in the EDOT directory in the `otel_samples` folder. + +5. Copy the command under Step 2 and run it in your terminal to start the EDOT Collector. + +::::{note} +Logs are collected from setup onward, so you won’t see logs that occurred before starting the EDOT Collector. +:::: + +::: + +:::{tab-item} Serverless +:sync: serverless + +1. [Create a new {{obs-serverless}} project](../../../solutions/observability/get-started/create-an-observability-project.md), or open an existing one. +2. To open the quickstart, go to **Add Data**. +3. Select **Collect and analyze logs**, and then select **OpenTelemetry**. +4. Under **What do you want to monitor?** select **Host**, and then select **Elastic Agent: Logs & Metrics**. + + :::{image} ../../../images/serverless-quickstart-monitor-hosts-otel-entry-point.png + :alt: Host monitoring entry point + :class: screenshot + ::: + +5. Select the appropriate platform, and complete the following: +6. For **MacOS and Linux**, copy the command, open a terminal on your host, and run the command to download and configure the OpenTelemetry collector. +7. For **Kubernetes**, download the manifest. +8. Copy the command under Step 2: +9. For **MacOS and Linux**, run the command in your terminal to start the EDOT Collector. +10. For **Kubernetes**, run the command from the directory where you downloaded the manifest to install the EDOT Collector on every node of your cluster. + +Logs are collected from setup onward, so you won’t see logs that occurred before starting the EDOT Collector. The default log path is `/var/log/*`. To update the path, modify `otel.yml`. + +::: + +:::: + + +Under **Visualize your data**, you’ll see links to **Logs Explorer** to view your logs and **Hosts** to view your host metrics. + + +## Gain deeper insight into your host data [_get_value_out_of_your_data] + +After using the Hosts page and Discover to confirm you’ve ingested all the host logs and metrics you want to monitor, use Elastic {{observability}} to gain deeper insight into your host data with the following capabilities and features: + +* In the [Infrastructure UI](../../../solutions/observability/infra-and-hosts/analyze-infrastructure-host-metrics.md), analyze and compare data collected from your hosts. You can also: + + * [Detect anomalies](../../../solutions/observability/infra-and-hosts/detect-metric-anomalies.md) for memory usage and network traffic on hosts. + * [Create alerts](../../../solutions/observability/incident-management/create-manage-rules.md) that notify you when an anomaly is detected or a metric exceeds a given value. + +* In the [Logs Explorer](../../../solutions/observability/logs/logs-explorer.md), search and filter your log data, get information about the structure of log fields, and display your findings in a visualization. You can also: + + * [Monitor log data set quality](../../../solutions/observability/data-set-quality-monitoring.md) to find degraded documents. + * [Run a pattern analysis](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#log-pattern-analysis) to find patterns in unstructured log messages. + * [Create alerts](../../../solutions/observability/incident-management/create-manage-rules.md) that notify you when an Observability data type reaches or exceeds a given value. + +* Use [machine learning](../../../explore-analyze/machine-learning/machine-learning-in-kibana.md) to apply predictive analytics to your data: + + * [Detect anomalies](../../../explore-analyze/machine-learning/anomaly-detection.md) by comparing real-time and historical data from different sources to look for unusual, problematic patterns. + * [Analyze log spikes and drops](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#log-rate-analysis). + * [Detect change points](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md#change-point-detection) in your time series data. + + +Refer to the [Elastic Observability](../../../solutions/observability.md) for a description of other useful features. \ No newline at end of file diff --git a/solutions/observability/get-started/quickstart-monitor-kubernetes-cluster-with-elastic-agent.md b/solutions/observability/get-started/quickstart-monitor-kubernetes-cluster-with-elastic-agent.md index 2eefce0d77..e118dfb8d1 100644 --- a/solutions/observability/get-started/quickstart-monitor-kubernetes-cluster-with-elastic-agent.md +++ b/solutions/observability/get-started/quickstart-monitor-kubernetes-cluster-with-elastic-agent.md @@ -4,11 +4,112 @@ mapped_urls: - https://www.elastic.co/guide/en/observability/current/monitor-k8s-logs-metrics-with-elastic-agent.html --- -# Quickstart: Monitor your Kubernetes cluster with Elastic Agent +# Quickstart: Monitor your Kubernetes cluster with {{agent}} [monitor-k8s-logs-metrics-with-elastic-agent] -% What needs to be done: Align serverless/stateful +In this quickstart guide, you’ll learn how to create the Kubernetes resources that are required to monitor your cluster infrastructure. -% Use migrated content from existing pages that map to this page: +This new approach requires minimal configuration and provides you with an easy setup to monitor your infrastructure. You no longer need to download, install, or configure the Elastic Agent, everything happens automatically when you run the kubectl command. -% - [ ] ./raw-migrated-files/docs-content/serverless/observability-quickstarts-k8s-logs-metrics.md -% - [ ] ./raw-migrated-files/observability-docs/observability/monitor-k8s-logs-metrics-with-elastic-agent.md \ No newline at end of file +The kubectl command installs the standalone Elastic Agent in your Kubernetes cluster, downloads all the Kubernetes resources needed to collect metrics from the cluster, and sends it to Elastic. + + +## Prerequisites [_prerequisites_2] + +::::{tab-set} +:group: stack-serverless + +:::{tab-item} Elastic Stack v9 +:sync: stack + +* An {{es}} cluster for storing and searching your data, and {{kib}} for visualizing and managing your data. This quickstart is available for all Elastic deployment models. To get started quickly, try out our hosted {{ess}} on [{{ecloud}}](https://cloud.elastic.co/registration?page=docs&placement=docs-body). +* A user with the `superuser` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) or the privileges required to onboard data. + + ::::{dropdown} Expand to view required privileges + * [**Cluster**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster): `['monitor', 'manage_own_api_key']` + * [**Index**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices): `{ names: ['logs-*-*', 'metrics-*-*'], privileges: ['auto_configure', 'create_doc'] }` + * [**Kibana**](../../../deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges.md): `{ spaces: ['*'], feature: { fleet: ['all'], fleetv2: ['all'] } }` + + :::: + +* A running Kubernetes cluster. +* [Kubectl](https://kubernetes.io/docs/reference/kubectl/). + +::: + +:::{tab-item} Serverless +:sync: serverless + +* An {{obs-serverless}} project. To learn more, refer to [Create an Observability project](../../../solutions/observability/get-started/create-an-observability-project.md). +* A user with the **Admin** role or higher—required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). +* A running Kubernetes cluster. +* [Kubectl](https://kubernetes.io/docs/reference/kubectl/). + +::: + +:::: + + +## Collect your data [_collect_your_data_2] + +::::{tab-set} +:group: stack-serverless + +:::{tab-item} Elastic Stack v9 +:sync: stack + +1. In {{kib}}, go to the **Observability** UI and click **Add Data**. +2. Under **What do you want to monitor?** select **Kubernetes**, and then select **Elastic Agent: Logs & Metrics**. + + :::{image} ../../../images/observability-quickstart-k8s-entry-point.png + :alt: Kubernetes entry point + :class: screenshot + ::: + +3. To install the Elastic Agent on your host, copy and run the install command. + + You will use the kubectl command to download a manifest file, inject user’s API key generated by {{kib}}, and create the Kubernetes resources. + +4. Go back to the **Add Observability Data** page. + + There might be a slight delay before data is ingested. When ready, you will see the message **We are monitoring your cluster**. + +5. Click **Explore Kubernetes cluster** to navigate to dashboards and explore your data. + +::: + +:::{tab-item} Serverless +:sync: serverless + +1. [Create a new {{obs-serverless}} project](../../../solutions/observability/get-started/create-an-observability-project.md), or open an existing one. +2. In your {{obs-serverless}} project, go to **Add Data**. +3. Under **What do you want to monitor?** select **Kubernetes**, and then select **Elastic Agent: Logs & Metrics**. + + :::{image} ../../../images/serverless-quickstart-k8s-entry-point.png + :alt: Kubernetes entry point + :class: screenshot + ::: + +4. To install the Elastic Agent on your host, copy and run the install command. + + You will use the kubectl command to download a manifest file, inject user’s API key generated by Kibana, and create the Kubernetes resources. + +5. Go back to the **Add Observability Data** page. There might be a slight delay before data is ingested. When ready, you will see the message **We are monitoring your cluster**. +6. Click **Explore Kubernetes cluster** to navigate to dashboards and explore your data. + +::: + +:::: + + +## Visualize your data [_visualize_your_data_2] + +After installation is complete and all relevant data is flowing into Elastic, the **Visualize your data** section allows you to access the Kubernetes Cluster Overview dashboard that can be used to monitor the health of the cluster. + +:::{image} ../../../images/observability-quickstart-k8s-overview.png +:alt: Kubernetes overview dashboard +:class: screenshot +::: + +Furthermore, you can access other useful prebuilt dashboards for monitoring Kubernetes resources, for example running pods per namespace, as well as the resources they consume, like CPU and memory. + +Refer to [Observability overview](../../../solutions/observability/get-started/what-is-elastic-observability.md) for a description of other useful features. \ No newline at end of file diff --git a/solutions/observability/get-started/quickstart-unified-kubernetes-observability-with-elastic-distributions-of-opentelemetry-edot.md b/solutions/observability/get-started/quickstart-unified-kubernetes-observability-with-elastic-distributions-of-opentelemetry-edot.md index de2bcda918..0fb7abb6b7 100644 --- a/solutions/observability/get-started/quickstart-unified-kubernetes-observability-with-elastic-distributions-of-opentelemetry-edot.md +++ b/solutions/observability/get-started/quickstart-unified-kubernetes-observability-with-elastic-distributions-of-opentelemetry-edot.md @@ -4,11 +4,136 @@ mapped_urls: - https://www.elastic.co/guide/en/serverless/current/monitor-k8s-otel-edot.html --- -# Quickstart: Unified Kubernetes Observability with Elastic Distributions of OpenTelemetry (EDOT) +# Quickstart: Unified Kubernetes Observability with Elastic Distributions of OpenTelemetry (EDOT) [monitor-k8s-otel-edot] -% What needs to be done: Align serverless/stateful +::::{warning} +This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. +:::: -% Use migrated content from existing pages that map to this page: -% - [ ] ./raw-migrated-files/observability-docs/observability/monitor-k8s-otel-edot.md -% - [ ] ./raw-migrated-files/docs-content/serverless/monitor-k8s-otel-edot.md \ No newline at end of file +In this quickstart guide, you’ll learn how to send Kubernetes logs, metrics, and application traces to Elasticsearch, using the [OpenTelemetry Operator](https://github.com/open-telemetry/opentelemetry-operator/) to orchestrate [Elastic Distributions of OpenTelemetry](https://github.com/elastic/opentelemetry/tree/main) (EDOT) Collectors and SDK instances. + +All the components will be deployed through the [opentelemetry-kube-stack](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-kube-stack) helm chart. They include: + +* [OpenTelemetry Operator](https://github.com/open-telemetry/opentelemetry-operator/). +* `DaemonSet` EDOT Collector configured for node level metrics. +* `Deployment` EDOT Collector configured for cluster level metrics. +* `Instrumentation` object for applications [auto-instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/). + +For a more detailed description of the components and advanced configuration, refer to the [elastic/opentelemetry](https://github.com/elastic/opentelemetry/blob/main/docs/kubernetes/operator/README.md) GitHub repository. + + +## Prerequisites [_prerequisites_2] + +::::{tab-set} +:group: stack-serverless + +:::{tab-item} Elastic Stack v9 +:sync: stack + +* An {{es}} cluster for storing and searching your data, and {{kib}} for visualizing and managing your data. This quickstart is available for all Elastic deployment models. To get started quickly, try out our hosted {{ess}} on [{{ecloud}}](https://cloud.elastic.co/registration?page=docs&placement=docs-body). +* A running Kubernetes cluster (v1.23 or newer). +* [Kubectl](https://kubernetes.io/docs/reference/kubectl/). +* [Helm](https://helm.sh/docs/intro/install/). +* (optional) [Cert-manager](https://cert-manager.io/docs/installation/), if you opt for automatic generation and renewal of TLS certificates. + +::: + +:::{tab-item} Serverless +:sync: serverless + +* An {{obs-serverless}} project. To learn more, refer to [Create an Observability project](../../../solutions/observability/get-started/create-an-observability-project.md). +* A running Kubernetes cluster (v1.23 or newer). +* [Kubectl](https://kubernetes.io/docs/reference/kubectl/). +* [Helm](https://helm.sh/docs/intro/install/). +* (optional) [Cert-manager](https://cert-manager.io/docs/installation/), if you opt for automatic generation and renewal of TLS certificates. + +::: + +:::: + +## Collect your data [_collect_your_data_2] + +::::{tab-set} +:group: stack-serverless + +:::{tab-item} Elastic Stack v9 +:sync: stack + +1. In {{kib}}, go to the **Observability** UI and click **Add Data**. +2. Under **What do you want to monitor?** select **Kubernetes**, and then select **OpenTelemetry: Full Observability**. + + :::{image} ../../../images/observability-quickstart-k8s-otel-entry-point.png + :alt: Kubernetes-OTel entry point + :class: screenshot + ::: + +3. Follow the on-screen instructions to install all needed components. + + ::::{note} + The default installation deploys the OpenTelemetry Operator with a self-signed TLS certificate valid for 365 days. This certificate **won’t be renewed** unless the Helm Chart release is manually updated. Refer to the [cert-manager integrated installation](https://github.com/elastic/opentelemetry/blob/main/docs/kubernetes/operator/README.md#cert-manager) guide to enable automatic certificate generation and renewal using [cert-manager](https://cert-manager.io/docs/installation/). + + :::: + + + Deploy the OpenTelemetry Operator and EDOT Collectors using the kube-stack Helm chart with the provided `values.yaml` file. You will run a few commands to: + + * Add the helm chart repository needed for the installation. + * Create a namespace. + * Create a secret with an API Key and the {{es}} endpoint to be used by the collectors. + * Install the `opentelemetry-kube-stack` helm chart with the provided `values.yaml`. + * Optionally, for instrumenting applications, apply the corresponding `annotations` as shown in {{kib}}. + +::: + +:::{tab-item} Serverless +:sync: serverless + +1. [Create a new {{obs-serverless}} project](../../../solutions/observability/get-started/create-an-observability-project.md), or open an existing one. +2. In your {{obs-serverless}} project, go to **Add Data**. +3. Under **What do you want to monitor?** select **Kubernetes**, and then select **OpenTelemetry: Full Observability**. + + :::{image} ../../../images/serverless-quickstart-k8s-otel-entry-point.png + :alt: Kubernetes-OTel entry point + :class: screenshot + ::: + +4. Follow the on-screen instructions to install all needed components. + + ::::{note} + The default installation deploys the OpenTelemetry Operator with a self-signed TLS certificate valid for 365 days. This certificate **won’t be renewed** unless the Helm Chart release is manually updated. Refer to the [cert-manager integrated installation](https://github.com/elastic/opentelemetry/blob/main/docs/kubernetes/operator/README.md#cert-manager) guide to enable automatic certificate generation and renewal using [cert-manager](https://cert-manager.io/docs/installation/). + + :::: + + + Deploy the OpenTelemetry Operator and EDOT Collectors using the kube-stack Helm chart with the provided `values.yaml` file. You will run a few commands to: + + * Add the helm chart repository needed for the installation. + * Create a namespace. + * Create a secret with an API Key and the {{es}} endpoint to be used by the collectors. + * Install the `opentelemetry-kube-stack` helm chart with the provided `values.yaml`. + * Optionally, for instrumenting applications, apply the corresponding `annotations` as shown in {{kib}}. + + +::: + +:::: + + + +## Visualize your data [_visualize_your_data] + +After installation is complete and all relevant data is flowing into Elastic, the **Visualize your data** section provides a link to the **[OTEL][Metrics Kubernetes]Cluster Overview** dashboard used to monitor the health of the cluster. + +:::{image} ../../../images/observability-quickstart-k8s-otel-dashboard.png +:alt: Kubernetes overview dashboard +:class: screenshot +::: + + +## Troubleshooting and more [_troubleshooting_and_more] + +* To troubleshoot deployment and installation, refer to [installation verification](https://github.com/elastic/opentelemetry/tree/main/docs/kubernetes/operator#installation-verification). +* For application instrumentation details, refer to [Instrumenting applications with EDOT SDKs on Kubernetes](https://github.com/elastic/opentelemetry/blob/main/docs/kubernetes/operator/instrumenting-applications.md). +* To customize the configuration, refer to [custom configuration](https://github.com/elastic/opentelemetry/tree/main/docs/kubernetes/operator#custom-configuration). +* Refer to [Observability overview](../../../solutions/observability/get-started/what-is-elastic-observability.md) for a description of other useful features. \ No newline at end of file diff --git a/solutions/observability/get-started/what-is-elastic-observability.md b/solutions/observability/get-started/what-is-elastic-observability.md index 2f75b52fab..8a4a301ab8 100644 --- a/solutions/observability/get-started/what-is-elastic-observability.md +++ b/solutions/observability/get-started/what-is-elastic-observability.md @@ -4,11 +4,145 @@ mapped_urls: - https://www.elastic.co/guide/en/serverless/current/observability-serverless-observability-overview.html --- -# What is Elastic Observability +# What is Elastic {{observability}}? [observability-introduction] -% What needs to be done: Align serverless/stateful +{{observability}} provides granular insights and context into the behavior of applications running in your environments. It’s an important part of any system that you build and want to monitor. Being able to detect and fix root cause events quickly within an observable system is a minimum requirement for any analyst. -% Use migrated content from existing pages that map to this page: +[Elastic {{observability}}](https://www.elastic.co/observability) provides a single stack to unify your logs, infrastructure metrics, application traces, user experience data, synthetics, and universal profiling. Ingest your data directly to {{es}}, where you can further process and enhance the data, before visualizing it and adding alerts in {{kib}}. -% - [ ] ./raw-migrated-files/observability-docs/observability/observability-introduction.md -% - [ ] ./raw-migrated-files/docs-content/serverless/observability-serverless-observability-overview.md \ No newline at end of file +:::{image} ../../../images/observability-what-is-observability.svg +:alt: Elastic {{observability}} overview diagram +::: + +## Log monitoring [apm-overview] + +Analyze log data from your hosts, services, Kubernetes, Apache, and many more. + +In **Logs Explorer** (powered by Discover), you can quickly search and filter your log data, get information about the structure of the fields, and display your findings in a visualization. + +:::{image} ../../../images/serverless-log-explorer-overview.png +:alt: Logs Explorer showing log events +:class: screenshot +::: + +[Learn more about log monitoring →](../../../solutions/observability/logs.md) + + +## Application performance monitoring (APM) [observability-serverless-observability-overview-application-performance-monitoring-apm] + +Instrument your code and collect performance data and errors at runtime by installing APM agents like Java, Go, .NET, and many more. Then use Observability to monitor your software services and applications in real time: + +* Visualize detailed performance information on your services. +* Identify and analyze errors. +* Monitor host-level and APM agent-specific metrics like JVM and Go runtime metrics. + +The **Service** inventory provides a quick, high-level overview of the health and general performance of all instrumented services. + +:::{image} ../../../images/serverless-services-inventory.png +:alt: Service inventory showing health and performance of instrumented services +:class: screenshot +::: + +[Learn more about Application performance monitoring (APM) →](../../../solutions/observability/apps/application-performance-monitoring-apm.md) + + +## Infrastructure monitoring [metrics-overview] + +Monitor system and service metrics from your servers, Docker, Kubernetes, Prometheus, and other services and applications. + +On the {{observability}} **Overview** page, the **Hosts** table shows your top hosts with the most significant resource footprints. These metrics help you evaluate host efficiency and determine if resource consumption is impacting end users. + +:::{image} ../../../images/observability-metrics-summary.png +:alt: Summary of Hosts on the {{observability}} overview page +:class: screenshot +::: + +You can then drill down into the {{infrastructure-app}} by clicking **Show inventory**. Here you can monitor and filter your data by hosts, pods, containers,or EC2 instances and create custom groupings such as availability zones or namespaces. + +[Learn more about infrastructure monitoring → ](../../../solutions/observability/infra-and-hosts/analyze-infrastructure-host-metrics.md) + + +% Stateful only for RUM. + +## Real user monitoring (RUM) [user-experience-overview] + +Quantify and analyze the perceived performance of your web application with {{user-experience}} data, powered by the APM RUM agent. Unlike testing environments, {{user-experience}} data reflects real-world user experiences. + +On the {{observability}} **Overview** page, the **{{user-experience}}** chart provides a snapshot of core web vitals for the service with the most traffic. + +:::{image} ../../../images/observability-obs-overview-ue.png +:alt: Summary of {{user-experience}} metrics on the {{observability}} overview page +:class: screenshot +::: + +You can then drill down into the {{user-experience}} dashboard by clicking **Show dashboard** too see data by URL, operating system, browser, and location. + + [Learn more about {{user-experience}} →](../../../solutions/observability/apps/real-user-monitoring-user-experience.md). + +## Synthetic monitoring [synthetic-monitoring-overview] + +Simulate actions and requests that an end user would perform on your site at predefined intervals and in a controlled environment. The end result is rich, consistent, and repeatable data that you can trend and alert on. + +[Learn more about Synthetic monitoring →](../../../solutions/observability/apps/synthetic-monitoring.md). + +% Stateful only for Universal Profiling. + +## Universal Profiling [universal-profiling-overview] + +Build stack traces to get visibility into your system without application source code changes or instrumentation. Use flamegraphs to explore system performance and identify the most expensive lines of code, increase CPU resource efficiency, debug performance regressions, and reduce cloud spend. + +[Learn more about Universal Profiling →](../../../solutions/observability/infra-and-hosts/universal-profiling.md). + + +## Alerting [observability-serverless-observability-overview-alerting] + +Stay aware of potential issues in your environments with Observability’s alerting and actions feature that integrates with log monitoring and APM. It provides a set of built-in actions and specific threshold rules and enables central management of all rules. + +On the **Alerts** page, the **Alerts** table provides a snapshot of alerts occurring within the specified time frame. The table includes the alert status, when it was last updated, the reason for the alert, and more. + +:::{image} ../../../images/serverless-observability-alerts-overview.png +:alt: Summary of Alerts on the Observability overview page +:class: screenshot +::: + +[Learn more about alerting → ](../../../solutions/observability/incident-management/alerting.md) + + +## Service-level objectives (SLOs) [observability-serverless-observability-overview-service-level-objectives-slos] + +Set clear, measurable targets for your service performance, based on factors like availability, response times, error rates, and other key metrics. Then monitor and track your SLOs in real time, using detailed dashboards and alerts that help you quickly identify and troubleshoot issues. + +From the SLO overview list, you can see all of your SLOs and a quick summary of what’s happening in each one: + +:::{image} ../../../images/serverless-slo-dashboard.png +:alt: Dashboard showing list of SLOs +:class: screenshot +::: + +[Learn more about SLOs → ](../../../solutions/observability/incident-management/service-level-objectives-slos.md) + +## Cases [observability-serverless-observability-overview-cases] + +Collect and share information about observability issues by creating cases. Cases allow you to track key investigation details, add assignees and tags to your cases, set their severity and status, and add alerts, comments, and visualizations. You can also send cases to third-party systems, such as ServiceNow and Jira. + +:::{image} ../../../images/serverless-cases.png +:alt: Screenshot showing list of cases +:class: screenshot +::: + +[Learn more about cases → ](../../../solutions/observability/incident-management/cases.md) + +## Machine learning and AIOps [observability-serverless-observability-overview-aiops] + +Reduce the time and effort required to detect, understand, investigate, and resolve incidents at scale by leveraging predictive analytics and machine learning: + +* Detect anomalies by comparing real-time and historical data from different sources to look for unusual, problematic patterns. +* Find and investigate the causes of unusual spikes or drops in log rates. +* Detect distribution changes, trend changes, and other statistically significant change points in a metric of your time series data. + +:::{image} ../../../images/serverless-log-rate-analysis.png +:alt: Log rate analysis page showing log rate spike +:class: screenshot +::: + +[Learn more about machine learning and AIOps →](../../../explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md) \ No newline at end of file