diff --git a/docset.yml b/docset.yml index 188f6d7612..15aa524251 100644 --- a/docset.yml +++ b/docset.yml @@ -277,3 +277,4 @@ subs: release-date: "2-April-2025" heroku: "Elasticsearch Add-on for Heroku" obs-ai-assistant: "Elastic AI Assistant for Observability and Search" + apm-server-or-mis: "APM Server or the managed intake service" diff --git a/solutions/observability/apps.md b/solutions/observability/apps.md index e1247f4aae..bc04e85e10 100644 --- a/solutions/observability/apps.md +++ b/solutions/observability/apps.md @@ -1,9 +1,11 @@ --- +navigation_title: "Applications and services" mapped_pages: - https://www.elastic.co/guide/en/serverless/current/application-and-service-monitoring.html - https://www.elastic.co/guide/en/observability/current/application-and-service-monitoring.html - -navigation_title: "Applications and services" +applies_to: + stack: + serverless: --- # Application and service monitoring [application-and-service-monitoring] @@ -15,9 +17,9 @@ Explore the topics in this section to learn how to observe and monitor software | | | | --- | --- | -| [LLM Observability](../../solutions/observability/apps/llm-observability.md) | Monitor LLM-powered applications to keep them reliable, efficient, cost-effective, and easy to troubleshoot.| -| [Application performance monitoring (APM)](../../solutions/observability/apps/application-performance-monitoring-apm.md) | Monitor software services and applications in real time, by collecting detailed performance information on response time for incoming requests, database queries, calls to caches, external HTTP requests, and more. | -| [Synthetic monitoring](../../solutions/observability/apps/synthetic-monitoring.md) | Monitor the availability of network endpoints and services. | -| [Real user monitoring](../../solutions/observability/apps/real-user-monitoring-user-experience.md) | Quantify and analyze the perceived performance of your web application using real-world user experiences. | -| [Uptime monitoring (deprecated)](../../solutions/observability/apps/uptime-monitoring-deprecated.md) | Periodically check the status of your services and applications. | -| [Tutorial: Monitor a Java application](../../solutions/observability/apps/tutorial-monitor-java-application.md) | Monitor a Java application using Elastic Observability: Logs, Infrastructure metrics, APM, and Uptime. | +| [LLM Observability](/solutions/observability/apps/llm-observability.md) | Monitor LLM-powered applications to keep them reliable, efficient, cost-effective, and easy to troubleshoot.| +| [Application performance monitoring (APM)](/solutions/observability/apps/application-performance-monitoring-apm.md) | Monitor software services and applications in real time, by collecting detailed performance information on response time for incoming requests, database queries, calls to caches, external HTTP requests, and more. | +| [Synthetic monitoring](/solutions/observability/apps/synthetic-monitoring.md) | Monitor the availability of network endpoints and services. | +| [Real user monitoring](/solutions/observability/apps/real-user-monitoring-user-experience.md) | Quantify and analyze the perceived performance of your web application using real-world user experiences. | +| [Uptime monitoring (deprecated)](/solutions/observability/apps/uptime-monitoring-deprecated.md) | Periodically check the status of your services and applications. | +| [Tutorial: Monitor a Java application](/solutions/observability/apps/tutorial-monitor-java-application.md) | Monitor a Java application using Elastic Observability: Logs, Infrastructure metrics, APM, and Uptime. | diff --git a/solutions/observability/apps/_snippets/apm-server-vs-mis.md b/solutions/observability/apps/_snippets/apm-server-vs-mis.md new file mode 100644 index 0000000000..5225feee4e --- /dev/null +++ b/solutions/observability/apps/_snippets/apm-server-vs-mis.md @@ -0,0 +1,3 @@ +:::{admonition} APM Server vs managed intake service +In {{ech}}, the _APM Server_ receives data from Elastic APM agents and transforms it into Elasticsearch documents. In {{serverless-full}} there is in fact no APM Server running, instead the _managed intake service_ receives and transforms data. +::: \ No newline at end of file diff --git a/solutions/observability/apps/act-on-data.md b/solutions/observability/apps/act-on-data.md index 850ac51ff3..9a47b487a0 100644 --- a/solutions/observability/apps/act-on-data.md +++ b/solutions/observability/apps/act-on-data.md @@ -10,7 +10,6 @@ applies_to: # Act on application data - In addition to exploring visualizations in the Applications UI in {{kib}}, you can make your application data more actionable with: | | | diff --git a/solutions/observability/apps/analyze-data-from-synthetic-monitors.md b/solutions/observability/apps/analyze-data-from-synthetic-monitors.md index 441f4eb684..a7604d5c64 100644 --- a/solutions/observability/apps/analyze-data-from-synthetic-monitors.md +++ b/solutions/observability/apps/analyze-data-from-synthetic-monitors.md @@ -16,7 +16,7 @@ The Synthetics UI gives you a high-level overview of your service’s availabili The Synthetics **Overview** tab provides you with a high-level view of all the services you are monitoring to help you quickly diagnose outages and other connectivity issues within your network. -To access this page, find `Synthetics` in the [global search field](../../../get-started/the-stack.md#kibana-navigation-search) and make sure you’re on the **Overview** tab. +To access this page, find `Synthetics` in the [global search field](/get-started/the-stack.md#kibana-navigation-search) and make sure you’re on the **Overview** tab. This overview includes a snapshot of the current status of all monitors, the number of errors that occurred over the last 6 hours, and the number of alerts over the last 12 hours. All monitors created using projects or using the UI will be listed below with information about the location, current status, and duration average. @@ -25,7 +25,6 @@ When you use a single monitor configuration to create monitors in multiple locat :::: - :::{image} /solutions/images/observability-synthetics-monitor-page.png :alt: Synthetics UI :screenshot: @@ -35,7 +34,6 @@ To get started with your analysis in the Overview tab, you can search for monito Then click an individual monitor to see some details in a flyout. From there, you can click **Go to monitor** to go to an individual monitor’s page to see more details (as described below). - ## All monitor types [synthetics-analyze-individual-monitors] When you go to an individual monitor’s page, you’ll see much more detail about the monitor’s performance over time. The details vary by monitor type, but for every monitor at the top of the page you’ll see: @@ -59,7 +57,6 @@ When you go to an individual monitor’s page, you’ll see much more detail abo Each individual monitor’s page has three tabs: Overview, History, and Errors. - ### Overview [synthetics-analyze-individual-monitors-overview] The **Overview** tab has information about the monitor availability, duration, and any errors that have occurred since the monitor was created. The *Duration trends* chart displays the timing for each check that was performed in the last 30 days. This visualization helps you to gain insights into how quickly requests resolve by the targeted endpoint and gives you a sense of how frequently a host or endpoint was down. @@ -69,40 +66,37 @@ The **Overview** tab has information about the monitor availability, duration, a :screenshot: ::: - ### History [synthetics-analyze-individual-monitors-history] The **History** tab has information on every time the monitor has run. It includes some high-level stats and a complete list of all test runs. Use the calendar icon (![Calendar icon](/solutions/images/observability-calendar.svg "")) and search bar to filter for runs that occurred in a specific time period. -For browser monitors, you can click on any run in the **Test runs** list to see the details for that run. Read more about what information is included the in [Details for one run](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-run) section below. +For browser monitors, you can click on any run in the **Test runs** list to see the details for that run. Read more about what information is included the in [Details for one run](/solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-run) section below. :::{image} /solutions/images/observability-synthetics-analyze-individual-monitor-history.png :alt: The History tab on the individual monitor page for all monitor types in the {synthetics-app} :screenshot: ::: -If the monitor is configured to [retest on failure](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor), you’ll see retests listed in the **Test runs** table. Runs that are retests include a rerun icon (![Refresh icon](/solutions/images/observability-refresh.svg "")) next to the result badge. +If the monitor is configured to [retest on failure](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor), you’ll see retests listed in the **Test runs** table. Runs that are retests include a rerun icon (![Refresh icon](/solutions/images/observability-refresh.svg "")) next to the result badge. :::{image} /solutions/images/observability-synthetics-retest.png :alt: A failed run and a retest in the table of test runs in the {synthetics-app} :screenshot: ::: - ### Errors [synthetics-analyze-individual-monitors-errors] -The **Errors** tab has information on failed runs. If the monitor is configured to [retest on failure](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor), failed runs will only result in an error if both the initial run and the rerun fail. This can reduce noise related to transient problems. +The **Errors** tab has information on failed runs. If the monitor is configured to [retest on failure](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor), failed runs will only result in an error if both the initial run and the rerun fail. This can reduce noise related to transient problems. The Errors tab includes a high-level overview of all alerts and a complete list of all failures. Use the calendar icon (![Calendar icon](/solutions/images/observability-calendar.svg "")) and search bar to filter for runs that occurred in a specific time period. -For browser monitors, you can click on any run in the **Error** list to open an **Error details** page that includes most of the same information that is included the in [Details for one run](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-run) section below. +For browser monitors, you can click on any run in the **Error** list to open an **Error details** page that includes most of the same information that is included the in [Details for one run](/solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-run) section below. :::{image} /solutions/images/observability-synthetics-analyze-individual-monitor-errors.png :alt: The Errors tab on the individual monitor page for all monitor types in the {synthetics-app} :screenshot: ::: - ## Browser monitors [synthetics-analyze-journeys] For browser monitors, you can look at results at various levels of granularity: @@ -111,7 +105,6 @@ For browser monitors, you can look at results at various levels of granularity: * Drill down into the details of a single run. * Drill down further into the details of a single *step* within a journey. - ### Journey runs over time [journey_runs_over_time] The journey page on the Overview tab includes: @@ -119,7 +112,7 @@ The journey page on the Overview tab includes: * An overview of the **last test run** including high-level information for each step. * **Alerts** to date including both active and recovered alerts. * **Duration by step** over the last 24 hours. -* A list of the **last 10 test runs** that link to the [details for each run](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-run). +* A list of the **last 10 test runs** that link to the [details for each run](/solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-run). :::{image} /solutions/images/observability-synthetics-analyze-journeys-over-time.png :alt: Individual journey page for browser monitors in the {synthetics-app} @@ -128,9 +121,8 @@ The journey page on the Overview tab includes: From here, you can either drill down into: -* The latest run of the full journey by clicking **![Inspect icon](/solutions/images/observability-inspect.svg "") View test run** or a past run in the list of **Last 10 test runs**. This will take you to the view described below in [Details for one run](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-run). -* An individual step in this run by clicking the performance breakdown icon (![Performance breakdown icon](/solutions/images/observability-apmTrace.svg "")) next to one of the steps. This will take you to the view described below in [Details for one step](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-step). - +* The latest run of the full journey by clicking **![Inspect icon](/solutions/images/observability-inspect.svg "") View test run** or a past run in the list of **Last 10 test runs**. This will take you to the view described below in [Details for one run](/solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-run). +* An individual step in this run by clicking the performance breakdown icon (![Performance breakdown icon](/solutions/images/observability-apmTrace.svg "")) next to one of the steps. This will take you to the view described below in [Details for one step](/solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-step). ### Details for one run [synthetics-analyze-one-run] @@ -145,30 +137,27 @@ Navigate through each step using **![Previous icon](/solutions/images/observabil :screenshot: ::: -Scroll down to dig into the steps in this journey run. Click the ![Arrow right icon](/solutions/images/observability-arrowRight.svg "") icon next to the step number to show details. The details include metrics for the step in the current run and the step in the last successful run. Read more about step-level metrics below in [Timing](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-step-timing) and [Metrics](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-step-metrics). +Scroll down to dig into the steps in this journey run. Click the ![Arrow right icon](/solutions/images/observability-arrowRight.svg "") icon next to the step number to show details. The details include metrics for the step in the current run and the step in the last successful run. Read more about step-level metrics below in [Timing](/solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-step-timing) and [Metrics](/solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-step-metrics). This is particularly useful to compare the metrics for a failed step to the last time it completed successfully when trying to diagnose the reason it failed. ![Step list on a page detailing one run of a browser monitor in the Synthetics UI](/solutions/images/observability-synthetics-analyze-one-run-compare-steps.png "") -Drill down to see even more details for an individual step by clicking the performance breakdown icon (![Performance breakdown icon](/solutions/images/observability-apmTrace.svg "")) next to one of the steps. This will take you to the view described below in [Details for one step](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-step). - +Drill down to see even more details for an individual step by clicking the performance breakdown icon (![Performance breakdown icon](/solutions/images/observability-apmTrace.svg "")) next to one of the steps. This will take you to the view described below in [Details for one step](/solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-step). ### Details for one step [synthetics-analyze-one-step] After clicking the performance breakdown icon (![Performance breakdown icon](/solutions/images/observability-apmTrace.svg "")) you’ll see more detail for an individual step. - #### Screenshot [synthetics-analyze-one-step-screenshot] By default the synthetics library will capture a screenshot for each step regardless of whether the step completed or failed. ::::{note} -Customize screenshot behavior for all monitors in the [configuration file](../../../solutions/observability/apps/configure-synthetics-projects.md), for one monitor using [`monitor.use`](../../../solutions/observability/apps/configure-individual-browser-monitors.md), or for a run using the [CLI](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-command). +Customize screenshot behavior for all monitors in the [configuration file](/solutions/observability/apps/configure-synthetics-projects.md), for one monitor using [`monitor.use`](/solutions/observability/apps/configure-individual-browser-monitors.md), or for a run using the [CLI](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-command). :::: - Screenshots can be particularly helpful to identify what went wrong when a step fails because of a change to the UI. You can compare the failed step to the last time the step successfully completed. :::{image} /solutions/images/observability-synthetics-analyze-one-step-screenshot.png @@ -176,7 +165,6 @@ Screenshots can be particularly helpful to identify what went wrong when a step :screenshot: ::: - #### Timing [synthetics-analyze-one-step-timing] The **Timing** visualization shows a breakdown of the time spent in each part of the resource loading process for the step including: @@ -191,14 +179,13 @@ The **Timing** visualization shows a breakdown of the time spent in each part of Next to each network timing metric, there’s an icon that indicates whether the value is higher (![Value is higher icon](/solutions/images/observability-sortUp.svg "")), lower (![Value is lower icon](/solutions/images/observability-sortDown.svg "")), or the same (![Value is the same](/solutions/images/observability-minus.svg "")) compared to the median of all runs in the last 24 hours. Hover over the icon to see more details in a tooltip. -This gives you an overview of how much time is spent (and how that time is spent) loading resources. This high-level information may not help you diagnose a problem on its own, but it could act as a signal to look at more granular information in the [Network requests](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-step-network) section. +This gives you an overview of how much time is spent (and how that time is spent) loading resources. This high-level information may not help you diagnose a problem on its own, but it could act as a signal to look at more granular information in the [Network requests](/solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-step-network) section. :::{image} /solutions/images/observability-synthetics-analyze-one-step-timing.png :alt: Network timing visualization for one step in a browser monitor in the {synthetics-app} :screenshot: ::: - #### Metrics [synthetics-analyze-one-step-metrics] The **Metrics** visualization gives you insight into the performance of the web page visited in the step and what a user would experience when going through the current step. Metrics include: @@ -214,7 +201,6 @@ Largest contentful paint and Cumulative layout shift are part of Google’s [Cor :::: - Next to each metric, there’s an icon that indicates whether the value is higher (![Value is higher icon](/solutions/images/observability-sortUp.svg "")), lower (![Value is lower icon](/solutions/images/observability-sortDown.svg "")), or the same (![Value is the same](/solutions/images/observability-minus.svg "")) compared to all runs over the last 24 hours. Hover over the icon to see more details in a tooltip. :::{image} /solutions/images/observability-synthetics-analyze-one-step-metrics.png @@ -222,7 +208,6 @@ Next to each metric, there’s an icon that indicates whether the value is highe :screenshot: ::: - #### Object weight and count [synthetics-analyze-one-step-object] The **Object weight** visualization shows the cumulative size of downloaded resources by type, and **Object count** shows the number of individual resources by type. @@ -234,12 +219,11 @@ This provides a different kind of analysis. For example, you might have a large :screenshot: ::: - #### Network requests [synthetics-analyze-one-step-network] The **Network requests** visualization is a waterfall chart that shows every request the page made when a user executed it. Each line in the chart represents an HTTP network request and helps you quickly identify what resources are taking the longest to load and in what order they are loading. -The colored bars within each line indicate the time spent per resource. Each color represents a different part of that resource’s loading process (as defined in the [Timing](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-step-timing) section above) and includes the time spent downloading content for specific Multipurpose Internet Mail Extensions (MIME) types: HTML, JS, CSS, Media, Font, XHR, and Other. +The colored bars within each line indicate the time spent per resource. Each color represents a different part of that resource’s loading process (as defined in the [Timing](/solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-one-step-timing) section above) and includes the time spent downloading content for specific Multipurpose Internet Mail Extensions (MIME) types: HTML, JS, CSS, Media, Font, XHR, and Other. Understanding each phase of a request can help you improve your site’s speed by reducing the time spent in each phase. diff --git a/solutions/observability/apps/analyze-monitors.md b/solutions/observability/apps/analyze-monitors.md index 386871fca8..ba5bea51c7 100644 --- a/solutions/observability/apps/analyze-monitors.md +++ b/solutions/observability/apps/analyze-monitors.md @@ -18,7 +18,6 @@ To access this page, go to **{{observability}} > Uptime > Monitors**. Click on a The monitor detail screen displays several panels of information. - ## Status panel [uptime-status-panel] The **Status** panel displays a summary of the latest information regarding your monitor. You can view its availability, monitor ID, type, and any assigned tags. You can click a link to visit the targeted URL, view when the TLS certificate expires, and determine the amount of time elapsed since the last check. @@ -32,7 +31,6 @@ The **Monitoring from** list displays service availability per monitoring locati To display a map with each location as a pinpoint, you can toggle the availability view from list view to map view. - ## Monitor duration [uptime-monitor-duration] The **Monitor duration** chart displays the timing for each check that was performed. The visualization helps you to gain insights into how quickly requests resolve by the targeted endpoint and give you a sense of how frequently a host or endpoint was down in your selected time span. @@ -44,7 +42,6 @@ Included on this chart is the {{anomaly-detect}} ({{ml}}) integration. For more :screenshot: ::: - ## Pings over time [uptime-pings-chart] The **Pings over time** chart is a graphical representation of the check statuses over time. Hover over the charts to display crosshairs with specific numeric data. @@ -54,7 +51,6 @@ The **Pings over time** chart is a graphical representation of the check statuse :screenshot: ::: - ## Check history [uptime-history-panel] The **History** table lists the total count of this monitor’s checks for the selected date range. To help find recent problems on a per-check basis, you can filter by `status` and `location`. diff --git a/solutions/observability/apps/analyze.md b/solutions/observability/apps/analyze.md index ce96d8f958..63c70c5962 100644 --- a/solutions/observability/apps/analyze.md +++ b/solutions/observability/apps/analyze.md @@ -23,6 +23,3 @@ Learn how to view and interpret data in the {{uptime-app}}: * [Analyze monitors](analyze-monitors.md) * [Inspect uptime duration anomalies](inspect-uptime-duration-anomalies.md) - - - diff --git a/solutions/observability/apps/anonymous-authentication.md b/solutions/observability/apps/anonymous-authentication.md index ac30a12e13..811c04be37 100644 --- a/solutions/observability/apps/anonymous-authentication.md +++ b/solutions/observability/apps/anonymous-authentication.md @@ -16,7 +16,6 @@ Elastic APM agents can send unauthenticated (anonymous) events to the APM Server In some cases, however, it makes sense to allow both authenticated and anonymous requests. For example, it isn’t possible to authenticate requests from front-end services as the secret token or API key can’t be protected. This is the case with the Real User Monitoring (RUM) agent running in a browser, or the Android or iOS/Swift agent running in a user application. However, you still likely want to authenticate requests from back-end services. To solve this problem, you can enable anonymous authentication in the APM Server to allow the ingestion of unauthenticated client-side APM data while still requiring authentication for server-side services. - ## Configuring anonymous auth for client-side services [apm-anonymous-auth-config] ::::{note} @@ -24,7 +23,6 @@ You can only enable and configure anonymous authentication if an [API key](api-k :::: - :::::::{tab-set} ::::::{tab-item} Fleet-managed @@ -57,7 +55,6 @@ The remote IP address of an incoming request might be different from the end-use If none of these headers are present, the remote address for the incoming request is used. - ### Using a reverse proxy or load balancer [apm-derive-client-ip-concerns] HTTP headers are easily modified; it’s possible for anyone to spoof the derived `client.ip` value by changing or setting, for example, the value of the `X-Forwarded-For` header. For this reason, if any of your clients are not trusted, we recommend setting up a reverse proxy or load balancer in front of the APM Server. diff --git a/solutions/observability/apps/api-keys.md b/solutions/observability/apps/api-keys.md index 4da797006a..38d744543b 100644 --- a/solutions/observability/apps/api-keys.md +++ b/solutions/observability/apps/api-keys.md @@ -7,26 +7,27 @@ applies_to: # API keys [apm-api-key] +:::{include} _snippets/apm-server-vs-mis.md +::: + ::::{important} API keys are sent as plain-text, so they only provide security when used in combination with [TLS](apm-agent-tls-communication.md). :::: - -When enabled, API keys are used to authorize requests to the APM Server. API keys are not applicable for APM agents running on clients, like the RUM agent, as there is no way to prevent them from being publicly exposed. +When enabled, API keys are used to authorize requests to {{apm-server-or-mis}}. API keys are not applicable for APM agents running on clients, like the RUM agent, as there is no way to prevent them from being publicly exposed. You can assign one or more unique privileges to each API key: * **Agent configuration** (`config_agent:read`): Required for agents to read [Agent configuration remotely](apm-agent-central-configuration.md). * **Ingest** (`event:write`): Required for ingesting agent events. -To secure the communication between APM Agents and the APM Server with API keys, make sure [TLS](apm-agent-tls-communication.md) is enabled, then complete these steps: +To secure the communication between APM Agents and either {{apm-server-or-mis}} with API keys, make sure [TLS](apm-agent-tls-communication.md) is enabled, then complete these steps: 1. [Enable API keys](#apm-enable-api-key) 2. [Create an API key user](#apm-create-api-key-user) 3. [Create an API key in {{kib}}](#apm-create-an-api-key) 4. [Set the API key in your APM agents](#apm-agent-api-key) - ## Enable API keys [apm-enable-api-key] :::::::{tab-set} @@ -50,11 +51,20 @@ apm-server.auth.api_key.limit: 50 <2> :::::: +::::::{tab-item} {{serverless-full}} +API keys are enabled by default. +:::::: + ::::::: ## Create an API key user in {{kib}} [apm-create-api-key-user] -API keys can only have the same or lower access rights than the user that creates them. Instead of using a superuser account to create API keys, you can create a role with the minimum required privileges. +API keys can only have the same or lower access rights than the user that creates them. + +:::::::{tab-set} + +::::::{tab-item} Fleet-managed or APM Server binary +Instead of using a superuser account to create API keys, you can create a role with the minimum required privileges. The user creating an {{apm-agent}} API key must have at least the `manage_own_api_key` cluster privilege and the APM application-level privileges that it wishes to grant. In addition, when creating an API key from the Applications UI, you’ll need the appropriate {{kib}} Space and Feature privileges. @@ -84,14 +94,23 @@ POST /_security/role/apm_agent_key_role 1. This example assigns privileges for the default space. - Assign the newly created `apm_agent_key_role` role to any user that wishes to create {{apm-agent}} API keys. +:::::: +::::::{tab-item} {{serverless-full}} +**For Observability Serverless projects**, the Editor role or higher is required to create and manage API keys. To learn more, refer to [Assign user roles and privileges](/deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). +:::::: + +::::::: ## Create an API key in the Applications UI [apm-create-an-api-key] The Applications UI has a built-in workflow that you can use to easily create and view {{apm-agent}} API keys. Only API keys created in the Applications UI will show up here. +:::::::{tab-set} + +::::::{tab-item} Fleet-managed or APM Server binary + Using a superuser account, or a user with the role created in the previous step, In {{kib}}, find **Applications** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). Go to **Settings** → **Agent keys**. Enter a name for your API key and select at least one privilege. For example, to create an API key that can be used to ingest APM events and read agent central configuration, select `config_agent:read` and `event:write`. @@ -103,6 +122,27 @@ Click **Create APM Agent key** and copy the Base64 encoded API key. You will nee :screenshot: ::: +:::::: + +::::::{tab-item} {{serverless-full}} +To create a new API key: + +1. In your Elastic Observability Serverless project, go to any Applications page. +1. Click **Settings**. +1. Select the **Agent keys** tab. +1. Click **Create APM agent key**. +1. Name the key and assign privileges to it. +1. Click **Create APM agent key**. +1. Copy the key now. You will not be able to see it again. API keys do not expire. + +To view all API keys for your project: + +1. Expand **Project settings**. +1. Select **Management**. +1. Select **API keys**. +:::::: + +::::::: ## Set the API key in your APM agents [apm-agent-api-key] @@ -118,15 +158,18 @@ You can now apply your newly created API keys in the configuration of each of yo * **Python agent**: [`api_key`](apm-agent-python://reference/configuration.md#config-api-key) * **Ruby agent**: [`api_key`](apm-agent-ruby://reference/configuration.md#config-api-key) - ## Alternate API key creation methods [apm-configure-api-key-alternative] +```{applies_to} +stack: +serverless: unavailable +``` + API keys can also be created and validated outside of {{kib}}: * [APM Server API key workflow](#apm-create-api-key-workflow-apm-server) * [{{es}} API key workflow](#apm-create-api-key-workflow-es) - ### APM Server API key workflow [apm-create-api-key-workflow-apm-server] This API creation method only works with the APM Server binary. @@ -137,10 +180,8 @@ This API creation method only works with the APM Server binary. Users should create API Keys through {{kib}} or the {{es}} REST API :::: - APM Server provides a command line interface for creating, retrieving, invalidating, and verifying API keys. Keys created using this method can only be used for communication with APM Server. - #### `apikey` subcommands [apm-create-api-key-subcommands] **`create`** @@ -163,10 +204,8 @@ APM Server provides a command line interface for creating, retrieving, invalidat * To **ingest agent data**, assign `event:write`. * To **upload source maps**, assign `sourcemap:write`. - :::: - **`info`** : Query API Key(s). `--id` or `--name` required. @@ -176,7 +215,6 @@ APM Server provides a command line interface for creating, retrieving, invalidat **`verify`** : Check if a credentials string has the given privilege(s). `--credentials` required. - #### Privileges [apm-create-api-key-privileges] If privileges are not specified at creation time, the created key will have all privileges. @@ -185,7 +223,6 @@ If privileges are not specified at creation time, the created key will have all * `--ingest` grants the `event:write` privilege * `--sourcemap` grants the `sourcemap:write` privilege - #### Create an API key [apm-create-api-key-workflow] Create an API key with the `create` subcommand. @@ -238,7 +275,6 @@ Error count ........ 0 A full list of `apikey` subcommands and flags is available in the [API key command reference](apm-server-command-reference.md#apm-apikey-command). - ### {{es}} API key workflow [apm-create-api-key-workflow-es] It is also possible to create API keys using the {{es}} [create API key API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key). @@ -268,7 +304,6 @@ POST /_security/api_key 2. The expiration time of the API key 3. Any assigned privileges - The response will look similar to this: ```console-result diff --git a/solutions/observability/apps/apm-agent-authorization.md b/solutions/observability/apps/apm-agent-authorization.md index e38fe12fa1..e0ef819c33 100644 --- a/solutions/observability/apps/apm-agent-authorization.md +++ b/solutions/observability/apps/apm-agent-authorization.md @@ -14,7 +14,6 @@ Most options in this section are supported by all APM Server deployment methods. :::: - Agent authorization APM Server configuration options. :::::::{tab-set} @@ -55,8 +54,6 @@ These settings apply to API key communication between the APM Server and APM Age These settings are different from the API key settings used for {{es}} output and monitoring. :::: - - ### API key for agent authentication [_api_key_for_agent_authentication] Enable API key authorization by setting `enabled` to `true`. By default, `enabled` is set to `false`, and API key support is disabled. (bool) @@ -70,8 +67,6 @@ Enable API key authorization by setting `enabled` to `true`. By default, `enable Not using Elastic APM agents? When enabled, third-party APM agents must include a valid API key in the following format: `Authorization: ApiKey `. The key must be the base64 encoded representation of the API key’s `id:name`. :::: - - ### API key limit [_api_key_limit] Each unique API key triggers one request to {{es}}. This setting restricts the number of unique API keys are allowed per minute. The minimum value for this setting should be the number of API keys configured in your monitored services. The default `limit` is `100`. (int) @@ -81,7 +76,6 @@ Each unique API key triggers one request to {{es}}. This setting restricts the n | APM Server binary | `auth.api_key.limit` | | Fleet-managed | `Number of keys` | - ### Secret token [_secret_token] Authorization token for sending APM data. The same token must also be set in each {{apm-agent}}. This token is not used for RUM endpoints. (text) @@ -91,7 +85,6 @@ Authorization token for sending APM data. The same token must also be set in eac | APM Server binary | `auth.api_key.token` | | Fleet-managed | `Secret token` | - ## `auth.api_key.elasticsearch.*` configuration options [_auth_api_key_elasticsearch_configuration_options] ::::{note} @@ -103,83 +96,66 @@ All of the `auth.api_key.elasticsearch.*` configurations are optional. If none a :::: - - ### `elasticsearch.hosts` [_elasticsearch_hosts] API keys are fetched from {{es}}. This configuration needs to point to a secured {{es}} cluster that is able to serve API key requests. - ### `elasticsearch.protocol` [_elasticsearch_protocol] The name of the protocol {{es}} is reachable on. The options are: `http` or `https`. The default is `http`. If nothing is configured, configuration settings from the `output` section will be reused. - ### `elasticsearch.path` [_elasticsearch_path] An optional HTTP path prefix that is prepended to the HTTP API calls. If nothing is configured, configuration settings from the `output` section will be reused. - ### `elasticsearch.proxy_url` [_elasticsearch_proxy_url] The URL of the proxy to use when connecting to the {{es}} servers. The value may be either a complete URL or a "host[:port]", in which case the "http"scheme is assumed. If nothing is configured, configuration settings from the `output` section will be reused. - ### `elasticsearch.timeout` [_elasticsearch_timeout] The HTTP request timeout in seconds for the {{es}} request. If nothing is configured, configuration settings from the `output` section will be reused. - ## `auth.api_key.elasticsearch.ssl.*` configuration options [_auth_api_key_elasticsearch_ssl_configuration_options] SSL is off by default. Set `elasticsearch.protocol` to `https` if you want to enable `https`. - ### `elasticsearch.ssl.enabled` [_elasticsearch_ssl_enabled] Enable custom SSL settings. Set to false to ignore custom SSL settings for secure communication. - ### `elasticsearch.ssl.verification_mode` [_elasticsearch_ssl_verification_mode] Configure SSL verification mode. If `none` is configured, all server hosts and certificates will be accepted. In this mode, SSL based connections are susceptible to man-in-the-middle attacks. **Use only for testing**. Default is `full`. - ### `elasticsearch.ssl.supported_protocols` [_elasticsearch_ssl_supported_protocols] List of supported/valid TLS versions. By default, all TLS versions from 1.0 to 1.2 are enabled. - ### `elasticsearch.ssl.certificate_authorities` [_elasticsearch_ssl_certificate_authorities] List of root certificates for HTTPS server verifications. - ### `elasticsearch.ssl.certificate` [_elasticsearch_ssl_certificate] The path to the certificate for SSL client authentication. - ### `elasticsearch.ssl.key` [_elasticsearch_ssl_key] The client certificate key used for client authentication. This option is required if certificate is specified. - ### `elasticsearch.ssl.key_passphrase` [_elasticsearch_ssl_key_passphrase] An optional passphrase used to decrypt an encrypted key stored in the configured key file. - ### `elasticsearch.ssl.cipher_suites` [_elasticsearch_ssl_cipher_suites] The list of cipher suites to use. The first entry has the highest priority. If this option is omitted, the Go crypto library’s default suites are used (recommended). - ### `elasticsearch.ssl.curve_types` [_elasticsearch_ssl_curve_types] The list of curve types for ECDHE (Elliptic Curve Diffie-Hellman ephemeral key exchange). - ### `elasticsearch.ssl.renegotiation` [_elasticsearch_ssl_renegotiation] Configure what types of renegotiation are supported. Valid options are `never`, `once`, and `freely`. Default is `never`. diff --git a/solutions/observability/apps/apm-agent-central-configuration.md b/solutions/observability/apps/apm-agent-central-configuration.md index 160882853c..3584143c96 100644 --- a/solutions/observability/apps/apm-agent-central-configuration.md +++ b/solutions/observability/apps/apm-agent-central-configuration.md @@ -6,8 +6,6 @@ applies_to: stack: --- - - # APM agent central configuration [apm-agent-configuration] APM Agent configuration allows you to fine-tune your APM agent configuration from within the Applications UI. Changes are automatically propagated to your APM agents, so there’s no need to redeploy. @@ -19,12 +17,10 @@ To get started, choose the services and environments you wish to configure. The :screenshot: ::: - ## Precedence [_precedence] Configurations set from the Applications UI take precedence over configurations set locally in each APM agent. However, if APM Server is slow to respond, is offline, reports an error, etc., APM agents will use local defaults until they’re able to update the configuration. For this reason, it is still essential to set custom default configurations locally in each of your APM agents. - ## Supported configurations [_supported_configurations] Each APM agent has a list of supported configurations. After selecting a Service name and environment in the Applications UI, a list of all supported configuration options, including descriptions and default values, will be displayed. @@ -61,7 +57,6 @@ Ruby agent Real User Monitoring (RUM) agent : [Configuration reference](apm-agent-rum-js://reference/configuration.md) - ## APM Server configuration [_apm_server_configuration] For most users, APM agent configuration should work out-of-the-box. If you run into trouble, it may be because you’re not using the {{es}} output, or because your {{es}} credentials don’t have sufficient privileges. diff --git a/solutions/observability/apps/apm-agent-compatibility.md b/solutions/observability/apps/apm-agent-compatibility.md index 486c0b600d..d490c0174e 100644 --- a/solutions/observability/apps/apm-agent-compatibility.md +++ b/solutions/observability/apps/apm-agent-compatibility.md @@ -9,21 +9,18 @@ applies_to: The chart below outlines the compatibility between different versions of Elastic APM agents and extensions with the APM integration. - ## APM AWS Lambda extension [_apm_aws_lambda_extension] | {{apm-agent}} version | APM integration version | | --- | --- | | `1.x` | ≥ `8.2` | - ## Android agent [_android_agent] | {{apm-agent}} version | APM integration version | | --- | --- | | `0.x` | ≥ `8.12` | - ## Go agent [_go_agent] | {{apm-agent}} version | APM integration version | @@ -31,14 +28,12 @@ The chart below outlines the compatibility between different versions of Elastic | `1.x` | ≥ `6.5` | | `2.x` | ≥ `6.5` | - ## iOS agent [_ios_agent] | {{apm-agent}} version | APM integration version | | --- | --- | | `1.x` | ≥ `8.12` | - ## Java agent [_java_agent] | {{apm-agent}} version | APM integration version | @@ -50,36 +45,30 @@ Java agent < 1.43.0 not fully compatible with APM Server >= 8.11.0. For more inf :::: - - ## .NET agent [_net_agent] | {{apm-agent}} version | APM integration version | | --- | --- | | `1.x` | ≥ `6.5` | - ## Node.js agent [_node_js_agent] | {{apm-agent}} version | APM integration version | | --- | --- | | `3.x` | ≥ `6.6` | - ## PHP agent [_php_agent] | {{apm-agent}} version | APM integration version | | --- | --- | | `1.x` | ≥ `7.0` | - ## Python agent [_python_agent] | {{apm-agent}} version | APM integration version | | --- | --- | | `6.x` | ≥ `6.6` | - ## Ruby agent [_ruby_agent] | {{apm-agent}} version | APM integration version | @@ -87,7 +76,6 @@ Java agent < 1.43.0 not fully compatible with APM Server >= 8.11.0. For more inf | `3.x` | ≥ `6.5` | | `4.x` | ≥ `6.5` | - ## JavaScript RUM agent [_javascript_rum_agent] | {{apm-agent}} version | APM integration version | diff --git a/solutions/observability/apps/apm-agent-explorer.md b/solutions/observability/apps/apm-agent-explorer.md index 066c832f3a..e1381a9175 100644 --- a/solutions/observability/apps/apm-agent-explorer.md +++ b/solutions/observability/apps/apm-agent-explorer.md @@ -7,12 +7,10 @@ applies_to: # APM Agent explorer [apm-agent-explorer] - ::::{important} This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. :::: - APM agent explorer provides a centralized panel to identify APM agent deployment details, like service name, environment, instances, and agent name, version, and documentation. :::{image} /solutions/images/observability-apm-agent-explorer.png diff --git a/solutions/observability/apps/apm-agent-tls-communication.md b/solutions/observability/apps/apm-agent-tls-communication.md index b139517c4d..9919023d20 100644 --- a/solutions/observability/apps/apm-agent-tls-communication.md +++ b/solutions/observability/apps/apm-agent-tls-communication.md @@ -11,10 +11,8 @@ TLS is disabled by default. When TLS is enabled for APM Server inbound communica When TLS is enabled, a certificate and corresponding private key are required. The certificate and private key can either be issued by a trusted certificate authority (CA) or be [self-signed](#apm-agent-self-sign). - ## Use a self-signed certificate [apm-agent-self-sign] - ### Step 1: Create a self-signed certificate [apm-agent-self-sign-1] The {{es}} distribution offers the `certutil` tool for the creation of self-signed certificates: @@ -24,7 +22,6 @@ The {{es}} distribution offers the `certutil` tool for the creation of self-sign 3. Create the self-signed certificate: `./bin/elasticsearch-certutil cert --ca-cert /ca.crt --ca-key /ca.key --pem --name localhost` 4. Extract the certificate and key from the resulted zip archive. - ### Step 2: Configure the APM Server [apm-agent-self-sign-2] Enable TLS and configure the APM Server to point to the extracted certificate and key: @@ -74,7 +71,6 @@ We do not recommend disabling {{apm-agent}} verification of the server’s certi * **Ruby agent**: [`verify_server_cert`](apm-agent-ruby://reference/configuration.md#config-verify-server-cert) * **Node.js agent**: [`verifyServerCert`](apm-agent-nodejs://reference/configuration.md#validate-server-cert) - ## Client certificate authentication [apm-agent-client-cert] APM Server does not require agents to provide a certificate for authentication, and there is no dedicated support for SSL/TLS client certificate authentication in Elastic’s backend agents. diff --git a/solutions/observability/apps/apm-apis.md b/solutions/observability/apps/apm-apis.md index 98c1416944..7fe8620a36 100644 --- a/solutions/observability/apps/apm-apis.md +++ b/solutions/observability/apps/apm-apis.md @@ -3,16 +3,19 @@ mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-apis.html applies_to: stack: + serverless: --- # APM APIs [apm-apis] +:::{include} _snippets/apm-server-vs-mis.md +::: + There are two kinds of APIs related to Elastic APM: | | | | --- | --- | | [APM UI API](apm-ui-api.md) | {{kib}} APIs specific to working with the Applications UI including updating configuration options, uploading real user monitoring (RUM) source maps, adding annotations, and more. | -| [APM Server API](apm-server-api.md) | APIs for working with APM Server. These are mainly intake APIs that accept data from APM agents and are used primarily by APM agent developers. | - - +| [APM Server API](apm-server-api.md) | APIs for working with APM Server. These are mainly intake APIs that accept data from APM agents and are used primarily by APM agent developers. | +| [Observability Intake Serverless API](/solutions/observability/apps/managed-intake-service-event-api.md) | The managed intake service exposes an API endpoint to query general server information. This lightweight endpoint is useful as a server up/down health check. This API is exclusively for APM agent developers. | diff --git a/solutions/observability/apps/apm-k8s-attacher.md b/solutions/observability/apps/apm-k8s-attacher.md index d3d8a3fa7e..1748ee5813 100644 --- a/solutions/observability/apps/apm-k8s-attacher.md +++ b/solutions/observability/apps/apm-k8s-attacher.md @@ -6,11 +6,8 @@ applies_to: stack: --- - - # APM K8S Attacher [apm-k8s-attacher] - The [APM attacher for Kubernetes](apm-k8s-attacher://reference/index.md) simplifies the instrumentation and configuration of your application pods. The attacher includes a webhook receiver that modifies pods so they are automatically instrumented by an Elastic APM agent. Ready to get started? See [Instrument and configure pods](apm-k8s-attacher://reference/apm-get-started-webhook.md) to get started. diff --git a/solutions/observability/apps/apm-reader-user.md b/solutions/observability/apps/apm-reader-user.md index 1fb3fd6c15..d8c93ccee0 100644 --- a/solutions/observability/apps/apm-reader-user.md +++ b/solutions/observability/apps/apm-reader-user.md @@ -6,14 +6,10 @@ applies_to: stack: --- - - # APM reader user [apm-app-reader] - APM reader users typically need to view the Applications UI and dashboards and visualizations that use APM data. These users might also need to create and edit dashboards, visualizations, and machine learning jobs. - ## APM reader [apm-app-reader-full] To create an APM reader user: @@ -56,8 +52,6 @@ To create an APM reader user: | `kibana_admin` | Grants access to all features in Kibana. | | `machine_learning_admin` | Grants the privileges required to create, update, and view machine learning jobs | - - ## Partial APM reader [apm-app-reader-partial] In some instances, you may wish to restrict certain Kibana apps that a user has access to. @@ -93,7 +87,6 @@ In some instances, you may wish to restrict certain Kibana apps that a user has :::: - 2. Assign feature privileges to any Kibana feature that the user needs access to. Here are two examples: | Type | Privilege | Purpose | @@ -107,4 +100,3 @@ In some instances, you may wish to restrict certain Kibana apps that a user has | --- | --- | | `machine_learning_admin` | Grants the privileges required to create, update, and view machine learning jobs | - diff --git a/solutions/observability/apps/apm-server-advanced-setup.md b/solutions/observability/apps/apm-server-advanced-setup.md index c6b77196e6..1c59ff90a4 100644 --- a/solutions/observability/apps/apm-server-advanced-setup.md +++ b/solutions/observability/apps/apm-server-advanced-setup.md @@ -6,11 +6,8 @@ applies_to: stack: --- - - # APM Server advanced setup [apm-setting-up-and-running] - Before reading this section, see the [getting started documentation](fleet-managed-apm-server.md) for basic installation and running instructions. This section includes additional information on how to set up and run APM Server, including: @@ -22,9 +19,3 @@ This section includes additional information on how to set up and run APM Server * [High Availability](high-availability.md) * [Run APM Server on Docker](apm-server-binary.md#apm-running-on-docker) - - - - - - diff --git a/solutions/observability/apps/apm-server-api.md b/solutions/observability/apps/apm-server-api.md index d569838929..174b0536c9 100644 --- a/solutions/observability/apps/apm-server-api.md +++ b/solutions/observability/apps/apm-server-api.md @@ -15,8 +15,3 @@ The APM Server exposes endpoints for: * [OpenTelemetry intake API](opentelemetry-intake-api.md) * [Jaeger event intake](jaeger-event-intake.md) - - - - - diff --git a/solutions/observability/apps/apm-server-binary.md b/solutions/observability/apps/apm-server-binary.md index 0bf2e38e29..01e6d0df6c 100644 --- a/solutions/observability/apps/apm-server-binary.md +++ b/solutions/observability/apps/apm-server-binary.md @@ -9,7 +9,6 @@ applies_to: This guide will explain how to set up and configure the APM Server binary. - ## Prerequisites [_prerequisites_7] First, see the [Elastic Support Matrix](https://www.elastic.co/support/matrix) for information about supported operating systems and product compatibility. @@ -19,20 +18,18 @@ You’ll need: * **{{es}}** for storing and indexing data. * **{{kib}}** for visualizing with the Applications UI. -We recommend you use the same version of {{es}}, {{kib}}, and APM Server. See [Installing the {{stack}}](../../../get-started/the-stack.md) for more information about installing these products. +We recommend you use the same version of {{es}}, {{kib}}, and APM Server. See [Installing the {{stack}}](/get-started/the-stack.md) for more information about installing these products. :::{image} /solutions/images/observability-apm-architecture-diy.png :alt: Install Elastic APM yourself ::: - ## Step 1: Install [apm-installing] ::::{note} -**Before you begin**: If you haven’t installed the {{stack}}, do that now. See [Learn how to install the {{stack}} on your own hardware](../../../get-started/the-stack.md). +**Before you begin**: If you haven’t installed the {{stack}}, do that now. See [Learn how to install the {{stack}} on your own hardware](/get-started/the-stack.md). :::: - To download and install APM Server, use the commands below that work with your system. If you use `apt` or `yum`, you can [install APM Server from our repositories](#apm-setup-repositories) to update to the newest version more easily. $$$apm-deb$$$ @@ -87,7 +84,6 @@ $$$apm-docker$$$ See [Running on Docker](#apm-running-on-docker) for deploying Docker containers. - ## Step 2: Set up and configure [apm-server-configuration] Configure APM by editing the `apm-server.yml` configuration file. The location of this file varies by platform—​see the [Installation layout](installation-layout.md) for help locating it. @@ -107,10 +103,8 @@ output.elasticsearch: 2. The {{es}} `host:port` to connect to. 3. This example uses basic authentication. The user provided here needs the privileges required to publish events to {{es}}. To create a dedicated user for this role, see [Create a *writer* role](create-assign-feature-roles-to-apm-server-users.md#apm-privileges-to-publish-events). - All available configuration options are outlined in [configuring APM Server](configure-apm-server.md). - ## Step 3: Start [apm-server-starting] In a production environment, you would put APM Server on its own machines, similar to how you run {{es}}. You *can* run it on the same machines as {{es}}, but this is not recommended, as the processes will be competing for resources. @@ -125,14 +119,12 @@ To start APM Server, run: The `-e` [global flag](apm-server-command-reference.md#apm-global-flags) enables logging to stderr and disables syslog/file output. Remove this flag if you’ve enabled logging in the configuration file. For Linux systems, see [APM Server status and logs](apm-server-systemd.md). :::: - You should see APM Server start up. It will try to connect to {{es}} on localhost port `9200` and expose an API to agents on port `8200`. You can change the defaults in `apm-server.yml` or by supplying a different address on the command line: ```bash ./apm-server -e -E output.elasticsearch.hosts=ElasticsearchAddress:9200 -E apm-server.host=localhost:8200 ``` - ### Debian Package / RPM [apm-running-deb-rpm] For Debian package and RPM installations, we recommend the `apm-server` process runs as a non-root user. Therefore, these installation methods create an `apm-server` user which you can use to start the process. In addition, APM Server will only start if the configuration file is [owned by the user running the process](apm-server-systemd.md#apm-config-file-ownership). @@ -145,7 +137,6 @@ sudo -u apm-server apm-server [] By default, APM Server loads its configuration file from `/etc/apm-server/apm-server.yml`. See the [deb & rpm default paths](installation-layout.md) for a full directory layout. - ## Step 4: Install APM agents [apm-next-steps] :::::::{tab-set} @@ -165,7 +156,6 @@ plugins { 1. The Elastic plugin declaration must be added below the Android app plugin declaration (`com.android.application`) and below the Kotlin plugin declaration (if used). - **2. Configure the agent** After adding the agent plugin, configure it. A minimal configuration sets the Elastic APM integration endpoint as shown below: @@ -195,12 +185,10 @@ elasticApm { 4. Defaults to null. More info on API Keys [here](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key). 5. Defaults to null. - ::::{note} When both `secretToken` and `apiKey` are provided, apiKey has priority and secretToken is ignored. :::: - **3. Initialize the agent** After syncing your project with the Gradle changes above, the Elastic APM agent needs to be initialized within your [Application class](https://developer.android.com/reference/android/app/Application). This example shows the simplest way to configure the agent: @@ -220,7 +208,6 @@ class MyApp extends android.app.Application { 1. Initialize the Elastic APM agent once. - All that’s left is to compile and run your application. That’s it! **Learn more in the agent reference** @@ -328,7 +315,6 @@ struct MyApp: App { 1. The APM integration host and port 2. Secret token for APM integration connection - If you’re not using `SwiftUI`, you can alternatively add the same thing to your `AppDelegate.swift` file: ```swift @@ -350,7 +336,6 @@ var config = AgentConfigBuilder() 1. The APM integration host and port 2. Secret token for APM integration connection - **Learn more in the agent reference** Read more in the [APM iOS Agent Reference](apm-agent-ios://reference/index.md). @@ -370,7 +355,6 @@ The first step in getting started with the Elastic APM Java agent is to retrieve curl -o 'elastic-apm-agent.jar' -L 'https://oss.sonatype.org/service/local/artifact/maven/redirect?r=releases&g=co.elastic.apm&a=elastic-apm-agent&v=LATEST' ``` - **2. Add `-javaagent` flag** When starting your application, add the JVM flag `-javaagent:/path/to/elastic-apm-agent-.jar` @@ -479,7 +463,6 @@ If you can’t find your distribution, you can install the agent by building it The agent is currently only available for Linux operating system. :::: - 1. Download the agent source from [https://github.com/elastic/apm-agent-php/](https://github.com/elastic/apm-agent-php/). 2. Execute the following commands to build the agent and install it: @@ -560,7 +543,6 @@ MIDDLEWARE = ( ) ``` - Flask : **1. Install the {{apm-agent}}** @@ -597,7 +579,6 @@ app.config['ELASTIC_APM'] = { apm = ElasticAPM(app) ``` - **Learn more in the agent reference** * [Supported technologies](apm-agent-python://reference/supported-technologies.md) @@ -632,7 +613,6 @@ secret_token: '' server_url: 'http://localhost:8200' ``` - Rack : For Rack or a compatible framework, like Sinatra, include the middleware in your app and start the agent. @@ -674,7 +654,6 @@ secret_token: '' server_url: 'http://localhost:8200' ``` - **Learn more in the agent reference** * [Supported technologies](apm-agent-ruby://reference/supported-technologies.md) @@ -770,7 +749,6 @@ Once you have at least one {{apm-agent}} sending data to APM Server, you can sta :screenshot: ::: - ## Repositories for APT and YUM [apm-setup-repositories] We have repositories available for APT and YUM-based distributions. Note that we provide binary packages, but no source packages. @@ -782,25 +760,21 @@ We use the PGP key [D88E42B4](https://pgp.mit.edu/pks/lookup?op=vindex&search=0x ``` to sign all our packages. It is available from [https://pgp.mit.edu](https://pgp.mit.edu). - ### APT [_apt] Version 9.0.0-beta1 of apm-server has not yet been released. - ### YUM [_yum] Version 9.0.0-beta1 of apm-server has not yet been released. - ## Run APM Server on Docker [apm-running-on-docker] Docker images for APM Server are available from the Elastic Docker registry. The base image is [ubuntu:22.04](https://hub.docker.com/_/ubuntu). A list of all published Docker images and tags is available at [www.docker.elastic.co](https://www.docker.elastic.co). -These images are free to use under the Elastic license. They contain open source and free commercial features and access to paid commercial features. [Start a 30-day trial](../../../deploy-manage/license/manage-your-license-in-self-managed-cluster.md) to try out all of the paid commercial features. See the [Subscriptions](https://www.elastic.co/subscriptions) page for information about Elastic license levels. - +These images are free to use under the Elastic license. They contain open source and free commercial features and access to paid commercial features. [Start a 30-day trial](/deploy-manage/license/manage-your-license-in-self-managed-cluster.md) to try out all of the paid commercial features. See the [Subscriptions](https://www.elastic.co/subscriptions) page for information about Elastic license levels. ### Pull the image [_pull_the_image] @@ -808,12 +782,10 @@ Obtaining APM Server for Docker is as simple as issuing a `docker pull` command However, version 9.0.0-beta1 of APM Server has not yet been released, so no Docker image is currently available for this version. - ### Configure APM Server on Docker [_configure_apm_server_on_docker] The Docker image provides several methods for configuring APM Server. The conventional approach is to provide a configuration file via a volume mount, but it’s also possible to create a custom image with your configuration included. - #### Example configuration file [_example_configuration_file] Download this example configuration file as a starting point: @@ -822,7 +794,6 @@ Download this example configuration file as a starting point: curl -L -O https://raw.githubusercontent.com/elastic/apm-server/master/apm-server.docker.yml ``` - #### Volume-mounted configuration [_volume_mounted_configuration] One way to configure APM Server on Docker is to provide `apm-server.docker.yml` via a volume mount. With `docker run`, the volume mount can be specified like this. @@ -841,13 +812,10 @@ docker run -d \ 1. Substitute your {{es}} hosts and ports. 2. If you are using {{ech}}, replace the `-E output.elasticsearch.hosts` line with the Cloud ID and elastic password using the syntax shown earlier. - - #### Customize your configuration [_customize_your_configuration] The `apm-server.docker.yml` downloaded earlier should be customized for your environment. See [Configure APM Server](configure-apm-server.md) for more details. Edit the configuration file and customize it to match your environment then re-deploy your APM Server container. - #### Custom image configuration [_custom_image_configuration] It’s possible to embed your APM Server configuration in a custom image. Here is an example Dockerfile to achieve this: diff --git a/solutions/observability/apps/apm-server-command-reference.md b/solutions/observability/apps/apm-server-command-reference.md index 5cd51ba28d..0b5559cf2a 100644 --- a/solutions/observability/apps/apm-server-command-reference.md +++ b/solutions/observability/apps/apm-server-command-reference.md @@ -6,16 +6,12 @@ applies_to: stack: --- - - # APM Server command reference [apm-command-line-options] - ::::{important} These commands only apply to the APM Server binary installation method. :::: - APM Server provides a command-line interface for starting APM Server and performing common tasks, like testing configuration files. The command-line also supports [global flags](#apm-global-flags) for controlling global behaviors. @@ -28,8 +24,7 @@ Use `sudo` to run the following commands if: :::: - -Some of the features described here require an Elastic license. For more information, see [https://www.elastic.co/subscriptions](https://www.elastic.co/subscriptions) and [License Management](../../../deploy-manage/license/manage-your-license-in-self-managed-cluster.md). +Some of the features described here require an Elastic license. For more information, see [https://www.elastic.co/subscriptions](https://www.elastic.co/subscriptions) and [License Management](/deploy-manage/license/manage-your-license-in-self-managed-cluster.md). | Commands | | | --- | --- | @@ -43,21 +38,18 @@ Some of the features described here require an Elastic license. For more informa Also see [Global flags](#apm-global-flags). - ## `apikey` command [apm-apikey-command] ::::{warning} This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. :::: - ::::{admonition} Deprecated in 8.6.0. :class: warning Users should create API Keys through {{kib}} or the {{es}} REST API. See [API keys](api-keys.md). :::: - Communication between APM agents and APM Server now supports sending an [API Key in the Authorization header](api-keys.md). APM Server provides an `apikey` command that can create, verify, invalidate, and show information about API Keys for agent/server communication. Most operations require the `manage_own_api_key` cluster privilege, and you must ensure that `apm-server.api_key` or `output.elasticsearch` are configured appropriately. **SYNOPSIS** @@ -88,10 +80,8 @@ apm-server apikey SUBCOMMAND [FLAGS] * To **ingest agent data**, assign `event:write`. * To **upload source maps**, assign `sourcemap:write`. - :::: - **`info`** : Query API Key(s). `--id` or `--name` required. @@ -142,7 +132,6 @@ apm-server apikey invalidate --name example-001 For more information, see [API keys](api-keys.md). - ## `export` command [apm-export-command] Exports the configuration, index template, or {{ilm-init}} policy to stdout. You can use this command to quickly view your configuration or see the contents of the index template or the {{ilm-init}} policy. @@ -189,7 +178,6 @@ apm-server export config apm-server export template --es.version 9.0.0-beta1 --index myindexname ``` - ## `help` command [apm-help-command] Shows help for any command. If no command is specified, shows help for the `run` command. @@ -216,7 +204,6 @@ Also see [Global flags](#apm-global-flags). apm-server help export ``` - ## `keystore` command [apm-keystore-command] Manages the [secrets keystore](secrets-keystore-for-secure-settings.md). @@ -265,7 +252,6 @@ apm-server keystore list See [Secrets keystore](secrets-keystore-for-secure-settings.md) for more examples. - ## `run` command [apm-run-command] Runs APM Server. This command is used by default if you start APM Server without specifying a command. @@ -316,7 +302,6 @@ Or: apm-server -e ``` - ## `test` command [apm-test-command] Tests the configuration. @@ -348,7 +333,6 @@ Also see [Global flags](#apm-global-flags). apm-server test config ``` - ## `version` command [apm-version-command] Shows information about the current version. @@ -372,7 +356,6 @@ Also see [Global flags](#apm-global-flags). apm-server version ``` - ## Global flags [apm-global-flags] These global flags are available whenever you run APM Server. @@ -386,7 +369,6 @@ These global flags are available whenever you run APM Server. This setting is applied to the currently running APM Server process. The APM Server configuration file is not changed. - **`-c, --c FILE`** : Specifies the configuration file to use for APM Server. The file you specify here is relative to `path.config`. If the `-c` flag is not specified, the default config file, `apm-server.yml`, is used. diff --git a/solutions/observability/apps/apm-server-information-api.md b/solutions/observability/apps/apm-server-information-api.md index 63d8d50843..c9485fcb86 100644 --- a/solutions/observability/apps/apm-server-information-api.md +++ b/solutions/observability/apps/apm-server-information-api.md @@ -9,7 +9,6 @@ applies_to: The APM Server exposes an API endpoint to query general server information. This lightweight endpoint is useful as a server up/down health check. - ## Server Information endpoint [apm-api-info-endpoint] This is the server information endpoint: @@ -24,7 +23,6 @@ To configure authenticated access to the APM server, the instructions at [APM AP If an [API keys](api-keys.md) or a [Secret token](secret-token.md) is passed along with the `HTTP GET` request, in addition to an HTTP 200, the response payload will include some information about the APM server. - ### Example: GET, without credentials [apm-api-info-example-get-without-credentials] Example APM Server status request with GET, without credentials: @@ -49,7 +47,6 @@ curl --verbose -X GET http://127.0.0.1:8200 * Connection #0 to host 127.0.0.1 left intact ``` - ### Example: GET, with secret token [apm-api-info-example-get-with-secret-token] Example APM Server information request with GET, with a [Secret token](secret-token.md): diff --git a/solutions/observability/apps/apm-server-performance-diagnostic.md b/solutions/observability/apps/apm-server-performance-diagnostic.md deleted file mode 100644 index 003d555e47..0000000000 --- a/solutions/observability/apps/apm-server-performance-diagnostic.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/observability/current/apm-performance-diagnostic.html -applies_to: - stack: ---- - -# APM Server performance diagnostic [apm-performance-diagnostic] - - -## Diagnosing backpressure from {{es}} [apm-es-backpressure] - -When {{es}} is under excessive load or indexing pressure, APM Server could experience the downstream backpressure when indexing new documents into {{es}}. Most commonly, backpressure from {{es}} will manifest itself in the form of higher indexing latency and/or rejected requests, which in return could lead APM Server to deny incoming requests. As a result, APM agents connected to the affected APM Server will suffer from throttling and/or request timeout when shipping APM events. - -To quickly identify possible issues try looking for similar error logs lines in APM Server logs: - -```json -... -{"log.level":"error","@timestamp":"2024-07-27T23:46:28.529Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"v2@v2.2.0/appender.go","file.line":370},"message":"bulk indexing request failed","service.name":"apm-server","error":{"message":"flush failed (429): [429 Too Many Requests]"},"ecs.version":"1.6.0"} -{"log.level":"error","@timestamp":"2024-07-27T23:55:38.612Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"v2@v2.2.0/appender.go","file.line":370},"message":"bulk indexing request failed","service.name":"apm-server","error":{"message":"flush failed (503): [503 Service Unavailable]"},"ecs.version":"1.6.0"} -... -``` - -To gain better insight into APM Server health and performance, consider enabling the monitoring feature by following the steps in [Monitor APM Server](monitor-apm-server.md). When enabled, APM Server will additionally report a set of vital metrics to help you identify any performance degradation. - -Pay careful attention to the next metric fields: - -* `beats_stats.metrics.libbeat.output.events.active` that represents the number of buffered pending documents waiting to be ingested; (*if this value is increasing rapidly it may indicate {{es}} backpressure*) -* `beats_stats.metrics.libbeat.output.events.acked` that represents the total number of documents that have been ingested successfully; -* `beats_stats.metrics.libbeat.output.events.failed` that represents the total number of documents that failed to ingest; (*if this value is increasing rapidly it may indicate {{es}} backpressure*) -* `beats_stats.metrics.libbeat.output.events.toomany` that represents the number of documents that failed to ingest due to {{es}} responding with 429 Too many Requests; (*if this value is increasing rapidly it may indicate {{es}} backpressure*) -* `beats_stats.output.elasticsearch.bulk_requests.available` that represents the number of bulk indexers available for making bulk index requests; (*if this value is equal to 0 it may indicate {{es}} backpressure*) -* `beats_stats.output.elasticsearch.bulk_requests.completed` that represents the number of already completed bulk requests; -* `beats_stats.metrics.output.elasticsearch.indexers.active` that represents the number of active bulk indexers that are concurrently processing batches; - -See [{{metricbeat}} documentation](beats://reference/metricbeat/exported-fields-beat.md) for the full list of exported metric fields. - -One likely cause of excessive indexing pressure or rejected requests is undersized {{es}}. To mitigate this, follow the guidance in [Rejected requests](../../../troubleshoot/elasticsearch/rejected-requests.md). - -(Not recommended) If scaling {{es}} resources up is not an option, you can adjust the `flush_bytes`, `flush_interval`, `max_retries` and `timeout` settings described in [Configure the Elasticsearch output](configure-elasticsearch-output.md) to reduce APM Server indexing pressure. However, consider that increasing number of buffered documents and/or reducing retries may lead to a higher rate of dropped APM events. Down below a custom configuration example is listed where the number of default buffered documents is roughly doubled while {{es}} indexing retries are decreased simultaneously. This configuration provides a generic example and might not be applicable to your situation. Try adjusting the settings further to see what works for you. - -```yaml -output.elasticsearch: - flush_bytes: "2MB" # double the default value - flush_interval: "2s" # double the default value - max_retries: 1 # decrease the default value - timeout: 60 # decrease the default value -``` - diff --git a/solutions/observability/apps/apm-server-response-codes.md b/solutions/observability/apps/apm-server-response-codes.md deleted file mode 100644 index ecabffb9e1..0000000000 --- a/solutions/observability/apps/apm-server-response-codes.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/observability/current/apm-common-response-codes.html -applies_to: - stack: ---- - -# APM Server response codes [apm-common-response-codes] - - -## HTTP 400: Data decoding error / Data validation error [apm-bad-request] - -The most likely cause for this error is using incompatible versions of {{apm-agent}} and APM Server. See the [agent/server compatibility matrix](apm-agent-compatibility.md) to verify compatibility. - - -## HTTP 400: Event too large [apm-event-too-large] - -APM agents communicate with the APM server by sending events in an HTTP request. Each event is sent as its own line in the HTTP request body. If events are too large, you should consider increasing the [Max event size](general-configuration-options.md#apm-max_event_size) setting in the APM integration, and adjusting relevant settings in the agent. - - -## HTTP 401: Invalid token [apm-unauthorized] - -Either the [Secret token](secret-token.md) in the request header doesn’t match the secret token configured in the APM integration, or the [API keys](api-keys.md) is invalid. - - -## HTTP 403: Forbidden request [apm-forbidden] - -Either you are sending requests to a [RUM](real-user-monitoring-rum.md) endpoint without RUM enabled, or a request is coming from an origin not specified in the APM integration settings. See the [Allowed origins](configure-real-user-monitoring-rum.md#apm-rum-allow-origins) setting for more information. - - -## HTTP 503: Request timed out waiting to be processed [apm-request-timed-out] - -This happens when APM Server exceeds the maximum number of requests that it can process concurrently. To alleviate this problem, you can try to: reduce the sample rate and/or reduce the collected stack trace information. See [Reduce storage](reduce-storage.md) for more information. - -Another option is to increase processing power. This can be done by either migrating your {{agent}} to a more powerful machine or adding more APM Server instances. - diff --git a/solutions/observability/apps/apm-server-systemd.md b/solutions/observability/apps/apm-server-systemd.md index ac8eb8db88..2e1b0bbf74 100644 --- a/solutions/observability/apps/apm-server-systemd.md +++ b/solutions/observability/apps/apm-server-systemd.md @@ -11,12 +11,10 @@ applies_to: These commands only apply to the APM Server binary installation method. Fleet-managed users should see [Start and stop {{agent}}s on edge hosts](/reference/fleet/start-stop-elastic-agent.md). :::: - The DEB and RPM packages include a service unit for Linux systems with systemd. On these systems, you can manage APM Server by using the usual systemd commands. We recommend that the apm-server process is run as a non-root user. Therefore, that is the default setup for APM Server’s DEB package and RPM installation. - ## Start and stop APM Server [_start_and_stop_apm_server] Use `systemctl` to start or stop APM Server: @@ -39,7 +37,6 @@ sudo systemctl enable apm-server sudo systemctl disable apm-server ``` - ## APM Server status and logs [_apm_server_status_and_logs] To get the service status, use `systemctl`: @@ -54,7 +51,6 @@ Logs are stored by default in journald. To view the Logs, use `journalctl`: journalctl -u apm-server.service ``` - ## Customize systemd unit for APM Server [_customize_systemd_unit_for_apm_server] The systemd service unit file includes environment variables that you can override to change the default options. @@ -69,7 +65,6 @@ The systemd service unit file includes environment variables that you can overri You can use `BEAT_LOG_OPTS` to set debug selectors for logging. However, to configure logging behavior, set the logging options described in [Configure logging](configure-logging.md). :::: - To override these variables, create a drop-in unit file in the `/etc/systemd/system/apm-server.service.d` directory. For example a file with the following content placed in `/etc/systemd/system/apm-server.service.d/debug.conf` would override `BEAT_LOG_OPTS` to enable debug for {{es}} output. @@ -90,8 +85,6 @@ systemctl restart apm-server It is recommended that you use a configuration management tool to include drop-in unit files. If you need to add a drop-in manually, use `systemctl edit apm-server.service`. :::: - - #### Configuration file ownership [apm-config-file-ownership] On systems with POSIX file permissions, the APM Server configuration file is subject to ownership and file permission checks. These checks prevent unauthorized users from providing or modifying configurations that are run by APM Server. @@ -119,7 +112,6 @@ can only be writable by the owner but the permissions are "-rw-rw-r--" To correct this problem, use `chmod go-w /etc/apm-server/apm-server.yml` to remove write privileges from anyone other than the owner. - ##### Disabling strict permission checks [_disabling_strict_permission_checks] You can disable strict permission checks from the command line by using `--strict.perms=false`, but we strongly encourage you to leave the checks enabled. diff --git a/solutions/observability/apps/apm-ui-api.md b/solutions/observability/apps/apm-ui-api.md index 9d768aaaff..1f4a887914 100644 --- a/solutions/observability/apps/apm-ui-api.md +++ b/solutions/observability/apps/apm-ui-api.md @@ -3,69 +3,12 @@ mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-app-api.html applies_to: stack: + serverless: --- # APM UI API [apm-app-api] -Some Applications UI features are provided via a REST API: - -* Agent Configuration API ([{{stack}}](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-apm-agent-configuration) or [{{serverless-short}}](https://www.elastic.co/docs/api/doc/serverless/group/endpoint-apm-agent-configuration)) -* Annotation API ([{{stack}}](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-apm-annotations) or [{{serverless-short}}](https://www.elastic.co/docs/api/doc/serverless/group/endpoint-apm-annotations)) -* RUM source map API ([{{stack}}](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-apm-sourcemaps) or [{{serverless-short}}](https://www.elastic.co/docs/api/doc/serverless/group/endpoint-apm-sourcemaps)) -* APM agent key API ([{{stack}}](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-apm-agent-keys) or [{{serverless-short}}](https://www.elastic.co/docs/api/doc/serverless/group/endpoint-apm-agent-keys)) - - -## Using the APIs [apm-api-example] - -Interact with APM APIs using cURL or another API tool. All APM APIs are Kibana APIs, not Elasticsearch APIs; because of this, the Kibana dev tools console cannot be used to interact with APM APIs. - -For all APM APIs, you must use a request header. Supported headers are `Authorization`, `kbn-xsrf`, and `Content-Type`. - -`Authorization: ApiKey {{credentials}}` -: Kibana supports token-based authentication with the Elasticsearch API key service. The API key returned by the [Elasticsearch create API key API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key) can be used by sending a request with an `Authorization` header that has a value of `ApiKey` followed by the `{{credentials}}`, where `{{credentials}}` is the base64 encoding of `id` and `api_key` joined by a colon. - - Alternatively, you can create a user and use their username and password to authenticate API access: `-u $USER:$PASSWORD`. - - Whether using `Authorization: ApiKey {{credentials}}`, or `-u $USER:$PASSWORD`, users interacting with APM APIs must have [sufficient privileges](applications-ui-api-user.md). - - -`kbn-xsrf: true` -: By default, you must use `kbn-xsrf` for all API calls, except in the following scenarios: - - * The API endpoint uses the `GET` or `HEAD` operations - * The path is allowed using the `server.xsrf.allowlist` setting - * XSRF protections are disabled using the `server.xsrf.disableProtection` setting - - -`Content-Type: application/json` -: Applicable only when you send a payload in the API request. {{kib}} API requests and responses use JSON. Typically, if you include the `kbn-xsrf` header, you must also include the `Content-Type` header. - -Here’s an example CURL request that adds an annotation to the Applications UI: - -```bash -curl -X POST \ - http://localhost:5601/api/apm/services/opbeans-java/annotation \ --H 'Content-Type: application/json' \ --H 'kbn-xsrf: true' \ --H 'Authorization: Basic YhUlubWZhM0FDbnlQeE6WRtaW49FQmSGZ4RUWXdX' \ --d '{ - "@timestamp": "2020-05-11T10:31:30.452Z", - "service": { - "version": "1.2" - }, - "message": "Revert upgrade", - "tags": [ - "elastic.co", "customer" - ] - }' -``` - - -## Kibana API [kibana-api] - -In addition to the APM specific API endpoints, Kibana provides its own [REST API](/solutions/observability/apps/apm-server-api.md) which you can use to automate certain aspects of configuring and deploying Kibana. - - - - +Some Applications UI features are provided via the Kibana API. Read more in: +* [Kibana APIs](https://www.elastic.co/docs/api/doc/kibana/) +* [Kibana Serverless APIs](https://www.elastic.co/docs/api/doc/serverless/) \ No newline at end of file diff --git a/solutions/observability/apps/application-data-security.md b/solutions/observability/apps/application-data-security.md index 4acc8522a7..4ac3d65709 100644 --- a/solutions/observability/apps/application-data-security.md +++ b/solutions/observability/apps/application-data-security.md @@ -4,13 +4,11 @@ mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-data-security.html applies_to: stack: + serverless: --- - - # Application data security [apm-data-security] - When setting up Elastic APM, it’s essential to review all captured data carefully to ensure it doesn’t contain sensitive information like passwords, credit card numbers, or health data. In addition, you may wish to filter out other identifiable information, like IP addresses, user agent information, or form field data. Depending on the type of data, we offer several different ways to filter, manipulate, or obfuscate sensitive information during or before ingestion: @@ -20,7 +18,6 @@ Depending on the type of data, we offer several different ways to filter, manipu In addition to utilizing filters, you should regularly review the [sensitive fields](#apm-sensitive-fields) table to ensure sensitive data is not being ingested. If it is, it’s possible to remove or redact it. See [Delete sensitive data](delete-sensitive-data.md) for more information. - ## Built-in data filters [apm-built-in-data-filters] Built-in data filters allow you to filter or turn off ingestion of the following types of data: @@ -33,7 +30,6 @@ Built-in data filters allow you to filter or turn off ingestion of the following | [Real user monitoring data](built-in-data-filters.md#apm-filters-real-user-data) | URLs visited, click events, user browser errors, resources used, etc. | | [Database statements](built-in-data-filters.md#apm-filters-database-statements) | Sensitive user or business information | - ## Custom filters [apm-custom-data-filters] Custom filters allow you to filter or redact other types of APM data on ingestion: @@ -43,7 +39,6 @@ Custom filters allow you to filter or redact other types of APM data on ingestio | [Ingest pipelines](custom-filters.md#apm-filters-ingest-pipeline) | Applied at ingestion time.All agents and fields are supported. Data leaves the instrumented service.There are no performance overhead implications on the instrumented service. | | [{{apm-agent}} filters](custom-filters.md#apm-filters-in-agent) | Not supported by all agents.Data is sanitized before leaving the instrumented service.Potential overhead implications on the instrumented service | - ## Sensitive fields [apm-sensitive-fields] You should review the following fields regularly to ensure sensitive data is not being captured: diff --git a/solutions/observability/apps/application-performance-monitoring-apm.md b/solutions/observability/apps/application-performance-monitoring-apm.md index 9ee3d3c727..88c6ffe8ba 100644 --- a/solutions/observability/apps/application-performance-monitoring-apm.md +++ b/solutions/observability/apps/application-performance-monitoring-apm.md @@ -20,7 +20,6 @@ Elastic APM also automatically collects unhandled errors and exceptions. Errors Metrics are another vital source of information when debugging production systems. Elastic APM agents automatically pick up basic host-level metrics and agent-specific metrics, like JVM metrics in the Java Agent, and Go runtime metrics in the Go Agent. - ## Give Elastic APM a try [give_elastic_apm_a_try] -Use [Get started with application traces and APM](../../../solutions/observability/apps/fleet-managed-apm-server.md) to quickly spin up an APM deployment. Want to host everything yourself instead? See [Get started](../../../solutions/observability/apps/get-started-with-apm.md). \ No newline at end of file +Use [Get started with application traces and APM](/solutions/observability/apps/fleet-managed-apm-server.md) to quickly spin up an APM deployment. Want to host everything yourself instead? See [Get started](/solutions/observability/apps/get-started-with-apm.md). \ No newline at end of file diff --git a/solutions/observability/apps/applications-ui-annotation-user.md b/solutions/observability/apps/applications-ui-annotation-user.md index ffa30c71b2..77bbdb754d 100644 --- a/solutions/observability/apps/applications-ui-annotation-user.md +++ b/solutions/observability/apps/applications-ui-annotation-user.md @@ -8,13 +8,10 @@ applies_to: # Applications UI annotation user [apm-app-annotation-user-create] - ::::{note} By default, the `viewer` and `editor` built-in roles provide read access to Observability annotations. You only need to create an annotation user to write to the annotations index ([`xpack.observability.annotations.index`](kibana://reference/configuration-reference/apm-settings.md)). :::: - - ## Annotation user [apm-app-annotation-user] View deployment annotations in the Applications UI. @@ -30,7 +27,6 @@ View deployment annotations in the Applications UI. 2. Assign the `annotation_user` created previously, and the roles and privileges necessary to create a [full](apm-reader-user.md#apm-app-reader-full) or [partial](apm-reader-user.md#apm-app-reader-partial) APM reader to any users that need to view annotations in the Applications UI - ## Annotation API [apm-app-annotation-api] See [Create an API user](applications-ui-api-user.md). diff --git a/solutions/observability/apps/applications-ui-api-user.md b/solutions/observability/apps/applications-ui-api-user.md index f82265b2df..37e20585e3 100644 --- a/solutions/observability/apps/applications-ui-api-user.md +++ b/solutions/observability/apps/applications-ui-api-user.md @@ -8,8 +8,6 @@ applies_to: # Applications UI API user [apm-app-api-user] - - ## Central configuration API [apm-app-api-config-manager] Users can list, search, create, update, and delete central configurations via the Applications UI API. @@ -20,8 +18,6 @@ Users can list, search, create, update, and delete central configurations via th | --- | --- | --- | | Kibana | `all` on the APM and User Experience feature | Allow all access to the Applications and User Experience UIs | - - ## Central configuration API reader [apm-app-api-config-reader] Sometimes a user only needs to list and search central configurations via the Applications UI API. @@ -32,8 +28,6 @@ Sometimes a user only needs to list and search central configurations via the Ap | --- | --- | --- | | Kibana | `read` on the APM and User Experience feature | Allow read access to the Applications and User Experience UIs | - - ## Annotation API [apm-app-api-annotation-manager] Users can use the annotation API to create annotations on their APM data. @@ -53,4 +47,3 @@ Users can use the annotation API to create annotations on their APM data. | --- | --- | --- | | Kibana | `all` on the APM and User Experience feature | Allow all access to the Applications and User Experience UIs | - diff --git a/solutions/observability/apps/applications-ui-central-config-user.md b/solutions/observability/apps/applications-ui-central-config-user.md index 3b6c8884bd..5307089163 100644 --- a/solutions/observability/apps/applications-ui-central-config-user.md +++ b/solutions/observability/apps/applications-ui-central-config-user.md @@ -8,8 +8,6 @@ applies_to: # Applications UI central config user [apm-app-central-config-user] - - ## Central configuration manager [apm-app-central-config-manager] Central configuration users need to be able to view, create, update, and delete APM agent configurations. @@ -57,8 +55,6 @@ Central configuration users need to be able to view, create, update, and delete | --- | --- | --- | | Kibana | `All` on the APM and User Experience feature | Allow full use of the Applications and User Experience UIs | - - ## Central configuration reader [apm-app-central-config-reader] In some instances, you may wish to create a user that can only read central configurations, but not create, update, or delete them. @@ -106,8 +102,6 @@ In some instances, you may wish to create a user that can only read central conf | --- | --- | --- | | Kibana | `read` on the APM and User Experience feature | Allow read access to the Applications and User Experience UIs | - - ## Central configuration API [apm-app-central-config-api] See [Create an API user](applications-ui-api-user.md). diff --git a/solutions/observability/apps/applications-ui-settings.md b/solutions/observability/apps/applications-ui-settings.md index c0e73853b9..40daaf4474 100644 --- a/solutions/observability/apps/applications-ui-settings.md +++ b/solutions/observability/apps/applications-ui-settings.md @@ -10,16 +10,14 @@ applies_to: # Applications UI settings [observability-apm-kibana-settings] - ::::{note} -The **Editor** role or higher is required to modify settings. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). +The **Editor** role or higher is required to modify settings. To learn more, refer to [Assign user roles and privileges](/deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). :::: You can adjust Application settings to fine-tune your experience in the Applications UI. - ## General settings [observability-apm-kibana-settings-general-settings] ```{applies_to} stack: ga 9.1 @@ -64,8 +62,7 @@ To change APM settings, select **Settings** from any **Applications** page. The The Applications UI uses data views to query APM indices. In non-serverless versions, change the default APM indices that the Applications UI queries by opening the Applications UI and select **Settings** → **Indices**. Index settings in the Applications UI take precedence over those set in `kibana.yml`. -APM indices are {{kib}} Spaces-aware; Changes to APM index settings will only apply to the currently enabled space. See [Control access to APM data](../../../solutions/observability/apps/control-access-to-apm-data.md) for more information. - +APM indices are {{kib}} Spaces-aware; Changes to APM index settings will only apply to the currently enabled space. See [Control access to APM data](/solutions/observability/apps/control-access-to-apm-data.md) for more information. ## APM Labs [observability-apm-kibana-settings-apm-labs] diff --git a/solutions/observability/apps/applications-ui-storage-explorer-user.md b/solutions/observability/apps/applications-ui-storage-explorer-user.md index 86c9fece4f..7cb3cffc99 100644 --- a/solutions/observability/apps/applications-ui-storage-explorer-user.md +++ b/solutions/observability/apps/applications-ui-storage-explorer-user.md @@ -8,8 +8,6 @@ applies_to: # Applications UI storage explorer user [apm-app-storage-explorer-user-create] - - ## Storage Explorer user [apm-app-storage-explorer-user] View the **Storage Explorer** in the Applications UI. diff --git a/solutions/observability/apps/built-in-data-filters.md b/solutions/observability/apps/built-in-data-filters.md index 1e5d74dd7c..37c2a6d4b0 100644 --- a/solutions/observability/apps/built-in-data-filters.md +++ b/solutions/observability/apps/built-in-data-filters.md @@ -3,10 +3,14 @@ mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-filtering.html applies_to: stack: + serverless: --- # Built-in data filters [apm-filtering] +:::{include} _snippets/apm-server-vs-mis.md +::: + Built-in data filters allow you to filter or turn off ingestion of the following types of data: | Data type | Common sensitive data | @@ -17,7 +21,6 @@ Built-in data filters allow you to filter or turn off ingestion of the following | [Real user monitoring data](#apm-filters-real-user-data) | URLs visited, click events, user browser errors, resources used, etc. | | [Database statements](#apm-filters-database-statements) | Sensitive user or business information | - ## HTTP headers [apm-filters-http-header] By default, APM agents capture HTTP request and response headers (including cookies). Most Elastic APM agents provide the ability to sanitize HTTP header fields, including cookies and `application/x-www-form-urlencoded` data (POST form fields). Query string and captured request bodies, like `application/json` data, are not sanitized. @@ -42,7 +45,6 @@ Alternatively, you can completely disable the capturing of HTTP headers. This se * Python: [`capture_headers`](apm-agent-python://reference/configuration.md#config-capture-headers) * Ruby: [`capture_headers`](apm-agent-ruby://reference/configuration.md#config-capture-headers) - ## HTTP bodies [apm-filters-http-body] By default, the body of HTTP requests is not recorded. Request bodies often contain sensitive data like passwords or credit card numbers, so use care when enabling this feature. @@ -56,10 +58,9 @@ This setting supports [Central configuration](apm-agent-central-configuration.md * Python: [`capture_body`](apm-agent-python://reference/configuration.md#config-capture-body) * Ruby: [`capture_body`](apm-agent-ruby://reference/configuration.md#config-capture-body) - ## Personal data [apm-filters-personal-data] -By default, the APM Server captures some personal data associated with trace events: +By default, {{apm-server-or-mis}} captures some personal data associated with trace events: * `client.ip`: The client’s IP address. Typically derived from the HTTP headers of incoming requests. `client.ip` is also used in conjunction with the [`geoip` processor](elasticsearch://reference/enrich-processor/geoip-processor.md) to assign geographical information to trace events. To learn more about how `client.ip` is derived, see [Deriving an incoming request’s `client.ip` address](anonymous-authentication.md#apm-derive-client-ip). * `user_agent`: User agent data, including the client operating system, device name, vendor, and version. @@ -67,12 +68,15 @@ By default, the APM Server captures some personal data associated with trace eve The capturing of this data can be turned off by setting **Capture personal data** to `false`. :::{note} -This setting only prevents APM Server from capturing already ingested personal data. It does not prevent such data from appearing in ingestion logs where applicable. See [{{apm-agent}} filters](custom-filters.md#apm-filters-in-agent) for redacting data on ingestion. +This setting only prevents {{apm-server-or-mis}} from capturing already ingested personal data. It does not prevent such data from appearing in ingestion logs where applicable. See [{{apm-agent}} filters](custom-filters.md#apm-filters-in-agent) for redacting data on ingestion. ::: - ## Real user monitoring data [apm-filters-real-user-data] +```{applies_to} +serverless: unavailable +``` + Protecting user data is important. For that reason, individual RUM instrumentations can be disabled in the RUM agent with the [`disableInstrumentations`](apm-agent-rum-js://reference/configuration.md#disable-instrumentations) configuration variable. Disabled instrumentations produce no spans or transactions. | Disable | Configuration value | @@ -83,14 +87,12 @@ Protecting user data is important. For that reason, individual RUM instrumentati | User click events including URLs visited, mouse clicks, and navigation events | `eventtarget` | | Single page application route changes | `history` | - ## Database statements [apm-filters-database-statements] For SQL databases, APM agents do not capture the parameters of prepared statements. Note that Elastic APM currently does not make an effort to strip parameters of regular statements. Not using prepared statements makes your code vulnerable to SQL injection attacks, so be sure to use prepared statements. For non-SQL data stores, such as {{es}} or MongoDB, Elastic APM captures the full statement for queries. For inserts or updates, the full document is not stored. To filter or obfuscate data in non-SQL database statements, or to remove the statement entirely, you can set up an ingest node pipeline. - ## Agent-specific options [apm-filters-agent-specific] Certain agents offer additional filtering and obfuscating options: diff --git a/solutions/observability/apps/collect-application-data.md b/solutions/observability/apps/collect-application-data.md index f5f211fc53..d13d29b98d 100644 --- a/solutions/observability/apps/collect-application-data.md +++ b/solutions/observability/apps/collect-application-data.md @@ -7,43 +7,38 @@ applies_to: serverless: --- - # Collect application data [apm-collect-application-data] % is required role serverless only? ::::{note} -**For Observability Serverless projects**, the **Admin** role or higher is required to send APM data to Elastic. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). +**For Observability Serverless projects**, the **Admin** role or higher is required to send APM data to Elastic. To learn more, refer to [Assign user roles and privileges](/deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). :::: ::::{note} - -![documentation icon](/solutions/images/serverless-documentation.svg "") Want to get started quickly? See [Get started with traces and APM](../../../solutions/observability/apps/get-started-with-apm.md). - +Want to get started quickly? See [Get started with traces and APM](/solutions/observability/apps/get-started-with-apm.md). :::: ## Language-specific options [_language_specific_options] Use Elastic APM agents or an OpenTelemetry language SDK to instrument a service in the language its written in: -* [**Elastic APM agents**](../../../solutions/observability/apps/elastic-apm-agents.md): Elastic APM agents are instrumentation libraries written in the same language as your service. -* [**OpenTelemetry**](../../../solutions/observability/apps/use-opentelemetry-with-apm.md): OpenTelemetry is an open source set of APIs, SDKs, tooling, and integrations that enable the capture and management of telemetry data from your services and applications. +* [**Elastic APM agents**](/solutions/observability/apps/elastic-apm-agents.md): Elastic APM agents are instrumentation libraries written in the same language as your service. +* [**OpenTelemetry**](/solutions/observability/apps/use-opentelemetry-with-apm.md): OpenTelemetry is an open source set of APIs, SDKs, tooling, and integrations that enable the capture and management of telemetry data from your services and applications. * This option includes Elastic Distributions of OpenTelemetry, which are customized versions of [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/) that are optimized to work with an Elastic backend. - **Not sure which method is right for you?** Compare the available options below. - ### Capabilities [_capabilities] | | Elastic APM agent | Elastic Distribution of OpenTelemetry | | --- | --- | --- | -| **Support level** | Fully supported | Mixed support
*Refer to the* [*availability table*](../../../solutions/observability/apps/collect-application-data.md#apm-collect-data-availability) | +| **Support level** | Fully supported | Mixed support
*Refer to the* [*availability table*](/solutions/observability/apps/collect-application-data.md#apm-collect-data-availability) | | **Data protocol** | Elastic protocol | [OpenTelemetry protocol (OTLP)](https://opentelemetry.io/docs/specs/otel/protocol/) | -| **Central configuration** | Supported
*Refer to* [*APM agent central configuration*](../../../solutions/observability/apps/apm-agent-central-configuration.md) | Not supported | +| **Central configuration** | Supported
*Refer to* [*APM agent central configuration*](/solutions/observability/apps/apm-agent-central-configuration.md) | Not supported | % Stateful only after this comment? @@ -61,11 +56,10 @@ Use Elastic APM agents or an OpenTelemetry language SDK to instrument a service | **Python** | Python agent | [preview] EDOT Python | | **Ruby** | Ruby agent | ![Not available](/solutions/images/observability-cross.svg "") | - ## Service-specific options [_service_specific_options] Elastic also offers several tools to help you collect data from specific services: * **Kubernetes**: The Elastic APM attacher for Kubernetes simplifies the instrumentation and configuration of your application pods. Read more in the [APM attacher for Kubernetes docs](apm-k8s-attacher://reference/index.md). * **AWS Lambda Functions**: Helps you monitor your AWS Lambda functions. Read more in the [APM Architecture for AWS Lambda docs](apm-aws-lambda://reference/index.md). -* [8.15.0] **Jaeger**: Helps you to switch an existing Jaeger setup from the default Jaeger backend to the {{stack}}. Read more in [Integrate with Jaeger](../../../solutions/observability/apps/integrate-with-jaeger-deprecated.md). \ No newline at end of file +* **Jaeger (deprecated)**: Helps you to switch an existing Jaeger setup from the default Jaeger backend to the {{stack}}. Read more in [Integrate with Jaeger](/solutions/observability/apps/integrate-with-jaeger-deprecated.md). \ No newline at end of file diff --git a/solutions/observability/apps/collect-metrics.md b/solutions/observability/apps/collect-metrics.md index 4063bf9202..c17884d55c 100644 --- a/solutions/observability/apps/collect-metrics.md +++ b/solutions/observability/apps/collect-metrics.md @@ -13,7 +13,6 @@ applies_to: When collecting metrics, please note that the [`DoubleValueRecorder`](https://www.javadoc.io/doc/io.opentelemetry/opentelemetry-api/latest/io/opentelemetry/api/metrics/DoubleValueRecorder.md) and [`LongValueRecorder`](https://www.javadoc.io/doc/io.opentelemetry/opentelemetry-api/latest/io/opentelemetry/api/metrics/LongValueObserver.md) metrics are not yet supported. :::: - Here’s an example of how to capture business metrics from a Java application. ```java @@ -32,18 +31,16 @@ public void createOrder(HttpServletRequest request) { See the [Open Telemetry Metrics API](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/api.md) for more information. - ## Verify OpenTelemetry metrics data [apm-open-telemetry-verify-metrics] Use **Discover** to validate that metrics are successfully reported to {{kib}}. 1. Open your Observability instance. -2. Find **Discover** in the main menu or use the [global search field](../../../get-started/the-stack.md#kibana-navigation-search). +2. Find **Discover** in the main menu or use the [global search field](/get-started/the-stack.md#kibana-navigation-search). 3. Click **All logs** → **Data Views** then select **APM**. 4. Filter the data to only show documents with metrics: `processor.name :"metric"` 5. Narrow your search with a known OpenTelemetry field. For example, if you have an `order_value` field, add `order_value: *` to your search to return only OpenTelemetry metrics documents. - ## Visualize your metrics[apm-open-telemetry-visualize] Use **Lens** to create visualizations for OpenTelemetry metrics. Lens enables you to build visualizations by dragging and dropping data fields. It makes smart visualization suggestions for your data, allowing you to switch between visualization types. @@ -54,4 +51,4 @@ To get started with a new Lens visualization: 2. Click **Create new visualization**. 3. Select **Lens**. -For more information on using Lens, refer to the [Lens documentation](../../../explore-analyze/visualize/lens.md). \ No newline at end of file +For more information on using Lens, refer to the [Lens documentation](/explore-analyze/visualize/lens.md). \ No newline at end of file diff --git a/solutions/observability/apps/configure-anonymous-authentication.md b/solutions/observability/apps/configure-anonymous-authentication.md index 6a21ccbcaa..8ce55bb5e4 100644 --- a/solutions/observability/apps/configure-anonymous-authentication.md +++ b/solutions/observability/apps/configure-anonymous-authentication.md @@ -8,7 +8,6 @@ applies_to: # Configure anonymous authentication [apm-configuration-anonymous] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-yes.svg "") @@ -16,7 +15,6 @@ Most options on this page are supported by all APM Server deployment methods. :::: - Elastic APM agents can send unauthenticated (anonymous) events to the APM Server. An event is considered to be anonymous if no authentication token can be extracted from the incoming request. This is useful for agents that run on clients, like the Real User Monitoring (RUM) agent running in a browser, or the Android or iOS/Swift agent running in a user application. Enable anonymous authentication in the APM Server to allow the ingestion of unauthenticated client-side APM data while still requiring authentication for server-side services. @@ -49,15 +47,12 @@ Configure and customize Fleet-managed APM settings directly in {{kib}}: All anonymous access configuration is ignored if [authenticated communication](secure-communication-with-apm-agents.md) is disabled. :::: - - ## Real User Monitoring (RUM) [apm-config-auth-anon-rum] If an [API key](api-keys.md) or [secret token](secret-token.md) is configured, then anonymous authentication must be enabled to collect RUM data. For this reason, anonymous auth will be enabled automatically if [Enable RUM](configure-real-user-monitoring-rum.md#apm-rum-enable) is set to `true`, and [Anonymous Agent access](#apm-config-auth-anon-enabled) is not explicitly defined. See [Real User Monitoring (RUM)](configure-real-user-monitoring-rum.md) for additional RUM configuration options. - ### Mitigating malicious requests [apm-config-auth-anon-mitigating] There are a few configuration variables that can mitigate the impact of malicious requests to an unauthenticated APM Server endpoint. @@ -66,7 +61,6 @@ Use the [Allowed anonymous agents](#apm-config-auth-anon-allow-agent) and [Allow Additionally, the APM Server can rate-limit unauthenticated requests based on the client IP address (`client.ip`) of the request with [Event limit](#apm-config-auth-anon-event-limit). This allows you to specify the maximum number of requests allowed per unique IP address, per second. - ### Deriving an incoming request’s `client.ip` address [apm-config-auth-anon-client-ip] The remote IP address of an incoming request might be different from the end-user’s actual IP address, for example, because of a proxy. For this reason, the APM Server attempts to derive the IP address of an incoming request from HTTP headers. The supported headers are parsed in the following order: @@ -77,17 +71,14 @@ The remote IP address of an incoming request might be different from the end-use If none of these headers are present, the remote address for the incoming request is used. - ### Using a reverse proxy or load balancer [apm-config-auth-anon-client-ip-concerns] HTTP headers are easily modified; it’s possible for anyone to spoof the derived `client.ip` value by changing or setting, for example, the value of the `X-Forwarded-For` header. For this reason, if any of your clients are not trusted, we recommend setting up a reverse proxy or load balancer in front of the APM Server. Using a proxy allows you to clear any existing IP-forwarding HTTP headers, and replace them with one set by the proxy. This prevents malicious users from cycling spoofed IP addresses to bypass the APM Server’s rate limiting feature. - ## Configuration reference [apm-config-auth-anon] - ### Anonymous Agent access [apm-config-auth-anon-enabled] Enable or disable anonymous authentication. Default: `false` (disabled). (bool) @@ -97,7 +88,6 @@ Enable or disable anonymous authentication. Default: `false` (disabled). (bool) | APM Server binary | `apm-server.auth.anonymous.enabled` | | Fleet-managed | `Anonymous Agent access` | - ### Allowed anonymous agents [apm-config-auth-anon-allow-agent] A list of permitted {{apm-agent}} names for anonymous authentication. Names in this list must match the agent’s `agent.name`. Default: `[rum-js, js-base]` (only RUM agent events are accepted). (array) @@ -107,7 +97,6 @@ A list of permitted {{apm-agent}} names for anonymous authentication. Names in t | APM Server binary | `apm-server.auth.anonymous.allow_agent` | | Fleet-managed | `Allowed Anonymous agents` | - ## Allowed services [apm-config-auth-anon-allow-service] A list of permitted service names for anonymous authentication. Names in this list must match the agent’s `service.name`. This can be used to limit the number of service-specific indices or data streams created. Default: Not set (any service name is accepted). (array) @@ -117,7 +106,6 @@ A list of permitted service names for anonymous authentication. Names in this li | APM Server binary | `apm-server.auth.anonymous.allow_service` | | Fleet-managed | `Allowed Anonymous services` | - ### IP limit [apm-config-auth-anon-ip-limit] The number of unique IP addresses to track in an LRU cache. IP addresses in the cache will be rate limited according to the [Event limit](#apm-config-auth-anon-event-limit) setting. Consider increasing this default if your application has many concurrent clients. Default: `1000`. (int) @@ -127,7 +115,6 @@ The number of unique IP addresses to track in an LRU cache. IP addresses in the | APM Server binary | `apm-server.auth.anonymous.rate_limit.ip_limit` | | Fleet-managed | `Anonymous Rate limit (IP limit)` | - ### Event limit [apm-config-auth-anon-event-limit] The maximum number of events allowed per second, per agent IP address. Default: `300`. (int) diff --git a/solutions/observability/apps/configure-apm-agent-central-configuration.md b/solutions/observability/apps/configure-apm-agent-central-configuration.md index 415e3e6b55..4134faede7 100644 --- a/solutions/observability/apps/configure-apm-agent-central-configuration.md +++ b/solutions/observability/apps/configure-apm-agent-central-configuration.md @@ -8,7 +8,6 @@ applies_to: # Configure APM agent central configuration [apm-configure-agent-config] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-yes.svg "") @@ -16,12 +15,10 @@ APM agent central configuration is supported by all APM Server deployment method :::: - APM agent central configuration allows you to fine-tune your APM agents from within the Applications UI. Changes are automatically propagated to your APM agents, so there’s no need to redeploy your applications. To learn more about this feature, see [APM agent central configuration](apm-agent-central-configuration.md). - ## APM agent configuration options [_apm_agent_configuration_options] **The following options are only supported for APM Server binary users**. @@ -35,23 +32,18 @@ apm-server.agent.config.elasticsearch.api_key: TiNAGG4BaaMdaH1tRfuU:KnR6yE41RrSo 1. You *must* set the API key to be configured to **Beats**. Base64 encoded API keys are not currently supported in this configuration. For details on how to create and configure a compatible API key, refer to [Create an API key for writing events](grant-access-using-api-keys.md#apm-beats-api-key-publish). - - ### `apm-server.agent.config.cache.expiration` [apm-agent-config-cache] When using APM agent central configuration, information fetched from {{es}} will be cached in memory for some time. Specify the cache expiration time via this setting. Defaults to `30s` (30 seconds). - ### `apm-server.agent.config.elasticsearch` [apm-agent-config-elasticsearch] Takes the same options as [output.elasticsearch](configure-elasticsearch-output.md). For APM Server binary users and Elastic Agent standalone-managed APM Server, APM agent central configuration is automatically fetched from {{es}} using the `output.elasticsearch` configuration. If `output.elasticsearch` isn’t set or doesn’t have sufficient privileges, use these {{es}} options to provide {{es}} access. - ### Common problems [_common_problems] - #### HTTP 403 errors [_http_403_errors] You may see either of the following HTTP 403 errors from APM Server when it attempts to fetch the APM agent central configuration: @@ -72,7 +64,6 @@ This occurs because the user or API key set in either `apm-server.agent.config.e To fix this error, ensure that APM Server has all the required privileges. For more details, refer to [Create a *central configuration management* role](create-assign-feature-roles-to-apm-server-users.md#apm-privileges-agent-central-config-server). - #### HTTP 401 errors [_http_401_errors] If you get an HTTP 401 error from APM Server, make sure that you’re using an API key that is configured to **Beats**. For details on how to create and configure a compatible API key, refer to [Create an API key for writing events](grant-access-using-api-keys.md#apm-beats-api-key-publish). diff --git a/solutions/observability/apps/configure-apm-instrumentation.md b/solutions/observability/apps/configure-apm-instrumentation.md index ff5b886fa2..3fd4e36b82 100644 --- a/solutions/observability/apps/configure-apm-instrumentation.md +++ b/solutions/observability/apps/configure-apm-instrumentation.md @@ -8,7 +8,6 @@ applies_to: # Configure APM instrumentation [apm-configuration-instrumentation] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,7 +15,6 @@ Instrumentation of APM Server is not yet supported for Fleet-managed APM. :::: - APM Server uses the Elastic APM Go Agent to instrument its publishing pipeline. To gain insight into the performance of APM Server, you can enable this instrumentation and send trace data to APM Server. Currently, only the {{es}} output is instrumented. Example configuration with instrumentation enabled: @@ -30,32 +28,26 @@ instrumentation: api_key: L5ER6FEvjkmlfalBealQ3f3fLqf03fazfOV ``` - ## Configuration options [_configuration_options] You can specify the following options in the `instrumentation` section of the `apm-server.yml` config file: - ### `enabled` [_enabled] Set to `true` to enable instrumentation of APM Server. Defaults to `false`. - ### `environment` [_environment] Set the environment in which APM Server is running, for example, `staging`, `production`, `dev`, etc. Environments can be filtered in the [{{kib}} Applications UI](overviews.md). - ### `hosts` [_hosts] The [APM Server](get-started-with-apm.md) hosts to report instrumentation data to. Defaults to `http://localhost:8200`. - ### `api_key` [_api_key] [API key](api-keys.md) used to secure communication with the APM Server(s). If `api_key` is set then `secret_token` will be ignored. - ### `secret_token` [_secret_token_2] [Secret token](secret-token.md) used to secure communication with the APM Server(s). diff --git a/solutions/observability/apps/configure-apm-server.md b/solutions/observability/apps/configure-apm-server.md index de6207680a..2a4be636d1 100644 --- a/solutions/observability/apps/configure-apm-server.md +++ b/solutions/observability/apps/configure-apm-server.md @@ -10,25 +10,25 @@ applies_to: How you configure the APM Server depends on your deployment method. -* **APM Server binary** users need to edit the `apm-server.yml` configuration file. The location of the file varies by platform. To locate the file, see [Installation layout](../../../solutions/observability/apps/installation-layout.md). +* **APM Server binary** users need to edit the `apm-server.yml` configuration file. The location of the file varies by platform. To locate the file, see [Installation layout](/solutions/observability/apps/installation-layout.md). * **Fleet-managed** users configure the APM Server directly in {{kib}}. Each configuration page describes the specific location. -* **Elastic cloud** users should see [Add APM user settings](../../../solutions/observability/apps/configure-apm-server.md) for information on how to configure Elastic APM. +* **Elastic cloud** users should see [Add APM user settings](/solutions/observability/apps/configure-apm-server.md) for information on how to configure Elastic APM. The following topics describe how to configure APM Server: -* [General configuration options](../../../solutions/observability/apps/general-configuration-options.md) -* [Anonymous authentication](../../../solutions/observability/apps/configure-anonymous-authentication.md) -* [APM agent authorization](../../../solutions/observability/apps/apm-agent-authorization.md) -* [APM agent central configuration](../../../solutions/observability/apps/configure-apm-agent-central-configuration.md) -* [Instrumentation](../../../solutions/observability/apps/configure-apm-instrumentation.md) -* [{{kib}} endpoint](../../../solutions/observability/apps/configure-kibana-endpoint.md) -* [Logging](../../../solutions/observability/apps/configure-logging.md) -* [Output](../../../solutions/observability/apps/configure-output.md) -* [Project paths](../../../solutions/observability/apps/configure-project-paths.md) -* [Real User Monitoring (RUM)](../../../solutions/observability/apps/configure-real-user-monitoring-rum.md) -* [SSL/TLS settings](../../../solutions/observability/apps/ssltls-settings.md) -* [Tail-based sampling](../../../solutions/observability/apps/tail-based-sampling.md) -* [Use environment variables in the configuration](../../../solutions/observability/apps/use-environment-variables-in-configuration.md) +* [General configuration options](/solutions/observability/apps/general-configuration-options.md) +* [Anonymous authentication](/solutions/observability/apps/configure-anonymous-authentication.md) +* [APM agent authorization](/solutions/observability/apps/apm-agent-authorization.md) +* [APM agent central configuration](/solutions/observability/apps/configure-apm-agent-central-configuration.md) +* [Instrumentation](/solutions/observability/apps/configure-apm-instrumentation.md) +* [{{kib}} endpoint](/solutions/observability/apps/configure-kibana-endpoint.md) +* [Logging](/solutions/observability/apps/configure-logging.md) +* [Output](/solutions/observability/apps/configure-output.md) +* [Project paths](/solutions/observability/apps/configure-project-paths.md) +* [Real User Monitoring (RUM)](/solutions/observability/apps/configure-real-user-monitoring-rum.md) +* [SSL/TLS settings](/solutions/observability/apps/ssltls-settings.md) +* [Tail-based sampling](/solutions/observability/apps/tail-based-sampling.md) +* [Use environment variables in the configuration](/solutions/observability/apps/use-environment-variables-in-configuration.md) ## Edit APM user settings [ec-manage-apm-settings] @@ -39,12 +39,10 @@ Change how Elastic APM runs by providing your own user settings. Starting in {{s Check [APM configuration reference](/solutions/observability/apps/configure-apm-server.md) for information on how to configure Elastic APM in this mode. - Standalone APM Server (legacy) : Deployments created prior to {{stack}} version 8.0 are in legacy mode. Upgrading to or past {{stack}} 8.0 will not remove you from legacy mode. - Check [Edit standalone APM settings (legacy)](../../../solutions/observability/apps/configure-apm-server.md#ec-edit-apm-standalone-settings) and [Supported standalone APM settings (legacy)](../../../solutions/observability/apps/configure-apm-server.md#ec-apm-settings) for information on how to configure Elastic APM in this mode. - + Check [Edit standalone APM settings (legacy)](/solutions/observability/apps/configure-apm-server.md#ec-edit-apm-standalone-settings) and [Supported standalone APM settings (legacy)](/solutions/observability/apps/configure-apm-server.md#ec-apm-settings) for information on how to configure Elastic APM in this mode. To learn more about the differences between these modes, or to switch from Standalone APM Server (legacy) mode to {{fleet}}-managed, check [Switch to the Elastic APM integration](/solutions/observability/apps/switch-to-elastic-apm-integration.md). @@ -68,8 +66,6 @@ To add user settings: If a setting is not supported on {{ecloud}}, you will get an error message when you try to save. :::: - - ## Supported standalone APM settings (legacy) [ec-apm-settings] {{ech}} supports the following setting when running APM in standalone mode (legacy). @@ -78,7 +74,6 @@ If a setting is not supported on {{ecloud}}, you will get an error message when Some settings that could break your cluster if set incorrectly are blocklisted. The following settings are generally safe in cloud environments. For detailed information about APM settings, check the [APM documentation](/solutions/observability/apps/configure-apm-server.md). :::: - ### Version 8.0+ [ec_version_8_0_3] This stack version removes support for some previously supported settings. These are all of the supported settings for this version: @@ -169,7 +164,6 @@ The following `apm-server.auth.anonymous.*` settings can be configured to restri `output.elasticsearch.flush_bytes` : The bulk request size threshold, in bytes, before flushing to Elasticsearch. The value must have a suffix. The default is 5MB. - ### Version 7.17+ [ec_version_7_17] This stack version includes all of the settings from 7.16 and the following: @@ -194,7 +188,6 @@ Allow anonymous access only for specified agents and/or services. This is primar `apm-server.auth.api_key.limit` : Restrict how many unique API keys are allowed per minute. Should be set to at least the amount of different API keys configured in your monitored services. Every unique API key triggers one request to Elasticsearch. This will be replacing the config option `apm-server.api_key.limit` from `8.0` on. - ### Supported versions before 8.x [ec_supported_versions_before_8_x_3] `apm-server.aggregation.transactions.*` @@ -340,7 +333,6 @@ Allow anonymous access only for specified agents and/or services. This is primar | 4GB | 762 | | 8GB | 1067 | - `output.elasticsearch.indices` : Array of index selector rules supporting conditionals and formatted string. @@ -358,7 +350,6 @@ Allow anonymous access only for specified agents and/or services. This is primar | 4GB | 14 | | 8GB | 20 | - `queue.mem.events` : Maximum number of events to concurrently store in the internal queue. By default, this number changes based on the size of the instance: @@ -370,14 +361,12 @@ Allow anonymous access only for specified agents and/or services. This is primar | 4GB | 16000 | | 8GB | 32000 | - `queue.mem.flush.min_events` : Minimum number of events to have before pushing them to Elasticsearch. By default, this number changes based on the size of the instance. `queue.mem.flush.timeout` : Maximum duration before sending the events to the output if the `min_events` is not crossed. - ### Logging settings [ec_logging_settings] `logging.level` @@ -393,5 +382,5 @@ Allow anonymous access only for specified agents and/or services. This is primar : The period after which to log the internal metrics. Defaults to *30s*. ::::{note} -To change logging settings you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). +To change logging settings you must first [enable deployment logging](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). :::: \ No newline at end of file diff --git a/solutions/observability/apps/configure-console-output.md b/solutions/observability/apps/configure-console-output.md index 01d31d0a2c..03560ee239 100644 --- a/solutions/observability/apps/configure-console-output.md +++ b/solutions/observability/apps/configure-console-output.md @@ -8,7 +8,6 @@ applies_to: # Configure the Console output [apm-console-output] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,14 +15,12 @@ The Console output is not yet supported by {{fleet}}-managed APM Server. :::: - The Console output writes events in JSON format to stdout. ::::{warning} The Console output should be used only for debugging issues as it can produce a large amount of logging data. :::: - To use this output, edit the APM Server configuration file to disable the {{es}} output by commenting it out, and enable the console output by adding `output.console`. Example configuration: @@ -33,31 +30,26 @@ output.console: pretty: true ``` - ## Configuration options [_configuration_options_7] You can specify the following `output.console` options in the `apm-server.yml` config file: - ### `enabled` [_enabled_6] The enabled config is a boolean setting to enable or disable the output. If set to false, the output is disabled. The default value is `true`. - ### `pretty` [_pretty] If `pretty` is set to true, events written to stdout will be nicely formatted. The default is false. - ### `codec` [_codec_3] Output codec configuration. If the `codec` section is missing, events will be JSON encoded using the `pretty` option. See [Change the output codec](#apm-configuration-output-codec) for more information. - ### `bulk_max_size` [_bulk_max_size_4] The maximum number of events to buffer internally during publishing. The default is 2048. @@ -66,7 +58,6 @@ Specifying a larger batch size may add some latency and buffering during publish Setting `bulk_max_size` to values less than or equal to 0 disables the splitting of batches. When splitting is disabled, the queue decides on the number of events to be contained in a batch. - ## Change the output codec [apm-configuration-output-codec] For outputs that do not require a specific encoding, you can change the encoding by using the codec configuration. You can specify either the `json` or `format` codec. By default the `json` codec is used. diff --git a/solutions/observability/apps/configure-elasticsearch-output.md b/solutions/observability/apps/configure-elasticsearch-output.md index bedc17e6ea..7f97cf7164 100644 --- a/solutions/observability/apps/configure-elasticsearch-output.md +++ b/solutions/observability/apps/configure-elasticsearch-output.md @@ -6,11 +6,8 @@ applies_to: stack: --- - - # Configure the Elasticsearch output [apm-elasticsearch-output] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -18,7 +15,6 @@ This documentation only applies to APM Server binary users. Fleet-managed users :::: - The {{es}} output sends events directly to {{es}} using the {{es}} HTTP API. Example configuration: @@ -30,7 +26,6 @@ output.elasticsearch: 1. To enable SSL, add `https` to all URLs defined under *hosts*. - When sending data to a secured cluster through the `elasticsearch` output, APM Server can use any of the following authentication methods: * Basic authentication credentials (username and password). @@ -56,7 +51,6 @@ output.elasticsearch: 1. You *must* set the API key to be configured to **Beats**. Base64 encoded API keys are not currently supported in this configuration. For details on how to create and configure a compatible API key, refer to [Create an API key for writing events](grant-access-using-api-keys.md#apm-beats-api-key-publish). - **PKI certificate authentication:** ```yaml @@ -68,24 +62,20 @@ output.elasticsearch: See [Secure communication with {{es}}](#apm-securing-communication-elasticsearch) for details on each authentication method. - ## Compatibility [_compatibility] This output works with all compatible versions of {{es}}. See the [Elastic Support Matrix](https://www.elastic.co/support/matrix#matrix_compatibility). - ## Configuration options [_configuration_options_3] You can specify the following options in the `elasticsearch` section of the `apm-server.yml` config file: - ### `enabled` [_enabled_2] The enabled config is a boolean setting to enable or disable the output. If set to `false`, the output is disabled. The default value is `true`. - ### `hosts` [apm-hosts-option] The list of {{es}} nodes to connect to. The events are distributed to these nodes in round robin order. If one node becomes unreachable, the event is automatically sent to another node. Each {{es}} node can be defined as a `URL` or `IP:PORT`. For example: `http://192.15.3.2`, `https://es.found.io:9230` or `192.24.3.2:9300`. If no port is specified, `9200` is used. @@ -94,7 +84,6 @@ The list of {{es}} nodes to connect to. The events are distributed to these node When a node is defined as an `IP:PORT`, the *scheme* and *path* are taken from the [`protocol`](#apm-protocol-option) and [`path`](#apm-path-option) config options. :::: - ```yaml output.elasticsearch: hosts: ["10.45.3.2:9220", "10.45.3.1:9230"] @@ -104,7 +93,6 @@ output.elasticsearch: In the previous example, the {{es}} nodes are available at `https://10.45.3.2:9220/elasticsearch` and `https://10.45.3.1:9230/elasticsearch`. - ### `compression_level` [_compression_level] The gzip compression level. Setting this value to `0` disables compression. The compression level must be in the range of `1` (best speed) to `9` (best compression). @@ -113,14 +101,12 @@ Increasing the compression level will reduce the network usage but will increase The default value is `0`. - ### `escape_html` [_escape_html] Configure escaping of HTML in strings. Set to `true` to enable escaping. The default value is `false`. - ### `api_key` [_api_key_2] Instead of using a username and password, you can use API keys to secure communication with {{es}}. The value must be the ID of the API key and the API key joined by a colon: `id:api_key`. @@ -131,34 +117,28 @@ You *must* set the API key to be configured to **Beats**. Base64 encoded API key :alt: API key dropdown highlighting the Beats option ::: - ### `username` [_username] The basic authentication username for connecting to {{es}}. This user needs the privileges required to publish events to {{es}}. To create a user like this, see [Create a *writer* role](create-assign-feature-roles-to-apm-server-users.md#apm-privileges-to-publish-events). - ### `password` [_password] The basic authentication password for connecting to {{es}}. - ### `parameters` [_parameters] Dictionary of HTTP parameters to pass within the URL with index operations. - ### `protocol` [apm-protocol-option] The name of the protocol {{es}} is reachable on. The options are: `http` or `https`. The default is `http`. However, if you specify a URL for [`hosts`](#apm-hosts-option), the value of `protocol` is overridden by whatever scheme you specify in the URL. - ### `path` [apm-path-option] An HTTP path prefix that is prepended to the HTTP API calls. This is useful for the cases where {{es}} listens behind an HTTP reverse proxy that exports the API under a custom prefix. - ### `headers` [_headers] Custom HTTP headers to add to each request created by the {{es}} output. Example: @@ -170,12 +150,10 @@ output.elasticsearch.headers: It is possible to specify multiple header values for the same header name by separating them with a comma. - ### `proxy_url` [_proxy_url] The URL of the proxy to use when connecting to the {{es}} servers. The value may be either a complete URL or a "host[:port]", in which case the "http" scheme is assumed. If a value is not specified through the configuration file then proxy environment variables are used. See the [Go documentation](https://golang.org/pkg/net/http/#ProxyFromEnvironment) for more information about the environment variables. - ### `max_retries` [_max_retries] The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped. @@ -184,39 +162,32 @@ Set `max_retries` to a value less than 0 to retry until all events are published The default is 3. - ### `flush_bytes` [_flush_bytes] The bulk request size threshold, in bytes, before flushing to {{es}}. The value must have a suffix, e.g. `"2MB"`. The default is `1MB`. - ### `flush_interval` [_flush_interval] The maximum duration to accumulate events for a bulk request before being flushed to {{es}}. The value must have a duration suffix, e.g. `"5s"`. The default is `1s`. - ### `backoff.init` [_backoff_init] The number of seconds to wait before trying to reconnect to {{es}} after a network error. After waiting `backoff.init` seconds, APM Server tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to `backoff.max`. After a successful connection, the backoff timer is reset. The default is `1s`. - ### `backoff.max` [_backoff_max] The maximum number of seconds to wait before attempting to connect to {{es}} after a network error. The default is `60s`. - ### `timeout` [_timeout] The HTTP request timeout in seconds for the {{es}} request. The default is 90. - ### `ssl` [_ssl] Configuration options for SSL parameters like the certificate authority to use for HTTPS-based connections. If the `ssl` section is missing, the host CAs are used for HTTPS connections to {{es}}. See the [secure communication with {{es}}](#apm-securing-communication-elasticsearch) guide or [SSL configuration reference](ssltls-output-settings.md) for more information. - ## Secure communication with {{es}} [apm-securing-communication-elasticsearch] When sending data to a secured cluster through the `elasticsearch` output, APM Server can use any of the following authentication methods: @@ -248,7 +219,6 @@ Authentication is specified in the APM Server configuration file: 1. This API key must have the privileges required to publish events to {{es}}. You *must* set the API key to be configured to **Beats**. Base64 encoded API keys are not currently supported in this configuration. For details on how to create and configure a compatible API key, refer to [Create an API key for writing events](grant-access-using-api-keys.md#apm-beats-api-key-publish). - * To use **Public Key Infrastructure (PKI) certificates** to authenticate users, specify the `certificate` and `key` settings under `output.elasticsearch`. For example: ```yaml @@ -261,8 +231,7 @@ Authentication is specified in the APM Server configuration file: 1. The path to the certificate for SSL client authentication 2. The client certificate key - - These settings assume that the distinguished name (DN) in the certificate is mapped to the appropriate roles in the `role_mapping.yml` file on each node in the {{es}} cluster. For more information, see [Using role mapping files](../../../deploy-manage/users-roles/cluster-or-deployment-auth/mapping-users-groups-to-roles.md#mapping-roles-file). + These settings assume that the distinguished name (DN) in the certificate is mapped to the appropriate roles in the `role_mapping.yml` file on each node in the {{es}} cluster. For more information, see [Using role mapping files](/deploy-manage/users-roles/cluster-or-deployment-auth/mapping-users-groups-to-roles.md#mapping-roles-file). By default, APM Server uses the list of trusted certificate authorities (CA) from the operating system where APM Server is running. If the certificate authority that signed your node certificates is not in the host system’s trusted certificate authorities list, you need to add the path to the `.pem` file that contains your CA’s certificate to the APM Server configuration. This will configure APM Server to use a specific list of CA certificates instead of the default list from the OS. @@ -282,14 +251,10 @@ Authentication is specified in the APM Server configuration file: 2. The path to the certificate for SSL client authentication 3. The client certificate key - ::::{note} For any given connection, the SSL/TLS certificates must have a subject that matches the value specified for `hosts`, or the SSL handshake fails. For example, if you specify `hosts: ["foobar:9200"]`, the certificate MUST include `foobar` in the subject (`CN=foobar`) or as a subject alternative name (SAN). Make sure the hostname resolves to the correct IP address. If no DNS is available, then you can associate the IP address with your hostname in `/etc/hosts` (on Unix) or `C:\Windows\System32\drivers\etc\hosts` (on Windows). :::: - - - ### Learn more about secure communication [apm-securing-communication-learn-more] More information on sending data to a secured cluster is available in the configuration reference: diff --git a/solutions/observability/apps/configure-individual-browser-monitors.md b/solutions/observability/apps/configure-individual-browser-monitors.md index 5de38c17b2..f241687aba 100644 --- a/solutions/observability/apps/configure-individual-browser-monitors.md +++ b/solutions/observability/apps/configure-individual-browser-monitors.md @@ -10,20 +10,18 @@ applies_to: # Configure individual browser monitors [synthetics-monitor-use] - ::::{note} -This is only relevant for monitors that are created and managed using a [Synthetics project](../../../solutions/observability/apps/get-started.md#observability-synthetics-get-started-synthetics-project). For more information on configuring browser monitors added in the UI, refer to [Create monitors in the Synthetics UI](../../../solutions/observability/apps/create-monitors-in-synthetics-app.md). +This is only relevant for monitors that are created and managed using a [Synthetics project](/solutions/observability/apps/get-started.md#observability-synthetics-get-started-synthetics-project). For more information on configuring browser monitors added in the UI, refer to [Create monitors in the Synthetics UI](/solutions/observability/apps/create-monitors-in-synthetics-app.md). :::: - -After [writing synthetic journeys](../../../solutions/observability/apps/write-synthetic-test.md), you can use `monitor.use` to configure the browser monitors that will run your tests. +After [writing synthetic journeys](/solutions/observability/apps/write-synthetic-test.md), you can use `monitor.use` to configure the browser monitors that will run your tests. You’ll need to set a few configuration options: * **Give your monitor a name.** Provide a human readable name and a unique ID for the monitor. This will appear in {{kib}} or your Observability Serverless project where you can view and manage monitors after they’re created. * **Set the schedule.** Specify the interval at which your tests will run. -* **Specify where the monitors should run.** You can run monitors on Elastic’s global managed testing infrastructure or [create a {{private-location}}](../../../solutions/observability/apps/monitor-resources-on-private-networks.md) to run monitors from your own premises. +* **Specify where the monitors should run.** You can run monitors on Elastic’s global managed testing infrastructure or [create a {{private-location}}](/solutions/observability/apps/monitor-resources-on-private-networks.md) to run monitors from your own premises. * **Set other options as needed.** There are several other options you can set to customize your implementation including params, tags, screenshot options, throttling options, and more. Configure each monitor directly in your `journey` code using `monitor.use`. The `monitor` API allows you to set unique options for each journey’s monitor directly through code. For example: @@ -54,4 +52,4 @@ journey('Ensure placeholder is correct', ({ page, params }) => { }); ``` -For each journey, you can specify its `schedule` and the `locations` in which it runs. When those options are not set, Synthetics will use the default values in the global configuration file. For more details, refer to [Configure a Synthetics project](../../../solutions/observability/apps/configure-synthetics-projects.md). +For each journey, you can specify its `schedule` and the `locations` in which it runs. When those options are not set, Synthetics will use the default values in the global configuration file. For more details, refer to [Configure a Synthetics project](/solutions/observability/apps/configure-synthetics-projects.md). diff --git a/solutions/observability/apps/configure-kafka-output.md b/solutions/observability/apps/configure-kafka-output.md index 74113039d2..9ad546a72c 100644 --- a/solutions/observability/apps/configure-kafka-output.md +++ b/solutions/observability/apps/configure-kafka-output.md @@ -8,7 +8,6 @@ applies_to: # Configure the Kafka output [apm-kafka-output] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,7 +15,6 @@ The Kafka output is not yet supported by {{fleet}}-managed APM Server. :::: - The Kafka output sends events to Apache Kafka. To use this output, edit the APM Server configuration file to disable the {{es}} output by commenting it out, and enable the Kafka output by uncommenting the Kafka section. @@ -42,30 +40,24 @@ output.kafka: Events bigger than [`max_message_bytes`](#apm-kafka-max_message_bytes) will be dropped. To avoid this problem, make sure APM Server does not generate events bigger than [`max_message_bytes`](#apm-kafka-max_message_bytes). :::: - - ## Compatibility [apm-kafka-compatibility] This output works with all Kafka versions in between 0.11 and 2.2.2. Older versions might work as well, but are not supported. - ## Configuration options [_configuration_options_5] You can specify the following options in the `kafka` section of the `apm-server.yml` config file: - ### `enabled` [_enabled_4] The `enabled` config is a boolean setting to enable or disable the output. If set to false, the output is disabled. The default value is `false`. - ### `hosts` [_hosts_2] The list of Kafka broker addresses from where to fetch the cluster metadata. The cluster metadata contain the actual Kafka brokers events are published to. - ### `version` [_version] Kafka version apm-server is assumed to run against. Defaults to 1.0.0. @@ -76,24 +68,20 @@ Valid values are all Kafka releases in between `0.8.2.0` and `2.0.0`. See [Compatibility](#apm-kafka-compatibility) for information on supported versions. - ### `username` [_username_2] The username for connecting to Kafka. If username is configured, the password must be configured as well. - ### `password` [_password_2] The password for connecting to Kafka. - ### `sasl.mechanism` [_sasl_mechanism] ::::{warning} This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. :::: - The SASL mechanism to use when connecting to Kafka. It can be one of: * `PLAIN` for SASL/PLAIN. @@ -102,7 +90,6 @@ The SASL mechanism to use when connecting to Kafka. It can be one of: If `sasl.mechanism` is not set, `PLAIN` is used if `username` and `password` are provided. Otherwise, SASL authentication is disabled. - ### `topic` [apm-topic-option-kafka] The Kafka topic used for produced events. @@ -115,7 +102,6 @@ topic: '%{[fields.log_topic]}' See the [`topics`](#apm-topics-option-kafka) setting for other ways to set the topic dynamically. - ### `topics` [apm-topics-option-kafka] An array of topic selector rules. Each rule specifies the `topic` to use for events that match the rule. During publishing, APM Server sets the `topic` for each event based on the first matching rule in the array. Rules can contain conditionals, format string-based fields, and name mappings. If the `topics` setting is missing or no rule matches, the [`topic`](#apm-topic-option-kafka) field is used. @@ -151,14 +137,12 @@ output.kafka: This configuration results in topics named `critical-9.0.0-beta1`, `error-9.0.0-beta1`, and `logs-9.0.0-beta1`. - ### `key` [_key] Optional formatted string specifying the Kafka event key. If configured, the event key can be extracted from the event using a format string. See the Kafka documentation for the implications of a particular choice of key; by default, the key is chosen by the Kafka cluster. - ### `partition` [_partition] Kafka output broker event partitioning strategy. Must be one of `random`, `round_robin`, or `hash`. By default the `hash` partitioner is used. @@ -177,25 +161,20 @@ All partitioners will try to publish events to all partitions by default. If a p Publishing to a subset of available partitions potentially increases resource usage because events may become unevenly distributed. :::: - - ### `client_id` [_client_id] The configurable client ID used for logging, debugging, and auditing purposes. The default is "beats". - ### `worker` [_worker_2] The number of concurrent load-balanced Kafka output workers. - ### `codec` [_codec] Output codec configuration. If the `codec` section is missing, events will be JSON encoded. See [Change the output codec](configure-console-output.md#apm-configuration-output-codec) for more information. - ### `metadata` [_metadata] Kafka metadata update settings. The metadata do contain information about brokers, topics, partition, and active leaders to use for publishing. @@ -212,7 +191,6 @@ Kafka metadata update settings. The metadata do contain information about broker **`retry.backoff`** : Waiting time between retries during leader elections. Default is `250ms`. - ### `max_retries` [_max_retries_3] The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped. @@ -221,47 +199,38 @@ Set `max_retries` to a value less than 0 to retry until all events are published The default is 3. - ### `backoff.init` [_backoff_init_3] The number of seconds to wait before trying to republish to Kafka after a network error. After waiting `backoff.init` seconds, APM Server tries to republish. If the attempt fails, the backoff timer is increased exponentially up to `backoff.max`. After a successful publish, the backoff timer is reset. The default is `1s`. - ### `backoff.max` [_backoff_max_3] The maximum number of seconds to wait before attempting to republish to Kafka after a network error. The default is `60s`. - ### `bulk_max_size` [_bulk_max_size_2] The maximum number of events to bulk in a single Kafka request. The default is 2048. - ### `bulk_flush_frequency` [_bulk_flush_frequency] Duration to wait before sending bulk Kafka request. 0 is no delay. The default is 0. - ### `timeout` [_timeout_3] The number of seconds to wait for responses from the Kafka brokers before timing out. The default is 30 (seconds). - ### `broker_timeout` [_broker_timeout] The maximum duration a broker will wait for number of required ACKs. The default is `10s`. - ### `channel_buffer_size` [_channel_buffer_size] Per Kafka broker number of messages buffered in output pipeline. The default is 256. - ### `keep_alive` [_keep_alive] The keep-alive period for an active network connection. If `0s`, keep-alives are disabled. The default is `0s`. - ### `compression` [_compression] Sets the output compression codec. Must be one of `none`, `snappy`, `lz4` and `gzip`. The default is `gzip`. @@ -273,8 +242,6 @@ When targeting Azure Event Hub for Kafka, set `compression` to `none` as the pro :::: - - ### `compression_level` [_compression_level_3] Sets the compression level used by gzip. Setting this value to 0 disables compression. The compression level must be in the range of 1 (best speed) to 9 (best compression). @@ -283,29 +250,24 @@ Increasing the compression level will reduce the network usage but will increase The default value is 4. - ### `max_message_bytes` [apm-kafka-max_message_bytes] The maximum permitted size of JSON-encoded messages. Bigger messages will be dropped. The default value is 1000000 (bytes). This value should be equal to or less than the broker’s `message.max.bytes`. - ### `required_acks` [_required_acks] The ACK reliability level required from broker. 0=no response, 1=wait for local commit, -1=wait for all replicas to commit. The default is 1. Note: If set to 0, no ACKs are returned by Kafka. Messages might be lost silently on error. - ### `enable_krb5_fast` [_enable_krb5_fast] ::::{warning} This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. :::: - Enable Kerberos FAST authentication. This may conflict with some Active Directory installations. It is separate from the standard Kerberos settings because this flag only applies to the Kafka output. The default is `false`. - ### `ssl` [_ssl_3] Configuration options for SSL parameters like the root CA for Kafka connections. The Kafka host keystore should be created with the `-keyalg RSA` argument to ensure it uses a cipher supported by [{{filebeat}}'s Kafka library](https://github.com/Shopify/sarama/wiki/Frequently-Asked-Questions#why-cant-sarama-connect-to-my-kafka-cluster-using-ssl). See [SSL/TLS output settings](ssltls-output-settings.md) for more information. diff --git a/solutions/observability/apps/configure-kibana-endpoint.md b/solutions/observability/apps/configure-kibana-endpoint.md index c977dae335..ca61ad46ab 100644 --- a/solutions/observability/apps/configure-kibana-endpoint.md +++ b/solutions/observability/apps/configure-kibana-endpoint.md @@ -8,7 +8,6 @@ applies_to: # Configure the Kibana endpoint [apm-setup-kibana-endpoint] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -18,7 +17,6 @@ For all other use-cases, starting in version 8.7.0, APM agent configurations is :::: - Here’s a sample configuration: ```yaml @@ -26,17 +24,14 @@ apm-server.kibana.enabled: true apm-server.kibana.host: "http://localhost:5601" ``` - ## {{kib}} endpoint configuration options [_kib_endpoint_configuration_options] You can specify the following options in the `apm-server.kibana` section of the `apm-server.yml` config file. These options are not required for a Fleet-managed APM Server. - ### `apm-server.kibana.enabled` [apm-kibana-enabled] Defaults to `false`. Must be `true` to use APM Agent configuration. - ### `apm-server.kibana.host` [apm-kibana-host] The {{kib}} host that APM Server will communicate with. The default is `127.0.0.1:5601`. The value of `host` can be a `URL` or `IP:PORT`. For example: `http://192.15.3.2`, `192:15.3.2:5601` or `http://192.15.3.2:6701/path`. If no port is specified, `5601` is used. @@ -45,10 +40,8 @@ The {{kib}} host that APM Server will communicate with. The default is `127.0.0. When a node is defined as an `IP:PORT`, the *scheme* and *path* are taken from the [apm-server.kibana.protocol](#apm-kibana-protocol-option) and [apm-server.kibana.path](#apm-kibana-path-option) config options. :::: - IPv6 addresses must be defined using the following format: `https://[2001:db8::1]:5601`. - ### `apm-server.kibana.protocol` [apm-kibana-protocol-option] The name of the protocol {{kib}} is reachable on. The options are: `http` or `https`. The default is `http`. However, if you specify a URL for host, the value of `protocol` is overridden by whatever scheme you specify in the URL. @@ -61,27 +54,22 @@ apm-server.kibana.protocol: "http" apm-server.kibana.path: /kibana ``` - ### `apm-server.kibana.username` [_apm_server_kibana_username] The basic authentication username for connecting to {{kib}}. - ### `apm-server.kibana.password` [_apm_server_kibana_password] The basic authentication password for connecting to {{kib}}. - ### `apm-server.kibana.api_key` [_apm_server_kibana_api_key] Authentication with an API key. Formatted as `id:api_key` - ### `apm-server.kibana.path` [apm-kibana-path-option] An HTTP path prefix that is prepended to the HTTP API calls. This is useful for the cases where {{kib}} listens behind an HTTP reverse proxy that exports the API under a custom prefix. - ### `apm-server.kibana.ssl.enabled` [_apm_server_kibana_ssl_enabled] Enables APM Server to use SSL settings when connecting to {{kib}} via HTTPS. If you configure APM Server to connect over HTTPS, this setting defaults to `true` and APM Server uses the default SSL settings. diff --git a/solutions/observability/apps/configure-lightweight-monitors.md b/solutions/observability/apps/configure-lightweight-monitors.md index 3e20508063..18197186e0 100644 --- a/solutions/observability/apps/configure-lightweight-monitors.md +++ b/solutions/observability/apps/configure-lightweight-monitors.md @@ -15,22 +15,20 @@ Monitor the status of network endpoints using the following lightweight checks: * **ICMP**: Check the availability of your hosts. The ICMP monitor uses ICMP (v4 and v6) Echo Requests to check the network reachability of the hosts you are pinging. This will tell you whether the host is available and connected to the network, but doesn’t tell you if a service on the host is running or not. * **TCP**: Monitor the services running on your hosts. The TCP monitor checks individual ports to make sure the service is accessible and running. -Lightweight monitors can be configured using either the [Synthetics UI](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-ui) or [{{project-monitors-cap}}](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-projects). - +Lightweight monitors can be configured using either the [Synthetics UI](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-ui) or [{{project-monitors-cap}}](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-projects). ## Synthetics UI [synthetics-lightweight-ui] -To use the UI, go to the Synthetics UI in {{kib}} or in your Observability Serverless project to create and configure monitors. For step-by-step instructions, refer to [Use the Synthetics UI](../../../solutions/observability/apps/create-monitors-in-synthetics-app.md). +To use the UI, go to the Synthetics UI in {{kib}} or in your Observability Serverless project to create and configure monitors. For step-by-step instructions, refer to [Use the Synthetics UI](/solutions/observability/apps/create-monitors-in-synthetics-app.md). :::{image} /solutions/images/observability-synthetics-get-started-ui-lightweight.png :alt: Synthetics Create monitor UI :screenshot: ::: - ## {{project-monitors-cap}} [synthetics-lightweight-projects] -To use YAML files to create lightweight monitors in a Synthetics project, [set up the Synthetics project](../../../solutions/observability/apps/create-monitors-with-project-monitors.md) and configure monitors in YAML files in the `lightweight` directory. +To use YAML files to create lightweight monitors in a Synthetics project, [set up the Synthetics project](/solutions/observability/apps/create-monitors-with-project-monitors.md) and configure monitors in YAML files in the `lightweight` directory. In each YAML file, define a set of `monitors` to check your remote hosts. Each `monitor` item is an entry in a YAML list and begins with a dash (`-`). You can define the type of monitor to use, the hosts to check, and other optional settings. @@ -58,17 +56,16 @@ heartbeat.monitors: ``` $$$synthetics-monitor-options$$$ -There are some common monitor configuration options that are the same for all lightweight monitor types. For a complete list, refer to [Common options](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-common-options). +There are some common monitor configuration options that are the same for all lightweight monitor types. For a complete list, refer to [Common options](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-common-options). Each monitor type also has additional configuration options that are specific to that type. Refer to: -* [HTTP options](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-http) -* [ICMP options](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-icmp) -* [TCP options](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-tcp) +* [HTTP options](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-http) +* [ICMP options](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-icmp) +* [TCP options](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-tcp) The `tcp` and `http` monitor types both support SSL/TLS and some proxy settings. - ### Common options [synthetics-lightweight-common-options] You can specify the following options when defining a synthetic monitor in any location. These options are the same for all monitors. Each monitor type has additional configuration options that are specific to that monitor type. @@ -84,11 +81,10 @@ $$$monitor-type$$$ * `icmp`: Uses an ICMP (v4 and v6) Echo Request to ping the configured hosts. Requires special permissions or root access. * `tcp`: Connects via TCP and optionally verifies the endpoint by sending and/or receiving a custom payload. - $$$monitor-id$$$ **`id`** -: Type: [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) +: Type: [string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) **Required**. A unique identifier for this configuration. This should not change with edits to the monitor configuration regardless of changes to any config fields. @@ -109,11 +105,10 @@ $$$monitor-id$$$ :::: - $$$monitor-name$$$ **`name`** -: Type: [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) +: Type: [string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) Human readable name for this monitor. @@ -127,19 +122,17 @@ $$$monitor-name$$$ name: Example website ``` - $$$monitor-service_name$$$ **`service.name`** -: Type: [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) +: Type: [string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) APM service name for this monitor. Corresponds to the `service.name` ECS field. Set this when monitoring an app that is also using APM to enable integrations between Synthetics and APM data in Kibana or your Observability Serverless project. - $$$monitor-enabled$$$ **`enabled`** -: Type: [boolean](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool) +: Type: [boolean](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool) Whether the monitor is enabled. @@ -151,11 +144,10 @@ $$$monitor-enabled$$$ enabled: false ``` - $$$monitor-schedule$$$ **`schedule`** -: Type: [duration](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-duration) +: Type: [duration](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-duration) **Required**. The task schedule. @@ -163,18 +155,16 @@ $$$monitor-schedule$$$ Schedules with less than 1 minute resolution will be saved to the nearest minute. For example, `@every 5s` will be changed to `@every 60s` when the monitor is pushed using the CLI. :::: - **Example**: Run the task every 5 minutes from the time the monitor was started. ```yaml schedule: @every 5m ``` - $$$monitor-timeout$$$ **`timeout`** -: Type: [duration](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-duration) +: Type: [duration](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-duration) The total running time for each ping test. This is the total time allowed for testing the connection and exchanging data. @@ -186,11 +176,10 @@ $$$monitor-timeout$$$ timeout: 2m ``` - $$$monitor-tags$$$ **`tags`** -: Type: list of [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string)s +: Type: list of [string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string)s A list of tags that will be sent with the monitor event. @@ -206,7 +195,6 @@ $$$monitor-tags$$$ tags: ["tag one", "tag two"] ``` - $$$monitor-mode$$$ **`mode`** @@ -221,11 +209,10 @@ $$$monitor-mode$$$ **Example**: If you’re using a DNS-load balancer and want to ping every IP address for the specified hostname, you should use `all`. - $$$monitor-ipv4$$$ **`ipv4`** -: Type: [boolean](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool) +: Type: [boolean](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool) Whether to ping using the ipv4 protocol if hostnames are configured. @@ -237,11 +224,10 @@ $$$monitor-ipv4$$$ ipv4: false ``` - $$$monitor-ipv6$$$ **`ipv6`** -: Type: [boolean](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool) +: Type: [boolean](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool) Whether to ping using the ipv6 protocol if hostnames are configured. @@ -253,13 +239,12 @@ $$$monitor-ipv6$$$ ipv6: false ``` - $$$monitor-alert$$$ **`alert`** -: Enable or disable alerts on this monitor. Read more about alerts in [Alerting](../../../solutions/observability/apps/configure-synthetics-settings.md#synthetics-settings-alerting). +: Enable or disable alerts on this monitor. Read more about alerts in [Alerting](/solutions/observability/apps/configure-synthetics-settings.md#synthetics-settings-alerting). - **`status.enabled`** ([boolean](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool)) + **`status.enabled`** ([boolean](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool)) : Enable monitor status alerts on this monitor. **Default**: `true` @@ -270,8 +255,7 @@ $$$monitor-alert$$$ alert.status.enabled: true ``` - - **`tls.enabled`** ([boolean](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool)) + **`tls.enabled`** ([boolean](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool)) : Enable TLS certificate alerts on this monitor. **Default**: `true` @@ -282,11 +266,10 @@ $$$monitor-alert$$$ alert.tls.enabled: true ``` - $$$monitor-retest_on_failure$$$ **`retest_on_failure`** -: Type: [boolean](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool) +: Type: [boolean](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool) Enable or disable retesting when a monitor fails. Default is `true`. @@ -298,7 +281,6 @@ $$$monitor-retest_on_failure$$$ retest_on_failure: false ``` - $$$monitor-locations$$$ **`locations`** @@ -308,7 +290,7 @@ $$$monitor-locations$$$ To list available locations you can: - * Run the [`elastic-synthetics locations` command](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command). + * Run the [`elastic-synthetics locations` command](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command). * Find `Synthetics` in the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md) and click **Create monitor**. Locations will be listed in *Locations*. **Examples**: @@ -324,23 +306,22 @@ $$$monitor-locations$$$ ``` ::::{note} - This can also be set using [`monitor.locations` in the Synthetics project configuration file](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor) or via the CLI using the [`--location` flag on `push`](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). + This can also be set using [`monitor.locations` in the Synthetics project configuration file](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor) or via the CLI using the [`--location` flag on `push`](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). The value defined via the CLI takes precedence over the value defined in the lightweight monitor configuration, and the value defined in the lightweight monitor configuration takes precedence over the value defined in the Synthetics project configuration file. :::: - $$$monitor-private_locations$$$ **`private_locations`** -: Type: list of [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string)s +: Type: list of [string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string)s - The [{{private-location}}s](../../../solutions/observability/apps/monitor-resources-on-private-networks.md) to which the monitors will be deployed. These {{private-location}}s refer to locations hosted and managed by you, whereas `locations` are hosted by Elastic. You can specify a {{private-location}} using the location’s name. + The [{{private-location}}s](/solutions/observability/apps/monitor-resources-on-private-networks.md) to which the monitors will be deployed. These {{private-location}}s refer to locations hosted and managed by you, whereas `locations` are hosted by Elastic. You can specify a {{private-location}} using the location’s name. To list available {{private-location}}s you can: - * Run the [`elastic-synthetics locations` command](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command) and specify the {{kib}} URL of the deployment. This will fetch all available private locations associated with the deployment. + * Run the [`elastic-synthetics locations` command](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command) and specify the {{kib}} URL of the deployment. This will fetch all available private locations associated with the deployment. * Find `Synthetics` in the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md) and click **Create monitor**. {{private-location}}s will be listed in *Locations*. **Examples**: @@ -356,17 +337,16 @@ $$$monitor-private_locations$$$ ``` ::::{note} - This can also be set using [`monitor.privateLocations` in the Synthetics project configuration file](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor) or via the CLI using the [`--privateLocations` flag on `push`](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). + This can also be set using [`monitor.privateLocations` in the Synthetics project configuration file](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor) or via the CLI using the [`--privateLocations` flag on `push`](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). The value defined via the CLI takes precedence over the value defined in the lightweight monitor configuration, and the value defined in the lightweight monitor configuration takes precedence over the value defined in Synthetics project configuration file. :::: - $$$monitor-fields$$$ **`fields`** -: A list of key-value pairs that will be sent with each monitor event. The `fields` are appended to {{es}} documents as `labels`, and those labels are displayed in {{kib}} in the *Monitor details* panel in the [individual monitor’s *Overview* tab](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-individual-monitors-overview). +: A list of key-value pairs that will be sent with each monitor event. The `fields` are appended to {{es}} documents as `labels`, and those labels are displayed in {{kib}} in the *Monitor details* panel in the [individual monitor’s *Overview* tab](/solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-individual-monitors-overview). **Examples**: @@ -381,26 +361,23 @@ $$$monitor-fields$$$ fields.team: synthetics ``` - - ### HTTP options [synthetics-lightweight-http] The options described here configure Synthetics to connect via HTTP and optionally verify that the host returns the expected response. -Valid options for HTTP monitors include [all common options](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-common-options) and the following HTTP-specific options: +Valid options for HTTP monitors include [all common options](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-common-options) and the following HTTP-specific options: $$$monitor-http-urls$$$ **`urls`** -: Type: [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) +: Type: [string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) **Required**. The URL to ping. - $$$monitor-http-max_redirects$$$ **`max_redirects`** -: Type: [number](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-numbers) +: Type: [number](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-numbers) The total number of redirections Synthetics will follow. @@ -410,7 +387,6 @@ $$$monitor-http-max_redirects$$$ **Default**: `0` - $$$monitor-http-proxy_headers$$$ **`proxy_headers`** @@ -419,7 +395,7 @@ $$$monitor-http-proxy_headers$$$ $$$monitor-http-proxy_url$$$ **`proxy_url`** -: ([string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string)) The HTTP proxy URL. This setting is optional. +: ([string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string)) The HTTP proxy URL. This setting is optional. **Example**: @@ -427,25 +403,22 @@ $$$monitor-http-proxy_url$$$ http://proxy.mydomain.com:3128 ``` - $$$monitor-http-username$$$ **`username`** -: Type: [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) +: Type: [string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) The username for authenticating with the server. The credentials are passed with the request. This setting is optional. You need to specify credentials when your `check.response` settings require it. For example, you can check for a 403 response (`check.response.status: [403]`) without setting credentials. - $$$monitor-http-password$$$ **`password`** -: Type: [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) +: Type: [string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) The password for authenticating with the server. This setting is optional. - $$$monitor-http-ssl$$$ **`ssl`** @@ -466,17 +439,15 @@ $$$monitor-http-ssl$$$ supported_protocols: ["TLSv1.0", "TLSv1.1", "TLSv1.2"] ``` - $$$monitor-http-headers$$$ **`headers`** -: Type: [boolean](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool) +: Type: [boolean](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool) Controls the indexing of the HTTP response headers `http.response.body.headers` field. Set `response.include_headers` to `false` to disable. **Default**: `true` - $$$monitor-http-response$$$ **`response`** @@ -489,13 +460,11 @@ $$$monitor-http-response$$$ * `never`: Never include the body. * `always`: Always include the body with checks. - - **`include_body_max_bytes`** ([number](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-numbers)) + **`include_body_max_bytes`** ([number](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-numbers)) : Set `response.include_body_max_bytes` to control the maximum size of the stored body contents. **Default**: `1024` - $$$monitor-http-check$$$ **`check`** @@ -507,19 +476,16 @@ $$$monitor-http-check$$$ The HTTP method to use. - **`headers`** : Type: [HTTP headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers) A dictionary of additional HTTP headers to send. By default Synthetics will set the *User-Agent* header to identify itself. - **`body`** - : Type: [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) + : Type: [string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) Optional request body content. - **Example**: This monitor POSTs an `x-www-form-urlencoded` string to the endpoint `/demo/add`. ```yaml @@ -531,14 +497,13 @@ check.request: body: "name=first&email=someemail%40someemailprovider.com" ``` - **`response`** : The expected `response`. Under `check.response`, specify these options: **`status`** - : Type: list of [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string)s + : Type: list of [string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string)s A list of expected status codes. 4xx and 5xx codes are considered `down` by default. Other codes are considered `up`. @@ -549,15 +514,13 @@ check.request: status: [200, 201] ``` - **`headers`** : Type: [HTTP headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers) The required response headers. - **`body.positive`** - : Type: list of [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string)s + : Type: list of [string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string)s A list of regular expressions to match the body output. Only a single expression needs to match. @@ -574,8 +537,7 @@ check.request: - Foo ``` - - **`body.negative`** (list of [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string)s) + **`body.negative`** (list of [string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string)s) : A list of regular expressions to match the body output negatively. Return match failed if single expression matches. HTTP response bodies of up to 100MiB are supported. This monitor examines match successfully if there is no *bar* or *Bar* at all, examines match failed if there is *bar* or *Bar* in the response body: @@ -607,7 +569,6 @@ check.request: - Bar ``` - **`json`** : A list of expressions executed against the body when parsed as JSON. Body sizes must be less than or equal to 100 MiB. @@ -627,8 +588,6 @@ check.request: expression: 'foo.bar == "myValue"' ``` - - ### ICMP options [synthetics-lightweight-icmp] The options described here configure Synthetics to use ICMP (v4 and v6) Echo Requests to check the configured hosts. On most platforms you must execute Synthetics with elevated permissions to perform ICMP pings. @@ -637,12 +596,12 @@ On Linux, regular users may perform pings if the right file capabilities are set Other platforms may require Synthetics to run as root or administrator to execute pings. -Valid options for ICMP monitors include [all common options](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-common-options) and the following ICMP-specific options: +Valid options for ICMP monitors include [all common options](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-common-options) and the following ICMP-specific options: $$$monitor-icmp-hosts$$$ **`hosts`** -: Type: [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) +: Type: [string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) **Required**. The host to ping. @@ -652,11 +611,10 @@ $$$monitor-icmp-hosts$$$ hosts: "myhost" ``` - $$$monitor-icmp-wait$$$ **`wait`** -: Type: [duration](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-duration) +: Type: [duration](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-duration) The duration to wait before emitting another ICMP Echo Request if no response is received. @@ -668,18 +626,16 @@ $$$monitor-icmp-wait$$$ wait: 1m ``` - - ### TCP options [synthetics-lightweight-tcp] The options described here configure Synthetics to connect via TCP and optionally verify the endpoint by sending and/or receiving a custom payload. -Valid options for TCP monitors include [all common options](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-common-options) and the following TCP-specific options: +Valid options for TCP monitors include [all common options](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-common-options) and the following TCP-specific options: $$$monitor-tcp-hosts$$$ **`hosts`** -: Type: [string](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) +: Type: [string](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-string) **Required**. The host to ping. The value can be: @@ -690,7 +646,6 @@ $$$monitor-tcp-hosts$$$ * `host` is the hostname. * `port` is the port number. - **Examples**: ```yaml @@ -701,7 +656,6 @@ $$$monitor-tcp-hosts$$$ hosts: "tcp://localhost:8000" ``` - $$$monitor-tcp-check$$$ **`check`** @@ -720,7 +674,6 @@ $$$monitor-tcp-check$$$ receive: 'Hello World' ``` - $$$monitor-tcp-proxy_url$$$ **`proxy_url`** @@ -738,17 +691,15 @@ $$$monitor-tcp-proxy_url$$$ proxy_url: socks5://user:password@socks5-proxy:2233 ``` - $$$monitor-tcp-proxy_use_local_resolver$$$ **`proxy_use_local_resolver`** -: Type: [boolean](../../../solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool) +: Type: [boolean](/solutions/observability/apps/configure-lightweight-monitors.md#synthetics-lightweight-data-bool) A Boolean value that determines whether hostnames are resolved locally instead of being resolved on the proxy server. The default value is `false`, which means that name resolution occurs on the proxy server. **Default**: `false` - $$$monitor-tcp-ssl$$$ **`ssl`** @@ -766,13 +717,10 @@ $$$monitor-tcp-ssl$$$ Also see [Configure SSL](beats://reference/heartbeat/configuration-ssl.md) for a full description of the `ssl` options. - - ### Data types reference [synthetics-lightweight-data-types] Values of configuration settings are interpreted as required by Synthetics. If a value can’t be correctly interpreted as the required type - for example a string is given when a number is required - Synthetics will fail to start up. - #### Boolean [synthetics-lightweight-data-bool] Boolean values can be either `true` or `false`. Alternative names for `true` are `yes` and `on`. Instead of `false` the values `no` and `off` can be used. @@ -782,7 +730,6 @@ enabled: true disabled: false ``` - #### Number [synthetics-lightweight-data-numbers] Number values require you to enter the number *without* single or double quotes. @@ -797,8 +744,6 @@ float: 5.4 Some settings only support a restricted number range. :::: - - #### String [synthetics-lightweight-data-string] In [YAML](http://www.yaml.org), multiple styles of string definitions are supported: double-quoted, single-quoted, unquoted. @@ -813,8 +758,6 @@ Unquoted style requires no quotes, but does not support any escaping and can’t Single-quoted style is recommended when defining regular expressions, event format strings, windows file paths, or non-alphabetical symbolic characters. :::: - - #### Duration [synthetics-lightweight-data-duration] Durations require a numeric value with optional fraction and required unit. Valid time units are `ns`, `us`, `ms`, `s`, `m`, `h`. Sometimes features based on durations can be disabled by using zero or negative durations. @@ -825,7 +768,6 @@ duration2: 6h duration_disabled: -1s ``` - #### Regular expression [synthetics-lightweight-data-regex] Regular expressions are special strings that are compiled into regular expressions at load time. diff --git a/solutions/observability/apps/configure-logging.md b/solutions/observability/apps/configure-logging.md index 58d8940518..cddbf9b419 100644 --- a/solutions/observability/apps/configure-logging.md +++ b/solutions/observability/apps/configure-logging.md @@ -8,7 +8,6 @@ applies_to: # Configure logging [apm-configuration-logging] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,7 +15,6 @@ These configuration options are only relevant to APM Server binary users. Fleet- :::: - The `logging` section of the `apm-server.yml` config file contains options for configuring the logging output. The logging system can write logs to the syslog or rotate log files. If logging is not explicitly configured the file output is used. @@ -35,23 +33,18 @@ logging.files: In addition to setting logging options in the config file, you can modify the logging output configuration from the command line. See [Command reference](apm-server-command-reference.md). :::: - ::::{warning} When APM Server is running on a Linux system with systemd, it uses by default the `-e` command line option, that makes it write all the logging output to stderr so it can be captured by journald. Other outputs are disabled. See [APM Server and systemd](apm-server-systemd.md) to know more and learn how to change this. :::: - - ## Configuration options [_configuration_options_2] You can specify the following options in the `logging` section of the `apm-server.yml` config file: - ### `logging.to_stderr` [_logging_to_stderr] When true, writes all logging output to standard error output. This is equivalent to using the `-e` command line option. - ### `logging.to_syslog` [_logging_to_syslog] When true, writes all logging output to the syslog. @@ -60,13 +53,10 @@ When true, writes all logging output to the syslog. This option is not supported on Windows. :::: - - ### `logging.to_eventlog` [_logging_to_eventlog] When true, writes all logging output to the Windows Event Log. - ### `logging.to_files` [_logging_to_files] When true, writes all logging output to files. The log files are automatically rotated when the log file size limit is reached. @@ -75,8 +65,6 @@ When true, writes all logging output to files. The log files are automatically r APM Server only creates a log file if there is logging output. For example, if you set the log [`level`](#apm-level) to `error` and there are no errors, there will be no log file in the directory specified for logs. :::: - - ### `logging.level` [apm-level] Minimum log level. One of `debug`, `info`, `warning`, or `error`. The default log level is `info`. @@ -93,7 +81,6 @@ Minimum log level. One of `debug`, `info`, `warning`, or `error`. The default lo `error` : Logs errors and critical errors. - ### `logging.selectors` [apm-selectors] The list of debugging-only selector tags used by different APM Server components. Use `*` to enable debug output for all components. Use `publisher` to display debug messages related to event publishing. @@ -105,7 +92,6 @@ To see which selectors are available, run APM Server in debug mode (set `logging :::: - To configure multiple selectors, use the following [YAML list syntax](beats://reference/libbeat/config-file-format.md): ```yaml @@ -114,7 +100,6 @@ logging.selectors: [ harvester, input ] To override selectors at the command line, use the `-d` global flag (`-d` also sets the debug log level). For more information, see [Command reference](apm-server-command-reference.md). - ### `logging.metrics.enabled` [_logging_metrics_enabled] By default, APM Server periodically logs its internal metrics that have changed in the last period. For each metric that changed, the delta from the value at the beginning of the period is logged. Also, the total values for all non-zero internal metrics are logged on shutdown. Set this to false to disable this behavior. The default is true. @@ -127,32 +112,26 @@ Here is an example log line: Note that we currently offer no backwards compatible guarantees for the internal metrics and for this reason they are also not documented. - ### `logging.metrics.period` [_logging_metrics_period] The period after which to log the internal metrics. The default is `30s`. - ### `logging.files.path` [_logging_files_path] The directory that log files are written to. The default is the logs path. See the [Installation layout](installation-layout.md) section for details. - ### `logging.files.name` [_logging_files_name] The name of the file that logs are written to. The default is *apm-server*. - ### `logging.files.rotateeverybytes` [_logging_files_rotateeverybytes] The maximum size of a log file. If the limit is reached, a new log file is generated. The default size limit is 10485760 (10 MB). - ### `logging.files.keepfiles` [_logging_files_keepfiles] The number of most recent rotated log files to keep on disk. Older files are deleted during log rotation. The default value is 7. The `keepfiles` options has to be in the range of 2 to 1024 files. - ### `logging.files.permissions` [_logging_files_permissions] The permissions mask to apply when rotating log files. The default value is 0600. The `permissions` option must be a valid Unix-style file permissions mask expressed in octal notation. In Go, numbers in octal notation must start with *0*. @@ -164,22 +143,18 @@ Examples: * 0640: give read and write access to the file owner, and read access to members of the group associated with the file. * 0600: give read and write access to the file owner, and no access to all others. - ### `logging.files.interval` [_logging_files_interval] Enable log file rotation on time intervals in addition to size-based rotation. Intervals must be at least `1s`. Values of `1m`, `1h`, `24h`, `7*24h`, `30*24h`, and `365*24h` are boundary-aligned with minutes, hours, days, weeks, months, and years as reported by the local system clock. All other intervals are calculated from the Unix epoch. Defaults to disabled. - ### `logging.files.rotateonstartup` [_logging_files_rotateonstartup] If the log file already exists on startup, immediately rotate it and start writing to a new file instead of appending to the existing one. Defaults to true. - ### `logging.files.redirect_stderr` [preview] [_logging_files_redirect_stderr] When true, diagnostic messages printed to APM Server’s standard error output will also be logged to the log file. This can be helpful in situations were APM Server terminates unexpectedly because an error has been detected by Go’s runtime but diagnostic information is not present in the log file. This feature is only available when logging to files (`logging.to_files` is true). Disabled by default. - ## Logging format [_logging_format] The logging format is generally the same for each logging output. The one exception is with the syslog output where the timestamp is not included in the message because syslog adds its own timestamp. diff --git a/solutions/observability/apps/configure-logstash-output.md b/solutions/observability/apps/configure-logstash-output.md index 016ad82ff1..6078c4bb72 100644 --- a/solutions/observability/apps/configure-logstash-output.md +++ b/solutions/observability/apps/configure-logstash-output.md @@ -8,7 +8,6 @@ applies_to: # Configure the Logstash output [apm-logstash-output] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,10 +15,8 @@ The {{ls}} output is not yet supported by {{fleet}}-managed APM Server. :::: - {{ls}} allows for additional processing and routing of APM events. The {{ls}} output sends events directly to {{ls}} using the lumberjack protocol, which runs over TCP. - ## Send events to {{ls}} [_send_events_to_ls] To send events to {{ls}}, you must: @@ -27,7 +24,6 @@ To send events to {{ls}}, you must: 1. [Enable the {{ls}} output in APM Server](#apm-ls-output-config) 2. [Create a {{ls}} configuration pipeline in {{ls}}](#apm-ls-config-pipeline) - ### {{ls}} output configuration [apm-ls-output-config] To enable the {{ls}} output in APM Server, edit the `apm-server.yml` file to: @@ -43,8 +39,6 @@ To enable the {{ls}} output in APM Server, edit the `apm-server.yml` file to: 1. The `hosts` option specifies the {{ls}} server and the port (`5044`) where {{ls}} is configured to listen for incoming APM Server connections. - - ### {{ls}} configuration pipeline [apm-ls-config-pipeline] Finally, you must create a {{ls}} configuration pipeline that listens for incoming APM Server connections and indexes received events into {{es}}. @@ -74,7 +68,6 @@ Finally, you must create a {{ls}} configuration pipeline that listens for incomi 1. Enables indexing into {{es}} data streams. 2. This example assumes you’re sending data to {{ecloud}}. If you’re using a self-hosted version of {{es}}, use `hosts` instead. See [{{es}} output plugin](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md) for more information. - Here’s what your basic {{ls}} configuration file will look like when we put everything together: ```json @@ -93,7 +86,6 @@ output { } ``` - ## Accessing the @metadata field [_accessing_the_metadata_field] Every event sent to {{ls}} contains a special field called [`@metadata`](logstash://reference/event-dependent-configuration.md#metadata) that you can use in {{ls}} for conditionals, filtering, indexing and more. APM Server sends the following `@metadata` to {{ls}}: @@ -111,7 +103,6 @@ Every event sent to {{ls}} contains a special field called [`@metadata`](logstas 1. To change the default `apm-server` value, set the [`index`](#apm-logstash-index) option in the APM Server config file. 2. The current version of APM Server. - In addition to `@metadata`, APM Server provides other potentially useful fields, like the `data_stream` field, which can be used to conditionally operate on [event types](learn-about-application-data-types.md), namespaces, or datasets. As an example, you might want to use {{ls}} to route all `metrics` events to the same custom metrics data stream, rather than to service-specific data streams. @@ -144,32 +135,26 @@ output { 3. Add a Logstash mutate filter to update the value of `data_stream.dataset`. 4. In this example, `cloud_id` and `cloud_auth` are stored as [environment variables](logstash://reference/environment-variables.md). - - ### Compatibility [_compatibility_2] This output works with all compatible versions of {{ls}}. See the [Elastic Support Matrix](https://www.elastic.co/support/matrix#matrix_compatibility). - ### Configuration options [_configuration_options_4] You can specify the following options in the `logstash` section of the `apm-server.yml` config file: - #### `enabled` [_enabled_3] The enabled config is a boolean setting to enable or disable the output. If set to false, the output is disabled. The default value is `false`. - #### `hosts` [apm-hosts] The list of known {{ls}} servers to connect to. If load balancing is disabled, but multiple hosts are configured, one host is selected randomly (there is no precedence). If one host becomes unreachable, another one is selected randomly. All entries in this list can contain a port number. The default port number 5044 will be used if no number is given. - #### `compression_level` [_compression_level_2] The gzip compression level. Setting this value to 0 disables compression. The compression level must be in the range of 1 (best speed) to 9 (best compression). @@ -178,19 +163,16 @@ Increasing the compression level will reduce the network usage but will increase The default value is 3. - #### `escape_html` [_escape_html_2] Configure escaping of HTML in strings. Set to `true` to enable escaping. The default value is `false`. - #### `worker` [_worker] The number of workers per configured host publishing events to {{ls}}. This is best used with load balancing mode enabled. Example: If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host). - #### `loadbalance` [apm-loadbalance] If set to true and multiple {{ls}} hosts are configured, the output plugin load balances published events onto all {{ls}} hosts. If set to false, the output plugin sends all events to only one host (determined at random) and will switch to another host if the selected one becomes unresponsive. The default value is false. @@ -202,7 +184,6 @@ output.logstash: index: apm-server ``` - #### `ttl` [_ttl] Time to live for a connection to {{ls}} after which the connection will be re-established. Useful when {{ls}} hosts represent load balancers. Since the connections to {{ls}} hosts are sticky, operating behind load balancers can lead to uneven load distribution between the instances. Specifying a TTL on the connection allows to achieve equal connection distribution between the instances. Specifying a TTL of 0 will disable this feature. @@ -213,13 +194,10 @@ The default value is 0. The "ttl" option is not yet supported on an asynchronous {{ls}} client (one with the "pipelining" option set). :::: - - #### `pipelining` [_pipelining] Configures the number of batches to be sent asynchronously to {{ls}} while waiting for ACK from {{ls}}. Output only becomes blocking once number of `pipelining` batches have been written. Pipelining is disabled if a value of 0 is configured. The default value is 2. - #### `proxy_url` [_proxy_url_2] The URL of the SOCKS5 proxy to use when connecting to the {{ls}} servers. The value must be a URL with a scheme of `socks5://`. The protocol used to communicate to {{ls}} is not based on HTTP so a web-proxy cannot be used. @@ -234,12 +212,10 @@ output.logstash: proxy_url: socks5://user:password@socks5-proxy:2233 ``` - #### `proxy_use_local_resolver` [apm-logstash-proxy-use-local-resolver] The `proxy_use_local_resolver` option determines if {{ls}} hostnames are resolved locally when using a proxy. The default value is false, which means that when a proxy is used the name resolution occurs on the proxy server. - #### `index` [apm-logstash-index] The index root name to write events to. The default is `apm-server`. For example `"apm"` generates `"[apm-]9.0.0-beta1-YYYY.MM.DD"` indices (for example, `"apm-9.0.0-beta1-2017.04.26"`). @@ -248,18 +224,14 @@ The index root name to write events to. The default is `apm-server`. For example This parameter’s value will be assigned to the `metadata.beat` field. It can then be accessed in {{ls}}'s output section as `%{[@metadata][beat]}`. :::: - - #### `ssl` [_ssl_2] Configuration options for SSL parameters like the root CA for {{ls}} connections. See [SSL/TLS output settings](ssltls-output-settings.md) for more information. To use SSL, you must also configure the [{{beats}} input plugin for {{ls}}](logstash-docs-md://lsr/plugins-inputs-beats.md) to use SSL/TLS. - #### `timeout` [_timeout_2] The number of seconds to wait for responses from the {{ls}} server before timing out. The default is 30 (seconds). - #### `max_retries` [_max_retries_2] The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped. @@ -268,7 +240,6 @@ Set `max_retries` to a value less than 0 to retry until all events are published The default is 3. - #### `bulk_max_size` [_bulk_max_size] The maximum number of events to bulk in a single {{ls}} request. The default is 2048. @@ -279,24 +250,20 @@ Specifying a larger batch size can improve performance by lowering the overhead Setting `bulk_max_size` to values less than or equal to 0 disables the splitting of batches. When splitting is disabled, the queue decides on the number of events to be contained in a batch. - #### `slow_start` [_slow_start] If enabled, only a subset of events in a batch of events is transferred per transaction. The number of events to be sent increases up to `bulk_max_size` if no error is encountered. On error, the number of events per transaction is reduced again. The default is `false`. - #### `backoff.init` [_backoff_init_2] The number of seconds to wait before trying to reconnect to {{ls}} after a network error. After waiting `backoff.init` seconds, APM Server tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to `backoff.max`. After a successful connection, the backoff timer is reset. The default is `1s`. - #### `backoff.max` [_backoff_max_2] The maximum number of seconds to wait before attempting to connect to {{ls}} after a network error. The default is `60s`. - ## Secure communication with {{ls}} [apm-configuring-ssl-logstash] You can use SSL mutual authentication to secure connections between APM Server and {{ls}}. This ensures that APM Server sends encrypted data to trusted {{ls}} servers only, and that the {{ls}} server receives data from trusted APM Server clients only. @@ -350,8 +317,6 @@ To use SSL mutual authentication: For more information about these options, see the [documentation for the {{beats}} input plugin](logstash-docs-md://lsr/plugins-inputs-beats.md). - - ### Validate the {{ls}} server’s certificate [apm-testing-ssl-logstash] Before running APM Server, you should validate the {{ls}} server’s certificate. You can use `curl` to validate the certificate even though the protocol used to communicate with {{ls}} is not based on HTTP. For example: @@ -397,7 +362,7 @@ Validation for this test fails because the certificate is not valid for the spec curl: (51) SSL: certificate verification failed (result: 5) ``` -See the [troubleshooting docs](../../../troubleshoot/observability/apm/common-problems.md#apm-ssl-client-fails) for info about resolving this issue. +See the [troubleshooting docs](/troubleshoot/observability/apm/common-problems.md#apm-ssl-client-fails) for info about resolving this issue. ### Test the APM Server to {{ls}} connection [_test_the_apm_server_to_ls_connection] @@ -407,5 +372,5 @@ If you have APM Server running as a service, first stop the service. Then test y apm-server -c apm-server.yml -e -v ``` -Any errors will be printed to the console. See the [troubleshooting docs](../../../troubleshoot/observability/apm/common-problems.md#apm-ssl-client-fails) for info about resolving common errors. +Any errors will be printed to the console. See the [troubleshooting docs](/troubleshoot/observability/apm/common-problems.md#apm-ssl-client-fails) for info about resolving common errors. diff --git a/solutions/observability/apps/configure-output-for-elasticsearch-service-on-elastic-cloud.md b/solutions/observability/apps/configure-output-for-elasticsearch-service-on-elastic-cloud.md index 66aae9d732..49634a3713 100644 --- a/solutions/observability/apps/configure-output-for-elasticsearch-service-on-elastic-cloud.md +++ b/solutions/observability/apps/configure-output-for-elasticsearch-service-on-elastic-cloud.md @@ -8,7 +8,6 @@ applies_to: # Configure the output for {{ech}} [apm-configure-cloud-id] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,12 +15,10 @@ This documentation only applies to APM Server binary users. :::: - ::::{note} -This page refers to using a separate instance of APM Server with an existing [{{ech}} deployment](https://www.elastic.co/cloud/elasticsearch-service?page=docs&placement=docs-body). If you want to use APM on {{ech}}, see: [Create your deployment](../../../deploy-manage/deploy/elastic-cloud/create-an-elastic-cloud-hosted-deployment.md) and [Add APM user settings](configure-apm-server.md). +This page refers to using a separate instance of APM Server with an existing [{{ech}} deployment](https://www.elastic.co/cloud/elasticsearch-service?page=docs&placement=docs-body). If you want to use APM on {{ech}}, see: [Create your deployment](/deploy-manage/deploy/elastic-cloud/create-an-elastic-cloud-hosted-deployment.md) and [Add APM user settings](configure-apm-server.md). :::: - APM Server comes with two settings that simplify the output configuration when used together with [{{ech}}](https://www.elastic.co/cloud/elasticsearch-service?page=docs&placement=docs-body). When defined, these setting overwrite settings from other parts in the configuration. Example: @@ -37,12 +34,10 @@ These settings can be also specified at the command line, like this: apm-server -e -E cloud.id="" -E cloud.auth="" ``` - ## `cloud.id` [_cloud_id] The Cloud ID, which can be found in the {{ecloud}} Console, is used by APM Server to resolve the {{es}} and {{kib}} URLs. This setting overwrites the `output.elasticsearch.hosts` and `setup.kibana.host` settings. - ## `cloud.auth` [_cloud_auth] When specified, the `cloud.auth` overwrites the `output.elasticsearch.username` and `output.elasticsearch.password` settings. Because the {{kib}} settings inherit the username and password from the {{es}} output, this can also be used to set the `setup.kibana.username` and `setup.kibana.password` options. diff --git a/solutions/observability/apps/configure-output.md b/solutions/observability/apps/configure-output.md index c7ceb8003a..70db626f2d 100644 --- a/solutions/observability/apps/configure-output.md +++ b/solutions/observability/apps/configure-output.md @@ -8,7 +8,6 @@ applies_to: # Configure the output [apm-configuring-output] - Output configuration options. * [{{ech}}](configure-output-for-elasticsearch-service-on-elastic-cloud.md) @@ -18,14 +17,7 @@ Output configuration options. * [Redis](configure-redis-output.md) * [Console](configure-console-output.md) - ## Source maps [apm-sourcemap-output] Source maps can be uploaded through all outputs but must eventually be stored in {{es}}. When using outputs other than {{es}}, `source_mapping.elasticsearch` must be set for source maps to be applied. Be sure to update `source_mapping.index_pattern` if source maps are stored in the non-default location. See [`source_mapping.elasticsearch`](configure-real-user-monitoring-rum.md#apm-config-sourcemapping-elasticsearch) for more details. - - - - - - diff --git a/solutions/observability/apps/configure-project-paths.md b/solutions/observability/apps/configure-project-paths.md index 0814617a5c..f6cfff9dc0 100644 --- a/solutions/observability/apps/configure-project-paths.md +++ b/solutions/observability/apps/configure-project-paths.md @@ -8,7 +8,6 @@ applies_to: # Configure project paths [apm-configuration-path] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,7 +15,6 @@ This documentation is only relevant for APM Server binary users. Fleet-managed p :::: - The `path` section of the `apm-server.yml` config file contains configuration options that define where APM Server looks for its files. For example, APM Server looks for the {{es}} template file in the configuration path and writes log files in the logs path. Please see the [Installation layout](installation-layout.md) section for more details. @@ -32,12 +30,10 @@ path.logs: /var/log/ Note that it is possible to override these options by using command line flags. - ## Configuration options [_configuration_options_8] You can specify the following options in the `path` section of the `apm-server.yml` config file: - ### `home` [_home] The home path for the APM Server installation. This is the default base path for all other path settings and for miscellaneous files that come with the distribution (for example, the sample dashboards). If not set by a CLI flag or in the configuration file, the default for the home path is the location of the APM Server binary. @@ -48,7 +44,6 @@ Example: path.home: /usr/share/beats ``` - ### `config` [_config] The configuration path for the APM Server installation. This is the default base path for configuration files, including the main YAML configuration file and the {{es}} template file. If not set by a CLI flag or in the configuration file, the default for the configuration path is the home path. @@ -59,7 +54,6 @@ Example: path.config: /usr/share/beats/config ``` - ### `data` [_data] The data path for the APM Server installation. This is the default base path for all the files in which APM Server needs to store its data. If not set by a CLI flag or in the configuration file, the default for the data path is a `data` subdirectory inside the home path. @@ -74,8 +68,6 @@ path.data: /var/lib/beats When running multiple APM Server instances on the same host, make sure they each have a distinct `path.data` value. :::: - - ### `logs` [_logs] The logs path for a APM Server installation. This is the default location for APM Server’s log files. If not set by a CLI flag or in the configuration file, the default for the logs path is a `logs` subdirectory inside the home path. @@ -86,7 +78,6 @@ Example: path.logs: /var/log/beats ``` - ### `system.hostfs` [_system_hostfs] Specifies the mount point of the host’s file system for use in monitoring a host. This can either be set in the config, or with the `--system.hostfs` CLI flag. This is used for cgroup self-monitoring. diff --git a/solutions/observability/apps/configure-real-user-monitoring-rum.md b/solutions/observability/apps/configure-real-user-monitoring-rum.md index ebbdf99559..91e5930fce 100644 --- a/solutions/observability/apps/configure-real-user-monitoring-rum.md +++ b/solutions/observability/apps/configure-real-user-monitoring-rum.md @@ -8,7 +8,6 @@ applies_to: # Configure Real User Monitoring (RUM) [apm-configuration-rum] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-yes.svg "") @@ -16,7 +15,6 @@ Most options in this section are supported by all APM Server deployment methods. :::: - The [Real User Monitoring (RUM) agent](apm-agent-rum-js://reference/index.md) captures user interactions with clients such as web browsers. These interactions are sent as events to the APM Server. Because the RUM agent runs on the client side, the connection between agent and server is unauthenticated. As a security precaution, RUM is therefore disabled by default. :::::::{tab-set} @@ -55,10 +53,8 @@ Configure and customize Fleet-managed APM settings directly in {{kib}}: ::::::: In addition, if APM Server is deployed in an origin different than the page’s origin, you will need to configure [Cross-Origin Resource Sharing (CORS)](apm-agent-rum-js://reference/configuring-cors.md) in the Agent. - ## Configuration reference [apm-enable-rum-support] - ### Enable RUM [apm-rum-enable] To enable RUM support, set to `true`. By default this is disabled. (bool) @@ -73,8 +69,6 @@ If an [API key](api-keys.md) or [secret token](secret-token.md) is configured, e :::: - - ### Allowed Origins [apm-rum-allow-origins] A list of permitted origins for RUM support. User-agents send an Origin header that will be validated against this list. This is done automatically by modern browsers as part of the [CORS specification](https://www.w3.org/TR/cors/). An origin is made of a protocol scheme, host and port, without the URL path. @@ -86,7 +80,6 @@ Default: `['*']` (allows everything). (text) | APM Server binary | `apm-server.rum.allow_origins` | | Fleet-managed | `Allowed Origins` | - ### Access-Control-Allow-Headers [apm-rum-allow-headers] HTTP requests made from the RUM agent to the APM Server are limited in the HTTP headers they are allowed to have. If any other headers are added, the request will be rejected by the browser due to Cross-Origin Resource Sharing (CORS) restrictions. Use this setting to allow additional headers. The default list of allowed headers includes "Content-Type", "Content-Encoding", and "Accept"; custom values configured here are appended to the default list and used as the value for the `Access-Control-Allow-Headers` header. @@ -98,7 +91,6 @@ Default: `[]`. (text) | APM Server binary | `apm-server.rum.allow_headers` | | Fleet-managed | `Access-Control-Allow-Headers` | - ### Custom HTTP response headers [apm-rum-response-headers] Custom HTTP headers to add to RUM responses. This can be useful for security policy compliance. @@ -112,7 +104,6 @@ Default: none. (text) | APM Server binary | `apm-server.rum.response_headers` | | Fleet-managed | `Custom HTTP response headers` | - ### Library Frame Pattern [apm-rum-library-pattern] RegExp to be matched against a stack trace frame’s `file_name` and `abs_path` attributes. If the RegExp matches, the stack trace frame is considered to be a library frame. When source mapping is applied, the `error.culprit` is set to reflect the *function* and the *filename* of the first non library frame. This aims to provide an entry point for identifying issues. @@ -124,7 +115,6 @@ Default: `"node_modules|bower_components|~"`. (text) | APM Server binary | `apm-server.rum.library_pattern` | | Fleet-managed | `Library Frame Pattern` | - ### Exclude from grouping [_exclude_from_grouping] RegExp to be matched against a stack trace frame’s `file_name`. If the RegExp matches, the stack trace frame is excluded from being used for calculating error groups. @@ -136,7 +126,6 @@ Default: `"^/webpack"` (excludes stack trace frames that have a filename startin | APM Server binary | `apm-server.rum.exclude_from_grouping` | | Fleet-managed | `Exclude from grouping` | - ## Source map configuration options [apm-rum-source-map] ::::{admonition} @@ -146,34 +135,28 @@ Source maps are supported by all APM Server deployment methods, however, the opt :::: - - ### `source_mapping.enabled` [apm-config-sourcemapping-enabled] Used to enable/disable [source mapping](create-upload-source-maps-rum.md) for RUM events. When enabled, the APM Server needs additional privileges to read source maps. See [Use feature roles](create-assign-feature-roles-to-apm-server-users.md#apm-privileges-rum-source-mapping) for more details. Default: `true` - ### `source_mapping.elasticsearch` [apm-config-sourcemapping-elasticsearch] Configure the {{es}} source map retrieval location, taking the same options as [output.elasticsearch](configure-elasticsearch-output.md). This must be set when using an output other than {{es}}, and that output is writing to {{es}}. Otherwise leave this section empty. - ### `source_mapping.cache.expiration` [apm-rum-sourcemap-cache] If a source map has been uploaded to the APM Server, [source mapping](create-upload-source-maps-rum.md) is automatically applied to documents sent to the RUM endpoint. Source maps are fetched from {{es}} and then kept in an in-memory cache for the configured time. Values configured without a time unit are treated as seconds. Default: `5m` (5 minutes) - ### `source_mapping.index_pattern` [_source_mapping_index_pattern] Previous versions of APM Server stored source maps in `apm-%{[observer.version]}-sourcemap` indices. Search source maps stored in an older version with this setting. Default: `"apm-*-sourcemap*"` - ## Ingest pipelines [_ingest_pipelines] The default APM Server pipeline includes processors that enrich RUM data prior to indexing in {{es}}. See [Parse data using ingest pipelines](parse-data-using-ingest-pipelines.md) for details on how to locate, edit, or disable this preprocessing. diff --git a/solutions/observability/apps/configure-redis-output.md b/solutions/observability/apps/configure-redis-output.md index 6df3d2a89a..a0c09abf65 100644 --- a/solutions/observability/apps/configure-redis-output.md +++ b/solutions/observability/apps/configure-redis-output.md @@ -8,7 +8,6 @@ applies_to: # Configure the Redis output [apm-redis-output] - ::::{note} ![supported deployment methods](/solutions/images/observability-binary-yes-fm-no.svg "") @@ -16,7 +15,6 @@ The Redis output is not yet supported by {{fleet}}-managed APM Server. :::: - The Redis output inserts the events into a Redis list or a Redis channel. This output plugin is compatible with the [Redis input plugin](logstash-docs-md://lsr/plugins-inputs-redis.md) for {{ls}}. To use this output, edit the APM Server configuration file to disable the {{es}} output by commenting it out, and enable the Redis output by adding `output.redis`. @@ -32,34 +30,28 @@ output.redis: timeout: 5 ``` - ## Compatibility [_compatibility_3] This output is expected to work with all Redis versions between 3.2.4 and 5.0.8. Other versions might work as well, but are not supported. - ## Configuration options [_configuration_options_6] You can specify the following `output.redis` options in the `apm-server.yml` config file: - ### `enabled` [_enabled_5] The enabled config is a boolean setting to enable or disable the output. If set to false, the output is disabled. The default value is `true`. - ### `hosts` [_hosts_3] The list of Redis servers to connect to. If load balancing is enabled, the events are distributed to the servers in the list. If one server becomes unreachable, the events are distributed to the reachable servers only. You can define each Redis server by specifying `HOST` or `HOST:PORT`. For example: `"192.15.3.2"` or `"test.redis.io:12345"`. If you don’t specify a port number, the value configured by `port` is used. Configure each Redis server with an `IP:PORT` pair or with a `URL`. For example: `redis://localhost:6379` or `rediss://localhost:6379`. URLs can include a server-specific password. For example: `redis://:password@localhost:6379`. The `redis` scheme will disable the `ssl` settings for the host, while `rediss` will enforce TLS. If `rediss` is specified and no `ssl` settings are configured, the output uses the system certificate store. - ### `index` [_index] The index name added to the events metadata for use by {{ls}}. The default is "apm-server". - ### `key` [apm-key-option-redis] The name of the Redis list or channel the events are published to. If not configured, the value of the `index` setting is used. @@ -74,7 +66,6 @@ output.redis: See the [`keys`](#apm-keys-option-redis) setting for other ways to set the key dynamically. - ### `keys` [apm-keys-option-redis] An array of key selector rules. Each rule specifies the `key` to use for events that match the rule. During publishing, APM Server uses the first matching rule in the array. Rules can contain conditionals, format string-based fields, and name mappings. If the `keys` setting is missing or no rule matches, the [`key`](#apm-key-option-redis) setting is used. @@ -113,54 +104,44 @@ output.redis: mysql: "backend_list" ``` - ### `password` [_password_3] The password to authenticate with. The default is no authentication. - ### `db` [_db] The Redis database number where the events are published. The default is 0. - ### `datatype` [_datatype] The Redis data type to use for publishing events.If the data type is `list`, the Redis RPUSH command is used and all events are added to the list with the key defined under `key`. If the data type `channel` is used, the Redis `PUBLISH` command is used and means that all events are pushed to the pub/sub mechanism of Redis. The name of the channel is the one defined under `key`. The default value is `list`. - ### `codec` [_codec_2] Output codec configuration. If the `codec` section is missing, events will be JSON encoded. See [Change the output codec](configure-console-output.md#apm-configuration-output-codec) for more information. - ### `worker` [_worker_3] The number of workers to use for each host configured to publish events to Redis. Use this setting along with the `loadbalance` option. For example, if you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host). - ### `loadbalance` [_loadbalance] If set to true and multiple hosts or workers are configured, the output plugin load balances published events onto all Redis hosts. If set to false, the output plugin sends all events to only one host (determined at random) and will switch to another host if the currently selected one becomes unreachable. The default value is true. - ### `timeout` [_timeout_4] The Redis connection timeout in seconds. The default is 5 seconds. - ### `backoff.init` [_backoff_init_4] The number of seconds to wait before trying to reconnect to Redis after a network error. After waiting `backoff.init` seconds, APM Server tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to `backoff.max`. After a successful connection, the backoff timer is reset. The default is `1s`. - ### `backoff.max` [_backoff_max_4] The maximum number of seconds to wait before attempting to connect to Redis after a network error. The default is `60s`. - ### `max_retries` [_max_retries_4] The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped. @@ -169,7 +150,6 @@ Set `max_retries` to a value less than 0 to retry until all events are published The default is 3. - ### `bulk_max_size` [_bulk_max_size_3] The maximum number of events to bulk in a single Redis request or pipeline. The default is 2048. @@ -180,12 +160,10 @@ Specifying a larger batch size can improve performance by lowering the overhead Setting `bulk_max_size` to values less than or equal to 0 disables the splitting of batches. When splitting is disabled, the queue decides on the number of events to be contained in a batch. - ### `ssl` [_ssl_4] Configuration options for SSL parameters like the root CA for Redis connections guarded by SSL proxies (for example [stunnel](https://www.stunnel.org)). See [SSL/TLS output settings](ssltls-output-settings.md) for more information. - ### `proxy_url` [_proxy_url_3] The URL of the SOCKS5 proxy to use when connecting to the Redis servers. The value must be a URL with a scheme of `socks5://`. You cannot use a web proxy because the protocol used to communicate with Redis is not based on HTTP. @@ -194,7 +172,6 @@ If the SOCKS5 proxy server requires client authentication, you can embed a usern When using a proxy, hostnames are resolved on the proxy server instead of on the client. You can change this behavior by setting the [`proxy_use_local_resolver`](#apm-redis-proxy-use-local-resolver) option. - ### `proxy_use_local_resolver` [apm-redis-proxy-use-local-resolver] This option determines whether Redis hostnames are resolved locally when using a proxy. The default value is false, which means that name resolution occurs on the proxy server. diff --git a/solutions/observability/apps/configure-settings.md b/solutions/observability/apps/configure-settings.md index dd2e930f2e..602326cb18 100644 --- a/solutions/observability/apps/configure-settings.md +++ b/solutions/observability/apps/configure-settings.md @@ -22,12 +22,10 @@ Uptime settings apply to the current space only. To segment different uptime use 2. At the top of the page, click **Settings**. ::::{important} - To modify items on this page, you must have the [`all`](../../../deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges.md) Uptime privilege granted to your role. + To modify items on this page, you must have the [`all`](/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges.md) Uptime privilege granted to your role. :::: - - ## Configure indices [configure-uptime-indices] Specify a comma-separated list of index patterns to match indices in {{es}} that contain {{heartbeat}} data. @@ -37,27 +35,24 @@ The pattern set here only restricts what the {{uptime-app}} displays. You can st :::: - :::{image} /solutions/images/observability-heartbeat-indices.png :alt: {{heartbeat}} indices :screenshot: ::: - ## Configure connectors [configure-uptime-alert-connectors] **Alerts** work by running checks on a schedule to detect conditions defined by a rule. When a condition is met, the rule tracks it as an **alert** and responds by triggering one or more **actions**. Actions typically involve interaction with {{kib}} services or third party integrations. **Connectors** allow actions to talk to these services and integrations. Click **Create connector** and follow the prompts to select a connector type and configure its properties. After you create a connector, it’s available to you anytime you set up a rule action in the current space. -For more information about each connector, see [action types and connectors](../../../deploy-manage/manage-connectors.md). +For more information about each connector, see [action types and connectors](/deploy-manage/manage-connectors.md). :::{image} /solutions/images/observability-alert-connector.png :alt: Rule connector :screenshot: ::: - ## Configure certificate thresholds [configure-cert-thresholds] You can modify certificate thresholds to control how Uptime displays your TLS values in the [TLS Certificates](uptime-monitoring-deprecated.md#view-certificate-status) page. These settings also determine which certificates are selected by any TLS rule you create. diff --git a/solutions/observability/apps/configure-synthetics-projects.md b/solutions/observability/apps/configure-synthetics-projects.md index 0eeee3772f..09cc2c4aa6 100644 --- a/solutions/observability/apps/configure-synthetics-projects.md +++ b/solutions/observability/apps/configure-synthetics-projects.md @@ -103,14 +103,11 @@ export default env => { :::: - -The configuration file can either export an object, or a function that when called should return the generated configuration. To know more about configuring the tests based on environments, look at the [dynamic configuration](../../../solutions/observability/apps/work-with-params-secrets.md#synthetics-dynamic-configs) documentation. - +The configuration file can either export an object, or a function that when called should return the generated configuration. To know more about configuring the tests based on environments, look at the [dynamic configuration](/solutions/observability/apps/work-with-params-secrets.md#synthetics-dynamic-configs) documentation. ## `params` [synthetics-configuration-params] -A JSON object that defines any variables your tests require. Read more in [Work with params and secrets](../../../solutions/observability/apps/work-with-params-secrets.md). - +A JSON object that defines any variables your tests require. Read more in [Work with params and secrets](/solutions/observability/apps/work-with-params-secrets.md). ## `playwrightOptions` [synthetics-configuration-playwright-options] @@ -121,10 +118,8 @@ Do not attempt to run in headful mode (using `headless:false`) when running thro :::: - Below are details on a few Playwright options that are particularly relevant to Elastic Synthetics including TLS client authentication, timeouts, timezones, and device emulation. - ### TLS client authentication [synthetics-configuration-playwright-options-client-certificates] To enable TLS client authentication, set the [`clientCertificates`](https://playwright.dev/docs/api/class-testoptions#test-options-client-certificates) option in the configuration file: @@ -134,7 +129,6 @@ Path-based options `{certPath, keyPath, pfxPath}` are only supported on Private :::: - ```js playwrightOptions: { clientCertificates: [ @@ -168,15 +162,12 @@ clientCertificates: [ :::: - - ### Timeouts [synthetics-configuration-playwright-options-timeouts] Playwright has two types of timeouts that are used in Elastic Synthetics: [action and navigation timeouts](https://playwright.dev/docs/test-timeouts#action-and-navigation-timeouts). Elastic Synthetics uses a default action and navigation timeout of 50 seconds. You can override this default using [`actionTimeout`](https://playwright.dev/docs/api/class-testoptions#test-options-action-timeout) and [`navigationTimeout`](https://playwright.dev/docs/api/class-testoptions#test-options-navigation-timeout) in `playwrightOptions`. - ### Timezones and locales [synthetics-configuration-playwright-options-timezones] The Elastic global managed testing infrastructure does not currently set the timezone. For {{private-location}}s, the monitors will use the timezone of the host machine running the {{agent}}. This is not always desirable if you want to test how a web application behaves across different timezones. To specify what timezone to use when the monitor runs, you can use `playwrightOptions` on a per monitor or global basis. @@ -190,8 +181,7 @@ playwrightOptions: { } ``` -To use a timezone and/or locale for a *specific* monitor, add these options to a journey using [`monitor.use`](../../../solutions/observability/apps/configure-individual-browser-monitors.md). - +To use a timezone and/or locale for a *specific* monitor, add these options to a journey using [`monitor.use`](/solutions/observability/apps/configure-individual-browser-monitors.md). ### Device emulation [synthetics-config-device-emulation] @@ -210,7 +200,6 @@ const config: SyntheticsConfig = { export default config; ``` - ## `project` [synthetics-configuration-project] Information about the Synthetics project. @@ -218,19 +207,17 @@ Information about the Synthetics project. `id` (`string`) : A unique id associated with your Synthetics project. It will be used for logically grouping monitors. - If you used [`init` to create a Synthetics project](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-init-command), this is the `` you specified. - + If you used [`init` to create a Synthetics project](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-init-command), this is the `` you specified. `url` (`string`) : The URL for the Serverless Observability project or the {{kib}} URL for the deployment to which you want to upload the monitors. space (`string`) -: For {{kib}} deployments 9.0 or higher, the identifier of the target [{{kib}} space](../../../deploy-manage/manage-spaces.md) for the pushed monitors. Spaces help you organize pushed monitors. Pushes to the "default" space if not specified. - +: For {{kib}} deployments 9.0 or higher, the identifier of the target [{{kib}} space](/deploy-manage/manage-spaces.md) for the pushed monitors. Spaces help you organize pushed monitors. Pushes to the "default" space if not specified. ## `monitor` [synthetics-configuration-monitor] -Default values to be applied to *all* monitors when using the [`@elastic/synthetics` `push` command](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). +Default values to be applied to *all* monitors when using the [`@elastic/synthetics` `push` command](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). `id` (`string`) : A unique identifier for this monitor. @@ -252,18 +239,16 @@ $$$synthetics-configuration-monitor-tags$$$ `tags` (`Array`) To list available locations you can: - * Run the [`elastic-synthetics locations` command](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command). - * Find `Synthetics` in the [global search field](../../../get-started/the-stack.md#kibana-navigation-search) and click **Create monitor**. Locations will be listed in *Locations*. - + * Run the [`elastic-synthetics locations` command](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command). + * Find `Synthetics` in the [global search field](/get-started/the-stack.md#kibana-navigation-search) and click **Create monitor**. Locations will be listed in *Locations*. `privateLocations` (`Array`) -: The [{{private-location}}s](../../../solutions/observability/apps/monitor-resources-on-private-networks.md) to which the monitors will be deployed. These {{private-location}}s refer to locations hosted and managed by you, whereas `locations` are hosted by Elastic. You can specify a {{private-location}} using the location’s name. +: The [{{private-location}}s](/solutions/observability/apps/monitor-resources-on-private-networks.md) to which the monitors will be deployed. These {{private-location}}s refer to locations hosted and managed by you, whereas `locations` are hosted by Elastic. You can specify a {{private-location}} using the location’s name. To list available {{private-location}}s you can: - * Run the [`elastic-synthetics locations` command](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command) with the URL for the Observability project or the {{kib}} URL for the deployment from which to fetch available locations. - * Find `Synthetics` in the [global search field](../../../get-started/the-stack.md#kibana-navigation-search) and click **Create monitor**. {{private-location}}s will be listed in *Locations*. - + * Run the [`elastic-synthetics locations` command](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command) with the URL for the Observability project or the {{kib}} URL for the deployment from which to fetch available locations. + * Find `Synthetics` in the [global search field](/get-started/the-stack.md#kibana-navigation-search) and click **Create monitor**. {{private-location}}s will be listed in *Locations*. `throttling` (`boolean` | [`ThrottlingOptions`](https://github.com/elastic/synthetics/blob/v1.3.0/src/common_types.ts#L194-L198)) : Control the monitor’s download speeds, upload speeds, and latency to simulate your application’s behavior on slower or laggier networks. Set to `false` to disable throttling altogether. @@ -272,7 +257,7 @@ $$$synthetics-configuration-monitor-tags$$$ `tags` (`Array`) : Control whether or not to capture screenshots. Options include `'on'`, `'off'`, or `'only-on-failure'`. `alert.status.enabled` (`boolean`) -: Enable or disable monitor status alerts. Read more about alerts in [Alerting](../../../solutions/observability/apps/configure-synthetics-settings.md#synthetics-settings-alerting). +: Enable or disable monitor status alerts. Read more about alerts in [Alerting](/solutions/observability/apps/configure-synthetics-settings.md#synthetics-settings-alerting). `retestOnFailure` (`boolean`) : Enable or disable retesting when a monitor fails. Default is `true`. @@ -281,9 +266,8 @@ $$$synthetics-configuration-monitor-tags$$$ `tags` (`Array`) Using `retestOnFailure` can reduce noise related to transient problems. - `fields` (`object`) -: A list of key-value pairs that will be sent with each monitor event. The `fields` are appended to {{es}} documents as `labels`, and those labels are displayed in {{kib}} in the *Monitor details* panel in the [individual monitor’s *Overview* tab](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-individual-monitors-overview). +: A list of key-value pairs that will be sent with each monitor event. The `fields` are appended to {{es}} documents as `labels`, and those labels are displayed in {{kib}} in the *Monitor details* panel in the [individual monitor’s *Overview* tab](/solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-individual-monitors-overview). For example: @@ -294,8 +278,7 @@ $$$synthetics-configuration-monitor-tags$$$ `tags` (`Array`) } ``` - For information on configuring monitors individually, refer to: -* [Configure individual browser monitors](../../../solutions/observability/apps/configure-individual-browser-monitors.md) for browser monitors -* [Configure lightweight monitors](../../../solutions/observability/apps/configure-lightweight-monitors.md) for lightweight monitors \ No newline at end of file +* [Configure individual browser monitors](/solutions/observability/apps/configure-individual-browser-monitors.md) for browser monitors +* [Configure lightweight monitors](/solutions/observability/apps/configure-lightweight-monitors.md) for lightweight monitors \ No newline at end of file diff --git a/solutions/observability/apps/configure-synthetics-settings.md b/solutions/observability/apps/configure-synthetics-settings.md index 50f0633fbf..89fa125b0b 100644 --- a/solutions/observability/apps/configure-synthetics-settings.md +++ b/solutions/observability/apps/configure-synthetics-settings.md @@ -11,7 +11,6 @@ applies_to: There are several Synthetics settings you can adjust in Observability. - ## Alerting [synthetics-settings-alerting] Alerting enables you to detect complex conditions using **rules** across Observability and send a notification using **connectors**. @@ -24,7 +23,6 @@ When you create a new synthetic monitor, new default synthetics rules will be ap * **Monitor status rule** for receiving notifications for errors and outages. * **TLS certificate rule** for receiving notifications when one or more of your HTTP or TCP lightweight monitors has a TLS certificate expiring within a specified threshold or when it exceeds an age limit. - However, the automatically created Synthetics internal alert is intentionally preconfigured, and some configuration options can’t be changed. For example, you can’t change how often it checks the rule. If you need specific alerting behavior, set up a different rule. To view all existing rules or create a new rule: @@ -42,71 +40,65 @@ On the *Rules* page, you can manage the default synthetics rules including snooz ::::{note} You can enable and disable default alerts for individual monitors in a few ways: -* In the Synthetics UI when you [create a monitor](../../../solutions/observability/apps/create-monitors-in-synthetics-app.md). +* In the Synthetics UI when you [create a monitor](/solutions/observability/apps/create-monitors-in-synthetics-app.md). * In the Synthetics UI *after* a monitor is already created, on the **Monitors** page or on the **Edit monitor** page for the monitor. -* In Synthetics projects when [configuring a lightweight monitor](../../../solutions/observability/apps/configure-lightweight-monitors.md). +* In Synthetics projects when [configuring a lightweight monitor](/solutions/observability/apps/configure-lightweight-monitors.md). :::: - -In the **Alerting** tab on the Synthetics Settings page, you can add and configure connectors. If you are running in Elastic Cloud, then an SMTP connector will automatically be configured, allowing you to easily set up email alerts. Read more about all available connectors in [Action types](../../../solutions/observability/incident-management/create-an-apm-anomaly-rule.md). +In the **Alerting** tab on the Synthetics Settings page, you can add and configure connectors. If you are running in Elastic Cloud, then an SMTP connector will automatically be configured, allowing you to easily set up email alerts. Read more about all available connectors in [Action types](/solutions/observability/incident-management/create-an-apm-anomaly-rule.md). :::{image} /solutions/images/observability-synthetics-settings-alerting.png :alt: Alerting tab on the Synthetics Settings page in {kib} :screenshot: ::: - ## {{private-location}}s [synthetics-settings-private-locations] {{private-location}}s allow you to run monitors from your own premises. -In the **{{private-location}}s** tab, you can add and manage {{private-location}}s. After you [Set up {{fleet-server}} and {{agent}}](../../../solutions/observability/apps/monitor-resources-on-private-networks.md#synthetics-private-location-fleet-agent) and [Connect to the {{stack}} or your serverless Observability project](../../../solutions/observability/apps/monitor-resources-on-private-networks.md#synthetics-private-location-connect), this is where you will add the {{private-location}} so you can specify it as the location for a monitor created using the Synthetics UI or a Synthetics project. +In the **{{private-location}}s** tab, you can add and manage {{private-location}}s. After you [Set up {{fleet-server}} and {{agent}}](/solutions/observability/apps/monitor-resources-on-private-networks.md#synthetics-private-location-fleet-agent) and [Connect to the {{stack}} or your serverless Observability project](/solutions/observability/apps/monitor-resources-on-private-networks.md#synthetics-private-location-connect), this is where you will add the {{private-location}} so you can specify it as the location for a monitor created using the Synthetics UI or a Synthetics project. :::{image} /solutions/images/observability-synthetics-settings-private-locations.png :alt: {{private-location}}s tab on the Synthetics Settings page in {kib} :screenshot: ::: - ## Global parameters [synthetics-settings-global-parameters] Global parameters can be defined once and used across the configuration of lightweight and browser-based monitors. -In the **Global parameters** tab, you can define variables and parameters. This is one of several methods you can use to define variables and parameters. To learn more about the other methods and which methods take precedence over others, see [Work with params and secrets](../../../solutions/observability/apps/work-with-params-secrets.md). +In the **Global parameters** tab, you can define variables and parameters. This is one of several methods you can use to define variables and parameters. To learn more about the other methods and which methods take precedence over others, see [Work with params and secrets](/solutions/observability/apps/work-with-params-secrets.md). :::{image} /solutions/images/observability-synthetics-settings-global-parameters.png :alt: Global parameters tab on the Synthetics Settings page in {kib} :screenshot: ::: - ## Data retention [synthetics-settings-data-retention] -When you set up a synthetic monitor, data from the monitor is saved in [Elasticsearch data streams](../../../manage-data/data-store/data-streams.md), an append-only structure in Elasticsearch. You can customize how long synthetics data is stored by creating your own index lifecycle policy and attaching it to the relevant custom Component Template in Stack Management. +When you set up a synthetic monitor, data from the monitor is saved in [Elasticsearch data streams](/manage-data/data-store/data-streams.md), an append-only structure in Elasticsearch. You can customize how long synthetics data is stored by creating your own index lifecycle policy and attaching it to the relevant custom Component Template in Stack Management. -In the **Data retention** tab, use the links to jump to the relevant policy for each data stream. Learn more about the data included in each data stream in [Manage data retention](../../../solutions/observability/apps/manage-data-retention.md). +In the **Data retention** tab, use the links to jump to the relevant policy for each data stream. Learn more about the data included in each data stream in [Manage data retention](/solutions/observability/apps/manage-data-retention.md). :::{image} /solutions/images/observability-synthetics-settings-data-retention.png :alt: Data retention tab on the Synthetics Settings page in {kib} :screenshot: ::: - ## Project API keys [synthetics-settings-api-keys] Project API keys are used to push {{project-monitors}} remotely from a CLI or CD pipeline. -In the **Project API keys** tab, you can generate project API keys to use with your projects. Learn more about using API keys in [Use {{project-monitors-cap}}](../../../solutions/observability/apps/create-monitors-with-project-monitors.md). +In the **Project API keys** tab, you can generate project API keys to use with your projects. Learn more about using API keys in [Use {{project-monitors-cap}}](/solutions/observability/apps/create-monitors-with-project-monitors.md). ::::{important} -**In an Elastic Stack deployment**, to create a Project API key you must be logged into {{kib}} as a user with the privileges described in [Writer role](../../../solutions/observability/apps/writer-role.md). +**In an Elastic Stack deployment**, to create a Project API key you must be logged into {{kib}} as a user with the privileges described in [Writer role](/solutions/observability/apps/writer-role.md). -In a serverless project, to create a Project API key you must be logged in as a user with [Editor](../../../solutions/observability/apps/grant-users-access-to-secured-resources.md) access. +In a serverless project, to create a Project API key you must be logged in as a user with [Editor](/solutions/observability/apps/grant-users-access-to-secured-resources.md) access. :::: - :::{image} /solutions/images/observability-synthetics-settings-api-keys.png :alt: Project API keys tab on the Synthetics Settings page in {kib} :screenshot: diff --git a/solutions/observability/apps/control-access-to-apm-data.md b/solutions/observability/apps/control-access-to-apm-data.md index 550fa9bfde..9896a9b2a8 100644 --- a/solutions/observability/apps/control-access-to-apm-data.md +++ b/solutions/observability/apps/control-access-to-apm-data.md @@ -7,13 +7,12 @@ applies_to: # Control access to APM data [apm-spaces] -Starting in version 8.2.0, the Applications UI is [Kibana space](../../../deploy-manage/manage-spaces.md) aware. This allows you to separate your data—​and access to that data—​by team, use case, service environment, or any other filter that you choose. +Starting in version 8.2.0, the Applications UI is [Kibana space](/deploy-manage/manage-spaces.md) aware. This allows you to separate your data—​and access to that data—​by team, use case, service environment, or any other filter that you choose. To take advantage of this feature, your APM data needs to be written to different data streams. One way to accomplish this is with different namespaces. For example, you can send production data to an APM integration with a namespace of `production`, while sending staging data to a different APM integration with a namespace of `staging`. Multiple APM integration instances is not required though. The simplest way to take advantage of this feature is by creating filtered aliases. See the guide below for more information. - ## Guide: Separate staging and production data [apm-spaces-example] This guide will explain how to separate your staging and production data. This can be helpful to either remove noise when troubleshooting a production issue, or to create more granular access control for certain data. @@ -23,7 +22,6 @@ This guide assumes that you: * Are sending both staging and production APM data to an {{es}} cluster. * Have configured the `environment` variable in your APM agent configurations. This variable sets the `service.environment` field in APM documents. You should have documents where `service.environment: production` and `service.environment: staging`. If this field is empty, see [service environment filter](filter-application-data.md#apm-filter-your-data-service-environment-filter) to learn how to set this value. - ### Step 1: Create filtered aliases [_step_1_create_filtered_aliases] The Applications UI uses index patterns to query your APM data. An index pattern can match data streams, indices, and/or aliases. The default values are: @@ -38,7 +36,6 @@ The Applications UI uses index patterns to query your APM data. An index pattern The default index settings also query the `apm-*` data view. This data view matches APM data shipped in earlier versions of APM (prior to v8.0). :::: - Instead of querying the default APM data views, we can create filtered aliases for the Applications UI to query. A filtered alias is a secondary name for a group of data streams that has a user-defined filter to limit the documents that the alias can access. To separate `staging` and `production` APM data, we’d need to create six filtered aliases—​three aliases for each service environment: @@ -53,7 +50,7 @@ The `production--apm` aliases will contain a filter that only provides ac To create these six filtered aliases, use the {{es}} [Aliases API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-update-aliases). In {{kib}}, open **Dev Tools** and run the following POST requests. -::::{dropdown} `traces-apm*` production alias example +::::{dropdown} traces-apm* production alias example ```console POST /_aliases?pretty { @@ -79,11 +76,9 @@ POST /_aliases?pretty 2. The alias must not match the default APM index (`traces-apm*,apm-*`) 3. Only match documents where `service.environment: production` - :::: - -::::{dropdown} `logs-apm*` production alias example +::::{dropdown} logs-apm* production alias example ```console POST /_aliases?pretty { @@ -109,11 +104,9 @@ POST /_aliases?pretty 2. The alias must not match the default APM index (`logs-apm*,apm-*`) 3. Only match documents where `service.environment: production` - :::: - -::::{dropdown} `metrics-apm*` production alias example +::::{dropdown} metrics-apm* production alias example ```console POST /_aliases?pretty { @@ -139,11 +132,9 @@ POST /_aliases?pretty 2. The alias must not match the default APM index (`metrics-apm*,apm-*`) 3. Only match documents where `service.environment: production` - :::: - -::::{dropdown} `traces-apm*` staging alias example +::::{dropdown} traces-apm* staging alias example ```console POST /_aliases?pretty { @@ -169,11 +160,9 @@ POST /_aliases?pretty 2. The alias must not match the default APM index (`traces-apm*,apm-*`) 3. Only match documents where `service.environment: staging` - :::: - -::::{dropdown} `logs-apm*` staging alias example +::::{dropdown} logs-apm* staging alias example ```console POST /_aliases?pretty { @@ -199,11 +188,9 @@ POST /_aliases?pretty 2. The alias must not match the default APM index (`logs-apm*,apm-*`) 3. Only match documents where `service.environment: staging` - :::: - -::::{dropdown} `metrics-apm*` staging alias example +::::{dropdown} metrics-apm* staging alias example ```console POST /_aliases?pretty { @@ -229,17 +216,13 @@ POST /_aliases?pretty 2. The alias must not match the default APM index (`metrics-apm*,apm-*`) 3. Only match documents where `service.environment: staging` - :::: - - ### Step 2: Create {{kib}} spaces [_step_2_create_kib_spaces] Next, you’ll need to create a {{kib}} space for each service environment. To open **Spaces**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). To create a new space, click **Create a space**. For this guide, we’ve created two Kibana spaces, one named `production` and one named `staging`. -See [Kibana spaces](../../../deploy-manage/manage-spaces.md) for more information on creating a space. - +See [Kibana spaces](/deploy-manage/manage-spaces.md) for more information on creating a space. ### Step 3: Update APM index settings in each space [_step_3_update_apm_index_settings_in_each_space] @@ -254,7 +237,6 @@ Open the Applications UI and navigate to **Settings** → **Indices**. Use the t | Transaction indices | `production-traces-apm` | `staging-traces-apm` | | Metrics indices | `production-metrics-apm` | `staging-metrics-apm` | - ### Step 4: Create {{kib}} access roles [_step_4_create_kib_access_roles] To open **Roles**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). Click **Create role**. @@ -275,7 +257,7 @@ Using the table below, assign each role the following privileges: Alternatively, you can use the {{es}} [Create or update roles API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role): -::::{dropdown} Create a `production_apm_viewer` role +::::{dropdown} Create a production_apm_viewer role This request creates a `production_apm_viewer` role: ```console @@ -306,11 +288,9 @@ POST /_security/role/production_apm_viewer 2. Assigns `read` privileges for the Applications and User Experience UIs. 3. Provides access to the space named `production`. - :::: - -::::{dropdown} Create a `staging_apm_viewer` role +::::{dropdown} Create a staging_apm_viewer role This request creates a `staging_apm_viewer` role: ```console @@ -341,16 +321,13 @@ POST /_security/role/staging_apm_viewer 2. Assigns `read` privileges for the Applications and User Experience UIs. 3. Provides access to the space named `staging`. - :::: - - ### Step 5: Assign users to roles [_step_5_assign_users_to_roles] The last thing to do is assign users to the newly created roles above. Users will only have access to the data within the spaces that they are granted. -For information on how to create users and assign them roles with the {{kib}} UI, see [Securing access to Kibana](../../../deploy-manage/users-roles/cluster-or-deployment-auth/quickstart.md). +For information on how to create users and assign them roles with the {{kib}} UI, see [Securing access to Kibana](/deploy-manage/users-roles/cluster-or-deployment-auth/quickstart.md). Alternatively, you can use the {{es}} [Create or update users API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-user). @@ -368,7 +345,6 @@ POST /_security/user/production-apm-user 1. Assigns the previously created `production_apm_viewer` role. - This example creates a new user and assigns them the `staging_apm_viewer` role created in the previous step. This user will only have access to the staging space and data with a `service.environment` of `staging`. Remember to change the `password`, `full_name`, and `email` fields. ```console @@ -383,8 +359,6 @@ POST /_security/user/staging-apm-user 1. Assigns the previously created `staging_apm_viewer` role. - - ### Step 6: Marvel [_step_6_marvel] That’s it! Head back to the Applications UI and marvel at your space-specific data. diff --git a/solutions/observability/apps/create-apm-rules-alerts.md b/solutions/observability/apps/create-apm-rules-alerts.md index 3756d7cd8d..eea5e1fcb1 100644 --- a/solutions/observability/apps/create-apm-rules-alerts.md +++ b/solutions/observability/apps/create-apm-rules-alerts.md @@ -10,23 +10,21 @@ applies_to: # Create APM rules and alerts [apm-alerts] - The Applications UI allows you to define **rules** to detect complex conditions within your APM data and trigger built-in **actions** when those conditions are met. - ## APM rules [apm_rules] The following APM rules are supported: | | | | --- | --- | -| **APM Anomaly** | Alert when either the latency, throughput, or failed transaction rate of a service is anomalous.Anomaly rules can be set at the environment level, service level, and/or transaction type level. Read more in [APM Anomaly rule →](../../../solutions/observability/incident-management/create-an-apm-anomaly-rule.md) | -| **Error count threshold** | Alert when the number of errors in a service exceeds a defined threshold. Error count rules can be set at theenvironment level, service level, and error group level. Read more in [Error count threshold rule →](../../../solutions/observability/incident-management/create-an-error-count-threshold-rule.md) | -| **Failed transaction rate threshold** | Alert when the rate of transaction errors in a service exceeds a defined threshold. Read more in [Failed transaction rate threshold rule →](../../../solutions/observability/incident-management/create-failed-transaction-rate-threshold-rule.md) | -| **Latency threshold** | Alert when the latency or failed transaction rate is abnormal.Threshold rules can be as broad or as granular as you’d like, enabling you to define exactly when you want to be alerted—​whether that’s at the environment level, service name level, transaction type level, and/or transaction name level. Read more in [Latency threshold rule →](../../../solutions/observability/incident-management/create-latency-threshold-rule.md) | +| **APM Anomaly** | Alert when either the latency, throughput, or failed transaction rate of a service is anomalous.Anomaly rules can be set at the environment level, service level, and/or transaction type level. Read more in [APM Anomaly rule →](/solutions/observability/incident-management/create-an-apm-anomaly-rule.md) | +| **Error count threshold** | Alert when the number of errors in a service exceeds a defined threshold. Error count rules can be set at theenvironment level, service level, and error group level. Read more in [Error count threshold rule →](/solutions/observability/incident-management/create-an-error-count-threshold-rule.md) | +| **Failed transaction rate threshold** | Alert when the rate of transaction errors in a service exceeds a defined threshold. Read more in [Failed transaction rate threshold rule →](/solutions/observability/incident-management/create-failed-transaction-rate-threshold-rule.md) | +| **Latency threshold** | Alert when the latency or failed transaction rate is abnormal.Threshold rules can be as broad or as granular as you’d like, enabling you to define exactly when you want to be alerted—​whether that’s at the environment level, service name level, transaction type level, and/or transaction name level. Read more in [Latency threshold rule →](/solutions/observability/incident-management/create-latency-threshold-rule.md) | ::::{tip} -For a complete walkthrough of the **Create rule** flyout panel, including detailed information on each configurable property, see Kibana’s [Create and manage rules](../../../explore-analyze/alerts-cases/alerts/create-manage-rules.md). +For a complete walkthrough of the **Create rule** flyout panel, including detailed information on each configurable property, see Kibana’s [Create and manage rules](/explore-analyze/alerts-cases/alerts/create-manage-rules.md). :::: @@ -37,22 +35,19 @@ stack: all View and manage rules and alerts in the Applications UI. - ### View active alerts [apm-alert-view-active] Active alerts are displayed and grouped in multiple ways in the Applications UI. - #### View alerts by service group [apm-alert-view-group] -If you’re using the [service groups](../../../solutions/observability/apps/services.md#service-groups) feature, you can view alerts by service group. From the service group overview page, click the red alert indicator to open the **Alerts** tab with a predefined filter that matches the filter used when creating the service group. +If you’re using the [service groups](/solutions/observability/apps/services.md#service-groups) feature, you can view alerts by service group. From the service group overview page, click the red alert indicator to open the **Alerts** tab with a predefined filter that matches the filter used when creating the service group. :::{image} /solutions/images/observability-apm-service-group.png :alt: Example view of service group in the Applications UI in Kibana :screenshot: ::: - #### View alerts by service [apm-alert-view-service] Alerts can be viewed within the context of any service. After selecting a service, go to the **Alerts** tab to view any alerts that are active for the selected service. @@ -62,16 +57,14 @@ Alerts can be viewed within the context of any service. After selecting a servic :screenshot: ::: - ### Manage alerts and rules [apm-alert-manage] From the Applications UI, select **Alerts and rules** → **Manage rules** to be taken to the {{kib}} **{{rules-ui}}** page. From this page, you can disable, mute, and delete APM alerts. - ### More information [apm-alert-more-info] -See [Alerting](../../../explore-analyze/alerts-cases.md) for more information. +See [Alerting](/explore-analyze/alerts-cases.md) for more information. ::::{note} -If you are using an **on-premise** Elastic Stack deployment with security, communication between Elasticsearch and Kibana must have TLS configured. More information is in the alerting [prerequisites](../../../explore-analyze/alerts-cases/alerts/alerting-setup.md#alerting-prerequisites). +If you are using an **on-premise** Elastic Stack deployment with security, communication between Elasticsearch and Kibana must have TLS configured. More information is in the alerting [prerequisites](/explore-analyze/alerts-cases/alerts/alerting-setup.md#alerting-prerequisites). :::: \ No newline at end of file diff --git a/solutions/observability/apps/create-assign-feature-roles-to-apm-server-users.md b/solutions/observability/apps/create-assign-feature-roles-to-apm-server-users.md index f13a4efadb..41f25bb749 100644 --- a/solutions/observability/apps/create-assign-feature-roles-to-apm-server-users.md +++ b/solutions/observability/apps/create-assign-feature-roles-to-apm-server-users.md @@ -6,17 +6,13 @@ applies_to: stack: --- - - # Create and assign feature roles to APM Server users [apm-feature-roles] - ::::{note} Kibana custom roles are *not* compatible with [{{serverless-full}}](https://docs.elastic.co/serverless). :::: - Manage access on a feature-by-feature basis by creating several custom feature-related *roles* and assigning one or more of these roles to each *user or group* based on which features they need to access. ::::{tip} @@ -30,7 +26,6 @@ In general, there are three types of privileges you’ll work with when creating :::: - The following are common roles that APM Server users might need: * [**Writer role**](#apm-privileges-to-publish-events): Allows a user to publish events collected by APM Server, which is **required** to write to {{es}}. @@ -47,8 +42,6 @@ If you want to create an APM Server user who can use the Elastic APM Real User M :::: - - ## Create a *writer* role [apm-privileges-to-publish-events] APM users that publish events to {{es}} *must* have privileges to write to APM data streams. @@ -58,7 +51,6 @@ This is not needed when APM Server doesn’t write to {{es}} directly. For examp :::: - To grant an APM Server user the required privileges for writing events to {{es}}: 1. Create a **general writer role**, called something like `apm_writer`, that has the following privileges: @@ -69,12 +61,10 @@ To grant an APM Server user the required privileges for writing events to {{es}} | Index | `create_doc` on `traces-apm*`, `logs-apm*`, and `metrics-apm*` indices | Write events into {{es}} | | Cluster | `monitor` | Allows cluster UUID checks, which are performed as part of APM server startup preconditionsif [Elasticsearch security](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md) is enabled (it is enabled by default), and allows a license check, which is required if [tail-based sampling](transaction-sampling.md#apm-tail-based-sampling) is enabled. | - ::::{note} If you have explicitly disabled Elastic security *and* you are *not* using tail-based sampling, the `monitor` privilege may not be necessary. :::: - 1. Assign the **general writer role** to APM Server users who need to publish APM data. ::::{note} @@ -82,8 +72,6 @@ Assign additional APM feature roles to users as needed including the *Central co :::: - - ## Create a *central configuration management* role [apm-privileges-agent-central-config] ::::{important} @@ -93,7 +81,6 @@ The privileges included in this role are **required** for all users when [centra :::: - $$$apm-privileges-agent-central-config-server$$$ APM Server acts as a proxy between your APM agents and the Applications UI. The Applications UI communicates any changed settings to APM Server so that your agents only need to poll the Server to determine which central configuration settings have changed. @@ -124,13 +111,10 @@ Assign additional APM feature roles to users as needed including the *Writer rol :::: - ::::{tip} Looking for privileges and roles needed to use central configuration from the Applications UI or APM UI API? See [Applications UI central configuration user](applications-ui-central-config-user.md). :::: - - ## Create a *monitoring* role [apm-privileges-to-publish-monitoring] {{es-security-features}} provides built-in users and roles for publishing and viewing monitoring data. The privileges and roles needed to publish monitoring data depend on the method used to collect that data. @@ -142,7 +126,6 @@ Looking for privileges and roles needed to use central configuration from the Ap * [View monitoring data](#apm-privileges-to-publish-monitoring-view) - ### Publish monitoring data [apm-privileges-to-publish-monitoring-write] ::::{important} @@ -150,8 +133,6 @@ Looking for privileges and roles needed to use central configuration from the Ap :::: - - #### Internal collection [apm-privileges-to-publish-monitoring-internal] If you’re using [internal collection](use-internal-collection-to-send-monitoring-data.md) to collect metrics about APM Server, either: @@ -161,7 +142,7 @@ If you’re using [internal collection](use-internal-collection-to-send-monitori **Use a built-in user or role** -{{es-security-features}} provides the `apm_system` [built-in user](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md) and `apm_system` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) to send monitoring information. You can use the built-in user, if it’s available in your environment, create a user who has the built-in role assigned, or create a user and manually assign the privileges needed to send monitoring information. +{{es-security-features}} provides the `apm_system` [built-in user](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md) and `apm_system` [built-in role](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) to send monitoring information. You can use the built-in user, if it’s available in your environment, create a user who has the built-in role assigned, or create a user and manually assign the privileges needed to send monitoring information. If you use the built-in `apm_system` user, make sure you set the password before using it. @@ -183,15 +164,12 @@ Assign additional APM feature roles to users as needed including the [*Writer ro :::: - - #### {{metricbeat}} collection [apm-privileges-to-publish-monitoring-metricbeat] ::::{note} When using {{metricbeat}} to collect metrics, no roles or users need to be created with APM Server. See [Use {{metricbeat}} collection](use-metricbeat-to-send-monitoring-data.md) for complete details on setting up {{metricbeat}} collection. :::: - If you’re [using {{metricbeat}}](use-metricbeat-to-send-monitoring-data.md) to collect metrics about APM Server, you can either: * Use the built-in `remote_monitoring_user` user or role @@ -199,7 +177,7 @@ If you’re [using {{metricbeat}}](use-metricbeat-to-send-monitoring-data.md) to **Use a built-in user or role** -{{es-security-features}} provides the `remote_monitoring_user` [built-in user](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md), and the `remote_monitoring_collector` and `remote_monitoring_agent` [built-in roles](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) for collecting and sending monitoring information. You can use the built-in user, if it’s available in your environment, or create a user who has the privileges needed to collect and send monitoring information. +{{es-security-features}} provides the `remote_monitoring_user` [built-in user](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md), and the `remote_monitoring_collector` and `remote_monitoring_agent` [built-in roles](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) for collecting and sending monitoring information. You can use the built-in user, if it’s available in your environment, or create a user who has the privileges needed to collect and send monitoring information. If you use the built-in `remote_monitoring_user` user, make sure you set the password before using it. @@ -214,14 +192,11 @@ If you don’t use the `remote_monitoring_user` user, you can create a custom us | `remote_monitoring_collector` | Collect monitoring metrics from APM Server | | `remote_monitoring_agent` | Send monitoring data to the monitoring cluster | - ::::{note} Assign additional APM feature roles to users as needed including the [*Writer role*](#apm-privileges-to-publish-events) and [*Central configuration management role*](#apm-central-config-role-note), both of which are required in most cases. :::: - - ### View monitoring data [apm-privileges-to-publish-monitoring-view] To grant users the required privileges for viewing monitoring data: @@ -239,14 +214,11 @@ To grant users the required privileges for viewing monitoring data: | --- | --- | | `monitoring_user` | Grants access to monitoring indices for APM Server | - ::::{note} Assign additional APM feature roles to users as needed including the [*Writer role*](#apm-privileges-to-publish-events) and [*Central configuration management role*](#apm-central-config-role-note), both of which are required in most cases. :::: - - ## Create a *source map* role [apm-privileges-rum-source-map] $$$apm-privileges-rum-source-mapping$$$ @@ -263,5 +235,4 @@ Assign additional APM feature roles to users as needed including the [*Writer ro :::: - The previous privileges should be sufficient for RUM source mapping to work properly as long as APM Server communicates with {{es}} successfully. If it fails, it may fallback to read source maps through {{kib}} if configured, which requires additional {{kib}} privileges. For more details, refer to the [{{stack}}](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-apm-sourcemaps) or [{{serverless-short}}](https://www.elastic.co/docs/api/doc/serverless/group/endpoint-apm-sourcemaps) API documentation. \ No newline at end of file diff --git a/solutions/observability/apps/create-custom-links.md b/solutions/observability/apps/create-custom-links.md index eddf88c559..d9042663d6 100644 --- a/solutions/observability/apps/create-custom-links.md +++ b/solutions/observability/apps/create-custom-links.md @@ -12,23 +12,20 @@ applies_to: ::::{note} -**For serverless Observability projects**, the **Editor** role or higher is required to create and manage custom links. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). +**For serverless Observability projects**, the **Editor** role or higher is required to create and manage custom links. To learn more, refer to [Assign user roles and privileges](/deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). :::: - Elastic’s custom link feature allows you to easily create up to 500 dynamic links based on your specific APM data. Custom links can be filtered to only appear in the Applications UI for relevant services, environments, transaction types, or transaction names. Ready to dive in? Jump straight to the [examples](#custom-links-examples). - ## Create a link [custom-links-create] Each custom link consists of a label, URL, and optional filter. The easiest way to create a custom link is from within the actions dropdown in the transaction detail page. This method will automatically apply filters, scoping the link to that specific service, environment, transaction type, and transaction name. Alternatively, you can create a custom link in the Applications UI by navigating to **Settings** > **Customize UI**, and selecting **Create custom link**. - ### Label [custom-links-label] The name of your custom link. The actions context menu displays this text, so keep it as short as possible. @@ -37,8 +34,6 @@ The name of your custom link. The actions context menu displays this text, so ke Custom links are displayed alphabetically in the actions menu. :::: - - ### URL [custom-links-url] The URL your link points to. URLs support dynamic field name variables, encapsulated in double curly brackets: `{{field.name}}`. These variables will be replaced with transaction metadata when the link is clicked. @@ -50,7 +45,6 @@ Because everyone’s data is different, you’ll need to examine your traces to :screenshot: ::: - ### Filters [custom-links-filters] Filter each link to only appear for specific services or transactions. You can filter on the following fields: @@ -62,12 +56,10 @@ Filter each link to only appear for specific services or transactions. You can f Multiple values are allowed when comma-separated. - ## Custom link examples [custom-links-examples] Not sure where to start with custom links? Take a look at the examples below and customize them to your liking! - ### Email [custom-links-examples-email] Email the owner of a service. @@ -88,7 +80,6 @@ This link opens an email addressed to the team or owner of `python-backend`. It | Link | `mailto:python_team@elastic.co` | | Filters | `service.name:python-backend` | - ### GitHub issue [custom-links-examples-gh] Open a GitHub issue with pre-populated metadata from the selected trace sample. @@ -116,7 +107,6 @@ This link opens a new GitHub issue in the apm-agent-rum repository. It populates See the [GitHub automation documentation](https://help.github.com/en/github/managing-your-work-on-github/about-automation-for-issues-and-pull-requests-with-query-parameters) for a full list of supported query parameters. - ### Jira task [custom-links-examples-jira] Create a Jira task with pre-populated metadata from the selected trace sample. @@ -142,7 +132,6 @@ This link creates a new task on the Engineering board in Jira. It populates the See the [Jira application administration knowledge base](https://confluence.atlassian.com/jirakb/how-to-create-issues-using-direct-html-links-in-jira-server-159474.html) for a full list of supported query parameters. - ### Dashboards [custom-links-examples-kib] Link to a custom dashboard. @@ -161,7 +150,6 @@ This link opens the current `transaction.id` in a custom dashboard. There are no | Label | `Open transaction in Python drilldown viz` | | URL | `https://kibana-instance/app/kibana#/dashboard?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-24h,to:now))&_a=(description:'',filters:!(),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!t),panels:!((embeddableConfig:(),gridData:(h:15,i:cb79c1c0-1af8-472c-aaf7-d158a76946fb,w:24,x:0,y:0),id:c8c74b20-6a30-11ea-92ab-b5d3feff11df,panelIndex:cb79c1c0-1af8-472c-aaf7-d158a76946fb,type:visualization,version:'7.7')),query:(language:kuery,query:'transaction.id:{{transaction.id}}'),timeRestore:!f,title:'',viewMode:edit)` | - ### Slack channel [custom-links-examples-slack] Open a specified slack channel. @@ -182,7 +170,6 @@ This link opens a company slack channel, #apm-user-support. It only appears when | Link | `https://COMPANY_SLACK.slack.com/archives/efk52kt23k` | | Filters | `transaction.name:GET user/login` | - ### Website [custom-links-examples-web] Open an internal or external website. diff --git a/solutions/observability/apps/create-monitors-in-synthetics-app.md b/solutions/observability/apps/create-monitors-in-synthetics-app.md index 20032ad0cc..caaa5261cf 100644 --- a/solutions/observability/apps/create-monitors-in-synthetics-app.md +++ b/solutions/observability/apps/create-monitors-in-synthetics-app.md @@ -16,23 +16,21 @@ You can create synthetic monitors directly in the UI by opening an Observability :alt: Diagram showing which pieces of software are used to configure monitors ::: -This is one of [two approaches](../../../solutions/observability/apps/get-started.md) you can use to set up a synthetic monitor. - +This is one of [two approaches](/solutions/observability/apps/get-started.md) you can use to set up a synthetic monitor. ## Prerequisites [synthetics-get-started-ui-prerequisites] -For **serverless Observability projects**, you must be signed in as a user with [Editor](../../../solutions/observability/apps/grant-users-access-to-secured-resources.md) access. +For **serverless Observability projects**, you must be signed in as a user with [Editor](/solutions/observability/apps/grant-users-access-to-secured-resources.md) access. -For **Elastic Stack deployments**, you must be signed into {{kib}} as a user with at least [synthetics write permissions](../../../solutions/observability/apps/writer-role.md), and Monitor Management must be enabled by an administrator as described in [Setup role](../../../solutions/observability/apps/setup-role.md). +For **Elastic Stack deployments**, you must be signed into {{kib}} as a user with at least [synthetics write permissions](/solutions/observability/apps/writer-role.md), and Monitor Management must be enabled by an administrator as described in [Setup role](/solutions/observability/apps/setup-role.md). You should decide where you want to run the monitors before getting started. You can run monitors on one or both of the following: * **Elastic’s global managed testing infrastructure**: With Elastic’s global managed testing infrastructure, you can create and run monitors in multiple locations without having to manage your own infrastructure. Elastic takes care of software updates and capacity planning for you. -* **{{private-location}}s**: {{private-location}}s allow you to run monitors from your own premises. To use {{private-location}}s you must create a {{private-location}} before continuing. For step-by-step instructions, refer to [Monitor resources on private networks](../../../solutions/observability/apps/monitor-resources-on-private-networks.md). +* **{{private-location}}s**: {{private-location}}s allow you to run monitors from your own premises. To use {{private-location}}s you must create a {{private-location}} before continuing. For step-by-step instructions, refer to [Monitor resources on private networks](/solutions/observability/apps/monitor-resources-on-private-networks.md). Executing synthetic tests on Elastic’s global managed testing infrastructure incurs an additional charge. Tests are charged under one of two new billing dimensions depending on the monitor type. For *browser monitor* usage, there is a fee per test run. For *lightweight monitor* usage, there is a fee per region in which you run any monitors regardless of the number of test runs. For more details, refer to the [{{obs-serverless}} pricing page](https://www.elastic.co/pricing/serverless-observability). - ## Add a lightweight monitor [synthetics-get-started-ui-add-a-lightweight-monitor] To use the UI to add a lightweight monitor: @@ -43,13 +41,12 @@ To use the UI to add a lightweight monitor: 4. In *Locations*, select one or more locations. ::::{note} - If you don’t see any locations listed, refer to the [troubleshooting guide](../../../troubleshoot/observability/troubleshooting-synthetics.md#synthetics-troubleshooting-no-locations) for guidance. + If you don’t see any locations listed, refer to the [troubleshooting guide](/troubleshoot/observability/troubleshooting-synthetics.md#synthetics-troubleshooting-no-locations) for guidance. :::: - :::::{note} - If you’ve [added a {{private-location}}](../../../solutions/observability/apps/monitor-resources-on-private-networks.md), you’ll see your the {{private-location}} in the list of *Locations*. + If you’ve [added a {{private-location}}](/solutions/observability/apps/monitor-resources-on-private-networks.md), you’ll see your the {{private-location}} in the list of *Locations*. :::{image} /solutions/images/serverless-private-locations-monitor-locations.png :alt: Screenshot of Monitor locations options including a {private-location} @@ -68,15 +65,13 @@ To use the UI to add a lightweight monitor: :screenshot: ::: - - ## Add a browser monitor [synthetics-get-started-ui-add-a-browser-monitor] You can also create a browser monitor in the UI using an **Inline script**. An inline script contains a single journey that you manage individually. Inline scripts can be quick to set up, but can also be more difficult to manage. Each browser monitor configured using an inline script can contain only *one* journey, which must be maintained directly in the UI. -If you depend on external packages, have your journeys next to your code repository, or want to embed and manage more than one journey from a single monitor configuration, use a [Synthetics project](../../../solutions/observability/apps/create-monitors-with-project-monitors.md) instead. +If you depend on external packages, have your journeys next to your code repository, or want to embed and manage more than one journey from a single monitor configuration, use a [Synthetics project](/solutions/observability/apps/create-monitors-with-project-monitors.md) instead. To use the UI to add a browser monitor: @@ -85,7 +80,7 @@ To use the UI to add a browser monitor: 3. In *Locations*, select one or more locations. ::::{note} - If you don’t see any locations listed, refer to the [troubleshooting guide](../../../troubleshoot/observability/troubleshooting-synthetics.md#synthetics-troubleshooting-no-locations) for guidance. + If you don’t see any locations listed, refer to the [troubleshooting guide](/troubleshoot/observability/troubleshooting-synthetics.md#synthetics-troubleshooting-no-locations) for guidance. :::: @@ -98,35 +93,33 @@ To use the UI to add a browser monitor: ::: ::::{note} - Alternatively, you can use the **Script recorder** option. You can use the Elastic Synthetics Recorder to interact with a web page, export journey code that reflects all the actions you took, and upload the results in the UI. For more information, refer to [Use the Synthetics Recorder](../../../solutions/observability/apps/use-synthetics-recorder.md). + Alternatively, you can use the **Script recorder** option. You can use the Elastic Synthetics Recorder to interact with a web page, export journey code that reflects all the actions you took, and upload the results in the UI. For more information, refer to [Use the Synthetics Recorder](/solutions/observability/apps/use-synthetics-recorder.md). :::: 6. Click **Advanced options** to see more ways to configure your monitor. * Use **Data options** to add context to the data coming from your monitors. - * Use the **Synthetics agent options** to provide fine-tuned configuration for the synthetics agent. Read more about available options in [Use the Synthetics CLI](../../../solutions/observability/apps/use-synthetics-cli.md). + * Use the **Synthetics agent options** to provide fine-tuned configuration for the synthetics agent. Read more about available options in [Use the Synthetics CLI](/solutions/observability/apps/use-synthetics-cli.md). 7. (Optional) Click **Run test** to verify that the test is valid. 8. Click **Create monitor**. - ## View in your Observability project [synthetics-get-started-ui-view-in-your-observability-project] Navigate to **Synthetics**, where you can see screenshots of each run, set up alerts in case of test failures, and more. -If a test does fail (shown as `down` in the Synthetics UI), you’ll be able to view the step script that failed, any errors, and a stack trace. For more information, refer to [Analyze data from synthetic monitors](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-journeys). +If a test does fail (shown as `down` in the Synthetics UI), you’ll be able to view the step script that failed, any errors, and a stack trace. For more information, refer to [Analyze data from synthetic monitors](/solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-journeys). ::::{note} When a monitor is created or updated, the first run might not occur immediately, but the time it takes for the first run to occur will be less than the monitor’s configured frequency. For example, if you create a monitor and configure it to run every 10 minutes, the first run will occur within 10 minutes of being created. After the first run, the monitor will begin running regularly based on the configured frequency. You can run a manual test if you want to see the results more quickly. :::: - ## Next steps [synthetics-get-started-ui-next-steps] Learn more about: -* [Writing user journeys](../../../solutions/observability/apps/write-synthetic-test.md) to use as inline scripts -* Using the [Synthetics Recorder](../../../solutions/observability/apps/use-synthetics-recorder.md) -* [Configuring lightweight monitors](../../../solutions/observability/apps/configure-lightweight-monitors.md) \ No newline at end of file +* [Writing user journeys](/solutions/observability/apps/write-synthetic-test.md) to use as inline scripts +* Using the [Synthetics Recorder](/solutions/observability/apps/use-synthetics-recorder.md) +* [Configuring lightweight monitors](/solutions/observability/apps/configure-lightweight-monitors.md) \ No newline at end of file diff --git a/solutions/observability/apps/create-monitors-with-project-monitors.md b/solutions/observability/apps/create-monitors-with-project-monitors.md index 6e8530b3a7..9e4243813f 100644 --- a/solutions/observability/apps/create-monitors-with-project-monitors.md +++ b/solutions/observability/apps/create-monitors-with-project-monitors.md @@ -10,21 +10,19 @@ applies_to: # Create monitors with a Synthetics project [observability-synthetics-get-started-project] - A Synthetics project is the most powerful and sophisticated way to configure synthetic monitors. A Synthetics project lets you define your infrastructure as code, more commonly known as IaaC or Git-ops. With monitors created and managed in Synthetics projects, you organize your YAML configuration and JavaScript- or TypeScript-defined monitors on the filesystem, use Git for version control, and deploy via a CLI tool, usually executed on a CI/CD platform. :::{image} /solutions/images/observability-synthetics-get-started-projects.png :alt: Diagram showing which pieces of software are used to configure monitors ::: -This is one of [two approaches](../../../solutions/observability/apps/get-started.md) you can use to set up a synthetic monitor. - +This is one of [two approaches](/solutions/observability/apps/get-started.md) you can use to set up a synthetic monitor. ## Prerequisites [synthetics-get-started-project-prerequisites] -For **serverless Observability projects**, you must be signed in as a user with [Editor](../../../solutions/observability/apps/grant-users-access-to-secured-resources.md) access. +For **serverless Observability projects**, you must be signed in as a user with [Editor](/solutions/observability/apps/grant-users-access-to-secured-resources.md) access. -For **Elastic Stack deployments**, you must be signed into {{kib}} as a user with at least [synthetics write permissions](../../../solutions/observability/apps/writer-role.md), and Monitor Management must be enabled by an administrator as described in [Setup role](../../../solutions/observability/apps/setup-role.md). +For **Elastic Stack deployments**, you must be signed into {{kib}} as a user with at least [synthetics write permissions](/solutions/observability/apps/writer-role.md), and Monitor Management must be enabled by an administrator as described in [Setup role](/solutions/observability/apps/setup-role.md). Working with a Synthetics project requires working with the Elastic Synthetics CLI tool, which can be invoked via the `npx @elastic/synthetics` command. Before getting started you’ll need to: @@ -41,20 +39,18 @@ Working with a Synthetics project requires working with the Elastic Synthetics C npx @elastic/synthetics -h ``` - You should also decide where you want to run the monitors before getting started. You can run monitors in Synthetics projects on one or both of the following: * **Elastic’s global managed testing infrastructure**: With Elastic’s global managed testing infrastructure, you can create and run monitors in multiple locations without having to manage your own infrastructure. Elastic takes care of software updates and capacity planning for you. -* **{{private-location}}s**: {{private-location}}s allow you to run monitors from your own premises. To use {{private-location}}s you must create a {{private-location}} before continuing. For step-by-step instructions, refer to [Monitor resources on private networks](../../../solutions/observability/apps/monitor-resources-on-private-networks.md). +* **{{private-location}}s**: {{private-location}}s allow you to run monitors from your own premises. To use {{private-location}}s you must create a {{private-location}} before continuing. For step-by-step instructions, refer to [Monitor resources on private networks](/solutions/observability/apps/monitor-resources-on-private-networks.md). % Stateful only for following note? ::::{note} -If you are setting up Synthetics for a deployment configured with [traffic filters](../../../deploy-manage/security/traffic-filtering.md), connections into {{es}} are restricted and results will not be able to be written back into {{es}} unless granted. For more details, refer to [Use Synthetics with traffic filters](../../../solutions/observability/apps/use-synthetics-with-traffic-filters.md). +If you are setting up Synthetics for a deployment configured with [traffic filters](/deploy-manage/security/traffic-filtering.md), connections into {{es}} are restricted and results will not be able to be written back into {{es}} unless granted. For more details, refer to [Use Synthetics with traffic filters](/solutions/observability/apps/use-synthetics-with-traffic-filters.md). :::: - ## Create a Synthetics project [synthetics-get-started-project-create-a-synthetics-project] Start by creating your first Synthetics project. Run the command below to create a new Synthetics project named `synthetic-project-test` in the current directory. @@ -79,11 +75,10 @@ Then, follow the prompts on screen to set up the correct default variables for y 4. Click **Generate Project API key**. ::::{important} - To generate a Project API key, you must be logged in as a user with [Editor](../../../solutions/observability/apps/grant-users-access-to-secured-resources.md) access. + To generate a Project API key, you must be logged in as a user with [Editor](/solutions/observability/apps/grant-users-access-to-secured-resources.md) access. :::: - :::{image} /solutions/images/serverless-synthetics-monitor-management-api-key.png :alt: Project API Keys tab in Synthetics settings :screenshot: @@ -103,7 +98,7 @@ Then, take a look at key files and directories inside your Synthetics project: * `synthetics.config.ts` contains settings for your Synthetics project. When you create a new Synthetics project, it will contain some basic configuration options that you can customize later. ::::{note} - The `synthetics.config.ts` in the sample Synthetics project uses a location on Elastic’s global managed testing infrastructure. Administrators can restrict access to Elastic’s global managed testing infrastructure. When you attempt to [`push` the sample monitors](../../../solutions/observability/apps/create-monitors-with-project-monitors.md#synthetics-get-started-project-test-and-connect-to-your-observability-project), if you see an error stating that you don’t have permission to use Elastic managed global locations, refer to the [troubleshooting guide](../../../troubleshoot/observability/troubleshooting-synthetics.md#synthetics-troubleshooting-no-locations) for guidance. + The `synthetics.config.ts` in the sample Synthetics project uses a location on Elastic’s global managed testing infrastructure. Administrators can restrict access to Elastic’s global managed testing infrastructure. When you attempt to [`push` the sample monitors](/solutions/observability/apps/create-monitors-with-project-monitors.md#synthetics-get-started-project-test-and-connect-to-your-observability-project), if you see an error stating that you don’t have permission to use Elastic managed global locations, refer to the [troubleshooting guide](/troubleshoot/observability/troubleshooting-synthetics.md#synthetics-troubleshooting-no-locations) for guidance. :::: @@ -122,11 +117,10 @@ Then, take a look at key files and directories inside your Synthetics project: 4. Click **Generate Project API key**. ::::{important} - To generate a Project API key, you must be logged in as a user with [Editor](../../../solutions/observability/apps/grant-users-access-to-secured-resources.md) access. + To generate a Project API key, you must be logged in as a user with [Editor](/solutions/observability/apps/grant-users-access-to-secured-resources.md) access. :::: - :::{image} /solutions/images/serverless-synthetics-monitor-management-api-key.png :alt: Project API Keys tab in Synthetics settings :screenshot: @@ -146,7 +140,7 @@ Then, take a look at key files and directories inside your Synthetics project: * `synthetics.config.ts` contains settings for your Synthetics project. When you create a new Synthetics project, it will contain some basic configuration options that you can customize later. ::::{note} - The `synthetics.config.ts` in the sample Synthetics project uses a location on Elastic’s global managed testing infrastructure. Administrators can restrict access to Elastic’s global managed testing infrastructure. When you attempt to [`push` the sample monitors](../../../solutions/observability/apps/create-monitors-with-project-monitors.md#synthetics-get-started-project-test-and-connect-to-your-observability-project), if you see an error stating that you don’t have permission to use Elastic managed global locations, refer to the [troubleshooting guide](../../../troubleshoot/observability/troubleshooting-synthetics.md#synthetics-troubleshooting-no-locations) for guidance. + The `synthetics.config.ts` in the sample Synthetics project uses a location on Elastic’s global managed testing infrastructure. Administrators can restrict access to Elastic’s global managed testing infrastructure. When you attempt to [`push` the sample monitors](/solutions/observability/apps/create-monitors-with-project-monitors.md#synthetics-get-started-project-test-and-connect-to-your-observability-project), if you see an error stating that you don’t have permission to use Elastic managed global locations, refer to the [troubleshooting guide](/troubleshoot/observability/troubleshooting-synthetics.md#synthetics-troubleshooting-no-locations) for guidance. :::: @@ -171,7 +165,7 @@ heartbeat.monitors: schedule: '@every 1m' ``` -For more details on lightweight monitor configuration options, refer to [Configure lightweight monitors](../../../solutions/observability/apps/configure-lightweight-monitors.md). +For more details on lightweight monitor configuration options, refer to [Configure lightweight monitors](/solutions/observability/apps/configure-lightweight-monitors.md). Inside the `journeys` directory you’ll find sample browser monitors. Here’s an example of a TypeScript file defining a browser monitor: @@ -195,8 +189,7 @@ journey('My Example Journey', ({ page, params }) => { }); ``` -For more details on writing journeys and configuring browser monitors, refer to [Scripting browser monitors](../../../solutions/observability/apps/scripting-browser-monitors.md). - +For more details on writing journeys and configuring browser monitors, refer to [Scripting browser monitors](/solutions/observability/apps/scripting-browser-monitors.md). ## Test and connect to your Observability project or Elastic Stack deployment[synthetics-get-started-project-test-and-connect-to-your-observability-project] @@ -220,13 +213,12 @@ While inside the project directory you can do two things with the `npx @elastic/ npx @elastic/synthetics push --auth $SYNTHETICS_API_KEY --url ``` - -One monitor will appear in the {{synthetics-app}} for each journey or lightweight monitor, and you’ll manage all monitors from your local environment. For more details on using the `push` command, refer to [`@elastic/synthetics push`](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). +One monitor will appear in the {{synthetics-app}} for each journey or lightweight monitor, and you’ll manage all monitors from your local environment. For more details on using the `push` command, refer to [`@elastic/synthetics push`](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). ::::{note} -If you’ve [added a {{private-location}}](../../../solutions/observability/apps/monitor-resources-on-private-networks.md), you can `push` to that {{private-location}}. +If you’ve [added a {{private-location}}](/solutions/observability/apps/monitor-resources-on-private-networks.md), you can `push` to that {{private-location}}. -To list available {{private-location}}s, run the [`elastic-synthetics locations` command](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command) with the {{kib}} URL for the deployment from which to fetch available locations. +To list available {{private-location}}s, run the [`elastic-synthetics locations` command](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command) with the {{kib}} URL for the deployment from which to fetch available locations. :::: @@ -249,13 +241,12 @@ While inside the Synthetics project directory you can do two things with the `np npx @elastic/synthetics push --auth $SYNTHETICS_API_KEY --url ``` - -One monitor will appear in the Synthetics UI for each journey or lightweight monitor, and you’ll manage all monitors from your local environment. For more details on using the `push` command, refer to [`@elastic/synthetics push`](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). +One monitor will appear in the Synthetics UI for each journey or lightweight monitor, and you’ll manage all monitors from your local environment. For more details on using the `push` command, refer to [`@elastic/synthetics push`](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). ::::{note} -If you’ve [added a {{private-location}}](../../../solutions/observability/apps/monitor-resources-on-private-networks.md), you can `push` to that {{private-location}}. +If you’ve [added a {{private-location}}](/solutions/observability/apps/monitor-resources-on-private-networks.md), you can `push` to that {{private-location}}. -To list available {{private-location}}s, run the [`elastic-synthetics locations` command](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command) with the URL for the Observability project from which to fetch available locations. +To list available {{private-location}}s, run the [`elastic-synthetics locations` command](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command) with the URL for the Observability project from which to fetch available locations. :::: @@ -263,7 +254,6 @@ To list available {{private-location}}s, run the [`elastic-synthetics locations` :::: - ## View in the Synthetics UI [synthetics-get-started-project-view-in-your-observability-project] Then, go to **Synthetics** in your serverless Observability project or in {{kib}}. You should see your newly pushed monitors running. You can also go to the **Management** tab to see the monitors' configuration settings. @@ -273,12 +263,10 @@ When a monitor is created or updated, the first run might not occur immediately, :::: - - ## Next steps [observability-synthetics-get-started-project-next-steps] Learn more about: -* [Configuring lightweight monitors](../../../solutions/observability/apps/configure-lightweight-monitors.md) -* [Configuring browser monitors](../../../solutions/observability/apps/write-synthetic-test.md) -* [Implementing best practices for working with Synthetics projects](../../../solutions/observability/apps/manage-monitors.md#synthetics-projects-best-practices) \ No newline at end of file +* [Configuring lightweight monitors](/solutions/observability/apps/configure-lightweight-monitors.md) +* [Configuring browser monitors](/solutions/observability/apps/write-synthetic-test.md) +* [Implementing best practices for working with Synthetics projects](/solutions/observability/apps/manage-monitors.md#synthetics-projects-best-practices) \ No newline at end of file diff --git a/solutions/observability/apps/create-upload-source-maps-rum.md b/solutions/observability/apps/create-upload-source-maps-rum.md index 69525748cd..0959dfbc0b 100644 --- a/solutions/observability/apps/create-upload-source-maps-rum.md +++ b/solutions/observability/apps/create-upload-source-maps-rum.md @@ -31,7 +31,6 @@ Follow the steps below to enable source mapping your error stack traces in the A * [Generate a source map](#apm-source-map-rum-generate) * [Upload the source map](#apm-source-map-rum-upload) - ## Initialize the RUM Agent [apm-source-map-rum-initialize] Set the service name and version of your application when initializing the RUM Agent. To make uploading subsequent source maps easier, the `serviceVersion` you choose might be the `version` from your `package.json`. For example: @@ -55,7 +54,6 @@ const serviceVersion = git.short() It can also be any other unique string that indicates a specific version of your application. The APM integration uses the service name and version to match the correct source map file to each stack trace. - ## Generate a source map [apm-source-map-rum-generate] To be compatible with Elastic APM, source maps must follow the [source map revision 3 proposal spec](https://sourcemaps.info/spec.html). @@ -84,15 +82,12 @@ module.exports = { 1. If you’re using a different method of defining `serviceVersion`, you can set it here. - - ## Upload the source map [apm-source-map-rum-upload] ::::{tip} When uploading a source map, ensure that RUM support is enabled in the APM integration. :::: - {{kib}} exposes a source map endpoint for uploading source maps. Source maps can be uploaded as a string, or as a file upload. Let’s look at two different ways to upload a source map: curl and a custom application. Each example includes the four fields necessary for APM Server to later map minified code to its source: @@ -104,7 +99,6 @@ Let’s look at two different ways to upload a source map: curl and a custom app If you have multiple source maps, you’ll need to upload each individually. - ### Upload via curl [apm-source-map-curl] Here’s an example curl request that uploads the source map file created in the previous step. This request uses an API key for authentication. @@ -152,7 +146,6 @@ request.post({url: 'http://localhost:5601/api/apm/sourcemaps',formData: formData }) ``` - ## What happens next [apm-source-map-next] Source maps are stored in {{es}}. When you upload a source map, a new {{es}} document is created containing the contents of the source map. When a RUM request comes in, APM Server will make use of these source map documents to apply the source map logic to the event’s stack traces. diff --git a/solutions/observability/apps/cross-cluster-search-with-application-data.md b/solutions/observability/apps/cross-cluster-search-with-application-data.md index 9b44279032..beff362592 100644 --- a/solutions/observability/apps/cross-cluster-search-with-application-data.md +++ b/solutions/observability/apps/cross-cluster-search-with-application-data.md @@ -8,15 +8,13 @@ applies_to: # Cross-cluster search with application data [apm-cross-cluster-search] - Elastic APM utilizes {{es}}'s cross-cluster search functionality. Cross-cluster search lets you run a single search request against one or more [remote clusters](/deploy-manage/remote-clusters/remote-clusters-self-managed.md) — making it easy to search APM data across multiple sources. This means you can also have deployments per data type, making sizing and scaling more predictable, and allowing for better performance while managing multiple observability use cases. - ## Set up cross-cluster search [apm-set-up-cross-cluster-search] **Step 1. Set up remote clusters.** -If you’re using the {{ech}}, see [Enable cross-cluster search](../../../deploy-manage/remote-clusters/ec-enable-ccs.md). +If you’re using the {{ech}}, see [Enable cross-cluster search](/deploy-manage/remote-clusters/ec-enable-ccs.md). To add remote clusters directly in {{kib}}, find `Remote Clusters` in the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). All you need is a name for the remote cluster and the seed node(s). Remember the names of your remote clusters, you’ll need them in step two. See [managing remote clusters](/deploy-manage/tools/cross-cluster-replication/set-up-cross-cluster-replication.md) for detailed information on the setup process. @@ -39,7 +37,7 @@ There are two ways to edit the default {{data-source}}: In a cross-cluster search (CCS) environment, it’s possible for different clusters to serve different data tiers in responses. If one of the requested clusters responds slowly, it can cause a timeout at the proxy after 320 seconds. This results in 502 Bad Gateway server error responses presented as failure toast messages in the UI, and no data loaded. -To prevent this, you can exclude [data tiers](../../../manage-data/lifecycle/data-tiers.md) that might slow down responses from search: the `data_frozen` and `data_cold` tiers. To exclude data tiers from search in the APM UI: +To prevent this, you can exclude [data tiers](/manage-data/lifecycle/data-tiers.md) that might slow down responses from search: the `data_frozen` and `data_cold` tiers. To exclude data tiers from search in the APM UI: 1. To open **Advanced settings**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). 2. In the **Observability** section, update the **Excluded data tiers from search** option with a list of data tiers. diff --git a/solutions/observability/apps/custom-filters.md b/solutions/observability/apps/custom-filters.md index d1037c0f61..6075a151a0 100644 --- a/solutions/observability/apps/custom-filters.md +++ b/solutions/observability/apps/custom-filters.md @@ -3,15 +3,20 @@ mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-custom-filter.html applies_to: stack: + serverless: --- # Custom filters [apm-custom-filter] Custom filters, including [ingest pipeline filters](#apm-filters-ingest-pipeline) and [APM agent filters](#apm-filters-in-agent), allow you to filter or redact APM data on ingestion. - ## Ingest pipeline filters [apm-filters-ingest-pipeline] +```{applies_to} +stack: +serverless: unavailable +``` + Ingest pipelines specify a series of processors that transform data in a specific way. Transformation happens prior to indexing—​inflicting no performance overhead on the monitored application. Pipelines are a flexible and easy way to filter or obfuscate Elastic APM data. Features of this approach: @@ -23,9 +28,13 @@ Features of this approach: For a step-by-step example, refer to [Tutorial: Use an ingest pipeline to redact sensitive information](#apm-filters-ingest-pipeline-tutorial). - ## APM agent filters [apm-filters-in-agent] +```{applies_to} +stack: +serverless: +``` + Some APM agents offer a way to manipulate or drop APM events *before* they are sent to APM Server. Features of this approach: @@ -41,9 +50,13 @@ Refer to the relevant agent’s documentation for more information and examples: * Python: [custom processors](apm-agent-python://reference/sanitizing-data.md). * Ruby: [`add_filter()`](apm-agent-ruby://reference/api-reference.md#api-agent-add-filter). - ## Tutorial: Use an ingest pipeline to redact sensitive information [apm-filters-ingest-pipeline-tutorial] +```{applies_to} +stack: +serverless: unavailable +``` + Say you decide to [capture HTTP request bodies](built-in-data-filters.md#apm-filters-http-body) but quickly notice that sensitive information is being collected in the `http.request.body.original` field: ```json @@ -55,14 +68,12 @@ Say you decide to [capture HTTP request bodies](built-in-data-filters.md#apm-fil To obfuscate the passwords stored in the request body, you can use a series of [ingest processors](elasticsearch://reference/enrich-processor/index.md). - ### Create a pipeline [_create_a_pipeline] ::::{tip} This tutorial uses the [Ingest APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ingest), but it’s also possible to create a pipeline using the UI. In Kibana, go to **Stack Management** → **Ingest Pipelines** → **Create pipeline** → **New pipeline** or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). :::: - To start, create a pipeline with a simple description and an empty array of processors: ```json @@ -76,8 +87,6 @@ To start, create a pipeline with a simple description and an empty array of proc 1. The processors defined below will go in this array - - #### Add a JSON processor [_add_a_json_processor] Add your first processor to the processors array. Because the agent captures the request body as a string, use the [JSON processor](elasticsearch://reference/enrich-processor/json-processor.md) to convert the original field value into a structured JSON object. Save this JSON object in a new field: @@ -92,7 +101,6 @@ Add your first processor to the processors array. Because the agent captures the } ``` - #### Add a set processor [_add_a_set_processor] If `body.original_json` is not `null`, i.e., it exists, we’ll redact the `password` with the [set processor](elasticsearch://reference/enrich-processor/set-processor.md), by setting the value of `body.original_json.password` to `"redacted"`: @@ -107,7 +115,6 @@ If `body.original_json` is not `null`, i.e., it exists, we’ll redact the `pass } ``` - #### Add a convert processor [_add_a_convert_processor] Use the [convert processor](elasticsearch://reference/enrich-processor/convert-processor.md) to convert the JSON value of `body.original_json` to a string and set it as the `body.original` value: @@ -124,7 +131,6 @@ Use the [convert processor](elasticsearch://reference/enrich-processor/convert-p } ``` - #### Add a remove processor [_add_a_remove_processor] Finally, use the [remove processor](elasticsearch://reference/enrich-processor/remove-processor.md) to remove the `body.original_json` field: @@ -139,7 +145,6 @@ Finally, use the [remove processor](elasticsearch://reference/enrich-processor/r } ``` - #### Register the pipeline [_register_the_pipeline] Then put it all together, and use the [create or update pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) to register the new pipeline in {{es}}. Name the pipeline `apm_redacted_body_password`: @@ -183,7 +188,6 @@ PUT _ingest/pipeline/apm_redacted_body_password } ``` - ### Test the pipeline [_test_the_pipeline] Prior to enabling this new pipeline, you can test it with the [simulate pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-simulate). This API allows you to run multiple documents through a pipeline to ensure it is working correctly. @@ -229,7 +233,6 @@ POST _ingest/pipeline/apm_redacted_body_password/_simulate 2. This document only contains an unrelated field 3. This document contains invalid JSON - The API response should be similar to this: ```json @@ -277,7 +280,6 @@ The API response should be similar to this: As expected, only the first simulated document has a redacted password field. All other documents are unaffected. - ### Create a `@custom` pipeline [_create_a_custom_pipeline] The final step in this process is to call the newly created `apm_redacted_body_password` pipeline from the `@custom` pipeline of the data stream you wish to edit. @@ -314,10 +316,8 @@ PUT _ingest/pipeline/traces-apm@custom 1. The name of the pipeline we previously created - That’s it! Passwords will now be redacted from your APM HTTP body data. - ### Next steps [_next_steps] To learn more about ingest pipelines, see [View the {{es}} index template](view-elasticsearch-index-template.md). diff --git a/solutions/observability/apps/data-streams.md b/solutions/observability/apps/data-streams.md index e51db657bb..184b411550 100644 --- a/solutions/observability/apps/data-streams.md +++ b/solutions/observability/apps/data-streams.md @@ -19,13 +19,10 @@ See the [{{fleet}} and {{agent}} Guide](/reference/fleet/data-streams.md) to lea :::: - - ## Data stream naming scheme [apm-data-streams-naming-scheme] APM data follows the `--` naming scheme. The `type` and `dataset` are predefined by the {{es}} apm-data plugin, but the `namespace` is your opportunity to customize how different types of data are stored in {{es}}. There is no recommendation for what to use as your namespace—​it is intentionally flexible. For example, you might create namespaces for each of your environments, like `dev`, `prod`, `production`, etc. Or, you might create namespaces that correspond to strategic business units within your organization. - ## APM data streams [apm-data-streams-list] By type, the APM data streams are: @@ -36,7 +33,6 @@ Traces * Application traces: `traces-apm-` * RUM and iOS agent application traces: `traces-apm.rum-` - Metrics : Metrics include application-based metrics, aggregation metrics, and basic system metrics. Metrics are stored in the following data streams: @@ -57,24 +53,19 @@ Metrics '\\', '/', '*', '?', '"', '<', '>', '|', ' ', ',', '#', ':', '-' ``` - :::: - ::::{important} Additional storage efficiencies provided by [Synthetic `_source`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md) are available to users with an [appropriate license](https://www.elastic.co/subscriptions). :::: - Logs : Logs include application error events and application logs. Logs are stored in the following data streams: * APM error/exception logging: `logs-apm.error-` * Applications UI logging: `logs-apm.app.-` - - ## APM data stream rerouting [apm-data-stream-rerouting] APM supports rerouting APM data to user-defined APM data stream names other than the defaults. This can be achieved by using a [`reroute` processor](elasticsearch://reference/enrich-processor/reroute-processor.md) in ingest pipelines to set the data stream dataset or namespace. The benefit of separating APM data streams is that custom retention and security policies can be used. @@ -94,11 +85,10 @@ For example, consider traces that would originally be indexed to `traces-apm-def ] ``` -To find other ingest pipelines from the {{es}} apm-data plugin that are called by default, go to **Stack management** → **Ingest pipelines** [in Kibana](../../../deploy-manage/index.md) and search for `apm`. Default APM ingest pipelines will follow the pattern `*-apm*@default-pipeline`. +To find other ingest pipelines from the {{es}} apm-data plugin that are called by default, go to **Stack management** → **Ingest pipelines** [in Kibana](/deploy-manage/index.md) and search for `apm`. Default APM ingest pipelines will follow the pattern `*-apm*@default-pipeline`. For more custom APM ingest pipeline guides, see [parse data using ingest pipelines](parse-data-using-ingest-pipelines.md). - ## What’s next? [apm-data-streams-next] * Data streams define not only how data is stored in {{es}}, but also how data is retained over time. See [{{ilm-cap}}](index-lifecycle-management.md) to learn how to create your own data retention policies. diff --git a/solutions/observability/apps/delete-sensitive-data.md b/solutions/observability/apps/delete-sensitive-data.md index 078d263a74..aaf0f25a95 100644 --- a/solutions/observability/apps/delete-sensitive-data.md +++ b/solutions/observability/apps/delete-sensitive-data.md @@ -15,8 +15,6 @@ If you accidentally ingest sensitive data, follow these steps to remove or redac * [Redact specific fields](#apm-redact-field-data) * [Delete {{es}} documents](#apm-delete-doc-data) - - ## Redact specific fields [apm-redact-field-data] To redact sensitive data in a specific field, use the [update by query API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update-by-query). @@ -63,14 +61,12 @@ POST /logs-apm.error-default/_update_by_query See [update by query API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update-by-query) for more information and examples. - ## Delete {{es}} documents [apm-delete-doc-data] ::::{warning} This will permanently delete your data. You should test your queries with the [search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) prior to deleting data. :::: - To delete an {{es}} document, you can use the [delete by query API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-delete-by-query). For example, to delete all documents in the `apm-traces-*` data stream with a `user.email` value, run the following query: diff --git a/solutions/observability/apps/dependencies.md b/solutions/observability/apps/dependencies.md index 9ae372f7bc..07c8c7a152 100644 --- a/solutions/observability/apps/dependencies.md +++ b/solutions/observability/apps/dependencies.md @@ -9,7 +9,7 @@ applies_to: # Dependencies [apm-dependencies] -APM agents collect details about external calls made from instrumented services. Sometimes, these external calls resolve into a downstream service that’s instrumented — in these cases, you can utilize [distributed tracing](../../../solutions/observability/apps/trace-sample-timeline.md#distributed-tracing) to drill down into problematic downstream services. Other times, though, it’s not possible to instrument a downstream dependency — like with a database or third-party service. **Dependencies** gives you a window into these uninstrumented, downstream dependencies. +APM agents collect details about external calls made from instrumented services. Sometimes, these external calls resolve into a downstream service that’s instrumented — in these cases, you can utilize [distributed tracing](/solutions/observability/apps/trace-sample-timeline.md#distributed-tracing) to drill down into problematic downstream services. Other times, though, it’s not possible to instrument a downstream dependency — like with a database or third-party service. **Dependencies** gives you a window into these uninstrumented, downstream dependencies. :::{image} /solutions/images/observability-dependencies.png :alt: Dependencies view in the Applications UI @@ -29,7 +29,6 @@ When viewing a dependency, consider your pattern of usage with that dependency. If your usage pattern *has* changed, the dependency view can quickly show you whether that pattern change exists in all upstream services, or just a subset of your services. You might then start digging into traces coming from impacted services to determine why that pattern change has occurred. - ## Operations [dependencies-operations] ::::{Warning} @@ -45,7 +44,7 @@ The Dependency operations functionality is in beta and is subject to change. The :screenshot: ::: -Selecting an operation displays the operation’s impact and performance trends over time, via key metrics like latency, throughput, and failed transaction rate. In addition, the [**Trace sample timeline**](../../../solutions/observability/apps/trace-sample-timeline.md) provides a visual drill-down into an end-to-end trace sample. +Selecting an operation displays the operation’s impact and performance trends over time, via key metrics like latency, throughput, and failed transaction rate. In addition, the [**Trace sample timeline**](/solutions/observability/apps/trace-sample-timeline.md) provides a visual drill-down into an end-to-end trace sample. :::{image} /solutions/images/observability-operations-detail.png :alt: operations detail view in the Applications UI diff --git a/solutions/observability/apps/drill-down-into-data.md b/solutions/observability/apps/drill-down-into-data.md index a41d1af453..8ae953df82 100644 --- a/solutions/observability/apps/drill-down-into-data.md +++ b/solutions/observability/apps/drill-down-into-data.md @@ -10,12 +10,11 @@ applies_to: # Drill down into application data [apm-ui-drill-down] - Notice something awry? Select a service or trace and dive deeper with: -* [Transactions](../../../solutions/observability/apps/transactions-2.md) -* [Trace sample timeline](../../../solutions/observability/apps/trace-sample-timeline.md) -* [Errors](../../../solutions/observability/apps/errors-2.md) -* [Metrics](../../../solutions/observability/apps/metrics-2.md) -* [Infrastructure](../../../solutions/observability/apps/infrastructure.md) -* [Logs](../../../solutions/observability/apps/logs.md) +* [Transactions](/solutions/observability/apps/transactions-2.md) +* [Trace sample timeline](/solutions/observability/apps/trace-sample-timeline.md) +* [Errors](/solutions/observability/apps/errors-2.md) +* [Metrics](/solutions/observability/apps/metrics-2.md) +* [Infrastructure](/solutions/observability/apps/infrastructure.md) +* [Logs](/solutions/observability/apps/logs.md) diff --git a/solutions/observability/apps/elastic-apm-agent-configuration-api.md b/solutions/observability/apps/elastic-apm-agent-configuration-api.md index 0afdd353fe..9986f994c4 100644 --- a/solutions/observability/apps/elastic-apm-agent-configuration-api.md +++ b/solutions/observability/apps/elastic-apm-agent-configuration-api.md @@ -9,7 +9,6 @@ applies_to: APM Server exposes API endpoints that allow Elastic APM agents to query the APM Server for configuration changes. More information on this feature is available in [{{apm-agent}} configuration in {{kib}}](apm-agent-central-configuration.md). - ## Agent configuration endpoints [apm-api-config-endpoint] | Name | Endpoint | @@ -19,7 +18,6 @@ APM Server exposes API endpoints that allow Elastic APM agents to query the APM The Agent configuration endpoints accepts both `HTTP GET` and `HTTP POST` requests. If an [API keys](api-keys.md) or [Secret token](secret-token.md) is configured, requests to this endpoint must be authenticated. - ### HTTP GET [apm-api-config-api-get] `service.name` is a required query string parameter. @@ -28,7 +26,6 @@ The Agent configuration endpoints accepts both `HTTP GET` and `HTTP POST` reques http(s)://{hostname}:{port}/config/v1/agents?service.name=SERVICE_NAME ``` - ### HTTP POST [apm-api-config-api-post] Encode parameters as a JSON object in the body. `service.name` is a required parameter. @@ -44,14 +41,12 @@ http(s)://{hostname}:{port}/config/v1/agents } ``` - ### Responses [apm-api-config-api-response] * Successful - `200` * APM Server is configured to fetch agent configuration from {{es}} but the configuration is invalid - `403` * APM Server is starting up or {{es}} is unreachable - `503` - ### Example request [apm-api-config-api-example] Example Agent configuration `GET` request including the service name "test-service": @@ -69,7 +64,6 @@ curl -X POST http://127.0.0.1:8200/config/v1/agents \ -d '{"service": {"name": "test-service"}}' ``` - ### Example response [apm-api-config-api-ex-response] ```sh diff --git a/solutions/observability/apps/elastic-apm-agents.md b/solutions/observability/apps/elastic-apm-agents.md index fcee458be4..4fb8d4f3ae 100644 --- a/solutions/observability/apps/elastic-apm-agents.md +++ b/solutions/observability/apps/elastic-apm-agents.md @@ -10,14 +10,12 @@ applies_to: # Elastic APM agents [observability-apm-agents-elastic-apm-agents] - ::::{note} -**For Observability Serverless projects**, the **Admin** role or higher is required to use APM agents. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). +**For Observability Serverless projects**, the **Admin** role or higher is required to use APM agents. To learn more, refer to [Assign user roles and privileges](/deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). :::: - Elastic APM agents automatically measure application performance and track errors. They offer built-in support for popular frameworks and technologies, and provide easy-to-use APIs that allow you to instrument any application. Elastic APM agents are built and maintained by Elastic. While they are similar, different programming languages have different nuances and requirements. Select your preferred language below to learn more about how each agent works. @@ -33,7 +31,7 @@ Spans are grouped in transactions—by default, one for each incoming HTTP reque **Learn more** -If you're ready to give Elastic APM a try, see [Get started with traces and APM](../../../solutions/observability/apps/get-started-with-apm.md). +If you're ready to give Elastic APM a try, see [Get started with traces and APM](/solutions/observability/apps/get-started-with-apm.md). See the [Java agent reference](apm-agent-java://reference/index.md) for full documentation, including: @@ -59,7 +57,7 @@ These events, called Transactions and Spans, are sent to Elastic, where they're **Learn more** -If you’re ready to give Elastic APM a try, see [Get started with traces and APM](../../../solutions/observability/apps/get-started-with-apm.md). +If you’re ready to give Elastic APM a try, see [Get started with traces and APM](/solutions/observability/apps/get-started-with-apm.md). See the [Node.js agent reference](apm-agent-nodejs://reference/index.md) for full documentation, including: @@ -89,7 +87,7 @@ In addition to APM and error data, the Python agent also collects system and app **Learn more** -If you’re ready to give Elastic APM a try, see [Get started with traces and APM](../../../solutions/observability/apps/get-started-with-apm.md). +If you’re ready to give Elastic APM a try, see [Get started with traces and APM](/solutions/observability/apps/get-started-with-apm.md). See the [Python agent reference](apm-agent-python://reference/index.md) for full documentation, including: @@ -115,7 +113,7 @@ These events, called Transactions and Spans, are sent to Elastic, where they're **Learn more** -If you're ready to give Elastic APM a try, see [Get started with traces and APM](../../../solutions/observability/apps/get-started-with-apm.md). +If you're ready to give Elastic APM a try, see [Get started with traces and APM](/solutions/observability/apps/get-started-with-apm.md). See the [Ruby agent reference](apm-agent-ruby://reference/index.md) for full documentation, including: @@ -147,7 +145,7 @@ In addition to capturing events like those mentioned here, the agent also collec **Learn more** -If you're ready to give Elastic APM a try, see [Get started with traces and APM](../../../solutions/observability/apps/get-started-with-apm.md). +If you're ready to give Elastic APM a try, see [Get started with traces and APM](/solutions/observability/apps/get-started-with-apm.md). See the [Go agent reference](apm-agent-go://reference/index.md) for full documentation, including: @@ -171,7 +169,7 @@ The Agent automatically registers callback methods for built-in Diagnostic Sourc **Learn more** -If you're ready to give Elastic APM a try, see [Get started with traces and APM](../../../solutions/observability/apps/get-started-with-apm.md). +If you're ready to give Elastic APM a try, see [Get started with traces and APM](/solutions/observability/apps/get-started-with-apm.md). See the [.NET agent reference](apm-agent-dotnet://reference/index.md) for full documentation, including: @@ -193,7 +191,7 @@ The Elastic APM PHP agent measures application performance and tracks errors. Th **Learn more** -If you're ready to give Elastic APM a try, see [Get started with traces and APM](../../../solutions/observability/apps/get-started-with-apm.md). +If you're ready to give Elastic APM a try, see [Get started with traces and APM](/solutions/observability/apps/get-started-with-apm.md). See the [PHP agent reference](apm-agent-php://reference/index.md) for full documentation, including: diff --git a/solutions/observability/apps/elastic-apm-events-intake-api.md b/solutions/observability/apps/elastic-apm-events-intake-api.md index 0684abb017..6fe5ff519d 100644 --- a/solutions/observability/apps/elastic-apm-events-intake-api.md +++ b/solutions/observability/apps/elastic-apm-events-intake-api.md @@ -11,7 +11,6 @@ applies_to: Most users do not need to interact directly with the events intake API. :::: - The events intake API is what we call the internal protocol that APM agents use to talk to the APM Server. Agents communicate with the Server by sending events — captured pieces of information — in an HTTP request. Events can be: * Transactions @@ -25,7 +24,6 @@ With NDJSON, agents can open an HTTP POST request and use chunked encoding to st See the [APM data model](learn-about-application-data-types.md) to learn more about the different types of events. - ### Endpoints [apm-api-events-endpoint] APM Server exposes the following endpoints for Elastic APM agent data intake: @@ -36,7 +34,6 @@ APM Server exposes the following endpoints for Elastic APM agent data intake: | RUM event intake (v2) | `/intake/v2/rum/events` | | RUM event intake (v3) | `/intake/v3/rum/events` | - ### Request [apm-api-events-example] Send an `HTTP POST` request to the APM Server `intake/v2/events` endpoint: @@ -55,21 +52,18 @@ http(s)://{hostname}:{port}/intake/v2/events?async=true Since asynchronous processing defers some of the event processing to the background and takes place after the client has closed the request, some errors can’t be communicated back to the client and are logged by the APM Server. Furthermore, asynchronous processing requests will only be scheduled if the APM Server can service the incoming request, requests that cannot be serviced will receive an internal error `503` "queue is full" error. :::: - For [RUM](real-user-monitoring-rum.md) send an `HTTP POST` request to the APM Server `intake/v3/rum/events` endpoint instead: ```bash http(s)://{hostname}:{port}/intake/v3/rum/events ``` - ### Response [apm-api-events-response] On success, the server will respond with a 202 Accepted status code and no body. Keep in mind that events can succeed and fail independently of each other. Only if all events succeed does the server respond with a 202. - ### Errors [apm-api-events-errors] There are two types of errors that the APM Server may return to an agent: @@ -108,10 +102,8 @@ An example error response might look something like this: 3. An immediately returning non-event related error 4. The number of accepted events - If you’re developing an agent, these errors can be useful for debugging. - ### Event API Schemas [apm-api-events-schema-definition] The APM Server uses a collection of JSON Schemas for validating requests to the intake API: @@ -133,11 +125,9 @@ Rather than send this metadata information from the agent multiple times, the AP Metadata is stored under `context` when viewing documents in {{es}}. :::: - * [Kubernetes data](#apm-api-kubernetes-data) * [Metadata Schema](#apm-api-metadata-schema) - #### Kubernetes data [apm-api-kubernetes-data] APM agents automatically read Kubernetes data and send it to the APM Server. In most instances, agents are able to read this data from inside the container. If this is not the case, or if you wish to override this data, you can set environment variables for the agents to read. These environment variable are set via the Kubernetes [Downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#use-pod-fields-as-values-for-environment-variables). Here’s how you would add the environment variables to your Kubernetes pod spec: @@ -170,7 +160,6 @@ The table below maps these environment variables to the APM metadata event field | `KUBERNETES_NAMESPACE` | system.kubernetes.namespace | | `KUBERNETES_POD_UID` | system.kubernetes.pod.uid | - #### Metadata Schema [apm-api-metadata-schema] APM Server uses JSON Schema to validate requests. The specification for metadata is defined on [GitHub](https://github.com/elastic/apm-server/blob/9.0/docs/spec/v2/metadata.json) and included below: @@ -753,12 +742,10 @@ APM Server uses JSON Schema to validate requests. The specification for metadata } ``` - ## Transactions [apm-api-transaction] Transactions are events corresponding to an incoming request or similar task occurring in a monitored service. - #### Transaction Schema [apm-api-transaction-schema] APM Server uses JSON Schema to validate requests. The specification for transactions is defined on [GitHub](https://github.com/elastic/apm-server/blob/9.0/docs/spec/v2/transaction.json) and included below: @@ -1897,12 +1884,10 @@ APM Server uses JSON Schema to validate requests. The specification for transact } ``` - ## Spans [apm-api-span] Spans are events captured by an agent occurring in a monitored service. - #### Span Schema [apm-api-span-schema] APM Server uses JSON Schema to validate requests. The specification for spans is defined on [GitHub](https://github.com/elastic/apm-server/blob/9.0/docs/spec/v2/span.json) and included below: @@ -2816,12 +2801,10 @@ APM Server uses JSON Schema to validate requests. The specification for spans is } ``` - ## Errors [apm-api-error] An error or a logged error message captured by an agent occurring in a monitored service. - #### Error Schema [apm-api-error-schema] APM Server uses JSON Schema to validate requests. The specification for errors is defined on [GitHub](https://github.com/elastic/apm-server/blob/9.0/docs/spec/v2/error.json) and included below: @@ -4122,12 +4105,10 @@ APM Server uses JSON Schema to validate requests. The specification for errors i } ``` - ## Metrics [apm-api-metricset] Metrics contain application metric data captured by an {{apm-agent}}. - #### Metric Schema [apm-api-metricset-schema] APM Server uses JSON Schema to validate requests. The specification for metrics is defined on [GitHub](https://github.com/elastic/apm-server/blob/9.0/docs/spec/v2/metricset.json) and included below: @@ -4436,7 +4417,6 @@ APM Server uses JSON Schema to validate requests. The specification for metrics } ``` - ## Example request body [apm-api-event-example] A request body example containing one event for all currently supported event types. @@ -4449,4 +4429,3 @@ A request body example containing one event for all currently supported event ty {"metricset":{"samples":{"transaction.breakdown.count":{"value":12},"transaction.duration.sum.us":{"value":12},"transaction.duration.count":{"value":2},"transaction.self_time.sum.us":{"value":10},"transaction.self_time.count":{"value":2},"span.self_time.count":{"value":1},"span.self_time.sum.us":{"value":633.288},"byte_counter":{"value":1},"short_counter":{"value":227},"integer_gauge":{"value":42767},"long_gauge":{"value":3147483648},"float_gauge":{"value":9.16},"double_gauge":{"value":3.141592653589793},"dotted.float.gauge":{"value":6.12},"negative.d.o.t.t.e.d":{"value":-1022}},"tags":{"code":200,"success":true},"transaction":{"type":"request","name":"GET/"},"span":{"type":"db","subtype":"mysql"},"timestamp":1571657444929001}} ``` - diff --git a/solutions/observability/apps/enable-apm-server-binary-debugging.md b/solutions/observability/apps/enable-apm-server-binary-debugging.md deleted file mode 100644 index d9f4d63e03..0000000000 --- a/solutions/observability/apps/enable-apm-server-binary-debugging.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -navigation_title: "APM Server binary debugging" -mapped_pages: - - https://www.elastic.co/guide/en/observability/current/apm-enable-apm-server-debugging.html -applies_to: - stack: ---- - - - -# Enable APM Server binary debugging [apm-enable-apm-server-debugging] - - -::::{note} -Fleet-managed users should see [View {{agent}} logs](/reference/fleet/monitor-elastic-agent.md) to learn how to view logs and change the logging level of {{agent}}. -:::: - - -By default, APM Server sends all its output to syslog. When you run APM Server in the foreground, you can use the `-e` command line flag to redirect the output to standard error instead. For example: - -```sh -apm-server -e -``` - -The default configuration file is apm-server.yml (the location of the file varies by platform). You can use a different configuration file by specifying the `-c` flag. For example: - -```sh -apm-server -e -c myapm-serverconfig.yml -``` - -You can increase the verbosity of debug messages by enabling one or more debug selectors. For example, to view publisher-related messages, start APM Server with the `publisher` selector: - -```sh -apm-server -e -d "publisher" -``` - -If you want all the debugging output (fair warning, it’s quite a lot), you can use `*`, like this: - -```sh -apm-server -e -d "*" -``` - diff --git a/solutions/observability/apps/errors.md b/solutions/observability/apps/errors.md index 35e7882a27..b34698e612 100644 --- a/solutions/observability/apps/errors.md +++ b/solutions/observability/apps/errors.md @@ -23,17 +23,14 @@ An Error contains: * URL - full, domain, port, query, etc. * [User](metadata.md#apm-data-model-user) - (if supplied) email, ID, username, etc. - In addition, agents provide options for users to capture custom [metadata](metadata.md). Metadata can be indexed - [`labels`](metadata.md#apm-data-model-labels), or not-indexed - [`custom`](metadata.md#apm-data-model-custom). ::::{tip} Most agents limit keyword fields (e.g. `error.id`) to 1024 characters, non-keyword fields (e.g. `error.exception.message`) to 10,000 characters. :::: - Errors are stored in error indices. - ## Data streams [_data_streams_3] Errors are stored in the following data streams: @@ -43,7 +40,6 @@ Errors are stored in the following data streams: See [Data streams](data-streams.md) to learn more. - ## Example error document [_example_error_document] This example shows what error documents can look like when indexed in {{es}}. @@ -643,4 +639,3 @@ This example shows what error documents can look like when indexed in {{es}}. :::: - diff --git a/solutions/observability/apps/explore-data-in-elasticsearch.md b/solutions/observability/apps/explore-data-in-elasticsearch.md index cc52979f29..0a0e016b80 100644 --- a/solutions/observability/apps/explore-data-in-elasticsearch.md +++ b/solutions/observability/apps/explore-data-in-elasticsearch.md @@ -9,12 +9,11 @@ applies_to: * [{{es}} query examples](#apm-elasticsearch-query-examples) - ## {{es}} query examples [apm-elasticsearch-query-examples] Elastic APM data is stored in [Data streams](data-streams.md). -The following examples enable you to interact with {{es}}'s REST API. One possible way to do this is using {{kib}}'s [{{dev-tools-app}} console](../../../explore-analyze/query-filter/tools/console.md). +The following examples enable you to interact with {{es}}'s REST API. One possible way to do this is using {{kib}}'s [{{dev-tools-app}} console](/explore-analyze/query-filter/tools/console.md). Data streams, templates, and index-level operations can also be manged via {{kib}}'s [Index management](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html) panel. diff --git a/solutions/observability/apps/explore-mobile-sessions-with-discover.md b/solutions/observability/apps/explore-mobile-sessions-with-discover.md index b4d5635368..b92b9dba1d 100644 --- a/solutions/observability/apps/explore-mobile-sessions-with-discover.md +++ b/solutions/observability/apps/explore-mobile-sessions-with-discover.md @@ -7,8 +7,7 @@ applies_to: # Explore mobile sessions with Discover [apm-mobile-session-explorer] -Elastic Mobile APM provides session tracking by attaching a `session.id`, a guid, to every span and event. This allows for the recall of the activities of a specific user during a specific period of time. The best way recall these data points is using [Discover](../../../explore-analyze/discover/document-explorer.md). This guide will explain how to do that. - +Elastic Mobile APM provides session tracking by attaching a `session.id`, a guid, to every span and event. This allows for the recall of the activities of a specific user during a specific period of time. The best way recall these data points is using [Discover](/explore-analyze/discover/document-explorer.md). This guide will explain how to do that. ## Viewing sessions with Discover [viewing-sessions-with-discover] diff --git a/solutions/observability/apps/filter-application-data.md b/solutions/observability/apps/filter-application-data.md index 651afe5137..0eab5b3112 100644 --- a/solutions/observability/apps/filter-application-data.md +++ b/solutions/observability/apps/filter-application-data.md @@ -10,7 +10,6 @@ applies_to: # Filter application data [apm-filter-your-data] - Global filters are ways you can filter your APM data based on a specific time range or environment. When viewing a specific service, the filter persists as you move between tabs. :::{image} /solutions/images/observability-global-filters.png @@ -19,17 +18,14 @@ Global filters are ways you can filter your APM data based on a specific time ra ::: ::::{note} -If you prefer to use advanced queries on your data to filter on specific pieces of information, see [Query your data](../../../solutions/observability/apps/use-advanced-queries-on-application-data.md). +If you prefer to use advanced queries on your data to filter on specific pieces of information, see [Query your data](/solutions/observability/apps/use-advanced-queries-on-application-data.md). :::: - - ## Global time range [apm-filter-your-data-global-time-range] The global time range filter restricts APM data to a specific time period. - ## Service environment filter [apm-filter-your-data-service-environment-filter] The environment selector is a global filter for `service.environment`. It allows you to view only relevant data and is especially useful for separating development from production environments. By default, all environments are displayed. If there are no environment options, you’ll see "not defined". diff --git a/solutions/observability/apps/filter-search-application-data.md b/solutions/observability/apps/filter-search-application-data.md index 3c7313b044..061176e072 100644 --- a/solutions/observability/apps/filter-search-application-data.md +++ b/solutions/observability/apps/filter-search-application-data.md @@ -10,9 +10,8 @@ applies_to: # Filter and search application data [apm-filter-and-search-data] - Because Elastic APM is built on top of the {{stack}}, you have the full power of Elastic’s powerful search capabilities to filter and search through your application data. Mastering how to filter and search your data can help you find bottlenecks in your code faster: -* Use global filters to [filter data](../../../solutions/observability/apps/filter-application-data.md) across the Applications UI based on a specific time range or environment. -* Use [advanced queries](../../../solutions/observability/apps/use-advanced-queries-on-application-data.md) on your data to filter on specific pieces of information. -* Use {{es}}'s [cross-cluster search](../../../solutions/observability/apps/cross-cluster-search-with-application-data.md) functionality to search APM data across multiple sources. \ No newline at end of file +* Use global filters to [filter data](/solutions/observability/apps/filter-application-data.md) across the Applications UI based on a specific time range or environment. +* Use [advanced queries](/solutions/observability/apps/use-advanced-queries-on-application-data.md) on your data to filter on specific pieces of information. +* Use {{es}}'s [cross-cluster search](/solutions/observability/apps/cross-cluster-search-with-application-data.md) functionality to search APM data across multiple sources. \ No newline at end of file diff --git a/solutions/observability/apps/find-transaction-latency-failure-correlations.md b/solutions/observability/apps/find-transaction-latency-failure-correlations.md index 60d93bc4a3..81df502c30 100644 --- a/solutions/observability/apps/find-transaction-latency-failure-correlations.md +++ b/solutions/observability/apps/find-transaction-latency-failure-correlations.md @@ -21,7 +21,6 @@ To find correlations: Select a service on the **Services** page in the Applications UI then select a transaction group from the **Transactions** tab. - ::: :::{tab-item} Serverless @@ -36,11 +35,10 @@ Select a service on the **Services** page in the Applications UI then select a t :::: -:::: {note} +::::{note} Active queries *are* applied to correlations. :::: - ## Find high transaction latency correlations [observability-apm-find-transaction-latency-and-failure-correlations-find-high-transaction-latency-correlations] The correlations on the **Latency correlations** tab help you discover which attributes are contributing to increased transaction latency. @@ -63,7 +61,6 @@ You can also click the icon beside the field name to view and filter its most po In this example screenshot, there are transactions that are skewed to the right with slower response times than the overall latency distribution. If you select the `+` filter in the appropriate row of the table, it creates a new query in the Applications UI for transactions with this attribute. With the "noise" now filtered out, you can begin viewing sample traces to continue your investigation. - ## Find failed transaction correlations [correlations-error-rate] The correlations on the **Failed transaction correlations** tab help you discover which attributes are most influential in distinguishing between transaction failures and successes. In this context, the success or failure of a transaction is determined by its [event.outcome](ecs://reference/ecs-event.md#field-event-outcome) value. For example, APM agents set the `event.outcome` to `failure` when an HTTP transaction returns a `5xx` status code. diff --git a/solutions/observability/apps/fleet-managed-apm-server.md b/solutions/observability/apps/fleet-managed-apm-server.md index 30bc00b353..19b92cf2a6 100644 --- a/solutions/observability/apps/fleet-managed-apm-server.md +++ b/solutions/observability/apps/fleet-managed-apm-server.md @@ -9,13 +9,12 @@ applies_to: This guide will explain how to set up and configure a Fleet-managed APM Server. - ## Prerequisites [_prerequisites_6] You need {{es}} for storing and searching your data, and {{kib}} for visualizing and managing it. When setting these components up, you need: -* {{es}} cluster and {{kib}} (version 9.0) with a basic license or higher. [Learn how to install the {{stack}} on your own hardware](../../../get-started/the-stack.md). -* Secure, encrypted connection between {{kib}} and {{es}}. For more information, see [Start the {{stack}} with security enabled](../../../deploy-manage/deploy/self-managed/installing-elasticsearch.md). +* {{es}} cluster and {{kib}} (version 9.0) with a basic license or higher. [Learn how to install the {{stack}} on your own hardware](/get-started/the-stack.md). +* Secure, encrypted connection between {{kib}} and {{es}}. For more information, see [Start the {{stack}} with security enabled](/deploy-manage/deploy/self-managed/installing-elasticsearch.md). * Internet connection for {{kib}} to download integration packages from the {{package-registry}}. Make sure the {{kib}} server can connect to `https://epr.elastic.co` on port `443`. If your environment has network traffic restrictions, there are ways to work around this requirement. See [Air-gapped environments](/reference/fleet/air-gapped.md) for more information. * {{kib}} user with `All` privileges on {{fleet}} and {{integrations}}. Since many Integrations assets are shared across spaces, users need the {{kib}} privileges in all spaces. * In the {{es}} configuration, the [built-in API key service](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#api-key-service-settings) must be enabled. (`xpack.security.authc.api_key.enabled: true`) @@ -39,9 +38,7 @@ elasticsearch.username: "kibana_system" <1> xpack.encryptedSavedObjects.encryptionKey: "something_at_least_32_characters" ``` -1. The password should be stored in the {{kib}} keystore as described in the [{{es}} security documentation](../../../deploy-manage/security/set-up-minimal-security.md). - - +1. The password should be stored in the {{kib}} keystore as described in the [{{es}} security documentation](/deploy-manage/security/set-up-minimal-security.md). ## Step 1: Set up Fleet [_step_1_set_up_fleet] @@ -52,7 +49,6 @@ If you already have a {{fleet-server}} set up, you can choose to skip this step. :::: - The first time you use {{fleet}}, you’ll need to set it up and add a {{fleet-server}} using the steps outlined below. To deploy a self-managed {{fleet-server}}, you install an {{agent}} and enroll it in an agent policy containing the {{fleet-server}} integration. @@ -61,7 +57,6 @@ To deploy a self-managed {{fleet-server}}, you install an {{agent}} and enroll i You can install only a single {{agent}} per host, which means you cannot run {{fleet-server}} and another {{agent}} on the same host unless you deploy a containerized {{fleet-server}}. :::: - 1. In {{fleet}}, open the **Settings** tab. For more information about these settings, see [{{fleet}} settings](/reference/fleet/fleet-settings.md). 2. Under **Fleet Server hosts**, click **Edit hosts** and specify one or more host URLs your {{agent}}s will use to connect to {{fleet-server}}. For example, `https://192.0.2.1:8220`, where `192.0.2.1` is the host IP where you will install {{fleet-server}}. Save and apply your settings. @@ -78,7 +73,6 @@ You can install only a single {{agent}} per host, which means you cannot run {{f :screenshot: ::: - **Notes:** * Choose **Quick Start** if you want {{fleet}} to generate a {{fleet-server}} policy and enrollment token for you. The {{fleet-server}} policy will include a {{fleet-server}} integration plus a system integration for monitoring {{agent}}. This option generates self-signed certificates and is not recommended for production use cases. @@ -101,10 +95,8 @@ If installation is successful, you’ll see confirmation that {{fleet-server}} c If you’re unable to add a {{fleet}}-managed agent, click the **Agents** tab and confirm that the agent running {{fleet-server}} is healthy. :::: - For more information, refer to [{{fleet-server}}](/reference/fleet/fleet-server.md). - ## Step 2: Add and configure the APM integration [add-apm-integration] ::::{note} @@ -112,7 +104,6 @@ If you don’t have a {{fleet}} setup already in place, the easiest way to get s :::: - 1. In {{kib}}, find **Integrations** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). 2. Select **Elastic APM**. @@ -145,7 +136,6 @@ If you don’t have a {{fleet}} setup already in place, the easiest way to get s Any {{agent}}s assigned to this policy will collect APM data from your instrumented services. - ::::{tip} An internet connection is required to install the APM integration via the Fleet UI in Kibana. @@ -163,7 +153,6 @@ xpack.fleet.packages: See [Configure Kibana](kibana://reference/configuration-reference/general-settings.md) to learn more about how to edit the Kibana configuration file. - Option 2: Use the {{fleet}} API : Use the {{fleet}} API to install the APM integration. To be successful, this needs to be run against the {{kib}} API, not the {{es}} API. @@ -174,14 +163,10 @@ POST kbn:/api/fleet/epm/packages/apm/9.0.0 See [Kibana API](https://www.elastic.co/guide/en/kibana/current/api.html) to learn more about how to use the Kibana APIs. - :::: - :::: - - ## Step 3: Install APM agents [_step_3_install_apm_agents] APM agents are written in the same language as your service. To monitor a new service, you must install the agent and configure it with a service name, APM Server host, and Secret token. @@ -194,7 +179,6 @@ APM agents are written in the same language as your service. To monitor a new se You can edit your APM integration settings if you need to change the APM Server URL or secret token to match your APM agents. :::: - :::::::{tab-set} ::::::{tab-item} Android @@ -212,7 +196,6 @@ plugins { 1. The Elastic plugin declaration must be added below the Android app plugin declaration (`com.android.application`) and below the Kotlin plugin declaration (if used). - **2. Configure the agent** After adding the agent plugin, configure it. A minimal configuration sets the Elastic APM integration endpoint as shown below: @@ -242,12 +225,10 @@ elasticApm { 4. Defaults to null. More info on API Keys [here](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key). 5. Defaults to null. - ::::{note} When both `secretToken` and `apiKey` are provided, apiKey has priority and secretToken is ignored. :::: - **3. Initialize the agent** After syncing your project with the Gradle changes above, the Elastic APM agent needs to be initialized within your [Application class](https://developer.android.com/reference/android/app/Application). This example shows the simplest way to configure the agent: @@ -267,7 +248,6 @@ class MyApp extends android.app.Application { 1. Initialize the Elastic APM agent once. - All that’s left is to compile and run your application. That’s it! **Learn more in the agent reference** @@ -375,7 +355,6 @@ struct MyApp: App { 1. The APM integration host and port 2. Secret token for APM integration connection - If you’re not using `SwiftUI`, you can alternatively add the same thing to your `AppDelegate.swift` file: ```swift @@ -397,7 +376,6 @@ var config = AgentConfigBuilder() 1. The APM integration host and port 2. Secret token for APM integration connection - **Learn more in the agent reference** Read more in the [APM iOS Agent Reference](apm-agent-ios://reference/index.md). @@ -417,7 +395,6 @@ The first step in getting started with the Elastic APM Java agent is to retrieve curl -o 'elastic-apm-agent.jar' -L 'https://oss.sonatype.org/service/local/artifact/maven/redirect?r=releases&g=co.elastic.apm&a=elastic-apm-agent&v=LATEST' ``` - **2. Add `-javaagent` flag** When starting your application, add the JVM flag `-javaagent:/path/to/elastic-apm-agent-.jar` @@ -526,7 +503,6 @@ If you can’t find your distribution, you can install the agent by building it The agent is currently only available for Linux operating system. :::: - 1. Download the agent source from [https://github.com/elastic/apm-agent-php/](https://github.com/elastic/apm-agent-php/). 2. Execute the following commands to build the agent and install it: @@ -607,7 +583,6 @@ MIDDLEWARE = ( ) ``` - Flask : **1. Install the {{apm-agent}}** @@ -644,7 +619,6 @@ app.config['ELASTIC_APM'] = { apm = ElasticAPM(app) ``` - **Learn more in the agent reference** * [Supported technologies](apm-agent-python://reference/supported-technologies.md) @@ -679,7 +653,6 @@ secret_token: '' server_url: 'http://localhost:8200' ``` - Rack : For Rack or a compatible framework, like Sinatra, include the middleware in your app and start the agent. @@ -721,7 +694,6 @@ secret_token: '' server_url: 'http://localhost:8200' ``` - **Learn more in the agent reference** * [Supported technologies](apm-agent-ruby://reference/supported-technologies.md) diff --git a/solutions/observability/apps/general-configuration-options.md b/solutions/observability/apps/general-configuration-options.md index a7989d933f..d4f1b405a0 100644 --- a/solutions/observability/apps/general-configuration-options.md +++ b/solutions/observability/apps/general-configuration-options.md @@ -14,7 +14,6 @@ Most options on this page are supported by all APM Server deployment methods. :::: - General APM Server configuration options. :::::::{tab-set} @@ -45,7 +44,6 @@ Configure and customize Fleet-managed APM settings directly in {{kib}}: ## Configuration options [apm-configuration-apm-server] - ### Host [apm-host] Defines the host and port the server is listening on. Use `"unix:/path/to.sock"` to listen on a Unix domain socket. Defaults to *localhost:8200*. (text) @@ -55,7 +53,6 @@ Defines the host and port the server is listening on. Use `"unix:/path/to.sock"` | APM Server binary | `apm-server.host` | | Fleet-managed | `Host` | - ### URL [_url] The publicly reachable server URL. For deployments on Elastic Cloud or ECK, the default is unchangeable. @@ -65,7 +62,6 @@ The publicly reachable server URL. For deployments on Elastic Cloud or ECK, the | APM Server binary | N/A | | Fleet-managed | `URL` | - ### Max header size [apm-max_header_size] Maximum permitted size of a request’s header accepted by the server to be processed (in Bytes). Defaults to 1048576 Bytes (1 MB). (int) @@ -75,7 +71,6 @@ Maximum permitted size of a request’s header accepted by the server to be proc | APM Server binary | `apm-server.max_header_size` | | Fleet-managed | `Maximum size of a request's header` | - ### Idle timeout [apm-idle_timeout] Maximum amount of time to wait for the next incoming request before underlying connection is closed. Defaults to `45s` (45 seconds). (text) @@ -85,7 +80,6 @@ Maximum amount of time to wait for the next incoming request before underlying c | APM Server binary | `apm-server.idle_timeout` | | Fleet-managed | `Idle time before underlying connection is closed` | - ### Read timeout [apm-read_timeout] Maximum permitted duration for reading an entire request. Defaults to `3600s` (3600 seconds). (text) @@ -95,7 +89,6 @@ Maximum permitted duration for reading an entire request. Defaults to `3600s` (3 | APM Server binary | `apm-server.read_timeout` | | Fleet-managed | `Maximum duration for reading an entire request` | - ### Write timeout [apm-write_timeout] Maximum permitted duration for writing a response. Defaults to `30s` (30 seconds). (text) @@ -105,7 +98,6 @@ Maximum permitted duration for writing a response. Defaults to `30s` (30 seconds | APM Server binary | `apm-server.write_timeout` | | Fleet-managed | `Maximum duration for writing a response` | - #### Shutdown timeout [apm-shutdown_timeout] Maximum duration in seconds before releasing resources when shutting down the server. Defaults to `30s` (30 seconds). (text) @@ -115,7 +107,6 @@ Maximum duration in seconds before releasing resources when shutting down the se | APM Server binary | `apm-server.shutdown_timeout` | | Fleet-managed | `Maximum duration before releasing resources when shutting down` | - ### Max event size [apm-max_event_size] Maximum permitted size of an event accepted by the server to be processed (in Bytes). Defaults to `307200` Bytes. (int) @@ -125,7 +116,6 @@ Maximum permitted size of an event accepted by the server to be processed (in By | APM Server binary | `apm-server.max_event_size` | | Fleet-managed | `Maximum size per event` | - ### Max connections [apm-max_connections] Maximum number of TCP connections to accept simultaneously. Default value is 0, which means *unlimited*. (int) @@ -135,7 +125,6 @@ Maximum number of TCP connections to accept simultaneously. Default value is 0, | APM Server binary | `apm-server.max_connections` | | Fleet-managed | `Simultaneously accepted connections` | - ### Custom HTTP response headers [apm-custom_http_headers] Custom HTTP headers to add to HTTP responses. Useful for security policy compliance. (text) @@ -145,7 +134,6 @@ Custom HTTP headers to add to HTTP responses. Useful for security policy complia | APM Server binary | `apm-server.response_headers` | | Fleet-managed | `Custom HTTP response headers` | - ### Capture personal data [apm-capture_personal_data] If true, APM Server captures the IP of the instrumented service and its User Agent if any. Enabled by default. (bool) @@ -155,7 +143,6 @@ If true, APM Server captures the IP of the instrumented service and its User Age | APM Server binary | `apm-server.capture_personal_data` | | Fleet-managed | `Capture personal data` | - ### Default service environment [apm-default_service_environment] Sets the default service environment to associate with data and requests received from agents which have no service environment defined. Default: none. (text) @@ -165,7 +152,6 @@ Sets the default service environment to associate with data and requests receive | APM Server binary | `apm-server.default_service_environment` | | Fleet-managed | `Default Service Environment` | - ### expvar support [apm-expvar.enabled] When set to true APM Server exposes [golang expvar](https://golang.org/pkg/expvar/) under `/debug/vars`. Disabled by default. @@ -175,7 +161,6 @@ When set to true APM Server exposes [golang expvar](https://golang.org/pkg/expva | APM Server binary | `apm-server.expvar.enabled` | | Fleet-managed | `Enable APM Server Golang expvar support` | - ### expvar URL [apm-expvar.url] Configure the URL to expose expvar. Defaults to `debug/vars`. @@ -185,7 +170,6 @@ Configure the URL to expose expvar. Defaults to `debug/vars`. | APM Server binary | `apm-server.expvar.url` | | Fleet-managed | N/A | - ### Data stream namespace [apm-data_streams.namespace] Change the default namespace. This setting changes the name of the data stream. diff --git a/solutions/observability/apps/get-started-apm-serverless.md b/solutions/observability/apps/get-started-apm-serverless.md new file mode 100644 index 0000000000..b8a2b2a07d --- /dev/null +++ b/solutions/observability/apps/get-started-apm-serverless.md @@ -0,0 +1,448 @@ +--- +navigation_title: "Elastic Cloud Serverless" +mapped_pages: + - https://www.elastic.co/guide/en/serverless/current/observability-apm-get-started.html +applies_to: + serverless: +--- + +# Get started with traces and APM [observability-apm-get-started] + +::::{admonition} Required role +:class: note + +**For Observability Serverless projects**, the **Admin** role or higher is required to send APM data to Elastic. To learn more, refer to [Assign user roles and privileges](/deploy-manage/users-roles/cloud-organization/user-roles.md). +:::: + +In this guide you’ll learn how to collect and send Application Performance Monitoring (APM) data to Elastic, then explore and visualize the data in real time. + +## Step 1: Add data [add-apm-integration-agents] + +You’ll use APM agents to send APM data from your application to Elastic. Elastic offers APM agents written in several languages and supports OpenTelemetry. Which agent you’ll use depends on the language used in your service. + +To send APM data to Elastic, you must install an APM agent and configure it to send data to your project: + +1. [Create a new {{obs-serverless}} project](/solutions/observability/get-started/create-an-observability-project.md), or open an existing one. +2. To install and configure one or more APM agents, do one of following: + + * In your Observability project, go to **Add data** → **Application** → **Elastic APM** and follow the prompts. + * Use the following instructions: + + ::::::{tab-set} + :::::{tab-item} Go + **1. Install the agent** + + Install the {{apm-agent}} package using `go get`: + + ```go + go get -u go.elastic.co/apm/v2 + ``` + + **2. Configure the agent** + + To simplify development and testing, the agent defaults to sending data to Elastic at `http://localhost:8200`. To send data to an alternative location, you must configure `ELASTIC_APM_SERVER_URL`. + + ```go + # The APM integration host and port + export ELASTIC_APM_SERVER_URL= + + # If you do not specify `ELASTIC_APM_SERVICE_NAME`, the Go agent will use the + # executable name. For example, if your executable is called "my-app.exe", then your + # service will be identified as "my-app". + export ELASTIC_APM_SERVICE_NAME= + + # API keys are used to authorize requests to the APM integration + export ELASTIC_APM_API_KEY= + ``` + + **3. Instrument your application** + + Instrumentation is the process of extending your application’s code to report trace data to Elastic APM. Go applications must be instrumented manually at the source code level. To instrument your applications, use one of the following approaches: + + * [Built-in instrumentation modules](apm-agent-go://reference/builtin-modules.md). + * [Custom instrumentation](apm-agent-go://reference/custom-instrumentation.md) and context propagation with the Go Agent API. + + **Learn more in the {{apm-agent}} reference** + + * [Supported technologies](apm-agent-go://reference/supported-technologies.md) + * [Advanced configuration](apm-agent-go://reference/configuration.md) + * [Detailed guide to instrumenting Go source code](apm-agent-go://reference/set-up-apm-go-agent.md) + ::::: + + :::::{tab-item} Java + Manually set up and configure the agent with the `-javaagent` JVM option. No application code change is required, but this requires an application restart. See below for more information on this setup method. + + **1. Download the {{apm-agent}}** + + The first step in getting started with the Elastic APM Java agent is to retrieve a copy of the agent JAR. Java agent releases are published to [Maven central](https://repo.maven.apache.org/maven2/). In order to get a copy you can either: + + * download the [latest agent](https://oss.sonatype.org/service/local/artifact/maven/redirect?r=releases&g=co.elastic.apm&a=elastic-apm-agent&v=LATEST) or a [previous release](https://mvnrepository.com/artifact/co.elastic.apm/elastic-apm-agent) from Maven central. + * download with `curl`: + + ```bash + curl -o 'elastic-apm-agent.jar' -L 'https://oss.sonatype.org/service/local/artifact/maven/redirect?r=releases&g=co.elastic.apm&a=elastic-apm-agent&v=LATEST' + ``` + + **2. Add `-javaagent` flag** + + When starting your application, add the JVM flag: `-javaagent:/path/to/elastic-apm-agent-.jar`. + + **3. Configure** + + Different application servers have different ways of setting the `-javaagent` flag and system properties. Start your application (for example a Spring Boot application or other embedded servers) and add the `-javaagent` JVM flag. Use the `-D` prefix to configure the agent using system properties: + + ```bash + java -javaagent:/path/to/elastic-apm-agent-.jar -Delastic.apm.service_name=my-cool-service -Delastic.apm.application_packages=org.example,org.another.example -Delastic.apm.server_url=http://127.0.0.1:8200 -jar my-application.jar + ``` + + Refer to [Manual setup with `-javaagent` flag](apm-agent-java://reference/setup-javaagent.md) to learn more. + + **Alternate setup methods** + + * **Automatic setup with `apm-agent-attach-cli.jar`** Automatically set up the agent without needing to alter the configuration of your JVM or application server. This method requires no changes to application code or JVM options, and allows attaching to a running JVM. Refer to the [Java agent documentation](apm-agent-java://reference/setup-attach-cli.md) for more information on this setup method. + * **Programmatic API setup to self-attach** Set up the agent with a one-line code change and an extra `apm-agent-attach` dependency. This method requires no changes to JVM options, and the agent artifact is embedded within the packaged application binary. Refer to the [Java agent documentation](apm-agent-java://reference/setup-attach-api.md) for more information on this setup method. + ::::: + + :::::{tab-item} .NET + **Set up the {{apm-agent}}** + + * **Profiler runtime instrumentation**: The agent supports auto instrumentation without any code change and without any recompilation of your projects. See [Profiler auto instrumentation](apm-agent-dotnet://reference/setup-auto-instrumentation.md). + * **NuGet packages**: The agent ships as a set of [NuGet packages](apm-agent-dotnet://reference/nuget-packages.md) available on [nuget.org](https://nuget.org). You can add the Agent and specific instrumentations to a .NET application by referencing one or more of these packages and following the package documentation. + * **Host startup hook**: On .NET Core 3.0+ or .NET 5+, the agent supports auto instrumentation without any code change and without any recompilation of your projects. See [Zero code change setup on .NET Core](apm-agent-dotnet://reference/setup-dotnet-net-core.md) for more details. + + **Learn more in the {{apm-agent}} reference** + + * [Supported technologies](apm-agent-dotnet://reference/supported-technologies.md) + * [Advanced configuration](apm-agent-dotnet://reference/configuration.md) + ::::: + + :::::{tab-item} Node.js + **1. Install the {{apm-agent}}** + + Install the {{apm-agent}} for Node.js as a dependency to your application. + + ```js + npm install elastic-apm-node --save + ``` + + **2. Initialization** + + It’s important that the agent is started before you require *any* other modules in your Node.js application - i.e. before `http` and before your router etc. + + This means that you should probably require and start the agent in your application’s main file (usually `index.js`, `server.js` or `app.js`). + + Here’s a simple example of how Elastic APM is normally required and started: + + ```js + // Add this to the VERY top of the first file loaded in your app + var apm = require('elastic-apm-node').start({ + // Override service name from package.json + // Allowed characters: a-z, A-Z, 0-9, -, _, and space + serviceName: '', + + // API keys are used to authorize requests to the APM integration + apiKey: '', + + // Set custom APM integration host and port (default: http://127.0.0.1:8200) + serverUrl: '', + }) + ``` + + The agent will now monitor the performance of your application and record any uncaught exceptions. + + **Learn more in the {{apm-agent}} reference** + + * [Supported technologies](apm-agent-nodejs://reference/supported-technologies.md) + * [Babel/ES Modules](apm-agent-nodejs://reference/advanced-setup.md) + * [Advanced configuration](apm-agent-nodejs://reference/configuring-agent.md) + ::::: + + :::::{tab-item} PHP + **1. Install the agent** + + Install the PHP agent using one of the [published packages](https://github.com/elastic/apm-agent-php/releases/latest). + + To use the RPM Package (RHEL/CentOS and Fedora): + + ```php + rpm -ivh .rpm + ``` + + To use the DEB package (Debian and Ubuntu): + + ```php + dpkg -i .deb + ``` + + To use the APK package (Alpine): + + ```php + apk add --allow-untrusted .apk + ``` + + If you can’t find your distribution, you can install the agent by building it from the source. The following instructions will build the APM agent using the same docker environment that Elastic uses to build our official packages. + + ::::{note} + The agent is currently only available for Linux operating system. + + :::: + + 1. Download the [agent source](https://github.com/elastic/apm-agent-php/). + 2. Execute the following commands to build the agent and install it: + + ```bash + cd apm-agent-php + # for linux glibc - libc distributions (Ubuntu, Redhat, etc) + export BUILD_ARCHITECTURE=linux-x86-64 + # for linux with musl - libc distributions (Alpine) + export BUILD_ARCHITECTURE=linuxmusl-x86-64 + # provide a path to php-config tool + export PHP_CONFIG=php-config + + # build extensions + make -f .ci/Makefile build + + # run extension tests + PHP_VERSION=`$PHP_CONFIG --version | cut -d'.' -f 1,2` make -f .ci/Makefile run-phpt-tests + + # install agent extensions + sudo cp agent/native/_build/${BUILD_ARCHITECTURE}-release/ext/elastic_apm-*.so `$PHP_CONFIG --extension-dir` + + # install automatic loader + sudo cp agent/native/_build/${BUILD_ARCHITECTURE}-release/loader/code/elastic_apm_loader.so `$PHP_CONFIG --extension-dir` + ``` + + **2. Enable and configure the APM agent** + + Enable and configure your agent inside of the `php.ini` file: + + ```ini + extension=elastic_apm_loader.so + elastic_apm.bootstrap_php_part_file=/agent/php/bootstrap_php_part.php + ``` + + **Learn more in the {{apm-agent}} reference** + + * [Supported technologies](apm-agent-php://reference/supported-technologies.md) + * [Configuration](apm-agent-php://reference/configuration.md) + ::::: + + :::::{tab-item} Python + Django and Flask are two of several frameworks that the Elastic APM Python Agent supports. For a complete list of supported technologies, refer to the [Elastic APM Python Agent documentation](apm-agent-python://reference/supported-technologies.md). + + *Django* + + ```python + $ pip install elastic-apm + ``` + + **1. Install the {{apm-agent}}** + + Install the {{apm-agent}} for Python as a dependency. + + ```python + $ pip install elastic-apm + ``` + + **2. Configure the {{apm-agent}}** + + Agents are libraries that run inside of your application process. APM services are created programmatically based on the `SERVICE_NAME`. + + ```python + # Add the agent to the installed apps + INSTALLED_APPS = ( + 'elasticapm.contrib.django', + # ... + ) + + ELASTIC_APM = { + # Set required service name. Allowed characters: + # a-z, A-Z, 0-9, -, _, and space + 'SERVICE_NAME': '', + + # API keys are used to authorize requests to the APM integration + 'API_KEY': '', + + # Set custom APM integration host and port (default: http://localhost:8200) + 'SERVER_URL': '', + } + + # To send performance metrics, add our tracing middleware: + MIDDLEWARE = ( + 'elasticapm.contrib.django.middleware.TracingMiddleware', + #... + ) + ``` + + *Flask* + + **1. Install the {{apm-agent}}** + + Install the {{apm-agent}} for Python as a dependency. + + ```python + $ pip install elastic-apm[flask] + ``` + + **2. Configure the {{apm-agent}}** + + Agents are libraries that run inside of your application process. APM services are created programmatically based on the `SERVICE_NAME`. + + ```python + # initialize using environment variables + from elasticapm.contrib.flask import ElasticAPM + app = Flask(__name__) + apm = ElasticAPM(app) + + # or configure to use ELASTIC_APM in your application settings + from elasticapm.contrib.flask import ElasticAPM + app.config['ELASTIC_APM'] = { + # Set required service name. Allowed characters: + # a-z, A-Z, 0-9, -, _, and space + 'SERVICE_NAME': '', + + # API keys are used to authorize requests to the APM integration + 'API_KEY': '', + + # Set custom APM integration host and port (default: http://localhost:8200) + 'SERVER_URL': '', + } + + apm = ElasticAPM(app) + ``` + + **Learn more in the {{apm-agent}} reference** + + * [Supported technologies](apm-agent-python://reference/supported-technologies.md) + * [Advanced configuration](apm-agent-python://reference/configuration.md) + ::::: + + :::::{tab-item} Ruby + **1. Install the {{apm-agent}}** + + Add the agent to your Gemfile. + + ```ruby + gem 'elastic-apm' + ``` + + **2. Configure the agent** + + *Ruby on Rails* + + APM is automatically started when your app boots. Configure the agent by creating the config file `config/elastic_apm.yml`: + + ```ruby + # config/elastic_apm.yml: + + # Set service name - allowed characters: a-z, A-Z, 0-9, -, _ and space + # Defaults to the name of your Rails app + service_name: 'my-service' + + # API keys are used to authorize requests to the APM integration + api_key: '' + + # Set custom APM integration host and port (default: http://localhost:8200) + server_url: 'http://localhost:8200' + ``` + + *Rack* + + For Rack or a compatible framework, like Sinatra, include the middleware in your app and start the agent. + + ```ruby + # config.ru + + app = lambda do |env| + [200, {'Content-Type' => 'text/plain'}, ['ok']] + end + + # Wraps all requests in transactions and reports exceptions + use ElasticAPM::Middleware + + # Start an instance of the Agent + ElasticAPM.start(service_name: 'NothingButRack') + + run app + + # Gracefully stop the agent when process exits. + # Makes sure any pending transactions are sent. + at_exit { ElasticAPM.stop } + ``` + + Create a config file `config/elastic_apm.yml`: + + ```ruby + # config/elastic_apm.yml: + + # Set service name - allowed characters: a-z, A-Z, 0-9, -, _ and space + # Defaults to the name of your Rack app's class. + service_name: 'my-service' + + # API keys are used to authorize requests to the APM integration + api_key: '' + + # Set custom APM integration host and port (default: http://localhost:8200) + server_url: 'http://localhost:8200' + ``` + + **Learn more in the {{apm-agent}} reference** + + * [Supported technologies](apm-agent-ruby://reference/supported-technologies.md) + * [Advanced configuration](apm-agent-ruby://reference/configuration.md) + ::::: + + :::::{tab-item} OpenTelemetry + Elastic integrates with OpenTelemetry, allowing you to reuse your existing instrumentation to easily send observability data to Elastic. + + For more information on how to combine Elastic and OpenTelemetry, refer to [OpenTelemetry](/solutions/observability/apps/use-opentelemetry-with-apm.md). + ::::: + :::::: + + While there are many configuration options, all APM agents require: + + * **Service name**: The APM integration maps an instrumented service’s name — defined in each {{apm-agent}}'s configuration — to the index where its data is stored. Service names are case-insensitive and must be unique. + + For example, you cannot have a service named `Foo` and another named `foo`. Special characters will be removed from service names and replaced with underscores (`_`). + + * **Server URL**: The host and port that the managed intake service listens for events on. + + To find the URL for your project: + + 1. Go to the [Cloud console](https://cloud.elastic.co/). + 2. Next to your project, select **Manage**. + 3. Next to *Endpoints*, select **View**. + 4. Copy the *APM endpoint*. + + * **API key**: Authentication method for communication between {{apm-agent}} and the managed intake service. + + You can create and delete API keys in Applications Settings: + + 1. Go to any page in the *Applications* section of the main menu. + 2. Click **Settings** in the top bar. + 3. Go to the **Agent keys** tab. + + * **Environment**: The name of the environment this service is deployed in, for example "production" or "staging". + + Environments allow you to easily filter data on a global level in the UI. It’s important to be consistent when naming environments across agents. + +3. If you’re using the step-by-step instructions in the UI, after you’ve installed and configured an agent, you can click **Check Agent Status** to verify that the agent is sending data. + +To learn more about APM agents, including how to fine-tune how agents send traces to Elastic, refer to [Collect application data](/solutions/observability/apps/collect-application-data.md). + +## Step 2: View your data [view-apm-integration-data] + +After one or more APM agents are installed and successfully sending data, you can view application performance monitoring data in the UI. + +In the *Applications* section of the main menu, select **Service Inventory**. This will show a high-level overview of the health and general performance of all your services. + +Learn more about visualizing APM data in [View and analyze data](/solutions/observability/apps/view-analyze-data.md). + +::::{tip} +Not seeing any data? Find helpful tips in [Troubleshooting](/troubleshoot/observability/apm.md). + +:::: + +## Next steps [observability-apm-get-started-next-steps] + +Now that data is streaming into your project, take your investigation to a deeper level. Learn how to use [Elastic’s built-in visualizations for APM data](/solutions/observability/apps/view-analyze-data.md), [alert on APM data](/solutions/observability/incident-management/alerting.md), or [fine-tune how agents send traces to Elastic](/solutions/observability/apps/collect-application-data.md). diff --git a/solutions/observability/apps/get-started-with-apm.md b/solutions/observability/apps/get-started-with-apm.md index f89d5c8214..f3f751d9ec 100644 --- a/solutions/observability/apps/get-started-with-apm.md +++ b/solutions/observability/apps/get-started-with-apm.md @@ -10,20 +10,30 @@ applies_to: # Get started with APM [apm-getting-started-apm-server] - ::::{note} Starting in version 8.15.0, the {{es}} apm-data plugin manages APM index templates, lifecycle policies, and ingest pipelines. :::: - The APM Server receives performance data from your APM agents, validates and processes it, and then transforms the data into {{es}} documents. If you’re on this page, then you’ve chosen to self-manage the Elastic Stack, and you now must decide how to run and configure the APM Server. There are two options, and the components required are different for each: -* **[Fleet-managed APM Server](../../../solutions/observability/apps/get-started-with-apm.md#apm-setup-fleet-managed-apm)** -* **[APM Server binary](../../../solutions/observability/apps/get-started-with-apm.md#apm-setup-apm-server-binary)** +* **[Elastic Cloud Serverless](/solutions/observability/apps/get-started-with-apm.md#get-started-apm-serverless)** +* **[Fleet-managed APM Server](/solutions/observability/apps/get-started-with-apm.md#apm-setup-fleet-managed-apm)** +* **[APM Server binary](/solutions/observability/apps/get-started-with-apm.md#apm-setup-apm-server-binary)** + +## Elastic Cloud Serverless [get-started-apm-serverless] + +```{applies_to} +serverless: +``` +Elastic Cloud Serverless is a fully managed solution that allows you to deploy and use Elastic for your use cases without managing the underlying infrastructure. ## Fleet-managed APM Server [apm-setup-fleet-managed-apm] +```{applies_to} +stack: +``` + Fleet is a web-based UI in {{kib}} that is used to centrally manage {{agent}}s. In this deployment model, use {{agent}} to spin up APM Server instances that can be centrally-managed in a custom-curated user interface. :::{image} /solutions/images/observability-fm-ov.png @@ -44,7 +54,6 @@ Fleet is a web-based UI in {{kib}} that is used to centrally manage {{agent}}s. Fleet-managed APM Server does *not* support all the outputs that are supported by the APM Server binary method of running Elastic APM. :::: - **Required components**: * APM agents @@ -52,9 +61,12 @@ Fleet-managed APM Server does *not* support all the outputs that are supported b **Configuration method**: {{kib}} UI - ## APM Server binary [apm-setup-apm-server-binary] +```{applies_to} +stack: +``` + Install, configure, and run the APM Server binary wherever you need it. :::{image} /solutions/images/observability-bin-ov.png @@ -85,18 +97,15 @@ Install, configure, and run the APM Server binary wherever you need it. **Configuration method**: YAML - ## Help me decide [_help_me_decide] +```{applies_to} +stack: +``` + This decision tree highlights key factors to help you make an informed decision about implementing Elastic APM. It provides practical guidance and is not intended to serve as a comprehensive reference of all possible implementations and capabilities. :::{image} /solutions/images/observability-apm-help-me-decide.svg :alt: APM decision tree :screenshot: ::: - -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [ ] ./raw-migrated-files/docs-content/serverless/observability-apm-get-started.md \ No newline at end of file diff --git a/solutions/observability/apps/get-started-with-uptime.md b/solutions/observability/apps/get-started-with-uptime.md index da0134e998..676763d32f 100644 --- a/solutions/observability/apps/get-started-with-uptime.md +++ b/solutions/observability/apps/get-started-with-uptime.md @@ -34,15 +34,12 @@ If you’ve used the Elastic Synthetics integration to create monitors in the pa :::: - - ## Pull the Docker image [uptime-set-up-docker] Elastic provides Docker images that you can use to run monitors. Start by pulling the {{heartbeat}} Docker image. Version 9.0.0-beta1 has not yet been released. - ## Configure [uptime-set-up-config] Next, create a `heartbeat.yml` configuration file. @@ -62,7 +59,6 @@ heartbeat.monitors: 1. Each `monitor` gets its own ID in the {{uptime-app}} and its own schedule entry. This allows tests to be run in parallel and analyzed separately. - Read more about configuration options in [Configure {{heartbeat}} monitors](beats://reference/heartbeat/configuration-heartbeat-options.md). ::::{warning} @@ -72,22 +68,18 @@ If you previously used {{heartbeat}} to set up **`browser`** monitor, you can fi :::: - - ## Connect to the {{stack}} [uptime-set-up-connect] After configuring the monitor, run it in Docker and connect the monitor to the {{stack}}. Version 9.0.0-beta1 has not yet been released. - ## View in {{kib}} [uptime-set-up-kibana] {{heartbeat}} is now sending synthetic monitoring data to the {{stack}}. Navigate to the {{uptime-app}} in {{kib}}, where you can see screenshots of each run, set up alerts in case of test failures, and more. If a test does fail (shown as `down` in the {{uptime-app}}), you’ll be able to view the step script that failed, any errors, and a stack trace. For more information, refer to [Analyze](analyze.md). - ## Manage monitors [uptime-manage] After you’ve created a monitor, you’ll need to manage that monitor over time. This might include updating or permanently deleting an existing monitor. diff --git a/solutions/observability/apps/get-started.md b/solutions/observability/apps/get-started.md index 021543f223..721c73d8bd 100644 --- a/solutions/observability/apps/get-started.md +++ b/solutions/observability/apps/get-started.md @@ -18,25 +18,23 @@ There are two ways to set up a synthetic monitor: Read more about each option below, and choose the approach that works best for you. - ## Synthetics project [observability-synthetics-get-started-synthetics-project] With a Synthetics project, you write tests in an external version-controlled Node.js project using YAML for lightweight monitors and JavaScript or TypeScript for browser monitors. Then, you use the `@elastic/synthetics` NPM library’s `push` command to create monitors. This approach works well if you want to create both browser monitors and lightweight monitors. It also allows you to configure and update monitors using a GitOps workflow. -Get started in [Create monitors in a Synthetics project](../../../solutions/observability/apps/create-monitors-with-project-monitors.md). +Get started in [Create monitors in a Synthetics project](/solutions/observability/apps/create-monitors-with-project-monitors.md). :::{image} /solutions/images/observability-synthetics-get-started-projects.png :alt: Diagram showing which pieces of software are used to configure monitors ::: - ## Synthetics UI [observability-synthetics-get-started-synthetics-ui] You can create monitors directly in the user interface. This approach works well if you want to create and manage your monitors in the browser. -Get started in [Create monitors in the Synthetics UI](../../../solutions/observability/apps/create-monitors-in-synthetics-app.md). +Get started in [Create monitors in the Synthetics UI](/solutions/observability/apps/create-monitors-in-synthetics-app.md). :::{image} /solutions/images/observability-synthetics-get-started-ui.png :alt: Diagram showing which pieces of software are used to configure monitors @@ -45,7 +43,7 @@ Get started in [Create monitors in the Synthetics UI](../../../solutions/observa ::::{note} The Elastic Synthetics integration is a method for creating synthetic monitors that is no longer recommended. **Do not use the Elastic Synthetics integration to set up new monitors.** -For details on how to migrate from Elastic Synthetics integration to {{project-monitors}} or the {{synthetics-app}}, refer to [Migrate from the Elastic Synthetics integration](../../../solutions/observability/apps/migrate-from-elastic-synthetics-integration.md). +For details on how to migrate from Elastic Synthetics integration to {{project-monitors}} or the {{synthetics-app}}, refer to [Migrate from the Elastic Synthetics integration](/solutions/observability/apps/migrate-from-elastic-synthetics-integration.md). If you’ve used the Elastic Synthetics integration to create monitors in the past and need to reference documentation about the integration, go to the [8.3 documentation](https://www.elastic.co/guide/en/observability/8.3/uptime-set-up.html#uptime-set-up-choose-agent). diff --git a/solutions/observability/apps/grant-access-using-api-keys.md b/solutions/observability/apps/grant-access-using-api-keys.md index 165a3474ce..b50e67cb42 100644 --- a/solutions/observability/apps/grant-access-using-api-keys.md +++ b/solutions/observability/apps/grant-access-using-api-keys.md @@ -15,8 +15,6 @@ APM Server instances typically send both collected data and monitoring informati For security reasons, we recommend using a unique API key per APM Server instance. You can create as many API keys per user as necessary. :::: - - ## Create an API key for writing events [apm-beats-api-key-publish] To create an API key: @@ -74,7 +72,6 @@ To create an API key: :alt: API key dropdown highlighting the Beats option ::: - You can now use this API key in your `apm-server.yml` configuration file: ```yaml @@ -84,8 +81,6 @@ output.elasticsearch: 1. Format is `id:api_key` (as shown in the {{beats}} dropdown) - - ## Create an API key for monitoring [apm-beats-api-key-monitor] To open **API keys**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). Click **Create API key**. @@ -114,7 +109,6 @@ Enter a name for your API key and select **Restrict privileges**. In the role de This example only provides privileges for **publishing monitoring data**. See [Use feature roles](create-assign-feature-roles-to-apm-server-users.md) for additional privileges and information. :::: - To set an expiration date for the API key, select **Expire after time** and input the lifetime of the API key in days. Click **Create API key**. In the dropdown, switch to **{{beats}}** and copy the API key. @@ -128,8 +122,6 @@ monitoring.elasticsearch: 1. Format is `id:api_key` (as shown in the {{beats}} dropdown) - - ## Create an API key with {{es}} APIs [apm-beats-api-key-es] You can also use {{es}}'s [Create API key API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key) to create a new API key. For example: @@ -172,10 +164,8 @@ POST /_security/api_key 1. Name of the API key 2. Granted privileges, see [Use feature roles](create-assign-feature-roles-to-apm-server-users.md) - See the [Create API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key) reference for more information. - ## Learn more about API keys [apm-learn-more-api-keys] See the {{es}} API key documentation for more information: diff --git a/solutions/observability/apps/grant-users-access-to-secured-resources.md b/solutions/observability/apps/grant-users-access-to-secured-resources.md index b3e88387ea..70a90c41b7 100644 --- a/solutions/observability/apps/grant-users-access-to-secured-resources.md +++ b/solutions/observability/apps/grant-users-access-to-secured-resources.md @@ -19,11 +19,11 @@ You can use role-based access control to grant users access to secured resources Typically you need the create the following separate roles: -* [Setup role](../../../solutions/observability/apps/setup-role.md) for enabling Monitor Management. -* [Writer role](../../../solutions/observability/apps/writer-role.md) for creating, modifying, and deleting monitors. -* [Reader role](../../../solutions/observability/apps/reader-role.md) for {{kib}} users who need to view and create visualizations that access Synthetics data. +* [Setup role](/solutions/observability/apps/setup-role.md) for enabling Monitor Management. +* [Writer role](/solutions/observability/apps/writer-role.md) for creating, modifying, and deleting monitors. +* [Reader role](/solutions/observability/apps/reader-role.md) for {{kib}} users who need to view and create visualizations that access Synthetics data. -{{es-security-features}} provides [built-in roles](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) that grant a subset of the privileges needed by Synthetics users. When possible, assign users the built-in roles to minimize the affect of future changes on your security strategy. If no built-in role is available, you can assign users the privileges needed to accomplish a specific task. +{{es-security-features}} provides [built-in roles](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) that grant a subset of the privileges needed by Synthetics users. When possible, assign users the built-in roles to minimize the affect of future changes on your security strategy. If no built-in role is available, you can assign users the privileges needed to accomplish a specific task. In general, these are types of privileges you’ll work with: @@ -51,8 +51,7 @@ In general, these are types of privileges you’ll work with: * Create, modify, and delete monitors. * View and create visualizations that access Synthetics data. - -Read more about user roles in [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md). +Read more about user roles in [Assign user roles and privileges](/deploy-manage/users-roles/cloud-organization/user-roles.md). ::: diff --git a/solutions/observability/apps/high-availability.md b/solutions/observability/apps/high-availability.md index 605e2045b6..f46fbbd1e4 100644 --- a/solutions/observability/apps/high-availability.md +++ b/solutions/observability/apps/high-availability.md @@ -19,4 +19,3 @@ If {{es}} goes down, the APM Server will eventually deny incoming requests. Both Fleet-managed APM Server users might also be interested in [Fleet/Agent proxy support](/reference/fleet/fleet-agent-proxy-support.md). :::: - diff --git a/solutions/observability/apps/index-lifecycle-management.md b/solutions/observability/apps/index-lifecycle-management.md index 87242a7535..2e8d253c3d 100644 --- a/solutions/observability/apps/index-lifecycle-management.md +++ b/solutions/observability/apps/index-lifecycle-management.md @@ -9,15 +9,13 @@ applies_to: Lifecycle policies allow you to automate the lifecycle of your APM indices as they grow and age. A default policy is applied to each APM data stream, but can be customized depending on your business needs. -In the latest version of Elastic APM, clusters are managed by [index lifecycle management (ILM)](../../../manage-data/lifecycle/index-lifecycle-management.md) to provide the default data retention settings for APM data as well as allow customization for lifecycle. +In the latest version of Elastic APM, clusters are managed by [index lifecycle management (ILM)](/manage-data/lifecycle/index-lifecycle-management.md) to provide the default data retention settings for APM data as well as allow customization for lifecycle. ::::{note} -Indices created in 8.15.x and 8.16.x might be managed by [data stream lifecycle (DSL)](../../../manage-data/lifecycle/data-stream.md). More details are available in [APM version 8.15](https://www.elastic.co/guide/en/observability/current/apm-release-notes-8.15.html). By default, new indices created in 8.17 are managed by ILM. More details are available in [APM version 8.17](https://www.elastic.co/guide/en/observability/current/apm-release-notes-8.17.html). If you have indices managed by DSL, any custom DSL settings that you specified will *not* automatically apply to the new indices managed by ILM. Instead, you can replicate custom DSL settings in ILM using this guide. +Indices created in 8.15.x and 8.16.x might be managed by [data stream lifecycle (DSL)](/manage-data/lifecycle/data-stream.md). More details are available in [APM version 8.15](https://www.elastic.co/guide/en/observability/current/apm-release-notes-8.15.html). By default, new indices created in 8.17 are managed by ILM. More details are available in [APM version 8.17](https://www.elastic.co/guide/en/observability/current/apm-release-notes-8.17.html). If you have indices managed by DSL, any custom DSL settings that you specified will *not* automatically apply to the new indices managed by ILM. Instead, you can replicate custom DSL settings in ILM using this guide. :::: - - ## Default policies [index-lifecycle-policies-default] Each APM data stream has its own default lifecycle policy including a delete definition and a rollover definition. @@ -51,15 +49,12 @@ Rollover (writing to a new index) prevents a single index from growing too large Default lifecycle policies can change between minor versions. This is not considered a breaking change as index management should continually improve and adapt to new features. :::: - - ## Configure a custom index lifecycle policy [apm-data-streams-custom-policy] Mappings and settings for data streams can be customized through the creation of `*@custom` component templates, which are referenced by the index templates created by the {{es}} apm-data plugin. The easiest way to configure a custom index lifecycle policy per data stream is to edit this template. This tutorial explains how to apply a custom index lifecycle policy to the `traces-apm` data stream. - ## Step 1: View data streams [apm-data-streams-custom-one] The **Data Streams** view in {{kib}} shows you data streams, index templates, and lifecycle policies: @@ -74,8 +69,6 @@ The **Data Streams** view in {{kib}} shows you data streams, index templates, an :screenshot: ::: - - ## Step 2: Create an index lifecycle policy [apm-data-streams-custom-two] 1. To open **Lifecycle Policies**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). @@ -83,7 +76,6 @@ The **Data Streams** view in {{kib}} shows you data streams, index templates, an Name your new policy; For this tutorial, I’ve chosen `custom-traces-apm-policy`. Customize the policy to your liking, and when you’re done, click **Save policy**. - ## Step 3: Apply the index lifecycle policy [apm-data-streams-custom-three] To apply your new index lifecycle policy to the `traces-apm-*` data stream, edit the `@custom` component template. @@ -108,8 +100,6 @@ To apply your new index lifecycle policy to the `traces-apm-*` data stream, edit :screenshot: ::: - - ## Step 4: Roll over the data stream (optional) [apm-data-streams-custom-four] To confirm that the data stream is now using the new index template and {{ilm-init}} policy, you can either repeat [step one](#apm-data-streams-custom-one), or navigate to **{{dev-tools-app}}** and run the following: @@ -120,7 +110,6 @@ GET /_data_stream/traces-apm-default <1> 1. The name of the data stream we’ve been hacking on appended with your - The result should include the following: ```json @@ -139,14 +128,12 @@ The result should include the following: 1. The name of the custom index template created in step three 2. The name of the {{ilm-init}} policy applied to the new component template in step two - New {{ilm-init}} policies only take effect when new indices are created, so you either must wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB), or force a rollover using the [{{es}} rollover API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover): ```bash POST /traces-apm-default/_rollover/ ``` - ## Namespace-level index lifecycle policies [apm-data-streams-custom-policy-namespace] It is also possible to create more granular index lifecycle policies that apply to individual namespaces. This process is similar to the above tutorial, but includes cloning and modify the existing index template to use a new `*@custom` component template. diff --git a/solutions/observability/apps/infrastructure.md b/solutions/observability/apps/infrastructure.md index dbf7915890..99656b926b 100644 --- a/solutions/observability/apps/infrastructure.md +++ b/solutions/observability/apps/infrastructure.md @@ -15,14 +15,12 @@ The Applications UI Infrastructure functionality is in beta and is subject to ch :::: - The **Infrastructure** tab provides information about the containers, pods, and hosts that the selected service is linked to. -* **Pods**: Uses the `kubernetes.pod.name` from the [APM metrics data streams](../../../solutions/observability/apps/metrics.md). -* **Containers**: Uses the `container.id` from the [APM metrics data streams](../../../solutions/observability/apps/metrics.md). +* **Pods**: Uses the `kubernetes.pod.name` from the [APM metrics data streams](/solutions/observability/apps/metrics.md). +* **Containers**: Uses the `container.id` from the [APM metrics data streams](/solutions/observability/apps/metrics.md). * **Hosts**: If the application is containerized—​if the APM metrics documents include `container.id`-- the `host.name` is used from the infrastructure data streams (filtered by `container.id`). If not, `host.hostname` is used from the APM metrics data streams. - :::{image} /solutions/images/serverless-infra.png :alt: Example view of the Infrastructure tab in the Applications UI :screenshot: @@ -33,6 +31,6 @@ IT ops and software reliability engineers (SREs) can use this tab to quickly fin ::::{tip} **Why is the infrastructure tab empty?** -If there is no data in the Application UI’s infrastructure tab for a selected service, you can read more about why this happens and how to fix it in the [troubleshooting docs](../../../troubleshoot/observability/apm/common-problems.md#troubleshooting-apm-infra-data). +If there is no data in the Application UI’s infrastructure tab for a selected service, you can read more about why this happens and how to fix it in the [troubleshooting docs](/troubleshoot/observability/apm/common-problems.md#troubleshooting-apm-infra-data). :::: \ No newline at end of file diff --git a/solutions/observability/apps/inspect-uptime-duration-anomalies.md b/solutions/observability/apps/inspect-uptime-duration-anomalies.md index d602ed5ea7..4230994b4f 100644 --- a/solutions/observability/apps/inspect-uptime-duration-anomalies.md +++ b/solutions/observability/apps/inspect-uptime-duration-anomalies.md @@ -16,7 +16,6 @@ Use [Synthetic monitoring](/solutions/observability/apps/synthetic-monitoring.md Each monitor location is modeled, and when a monitor runs for an unusual amount of time, at a particular time, an anomaly is recorded and highlighted on the **Monitor duration** chart. - ## Enable uptime duration {{anomaly-detect}} [uptime-anomaly-detection] Create a {{ml}} job to detect anomalous monitor duration rates automatically. diff --git a/solutions/observability/apps/installation-layout.md b/solutions/observability/apps/installation-layout.md index a8524f63fa..87d7387b6c 100644 --- a/solutions/observability/apps/installation-layout.md +++ b/solutions/observability/apps/installation-layout.md @@ -9,7 +9,6 @@ applies_to: View the installation layout and default paths for both Fleet-managed APM Server and the APM Server binary. - ## Fleet-managed [_fleet_managed] {{agent}} files are installed in the following locations. You cannot override these installation paths because they are required for upgrades. diff --git a/solutions/observability/apps/integrate-with-jaeger-deprecated.md b/solutions/observability/apps/integrate-with-jaeger-deprecated.md index 35a60c42c9..80e6a079cc 100644 --- a/solutions/observability/apps/integrate-with-jaeger-deprecated.md +++ b/solutions/observability/apps/integrate-with-jaeger-deprecated.md @@ -6,19 +6,14 @@ applies_to: stack: deprecated --- - - # Integrate with Jaeger (deprecated) [apm-jaeger-integration] - ::::{warning} Support for Jaeger is deprecated and will be removed in a future version of Elastic APM. [Jaeger clients are deprecated](https://www.jaegertracing.io/docs/1.35/client-libraries/) in favor of OpenTelemetry SDKs, and OpenTelemetry has removed all Jaeger exporters from their [specification](https://github.com/open-telemetry/opentelemetry-specification/pull/2858). :::: - Elastic APM integrates with [Jaeger](https://www.jaegertracing.io/), an open-source, distributed tracing system. This integration allows users with an existing Jaeger setup to switch from the default Jaeger backend, to the {{stack}}. Best of all, no instrumentation changes are needed in your application code. - ## Supported architecture [apm-jaeger-architecture] Jaeger architecture supports different data formats and transport protocols that define how data can be sent to a collector. Elastic APM, as a Jaeger collector, supports communication with **Jaeger agents** via gRPC. @@ -29,7 +24,6 @@ Jaeger architecture supports different data formats and transport protocols that See the [Jaeger docs](https://www.jaegertracing.io/docs/1.27/architecture) for more information on Jaeger architecture. - ## Get started [apm-get-started-jaeger] Connect your preexisting Jaeger setup to Elastic APM in three steps: @@ -42,8 +36,6 @@ Connect your preexisting Jaeger setup to Elastic APM in three steps: There are [caveats](#apm-caveats-jaeger) to this integration. :::: - - ### Configure Jaeger agents [apm-configure-agent-client-jaeger] The APM integration serves Jaeger gRPC over the same host and port as the Elastic {{apm-agent}} protocol. @@ -64,7 +56,6 @@ The APM integration serves Jaeger gRPC over the same host and port as the Elasti --agent.tags="elastic-apm-auth=Bearer " ``` - ::::{tip} For the equivalent environment variables, change all letters to upper-case and replace punctuation with underscores (`_`). See the [Jaeger CLI flags documentation](https://www.jaegertracing.io/docs/1.27/cli/) for more information. :::: @@ -95,7 +86,6 @@ For the equivalent environment variables, change all letters to upper-case and r --agent.tags="elastic-apm-auth=Bearer " ``` - ::::{tip} For the equivalent environment variables, change all letters to upper-case and replace punctuation with underscores (`_`). See the [Jaeger CLI flags documentation](https://www.jaegertracing.io/docs/1.27/cli/) for more information. :::: @@ -112,22 +102,18 @@ There are two different ways to configure the sampling rate of your Jaeger agent * [{{apm-agent}} central configuration (default)](#apm-configure-sampling-central-jaeger) * [Local sampling in each Jaeger client](#apm-configure-sampling-local-jaeger) - #### {{apm-agent}} central configuration (default) [apm-configure-sampling-central-jaeger] Central sampling, with {{apm-agent}} central configuration, allows Jaeger clients to poll APM Server for the sampling rate. This means sample rates can be configured on the fly, on a per-service and per-environment basis. See [Central configuration](apm-agent-central-configuration.md) to learn more. - #### Local sampling in each Jaeger client [apm-configure-sampling-local-jaeger] If you don’t have access to the Applications UI, you’ll need to change the Jaeger client’s `sampler.type` and `sampler.param`. This enables you to set the sampling configuration locally in each Jaeger client. See the official [Jaeger sampling documentation](https://www.jaegertracing.io/docs/1.27/sampling/) for more information. - ### Start sending data [apm-configure-start-jaeger] That’s it! Data sent from Jaeger clients to the APM Server can now be viewed in the Applications UI. - ## Caveats [apm-caveats-jaeger] There are some limitations and differences between Elastic APM and Jaeger that you should be aware of. diff --git a/solutions/observability/apps/integrate-with-machine-learning.md b/solutions/observability/apps/integrate-with-machine-learning.md index 66b687cf46..ee780efd0d 100644 --- a/solutions/observability/apps/integrate-with-machine-learning.md +++ b/solutions/observability/apps/integrate-with-machine-learning.md @@ -29,7 +29,6 @@ Results from machine learning jobs are shown in multiple places throughout the A :screenshot: ::: - ## Enable anomaly detection [observability-apm-integrate-with-machine-learning-enable-anomaly-detection] To enable machine learning anomaly detection: @@ -42,16 +41,14 @@ To enable machine learning anomaly detection: That’s it! After a few minutes, the job will begin calculating results; it might take additional time for results to appear on your service maps. To manage existing jobs, click **Manage jobs** (or go to **Machine learning** → **Jobs**). - ## Anomaly detection warning [observability-apm-integrate-with-machine-learning-anomaly-detection-warning] To make machine learning as easy as possible to set up, Elastic will warn you when filtered to an environment without a machine learning job. - ## Unknown service health [observability-apm-integrate-with-machine-learning-unknown-service-health] After enabling anomaly detection, service health may display as "Unknown". Here are some reasons why this can occur: -1. No machine learning job exists. See [Enable anomaly detection](../../../solutions/observability/apps/integrate-with-machine-learning.md#observability-apm-integrate-with-machine-learning-enable-anomaly-detection) to enable anomaly detection and create a machine learning job. +1. No machine learning job exists. See [Enable anomaly detection](/solutions/observability/apps/integrate-with-machine-learning.md#observability-apm-integrate-with-machine-learning-enable-anomaly-detection) to enable anomaly detection and create a machine learning job. 2. There is no machine learning data for the job. If you just created the machine learning job you’ll need to wait a few minutes for data to be available. Alternatively, if the service or its environment are new, you’ll need to wait for more trace data. 3. No "request" or "page-load" transaction type exists for this service; service health is only available for these transaction types. \ No newline at end of file diff --git a/solutions/observability/apps/interpret-application-data.md b/solutions/observability/apps/interpret-application-data.md index 2fc8ac1017..2ae923c0ab 100644 --- a/solutions/observability/apps/interpret-application-data.md +++ b/solutions/observability/apps/interpret-application-data.md @@ -10,14 +10,13 @@ applies_to: # Interpret application data [observability-apm-interpret-data] - Learn how to get the most out of your data using the Applications UI. % Stateful only for exploring mobile sessions with Discover? | | | | --- | --- | -| [Finding transaction latency and failure correlations](../../../solutions/observability/apps/find-transaction-latency-failure-correlations.md) | Surface characteristics of your data that are potentially correlated with high-latency or erroneous transactions. | -| [Tracking deployments with annotations](../../../solutions/observability/apps/track-deployments-with-annotations.md) | Annotations enable you to easily determine if your deployment has increased response times for an end-user or if the memory/CPU footprint of your application has changed. | -| [Exploring mobile sessions with Discover](../../../solutions/observability/apps/explore-mobile-sessions-with-discover.md) | **Elastic Stack only:** Use session tracking via a globally unique identifier to explore the activities of a specific user during a specific period of time. | -| [Observing Lambda functions](../../../solutions/observability/apps/observe-lambda-functions.md) | Learn how your AWS Lambda functions relate to and depend on other services, and get insight into function execution and runtime behavior, like lambda duration, cold start rate, cold start duration, compute usage, memory usage, and more. | \ No newline at end of file +| [Finding transaction latency and failure correlations](/solutions/observability/apps/find-transaction-latency-failure-correlations.md) | Surface characteristics of your data that are potentially correlated with high-latency or erroneous transactions. | +| [Tracking deployments with annotations](/solutions/observability/apps/track-deployments-with-annotations.md) | Annotations enable you to easily determine if your deployment has increased response times for an end-user or if the memory/CPU footprint of your application has changed. | +| [Exploring mobile sessions with Discover](/solutions/observability/apps/explore-mobile-sessions-with-discover.md) | **Elastic Stack only:** Use session tracking via a globally unique identifier to explore the activities of a specific user during a specific period of time. | +| [Observing Lambda functions](/solutions/observability/apps/observe-lambda-functions.md) | Learn how your AWS Lambda functions relate to and depend on other services, and get insight into function execution and runtime behavior, like lambda duration, cold start rate, cold start duration, compute usage, memory usage, and more. | \ No newline at end of file diff --git a/solutions/observability/apps/inventory.md b/solutions/observability/apps/inventory.md index 769cdfabfd..f141910be3 100644 --- a/solutions/observability/apps/inventory.md +++ b/solutions/observability/apps/inventory.md @@ -44,7 +44,6 @@ Inventory allows you to: * Debug resource bottlenecks with your service caused by their containers and the hosts they run on. * Easily discover all entities related to the host, container or service you are viewing by leveraging your tags and labels - ## Explore your entities [explore-your-entities] 1. To view all your entities, find **Inventory** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). @@ -77,12 +76,10 @@ For each entity, you can click the entity name and get a detailed view. For exam If you open an entity of type `host` or `container` that does not have infrastructure data, some of the visualizations will be blank and some features on the page will not be fully populated. - ## Add entities to the Inventory [add-entities-to-inventory] You can add entities to the Inventory through one of the following approaches: **Add data** or **Associate existing service logs**. - ## Add data [add-data-entities] To add entities, select **Add data** and choose one of the following onboarding journeys: @@ -92,7 +89,6 @@ To add entities, select **Add data** and choose one of the following onboarding * **Application** Detects services * **Cloud** Ingests telemetry data from the Cloud - ## Associate existing service logs [associate-existing-service-logs] To learn how, refer to [Add a service name to logs](../logs/add-service-name-to-logs.md). diff --git a/solutions/observability/apps/jaeger-event-intake.md b/solutions/observability/apps/jaeger-event-intake.md index 77412be6f3..3a353c87e3 100644 --- a/solutions/observability/apps/jaeger-event-intake.md +++ b/solutions/observability/apps/jaeger-event-intake.md @@ -12,7 +12,6 @@ applies_to: :::: - Elastic APM natively supports Jaeger, an open-source, distributed tracing system. [Learn more](integrate-with-jaeger-deprecated.md). **Jaeger/gRPC paths** diff --git a/solutions/observability/apps/learn-about-application-data-types.md b/solutions/observability/apps/learn-about-application-data-types.md index 0669dceb38..c683760453 100644 --- a/solutions/observability/apps/learn-about-application-data-types.md +++ b/solutions/observability/apps/learn-about-application-data-types.md @@ -13,9 +13,9 @@ Elastic APM agents capture different types of information from within their inst Elastic APM helps you see what happens from start to finish when a request is made to an application: -* [**Spans**](../../../solutions/observability/apps/spans.md): A span contain information about the execution of a specific code path. They are the building blocks of *transactions* and *traces*. -* [**Transactions**](../../../solutions/observability/apps/transactions.md): A transaction describes an event captured by an Elastic APM agent instrumenting a service. A transaction is technically a type of span that has additional attributes associated with it and often contains multiple child *spans*. You can think of transactions as the highest level of work you’re measuring within a service. -* [**Traces**](../../../solutions/observability/apps/traces.md#apm-distributed-tracing): A trace is a group of *transactions* and *spans* with a common root. Each trace tracks the entirety of a single request. When a trace travels through multiple services, it is known as a *distributed trace*. +* [**Spans**](/solutions/observability/apps/spans.md): A span contain information about the execution of a specific code path. They are the building blocks of *transactions* and *traces*. +* [**Transactions**](/solutions/observability/apps/transactions.md): A transaction describes an event captured by an Elastic APM agent instrumenting a service. A transaction is technically a type of span that has additional attributes associated with it and often contains multiple child *spans*. You can think of transactions as the highest level of work you’re measuring within a service. +* [**Traces**](/solutions/observability/apps/traces.md#apm-distributed-tracing): A trace is a group of *transactions* and *spans* with a common root. Each trace tracks the entirety of a single request. When a trace travels through multiple services, it is known as a *distributed trace*. :::{image} /solutions/images/observability-spans-transactions-and-traces.png :alt: Diagram illustrating the relationship between spans @@ -23,7 +23,7 @@ Elastic APM helps you see what happens from start to finish when a request is ma In addition to the building blocks of traces, Elastic APM agents also capture: -* [**Errors**](../../../solutions/observability/apps/errors.md): An error is created when something goes wrong with a request to an application. This event contains information to help you determine where and why an error occurred, often including in which *transaction* the error occurred. -* [**Metrics**](../../../solutions/observability/apps/metrics.md): Metrics measure the state of a system by gathering information on a regular interval. +* [**Errors**](/solutions/observability/apps/errors.md): An error is created when something goes wrong with a request to an application. This event contains information to help you determine where and why an error occurred, often including in which *transaction* the error occurred. +* [**Metrics**](/solutions/observability/apps/metrics.md): Metrics measure the state of a system by gathering information on a regular interval. -Events can contain additional [metadata](../../../solutions/observability/apps/metadata.md) which further enriches your data. \ No newline at end of file +Events can contain additional [metadata](/solutions/observability/apps/metadata.md) which further enriches your data. \ No newline at end of file diff --git a/solutions/observability/apps/limitations.md b/solutions/observability/apps/limitations.md index 8f912dc69e..ebed4f3f62 100644 --- a/solutions/observability/apps/limitations.md +++ b/solutions/observability/apps/limitations.md @@ -9,30 +9,26 @@ applies_to: # Limitations [apm-open-telemetry-known-limitations] - ## OpenTelemetry traces [apm-open-telemetry-traces-limitations] * Traces of applications using `messaging` semantics might be wrongly displayed as `transactions` in the Applications UI, while they should be considered `spans` (see issue [#7001](https://github.com/elastic/apm-server/issues/7001)). * Inability to see Stack traces in spans. * Inability in APM views to view the "Time Spent by Span Type" (see issue [#5747](https://github.com/elastic/apm-server/issues/5747)). - ## OpenTelemetry logs [apm-open-telemetry-logs-intake] * [preview] The OpenTelemetry logs intake via Elastic is in technical preview. * The application logs data stream (`app_logs`) has dynamic mapping disabled. This means the automatic detection and mapping of new fields is disabled (see issue [#9093](https://github.com/elastic/apm-server/issues/9093)). - ## OpenTelemetry Line Protocol (OTLP) [apm-open-telemetry-otlp-limitations] Elastic supports both the [OTLP/gRPC](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc) and [OTLP/HTTP](https://opentelemetry.io/docs/specs/otlp/#otlphttp) protocol with ProtoBuf payload. Elastic does not yet support JSON Encoding for OTLP/HTTP. - ## OpenTelemetry Collector exporter for Elastic [apm-open-telemetry-collector-exporter] The [OpenTelemetry Collector exporter for Elastic](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.57.2/exporter/elasticexporter) has been deprecated and replaced by the native support of the OpenTelemetry Line Protocol in Elastic Observability (OTLP). -The [OpenTelemetry Collector exporter for Elastic](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter) (which is different from the legacy exporter mentioned above) is not intended to be used with Elastic APM and Elastic Observability. Use [Elastic’s native OTLP support](../../../solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md) instead. +The [OpenTelemetry Collector exporter for Elastic](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter) (which is different from the legacy exporter mentioned above) is not intended to be used with Elastic APM and Elastic Observability. Use [Elastic’s native OTLP support](/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md) instead. % Statefull only for tail-based sampling? @@ -46,7 +42,7 @@ Tail-based sampling allows to make sampling decisions after all spans of a trace When using OpenTelemetry with Elastic APM, there are two different implementations available for tail-based sampling: * Tail-based sampling using the [tailsamplingprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor) in the OpenTelemetry Collector -* Native [tail-based sampling in the Elastic APM backend](../../../solutions/observability/apps/transaction-sampling.md#apm-tail-based-sampling) +* Native [tail-based sampling in the Elastic APM backend](/solutions/observability/apps/transaction-sampling.md#apm-tail-based-sampling) Using the [tailsamplingprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor) in the OpenTelemetry Collector comes with an important limitation. Elastic’s APM backend calculates span and transaction metrics based on the incoming span events. These metrics are accurate for 100% sampling scenarios. In scenarios with probabilistic sampling, Elastic’s APM backend is being informed about the sampling rate of spans and can extrapolate throughput metrics based on the incoming, partial data. However, with tail-based sampling there’s no clear probability for sampling decisions as the rules can be more complex and the OpenTelemetry Collector does not provide sampling probability information to the Elastic backend that could be used for extrapolation of data. Therefore, there’s no way for Elastic APM to properly extrapolate throughput and count metrics that are derived from span events that have been tail-based sampled in the OpenTelemetry Collector. In these scenarios, derived throughput and count metrics are likely to be inaccurate. diff --git a/solutions/observability/apps/logs.md b/solutions/observability/apps/logs.md index 21c15ab4ea..8eb213a59c 100644 --- a/solutions/observability/apps/logs.md +++ b/solutions/observability/apps/logs.md @@ -15,7 +15,7 @@ Logs provide detailed information about specific events, and are crucial to succ If you’ve correlated your application’s logs and traces, you never have to search for relevant data; it’s already available to you. Viewing log and trace data together allows you to quickly diagnose and solve problems. -To learn how to correlate your logs with your instrumented services, refer to [Stream application logs](../../../solutions/observability/logs/stream-application-logs.md). +To learn how to correlate your logs with your instrumented services, refer to [Stream application logs](/solutions/observability/logs/stream-application-logs.md). :::{image} /solutions/images/observability-logs.png :alt: Example view of the Logs tab in Applications UI @@ -37,4 +37,4 @@ Elastic APM integrates with popular logging frameworks, making it easy to correl * View all logs belonging to a particular trace * Easily move between logs and traces when debugging application issues -See the [Stream application logs](../../../solutions/observability/logs/stream-application-logs.md) guide to get started. \ No newline at end of file +See the [Stream application logs](/solutions/observability/logs/stream-application-logs.md) guide to get started. \ No newline at end of file diff --git a/solutions/observability/apps/manage-data-retention.md b/solutions/observability/apps/manage-data-retention.md index dafa0f0064..2146baebe3 100644 --- a/solutions/observability/apps/manage-data-retention.md +++ b/solutions/observability/apps/manage-data-retention.md @@ -9,11 +9,10 @@ applies_to: # Manage data retention [synthetics-manage-retention] -When you set up a synthetic monitor, data from the monitor is saved in [{{es}} data streams](../../../manage-data/data-store/data-streams.md), an append-only structure in {{es}}. +When you set up a synthetic monitor, data from the monitor is saved in [{{es}} data streams](/manage-data/data-store/data-streams.md), an append-only structure in {{es}}. There are six data streams recorded by synthetic monitors: `http`, `tcp`, `icmp`, `browser`, `browser.network`, `browser.screenshot`. Elastic will retain data from each data stream for some time period, and the default time period varies by data stream. If you want to reduce the amount of storage required or store data for longer, you can customize how long to retain data for each data stream. - ## Synthetics data streams [synthetics-manage-retention-data-streams] There are six data streams recorded by synthetic monitors: @@ -29,7 +28,6 @@ There are six data streams recorded by synthetic monitors: All types of checks record core metadata. Browser-based checks store two additional types of data: network and screenshot documents. These browser-specific indices are usually many times larger than the core metadata. The relative sizes of each vary depending on the sites being checked with network data usually being the larger of the two by a significant factor. - ## Customize data stream lifecycles [synthetics-manage-retention-customize] If Synthetics browser data streams are storing data longer than necessary, you can opt to retain data for a shorter period. @@ -42,7 +40,7 @@ To find Synthetics data streams: :::{tab-item} Elastic Stack :sync: stack -1. Navigate to [{{kib}} index management](../../../manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md). +1. Navigate to [{{kib}} index management](/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md). 2. Filter the list of data streams for those containing the term `synthetics`. 1. In the UI there will be three types of browser data streams: `synthetics-browser-*`, `synthetics-browser.network-*`, and `synthetics-browser.screenshot-*`. diff --git a/solutions/observability/apps/manage-monitors.md b/solutions/observability/apps/manage-monitors.md index 745a4d24d1..2aeb64592e 100644 --- a/solutions/observability/apps/manage-monitors.md +++ b/solutions/observability/apps/manage-monitors.md @@ -9,10 +9,9 @@ applies_to: # Manage monitors [synthetics-manage-monitors] -After you’ve [created a synthetic monitor](../../../solutions/observability/apps/get-started.md), you’ll need to manage that monitor over time. This might include updating or permanently deleting an existing monitor. - -If you’re using {{project-monitors}}, you should also set up a workflow that uses [best practices for managing monitors effectively](../../../solutions/observability/apps/manage-monitors.md#synthetics-projects-best-practices) in a production environment. +After you’ve [created a synthetic monitor](/solutions/observability/apps/get-started.md), you’ll need to manage that monitor over time. This might include updating or permanently deleting an existing monitor. +If you’re using {{project-monitors}}, you should also set up a workflow that uses [best practices for managing monitors effectively](/solutions/observability/apps/manage-monitors.md#synthetics-projects-best-practices) in a production environment. ## Update a monitor [manage-monitors-config] @@ -23,18 +22,18 @@ You can also update the journey used in a browser monitor. For example, if you u :::::::{tab-set} ::::::{tab-item} Project monitors -If you [set up the monitor using a Synthetic project](../../../solutions/observability/apps/create-monitors-with-project-monitors.md), you’ll update the monitor in the Synthetic project source and then `push` changes. +If you [set up the monitor using a Synthetic project](/solutions/observability/apps/create-monitors-with-project-monitors.md), you’ll update the monitor in the Synthetic project source and then `push` changes. For lightweight monitors, make changes to the YAML file. For browser monitors, you can update the configuration of one or more monitors: * To update the configuration of an individual monitor, edit the journey directly in the JavaScript or TypeScript files, specifically the options in `monitor.use`. -* To update the configuration of *all* monitors in a Synthetic project, edit the [global synthetics configuration file](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). +* To update the configuration of *all* monitors in a Synthetic project, edit the [global synthetics configuration file](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). -To update the journey that a browser monitor runs, edit the journey code directly and [test the updated journey locally](../../../solutions/observability/apps/write-synthetic-test.md#synthetics-test-locally) to make sure it’s valid. +To update the journey that a browser monitor runs, edit the journey code directly and [test the updated journey locally](/solutions/observability/apps/write-synthetic-test.md#synthetics-test-locally) to make sure it’s valid. -After making changes to the monitors, run the [`push` command](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command) to replace the existing monitors with new monitors using the updated configuration or journey code. +After making changes to the monitors, run the [`push` command](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command) to replace the existing monitors with new monitors using the updated configuration or journey code. ::::{note} Updates are linked to a monitor’s `id`. To update a monitor you must keep its `id` the same. @@ -42,7 +41,7 @@ Updates are linked to a monitor’s `id`. To update a monitor you must keep its :::::: ::::::{tab-item} Synthetics UI -If you [set up the monitor using the Synthetics UI](../../../solutions/observability/apps/create-monitors-in-synthetics-app.md), you can update the monitor configuration of both lightweight and browser monitors in the {{synthetics-app}}: +If you [set up the monitor using the Synthetics UI](/solutions/observability/apps/create-monitors-in-synthetics-app.md), you can update the monitor configuration of both lightweight and browser monitors in the {{synthetics-app}}: 1. Go to **Management**. 2. Click the pencil icon next to the monitor you want to edit. @@ -63,17 +62,17 @@ Eventually you might want to delete a monitor altogether. For example, if the us :::::::{tab-set} ::::::{tab-item} Project monitors -If you [set up the monitor using a Synthetics project](../../../solutions/observability/apps/create-monitors-with-project-monitors.md), you’ll delete the monitor from the project source and push changes. +If you [set up the monitor using a Synthetics project](/solutions/observability/apps/create-monitors-with-project-monitors.md), you’ll delete the monitor from the project source and push changes. For lightweight monitors, delete the monitor from the YAML file. For browser monitors, delete the full journey from the JavaScript or TypeScript file. -Then, run the [`push` command](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). The monitor associated with that journey that existed will be deleted. +Then, run the [`push` command](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-push-command). The monitor associated with that journey that existed will be deleted. :::::: ::::::{tab-item} Synthetics UI -If you [set up the monitor using the Synthetics UI](../../../solutions/observability/apps/create-monitors-in-synthetics-app.md), you can delete a lightweight or browser monitor in the Synthetics UI: +If you [set up the monitor using the Synthetics UI](/solutions/observability/apps/create-monitors-in-synthetics-app.md), you can delete a lightweight or browser monitor in the Synthetics UI: 1. Go to **Management**. 2. Click the trash can icon next to the monitor you want to delete. @@ -82,22 +81,18 @@ If you [set up the monitor using the Synthetics UI](../../../solutions/observabi ::::::: Alternatively, you can temporarily disable a monitor by updating the monitor’s configuration in your journey’s code or in the Synthetics UI using the *Enabled* toggle. - ## Implement best practices for Synthetics projects [synthetics-projects-best-practices] ::::{important} This is only relevant to monitors created using projects. :::: - -After you’ve [set up a project](../../../solutions/observability/apps/create-monitors-with-project-monitors.md), there are some best practices you can implement to manage the Synthetics project effectively. - +After you’ve [set up a project](/solutions/observability/apps/create-monitors-with-project-monitors.md), there are some best practices you can implement to manage the Synthetics project effectively. ### Use version control [synthetics-version-control] First, it’s recommended that you version control all files in Git. If your Synthetics project is not already in a version controlled directory add it and push it to your Git host. - ### Set up recommended workflow [synthetics-workflow] While it can be convenient to run the `push` command directly from your workstation, especially when setting up a new Synthetics project, it is not recommended for production environments. diff --git a/solutions/observability/apps/manage-storage.md b/solutions/observability/apps/manage-storage.md index c598f991bd..7bcbc3a9c6 100644 --- a/solutions/observability/apps/manage-storage.md +++ b/solutions/observability/apps/manage-storage.md @@ -13,11 +13,3 @@ The [storage and sizing guide](storage-sizing-guide.md) attempts to define a "ty In addition, the Applications UI makes it easy to visualize your APM data usage with [storage explorer](storage-explorer.md). Storage explorer allows you to analyze the storage footprint of each of your services to see which are producing large amounts of data—​so you can better reduce the data you’re collecting or forecast and prepare for future storage needs. - - - - - - - - diff --git a/solutions/observability/apps/managed-intake-service-event-api.md b/solutions/observability/apps/managed-intake-service-event-api.md index 8dc154b6da..8a69cae58e 100644 --- a/solutions/observability/apps/managed-intake-service-event-api.md +++ b/solutions/observability/apps/managed-intake-service-event-api.md @@ -9,22 +9,18 @@ applies_to: ::::{warning} This API is exclusively for APM agent developers. The vast majority of users should have no reason to interact with this API. - :::: - The managed intake service exposes endpoints for: * [The managed intake service information API](#observability-apm-server-api-server-information-api) * [Elastic APM events intake API](#observability-apm-server-api-events-intake-api) * [OpenTelemetry intake API](#observability-apm-server-api-opentelemetry-api) - ## Server information API [observability-apm-server-api-server-information-api] The managed intake service exposes an API endpoint to query general server information. This lightweight endpoint is useful as a server up/down health check. - ### Server Information endpoint [api-info-endpoint] Send an `HTTP GET` request to the server information endpoint: @@ -37,7 +33,6 @@ This endpoint always returns an HTTP 200. Requests to this endpoint must be authenticated. - #### Example [api-info-examples] Example managed intake service information request: @@ -54,7 +49,6 @@ curl -X POST http://127.0.0.1:8200/ \ } ``` - ## Events intake API [observability-apm-server-api-events-intake-api] ::::{note} @@ -62,7 +56,6 @@ Most users do not need to interact directly with the events intake API. :::: - The events intake API is what we call the internal protocol that APM agents use to talk to the managed intake service. Agents communicate with the Server by sending events — captured pieces of information — in an HTTP request. Events can be: * Transactions @@ -76,7 +69,6 @@ With NDJSON, agents can open an HTTP POST request and use chunked encoding to st Refer to [Learn about data types](learn-about-application-data-types.md) to learn more about the different types of events. - ### Endpoints [api-events-endpoint] The managed intake service exposes the following endpoints for Elastic APM agent data intake: @@ -85,7 +77,6 @@ The managed intake service exposes the following endpoints for Elastic APM agent | --- | --- | | APM agent event intake | `/intake/v2/events` | - ### Request [api-events-example] Send an `HTTP POST` request to the managed intake service `intake/v2/events` endpoint: @@ -105,15 +96,12 @@ Since asynchronous processing defers some of the event processing to the backgro :::: - - ### Response [api-events-response] On success, the server will respond with a 202 Accepted status code and no body. Keep in mind that events can succeed and fail independently of each other. Only if all events succeed does the server respond with a 202. - ### API Errors [api-events-errors] There are two types of errors that the managed intake service may return to an agent: @@ -152,15 +140,12 @@ An example error response might look something like this: 3. An immediately returning non-event related error 4. The number of accepted events - If you’re developing an agent, these errors can be useful for debugging. - ### Event API Schemas [api-events-schema-definition] The managed intake service uses a collection of JSON Schemas for validating requests to the intake API. - ### Metadata [observability-apm-server-api-metadata] Every new connection to the managed intake service starts with a `metadata` stanza. This provides general metadata concerning the other objects in the stream. @@ -172,8 +157,6 @@ Metadata is stored under `context` when viewing documents in {{es}}. :::: - - #### Metadata Schema [metadata-schema] The managed intake service uses JSON Schema to validate requests. The specification for metadata is defined on [GitHub](https://github.com/elastic/apm-server/blob/main/docs/spec/v2/metadata.json) and included below. @@ -751,8 +734,6 @@ The managed intake service uses JSON Schema to validate requests. The specificat :::: - - #### Kubernetes data [kubernetes-data] APM agents automatically read Kubernetes data and send it to the managed intake service. In most instances, agents are able to read this data from inside the container. If this is not the case, or if you wish to override this data, you can set environment variables for the agents to read. These environment variable are set via the Kubernetes [Downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#use-pod-fields-as-values-for-environment-variables). Here’s how you would add the environment variables to your Kubernetes pod spec: @@ -785,12 +766,10 @@ The table below maps these environment variables to the APM metadata event field | `KUBERNETES_NAMESPACE` | system.kubernetes.namespace | | `KUBERNETES_POD_UID` | system.kubernetes.pod.uid | - ### Transactions [observability-apm-server-api-transactions] Transactions are events corresponding to an incoming request or similar task occurring in a monitored service. - #### Transaction Schema [api-transaction-schema] The managed intake service uses JSON Schema to validate requests. The specification for transactions is defined on [GitHub](https://github.com/elastic/apm-server/blob/main/docs/spec/v2/transaction.json) and included below. @@ -1932,13 +1911,10 @@ The managed intake service uses JSON Schema to validate requests. The specificat :::: - - ### Spans [observability-apm-server-api-spans] Spans are events captured by an agent occurring in a monitored service. - #### Span Schema [api-span-schema] The managed intake service uses JSON Schema to validate requests. The specification for spans is defined on [GitHub](https://github.com/elastic/apm-server/blob/main/docs/spec/v2/span.json) and included below. @@ -2852,13 +2828,10 @@ The managed intake service uses JSON Schema to validate requests. The specificat :::: - - ### Errors [observability-apm-server-api-errors] An error or a logged error message captured by an agent occurring in a monitored service. - #### Error Schema [api-error-schema] The managed intake service uses a JSON Schema to validate requests. The specification for errors is defined on [GitHub](https://github.com/elastic/apm-server/blob/main/docs/spec/v2/error.json) and included below. @@ -4162,13 +4135,10 @@ The managed intake service uses a JSON Schema to validate requests. The specific :::: - - ### Metrics [observability-apm-server-api-metrics] Metrics contain application metric data captured by an {{apm-agent}}. - #### Metric Schema [api-metricset-schema] The managed intake service uses JSON Schema to validate requests. The specification for metrics is defined on [GitHub](https://github.com/elastic/apm-server/blob/main/docs/spec/v2/metricset.json) and included below. @@ -4480,8 +4450,6 @@ The managed intake service uses JSON Schema to validate requests. The specificat :::: - - ## OpenTelemetry API [observability-apm-server-api-opentelemetry-api] Elastic supports receiving traces, metrics, and logs over the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp/). OTLP is the default transfer protocol for OpenTelemetry and is supported natively by the managed intake service. @@ -4491,7 +4459,6 @@ The managed intake service supports two OTLP communication protocols on the same * OTLP/HTTP (protobuf) * OTLP/gRPC - ### OTLP/gRPC paths [otlpgrpc-paths] | Name | Endpoint | @@ -4500,7 +4467,6 @@ The managed intake service supports two OTLP communication protocols on the same | OTLP trace intake | `/opentelemetry.proto.collector.trace.v1.TraceService/Export` | | OTLP logs intake | `/opentelemetry.proto.collector.logs.v1.LogsService/Export` | - ### OTLP/HTTP paths [otlphttp-paths] | Name | Endpoint | @@ -4514,4 +4480,3 @@ See our [OpenTelemetry docs](upstream-opentelemetry-collectors-language-sdks.md) :::: - diff --git a/solutions/observability/apps/metadata.md b/solutions/observability/apps/metadata.md index 9cd46c5394..903ac1ca72 100644 --- a/solutions/observability/apps/metadata.md +++ b/solutions/observability/apps/metadata.md @@ -10,7 +10,6 @@ applies_to: Metadata can enrich your events and make application performance monitoring even more useful. Let’s explore the different types of metadata that Elastic APM offers. - ## Labels [apm-data-model-labels] Labels add **indexed** information to transactions, spans, and errors. Indexed means the data is searchable and aggregatable in {{es}}. Add additional key-value pairs to define multiple labels. @@ -23,11 +22,9 @@ Labels add **indexed** information to transactions, spans, and errors. Indexed m Label values can be a string, boolean, or number, although some agents only support string values at this time. Because labels for a given key, regardless of agent used, are stored in the same place in {{es}}, all label values of a given key must have the same data type. Multiple data types per key will throw an exception, for example: `{foo: bar}` and `{foo: 42}` is not allowed. ::::{important} -Avoid defining too many user-specified labels. Defining too many unique fields in an index is a condition that can lead to a [mapping explosion](../../../manage-data/data-store/mapping.md#mapping-limit-settings). +Avoid defining too many user-specified labels. Defining too many unique fields in an index is a condition that can lead to a [mapping explosion](/manage-data/data-store/mapping.md#mapping-limit-settings). :::: - - ### Agent API reference [_agent_api_reference] * Go: [`SetLabel`](apm-agent-go://reference/api-documentation.md#context-set-label) @@ -39,10 +36,9 @@ Avoid defining too many user-specified labels. Defining too many unique fields i * Ruby: [`set_label`](apm-agent-ruby://reference/api-reference.md#api-agent-set-label) * Rum: [`addLabels`](apm-agent-rum-js://reference/agent-api.md#apm-add-labels) - ## Custom context [apm-data-model-custom] -Custom context adds **non-indexed**, custom contextual information to transactions and errors. Non-indexed means the data is not searchable or aggregatable in {{es}}, and you cannot build dashboards on top of the data. This also means you don’t have to worry about [mapping explosions](../../../manage-data/data-store/mapping.md#mapping-limit-settings), as these fields are not added to the mapping. +Custom context adds **non-indexed**, custom contextual information to transactions and errors. Non-indexed means the data is not searchable or aggregatable in {{es}}, and you cannot build dashboards on top of the data. This also means you don’t have to worry about [mapping explosions](/manage-data/data-store/mapping.md#mapping-limit-settings), as these fields are not added to the mapping. Non-indexed information is useful for providing contextual information to help you quickly debug performance issues or errors. @@ -55,8 +51,6 @@ Non-indexed information is useful for providing contextual information to help y Setting a circular object, a large object, or a non JSON serializable object can lead to errors. :::: - - ### Agent API reference [_agent_api_reference_2] * Go: [`SetCustom`](apm-agent-go://reference/api-documentation.md#context-set-custom) @@ -69,7 +63,6 @@ Setting a circular object, a large object, or a non JSON serializable object can * Ruby: [`set_custom_context`](apm-agent-ruby://reference/api-reference.md#api-agent-set-custom-context) * Rum: [`setCustomContext`](apm-agent-rum-js://reference/agent-api.md#apm-set-custom-context) - ## User context [apm-data-model-user] User context adds **indexed** user information to transactions and errors. Indexed means the data is searchable and aggregatable in {{es}}. @@ -79,7 +72,6 @@ User context adds **indexed** user information to transactions and errors. Index * {{es}} fields: `user.email` | `user.name` | `user.id` * Applies to: [Transactions](transactions.md) | [Errors](errors.md) - ### Agent API reference [_agent_api_reference_3] * Go: [`SetUsername`](apm-agent-go://reference/api-documentation.md#context-set-user-email) diff --git a/solutions/observability/apps/metrics.md b/solutions/observability/apps/metrics.md index 0f33429c3d..2590aa2ef9 100644 --- a/solutions/observability/apps/metrics.md +++ b/solutions/observability/apps/metrics.md @@ -8,12 +8,14 @@ applies_to: # Metrics [apm-data-model-metrics] +:::{include} _snippets/apm-server-vs-mis.md +::: + **Metrics** measure the state of a system by gathering information on a regular interval. There are two types of APM metrics: * **System metrics**: Basic infrastructure and application metrics. * **Calculated metrics**: Aggregated trace event metrics used to power visualizations in the Applications UI. - ## System metrics [_system_metrics] APM agents automatically pick up basic host-level metrics, including system and process-level CPU and memory metrics. Agent specific metrics are also available, like [JVM metrics](apm-agent-java://reference/metrics.md) in the Java Agent, and [Go runtime](apm-agent-go://reference/metrics.md) metrics in the Go Agent. @@ -24,7 +26,6 @@ Infrastructure and application metrics are important sources of information when Most agents limit keyword fields to 1024 characters, non-keyword fields (e.g. `system.memory.total`) to 10,000 characters. :::: - Metrics are stored in metric indices. For a full list of tracked metrics, see the relevant agent documentation: @@ -35,7 +36,6 @@ For a full list of tracked metrics, see the relevant agent documentation: * [Python](apm-agent-python://reference/metrics.md) * [Ruby](apm-agent-ruby://reference/metrics.md) - ### Example system metric document [_example_system_metric_document] This example shows what system metric documents can look like when indexed in {{es}}. @@ -117,17 +117,14 @@ This example contains JVM metrics produced by the {{apm-java-agent}}. and contai :::: - - ## Calculated metrics [_calculated_metrics] -APM agents and APM Server calculate metrics from trace events to power visualizations in the Applications UI. +APM agents and either {{apm-server-or-mis}} calculate metrics from trace events to power visualizations in the Applications UI. Calculated metrics are an implementation detail and while we aim for stability for these data models, the dimensions and concrete limits for aggregations are subject to change within minor version updates. These metrics are described below. - ### Breakdown metrics [_breakdown_metrics] To power the [Time spent by span type](transactions-2.md) graph, agents collect summarized metrics about the timings of spans and transactions, broken down by span type. @@ -144,8 +141,6 @@ You can filter and group by these dimensions: * `span.type`: The type of the span, for example `app`, `template` or `db` * `span.subtype`: The sub-type of the span, for example `mysql` (optional) - - ### Example breakdown metric document [_example_breakdown_metric_document] This example shows what breakdown metric documents can look like when indexed in {{es}}. @@ -238,11 +233,9 @@ This example shows what breakdown metric documents can look like when indexed in :::: - - ### Transaction metrics [_transaction_metrics] -To power [{{kib}} Applications UI](overviews.md) visualizations, APM Server aggregates transaction events into latency distribution metrics. +To power [{{kib}} Applications UI](overviews.md) visualizations, either {{apm-server-or-mis}} aggregates transaction events into latency distribution metrics. **`transaction.duration.summary`** and **`transaction.duration.histogram`** : These metrics represent the latency summary and latency distribution of transaction groups, used to power transaction-oriented visualizations and analytics in Elastic APM. @@ -288,10 +281,8 @@ You can filter and group by these dimensions (some of which are optional, for ex * `transaction.root`: A boolean flag indicating whether the transaction is the root of a trace * `transaction.type`: The type of the transaction, for example `request` - The `@timestamp` field of these documents holds the start of the aggregation interval. - ### Example transaction document [_example_transaction_document_2] This example shows what transaction documents can look like when indexed in {{es}}. @@ -428,11 +419,9 @@ This example shows what transaction documents can look like when indexed in {{es :::: - - ### Service-transaction metrics [_service_transaction_metrics] -To power [{{kib}} Applications UI](overviews.md) visualizations, APM Server aggregates transaction events into service-transaction metrics. Service-transaction metrics are similar to transaction metrics, but they usually have a much lower cardinality as they have significantly fewer dimensions. The UI uses them when fewer details of the transactions are needed. +To power [{{kib}} Applications UI](overviews.md) visualizations, either {{apm-server-or-mis}} aggregates transaction events into service-transaction metrics. Service-transaction metrics are similar to transaction metrics, but they usually have a much lower cardinality as they have significantly fewer dimensions. The UI uses them when fewer details of the transactions are needed. **`transaction.duration.summary`** and **`transaction.duration.histogram`** : These metrics represent the latency summary and latency distribution of service transaction groups, used to power service-oriented visualizations and analytics in Elastic APM. @@ -450,10 +439,8 @@ You can filter and group by these dimensions: * `service.name`: The name of the service that made the request * `transaction.type`: The type of the enclosing transaction, for example `request` - The `@timestamp` field of these documents holds the start of the aggregation interval. - ### Example service-transaction document [_example_service_transaction_document] This example shows what service-transaction documents can look like when indexed in {{es}}. @@ -966,11 +953,9 @@ This example shows what service-transaction documents can look like when indexed :::: - - ### Service-destination metrics [_service_destination_metrics] -To power [{{kib}} Applications UI](overviews.md) visualizations, APM Server aggregates span events into service-destination metrics. +To power [{{kib}} Applications UI](overviews.md) visualizations, either {{apm-server-or-mis}} aggregates span events into service-destination metrics. **`span.destination.service.response_time.count`** and **`span.destination.service.response_time.sum.us`** : These metrics measure the count and total duration of requests from one service to another service. These are used to calculate the throughput and latency of requests to backend services such as databases in [Service maps](service-map.md). @@ -992,10 +977,8 @@ You can filter and group by these dimensions: * `span.destination.service.resource`: The destination service resource, for example `mysql` * `span.name`: The name of the operation, for example `SELECT FROM table_name`. - The `@timestamp` field of these documents holds the start of the aggregation interval. - ### Example service-destination document [_example_service_destination_document] This example shows what service-destination documents can look like when indexed in {{es}}. @@ -1063,11 +1046,9 @@ This example shows what service-destination documents can look like when indexed :::: - - ### Service-summary metrics [_service_summary_metrics] -To power [{{kib}} Applications UI](overviews.md) visualizations, APM Server aggregates transaction, error, log, and metric events into service-summary metrics. +To power [{{kib}} Applications UI](overviews.md) visualizations, either {{apm-server-or-mis}} aggregates transaction, error, log, and metric events into service-summary metrics. These metric documents can be identified by searching for `metricset.name: service_summary`. @@ -1083,7 +1064,6 @@ You can filter and group by these dimensions: The `@timestamp` field of these documents holds the start of the aggregation interval. - ### Example service-summary document [_example_service_summary_document] This example shows what service-summary documents can look like when indexed in {{es}}. @@ -1132,8 +1112,6 @@ This example shows what service-summary documents can look like when indexed in :::: - - ## Data streams [_data_streams_4] Metrics are stored in the following data streams: @@ -1147,12 +1125,10 @@ Metrics are stored in the following data streams: See [Data streams](data-streams.md) to learn more. - ## Aggregated metrics: limits and overflows [_aggregated_metrics_limits_and_overflows] For all aggregated metrics, namely transaction, service-transaction, service-destination, and service-summary metrics, there are limits on the number of unique groups tracked at any given time. - ### Limits [_limits] Note that all the below limits may change in the future with further improvements. @@ -1164,10 +1140,8 @@ Note that all the below limits may change in the future with further improvement * For service-destination metrics, there is an additional limit of 5000 total service destination groups per GB of APM Server starting with 10000 service destination groups for 1 GB APM Server, and each service may only consume up to 10% of the service destination groups, which is 1000 service destination groups for 1GB APM Server with 500 increment per GB of APM Server. * For service-summary metrics, there is no additional limit. - In the previous metrics, a service is defined as a combination of `service.name`, `service.environment`, `service.language.name` and `agent.name`. - ### Overflows [_overflows] When a dimension has a high cardinality and exceeds the limit, the metrics will be aggregated under a dedicated overflow bucket. diff --git a/solutions/observability/apps/migrate-from-elastic-synthetics-integration.md b/solutions/observability/apps/migrate-from-elastic-synthetics-integration.md index 131cbf2341..6f89782ca9 100644 --- a/solutions/observability/apps/migrate-from-elastic-synthetics-integration.md +++ b/solutions/observability/apps/migrate-from-elastic-synthetics-integration.md @@ -14,7 +14,6 @@ You should *not* use the Elastic Synthetics integration to set up new monitors a * With Elastic **{{project-monitors-cap}}**, you write tests in an external version-controlled project and use a CLI tool to push monitors to the {{stack}}. * The **{{synthetics-app}}** is an application in {{kib}} that you can use to configure and create monitors using a user interface. - ## Compare approaches [synthetics-migrate-integration-compare] Below is a comparison of how you used the {{agent}} integration to create monitors and how you’ll use the {{synthetics-app}} or projects to create monitors: @@ -32,7 +31,6 @@ Below is a comparison of how you used the {{agent}} integration to create monito * Your infrastructure using [{{private-location}}s](monitor-resources-on-private-networks.md) * Elastic’s global managed infrastructure - **Where you configure monitors**: * **{{agent}} integration**: You could configure monitors using: @@ -45,7 +43,6 @@ Below is a comparison of how you used the {{agent}} integration to create monito * A user interface in {{kib}} (all lightweight monitors, browser monitors via inline script only) * Code in an external, version-controlled project (all monitor types) - **How to use projects ([read more](#synthetics-migrate-integration-projects))**: * **{{agent}} integration**: @@ -62,7 +59,6 @@ Below is a comparison of how you used the {{agent}} integration to create monito 3. Write journeys in JavaScript or TypeScript files and configure individual monitors in your journey code using `monitor.use` or configure all monitors using the `synthetics.config.ts` file. 4. Use the `elastic/synthetics push` command to create monitors. - Find more details in [Use {{project-monitors-cap}}](create-monitors-with-project-monitors.md). **How to use the UI ([read more](#synthetics-migrate-integration-ui))**: @@ -82,10 +78,8 @@ Find more details in [Use {{project-monitors-cap}}](create-monitors-with-project 4. Configure the monitor. 5. Create the monitor. - Find more details in [Use the {{synthetics-app}}](create-monitors-in-synthetics-app.md). - ## Where monitors run [synthetics-migrate-integration-location] If you want to continue hosting on your infrastructure, you will need to create a {{private-location}} before creating monitors. If you have already have an {{agent}} running using `elastic-agent-complete`, you can [add it as a new {{private-location}}](monitor-resources-on-private-networks.md#synthetics-private-location-add) in the {{synthetics-app}}. To create a new {{private-location}} from scratch, follow all instructions in [Monitor resources on private networks](monitor-resources-on-private-networks.md). @@ -94,7 +88,6 @@ Alternatively, you can start hosting on Elastic’s global managed infrastructur Executing synthetic tests on Elastic’s global managed testing infrastructure incurs an additional charge. Tests are charged under one of two new billing dimensions depending on the monitor type. For *browser monitor* usage, there is a fee per test run. For *lightweight monitor* usage, there is a fee per region in which you run any monitors regardless of the number of test runs. For more details, refer to [full details and current pricing](https://www.elastic.co/pricing). - ## How to use projects [synthetics-migrate-integration-projects] If you already have an external project you were adding via a ZIP URL you can use the same project, but you will have to make some changes. @@ -128,7 +121,6 @@ First, upgrade the existing project to use the latest version of `@elastic/synth 4. `.github/` will contain sample workflow files to use with GitHub Actions. - Then, you can further configure monitors as needed. In the upgraded project, you’ll use code (instead of the Integrations UI) to define settings like the name of the monitor and the frequency at which it will run. There are two ways you can configure monitors using code: * For individual monitors, use `monitor.use` directly in the journey code. Read more in [Configure individual monitors](configure-individual-browser-monitors.md). @@ -140,7 +132,6 @@ Optionally, you can also add lightweight monitors to the project in YAML files. For more information on getting started with projects, refer to [Use {{project-monitors-cap}}](create-monitors-with-project-monitors.md). - ## How to use the UI [synthetics-migrate-integration-ui] If you created monitors solely via the Elastic Synthetics integration UI, you can recreate monitors in the {{synthetics-app}}. diff --git a/solutions/observability/apps/mobile-service-overview.md b/solutions/observability/apps/mobile-service-overview.md index 9160fe7808..a0f0337cb3 100644 --- a/solutions/observability/apps/mobile-service-overview.md +++ b/solutions/observability/apps/mobile-service-overview.md @@ -22,31 +22,26 @@ For example, see: All of these metrics & insights can help SREs and developers better understand the health of their mobile application environment and the impact of backend errors and bottlenecks on end-user experience. - ## Quick stats [mobile-service-stats] Understand the impact of slow application load times and variations in application crash rate on user traffic (coming soon). Visualize session and HTTP trends, and see where your users are located—​enabling you to optimize your infrastructure deployment and routing topology. Note: due to the way crash rate is calculated (crashes per session) it is possible to have greater than 100% rate, due to fact that a session may contain multiple crashes. - :::{image} /solutions/images/observability-mobile-location.png :alt: mobile service overview centered on location map :screenshot: ::: - ## Most used [mobile-service-most-used] Optimize your end-user experience and your application QA strategy based on your most used device models and operating system versions. - :::{image} /solutions/images/observability-mobile-most-used.png :alt: mobile service overview showing most used devices :screenshot: ::: - ## Throughput and transactions [mobile-throughput-transactions] The **Throughput** chart visualizes the average number of transactions per minute for the selected service. @@ -55,13 +50,11 @@ The **Transactions** table displays a list of *transaction groups* for the selec By default, transaction groups are sorted by *Impact* to show the most used and slowest endpoints in your service. If there is a particular endpoint you are interested in, click **View transactions** to view a list of similar transactions on the [transactions overview](transactions-2.md) page. - :::{image} /solutions/images/observability-traffic-transactions.png :alt: Traffic and transactions :screenshot: ::: - ## Failed transaction rate and dependencies [mobile-error-and-dependencies] The failed transaction rate represents the percentage of failed transactions from the perspective of the selected service. It’s useful for visualizing unexpected increases, decreases, or irregular patterns in a service’s transactions. @@ -75,21 +68,17 @@ If there is no HTTP status, both transactions and spans are considered successfu :::: - The **Dependencies** table displays a list of downstream services or external connections relevant to the service at the selected time range. The table displays latency, throughput, failed transaction rate, and the impact of each dependency. By default, dependencies are sorted by *Impact* to show the most used and the slowest dependency. If there is a particular dependency you are interested in, click **[View dependencies](dependencies.md)** to learn more about it. - ::::{note} Displaying dependencies for services instrumented with the Real User Monitoring (RUM) agent requires an agent version ≥ v5.6.3. :::: - :::{image} /solutions/images/observability-spans-dependencies.png :alt: Span type duration and dependencies :screenshot: ::: - :::{image} /solutions/images/observability-mobile-tp.png :alt: mobile service overview showing latency :screenshot: diff --git a/solutions/observability/apps/monitor-apm-server-binary.md b/solutions/observability/apps/monitor-apm-server-binary.md index 20e28434d6..32fde3bd29 100644 --- a/solutions/observability/apps/monitor-apm-server-binary.md +++ b/solutions/observability/apps/monitor-apm-server-binary.md @@ -8,7 +8,6 @@ applies_to: # Monitor the APM Server binary [apm-monitoring] - There are two methods to monitor the APM Server binary. Make sure monitoring is enabled on your {{es}} cluster, then configure one of these methods to collect APM Server metrics: * [Internal collection](use-internal-collection-to-send-monitoring-data.md) - Internal collectors send monitoring data directly to your monitoring cluster. diff --git a/solutions/observability/apps/monitor-apm-server.md b/solutions/observability/apps/monitor-apm-server.md index fc8f14fe29..0372f6fb92 100644 --- a/solutions/observability/apps/monitor-apm-server.md +++ b/solutions/observability/apps/monitor-apm-server.md @@ -15,10 +15,9 @@ Select your deployment method to get started: * [Fleet-managed](monitor-fleet-managed-apm-server.md) * [APM Server binary](monitor-apm-server-binary.md) - ## {{ecloud}} [apm-monitor-apm-cloud] {{ecloud}} manages the installation and configuration of a monitoring agent for you — so all you have to do is flip a switch and watch the data pour in. -* **{{ech}}** user? See [Stack Monitoring on {{ecloud}} deployments](../../../deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). +* **{{ech}}** user? See [Stack Monitoring on {{ecloud}} deployments](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). * **{{ece}}** user? See [Enable stack monitoring on ECE deployments](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). diff --git a/solutions/observability/apps/monitor-fleet-managed-apm-server.md b/solutions/observability/apps/monitor-fleet-managed-apm-server.md index f022329e55..6471de7a67 100644 --- a/solutions/observability/apps/monitor-fleet-managed-apm-server.md +++ b/solutions/observability/apps/monitor-fleet-managed-apm-server.md @@ -6,30 +6,24 @@ applies_to: stack: all --- - - # Monitor a Fleet-managed APM Server [apm-monitor-apm-self-install] - ::::{note} This guide assumes you are already ingesting APM data into the {{stack}}. :::: - In 8.0 and later, you can use {{metricbeat}} to collect data about APM Server and ship it to a monitoring cluster. To collect and ship monitoring data: 1. [Configure {{agent}} to send monitoring data](#apm-configure-ea-monitoring-data) 2. [Install and configure {{metricbeat}} to collect monitoring data](#apm-install-config-metricbeat) - ## Configure {{agent}} to send monitoring data [apm-configure-ea-monitoring-data] ::::{admonition} -Before you can monitor APM, you must have monitoring data for the {{es}} production cluster. To learn how, see [Collect {{es}} monitoring data with {{metricbeat}}](../../../deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat.md). Alternatively, open the **{{stack-monitor-app}}** app in {{kib}} and follow the in-product guide. +Before you can monitor APM, you must have monitoring data for the {{es}} production cluster. To learn how, see [Collect {{es}} monitoring data with {{metricbeat}}](/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat.md). Alternatively, open the **{{stack-monitor-app}}** app in {{kib}} and follow the in-product guide. :::: - 1. Enable monitoring of {{agent}} by adding the following settings to your `elastic-agent.yml` configuration file: ```yaml @@ -72,10 +66,8 @@ Before you can monitor APM, you must have monitoring data for the {{es}} product [global-flags] ``` - See the [{{agent}} command reference](/reference/fleet/agent-command-reference.md) for more information on the enroll command. - ## Install and configure {{metricbeat}} to collect monitoring data [apm-install-config-metricbeat] 1. Install {{metricbeat}} on the same server as {{agent}}. To learn how, see [Get started with {{metricbeat}}](beats://reference/metricbeat/metricbeat-installation-configuration.md). If you already have {{metricbeat}} installed, skip this step. @@ -117,7 +109,7 @@ See the [{{agent}} command reference](/reference/fleet/agent-command-reference.m 3. APM Server metrics are exposed at `/processes/apm-server-default`. Add this location as the `basepath`. 4. Set the `username` and `password` settings as required by your environment. If Elastic {{security-features}} are enabled, you must provide a username and password so that {{metricbeat}} can collect metrics successfully: - 1. Create a user on the {{es}} cluster that has the `remote_monitoring_collector` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). Alternatively, if it’s available in your environment, use the `remote_monitoring_user` [built-in user](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). + 1. Create a user on the {{es}} cluster that has the `remote_monitoring_collector` [built-in role](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). Alternatively, if it’s available in your environment, use the `remote_monitoring_user` [built-in user](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). 2. Add the `username` and `password` settings to the beat module configuration file. 4. Optional: Disable the system module in the {{metricbeat}}. @@ -134,7 +126,6 @@ See the [{{agent}} command reference](/reference/fleet/agent-command-reference.m In production environments, you should send your deployment logs and metrics to a dedicated monitoring deployment (referred to as the *monitoring cluster*). Monitoring indexes logs and metrics into {{es}} and these indexes consume storage, memory, and CPU cycles like any other index. By using a separate monitoring deployment, you avoid affecting your other production deployments and can view the logs and metrics even when a production deployment is unavailable. :::: - For example, specify the {{es}} output information in the {{metricbeat}} configuration file (`metricbeat.yml`): ```yaml @@ -152,20 +143,18 @@ See the [{{agent}} command reference](/reference/fleet/agent-command-reference.m 1. In this example, the data is stored on a monitoring cluster with nodes `es-mon-1` and `es-mon-2`. 2. Specify one of `api_key` or `username`/`password`. - If you configured the monitoring cluster to use encrypted communications, you must access it via HTTPS. For example, use a `hosts` setting like `https://es-mon-1:9200`. ::::{important} The {{es}} {{monitor-features}} use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one ingest node. :::: - If the {{es}} {{security-features}} are enabled on the monitoring cluster, you must provide a valid user ID and password so that {{metricbeat}} can send metrics successfully: - 1. Create a user on the monitoring cluster that has the `remote_monitoring_agent` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). Alternatively, if it’s available in your environment, use the `remote_monitoring_user` [built-in user](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). + 1. Create a user on the monitoring cluster that has the `remote_monitoring_agent` [built-in role](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). Alternatively, if it’s available in your environment, use the `remote_monitoring_user` [built-in user](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). 2. Add the `username` and `password` settings to the {{es}} output information in the {{metricbeat}} configuration file. For more information about these configuration options, see [Configure the {{es}} output](beats://reference/metricbeat/elasticsearch-output.md). 6. [Start {{metricbeat}}](beats://reference/metricbeat/metricbeat-starting.md) to begin collecting APM monitoring data. -7. [View the monitoring data in {{kib}}](../../../deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md). +7. [View the monitoring data in {{kib}}](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md). diff --git a/solutions/observability/apps/monitor-resources-on-private-networks.md b/solutions/observability/apps/monitor-resources-on-private-networks.md index d57011f80b..196c9d9783 100644 --- a/solutions/observability/apps/monitor-resources-on-private-networks.md +++ b/solutions/observability/apps/monitor-resources-on-private-networks.md @@ -16,28 +16,25 @@ To monitor resources on private networks you can either: {{private-location}}s via Elastic Agent require only outbound connections from your network, while allowing Elastic’s global managed infrastructure to access a private endpoint requires inbound access, thus posing an additional risk that users must assess. - ## Allow access to your private network [monitor-via-access-control] To give Elastic’s global managed infrastructure access to a private endpoint, use IP address filtering, HTTP authentication, or both. To grant access via IP, use [this list of egress IPs](https://manifest.synthetics.elastic-cloud.com/v1/ip-ranges.json). The addresses and locations on this list may change, so automating updates to filtering rules is recommended. IP filtering alone will allow all users of Elastic’s global managed infrastructure access to your endpoints, if this is a concern consider adding additional protection via user/password authentication via a proxy like nginx. - ## Monitor via a private agent [monitor-via-private-agent] {{private-location}}s allow you to run monitors from your own premises. Before running a monitor on a {{private-location}}, you’ll need to: -* [Set up {{fleet-server}} and {{agent}}](../../../solutions/observability/apps/monitor-resources-on-private-networks.md#synthetics-private-location-fleet-agent). -* [Connect {{fleet}} to the {{stack}}](../../../solutions/observability/apps/monitor-resources-on-private-networks.md#synthetics-private-location-connect) and enroll an {{agent}} in {{fleet}}. -* [Add a {{private-location}}](../../../solutions/observability/apps/monitor-resources-on-private-networks.md#synthetics-private-location-add) in the Synthetics UI. +* [Set up {{fleet-server}} and {{agent}}](/solutions/observability/apps/monitor-resources-on-private-networks.md#synthetics-private-location-fleet-agent). +* [Connect {{fleet}} to the {{stack}}](/solutions/observability/apps/monitor-resources-on-private-networks.md#synthetics-private-location-connect) and enroll an {{agent}} in {{fleet}}. +* [Add a {{private-location}}](/solutions/observability/apps/monitor-resources-on-private-networks.md#synthetics-private-location-add) in the Synthetics UI. ::::{important} -{{private-location}}s running through {{agent}} must have a direct connection to {{es}}. Do not configure any ingest pipelines, or output via Logstash as this will prevent Synthetics from working properly and is not [supported](../../../solutions/observability/apps/synthetics-support-matrix.md). +{{private-location}}s running through {{agent}} must have a direct connection to {{es}}. Do not configure any ingest pipelines, or output via Logstash as this will prevent Synthetics from working properly and is not [supported](/solutions/observability/apps/synthetics-support-matrix.md). :::: - ## Set up {{fleet-server}} and {{agent}} [synthetics-private-location-fleet-agent] Start by setting up {{fleet-server}} and {{agent}}: @@ -46,13 +43,12 @@ Start by setting up {{fleet-server}} and {{agent}}: * **Create an agent policy**: For more information on agent policies and creating them, refer to [{{agent}} policy](/reference/fleet/agent-policy.md#create-a-policy). ::::{important} -A {{private-location}} should be set up against an agent policy that runs on a single {{agent}}. The {{agent}} must be **enrolled in Fleet** ({{private-location}}s cannot be set up using **standalone** {{agents}}). Do *not* run the same agent policy on multiple agents being used for {{private-location}}s, as you may end up with duplicate or missing tests. {{private-location}}s do not currently load balance tests across multiple {{agents}}. See [Scaling {{private-location}}s](../../../solutions/observability/apps/monitor-resources-on-private-networks.md#synthetics-private-location-scaling) for information on increasing the capacity within a {{private-location}}. +A {{private-location}} should be set up against an agent policy that runs on a single {{agent}}. The {{agent}} must be **enrolled in Fleet** ({{private-location}}s cannot be set up using **standalone** {{agents}}). Do *not* run the same agent policy on multiple agents being used for {{private-location}}s, as you may end up with duplicate or missing tests. {{private-location}}s do not currently load balance tests across multiple {{agents}}. See [Scaling {{private-location}}s](/solutions/observability/apps/monitor-resources-on-private-networks.md#synthetics-private-location-scaling) for information on increasing the capacity within a {{private-location}}. By default {{private-location}}s are configured to allow two simultaneous browser tests and an unlimited number of lightweight checks. As a result, if more than two browser tests are assigned to a particular {{private-location}}, there may be a delay to run them. :::: - ## Connect to the {{stack}} or your Observability Serverless project [synthetics-private-location-connect] After setting up {{fleet}}, you’ll connect {{fleet}} to the {{stack}} or your Observability Serverless project and enroll an {{agent}} in {{fleet}}. @@ -64,7 +60,6 @@ The `elastic-agent-complete` Docker image is the only way to have all available :::: - To pull the Docker image run: ```sh @@ -90,13 +85,11 @@ The `elastic-agent-complete` Docker image requires additional capabilities to op :::: - ::::{note} You may need to set other environment variables. Learn how in [{{agent}} environment variables guide](/reference/fleet/agent-environment-variables.md). :::: - ## Add a {{private-location}} [synthetics-private-location-add] When the {{agent}} is running you can add a new {{private-location}} in the UI: @@ -112,7 +105,6 @@ When the {{agent}} is running you can add a new {{private-location}} in the UI: It is not currently possible to use custom CAs for synthetics browser tests in private locations without following a workaround. To learn more about the workaround, refer to the following GitHub issue: [elastic/synthetics#717](https://github.com/elastic/synthetics/issues/717). :::: - ## Scaling {{private-location}}s [synthetics-private-location-scaling] By default {{private-location}}s are configured to allow two simultaneous browser tests, and an unlimited number of lightweight checks. These limits can be set via the environment variables `SYNTHETICS_LIMIT_{{TYPE}}`, where `{{TYPE}}` is one of `BROWSER`, `HTTP`, `TCP`, and `ICMP` for the container running the {{agent}} docker image. @@ -121,7 +113,6 @@ It is critical to allocate enough memory and CPU capacity to handle configured l These limits are for simultaneous tests, not total tests. For example, if 60 browser tests were scheduled to run once per hour and each took 1 minute to run, that would fully occupy one execution slot. However, it is a good practice to set up execution slots with extra capacity. A good starting point would be to over-allocate by a factor of 5. In the previous example that would mean allocating 5 slots. - ## Next steps [synthetics-private-location-next] -Now you can add monitors to your {{private-location}} in [the Synthetics UI](../../../solutions/observability/apps/create-monitors-in-synthetics-app.md) or using the [Elastic Synthetics library’s `push` method](../../../solutions/observability/apps/create-monitors-with-project-monitors.md). \ No newline at end of file +Now you can add monitors to your {{private-location}} in [the Synthetics UI](/solutions/observability/apps/create-monitors-in-synthetics-app.md) or using the [Elastic Synthetics library’s `push` method](/solutions/observability/apps/create-monitors-with-project-monitors.md). \ No newline at end of file diff --git a/solutions/observability/apps/monitoring-aws-lambda-functions.md b/solutions/observability/apps/monitoring-aws-lambda-functions.md index e9398ce822..86bacd58ca 100644 --- a/solutions/observability/apps/monitoring-aws-lambda-functions.md +++ b/solutions/observability/apps/monitoring-aws-lambda-functions.md @@ -10,8 +10,7 @@ applies_to: # Monitoring AWS Lambda Functions [apm-monitoring-aws-lambda] - -Elastic APM lets you monitor your AWS Lambda functions. The natural integration of [distributed tracing](../../../solutions/observability/apps/traces.md#apm-distributed-tracing) into your AWS Lambda functions provides insights into the function’s execution and runtime behavior as well as its relationships and dependencies to other services. +Elastic APM lets you monitor your AWS Lambda functions. The natural integration of [distributed tracing](/solutions/observability/apps/traces.md#apm-distributed-tracing) into your AWS Lambda functions provides insights into the function’s execution and runtime behavior as well as its relationships and dependencies to other services. ## AWS Lambda architecture [aws-lambda-arch] diff --git a/solutions/observability/apps/multi-factor-authentication-mfa-for-browser-monitors.md b/solutions/observability/apps/multi-factor-authentication-mfa-for-browser-monitors.md index 0dd1224cc1..006267ec22 100644 --- a/solutions/observability/apps/multi-factor-authentication-mfa-for-browser-monitors.md +++ b/solutions/observability/apps/multi-factor-authentication-mfa-for-browser-monitors.md @@ -10,15 +10,13 @@ applies_to: # Multi-factor Authentication (MFA) for browser monitors [synthetics-mfa] - Multi-factor Authentication (MFA) adds an essential layer of security to applications login processes, protecting against unauthorized access. A very common use case in Synthetics is testing user journeys involving websites protected by MFA. Synthetics supports testing websites secured by Time-based One-Time Password (TOTP), a common MFA method that provides short-lived one-time tokens to enhance security. - ## Configuring TOTP for MFA [configuring_totp_for_mfa] -To test a browser journey that uses TOTP for MFA, first configure the Synthetics authenticator token in the target application. To do this, generate a One-Time Password (OTP) using the Synthetics CLI; refer to [`@elastic/synthetics totp `](../../../solutions/observability/apps/use-synthetics-cli.md). +To test a browser journey that uses TOTP for MFA, first configure the Synthetics authenticator token in the target application. To do this, generate a One-Time Password (OTP) using the Synthetics CLI; refer to [`@elastic/synthetics totp `](/solutions/observability/apps/use-synthetics-cli.md). ```sh npx @elastic/synthetics totp @@ -27,7 +25,6 @@ npx @elastic/synthetics totp OTP Token: 123456 ``` - ## Applying the TOTP Token in Browser Journeys [applying_the_totp_token_in_browser_journeys] Once the Synthetics TOTP Authentication is configured in your application, you can now use the OTP token in the synthetics browser journeys using the `mfa` object imported from `@elastic/synthetics`. diff --git a/solutions/observability/apps/observe-lambda-functions.md b/solutions/observability/apps/observe-lambda-functions.md index ce583543bd..c7b393ac51 100644 --- a/solutions/observability/apps/observe-lambda-functions.md +++ b/solutions/observability/apps/observe-lambda-functions.md @@ -18,25 +18,21 @@ To set up Lambda monitoring, refer to [AWS Lambda functions](/solutions/observab :screenshot: ::: - ## Cold starts [apm-lambda-cold-start-info] A cold start occurs when a Lambda function has not been used for a certain period of time. A lambda worker receives a request to run the function and prepares an execution environment. Cold starts are an unavoidable byproduct of the serverless world, but visibility into how they impact your services can help you make better decisions about factors like how much memory to allocate to a function, whether to enable provisioned concurrency, or if it’s time to consider removing a large dependency. - ### Cold start rate [apm-lambda-cold-start-rate] The cold start rate (i.e. proportion of requests that experience a cold start) is displayed per service and per transaction. Cold start is also displayed in the trace waterfall, where you can drill-down into individual traces and see trace metadata like AWS request ID, trigger type, and trigger request ID. - ### Latency distribution correlation [apm-lambda-cold-start-latency] -The [latency correlations](../../../solutions/observability/apps/find-transaction-latency-failure-correlations.md) feature can be used to visualize the impact of Lambda cold starts on latency—​just select the `faas.coldstart` field. - +The [latency correlations](/solutions/observability/apps/find-transaction-latency-failure-correlations.md) feature can be used to visualize the impact of Lambda cold starts on latency—​just select the `faas.coldstart` field. ## AWS Lambda function grouping [apm-lambda-service-config] diff --git a/solutions/observability/apps/opentelemetry-intake-api.md b/solutions/observability/apps/opentelemetry-intake-api.md index f051269423..815a0c065f 100644 --- a/solutions/observability/apps/opentelemetry-intake-api.md +++ b/solutions/observability/apps/opentelemetry-intake-api.md @@ -14,7 +14,6 @@ APM Server supports two OTLP communication protocols on the same port: * OTLP/HTTP (protobuf) * OTLP/gRPC - ## OTLP/gRPC paths [_otlpgrpc_paths] | Name | Endpoint | @@ -23,7 +22,6 @@ APM Server supports two OTLP communication protocols on the same port: | OTLP trace intake | `/opentelemetry.proto.collector.trace.v1.TraceService/Export` | | OTLP logs intake | `/opentelemetry.proto.collector.logs.v1.LogsService/Export` | - ## OTLP/HTTP paths [_otlphttp_paths] | Name | Endpoint | @@ -36,4 +34,3 @@ APM Server supports two OTLP communication protocols on the same port: See our OpenTelemetry documentation to learn how to send data to the APM Server from an [OpenTelemetry agent](upstream-opentelemetry-collectors-language-sdks.md#apm-instrument-apps-otel) or [OpenTelemetry collector](upstream-opentelemetry-collectors-language-sdks.md#apm-connect-open-telemetry-collector). :::: - diff --git a/solutions/observability/apps/overviews.md b/solutions/observability/apps/overviews.md index 80e991c331..a97eab1e6c 100644 --- a/solutions/observability/apps/overviews.md +++ b/solutions/observability/apps/overviews.md @@ -10,21 +10,16 @@ applies_to: # High-level overviews of application data [apm-ui] - For a quick, high-level overview of the health and performance of your application, start with: -* [Services](../../../solutions/observability/apps/services.md) -* [Traces](../../../solutions/observability/apps/traces-2.md) -* [Dependencies](../../../solutions/observability/apps/dependencies.md) -* [Service Map](../../../solutions/observability/apps/service-map.md) - +* [Services](/solutions/observability/apps/services.md) +* [Traces](/solutions/observability/apps/traces-2.md) +* [Dependencies](/solutions/observability/apps/dependencies.md) +* [Service Map](/solutions/observability/apps/service-map.md) ## View an individual service [apm-ui-individual-service] -```{applies_to} -stack: all -``` View an individual service: -* [Service overview](../../../solutions/observability/apps/service-overview.md) -* [Mobile service overview](../../../solutions/observability/apps/mobile-service-overview.md) +* [Service overview](/solutions/observability/apps/service-overview.md) +* [Mobile service overview](/solutions/observability/apps/mobile-service-overview.md) diff --git a/solutions/observability/apps/parse-data-using-ingest-pipelines.md b/solutions/observability/apps/parse-data-using-ingest-pipelines.md index 6055144a4b..354b385237 100644 --- a/solutions/observability/apps/parse-data-using-ingest-pipelines.md +++ b/solutions/observability/apps/parse-data-using-ingest-pipelines.md @@ -11,7 +11,6 @@ Ingest pipelines preprocess and enrich APM documents before indexing them. For e The default APM pipelines are defined in {{es}} apm-data plugin index templates. {{es}} then uses the index pattern in these index templates to match pipelines to APM data streams. - ## Custom ingest pipelines [custom-ingest-pipelines] Elastic APM supports custom ingest pipelines. A custom pipeline allows you to transform data to better match your specific use case. This can be useful, for example, to ensure data security by removing or obfuscating sensitive information. @@ -20,7 +19,6 @@ Each data stream ships with a default pipeline. This default pipeline calls an i In addition, ingest pipelines can also be used to direct application metrics (`metrics-apm.app.*`) to a data stream with a different dataset, e.g. to combine metrics for two applications. Sending other APM data to alternate data streams, like traces (`traces-apm.*`), logs (`logs-apm.*`), and internal metrics (`metrics-apm.internal*`) is not currently supported. - ## `@custom` ingest pipeline naming convention [custom-ingest-pipeline-naming] `@custom` pipelines are specific to each data stream and follow a similar naming convention: `-@custom`. As a reminder, the default APM data streams are: @@ -40,7 +38,6 @@ To match a custom ingest pipeline with a data stream, follow the `-(10 agents) | 15,000
events/second | 14,000
events/second | 17,000
events/second | -| **4 GB**
(30 agents) | 29,000
events/second | 26,000
events/second | 35,000
events/second | -| **8 GB**
(60 agents) | 50,000
events/second | 34,000
events/second | 48,000
events/second | -| **16 GB**
(120 agents) | 96,000
events/second | 57,000
events/second | 90,000
events/second | -| **32 GB**
(240 agents) | 133,000
events/second | 89,000
events/second | 143,000
events/second | - -Don’t forget that the APM Server is stateless. Several instances running do not need to know about each other. This means that with a properly sized {{es}} instance, APM Server scales out linearly. - -::::{note} -RUM deserves special consideration. The RUM agent runs in browsers, and there can be many thousands reporting to an APM Server with very variable network latency. -:::: - - -Alternatively or in addition to scaling the APM Server, consider decreasing the ingestion volume. Read more in [Reduce storage](reduce-storage.md). - diff --git a/solutions/observability/apps/reader-role.md b/solutions/observability/apps/reader-role.md index e157e33007..8366e09b58 100644 --- a/solutions/observability/apps/reader-role.md +++ b/solutions/observability/apps/reader-role.md @@ -12,21 +12,19 @@ For users who need to view and create visualizations that access Synthetics data * **General read access**: For most users, you can use [General read access](#synthetics-read-privileges-general), which grants read access to all {{kib}} apps and requires little configuration. * **Limited read access**: If you want to limit read access to the {{synthetics-app}} only, you can use [Limited read access](#synthetics-read-privileges-limited), which requires additional configuration. - ## General read access [synthetics-read-privileges-general] -For users who only need to view results in {{kib}}, use the `viewer` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). - +For users who only need to view results in {{kib}}, use the `viewer` [built-in role](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). ## Limited read access [synthetics-read-privileges-limited] -If you want to limit read access to the {{synthetics-app}} only, do *not* use the `viewer` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). +If you want to limit read access to the {{synthetics-app}} only, do *not* use the `viewer` [built-in role](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). Instead to you can create a reader role, called something like `synthetics_reader_limited`, and grant the following privileges: | Type | Privilege | Purpose | | --- | --- | --- | -| [Index](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) | `synthetics-*`: `read` | Read-only access to synthetics indices. | -| [Index](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) | `.alerts-observability.uptime.alerts-*`: `read` | Read-only access to synthetics alert indices. | -| [Kibana](../../../deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges.md) | `Synthetics and Uptime`: `All` | Access to the {{synthetics-app}} in {{kib}}. | +| [Index](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) | `synthetics-*`: `read` | Read-only access to synthetics indices. | +| [Index](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) | `.alerts-observability.uptime.alerts-*`: `read` | Read-only access to synthetics alert indices. | +| [Kibana](/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges.md) | `Synthetics and Uptime`: `All` | Access to the {{synthetics-app}} in {{kib}}. | diff --git a/solutions/observability/apps/real-user-monitoring-user-experience.md b/solutions/observability/apps/real-user-monitoring-user-experience.md index e58c8d1cf0..798c53f50d 100644 --- a/solutions/observability/apps/real-user-monitoring-user-experience.md +++ b/solutions/observability/apps/real-user-monitoring-user-experience.md @@ -6,11 +6,8 @@ applies_to: stack: all --- - - # Real user monitoring (User Experience) [user-experience] - {{user-experience}} provides a way to quantify and analyze the perceived performance of your web application. Unlike testing environments, {{user-experience}} data reflects real-world user experiences. Drill down further by looking at data by URL, operating system, browser, and location — all of which can impact how your application performs on end-user machines. Powered by the APM Real user monitoring (RUM) agent, all it takes is a few lines of code to begin surfacing key user experience metrics. @@ -20,22 +17,18 @@ Powered by the APM Real user monitoring (RUM) agent, all it takes is a few lines :screenshot: ::: - ## Why is {{user-experience}} important? [why-user-experience] Search engines are placing increasing importance on user experience when organically ranking websites. Elastic makes it easy to view your website data in the context of Google Core Web Vitals — metrics that score three key areas of user experience: loading performance, visual stability, and interactivity. These Core Web Vitals are set to become the main performance measurement in Google ranking factors. If you’re a content-based site that wants to appear in the “Top Stories” section of Google search results, you must have good Core Web Vitals. - ## How does {{user-experience}} work? [how-user-experience-works] {{user-experience}} metrics are powered by the [APM Real User Monitoring (RUM) agent](https://www.elastic.co/guide/en/apm/agent/rum-js/current). The RUM agent uses browser timing APIs, like [Navigation Timing](https://w3c.github.io/navigation-timing/), [Resource Timing](https://w3c.github.io/resource-timing/), [Paint Timing](https://w3c.github.io/paint-timing/), and [User Timing](https://w3c.github.io/user-timing/), to capture {{user-experience}} metrics every time a user hits one of your pages. This data is stored in {{es}}, where it can be visualized using {{kib}}. The RUM agent can be installed as a dependency to your application, or with just a few lines of JavaScript. It only takes a few minutes to [get started](real-user-monitoring-rum.md). - ## {{user-experience}} in {{kib}} [user-experience-tab] - ### Page load duration [user-experience-page-load] This high-level overview is your analysis starting point and answers questions like: How long is my server taking to respond to requests? How much time is spent parsing and painting that content? How many page views has my site received? @@ -47,7 +40,6 @@ You won’t be able to fix any problems from viewing these metrics alone, but yo :screenshot: ::: - ### {{user-experience}} metrics [user-experience-metrics] {{user-experience}} metrics help you understand the perceived performance of your website. For example, first contentful paint is the timestamp when the browser begins rendering content. In other words, it’s around this time that a user first gets feedback that the page is loading. @@ -78,10 +70,8 @@ Total long tasks duration :::: - These metrics tell an important story about how users experience your website. But developers shouldn’t have to become experts at interpreting and acting on these signals; they should spend their time reacting to the opportunities that these metrics present. For that reason (and many others), Elastic has embraced Google Core Web Vitals. - #### Core Web Vitals [user-experience-core-vitals] [Core Web Vitals](https://web.dev/vitals/) is a recent initiative from Google to introduce a new set of metrics that better categorize good and bad sites by quantifying the real-world user experience. This is done by looking at three key metrics: loading performance, visual stability, and interactivity: @@ -101,11 +91,10 @@ Previous {{kib}} versions included the metric [First input delay (FID)](https:// | **RUM agent version ≥ 5.16.0** | INP data will be visible. | FID data will be visible. | | **RUM agent version < 5.16.0** | The INP section will be empty. | FID data will be visible. | -RUM agent version ≥ 5.16.0 will continue to collect FID metrics so, while FID metrics are not shown in the User Experience app in {{kib}} versions 8.12 and later, you can choose to visualize FID metrics in a [custom dashboard](../../../explore-analyze/dashboards/create-dashboard-of-panels-with-web-server-data.md) or using [Lens](../../../explore-analyze/visualize/lens.md). +RUM agent version ≥ 5.16.0 will continue to collect FID metrics so, while FID metrics are not shown in the User Experience app in {{kib}} versions 8.12 and later, you can choose to visualize FID metrics in a [custom dashboard](/explore-analyze/dashboards/create-dashboard-of-panels-with-web-server-data.md) or using [Lens](/explore-analyze/visualize/lens.md). :::: - Cumulative layout shift (CLS) : Visual stability. Is content moving around because of `async` resource loading or dynamic content additions? CLS measures these frustrating unexpected layout shifts. To provide a good user experience, Google recommends a CLS score of less than `.1`. [7](https://web.dev/cls/)] @@ -113,8 +102,6 @@ Cumulative layout shift (CLS) [Beginning in May 2021](https://webmasters.googleblog.com/2020/11/timing-for-page-experience.md), Google will start using Core Web Vitals as part of their ranking algorithm and will open up the opportunity for websites to rank in the "top stories" position without needing to leverage [AMP](https://amp.dev/). [8](https://webmasters.googleblog.com/2020/05/evaluating-page-experience.md)] :::: - - ### Load/view distribution [user-experience-distribution] Operating system, browser family, and geographic location can all have a massive impact on how visitors experience your website. This data can help you understand when and where your users are visiting from, and can help you prioritize optimizations — for example, prioritizing improvements for the most popular browsers visiting your site. @@ -126,7 +113,6 @@ Don’t forget, this data also influences search engine page rankings and placem :screenshot: ::: - ### Error breakdown [user-experience-errors] JavaScript errors can be detrimental to a users experience on your website. But variation in users' software and hardware makes it nearly impossible to test for every combination. And, as JavaScript continues to get more and more complex, the need for user experience monitoring and error reporting only increases. Error monitoring makes this visible by surfacing JavaScript errors that are occurring on your website in production. @@ -138,11 +124,9 @@ JavaScript errors can be detrimental to a users experience on your website. But Open error messages in APM for additional analysis tools, like occurrence rates, transaction ids, user data, and more. - ### Feedback and troubleshooting [user-experience-feedback] Have a question? Want to leave feedback? Visit the [{{user-experience}} discussion forum](https://discuss.elastic.co/c/observability/user-experience/87). - #### References [user-experience-references] diff --git a/solutions/observability/apps/reduce-storage.md b/solutions/observability/apps/reduce-storage.md index df19ecccbe..5e173d50d2 100644 --- a/solutions/observability/apps/reduce-storage.md +++ b/solutions/observability/apps/reduce-storage.md @@ -11,13 +11,11 @@ applies_to: The richness and volume of APM data provides unique insights into your applications, but it can also mean higher costs and more noise when analyzing data. There are a couple strategies you can use to reduce your data usage while continuing to get the full value of APM data. - ## Reduce the sample rate [apm-reduce-sample-rate] Distributed tracing can generate a substantial amount of data. More data can mean higher costs and more noise. Sampling aims to lower the amount of data ingested and the effort required to analyze that data. -See [Transaction sampling](../../../solutions/observability/apps/transaction-sampling.md) to learn more. - +See [Transaction sampling](/solutions/observability/apps/transaction-sampling.md) to learn more. ## Enable span compression [enable_span_compression] @@ -25,7 +23,6 @@ In some cases, APM agents may collect large amounts of very similar or identical See [Span compression](/solutions/observability/apps/spans.md#apm-spans-span-compression) to learn more. - ## Reduce collected stack trace information [observability-apm-reduce-stacktrace] Elastic APM agents collect `stacktrace` information under certain circumstances. This can be very helpful in identifying issues in your code, but it also comes with an overhead at collection time and increases your storage usage. @@ -35,25 +32,25 @@ Stack trace collection settings are managed in each APM agent. You can enable an % Stateful only after this? ## Delete data [delete_data] + ```{applies_to} -stack: all +stack: +serverless: unavailable ``` You might want to only keep data for a defined time period. This might mean deleting old documents periodically, deleting data collected for specific services or customers, or deleting specific indices. Depending on your use case, you can delete data: -* periodically with [{{ilm}}](../../../solutions/observability/apps/reduce-storage.md#apm-delete-data-with-ilm) -* [matching a query](../../../solutions/observability/apps/reduce-storage.md#apm-delete-data-query) -* with the [{{kib}} Index Management UI](../../../solutions/observability/apps/reduce-storage.md#apm-delete-data-in-kibana) - -If you want to delete data for security or privacy reasons, see [Secure data](../../../solutions/observability/apps/application-data-security.md). +* periodically with [{{ilm}}](/solutions/observability/apps/reduce-storage.md#apm-delete-data-with-ilm) +* [matching a query](/solutions/observability/apps/reduce-storage.md#apm-delete-data-query) +* with the [{{kib}} Index Management UI](/solutions/observability/apps/reduce-storage.md#apm-delete-data-in-kibana) +If you want to delete data for security or privacy reasons, see [Secure data](/solutions/observability/apps/application-data-security.md). ### Delete data with {{ilm}} ({{ilm-init}}) [apm-delete-data-with-ilm] -Index lifecycle management enables you to automate how you want to manage your indices over time. You can base actions on factors such as shard size and performance requirements. See [{{ilm-cap}}](../../../solutions/observability/apps/index-lifecycle-management.md) to learn more. - +Index lifecycle management enables you to automate how you want to manage your indices over time. You can base actions on factors such as shard size and performance requirements. See [{{ilm-cap}}](/solutions/observability/apps/index-lifecycle-management.md) to learn more. ### Delete data matching a query [apm-delete-data-query] @@ -72,17 +69,17 @@ POST /.ds-*-apm*/_delete_by_query } ``` - ### Delete data with {{kib}} Index Management [apm-delete-data-in-kibana] -{{kib}}'s [Index Management](../../../manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) allows you to manage your cluster’s indices, data streams, index templates, and much more. +{{kib}}'s [Index Management](/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) allows you to manage your cluster’s indices, data streams, index templates, and much more. To open **Index Management**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). Select **Data Streams**. Select the data streams you want to delete, and click **Delete data streams**. - ## Update existing data [apm-update-data] + ```{applies_to} -stack: all +stack: +serverless: unavailable ``` You might want to update documents that are already indexed. For example, if you your service name was set incorrectly. diff --git a/solutions/observability/apps/resource-atrributes.md b/solutions/observability/apps/resource-atrributes.md index 7a5261e2f1..b1771ee360 100644 --- a/solutions/observability/apps/resource-atrributes.md +++ b/solutions/observability/apps/resource-atrributes.md @@ -45,4 +45,4 @@ Need to add event attributes instead? Use attributes—​not to be confused wit Elastic integrates with OpenTelemetry, allowing you to reuse your existing instrumentation to easily send observability data to the {{stack}}. -For more information on how to combine Elastic and OpenTelemetry, see [OpenTelemetry integration](../../../solutions/observability/apps/use-opentelemetry-with-apm.md). \ No newline at end of file +For more information on how to combine Elastic and OpenTelemetry, see [OpenTelemetry integration](/solutions/observability/apps/use-opentelemetry-with-apm.md). \ No newline at end of file diff --git a/solutions/observability/apps/scale-architect-synthetics-deployment.md b/solutions/observability/apps/scale-architect-synthetics-deployment.md index d7453d027a..b1cccdd390 100644 --- a/solutions/observability/apps/scale-architect-synthetics-deployment.md +++ b/solutions/observability/apps/scale-architect-synthetics-deployment.md @@ -25,12 +25,10 @@ You may, however, use Dashboards and the Discover feature with CCS to view `synt The Synthetics UI does *not* work with CCS/CCR because it would limit the rich user experience that the Synthetics UI provides. Unlike the majority of {{kib}} apps, the Synthetics UI relies heavily on state stored in {{kib}} shared objects in order to provide a rich user experience. Because {{kib}} saved objects cannot be shared via CCS/CCR, the Synthetics UI will not show any user data even if CCS/CCR is configured. - ## Manage large numbers of Synthetic monitors with tags [synthetics-tagging] When managing larger numbers of synthetic monitors, use tags to keep them organized. Many of the views in the Synthetics UI are tag-aware and can group data by tag. - ## Create custom dashboards [synthetics-custom-dashboards] -If we don't provide a UI for your exact needs, you can use [dashboards](../../../explore-analyze/dashboards.md) to build custom visualizations. For a complete accounting of fields used by the Synthetics UI, refer to [{{heartbeat}}'s exported fields](beats://reference/heartbeat/exported-fields.md). \ No newline at end of file +If we don't provide a UI for your exact needs, you can use [dashboards](/explore-analyze/dashboards.md) to build custom visualizations. For a complete accounting of fields used by the Synthetics UI, refer to [{{heartbeat}}'s exported fields](beats://reference/heartbeat/exported-fields.md). \ No newline at end of file diff --git a/solutions/observability/apps/scripting-browser-monitors.md b/solutions/observability/apps/scripting-browser-monitors.md index a409faec57..9d9c3af3cf 100644 --- a/solutions/observability/apps/scripting-browser-monitors.md +++ b/solutions/observability/apps/scripting-browser-monitors.md @@ -15,11 +15,11 @@ You can use synthetic monitors to detect bugs caused by invalid states you could Start by learning the basics of synthetic monitoring, including how to: -* [Write a synthetic test](../../../solutions/observability/apps/write-synthetic-test.md) -* [Test locally](../../../solutions/observability/apps/write-synthetic-test.md#synthetics-test-locally) -* [Configure individual browser monitors](../../../solutions/observability/apps/configure-individual-browser-monitors.md) -* [Work with params and secrets](../../../solutions/observability/apps/work-with-params-secrets.md) -* [Use the Synthetics Recorder](../../../solutions/observability/apps/use-synthetics-recorder.md) +* [Write a synthetic test](/solutions/observability/apps/write-synthetic-test.md) +* [Test locally](/solutions/observability/apps/write-synthetic-test.md#synthetics-test-locally) +* [Configure individual browser monitors](/solutions/observability/apps/configure-individual-browser-monitors.md) +* [Work with params and secrets](/solutions/observability/apps/work-with-params-secrets.md) +* [Use the Synthetics Recorder](/solutions/observability/apps/use-synthetics-recorder.md) :::{image} /solutions/images/observability-synthetic-monitor-lifecycle.png :alt: Diagram of the lifecycle of a synthetic monitor: write a test diff --git a/solutions/observability/apps/secret-token.md b/solutions/observability/apps/secret-token.md index 28f4c91e9a..32c20644e8 100644 --- a/solutions/observability/apps/secret-token.md +++ b/solutions/observability/apps/secret-token.md @@ -11,7 +11,6 @@ applies_to: Secret tokens are sent as plain-text, so they only provide security when used in combination with [TLS](apm-agent-tls-communication.md). :::: - When defined, secret tokens are used to authorize requests to the APM Server. Both the {{apm-agent}} and APM Server must be configured with the same secret token for the request to be accepted. To secure the communication between APM agents and the APM Server with a secret token: @@ -24,8 +23,6 @@ To secure the communication between APM agents and the APM Server with a secret Secret tokens are not applicable for the RUM Agent, as there is no way to prevent them from being publicly exposed. :::: - - ## Create a secret token [apm-create-secret-token] % This should be checked and updated, it doesn't looks like it's there by default vv @@ -34,7 +31,6 @@ Secret tokens are not applicable for the RUM Agent, as there is no way to preven {{ech}} and {{ece}} deployments provision a secret token when the deployment is created. The secret token can be found and reset in the {{ecloud}} Console under the deployment's **APM & Fleet** page. :::: - :::::::{tab-set} ::::::{tab-item} Fleet-managed diff --git a/solutions/observability/apps/secrets-keystore-for-secure-settings.md b/solutions/observability/apps/secrets-keystore-for-secure-settings.md index 559d79fbab..d9445b30db 100644 --- a/solutions/observability/apps/secrets-keystore-for-secure-settings.md +++ b/solutions/observability/apps/secrets-keystore-for-secure-settings.md @@ -6,16 +6,12 @@ applies_to: stack: all --- - - # Secrets keystore for secure settings [apm-keystore] - ::::{important} The APM Server keystore only applies to the APM Server binary installation method. :::: - When you configure APM Server, you might need to specify sensitive settings, such as passwords. Rather than relying on file system permissions to protect these values, you can use the APM Server keystore to securely store secret values for use in configuration settings. After adding a key and its secret value to the keystore, you can use the key in place of the secret value when you configure sensitive settings. @@ -41,8 +37,6 @@ To create and manage keys, use the `keystore` command. See the [command referenc The `keystore` command must be run by the same user who will run APM Server. :::: - - ## Create a keystore [apm-creating-keystore] To create a secrets keystore, use: @@ -53,7 +47,6 @@ apm-server keystore create APM Server creates the keystore in the directory defined by the `path.data` configuration setting. - ## Add keys [apm-add-keys-to-keystore] To store sensitive values, such as authentication credentials for {{es}}, use the `keystore add` command: @@ -76,7 +69,6 @@ To pass the value through stdin, use the `--stdin` flag. You can also use `--for cat /file/containing/setting/value | apm-server keystore add ES_PWD --stdin --force ``` - ## List keys [apm-list-settings] To list the keys defined in the keystore, use: @@ -85,7 +77,6 @@ To list the keys defined in the keystore, use: apm-server keystore list ``` - ## Remove keys [apm-remove-settings] To remove a key from the keystore, use: diff --git a/solutions/observability/apps/secure-access-to-applications-ui.md b/solutions/observability/apps/secure-access-to-applications-ui.md index a600cdcced..b875bd6803 100644 --- a/solutions/observability/apps/secure-access-to-applications-ui.md +++ b/solutions/observability/apps/secure-access-to-applications-ui.md @@ -8,10 +8,9 @@ applies_to: # Secure access to the Applications UI [apm-app-users] - Use role-based access control to grant users access to secured resources. The roles that you set up depend on your organization’s security requirements and the minimum privileges required to use specific features. -{{es-security-features}} provides [built-in roles](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) that grant a subset of the privileges needed by APM users. When possible, assign users the built-in roles to minimize the affect of future changes on your security strategy. If no built-in role is available, you can assign users the privileges needed to accomplish a specific task. In general, there are three types of privileges you’ll work with: +{{es-security-features}} provides [built-in roles](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) that grant a subset of the privileges needed by APM users. When possible, assign users the built-in roles to minimize the affect of future changes on your security strategy. If no built-in role is available, you can assign users the privileges needed to accomplish a specific task. In general, there are three types of privileges you’ll work with: * **Elasticsearch cluster privileges**: Manage the actions a user can perform against your cluster. * **Elasticsearch index privileges**: Control access to the data in specific indices your cluster. diff --git a/solutions/observability/apps/secure-communication-with-apm-agents.md b/solutions/observability/apps/secure-communication-with-apm-agents.md index 3aed4796d3..ab525b0559 100644 --- a/solutions/observability/apps/secure-communication-with-apm-agents.md +++ b/solutions/observability/apps/secure-communication-with-apm-agents.md @@ -3,6 +3,7 @@ mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-secure-agent-communication.html applies_to: stack: + serverless: --- # Secure communication with APM agents [apm-secure-agent-communication] @@ -17,7 +18,3 @@ As soon as an authenticated communication is enabled, requests without a valid t In some use-cases, like when an {{apm-agent}} is running on the client side, authentication is not possible. See [Anonymous authentication](anonymous-authentication.md) for more information. - - - - diff --git a/solutions/observability/apps/secure-communication-with-elastic-stack.md b/solutions/observability/apps/secure-communication-with-elastic-stack.md index 6e51dec046..587bf885a8 100644 --- a/solutions/observability/apps/secure-communication-with-elastic-stack.md +++ b/solutions/observability/apps/secure-communication-with-elastic-stack.md @@ -11,24 +11,20 @@ applies_to: This documentation only applies to the APM Server binary. :::: - Use role-based access control or API keys to grant APM Server users access to secured resources. - ## Role-based access [apm-secure-comms-stack-role-based] Manage access on a feature-by-feature basis by creating several custom feature-related roles and assigning one or more of these roles to each APM Server user based on which features they need to access. [**Read more in Use feature roles →**](create-assign-feature-roles-to-apm-server-users.md) - ## API keys [apm-secure-comms-stack-api-keys] Instead of using usernames and passwords, you can use API keys to grant access to Elasticsearch resources. You can set API keys to expire at a certain time, and you can explicitly invalidate them. [**Read more in Grant access using API keys →**](grant-access-using-api-keys.md) - ## More resources [_more_resources] After privileged users have been created, use authentication to connect to a secured Elastic cluster. @@ -47,6 +43,3 @@ APM Server exposes an HTTP endpoint, and as with anything that opens ports on yo :::: - - - diff --git a/solutions/observability/apps/service-map.md b/solutions/observability/apps/service-map.md index 3b7db18c9e..7c7da48290 100644 --- a/solutions/observability/apps/service-map.md +++ b/solutions/observability/apps/service-map.md @@ -16,27 +16,23 @@ We currently surface two types of service maps: * **Global**: All services instrumented with APM agents and the connections between them are shown. * **Service-specific**: Highlight connections for a selected service. - ## How do service maps work? [service-maps-how] Service Maps rely on distributed traces to draw connections between services. As [distributed tracing](/solutions/observability/apps/traces.md) is enabled out-of-the-box for supported technologies, so are service maps. However, if a service isn’t instrumented, or a `traceparent` header isn’t being propagated to it, distributed tracing will not work, and the connection will not be drawn on the map. - ## Visualize your architecture [visualize-your-architecture] Select the **Service Map** tab to get started. By default, all instrumented services and connections are shown. Whether you’re onboarding a new engineer, or just trying to grasp the big picture, drag things around, zoom in and out, and begin to visualize how your services are connected. -Customize what the service map displays using either the query bar or the environment selector. The query bar enables you to use [advanced queries](../../../solutions/observability/apps/use-advanced-queries-on-application-data.md) to customize the service map based on your needs. The environment selector allows you to narrow displayed results to a specific environment. This can be useful if you have two or more services, in separate environments, but with the same name. Use the environment drop-down to only see the data you’re interested in, like `dev` or `production`. +Customize what the service map displays using either the query bar or the environment selector. The query bar enables you to use [advanced queries](/solutions/observability/apps/use-advanced-queries-on-application-data.md) to customize the service map based on your needs. The environment selector allows you to narrow displayed results to a specific environment. This can be useful if you have two or more services, in separate environments, but with the same name. Use the environment drop-down to only see the data you’re interested in, like `dev` or `production`. If there’s a specific service that interests you, select that service to highlight its connections. Click **Focus map** to refocus the map on the selected service and lock the connection highlighting. Click the **Transactions** tab to jump to the Transaction overview for the selected service. You can also use the tabs at the top of the page to easily jump to the **Errors** or **Metrics** overview. - :::{image} /solutions/images/observability-service-maps-java.png :alt: Example view of service maps in the Applications UI in Kibana :screenshot: ::: - ## Anomaly detection with machine learning [service-map-anomaly-detection] You can create machine learning jobs to calculate anomaly scores on APM transaction durations within the selected service. When these jobs are active, service maps will display a color-coded anomaly indicator based on the detected anomaly score: @@ -54,8 +50,7 @@ You can create machine learning jobs to calculate anomaly scores on APM transact If an anomaly has been detected, click **View anomalies** to view the anomaly detection metric viewer. This time series analysis will display additional details on the severity and time of the detected anomalies. -To learn how to create a machine learning job, refer to [Integrate with machine learning](../../../solutions/observability/apps/integrate-with-machine-learning.md). - +To learn how to create a machine learning job, refer to [Integrate with machine learning](/solutions/observability/apps/integrate-with-machine-learning.md). ## Legend [service-maps-legend] @@ -64,7 +59,6 @@ Nodes appear on the map in one of two shapes: * **Circle**: Instrumented services. Interior icons are based on the language of the APM agent used. * **Diamond**: Databases, external, and messaging. Interior icons represent the generic type, with specific icons for known entities, like Elasticsearch. Type and subtype are based on `span.type`, and `span.subtype`. - ## Supported APM agents [service-maps-supported] Service Maps are supported for the following APM agent versions: diff --git a/solutions/observability/apps/service-overview.md b/solutions/observability/apps/service-overview.md index f8eba3d080..a800020053 100644 --- a/solutions/observability/apps/service-overview.md +++ b/solutions/observability/apps/service-overview.md @@ -9,7 +9,7 @@ applies_to: # Service overview [apm-service-overview] -Selecting a non-mobile [**service**](../../../solutions/observability/apps/services.md) brings you to the **Service overview**. The **Service overview** contains a wide variety of charts and tables that provide high-level visibility into how a service is performing across your infrastructure: +Selecting a non-mobile [**service**](/solutions/observability/apps/services.md) brings you to the **Service overview**. The **Service overview** contains a wide variety of charts and tables that provide high-level visibility into how a service is performing across your infrastructure: * Service details like service version, runtime version, framework, and APM agent name and version * Container and orchestration information @@ -35,8 +35,7 @@ Select the **Comparison** box to apply a time-based or expected bounds compariso | > 24 hours and ≤ 7 days | One week | | > 7 days | An identical amount of time immediately before the selected time range | -The expected bounds comparison is powered by [machine learning](../../../solutions/observability/apps/integrate-with-machine-learning.md) and requires anomaly detection to be enabled. - +The expected bounds comparison is powered by [machine learning](/solutions/observability/apps/integrate-with-machine-learning.md) and requires anomaly detection to be enabled. ## Latency [service-latency] @@ -47,15 +46,13 @@ Response times for the service. You can filter the **Latency** chart to display :screenshot: ::: - ## Throughput and transactions [service-throughput-transactions] The **Throughput** chart visualizes the average number of transactions per minute for the selected service. The **Transactions** table displays a list of *transaction groups* for the selected service and includes the latency, traffic, error rate, and the impact for each transaction. Transactions that share the same name are grouped, and only one entry is displayed for each group. -By default, transaction groups are sorted by *Impact* to show the most used and slowest endpoints in your service. If there is a particular endpoint you are interested in, click **View transactions** to view a list of similar transactions on the [transactions overview](../../../solutions/observability/apps/transactions-2.md) page. - +By default, transaction groups are sorted by *Impact* to show the most used and slowest endpoints in your service. If there is a particular endpoint you are interested in, click **View transactions** to view a list of similar transactions on the [transactions overview](/solutions/observability/apps/transactions-2.md) page. ## Failed transaction rate and errors [service-error-rates] @@ -70,7 +67,6 @@ If there is no HTTP status, both transactions and spans are considered successfu :::: - The **Errors** table provides a high-level view of each error message when it first and last occurred, along with the total number of occurrences. This makes it very easy to quickly see which errors affect your services and take actions to rectify them. To do so, click **View errors**. :::{image} /solutions/images/observability-error-rate.png @@ -78,12 +74,11 @@ The **Errors** table provides a high-level view of each error message when it fi :screenshot: ::: - ## Span types average duration and dependencies [service-span-duration] The **Time spent by span type** chart visualizes each span type’s average duration and helps you determine which spans could be slowing down transactions. The "app" label displayed under the chart indicates that something was happening within the application. This could signal that the APM agent does not have auto-instrumentation for whatever was happening during that time or that the time was spent in the application code and not in database or external requests. -The **Dependencies** table displays a list of downstream services or external connections relevant to the service at the selected time range. The table displays latency, throughput, failed transaction rate, and the impact of each dependency. By default, dependencies are sorted by *Impact* to show the most used and the slowest dependency. If there is a particular dependency you are interested in, click **[View dependencies](../../../solutions/observability/apps/dependencies.md)** to learn more about it. +The **Dependencies** table displays a list of downstream services or external connections relevant to the service at the selected time range. The table displays latency, throughput, failed transaction rate, and the impact of each dependency. By default, dependencies are sorted by *Impact* to show the most used and the slowest dependency. If there is a particular dependency you are interested in, click **[View dependencies](/solutions/observability/apps/dependencies.md)** to learn more about it. % Stateful only for following note? @@ -91,30 +86,25 @@ The **Dependencies** table displays a list of downstream services or external co **In Elastic Stack deployments**, displaying dependencies for services instrumented with the Real User Monitoring (RUM) agent requires an agent version ≥ v5.6.3. :::: - ## Cold start rate [service-cold-start] The cold start rate chart is specific to serverless services, and displays the percentage of requests that trigger a cold start of a serverless function. A cold start occurs when a serverless function has not been used for a certain period of time. Analyzing the cold start rate can be useful for deciding how much memory to allocate to a function, or when to remove a large dependency. -The cold start rate chart is currently supported for [AWS Lambda](../../../solutions/observability/apps/observe-lambda-functions.md#apm-lambda-cold-start-info) functions and Azure functions. - +The cold start rate chart is currently supported for [AWS Lambda](/solutions/observability/apps/observe-lambda-functions.md#apm-lambda-cold-start-info) functions and Azure functions. ## Instances [service-instances] The **Instances** table displays a list of all the available service instances within the selected time range. Depending on how the service runs, the instance could be a host or a container. The table displays latency, throughput, failed transaction, CPU usage, and memory usage for each instance. By default, instances are sorted by *Throughput*. - :::{image} /solutions/images/observability-all-instances.png :alt: All instances :screenshot: ::: - ## Service metadata [service-metadata] To view metadata relating to the service agent, and if relevant, the container and cloud provider, click on each icon located at the top of the page beside the service name. - :::{image} /solutions/images/observability-metadata-icons.png :alt: Service metadata :screenshot: diff --git a/solutions/observability/apps/services.md b/solutions/observability/apps/services.md index 97d00b7476..cd704add20 100644 --- a/solutions/observability/apps/services.md +++ b/solutions/observability/apps/services.md @@ -11,9 +11,9 @@ applies_to: The **Services** inventory provides a quick, high-level overview of the health and general performance of all instrumented services. -To help surface potential issues, services are sorted by their health status: **critical** → **warning** → **healthy** → **unknown**. Health status is powered by [machine learning](../../../solutions/observability/apps/integrate-with-machine-learning.md) and requires anomaly detection to be enabled. +To help surface potential issues, services are sorted by their health status: **critical** → **warning** → **healthy** → **unknown**. Health status is powered by [machine learning](/solutions/observability/apps/integrate-with-machine-learning.md) and requires anomaly detection to be enabled. -In addition to health status, active alerts for each service are prominently displayed in the service inventory table. Selecting an active alert badge brings you to the [**Alerts**](../../../solutions/observability/apps/create-apm-rules-alerts.md) tab where you can learn more about the active alert and take action. +In addition to health status, active alerts for each service are prominently displayed in the service inventory table. Selecting an active alert badge brings you to the [**Alerts**](/solutions/observability/apps/create-apm-rules-alerts.md) tab where you can learn more about the active alert and take action. :::{image} /solutions/images/observability-apm-services-overview.png :alt: Example view of services table the Applications UI in Kibana @@ -24,18 +24,16 @@ In addition to health status, active alerts for each service are prominently dis ::::{note} -For Observability Serverless projects, the **Editor** role or higher is required to create and manage service groups. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). +For Observability Serverless projects, the **Editor** role or higher is required to create and manage service groups. To learn more, refer to [Assign user roles and privileges](/deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). :::: - ::::{important} The Service grouping functionality is in beta and is subject to change. The design and code is less mature than official generally available features and is being provided as-is with no warranties. :::: - Group services together to build meaningful views that remove noise, simplify investigations across services, and combine related alerts. :::{image} /solutions/images/observability-apm-service-group.png @@ -50,7 +48,7 @@ To create a service group: 3. Click **Create group**. 4. Specify a name, color, and description. 5. Click **Select services**. -6. Specify a [{{kib}} Query Language (KQL)](../../../explore-analyze/query-filter/languages/kql.md) query to filter services by one or more of the following dimensions: `agent.name`, `service.name`, `service.language.name`, `service.environment`, `labels.`. Services that match the query within the last 24 hours will be assigned to the group. +6. Specify a [{{kib}} Query Language (KQL)](/explore-analyze/query-filter/languages/kql.md) query to filter services by one or more of the following dimensions: `agent.name`, `service.name`, `service.language.name`, `service.environment`, `labels.`. Services that match the query within the last 24 hours will be assigned to the group. ### Examples [apm-services-examples] diff --git a/solutions/observability/apps/setup-role.md b/solutions/observability/apps/setup-role.md index 8bd84e4764..2344fca355 100644 --- a/solutions/observability/apps/setup-role.md +++ b/solutions/observability/apps/setup-role.md @@ -15,10 +15,10 @@ As a best practice, **grant the setup role to administrators only**, and use a m Create a **setup role**, called something like `synthetics_setup`: -1. Start with the `editor` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). This role grants full access to all features in {{kib}} (including the {{observability}} solution) and read-only access to data indices. +1. Start with the `editor` [built-in role](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). This role grants full access to all features in {{kib}} (including the {{observability}} solution) and read-only access to data indices. ::::{note} - The `editor` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) will grant write access to *all* {{kib}} apps. If you want to limit write access to the {{synthetics-app}} only, refer to [Limited write access](writer-role.md#synthetics-write-privileges-limited). + The `editor` [built-in role](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) will grant write access to *all* {{kib}} apps. If you want to limit write access to the {{synthetics-app}} only, refer to [Limited write access](writer-role.md#synthetics-write-privileges-limited). If you choose this approach, you will still need to grant the privileges in the next step. @@ -30,17 +30,15 @@ Create a **setup role**, called something like `synthetics_setup`: | Type | Privilege | Purpose | | --- | --- | --- | - | [Cluster](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster) | `monitor` | Allows the user to retrieve cluster details. | - | [Cluster](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster) | `manage_ilm` | Allows the user access to all index lifecycle management operations related to managing policies. | - | [Cluster](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster) | `read_pipeline` | Gives the user read-only access to the ingest pipline. | - | [Index](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) | `synthetics-*`: `view_index_metadata` | Gives the user read-only access to index and data stream metadata. | - | [Index](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) | `synthetics-*`: `create_doc` | Allows the user to index documents. | - | [Index](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) | `synthetics-*`: `auto_configure` | Permits auto-creation of indices and data streams. | - | [Index](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) | `synthetics-*`: `monitor` | Gives access to all actions that are required for monitoring (recovery, segments info, index stats, and status). | - + | [Cluster](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster) | `monitor` | Allows the user to retrieve cluster details. | + | [Cluster](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster) | `manage_ilm` | Allows the user access to all index lifecycle management operations related to managing policies. | + | [Cluster](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster) | `read_pipeline` | Gives the user read-only access to the ingest pipline. | + | [Index](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) | `synthetics-*`: `view_index_metadata` | Gives the user read-only access to index and data stream metadata. | + | [Index](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) | `synthetics-*`: `create_doc` | Allows the user to index documents. | + | [Index](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) | `synthetics-*`: `auto_configure` | Permits auto-creation of indices and data streams. | + | [Index](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) | `synthetics-*`: `monitor` | Gives access to all actions that are required for monitoring (recovery, segments info, index stats, and status). | ::::{note} If users with the setup role also need to create, modify, and delete monitors, add the privileges defined in the [writer role](writer-role.md). :::: - diff --git a/solutions/observability/apps/spans.md b/solutions/observability/apps/spans.md index aa4f5640ee..34cba0a405 100644 --- a/solutions/observability/apps/spans.md +++ b/solutions/observability/apps/spans.md @@ -9,6 +9,9 @@ applies_to: # Spans [apm-data-model-spans] +:::{include} _snippets/apm-server-vs-mis.md +::: + % Stateful only until Span Compressions **Spans** contain information about the execution of a specific code path. They measure from the start to the end of an activity, and they can have a parent/child relationship with other spans. @@ -17,7 +20,7 @@ Agents automatically instrument a variety of libraries to capture these spans fr Among other things, spans can contain: -* A `transaction.id` attribute that refers to its parent [transaction](../../../solutions/observability/apps/transactions.md). +* A `transaction.id` attribute that refers to its parent [transaction](/solutions/observability/apps/transactions.md). * A `parent.id` attribute that refers to its parent span or transaction. * Its start time and duration. * A `name`, `type`, `subtype`, and `action`—see the [span name/type alignment](https://docs.google.com/spreadsheets/d/1SmWeX5AeqUcayrArUauS_CxGgsjwRgMYH4ZY8yQsMhQ/edit#gid=644582948) sheet for span name patterns and examples by {{apm-agent}}. In addition, some APM agents test against a public [span type/subtype spec](https://github.com/elastic/apm/blob/main/tests/agents/json-specs/span_types.json). @@ -27,11 +30,9 @@ Among other things, spans can contain: Most agents limit keyword fields, like `span.id`, to 1024 characters, and non-keyword fields, like `span.start.us`, to 10,000 characters. :::: - - ## Dropped spans [apm-data-model-dropped-spans] -For performance reasons, APM agents can choose to sample or omit spans purposefully. This can be useful in preventing edge cases, like long-running transactions with over 100 spans, that would otherwise overload both the Agent and the APM Server. When this occurs, the Applications UI will display the number of spans dropped. +For performance reasons, APM agents can choose to sample or omit spans purposefully. This can be useful in preventing edge cases, like long-running transactions with over 100 spans, that would otherwise overload both the Agent and the {{apm-server-or-mis}}. When this occurs, the Applications UI will display the number of spans dropped. To configure the number of spans recorded per transaction, see the relevant Agent documentation: @@ -45,11 +46,9 @@ To configure the number of spans recorded per transaction, see the relevant Agen * Python: [`transaction_max_spans`](apm-agent-python://reference/configuration.md#config-transaction-max-spans) * Ruby: [`transaction_max_spans`](apm-agent-ruby://reference/configuration.md#config-transaction-max-spans) - ## Missing spans [apm-data-model-missing-spans] -Agents stream spans to the APM Server separately from their transactions. Because of this, unforeseen errors may cause spans to go missing. Agents know how many spans a transaction should have; if the number of expected spans does not equal the number of spans received by the APM Server, the Applications UI will calculate the difference and display a message. - +Agents stream spans to the {{apm-server-or-mis}} separately from their transactions. Because of this, unforeseen errors may cause spans to go missing. Agents know how many spans a transaction should have; if the number of expected spans does not equal the number of spans received by the {{apm-server-or-mis}}, the Applications UI will calculate the difference and display a message. ## Data streams [_data_streams] @@ -58,8 +57,7 @@ Spans are stored with transactions in the following data streams: * Application traces: `traces-apm-` * RUM and iOS agent application traces: `traces-apm.rum-` -See [Data streams](../../../solutions/observability/apps/data-streams.md) to learn more. - +See [Data streams](/solutions/observability/apps/data-streams.md) to learn more. ## Example span document [_example_span_document] @@ -424,7 +422,6 @@ This example shows what span documents can look like when indexed in {{es}}. :::: - ## Span compression [apm-spans-span-compression] In some cases, APM agents may collect large amounts of very similar or identical spans in a transaction. For example, this can happen if spans are captured inside a loop or in unoptimized SQL queries that use multiple queries instead of joins to fetch related data. @@ -441,12 +438,10 @@ Regardless of the compression strategy, a span is eligible for compression if: * It is an *exit* span (such as database query spans). * Its outcome is not `"failure"`. - ### Compression strategies [apm-span-compression-strategy] The {{apm-agent}} selects between two strategies to decide if adjacent spans can be compressed. In both strategies, only one previous span needs to be kept in memory. This ensures that the agent doesn’t require large amounts of memory to enable span compression. - #### Same-Kind strategy [apm-span-compression-same] The agent uses the same-kind strategy if two adjacent spans have the same: @@ -464,14 +459,12 @@ The agent uses the exact-match strategy if two adjacent spans have the same: * span subtype * `destination.service.resource` (e.g. database name) - ### Settings [apm-span-compression-settings] You can specify the maximum span duration in the agent’s configuration settings. Spans with a duration longer than the specified value will not be compressed. For the "Same-Kind" strategy, the default maximum span duration is 0 milliseconds, which means that the "Same-Kind" strategy is disabled by default. For the "Exact-Match" strategy, the default limit is 50 milliseconds. - ### Agent support [apm-span-compression-support] Support for span compression is available in the following agents and can be configured using the options listed below: diff --git a/solutions/observability/apps/ssltls-input-settings.md b/solutions/observability/apps/ssltls-input-settings.md index 1bf160abdd..ea734d4efd 100644 --- a/solutions/observability/apps/ssltls-input-settings.md +++ b/solutions/observability/apps/ssltls-input-settings.md @@ -14,7 +14,6 @@ Most options on this page are supported by all APM Server deployment methods. :::: - These settings apply to SSL/TLS communication between the APM Server and APM Agents. See [{{apm-agent}} TLS communication](apm-agent-tls-communication.md) to learn more. :::::::{tab-set} @@ -48,7 +47,6 @@ Enable or disable TLS. Disabled by default. | APM Server binary | `apm-server.ssl.enabled` | | Fleet-managed | `Enable TLS` | - ## File path to server certificate [_file_path_to_server_certificate] The path to the file containing the certificate for Server authentication. Required if TLS is enabled. @@ -58,7 +56,6 @@ The path to the file containing the certificate for Server authentication. Requi | APM Server binary | `apm-server.ssl.certificate` | | Fleet-managed | `File path to server certificate` | - ## File path to server certificate key [_file_path_to_server_certificate_key] The path to the file containing the Server certificate key. Required if TLS is enabled. @@ -68,7 +65,6 @@ The path to the file containing the Server certificate key. Required if TLS is e | APM Server binary | `apm-server.ssl.key` | | Fleet-managed | `File path to server certificate key` | - ## Key passphrase [_key_passphrase] The passphrase used to decrypt an encrypted key stored in the configured `apm-server.ssl.key` file. @@ -78,7 +74,6 @@ The passphrase used to decrypt an encrypted key stored in the configured `apm-se | APM Server binary | `apm-server.ssl.key_passphrase` | | Fleet-managed | N/A | - ## Supported protocol versions [_supported_protocol_versions] This setting is a list of allowed protocol versions: `SSLv3`, `TLSv1.0`, `TLSv1.1`, `TLSv1.2` and `TLSv1.3`. We do not recommend using `SSLv3` or `TLSv1.0`. The default value is `[TLSv1.1, TLSv1.2, TLSv1.3]`. @@ -88,7 +83,6 @@ This setting is a list of allowed protocol versions: `SSLv3`, `TLSv1.0`, `TLSv1. | APM Server binary | `apm-server.ssl.supported_protocols` | | Fleet-managed | `Supported protocol versions` | - ## Cipher suites for TLS connections [_cipher_suites_for_tls_connections] The list of cipher suites to use. The first entry has the highest priority. If this option is omitted, the Go crypto library’s [default suites](https://golang.org/pkg/crypto/tls/) are used (recommended). Note that TLS 1.3 cipher suites are not individually configurable in Go, so they are not included in this list. @@ -137,7 +131,6 @@ Here is a list of acronyms used in defining the cipher suites: * RSA: Cipher suites using RSA. * SHA, SHA256, SHA384: Cipher suites using SHA-1, SHA-256 or SHA-384. - ## Curve types for ECDHE based cipher suites [_curve_types_for_ecdhe_based_cipher_suites] The list of curve types for ECDHE (Elliptic Curve Diffie-Hellman ephemeral key exchange). @@ -147,7 +140,6 @@ The list of curve types for ECDHE (Elliptic Curve Diffie-Hellman ephemeral key e | APM Server binary | `apm-server.ssl.curve_types` | | Fleet-managed | `Curve types for ECDHE based cipher suites` | - ## List of root certificates for verifying client certificates [_list_of_root_certificates_for_verifying_client_certificates] The list of root certificates for verifying client certificates. If `certificate_authorities` is empty or not set, the trusted certificate authorities of the host system are used. If `certificate_authorities` is set, `client_authentication` will be automatically set to `required`. Sending client certificates is currently only supported by the RUM agent through the browser, the Java agent (see [Agent certificate authentication](apm-agent-java://reference/ssl-configuration.md)), and the Jaeger agent. @@ -157,7 +149,6 @@ The list of root certificates for verifying client certificates. If `certificate | APM Server binary | `apm-server.ssl.certificate_authorities` | | Fleet-managed | N/A | - ## Client authentication [_client_authentication] This configures what types of client authentication are supported. The valid options are `none`, `optional`, and `required`. The default is `none`. If `certificate_authorities` has been specified, this setting will automatically change to `required`. This option only needs to be configured when the agent is expected to provide a client certificate. Sending client certificates is currently only supported by the RUM agent through the browser, the Java agent (see [Agent certificate authentication](apm-agent-java://reference/ssl-configuration.md)), and the Jaeger agent. diff --git a/solutions/observability/apps/ssltls-output-settings.md b/solutions/observability/apps/ssltls-output-settings.md index c17a098e53..eb08880893 100644 --- a/solutions/observability/apps/ssltls-output-settings.md +++ b/solutions/observability/apps/ssltls-output-settings.md @@ -14,7 +14,6 @@ These configuration options are only relevant to APM Server binary users. Fleet- :::: - You can specify SSL/TLS options with any output that supports SSL, like {{es}}, {{ls}}, or Kafka. Example output config with SSL/TLS enabled: ```yaml @@ -30,12 +29,10 @@ There are a number of SSL/TLS configuration options available to you: * [Client configuration options](#apm-ssl-client-config) * [Server configuration options](#apm-ssl-server-config) - ## Common configuration options [apm-ssl-common-config] Common SSL configuration options can be used in both client and server configurations. You can specify the following options in the `ssl` section of each subsystem that supports SSL. - ### `enabled` [apm-enabled] To disable SSL configuration, set the value to `false`. The default value is `true`. @@ -45,15 +42,12 @@ SSL settings are disabled if either `enabled` is set to `false` or the `ssl` sec :::: - - ### `supported_protocols` [apm-supported-protocols] List of allowed SSL/TLS versions. If SSL/TLS server decides for protocol versions not configured, the connection will be dropped during or after the handshake. The setting is a list of allowed protocol versions: `SSLv3`, `TLSv1` for TLS version 1.0, `TLSv1.0`, `TLSv1.1`, `TLSv1.2`, and `TLSv1.3`. The default value is `[TLSv1.1, TLSv1.2, TLSv1.3]`. - ### `cipher_suites` [apm-cipher-suites] The list of cipher suites to use. The first entry has the highest priority. If this option is omitted, the Go crypto library’s [default suites](https://golang.org/pkg/crypto/tls/) are used (recommended). Note that TLS 1.3 cipher suites are not individually configurable in Go, so they are not included in this list. @@ -97,7 +91,6 @@ Here is a list of acronyms used in defining the cipher suites: * RSA: Cipher suites using RSA. * SHA, SHA256, SHA384: Cipher suites using SHA-1, SHA-256 or SHA-384. - ### `curve_types` [apm-curve-types] The list of curve types for ECDHE (Elliptic Curve Diffie-Hellman ephemeral key exchange). @@ -109,7 +102,6 @@ The following elliptic curve types are available: * P-521 * X25519 - ### `ca_sha256` [apm-ca-sha256] This configures a certificate pin that you can use to ensure that a specific certificate is part of the verified chain. @@ -120,13 +112,10 @@ The pin is a base64 encoded string of the SHA-256 of the certificate. This check is not a replacement for the normal SSL validation, but it adds additional validation. If this option is used with `verification_mode` set to `none`, the check will always fail because it will not receive any verified chains. :::: - - ## Client configuration options [apm-ssl-client-config] You can specify the following options in the `ssl` section of each subsystem that supports SSL. - ### `certificate_authorities` [apm-client-certificate-authorities] The list of root certificates for verifications is required. If `certificate_authorities` is self-signed, the host system needs to trust that CA cert as well. @@ -157,7 +146,6 @@ certificate_authorities: -----END CERTIFICATE----- ``` - ### `certificate: "/etc/pki/client/cert.pem"` [apm-client-certificate] The path to the certificate for SSL client authentication is only required if `client_authentication` is specified. If the certificate is not specified, client authentication is not available. The connection might fail if the server requests client authentication. If the SSL server does not require client authentication, the certificate will be loaded, but not requested or used by the server. @@ -187,7 +175,6 @@ certificate: | -----END CERTIFICATE----- ``` - ### `key: "/etc/pki/client/cert.key"` [apm-client-key] The client certificate key used for client authentication and is only required if `client_authentication` is configured. The key option support embedding of the private key: @@ -224,12 +211,10 @@ key: | -----END PRIVATE KEY----- ``` - ### `key_passphrase` [apm-client-key-passphrase] The passphrase used to decrypt an encrypted key stored in the configured `key` file. - ### `verification_mode` [apm-client-verification-mode] Controls the verification of server certificates. Valid values are: @@ -248,13 +233,10 @@ Controls the verification of server certificates. Valid values are: The default value is `full`. - - ## Server configuration options [apm-ssl-server-config] You can specify the following options in the `ssl` section of each subsystem that supports SSL. - ### `certificate_authorities` [apm-server-certificate-authorities] The list of root certificates for client verifications is only required if `client_authentication` is configured. If `certificate_authorities` is empty or not set, and `client_authentication` is configured, the system keystore is used. @@ -285,7 +267,6 @@ certificate_authorities: -----END CERTIFICATE----- ``` - ### `certificate: "/etc/pki/server/cert.pem"` [apm-server-certificate] For server authentication, the path to the SSL authentication certificate must be specified for TLS. If the certificate is not specified, startup will fail. @@ -315,7 +296,6 @@ certificate: | -----END CERTIFICATE----- ``` - ### `key: "/etc/pki/server/cert.key"` [apm-server-key] The server certificate key used for authentication is required. The key option support embedding of the private key: @@ -352,12 +332,10 @@ key: | -----END PRIVATE KEY----- ``` - ### `key_passphrase` [apm-server-key-passphrase] The passphrase is used to decrypt an encrypted key stored in the configured `key` file. - ### `verification_mode` [apm-server-verification-mode] Controls the verification of client certificates. Valid values are: @@ -376,8 +354,6 @@ Controls the verification of client certificates. Valid values are: The default value is `full`. - - ### `renegotiation` [apm-server-renegotiation] This configures what types of TLS renegotiation are supported. The valid options are: @@ -393,4 +369,3 @@ This configures what types of TLS renegotiation are supported. The valid options The default value is `never`. - diff --git a/solutions/observability/apps/storage-explorer.md b/solutions/observability/apps/storage-explorer.md index e699b43f2c..f50127f53d 100644 --- a/solutions/observability/apps/storage-explorer.md +++ b/solutions/observability/apps/storage-explorer.md @@ -14,7 +14,6 @@ Analyze your APM data and manage costs with **storage explorer**. For example, a :screenshot: ::: - ## Index lifecycle phases [_index_lifecycle_phases] A default [index lifecycle policy](/solutions/observability/apps/index-lifecycle-management.md) is applied to each APM data stream, but can be customized depending on your business needs. Use the **Index lifecycle phase** dropdown to visualize and analyze your storage by phase. @@ -27,12 +26,10 @@ Customizing the default APM index lifecycle policies can save money by specifyin See [Index lifecycle management](/solutions/observability/apps/index-lifecycle-management.md) to learn more about customizing the default APM index lifecycle policies. - ## Service size chart [_service_size_chart] The service size chart displays the estimated size of each service over time. Expand the time filter to visualize data trends and estimate daily data generation. - ## Service statistics table [_service_statistics_table] The service statistics table provides detailed information on each service: @@ -49,32 +46,26 @@ The service statistics table provides detailed information on each service: As you explore your service statistics, you might want to take action to reduce the number of documents and therefore storage size of a particular service. - ### Reduce the number of transactions [_reduce_the_number_of_transactions] To reduce the number of transactions a service generates, configure a more aggressive [transaction sampling policy](/solutions/observability/apps/transaction-sampling.md). Transaction sampling lowers the amount of data ingested without negatively impacting the usefulness of your data. - ### Reduce the number of spans [_reduce_the_number_of_spans] To reduce the number of spans a service generates, enable [span compression](/solutions/observability/apps/spans.md#apm-spans-span-compression). Span compression saves on data and transfer costs by compressing multiple, similar spans into a single span. - ### Reduce the number of metrics [_reduce_the_number_of_metrics] To reduce the number of system, runtime, and application metrics, tune the APM agent or agents that are collecting the data. You can disable the collection of specific metrics with the **disable metrics** configuration. Or, you can set the **metrics interval** to zero seconds to deactivate metrics entirely. Most APM agents support both options. See the relevant [APM agent configuration options](https://www.elastic.co/guide/en/apm/agent) for more details. - ### Reduce the number of errors [_reduce_the_number_of_errors] To reduce the number of errors a service generate, work with your developers to change how exceptions are handled in your code. - ## Privileges [_privileges] Storage Explorer requires expanded privileges to view. See [Create a storage explorer user](applications-ui-storage-explorer-user.md) for more information. - ## Limitations [_limitations_4] Multi-cluster deployments are not supported. diff --git a/solutions/observability/apps/storage-sizing-guide.md b/solutions/observability/apps/storage-sizing-guide.md index 548ec85df3..19875d1c50 100644 --- a/solutions/observability/apps/storage-sizing-guide.md +++ b/solutions/observability/apps/storage-sizing-guide.md @@ -13,7 +13,6 @@ APM processing and storage costs are largely dominated by transactions, spans, a * [**Spans**](spans.md) belong to transactions. They measure from the start to end of an activity, and contain information about a specific code path that has been executed. * **Stack frames** belong to spans. Stack frames represent a function call on the call stack, and include attributes like function name, file name and path, line number, etc. Stack frames can heavily influence the size of a span. - ## Typical transactions [_typical_transactions] Due to the high variability of APM data, it’s difficult to classify a transaction as typical. Regardless, this guide will attempt to classify Transactions as *Small*, *Medium*, or *Large*, and make recommendations based on those classifications. @@ -34,7 +33,6 @@ Here’s a speculative reference: There will always be transaction outliers with hundreds of spans or stack frames, but those are very rare. Small transactions are the most common. - ## Typical storage [_typical_storage] Consider the following typical storage reference. These numbers do not account for {{es}} compression. diff --git a/solutions/observability/apps/switch-an-elastic-cloud-cluster-to-apm-integration.md b/solutions/observability/apps/switch-an-elastic-cloud-cluster-to-apm-integration.md index b96beba06a..8e9208805f 100644 --- a/solutions/observability/apps/switch-an-elastic-cloud-cluster-to-apm-integration.md +++ b/solutions/observability/apps/switch-an-elastic-cloud-cluster-to-apm-integration.md @@ -6,21 +6,16 @@ applies_to: stack: --- - - # Switch an Elastic Cloud cluster to the APM integration [apm-integration-upgrade-steps-ess] - 1. [Upgrade the {{stack}}](#apm-integration-upgrade-ess-1) 2. [Switch to {{agent}}](#apm-integration-upgrade-ess-2) 3. [Configure the APM integration](#apm-integration-upgrade-ess-3) 4. [Scale APM and {{fleet}}](#apm-integration-upgrade-ess-4) - ## Upgrade the {{stack}} [apm-integration-upgrade-ess-1] -Use the {{ecloud}} Console to upgrade the {{stack}} to version 9.0.0-beta1. See the [Upgrade guide](../../../deploy-manage/upgrade/deployment-or-cluster.md) for details. - +Use the {{ecloud}} Console to upgrade the {{stack}} to version 9.0.0-beta1. See the [Upgrade guide](/deploy-manage/upgrade/deployment-or-cluster.md) for details. ## Switch to {{agent}} [apm-integration-upgrade-ess-2] @@ -40,17 +35,14 @@ With a Superuser account, complete the following steps: :alt: {{agent}} settings migration ::: - {{ecloud}} will now create a {{fleet}} Server instance to contain the new APM integration, and then will shut down the old APM server instance. Within minutes your data should begin appearing in the Applications UI again. - ## Configure the APM integration [apm-integration-upgrade-ess-3] You can now update settings that were removed during the upgrade. See [Configure APM Server](configure-apm-server.md) for a reference of all available settings. In {{kib}}, navigate to **Management** > **Fleet**. Select the **Elastic Cloud Agent Policy**. Next to the **Elastic APM** integration, select **Actions** > **Edit integration**. - ## Scale APM and {{fleet}} [apm-integration-upgrade-ess-4] Certain {{es}} output configuration options are not available with the APM integration. To ensure data is not lost, you can scale APM and {{fleet}} up and out. APM’s capacity to process events increases with the instance memory size. diff --git a/solutions/observability/apps/switch-self-installation-to-apm-integration.md b/solutions/observability/apps/switch-self-installation-to-apm-integration.md index 45402caa25..a31e23bad5 100644 --- a/solutions/observability/apps/switch-self-installation-to-apm-integration.md +++ b/solutions/observability/apps/switch-self-installation-to-apm-integration.md @@ -6,49 +6,40 @@ applies_to: stack: --- - - # Switch a self-installation to the APM integration [apm-integration-upgrade-steps] - 1. [Upgrade the {{stack}}](#apm-integration-upgrade-1) 2. [Add a {{fleet}} Server](#apm-integration-upgrade-2) 3. [Install a {{fleet}}-managed {{agent}}](#apm-integration-upgrade-3) 4. [Add the APM integration](#apm-integration-upgrade-4) 5. [Stop the APM Server](#apm-integration-upgrade-5) - ## Upgrade the {{stack}} [apm-integration-upgrade-1] -The {{stack}} ({{es}} and {{kib}}) must be upgraded to version 7.14 or higher. See the [{{stack}} Installation and Upgrade Guide](../../../deploy-manage/upgrade/deployment-or-cluster.md) for guidance. +The {{stack}} ({{es}} and {{kib}}) must be upgraded to version 7.14 or higher. See the [{{stack}} Installation and Upgrade Guide](/deploy-manage/upgrade/deployment-or-cluster.md) for guidance. Review the [Elastic APM release notes](apm-server://release-notes/index.md) and [Elastic {{observability}} release notes](/release-notes/elastic-observability/index.md) for important changes between your current APM version and this one. - ## Add a {{fleet}} Server [apm-integration-upgrade-2] {{fleet}} Server is a component of the {{stack}} used to centrally manage {{agent}}s. The APM integration requires a {{fleet}} Server to be running and accessible to your hosts. Add a {{fleet}} Server by following [this guide](/reference/fleet/deployment-models.md). ::::{tip} -If you’re upgrading a self-managed deployment of the {{stack}}, you’ll need to enable [{{es}} security](../../../deploy-manage/deploy/self-managed/installing-elasticsearch.md) and the [API key service](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md). +If you’re upgrading a self-managed deployment of the {{stack}}, you’ll need to enable [{{es}} security](/deploy-manage/deploy/self-managed/installing-elasticsearch.md) and the [API key service](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md). :::: - After adding your {{fleet}} Server host and generating a service token, the in-product help in {{kib}} will provide a command to run to start an {{agent}} as a {{fleet}} Server. Commands may require administrator privileges. Verify {{fleet}} Server is running by navigating to **{{fleet}}** > **Agents** in {{kib}}. - ## Install a {{fleet}}-managed {{agent}} [apm-integration-upgrade-3] ::::{note} It’s possible to install the Elastic APM integration on the same {{agent}} that is running the {{fleet}} Server integration. For this use case, skip this step. :::: - The {{fleet}}-managed {{agent}} will run the Elastic APM integration on your edge nodes, next to your applications. To install a {{fleet}}-managed {{agent}}, follow [this guide](/reference/fleet/install-fleet-managed-elastic-agent.md). - ## Add the APM integration [apm-integration-upgrade-4] The APM integration receives performance data from your APM agents, validates and processes it, and then transforms the data into {{es}} documents. @@ -59,8 +50,6 @@ To add the APM integration, see [Step 2: Add and configure the APM integration]( You’ll configure the APM integration in this step. See [Configure APM Server](configure-apm-server.md) for a reference of all available settings. As long as the APM integration is configured with the same secret token or you have API keys enabled on the same host, no reconfiguration is required in your APM agents. :::: - - ## Stop the APM Server [apm-integration-upgrade-5] Once data from upgraded APM agents is visible in the Applications UI, it’s safe to stop the APM Server process. diff --git a/solutions/observability/apps/switch-to-elastic-apm-integration.md b/solutions/observability/apps/switch-to-elastic-apm-integration.md index b261d3fb9d..f732d51735 100644 --- a/solutions/observability/apps/switch-to-elastic-apm-integration.md +++ b/solutions/observability/apps/switch-to-elastic-apm-integration.md @@ -30,7 +30,6 @@ The APM integration offers a number of benefits over the standalone method of ru * Easier and less error-prone upgrade path * Zero-downtime configuration changes - ## APM integration architecture [apm-arch-upgrade] Elastic APM consists of four components: **APM agents**, the **Elastic APM integration**, **{{es}}**, and **{{kib}}**. Generally, there are two ways that these four components can work together: @@ -47,7 +46,6 @@ Or, APM agents and the APM integration live on edge machines and enroll via a ce In order to collect data from RUM and mobile agents, which run in browser and mobile applications, you must run {{agent}} centrally. For other applications, such as backend services, {{agent}} may be co-located on the edge machine. :::: - ## Limitations [apm-integration-upgrade-limitations] There are some limitations to be aware of: @@ -57,7 +55,6 @@ There are some limitations to be aware of: * APM runs under {{agent}} which, depending on the installation method, might require root privileges * An {{agent}} with the APM integration enabled must be managed by {{fleet}}. - ## Make the switch [_make_the_switch] Select a guide below to get started. diff --git a/solutions/observability/apps/synthetic-monitoring.md b/solutions/observability/apps/synthetic-monitoring.md index b15673da80..38f734f4d2 100644 --- a/solutions/observability/apps/synthetic-monitoring.md +++ b/solutions/observability/apps/synthetic-monitoring.md @@ -10,22 +10,20 @@ applies_to: # Synthetic monitoring [monitor-uptime-synthetics] ::::{note} -The Synthetics UI is for viewing result data from monitors created and managed directly in the [Synthetics UI](../../../solutions/observability/apps/create-monitors-in-synthetics-app.md) or managed externally using a [Synthetics project](../../../solutions/observability/apps/create-monitors-with-project-monitors.md). This can include both lightweight and browser-based monitors, and can include monitors running from either Elastic’s global managed testing infrastructure or from [{{private-location}}s](../../../solutions/observability/apps/monitor-resources-on-private-networks.md). +The Synthetics UI is for viewing result data from monitors created and managed directly in the [Synthetics UI](/solutions/observability/apps/create-monitors-in-synthetics-app.md) or managed externally using a [Synthetics project](/solutions/observability/apps/create-monitors-with-project-monitors.md). This can include both lightweight and browser-based monitors, and can include monitors running from either Elastic’s global managed testing infrastructure or from [{{private-location}}s](/solutions/observability/apps/monitor-resources-on-private-networks.md). :::: - Synthetics periodically checks the status of your services and applications. Monitor the availability of network endpoints and services using the following types of monitors: -* [Lightweight HTTP/S, TCP, and ICMP monitors](../../../solutions/observability/apps/synthetic-monitoring.md#monitoring-uptime) -* [Browser monitors](../../../solutions/observability/apps/synthetic-monitoring.md#monitoring-synthetics) +* [Lightweight HTTP/S, TCP, and ICMP monitors](/solutions/observability/apps/synthetic-monitoring.md#monitoring-uptime) +* [Browser monitors](/solutions/observability/apps/synthetic-monitoring.md#monitoring-synthetics) :::{image} /solutions/images/observability-synthetics-monitor-page.png :alt: {{synthetics-app}} in {{kib}} :screenshot: ::: - ## Lightweight HTTP/S, TCP, and ICMP monitors [monitoring-uptime] You can monitor the status of network endpoints using the following lightweight checks: @@ -36,8 +34,7 @@ You can monitor the status of network endpoints using the following lightweight | **ICMP monitor** | Check the availability of your hosts. The ICMP monitor uses ICMP (v4 and v6) EchoRequests to check the network reachability of the hosts you are pinging. This will tell you whether thehost is available and connected to the network, but doesn’t tell you if a service on the host is running ornot. | | **TCP monitor** | Monitor the services running on your hosts. The TCP monitor checks individual portsto make sure the service is accessible and running. | -To set up your first monitor, refer to [Get started](../../../solutions/observability/apps/get-started.md). - +To set up your first monitor, refer to [Get started](/solutions/observability/apps/get-started.md). ## Browser monitors [monitoring-synthetics] @@ -49,4 +46,4 @@ You can run an automated Synthetics project on a real Chromium browser and view Alerting helps you detect degraded performance or broken actions before your users do. By receiving alerts early, you can fix issues before they impact your bottom line or customer experience. -To set up your first monitor, refer to [Get started](../../../solutions/observability/apps/get-started.md). \ No newline at end of file +To set up your first monitor, refer to [Get started](/solutions/observability/apps/get-started.md). \ No newline at end of file diff --git a/solutions/observability/apps/synthetics-encryption-security.md b/solutions/observability/apps/synthetics-encryption-security.md index fb55d279a5..6b6b3792a1 100644 --- a/solutions/observability/apps/synthetics-encryption-security.md +++ b/solutions/observability/apps/synthetics-encryption-security.md @@ -11,19 +11,16 @@ applies_to: Elastic Synthetics was designed with security in mind encrypting both persisted and transmitted data. This page catalogs the points within Elastic Synthetics where data is either stored or transmitted in an encrypted fashion. - ## Synthetics UI (Kibana App) [_synthetics_ui_kibana_app] -Data is stored in [Kibana Secure Saved Objects](../../../deploy-manage/security/secure-saved-objects.md), with sensitive fields encrypted. These fields include your script source, params, and global params. - +Data is stored in [Kibana Secure Saved Objects](/deploy-manage/security/secure-saved-objects.md), with sensitive fields encrypted. These fields include your script source, params, and global params. ## Synthetics Service [_synthetics_service] The Global Elastic Synthetics Service performs all communication of sensitive data (both internally, and with Kibana) over encrypted connections and encrypts all data persisted to disk as well. - ## Synthetics Private Locations [_synthetics_private_locations] -In Kibana configuration for private locations is stored in two places, Synthetics saved objects which always encrypt sensitive fields using [Kibana Secure Saved Objects](../../../deploy-manage/security/secure-saved-objects.md) and also in Fleet, which uses unencrypted saved objects restricted by user permissions. For Elastic Cloud customers all data is secured on disk regardless of whether additional saved object encryption is present. See our [Cloud Security Statement](https://www.elastic.co/cloud/security) for more information. We recommend that self-managed customers encrypt disks for their Elasticsearch instances if this is a concern. +In Kibana configuration for private locations is stored in two places, Synthetics saved objects which always encrypt sensitive fields using [Kibana Secure Saved Objects](/deploy-manage/security/secure-saved-objects.md) and also in Fleet, which uses unencrypted saved objects restricted by user permissions. For Elastic Cloud customers all data is secured on disk regardless of whether additional saved object encryption is present. See our [Cloud Security Statement](https://www.elastic.co/cloud/security) for more information. We recommend that self-managed customers encrypt disks for their Elasticsearch instances if this is a concern. All data is encrypted in transit. See [Elastic Agent configuration encryption](/reference/fleet/_agent_configuration_encryption.md) for more details. \ No newline at end of file diff --git a/solutions/observability/apps/synthetics-support-matrix.md b/solutions/observability/apps/synthetics-support-matrix.md index b62ba2ab53..53bed3ad7e 100644 --- a/solutions/observability/apps/synthetics-support-matrix.md +++ b/solutions/observability/apps/synthetics-support-matrix.md @@ -9,7 +9,6 @@ applies_to: There are various components that make up the Synthetics solution, which are supported in the following configurations: - ## {{synthetics-app}} [_synthetics_app] * **GA support**: 8.8.0 and higher @@ -18,14 +17,11 @@ There are various components that make up the Synthetics solution, which are sup * For creating and managing lightweight and browser monitors configured through the [{{synthetics-app}}](create-monitors-in-synthetics-app.md) * For reporting for lightweight and browser monitors configured through the [{{synthetics-app}}](create-monitors-in-synthetics-app.md) and/or [{{project-monitors-cap}}](create-monitors-with-project-monitors.md) - - ## {{project-monitors-cap}} [_project_monitors_cap] * **GA support**: 8.8.0 and higher * **Notes**: For creating and managing lightweight and browser monitors configured as [{{project-monitors-cap}}](create-monitors-with-project-monitors.md) - ## Elastic’s global managed testing infrastructure [_elastics_global_managed_testing_infrastructure_2] * **GA support**: 8.8.0 and higher @@ -33,8 +29,6 @@ There are various components that make up the Synthetics solution, which are sup Executing synthetic tests on Elastic’s global managed testing infrastructure incurs an additional charge. Tests are charged under one of two new billing dimensions depending on the monitor type. For *browser monitor* usage, there is a fee per test run. For *lightweight monitor* usage, there is a fee per region in which you run any monitors regardless of the number of test runs. For more details, refer to [full details and current pricing](https://www.elastic.co/pricing). - - ## {{private-location}}s [_private_locations_2] * **GA support**: 8.8.0 and higher @@ -50,8 +44,6 @@ There are various components that make up the Synthetics solution, which are sup * Do not configure any ingest pipelines or Logstash output - - ## Heartbeat with Uptime [_heartbeat_with_uptime] ```{applies_to} @@ -65,8 +57,6 @@ serverless: unavailable * For running lightweight monitors via YML configuration running on self-managed infrastructure * Browser-based monitors are not supported in this configuration - - ## Standalone {{agent}} [_standalone_agent] * **GA support**: As defined in the standard [Support matrix](https://www.elastic.co/support/matrix) @@ -76,8 +66,6 @@ serverless: unavailable * Browser-based monitors are not supported in this configuration * Results for monitors configured using the standalone {{agent}} are available in the {{uptime-app}} (*not* the {{synthetics-app}}) - - ## Synthetics Recorder [_synthetics_recorder] System requirements: @@ -90,8 +78,6 @@ System requirements: * Fedora - 24 and newer * Debian - 8 and newer - - ## Output to Elasticsearch [_output_to_elasticsearch] Synthetics must have a direct connection to {{es}}, whether running monitors from Elastic’s global managed testing infrastructure or from {{private-location}}s. diff --git a/solutions/observability/apps/tail-based-sampling.md b/solutions/observability/apps/tail-based-sampling.md index 922e30e8ef..e0a7e4258c 100644 --- a/solutions/observability/apps/tail-based-sampling.md +++ b/solutions/observability/apps/tail-based-sampling.md @@ -14,7 +14,6 @@ Most options on this page are supported by all APM Server deployment methods whe :::: - Tail-based sampling configuration options. :::::::{tab-set} @@ -51,7 +50,6 @@ Configure and customize Fleet-managed APM settings directly in {{kib}}: See [Tail-based sampling](transaction-sampling.md#apm-tail-based-sampling) to learn more. - ### Enable tail-based sampling [sampling-tail-enabled-ref] Set to `true` to enable tail based sampling. Disabled by default. (bool) @@ -61,7 +59,6 @@ Set to `true` to enable tail based sampling. Disabled by default. (bool) | APM Server binary | `sampling.tail.enabled` | | Fleet-managed | `Enable tail-based sampling` | - ### Interval [sampling-tail-interval-ref] Synchronization interval for multiple APM Servers. Should be in the order of tens of seconds or low minutes. Default: `1m` (1 minute). (duration) @@ -71,7 +68,6 @@ Synchronization interval for multiple APM Servers. Should be in the order of ten | APM Server binary | `sampling.tail.interval` | | Fleet-managed | `Interval` | - ### Policies [sampling-tail-policies-ref] Criteria used to match a root transaction to a sample rate. @@ -83,7 +79,6 @@ Policies map trace events to a sample rate. Each policy must specify a sample ra | APM Server binary | `sampling.tail.policies` | | Fleet-managed | `Policies` | - ### Storage limit [sampling-tail-storage_limit-ref] The amount of storage space allocated for trace events matching tail sampling policies. Caution: Setting this limit higher than the allowed space may cause APM Server to become unhealthy. @@ -101,39 +96,32 @@ Default: `0GB`. (text) | APM Server binary | `sampling.tail.storage_limit` | | Fleet-managed | `Storage limit` | - ## Policy-level tail-based sampling settings [apm-configuration-tbs-policy] See [Tail-based sampling](transaction-sampling.md#apm-tail-based-sampling) to learn more. - ### **`sample_rate`** [sampling-tail-sample-rate-ref] The sample rate to apply to trace events matching this policy. Required in each policy. The sample rate must be greater than or equal to `0` and less than or equal to `1`. For example, a `sample_rate` of `0.01` means that 1% of trace events matching the policy will be sampled. A `sample_rate` of `1` means that 100% of trace events matching the policy will be sampled. (int) - ### **`trace.name`** [sampling-tail-trace-name-ref] The trace name for events to match a policy. A match occurs when the configured `trace.name` matches the `transaction.name` of the root transaction of a trace. A root transaction is any transaction without a `parent.id`. (string) - ### **`trace.outcome`** [sampling-tail-trace-outcome-ref] The trace outcome for events to match a policy. A match occurs when the configured `trace.outcome` matches a trace’s `event.outcome` field. Trace outcome can be `success`, `failure`, or `unknown`. (string) - ### **`service.name`** [sampling-tail-service-name-ref] The service name for events to match a policy. (string) - ### **`service.environment`** [sampling-tail-service-environment-ref] The service environment for events to match a policy. (string) - ## Monitoring tail-based sampling [sampling-tail-monitoring-ref] APM Server produces metrics to monitor the performance and estimate the workload being processed by tail-based sampling. In order to use these metrics, you need to [enable monitoring for the APM Server](/solutions/observability/apps/monitor-apm-server.md). The following metrics are produced by the tail-based sampler (note that the metrics might have a different prefix, for example `beat.stats` for ECH deployments, based on how the APM Server is running): diff --git a/solutions/observability/apps/trace-sample-timeline.md b/solutions/observability/apps/trace-sample-timeline.md index d1393f9db9..e85e773fbd 100644 --- a/solutions/observability/apps/trace-sample-timeline.md +++ b/solutions/observability/apps/trace-sample-timeline.md @@ -22,13 +22,11 @@ View a span in detail by clicking on it in the timeline waterfall. For example, A [span](/solutions/observability/apps/spans.md) is the duration of a single event. Spans are automatically captured by APM agents, and you can also define custom spans. Each span has a type and is defined by a different color in the timeline/waterfall visualization. :::: - :::{image} /solutions/images/observability-apm-span-detail.png :alt: Example view of a span detail in the Applications UI :screenshot: ::: - ## Investigate [trace-sample-investigate] The trace sample timeline features an **Investigate** button which provides a quick way to jump to other areas of the Elastic Observability UI while maintaining the context of the currently selected trace sample. For example, quickly view: @@ -37,10 +35,9 @@ The trace sample timeline features an **Investigate** button which provides a qu * logs and metrics for the selected host * trace logs for the selected `trace.id` * uptime status of the selected domain -* the [service map](../../../solutions/observability/apps/service-map.md) filtered by the selected trace +* the [service map](/solutions/observability/apps/service-map.md) filtered by the selected trace * the selected transaction in **Discover** -* your [custom links](../../../solutions/observability/apps/create-custom-links.md) - +* your [custom links](/solutions/observability/apps/create-custom-links.md) ## Distributed tracing [distributed-tracing] diff --git a/solutions/observability/apps/traces-2.md b/solutions/observability/apps/traces-2.md index 391fc39185..d00ce74dbc 100644 --- a/solutions/observability/apps/traces-2.md +++ b/solutions/observability/apps/traces-2.md @@ -10,13 +10,12 @@ applies_to: # Traces [apm-traces] ::::{tip} -Traces link together related transactions to show an end-to-end performance of how a request was served and which services were part of it. In addition to the Traces overview, you can view your application traces in the [trace sample timeline waterfall](../../../solutions/observability/apps/trace-sample-timeline.md). +Traces link together related transactions to show an end-to-end performance of how a request was served and which services were part of it. In addition to the Traces overview, you can view your application traces in the [trace sample timeline waterfall](/solutions/observability/apps/trace-sample-timeline.md). :::: +**Traces** displays your application’s entry (root) transactions. Transactions with the same name are grouped together and only shown once in this table. If you’re using [distributed tracing](/solutions/observability/apps/trace-sample-timeline.md#distributed-tracing), this view is key to finding the critical paths within your application. -**Traces** displays your application’s entry (root) transactions. Transactions with the same name are grouped together and only shown once in this table. If you’re using [distributed tracing](../../../solutions/observability/apps/trace-sample-timeline.md#distributed-tracing), this view is key to finding the critical paths within your application. - -By default, transactions are sorted by *Impact*. Impact helps show the most used and slowest endpoints in your service — in other words, it’s the collective amount of pain a specific endpoint is causing your users. If there’s a particular endpoint you’re worried about, select it to view its [transaction details](../../../solutions/observability/apps/transactions-2.md#transaction-details). +By default, transactions are sorted by *Impact*. Impact helps show the most used and slowest endpoints in your service — in other words, it’s the collective amount of pain a specific endpoint is causing your users. If there’s a particular endpoint you’re worried about, select it to view its [transaction details](/solutions/observability/apps/transactions-2.md#transaction-details). You can also use queries to filter and search the transactions shown on this page. Note that only properties available on root transactions are searchable. For example, you can’t search for `label.tier: 'high'`, as that field is only available on non-root transactions. @@ -25,17 +24,15 @@ You can also use queries to filter and search the transactions shown on this pag :screenshot: ::: - ## Trace explorer [trace-explorer] ::::{warning} This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. :::: +**Trace explorer** is an experimental top-level search tool that allows you to query your traces using [{{kib}} Query Language (KQL)](/explore-analyze/query-filter/languages/kql.md) or [Event Query Language (EQL)](/explore-analyze/query-filter/languages/eql.md). -**Trace explorer** is an experimental top-level search tool that allows you to query your traces using [{{kib}} Query Language (KQL)](../../../explore-analyze/query-filter/languages/kql.md) or [Event Query Language (EQL)](../../../explore-analyze/query-filter/languages/eql.md). - -Curate your own custom queries, or use the [**Service Map**](../../../solutions/observability/apps/service-map.md) to find and select edges to automatically generate queries based on your selection: +Curate your own custom queries, or use the [**Service Map**](/solutions/observability/apps/service-map.md) to find and select edges to automatically generate queries based on your selection: :::{image} /solutions/images/observability-trace-explorer.png :alt: Trace explorer diff --git a/solutions/observability/apps/traces.md b/solutions/observability/apps/traces.md index a131ffef85..69a4e2b110 100644 --- a/solutions/observability/apps/traces.md +++ b/solutions/observability/apps/traces.md @@ -9,8 +9,7 @@ applies_to: # Traces [apm-data-model-traces] -A trace is a group of [transactions](../../../solutions/observability/apps/transactions.md) and [spans](../../../solutions/observability/apps/spans.md) with a common root. Each trace tracks the entirety of a single request. It describes the individual operations and their causality that ensue from a single logical operation. - +A trace is a group of [transactions](/solutions/observability/apps/transactions.md) and [spans](/solutions/observability/apps/spans.md) with a common root. Each trace tracks the entirety of a single request. It describes the individual operations and their causality that ensue from a single logical operation. ## Distributed tracing [apm-distributed-tracing] @@ -24,14 +23,12 @@ Tracking requests as they propagate through your services provides an end-to-end For supported technologies, distributed tracing works out-of-the-box, with no additional configuration required. - ### How distributed tracing works [apm-how-distributed-tracing] Distributed tracing works by injecting a custom `traceparent` HTTP header into outgoing requests. This header includes information, like `trace-id`, which is used to identify the current trace, and `parent-id`, which is used to identify the parent of the current span on incoming requests or the current span on an outgoing request. When a service is working on a request, it checks for the existence of this HTTP header. If it’s missing, the service starts a new trace. If it exists, the service ensures the current action is added as a child of the existing trace, and continues to propagate the trace. - ### Trace propagation examples [apm-trace-propagation] In this example, Elastic’s Ruby agent communicates with Elastic’s Java agent. Both support the `traceparent` header, and trace data is successfully propagated. @@ -52,7 +49,6 @@ In this example, the trace meets a piece of middleware that doesn’t propagate :alt: How traceparent propagation works ::: - ### W3C Trace Context specification [apm-w3c-tracecontext-spec] All Elastic agents now support the official W3C Trace Context specification and `traceparent` header. See the table below for the minimum required agent version: @@ -74,8 +70,6 @@ All Elastic agents now support the official W3C Trace Context specification and Older Elastic agents use a unique `elastic-apm-traceparent` header. For backward-compatibility purposes, new versions of Elastic agents still support this header. :::: - - ### Visualize distributed tracing [apm-visualize-distributed-tracing] APM's timeline visualization provides a visual deep-dive into each of your application’s traces: @@ -85,12 +79,10 @@ APM's timeline visualization provides a visual deep-dive into each of your appli :screenshot: ::: - ### Manual distributed tracing [apm-manual-distributed-tracing] Elastic agents automatically propagate distributed tracing context for supported technologies. If your service communicates over a different, unsupported protocol, you can manually propagate distributed tracing context from a sending service to a receiving service with each agent’s API. - #### Add the `traceparent` header to outgoing requests [apm-distributed-tracing-outgoing] Sending services must add the `traceparent` header to outgoing requests. @@ -380,7 +372,6 @@ $receiverTransaction = ElasticApm::beginCurrentTransaction( <1> 1. Start a new transaction 2. Pass in the received distributed tracing data (serialized as string) - Once this new transaction has been created in the receiving service, you can create child spans, or use any other agent API methods as you typically would. :::::: @@ -398,7 +389,6 @@ client.begin_transaction('processors', trace_parent=parent) <2> 1. Create a `TraceParent` object from HTTP headers formed as a dictionary 2. Begin a new transaction as a child of the received `TraceParent` - ::::{tip} See the [`TraceParent` API](apm-agent-python://reference/api-reference.md#traceparent-api) for additional examples. :::: diff --git a/solutions/observability/apps/track-deployments-with-annotations.md b/solutions/observability/apps/track-deployments-with-annotations.md index d8bb9a86b7..66b0912cd3 100644 --- a/solutions/observability/apps/track-deployments-with-annotations.md +++ b/solutions/observability/apps/track-deployments-with-annotations.md @@ -12,7 +12,7 @@ applies_to: ::::{note} -**For Observability Serverless projects**, the **Admin** role or higher is required to create and manage annotations. To learn more, refer to [Assign user roles and privileges](../../../deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). +**For Observability Serverless projects**, the **Admin** role or higher is required to create and manage annotations. To learn more, refer to [Assign user roles and privileges](/deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles). :::: diff --git a/solutions/observability/apps/transaction-sampling.md b/solutions/observability/apps/transaction-sampling.md index e7497a0141..9b3e25cf86 100644 --- a/solutions/observability/apps/transaction-sampling.md +++ b/solutions/observability/apps/transaction-sampling.md @@ -9,26 +9,32 @@ applies_to: # Transaction sampling [apm-sampling] -[Distributed tracing](../../../solutions/observability/apps/traces.md) can generate a substantial amount of data. More data can mean higher costs and more noise. Sampling aims to lower the amount of data ingested and the effort required to analyze that data — all while still making it easy to find anomalous patterns in your applications, detect outages, track errors, and lower mean time to recovery (MTTR). +:::{include} _snippets/apm-server-vs-mis.md +::: + +[Distributed tracing](/solutions/observability/apps/traces.md) can generate a substantial amount of data. More data can mean higher costs and more noise. Sampling aims to lower the amount of data ingested and the effort required to analyze that data — all while still making it easy to find anomalous patterns in your applications, detect outages, track errors, and lower mean time to recovery (MTTR). Elastic APM supports two types of sampling: % Stateful only for Tail-based sampling -* [Head-based sampling](../../../solutions/observability/apps/transaction-sampling.md#apm-head-based-sampling) -* [Tail-based sampling](../../../solutions/observability/apps/transaction-sampling.md#apm-tail-based-sampling) - +* [Head-based sampling](/solutions/observability/apps/transaction-sampling.md#apm-head-based-sampling) +* [Tail-based sampling](/solutions/observability/apps/transaction-sampling.md#apm-tail-based-sampling) ## Head-based sampling [apm-head-based-sampling] +```{applies_to} +stack: +serverless: +``` + In head-based sampling, the sampling decision for each trace is made when the trace is initiated. Each trace has a defined and equal probability of being sampled. For example, a sampling value of `.2` indicates a transaction sample rate of `20%`. This means that only `20%` of traces will send and retain all of their associated information. The remaining traces will drop contextual information to reduce the transfer and storage size of the trace. Head-based sampling is quick and easy to set up. Its downside is that it’s entirely random — interesting data might be discarded purely due to chance. -See [Configure head-based sampling](../../../solutions/observability/apps/transaction-sampling.md#apm-configure-head-based-sampling) to get started. - +See [Configure head-based sampling](/solutions/observability/apps/transaction-sampling.md#apm-configure-head-based-sampling) to get started. ### Distributed tracing [distributed-tracing-examples] @@ -52,12 +58,11 @@ In the example in *Figure 2*, `Service A` initiates four transactions and has a :screenshot: ::: - #### Trace continuation strategies with distributed tracing [_trace_continuation_strategies_with_distributed_tracing] In addition to setting the sample rate, you can also specify which *trace continuation strategy* to use. There are three trace continuation strategies: `continue`, `restart`, and `restart_external`. -The **`continue`** trace continuation strategy is the default and will behave similar to the examples in the [Distributed tracing section](../../../solutions/observability/apps/transaction-sampling.md#distributed-tracing-examples). +The **`continue`** trace continuation strategy is the default and will behave similar to the examples in the [Distributed tracing section](/solutions/observability/apps/transaction-sampling.md#distributed-tracing-examples). Use the **`restart_external`** trace continuation strategy on an Elastic-monitored service to start a new trace if the previous service did not have a `traceparent` header with `es` vendor data. This can be helpful if a transaction includes an Elastic-monitored service that is receiving requests from an unmonitored service. @@ -79,7 +84,6 @@ In the example in *Figure 4*, `Service A` and `Service B` are Elastic-monitored :title: Using the `restart` trace continuation strategy ::: - ### OpenTelemetry [_opentelemetry] Head-based sampling is implemented directly in the APM agents and SDKs. The sample rate must be propagated between services and the managed intake service in order to produce accurate metrics. @@ -98,26 +102,26 @@ Refer to the documentation of your favorite OpenTelemetry agent or SDK for more % Stateful only for tail-based sampling ## Tail-based sampling [apm-tail-based-sampling] + ```{applies_to} -stack: all +stack: +serverless: unavailable ``` ::::{note} **Support for tail-based sampling** -Tail-based sampling is only supported when writing to {{es}}. If you are using a different [output](../../../solutions/observability/apps/configure-output.md), tail-based sampling is *not* supported. +Tail-based sampling is only supported when writing to {{es}}. If you are using a different [output](/solutions/observability/apps/configure-output.md), tail-based sampling is *not* supported. :::: - In tail-based sampling, the sampling decision for each trace is made after the trace has completed. This means all traces will be analyzed against a set of rules, or policies, which will determine the rate at which they are sampled. Unlike head-based sampling, each trace does not have an equal probability of being sampled. Because slower traces are more interesting than faster ones, tail-based sampling uses weighted random sampling — so traces with a longer root transaction duration are more likely to be sampled than traces with a fast root transaction duration. A downside of tail-based sampling is that it results in more data being sent from APM agents to the APM Server. The APM Server will therefore use more CPU, memory, and disk than with head-based sampling. However, because the tail-based sampling decision happens in APM Server, there is less data to transfer from APM Server to {{es}}. So running APM Server close to your instrumented services can reduce any increase in transfer costs that tail-based sampling brings. -See [Configure tail-based sampling](../../../solutions/observability/apps/transaction-sampling.md#apm-configure-tail-based-sampling) to get started. - +See [Configure tail-based sampling](/solutions/observability/apps/transaction-sampling.md#apm-configure-tail-based-sampling) to get started. ### Distributed tracing with tail-based sampling [_distributed_tracing_with_tail_based_sampling] @@ -129,12 +133,11 @@ In this example, `Service A` initiates four transactions. If our sample rate is :alt: Distributed tracing and tail based sampling example one ::: - ### OpenTelemetry with tail-based sampling [_opentelemetry_with_tail_based_sampling] Tail-based sampling is implemented entirely in APM Server, and will work with traces sent by either Elastic APM agents or OpenTelemetry SDKs. -Due to [OpenTelemetry tail-based sampling limitations](../../../solutions/observability/apps/limitations.md#apm-open-telemetry-tbs) when using [tailsamplingprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor), we recommend using APM Server tail-based sampling instead. +Due to [OpenTelemetry tail-based sampling limitations](/solutions/observability/apps/limitations.md#apm-open-telemetry-tbs) when using [tailsamplingprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor), we recommend using APM Server tail-based sampling instead. ### Tail-based sampling performance and requirements [_tail_based_sampling_performance_and_requirements] @@ -187,9 +190,14 @@ The tail-based sampling implementation in version 9.0 offers significantly bette ## Sampled data and visualizations [_sampled_data_and_visualizations] -A sampled trace retains all data associated with it. A non-sampled trace drops all [span](../../../solutions/observability/apps/spans.md) and [transaction](../../../solutions/observability/apps/transactions.md) data1. Regardless of the sampling decision, all traces retain [error](../../../solutions/observability/apps/errors.md) data. +```{applies_to} +stack: +serverless: +``` + +A sampled trace retains all data associated with it. A non-sampled trace drops all [span](/solutions/observability/apps/spans.md) and [transaction](/solutions/observability/apps/transactions.md) data1. Regardless of the sampling decision, all traces retain [error](/solutions/observability/apps/errors.md) data. -Some visualizations in the {{apm-app}}, like latency, are powered by aggregated transaction and span [metrics](../../../solutions/observability/apps/metrics.md). The way these metrics are calculated depends on the sampling method used: +Some visualizations in the {{apm-app}}, like latency, are powered by aggregated transaction and span [metrics](/solutions/observability/apps/metrics.md). The way these metrics are calculated depends on the sampling method used: * **Head-based sampling**: Metrics are calculated based on all sampled events. * **Tail-based sampling**: Metrics are calculated based on all events, regardless of whether they are ultimately sampled or not. @@ -201,9 +209,13 @@ These calculation methods ensure that the APM app provides the most accurate met 1 Real User Monitoring (RUM) traces are an exception to this rule. The {{kib}} apps that utilize RUM data depend on transaction events, so non-sampled RUM traces retain transaction data — only span data is dropped. - ## Sample rates [_sample_rates] +```{applies_to} +stack: +serverless: +``` + What’s the best sampling rate? Unfortunately, there isn’t one. Sampling is dependent on your data, the throughput of your application, data retention policies, and other factors. A sampling rate from `.1%` to `100%` would all be considered normal. You’ll likely decide on a unique sample rate for different scenarios. Here are some examples: * Services with considerably more traffic than others might be safe to sample at lower rates @@ -213,22 +225,23 @@ What’s the best sampling rate? Unfortunately, there isn’t one. Sampling is d Regardless of the above, cost conscious customers are likely to be fine with a lower sample rate. - ## Configure head-based sampling [apm-configure-head-based-sampling] -There are three ways to adjust the head-based sampling rate of your APM agents: +```{applies_to} +stack: +serverless: +``` +There are three ways to adjust the head-based sampling rate of your APM agents: ### Dynamic configuration [_dynamic_configuration] -The transaction sample rate can be changed dynamically (no redeployment necessary) on a per-service and per-environment basis with [{{apm-agent}} Configuration](../../../solutions/observability/apps/apm-agent-central-configuration.md) in {{kib}}. - +The transaction sample rate can be changed dynamically (no redeployment necessary) on a per-service and per-environment basis with [{{apm-agent}} Configuration](/solutions/observability/apps/apm-agent-central-configuration.md) in {{kib}}. ### {{kib}} API configuration [_kib_api_configuration] {{apm-agent}} configuration exposes an API that can be used to programmatically change your agents' sampling rate. For examples, refer to the [Agent configuration API reference](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-apm-agent-configuration). - ### {{apm-agent}} configuration [_apm_agent_configuration] Each agent provides a configuration value used to set the transaction sample rate. See the relevant agent’s documentation for more details: @@ -241,10 +254,14 @@ Each agent provides a configuration value used to set the transaction sample rat * Python: [`transaction_sample_rate`](apm-agent-python://reference/configuration.md#config-transaction-sample-rate) * Ruby: [`transaction_sample_rate`](apm-agent-ruby://reference/configuration.md#config-transaction-sample-rate) - ## Configure tail-based sampling [apm-configure-tail-based-sampling] -Enable tail-based sampling with [Enable tail-based sampling](../../../solutions/observability/apps/tail-based-sampling.md#sampling-tail-enabled-ref). When enabled, trace events are mapped to sampling policies. Each sampling policy must specify a sample rate, and can optionally specify other conditions. All of the policy conditions must be true for a trace event to match it. +```{applies_to} +stack: +serverless: unavailable +``` + +Enable tail-based sampling with [Enable tail-based sampling](/solutions/observability/apps/tail-based-sampling.md#sampling-tail-enabled-ref). When enabled, trace events are mapped to sampling policies. Each sampling policy must specify a sample rate, and can optionally specify other conditions. All of the policy conditions must be true for a trace event to match it. Trace events are matched to policies in the order specified. Each policy list must conclude with a default policy — one that only specifies a sample rate. This default policy is used to catch remaining trace events that don’t match a stricter policy. Requiring this default policy ensures that traces are only dropped intentionally. If you enable tail-based sampling and send a transaction that does not match any of the policies, APM Server will reject the transaction with the error `no matching policy`. @@ -252,8 +269,6 @@ Trace events are matched to policies in the order specified. Each policy list mu Please note that from version `9.0.0` APM Server has an unlimited storage limit, but will stop writing when the disk where the database resides reaches 80% usage. Due to how the limit is calculated and enforced, the actual disk space may still grow slightly over this disk usage based limit, or any configured storage limit. :::: - - ### Example configuration [_example_configuration] This example defines three tail-based sampling polices: @@ -272,14 +287,10 @@ This example defines three tail-based sampling polices: 2. Samples 1% of traces in `production` with the trace name `"GET /not_important_route"` 3. Default policy to sample all remaining traces at 10%, e.g. traces in a different environment, like `dev`, or traces with any other name - - ### Configuration reference [_configuration_reference] - #### Top-level tail-based sampling settings [_top_level_tail_based_sampling_settings] - ##### Enable tail-based sampling [sampling-tail-enabled] Set to `true` to enable tail based sampling. Disabled by default. (bool) @@ -289,7 +300,6 @@ Set to `true` to enable tail based sampling. Disabled by default. (bool) | APM Server binary | `sampling.tail.enabled` | | Fleet-managed | `Enable tail-based sampling` | - ##### Interval [sampling-tail-interval] Synchronization interval for multiple APM Servers. Should be in the order of tens of seconds or low minutes. Default: `1m` (1 minute). (duration) @@ -299,7 +309,6 @@ Synchronization interval for multiple APM Servers. Should be in the order of ten | APM Server binary | `sampling.tail.interval` | | Fleet-managed | `Interval` | - ##### Policies [sampling-tail-policies] Criteria used to match a root transaction to a sample rate. @@ -311,7 +320,6 @@ Policies map trace events to a sample rate. Each policy must specify a sample ra | APM Server binary | `sampling.tail.policies` | | Fleet-managed | `Policies` | - ##### Storage limit [sampling-tail-storage_limit] The amount of storage space allocated for trace events matching tail sampling policies. Caution: Setting this limit higher than the allowed space may cause APM Server to become unhealthy. @@ -329,32 +337,26 @@ Default: `0GB`. (text) | APM Server binary | `sampling.tail.storage_limit` | | Fleet-managed | `Storage limit` | - #### Policy settings [_policy_settings] - ##### **`sample_rate`** [sampling-tail-sample-rate] The sample rate to apply to trace events matching this policy. Required in each policy. The sample rate must be greater than or equal to `0` and less than or equal to `1`. For example, a `sample_rate` of `0.01` means that 1% of trace events matching the policy will be sampled. A `sample_rate` of `1` means that 100% of trace events matching the policy will be sampled. (int) - ##### **`trace.name`** [sampling-tail-trace-name] The trace name for events to match a policy. A match occurs when the configured `trace.name` matches the `transaction.name` of the root transaction of a trace. A root transaction is any transaction without a `parent.id`. (string) - ##### **`trace.outcome`** [sampling-tail-trace-outcome] The trace outcome for events to match a policy. A match occurs when the configured `trace.outcome` matches a trace’s `event.outcome` field. Trace outcome can be `success`, `failure`, or `unknown`. (string) - ##### **`service.name`** [sampling-tail-service-name] The service name for events to match a policy. (string) - ##### **`service.environment`** [sampling-tail-service-environment] The service environment for events to match a policy. (string) diff --git a/solutions/observability/apps/transactions-2.md b/solutions/observability/apps/transactions-2.md index 163973f301..d1e38e68b2 100644 --- a/solutions/observability/apps/transactions-2.md +++ b/solutions/observability/apps/transactions-2.md @@ -11,7 +11,6 @@ applies_to: A *transaction* describes an event captured by an Elastic APM agent instrumenting a service. APM agents automatically collect performance metrics on HTTP requests, database queries, and much more. - :::{image} /solutions/images/observability-apm-transactions-overview.png :alt: Example view of transactions table in the Applications UI :screenshot: @@ -37,7 +36,6 @@ The **Latency**, **Throughput**, **Failed transaction rate**, **Time spent by sp :::: - **Time spent by span type** Visualize where your application is spending most of its time. For example, is your app spending time in external calls, database processing, or application code execution? @@ -45,10 +43,8 @@ The **Latency**, **Throughput**, **Failed transaction rate**, **Time spent by sp It’s important to note that if you have asynchronous spans, the sum of all span times may exceed the duration of the transaction. - **Cold start rate** - Only applicable to serverless transactions, this chart displays the percentage of requests that trigger a cold start of a serverless function. See [Cold starts](../../../solutions/observability/apps/observe-lambda-functions.md#apm-lambda-cold-start-info) for more information. - + Only applicable to serverless transactions, this chart displays the percentage of requests that trigger a cold start of a serverless function. See [Cold starts](/solutions/observability/apps/observe-lambda-functions.md#apm-lambda-cold-start-info) for more information. ## Transactions table [transactions-table] @@ -59,7 +55,7 @@ The **Transactions** table displays a list of *transaction groups* for the selec :screenshot: ::: -By default, transaction groups are sorted by *Impact*. Impact helps show the most used and slowest endpoints in your service - in other words, it’s the collective amount of pain a specific endpoint is causing your users. If there’s a particular endpoint you’re worried about, you can click on it to view the [transaction details](../../../solutions/observability/apps/transactions-2.md#transaction-details). +By default, transaction groups are sorted by *Impact*. Impact helps show the most used and slowest endpoints in your service - in other words, it’s the collective amount of pain a specific endpoint is causing your users. If there’s a particular endpoint you’re worried about, you can click on it to view the [transaction details](/solutions/observability/apps/transactions-2.md#transaction-details). ::::{important} If you only see one route in the Transactions table, or if you have transactions named "unknown route", it could be a symptom that the APM agent either wasn’t installed correctly or doesn’t support your framework. @@ -81,7 +77,6 @@ The transaction overview page is customized for the JavaScript RUM agent. Specif Additional RUM goodies, like core vitals, and visitor breakdown by browser, location, and device, are available in the Observability User Experience tab. - ## Transaction details [transaction-details] Selecting a transaction group will bring you to the **transaction** details. This page is visually similar to the transaction overview, but it shows data from all transactions within the selected transaction group. @@ -91,7 +86,6 @@ Selecting a transaction group will bring you to the **transaction** details. Thi :screenshot: ::: - ### Latency distribution [transaction-duration-distribution] The latency distribution shows a plot of all transaction durations for the given time period. The following screenshot shows a typical distribution and indicates most of our requests were served quickly — awesome! The requests on the right are taking longer than average; we probably need to focus on them. @@ -103,7 +97,6 @@ The latency distribution shows a plot of all transaction durations for the given Click and drag to select a latency duration *bucket* to display up to 500 trace samples. - ### Trace samples [transaction-trace-sample] Trace samples are based on the *bucket* selection in the **Latency distribution** chart; update the samples by selecting a new *bucket*. The number of requests per bucket is displayed when hovering over the graph, and the selected bucket is highlighted to stand out. @@ -120,11 +113,10 @@ Each sample has a trace timeline waterfall that shows how a typical request in t ::: ::::{note} -More information on timeline waterfalls is available in [spans](../../../solutions/observability/apps/trace-sample-timeline.md). +More information on timeline waterfalls is available in [spans](/solutions/observability/apps/trace-sample-timeline.md). :::: - **Trace sample metadata** Learn more about a trace sample in the **Metadata** tab: @@ -145,7 +137,6 @@ All of this data is stored in documents in Elasticsearch. This means you can sel :::: - **Trace sample logs** The **Logs** tab displays logs related to the sampled trace. @@ -154,17 +145,16 @@ Logs provide detailed information about specific events, and are crucial to succ If you’ve correlated your application’s logs and traces, you never have to search for relevant data; it’s already available to you. Viewing log and trace data together allows you to quickly diagnose and solve problems. -To learn how to correlate your logs with your instrumented services, see [Stream application logs](../../../solutions/observability/logs/stream-application-logs.md) +To learn how to correlate your logs with your instrumented services, see [Stream application logs](/solutions/observability/logs/stream-application-logs.md) :::{image} /solutions/images/observability-apm-logs-tab.png :alt: APM logs tab :screenshot: ::: - ### Correlations [transaction-latency-correlations] -Correlations surface attributes of your data that are potentially correlated with high-latency or erroneous transactions. To learn more, see [Find transaction latency and failure correlations](../../../solutions/observability/apps/find-transaction-latency-failure-correlations.md). +Correlations surface attributes of your data that are potentially correlated with high-latency or erroneous transactions. To learn more, see [Find transaction latency and failure correlations](/solutions/observability/apps/find-transaction-latency-failure-correlations.md). :::{image} /solutions/images/observability-correlations-hover.png :alt: APM lattency correlations diff --git a/solutions/observability/apps/transactions.md b/solutions/observability/apps/transactions.md index 63dc3d9603..00818dc3d4 100644 --- a/solutions/observability/apps/transactions.md +++ b/solutions/observability/apps/transactions.md @@ -42,8 +42,6 @@ Transactions are grouped by their `type` and `name` in the Applications UI’s [ Most agents limit keyword fields (e.g. `labels`) to 1024 characters, non-keyword fields (e.g. `span.db.statement`) to 10,000 characters. :::: - - ## Data streams [_data_streams_2] Transactions are stored with spans in the following data streams: @@ -53,7 +51,6 @@ Transactions are stored with spans in the following data streams: See [Data streams](data-streams.md) to learn more. - ## Example transaction document [_example_transaction_document] This example shows what transaction documents can look like when indexed in {{es}}. @@ -467,5 +464,3 @@ This example shows what transaction documents can look like when indexed in {{es :::: - - diff --git a/solutions/observability/apps/tune-data-ingestion.md b/solutions/observability/apps/tune-data-ingestion.md index c0e257923f..789635262c 100644 --- a/solutions/observability/apps/tune-data-ingestion.md +++ b/solutions/observability/apps/tune-data-ingestion.md @@ -9,21 +9,18 @@ applies_to: This section explains how to adapt data ingestion according to your needs. - ## Tune APM Server [apm-tune-apm-server] * [Add APM Server or {{agent}} instances](#apm-add-apm-server-instances) * [Reduce the payload size](#apm-reduce-payload-size) * [Adjust anonymous auth rate limit](#apm-adjust-event-rate) - ### Add APM Server or {{agent}} instances [apm-add-apm-server-instances] If the APM Server cannot process data quickly enough, you will see request timeouts. One way to solve this problem is to increase processing power. Increase processing power by either migrating to a more powerful machine or adding more APM Server/Elastic Agent instances. Having several instances will also increase [availability](high-availability.md). - ### Reduce the payload size [apm-reduce-payload-size] Large payloads may result in request timeouts. You can reduce the payload size by decreasing the flush interval in the agents. This will cause agents to send smaller and more frequent requests. @@ -32,7 +29,6 @@ Optionally you can also [reduce the sample rate](reduce-storage.md#apm-reduce-sa Read more in the [agents documentation](https://www.elastic.co/guide/en/apm/agent/index.html). - ### Adjust anonymous auth rate limit [apm-adjust-event-rate] Agents make use of long running requests and flush as many events over a single request as possible. Thus, the rate limiter for anonymous authentication is bound to the number of *events* sent per second, per IP. @@ -46,12 +42,11 @@ Increasing the default value for the following configuration variable will help | APM Server binary | [`rate_limit.event_limit`](configure-anonymous-authentication.md#apm-config-auth-anon-event-limit) | | Fleet-managed | `Anonymous Event rate limit (event limit)` | - ## Tune {{es}} [apm-tune-elasticsearch] The {{es}} Reference provides insight on tuning {{es}}. -[Tune for indexing speed](../../../deploy-manage/production-guidance/optimize-performance/indexing-speed.md) provides information on: +[Tune for indexing speed](/deploy-manage/production-guidance/optimize-performance/indexing-speed.md) provides information on: * Refresh interval * Disabling swapping @@ -59,7 +54,7 @@ The {{es}} Reference provides insight on tuning {{es}}. * Considerations regarding faster hardware * Setting the indexing buffer size -[Tune for disk usage](../../../deploy-manage/production-guidance/optimize-performance/disk-usage.md) provides information on: +[Tune for disk usage](/deploy-manage/production-guidance/optimize-performance/disk-usage.md) provides information on: * Disabling unneeded features * Shard size diff --git a/solutions/observability/apps/tutorial-monitor-java-application.md b/solutions/observability/apps/tutorial-monitor-java-application.md index 888ec0f693..c96d55708f 100644 --- a/solutions/observability/apps/tutorial-monitor-java-application.md +++ b/solutions/observability/apps/tutorial-monitor-java-application.md @@ -9,7 +9,6 @@ applies_to: In this guide, you’ll learn how to monitor a Java application using Elastic {{observability}}: Logs, Infrastructure metrics, APM, and Uptime. - ## What you’ll learn [_what_youll_learn] You’ll learn how to: @@ -20,12 +19,10 @@ You’ll learn how to: * Instrument your application using the [Elastic APM Java agent](apm-agent-java://reference/index.md). * Monitor your services using {{heartbeat}} and view your uptime data in {{kib}}. - ## Before you begin [_before_you_begin] Create an [{{ech}}](https://cloud.elastic.co/registration?page=docs&placement=docs-body) deployment. The deployment includes an {{es}} cluster for storing and searching your data, {{kib}} for visualizing and managing your data, and an APM server. If you do not want to follow all those steps listed here and take a look at the final java code, check out the [observability-contrib GitHub repository](https://github.com/elastic/observability-contrib/tree/main/monitor-java-app) for the sample application. - ## Step 1: Create a Java application [_step_1_create_a_java_application] To create the Java application, you require OpenJDK 14 (or higher) and the [Javalin](https://javalin.io/) web framework. The application will include the main endpoint, an artificially long-running endpoint, and an endpoint that needs to poll another data source. There will also be a background job running. @@ -201,13 +198,10 @@ To create the Java application, you require OpenJDK 14 (or higher) and the [Java ./gradlew clean check shadowJar ``` - - ## Step 2: Ingest logs [_step_2_ingest_logs] Logs can be events such as checkout, an exception, or an HTTP request. For this tutorial, let’s use log4j2 as our logging implementation. - ### Add logging implementation [_add_logging_implementation] 1. Add the dependency to the `build.gradle` file. @@ -228,7 +222,6 @@ Logs can be events such as checkout, an exception, or an HTTP request. For this :::: - ```java package de.spinscale.javalin; @@ -281,8 +274,6 @@ Logs can be events such as checkout, an exception, or an HTTP request. For this 17:17:40.019 [INFO ] de.spinscale.javalin.App - This is an informative logging message, user agent [curl/7.64.1] ``` - - ### Log requests [_log_requests] Depending on the application traffic and whether it happens outside of the application, it makes sense to log each request on the application level. @@ -321,8 +312,6 @@ Depending on the application traffic and whether it happens outside of the appli 10:43:50.066 [INFO ] de.spinscale.javalin.App - GET / 200 0:0:0:0:0:0:0:1 "curl/7.64.1" 7 ``` - - ### Create an ISO8601 timestamp [_create_an_iso8601_timestamp] Before ingesting logs into {{ech}}, create an ISO8601 timestamp by editing the `log4j2.xml` file. @@ -332,7 +321,6 @@ Creating an ISO8601 timestamp removes the need to do any calculation for timesta :::: - ```text ``` @@ -343,7 +331,6 @@ The log entries are ingested containing timestamps like the following. 2020-07-03T14:25:40,378+02:00 [INFO ] de.spinscale.javalin.App GET / 200 0:0:0:0:0:0:0:1 "curl/7.64.1" 0 ``` - ### Log to a file and stdout [_log_to_a_file_and_stdout] 1. To read the logging output, let’s write data into a file and to stdout. This is a new `log4j2.xml` file. @@ -371,7 +358,6 @@ The log entries are ingested containing timestamps like the following. 2. Restart the application and send a request. The logs will be sent to `/tmp/javalin/app.log`. - ### Install and configure {{filebeat}} [_install_and_configure_filebeat] To read the log file and send it to {{es}}, {{filebeat}} is required. To download and install {{filebeat}}, use the commands that work with your system: @@ -433,7 +419,6 @@ NOTE: If script execution is disabled on your system, you need to set the execut :::: - ```bash ./filebeat keystore create echo -n "" | ./filebeat keystore add CLOUD_ID --stdin @@ -472,7 +457,6 @@ NOTE: If script execution is disabled on your system, you need to set the execut :::: - To see if both settings have been stored, run `./filebeat keystore list`. 3. To load the {{filebeat}} dashboards, use the `elastic` super user. @@ -501,8 +485,6 @@ NOTE: If script execution is disabled on your system, you need to set the execut api_key: ${ES_API_KEY} ``` - - ### Send data to {{es}} [_send_data_to_es] To send data to {{es}}, start {{filebeat}}. @@ -518,7 +500,6 @@ sudo service filebeat start If you use an `init.d` script to start Filebeat, you can’t specify command line flags (see [Command reference](https://www.elastic.co/guide/en/beats/filebeat/master/command-line-options.html)). To specify flags, start Filebeat in the foreground. :::: - Also see [Filebeat and systemd](https://www.elastic.co/guide/en/beats/filebeat/master/running-with-systemd.html). :::::: @@ -531,7 +512,6 @@ sudo service filebeat start If you use an `init.d` script to start Filebeat, you can’t specify command line flags (see [Command reference](https://www.elastic.co/guide/en/beats/filebeat/master/command-line-options.html)). To specify flags, start Filebeat in the foreground. :::: - Also see [Filebeat and systemd](https://www.elastic.co/guide/en/beats/filebeat/master/running-with-systemd.html). :::::: @@ -570,7 +550,6 @@ wrk -t1 -c 100 -d10s http://localhost:7000 This command results in roughly 8,000 requests per second, and the equivalent number of log lines are also written. - ## Step 3: View logs in {{kib}} [_step_3_view_logs_in_kib] 1. Log into {{kib}} and select the **Discover** app. @@ -632,11 +611,8 @@ This command results in roughly 8,000 requests per second, and the equivalent nu * When you compare the `@timestamp` field with the timestamp of the log message, you will notice that it differs. This means that you don’t get the results you expect when filtering based on the `@timestamp` field. The current `@timestamp` field reflects the timestamp when the event was created within {{filebeat}}, not the timestamp of when the log event occurred in the application. * One cannot filter on specific fields like the HTTP verb, the HTTP status code, the log level or the class that generated the log message - - ## Step 4: Work with your logs [_step_4_work_with_your_logs] - ### Structure logs [_structure_logs] To extract more data from a single log line into several fields requires additional structuring of the logs. @@ -649,7 +625,7 @@ Let’s take another look at a log message generated by our app. This message has four parts: `timestamp`, `log level`, `class`, and `message`. The rules of splitting are apparent as well, as most of them involve whitespace. -The good news is that all {{beats}} can process a log line before sending it to {{es}} by using [processors](beats://reference/filebeat/filtering-enhancing-data.md). If the capabilities of these processors are not enough, you can always let {{es}} do the heavy lifting by using [an ingest node](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md). This is what many modules in {{filebeat}} do. A module in {{filebeat}} is a way to parse a specific log file format for a particular software. +The good news is that all {{beats}} can process a log line before sending it to {{es}} by using [processors](beats://reference/filebeat/filtering-enhancing-data.md). If the capabilities of these processors are not enough, you can always let {{es}} do the heavy lifting by using [an ingest node](/manage-data/ingest/transform-enrich/ingest-pipelines.md). This is what many modules in {{filebeat}} do. A module in {{filebeat}} is a way to parse a specific log file format for a particular software. Let’s try this by using a couple of processors and only a {{filebeat}} configuration. @@ -683,7 +659,6 @@ The removal of parts of the original message is debatable. Keeping the original :::: - There is also a slight difference in the parsing of a timestamp as the go time parser only accepts dots as a separator between seconds and milliseconds. Still, our default output of the log4j2 is using a comma. Either one can fix the timestamp in the logging output to look like one expected from {{filebeat}}. This results in the following pattern layout. @@ -779,7 +754,6 @@ When the log level is `INFO`, it is logged with additional space at the end. You ``` - ### Parse exceptions [_parse_exceptions] Exceptions are a special treat in the case of logging. They span multiple lines, so the old rule of one message per line does not exist in exceptions. @@ -830,10 +804,8 @@ This introduces state into your logging. You cannot split a log file among sever :::: - After restarting {{filebeat}} and your Javalin app, trigger an exception, and you will see a long stack trace in the `message` field of your logs. - ### Configure log rotation [_configure_log_rotation] To ensure that logs don’t grow infinitely, let’s add some log rotation to your logging configuration. @@ -870,10 +842,9 @@ The sample added a `JavalinAppLogRolling` appender to our configuration that use If a new log file is created, older log files are gzipped as well to take less space on disk. The size of 50 megabytes refers to the unpacked file size, so the potentially twenty files on disk will be much smaller each. - ### Ingest node [_ingest_node] -The built-in modules are almost entirely using the [Ingest node](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md) feature of {{es}} instead of the {{beats}} processors. +The built-in modules are almost entirely using the [Ingest node](/manage-data/ingest/transform-enrich/ingest-pipelines.md) feature of {{es}} instead of the {{beats}} processors. One of the most helpful parts of the ingest pipeline is the ability to debug by using the [Simulate Pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-simulate). @@ -939,7 +910,6 @@ One of the most helpful parts of the ingest pipeline is the ability to debug by 4. Restart {{filebeat}} and see if logs are flowing in as expected. - ### Write logs as JSON [_write_logs_as_json] You have now learned about parsing logs in either {{beats}} or {{es}}. What if we didn’t need to think about parsing our logs and extract data manually? @@ -1007,8 +977,6 @@ While log4j2 has a [JSONLayout](https://logging.apache.org/log4j/2.x/manual/layo As you can see, just by writing out logs as JSON, our whole logging setup got a ton easier, so whenever possible, try to directly write your logs as JSON. - - ## Step 5: Ingest metrics [_step_5_ingest_metrics] A metric is considered a point in time value that can change anytime. The number of current requests can change any millisecond. You could have a spike of 1000 requests, and then everything goes back to one request. This also means that these kinds of metrics may not be accurate, and you also want to pull min/max values to get some more indication. Furthermore, this implies that you need to think about the duration of those metrics as well. Do you need those once per minute or every 10 seconds? @@ -1017,7 +985,6 @@ To get a different angled view of your application, let’s ingest some metrics. The underlying library used in our app is [micrometer.io](http://micrometer.io/), a vendor-neutral application metrics facade in combination with its [Prometheus support](http://micrometer.io/docs/registry/prometheus) to implement a pull-based model. You could use the [elastic support](http://micrometer.io/docs/registry/elastic) to achieve a push-based model. This would require users to store credential data of the {{es}} cluster in our app. This example keeps this data in the surrounding tools. - ### Add metrics to the application [_add_metrics_to_the_application] 1. Add the dependencies to our `build.gradle` file. @@ -1117,8 +1084,6 @@ The underlying library used in our app is [micrometer.io](http://micrometer.io/) This returns a line based response with one metric per line. This is the standard Prometheus format. - - ### Install and configure {{metricbeat}} [_install_and_configure_metricbeat] To send metrics to {{es}}, {{metricbeat}} is required. To download and install {{metricbeat}}, use the commands that work with your system: @@ -1260,7 +1225,6 @@ sudo service metricbeat start If you use an `init.d` script to start Metricbeat, you can’t specify command line flags (see [Command reference](https://www.elastic.co/guide/en/beats/metricbeat/master/command-line-options.html)). To specify flags, start Metricbeat in the foreground. :::: - Also see [Metricbeat and systemd](https://www.elastic.co/guide/en/beats/metricbeat/master/running-with-systemd.html). :::::: @@ -1273,7 +1237,6 @@ sudo service metricbeat start If you use an `init.d` script to start Metricbeat, you can’t specify command line flags (see [Command reference](https://www.elastic.co/guide/en/beats/metricbeat/master/command-line-options.html)). To specify flags, start Metricbeat in the foreground. :::: - Also see [Metricbeat and systemd](https://www.elastic.co/guide/en/beats/metricbeat/master/running-with-systemd.html). :::::: @@ -1321,7 +1284,6 @@ GET metricbeat-*/_search?filter_path=**.prometheus,hits.total } ``` - ## Step 6: View metrics in {{kib}} [_step_6_view_metrics_in_kib] As this is custom data from our Javalin app, there is no predefined dashboard for displaying this data. @@ -1352,8 +1314,6 @@ Visualize the number of log messages over time, split by the log level. Since th ![Date Histogram of the log rate per log level](/solutions/images/observability-monitor-java-app-metrics-kibana-create-visualization-log-rate.png "") - - ### Visualize open files over time [_visualize_open_files_over_time] The second visualization is to be a check for the number of open files in our application. @@ -1418,7 +1378,7 @@ Let’s look at the `process_files_open_files` metric. This should be a rather s Results show that only twenty requests were sent, which makes sense given the processing time. - Now let’s build a visualization using [Lens](../../../explore-analyze/dashboards.md) in {{kib}}. + Now let’s build a visualization using [Lens](/explore-analyze/dashboards.md) in {{kib}}. ![Lens visualization](/solutions/images/observability-monitor-java-app-metrics-kibana-create-visualization-open-files.png "") @@ -1436,8 +1396,6 @@ Let’s look at the `process_files_open_files` metric. This should be a rather s This is an area chart of the total events counter the Javalin app emits. It’s rising because there is a component polling an endpoint that, in turn, produces another log message. The steeper peek was due to sending more requests. But where is the sudden drop-off coming from? A JVM restart. As those metrics are not persisted, they reset on a JVM restart. With that in mind, it’s often better to log the `rate` instead of the `counter` field. - - ## Step 7: Instrument the application [_step_7_instrument_the_application] The third piece of {{observability}} is Application Performance Management (APM). An APM setup consists of an APM server which accepts the data (and is already running within our {{ecloud}} setup) and an agent delivering the data to the server. @@ -1448,7 +1406,6 @@ One of the APM’s core ideas is the ability to follow the flow of a user sessio To fully capture user activity, you need to start in the browser of the user using Real User Monitoring (RUM) down to your application, which sends a SQL query to your database. - ### Data Model [_data_model] Despite a heavily fragmented APM landscape, the terminology often is similar. The two most important terms are **Spans** and **Transactions**. @@ -1463,7 +1420,6 @@ The Java {{apm-agent}} can instrument specific frameworks automatically. Spring There are currently agents for Go, .NET, Node, Python, Ruby, and the browser (RUM). Agents keep getting added so you may want to check the [APM agent documentation](https://www.elastic.co/guide/en/apm/agent/index.html). - ### Add the {{apm-agent}} to your code [_add_the_apm_agent_to_your_code] You have two options to add Java agent instrumentation to your application. @@ -1491,7 +1447,6 @@ java -javaagent:/path/to/elastic-apm-agent-1.17.0.jar\ You could now go ahead and open up the Applications UI and you should see the data flowing in. - ### Automatic attachment [_automatic_attachment] If you do not want to change the startup options of your application, the standalone agent allows you to attach to running JVMs on a host. @@ -1525,7 +1480,6 @@ So now the agent was attached to a running application with a special configurat While both of the first two possibilities work, you can also use the third one: using the {{apm-agent}} as a direct dependency. This allows you to write custom spans and transactions within our application. - ### Programmatic setup [_programmatic_setup] A programmatic setup allows you to attach the agent via a line of java in your source code. @@ -1589,8 +1543,6 @@ A programmatic setup allows you to attach the agent via a line of java in your s The only transaction comes from a single servlet, which is not too helpful. Let’s try to fix this by introducing custom programmatic transactions. - - ### Custom transactions [_custom_transactions] 1. Add another dependency. @@ -1660,8 +1612,6 @@ A programmatic setup allows you to attach the agent via a line of java in your s + " " + ctx.endpointHandlerPath())); ``` - - ### Method tracing via agent configuration [_method_tracing_via_agent_configuration] Instead of writing code to trace methods, you can also configure the agent to do this. Let’s try to figure out if logging is a bottleneck for our application and trace the request logger statements we added earlier. @@ -1713,8 +1663,6 @@ ELASTIC_APM_TRACE_METHODS="de.spinscale.javalin.Log4j2RequestLogger#handle" If you are on the quest for a faster service, you may want to rethink logging. However, this logging happens after the result is written to the client, so while the total processing time increases with logging, responding to the client does not (closing the connection however might be). Also note that these tests were conducted without a proper warm-up. I assume that after appropriate JVM warm-up, you will have much faster processing of requests. - - ### Automatic profiling of inferred spans [_automatic_profiling_of_inferred_spans] Once you have a bigger application with more code paths than our sample app, you can try to enable the [automatic profiling of inferred spans](apm-agent-java://reference/config-profiling.md#config-profiling-inferred-spans-enabled) by setting the following. @@ -1725,7 +1673,6 @@ ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED=true This mechanism uses the [async profiler](https://github.com/jvm-profiling-tools/async-profiler) to create spans without you having to instrument anything allowing you to find bottlenecks faster. - ### Log correlation [_log_correlation] Transaction ids are automatically added to logs. You can check the generated log files that are sent to {{es}} via {{filebeat}}. An entry looks like this. @@ -1751,8 +1698,6 @@ We have not covered the [Elastic APM OpenTracing bridge](apm-agent-java://refere :::: - - ## Step 8: Ingest Uptime data [_step_8_ingest_uptime_data] ```{applies_to} @@ -1765,7 +1710,6 @@ How about checking if our users have the same experience that our APM data is su [Uptime](https://www.elastic.co/uptime-monitoring) not only enables you to monitor the availability of a service, but also graph latencies over time, and get notified about expiring TLS certificates. - ### Setup [_setup] To send uptime data to {{es}}, {{heartbeat}} (the polling component) is required. To download and install {{heartbeat}}, use the commands that work with your system: @@ -1902,7 +1846,6 @@ sudo service heartbeat start If you use an `init.d` script to start Heartbeat, you can’t specify command line flags (see [Command reference](https://www.elastic.co/guide/en/beats/heartbeat/master/command-line-options.html)). To specify flags, start Heartbeat in the foreground. :::: - Also see [Heartbeat and systemd](https://www.elastic.co/guide/en/beats/heartbeat/master/running-with-systemd.html). :::::: @@ -1915,7 +1858,6 @@ sudo service heartbeat start If you use an `init.d` script to start Heartbeat, you can’t specify command line flags (see [Command reference](https://www.elastic.co/guide/en/beats/heartbeat/master/command-line-options.html)). To specify flags, start Heartbeat in the foreground. :::: - Also see [Heartbeat and systemd](https://www.elastic.co/guide/en/beats/heartbeat/master/running-with-systemd.html). :::::: @@ -1960,7 +1902,6 @@ The duration of the monitor is in the low milliseconds, as it is really fast. Ch Do not underestimate the importance of this kind of monitoring. Also, consider this just the beginning as the next step is to have synthetics that monitor the correct behavior of your application, for example, to ensure that your checkout process works all the time. - ## What’s next? [_whats_next] For more information about using Elastic {{observability}}, see the [{{observability}} documentation](../get-started/what-is-elastic-observability.md). diff --git a/solutions/observability/apps/upgrade-elastic-cloud-apm-server-standalone-to-90.md b/solutions/observability/apps/upgrade-elastic-cloud-apm-server-standalone-to-90.md index f26ced3e9b..98582c5b89 100644 --- a/solutions/observability/apps/upgrade-elastic-cloud-apm-server-standalone-to-90.md +++ b/solutions/observability/apps/upgrade-elastic-cloud-apm-server-standalone-to-90.md @@ -2,13 +2,12 @@ navigation_title: "{{ecloud}} standalone" mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-upgrade-8.0-cloud-standalone.html +applies_to: + stack: --- - - # Upgrade Elastic Cloud APM Server standalone to 8. [apm-upgrade-8.0-cloud-standalone] - This upgrade guide is for the standalone method of running APM Server. Only use this guide if both of the following are true: * You’re using {{ecloud}}. @@ -18,6 +17,6 @@ Follow these steps to upgrade: 1. Review the [Elastic APM release notes](apm-server://release-notes/index.md) and [Elastic {{observability}} release notes](/release-notes/elastic-observability/index.md). 2. Review the [Elastic APM breaking changes](apm-server://release-notes/breaking-changes.md). -3. Upgrade {{ecloud}} to 9.0, refer to [Upgrade your deployment](../../../deploy-manage/upgrade/deployment-or-cluster.md) for instructions. +3. Upgrade {{ecloud}} to 9.0, refer to [Upgrade your deployment](/deploy-manage/upgrade/deployment-or-cluster.md) for instructions. 4. (Optional) Upgrade to the APM integration. Got time for one more upgrade? Refer to [Switch to the Elastic APM integration](switch-to-elastic-apm-integration.md). diff --git a/solutions/observability/apps/upgrade-elastic-cloud-with-apm-integration-to-90.md b/solutions/observability/apps/upgrade-elastic-cloud-with-apm-integration-to-90.md index 6d691df7f9..f96ab7000f 100644 --- a/solutions/observability/apps/upgrade-elastic-cloud-with-apm-integration-to-90.md +++ b/solutions/observability/apps/upgrade-elastic-cloud-with-apm-integration-to-90.md @@ -2,13 +2,12 @@ navigation_title: "{{ecloud}} APM integration" mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-upgrade-8.0-cloud-integration.html +applies_to: + stack: --- - - # Upgrade Elastic Cloud with the APM integration to 8.0 [apm-upgrade-8.0-cloud-integration] - This upgrade guide is for the Elastic APM integration. Only use this guide if both of the following are true: * You’re using {{ecloud}}. @@ -18,10 +17,9 @@ Follow these steps to upgrade: 1. Review the [Elastic APM release notes](apm-server://release-notes/index.md) and [Elastic {{observability}} release notes](/release-notes/elastic-observability/index.md). 2. Review the [Elastic APM breaking changes](apm-server://release-notes/breaking-changes.md). -3. Upgrade your {{ecloud}} instance to 9.0. Refer to [Upgrade your deployment](../../../deploy-manage/upgrade/deployment-or-cluster.md) for details. As a part of this process, the APM integration will automatically upgrade to version 9.0. +3. Upgrade your {{ecloud}} instance to 9.0. Refer to [Upgrade your deployment](/deploy-manage/upgrade/deployment-or-cluster.md) for details. As a part of this process, the APM integration will automatically upgrade to version 9.0. ::::{note} {{ece}} users require additional TLS setup. Refer to [Add APM user settings](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-manage-apm-settings.html) for more information. :::: - diff --git a/solutions/observability/apps/upgrade-self-installation-of-apm-integration-to-90.md b/solutions/observability/apps/upgrade-self-installation-of-apm-integration-to-90.md index 27bcf9202d..f916ed22ba 100644 --- a/solutions/observability/apps/upgrade-self-installation-of-apm-integration-to-90.md +++ b/solutions/observability/apps/upgrade-self-installation-of-apm-integration-to-90.md @@ -2,31 +2,28 @@ navigation_title: "Self-installation APM integration" mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-upgrade-8.0-self-integration.html +applies_to: + stack: --- - - # Upgrade a self-installation of the APM integration to 9.0 [apm-upgrade-9.0-self-integration] - This upgrade guide is for the Elastic APM integration. Only use this guide if both of the following are true: * You have a self-installation of the {{stack}}, i.e. you’re not using {{ecloud}}. * You have already switched to and are running {{fleet}} and the Elastic APM integration. - ## Prerequisites [_prerequisites_9] 1. Before you upgrade to version 9.0, you must upgrade {{es}} and {{kib}} to the latest patch version of 8.18. For more details, refer to the [{{stack}} Installation and Upgrade Guide](https://www.elastic.co/guide/en/elastic-stack/8.18/upgrading-elastic-stack.html). 2. Review the [Elastic APM release notes](apm-server://release-notes/index.md) and [Elastic {{observability}} release notes](/release-notes/elastic-observability/index.md). 3. Review the [Elastic APM breaking changes](apm-server://release-notes/breaking-changes.md). - ## Upgrade steps [_upgrade_steps_2] 1. Upgrade the {{stack}} to version 9.0. - The {{stack}} ({{es}} and {{kib}}) must be upgraded before {{agent}}. refer to the [{{stack}} Installation and Upgrade Guide](../../../deploy-manage/upgrade/deployment-or-cluster.md) for guidance. + The {{stack}} ({{es}} and {{kib}}) must be upgraded before {{agent}}. refer to the [{{stack}} Installation and Upgrade Guide](/deploy-manage/upgrade/deployment-or-cluster.md) for guidance. 2. Upgrade {{agent}} to version 9.0 As a part of this process, the APM integration will automatically upgrade to version 9.0. diff --git a/solutions/observability/apps/upgrade-self-installation-of-apm-server-standalone-to-90.md b/solutions/observability/apps/upgrade-self-installation-of-apm-server-standalone-to-90.md index 0d4bdd8484..d11931f57d 100644 --- a/solutions/observability/apps/upgrade-self-installation-of-apm-server-standalone-to-90.md +++ b/solutions/observability/apps/upgrade-self-installation-of-apm-server-standalone-to-90.md @@ -2,19 +2,17 @@ navigation_title: "Self-installation standalone" mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-upgrade-8.0-self-standalone.html +applies_to: + stack: --- - - # Upgrade a self-installation of APM Server standalone to 9.0 [apm-upgrade-9.0-self-standalone] - This upgrade guide is for the standalone method of running APM Server. Only use this guide if both of the following are true: * You have a self-installation of the {{stack}}, i.e. you’re not using {{ecloud}}. * You’re running the APM Server binary, i.e. you haven’t switched to the Elastic APM integration. - ## Prerequisites [_prerequisites_8] 1. Prior to upgrading to version 9.0, {{es}}, {{kib}}, and APM Server must be upgraded to version 8.18. @@ -25,13 +23,11 @@ This upgrade guide is for the standalone method of running APM Server. Only use 2. Review the [Elastic APM release notes](apm-server://release-notes/index.md) and [Elastic {{observability}} release notes](/release-notes/elastic-observability/index.md). 3. Review the [Elastic APM breaking changes](apm-server://release-notes/breaking-changes.md). - - ## Upgrade steps [_upgrade_steps] 1. **Upgrade the {{stack}} to version 9.0** - The {{stack}} ({{es}} and {{kib}}) must be upgraded before APM Server. Refer to the [{{stack}} Installation and Upgrade Guide](../../../deploy-manage/upgrade/deployment-or-cluster.md) for guidance. + The {{stack}} ({{es}} and {{kib}}) must be upgraded before APM Server. Refer to the [{{stack}} Installation and Upgrade Guide](/deploy-manage/upgrade/deployment-or-cluster.md) for guidance. 2. **Install the 9.0 APM Server release** diff --git a/solutions/observability/apps/upgrade-to-version-90.md b/solutions/observability/apps/upgrade-to-version-90.md index 97c05e30e9..ef73c5f313 100644 --- a/solutions/observability/apps/upgrade-to-version-90.md +++ b/solutions/observability/apps/upgrade-to-version-90.md @@ -1,6 +1,8 @@ --- mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-upgrading-to-8.x.html +applies_to: + stack: --- # Upgrade to version 9.0 [apm-upgrading-to-9.0] @@ -11,7 +13,6 @@ This guide explains the upgrade process for version 9.0. For a detailed look at * [What’s new in {{kib}}](https://www.elastic.co/guide/en/kibana/current/whats-new.html) * [{{es}} release highlights](https://www.elastic.co/guide/en/elasticsearch/reference/current/release-highlights.html) - ## Notable APM changes [_notable_apm_changes] * All index management has been removed from APM Server; the built-in {{es}} apm-data plugin is entirely responsible for setting up index templates, index lifecycle polices, and index pipelines. @@ -20,7 +21,6 @@ This guide explains the upgrade process for version 9.0. For a detailed look at As a result of the above changes, a number of index management and index tuning configuration variables have been removed. See the APM [release notes](apm-server://release-notes/index.md), [breaking changes](https://www.elastic.co/guide/en/observability/current/apm-breaking.html) for full details. - ## Find your upgrade guide [_find_your_upgrade_guide] Starting in version 7.14, there are two ways to run Elastic APM. Determine which method you’re using, then use the links below to find the correct upgrading guide. diff --git a/solutions/observability/apps/upgrade.md b/solutions/observability/apps/upgrade.md index 0fb85bd7f0..ea75c5625f 100644 --- a/solutions/observability/apps/upgrade.md +++ b/solutions/observability/apps/upgrade.md @@ -1,6 +1,8 @@ --- mapped_pages: - https://www.elastic.co/guide/en/observability/current/apm-upgrade.html +applies_to: + stack: --- # Upgrade [apm-upgrade] diff --git a/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md b/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md index 91bd4a0a9f..7bdd08fd8b 100644 --- a/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md +++ b/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md @@ -9,17 +9,17 @@ applies_to: # Upstream OpenTelemetry Collectors and language SDKs [apm-open-telemetry-direct] -::::{note} -This is one of several approaches you can use to integrate Elastic with OpenTelemetry. **To compare approaches and choose the best approach for your use case, refer to [OpenTelemetry](../../../solutions/observability/apps/use-opentelemetry-with-apm.md).** +:::{include} _snippets/apm-server-vs-mis.md +::: +::::{note} +This is one of several approaches you can use to integrate Elastic with OpenTelemetry. **To compare approaches and choose the best approach for your use case, refer to [OpenTelemetry](/solutions/observability/apps/use-opentelemetry-with-apm.md).** :::: - The {{stack}} natively supports the OpenTelemetry protocol (OTLP). This means trace data and metrics collected from your applications and infrastructure can be sent directly to the {{stack}}. -* Send data to Elastic from an upstream [OpenTelemetry Collector](../../../solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md#apm-connect-open-telemetry-collector) -* Send data to Elastic from an upstream [OpenTelemetry language SDK](../../../solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md#apm-instrument-apps-otel) - +* Send data to Elastic from an upstream [OpenTelemetry Collector](/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md#apm-connect-open-telemetry-collector) +* Send data to Elastic from an upstream [OpenTelemetry language SDK](/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md#apm-instrument-apps-otel) ## Send data from an upstream OpenTelemetry Collector [apm-connect-open-telemetry-collector] @@ -31,7 +31,6 @@ Connect your OpenTelemetry Collector instances to Elastic {{observability}} or { :::{tab-item} Elastic Stack :sync: stack - ```yaml receivers: <1> # ... @@ -79,7 +78,7 @@ service: 3. The [debug exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/debugexporter) is helpful for troubleshooting, and supports configurable verbosity levels: `basic` (default), `normal`, and `detailed`. 4. Elastic {{observability}} endpoint configuration. APM Server supports a ProtoBuf payload via both the OTLP protocol over gRPC transport [(OTLP/gRPC)](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc) and the OTLP protocol over HTTP transport [(OTLP/HTTP)](https://opentelemetry.io/docs/specs/otlp/#otlphttp). To learn more about these exporters, see the OpenTelemetry Collector documentation: [OTLP/HTTP Exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter) or [OTLP/gRPC exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter). When adding an endpoint to an existing configuration an optional name component can be added, like `otlp/elastic`, to distinguish endpoints as described in the [OpenTelemetry Collector Configuration Basics](https://opentelemetry.io/docs/collector/configuration/#basics). 5. Hostname and port of the APM Server endpoint. For example, `elastic-apm-server:8200`. -6. Credential for Elastic APM [secret token authorization](../../../solutions/observability/apps/secret-token.md) (`Authorization: "Bearer a_secret_token"`) or [API key authorization](../../../solutions/observability/apps/api-keys.md) (`Authorization: "ApiKey an_api_key"`). +6. Credential for Elastic APM [secret token authorization](/solutions/observability/apps/secret-token.md) (`Authorization: "Bearer a_secret_token"`) or [API key authorization](/solutions/observability/apps/api-keys.md) (`Authorization: "ApiKey an_api_key"`). 7. Environment-specific configuration parameters can be conveniently passed in as environment variables documented [here](https://opentelemetry.io/docs/collector/configuration/#environment-variables) (e.g. `ELASTIC_APM_SERVER_ENDPOINT` and `ELASTIC_APM_SECRET_TOKEN`). 8. [preview] To send OpenTelemetry logs to {{stack}} version 8.0+, declare a `logs` pipeline. @@ -139,14 +138,12 @@ service: :::: - You’re now ready to export traces and metrics from your services and applications. ::::{tip} When using the OpenTelemetry Collector, you should always prefer sending data via the [`OTLP` exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter). Using other methods, like the [`elasticsearch` exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter), will bypass all of the validation and data processing that Elastic performs. In addition, your data will not be viewable in your Observability project if you use the `elasticsearch` exporter. :::: - ## Send data from an upstream OpenTelemetry SDK [apm-instrument-apps-otel] ::::{note} @@ -154,7 +151,6 @@ This document outlines how to send data directly from an upstream OpenTelemetry :::: - To export traces and metrics to Elastic, instrument your services and applications with the OpenTelemetry API, SDK, or both. For example, if you are a Java developer, you need to instrument your Java app with the [OpenTelemetry agent for Java](https://github.com/open-telemetry/opentelemetry-java-instrumentation). See the [OpenTelemetry Instrumentation guides](https://opentelemetry.io/docs/instrumentation/) to download the OpenTelemetry agent or SDK for your language. Define environment variables to configure the OpenTelemetry agent or SDK and enable communication with Elastic APM. For example, if you are instrumenting a Java app, define the following environment variables: @@ -179,7 +175,7 @@ java -javaagent:/path/to/opentelemetry-javaagent-all.jar \ 1. [preview] The OpenTelemetry logs intake via APM Server is currently in technical preview. `OTEL_RESOURCE_ATTRIBUTES` -: Fields that describe the service and the environment that the service runs in. See [resource attributes](../../../solutions/observability/apps/resource-atrributes.md) for more information. +: Fields that describe the service and the environment that the service runs in. See [resource attributes](/solutions/observability/apps/resource-atrributes.md) for more information. `OTEL_EXPORTER_OTLP_ENDPOINT` : APM Server URL. The host and port that APM Server listens for events on. @@ -187,7 +183,7 @@ java -javaagent:/path/to/opentelemetry-javaagent-all.jar \ `OTEL_EXPORTER_OTLP_HEADERS` : Authorization header that includes the Elastic APM Secret token or API key: `"Authorization=Bearer an_apm_secret_token"` or `"Authorization=ApiKey an_api_key"`. - For information on how to format an API key, see [API keys](../../../solutions/observability/apps/api-keys.md). + For information on how to format an API key, see [API keys](/solutions/observability/apps/api-keys.md). Please note the required space between `Bearer` and `an_apm_secret_token`, and `ApiKey` and `an_api_key`. @@ -195,7 +191,6 @@ java -javaagent:/path/to/opentelemetry-javaagent-all.jar \ If you are using a version of the Python OpenTelemetry agent *before* 1.27.0, the content of the header *must* be URL-encoded. You can use the Python standard library’s `urllib.parse.quote` function to encode the content of the header. :::: - `OTEL_METRICS_EXPORTER` : Metrics exporter to use. See [exporter selection](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection) for more information. @@ -221,7 +216,7 @@ java -javaagent:/path/to/opentelemetry-javaagent-all.jar \ 1. [preview] The OpenTelemetry logs intake via Elastic is currently in technical preview. `OTEL_RESOURCE_ATTRIBUTES` -: Fields that describe the service and the environment that the service runs in. See [resource attributes](../../../solutions/observability/apps/resource-atrributes.md) for more information. +: Fields that describe the service and the environment that the service runs in. See [resource attributes](/solutions/observability/apps/resource-atrributes.md) for more information. `OTEL_EXPORTER_OTLP_ENDPOINT` : Elastic URL. The host and port that Elastic listens for APM events on. @@ -229,28 +224,24 @@ java -javaagent:/path/to/opentelemetry-javaagent-all.jar \ `OTEL_EXPORTER_OTLP_HEADERS` : Authorization header that includes the Elastic APM API key: `"Authorization=ApiKey an_api_key"`. Note the required space between `ApiKey` and `an_api_key`. - For information on how to format an API key, refer to [Secure communication with APM agents](../../../solutions/observability/apps/use-apm-securely.md). + For information on how to format an API key, refer to [Secure communication with APM agents](/solutions/observability/apps/use-apm-securely.md). ::::{note} If you are using a version of the Python OpenTelemetry agent *before* 1.27.0, the content of the header *must* be URL-encoded. You can use the Python standard library’s `urllib.parse.quote` function to encode the content of the header. :::: - `OTEL_METRICS_EXPORTER` : Metrics exporter to use. See [exporter selection](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection) for more information. `OTEL_LOGS_EXPORTER` : Logs exporter to use. See [exporter selection](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection) for more information. - ::: :::: - -You are now ready to collect traces and [metrics](../../../solutions/observability/apps/collect-metrics.md) before [verifying metrics](../../../solutions/observability/apps/collect-metrics.md#apm-open-telemetry-verify-metrics) and [visualizing metrics](../../../solutions/observability/apps/collect-metrics.md#apm-open-telemetry-visualize). - +You are now ready to collect traces and [metrics](/solutions/observability/apps/collect-metrics.md) before [verifying metrics](/solutions/observability/apps/collect-metrics.md#apm-open-telemetry-verify-metrics) and [visualizing metrics](/solutions/observability/apps/collect-metrics.md#apm-open-telemetry-visualize). ## Proxy requests to APM Server [apm-open-telemetry-proxy-apm] @@ -271,9 +262,8 @@ For more information on how to configure an AWS ALB to support gRPC, see this AW For more information on how APM Server services gRPC requests, see [Muxing gRPC and HTTP/1.1](https://github.com/elastic/apm-server/blob/main/dev_docs/otel.md#muxing-grpc-and-http11). - ## Next steps [apm-open-telemetry-direct-next] -* [Collect metrics](../../../solutions/observability/apps/collect-metrics.md) -* Add [Resource attributes](../../../solutions/observability/apps/resource-atrributes.md) -* Learn about the [limitations of this integration](../../../solutions/observability/apps/limitations.md) +* [Collect metrics](/solutions/observability/apps/collect-metrics.md) +* Add [Resource attributes](/solutions/observability/apps/resource-atrributes.md) +* Learn about the [limitations of this integration](/solutions/observability/apps/limitations.md) diff --git a/solutions/observability/apps/uptime-monitoring-deprecated.md b/solutions/observability/apps/uptime-monitoring-deprecated.md index 7997d7819c..7b5ccc8001 100644 --- a/solutions/observability/apps/uptime-monitoring-deprecated.md +++ b/solutions/observability/apps/uptime-monitoring-deprecated.md @@ -14,7 +14,6 @@ applies_to: Use [Synthetic monitoring](/solutions/observability/apps/synthetic-monitoring.md) instead of the {{uptime-app}}. :::: - ::::{important} The {{uptime-app}} is for viewing result data from lightweight monitors running through {{heartbeat}} and [configured with a traditional `heartbeat.yml` file](get-started-with-uptime.md). This is for TCP, HTTP or ICMP monitors that you have configured and run from your own infrastructure with {{heartbeat}} natively. @@ -24,10 +23,8 @@ Note that the {{uptime-app}} is hidden from the interface when there is no recen :::: - The {{uptime-app}} uses {{agent}} to periodically check the status of your services and applications. Monitor the availability of network endpoints and services using [Lightweight HTTP/S, TCP, and ICMP monitors](synthetic-monitoring.md#monitoring-uptime). - ## Lightweight HTTP/S, TCP, and ICMP monitors [uptime-lightweight] In the {{uptime-app}}, you can monitor the status of network endpoints using the following lightweight checks: @@ -45,7 +42,6 @@ In the {{uptime-app}}, you can monitor the status of network endpoints using the To set up your first monitor, refer to [Get started with Uptime](get-started-with-uptime.md). - ## TLS Certificates [view-certificate-status] The TLS Certificates page in the {{uptime-app}} lists the TLS certificates that are being monitored and shows the TLS certificate data in your indices. diff --git a/solutions/observability/apps/use-advanced-queries-on-application-data.md b/solutions/observability/apps/use-advanced-queries-on-application-data.md index c4820bb1bc..076b864a6c 100644 --- a/solutions/observability/apps/use-advanced-queries-on-application-data.md +++ b/solutions/observability/apps/use-advanced-queries-on-application-data.md @@ -10,7 +10,6 @@ applies_to: # Use advanced queries on your application data [apm-advanced-queries] - Querying your APM data is an essential tool that can make finding bottlenecks in your code even more straightforward. Using the query bar, a powerful data query feature, you can pass advanced queries on your data to filter on specific pieces of information you’re interested in. APM queries entered into the query bar are added as parameters to the URL, so it’s easy to share a specific query or view with others. @@ -25,14 +24,13 @@ When you type, you can begin to see some of the transaction fields available for ::: ::::{tip} -To learn more about the {{kib}} query language capabilities, see the [Kibana Query Language Enhancements](../../../explore-analyze/query-filter/languages/kql.md) documentation. +To learn more about the {{kib}} query language capabilities, see the [Kibana Query Language Enhancements](/explore-analyze/query-filter/languages/kql.md) documentation. :::: - ### APM queries [apm-app-queries] -APM queries can be handy for removing noise from your data in the [Services](../../../solutions/observability/apps/services.md), [Transactions](../../../solutions/observability/apps/transactions-2.md), [Errors](../../../solutions/observability/apps/errors-2.md), [Metrics](../../../solutions/observability/apps/metrics-2.md), and [Traces](../../../solutions/observability/apps/traces-2.md) views. +APM queries can be handy for removing noise from your data in the [Services](/solutions/observability/apps/services.md), [Transactions](/solutions/observability/apps/transactions-2.md), [Errors](/solutions/observability/apps/errors-2.md), [Metrics](/solutions/observability/apps/metrics-2.md), and [Traces](/solutions/observability/apps/traces-2.md) views. For example, in the **Services** view, you can quickly view a list of all the instrumented services running on your production environment: `service.environment : production`. Or filter the list by including the APM agent’s name and the host it’s running on: `service.environment : "production" and agent.name : "java" and host.name : "prod-server1"`. @@ -40,11 +38,9 @@ On the **Traces** view, you might want to view failed transaction results from a On the **Transactions** view, you may want to list only the slower transactions than a specified time threshold: `transaction.duration.us > 2000000`. Or filter the list by including the service version and the Kubernetes pod it’s running on: `transaction.duration.us > 2000000 and service.version : "7.12.0" and kubernetes.pod.name : "pod-5468b47f57-pqk2m"`. - ## Querying in Discover [discover-advanced-queries] -Alternatively, you can query your APM documents in [**Discover**](../../../explore-analyze/discover.md). Querying documents in **Discover** works the same way as queries in the Applications UI, and **Discover** supports all of the example APM queries shown on this page. - +Alternatively, you can query your APM documents in [**Discover**](/explore-analyze/discover.md). Querying documents in **Discover** works the same way as queries in the Applications UI, and **Discover** supports all of the example APM queries shown on this page. ### Discover queries [discover-queries] diff --git a/solutions/observability/apps/use-apm-securely.md b/solutions/observability/apps/use-apm-securely.md index 9d4b158976..0b01917d4e 100644 --- a/solutions/observability/apps/use-apm-securely.md +++ b/solutions/observability/apps/use-apm-securely.md @@ -9,11 +9,14 @@ applies_to: # Use APM securely [apm-securing-apm-server] +:::{include} _snippets/apm-server-vs-mis.md +::: + When setting up Elastic APM, it’s critical to ensure that application data is secure from start to finish. You should approach securing your application data from different perspectives: | | | | --- | --- | -| **What kind of data is collected?** | Ensure that data doesn’t contain sensitive information like passwords, credit card numbers, health data, or other identifiable information.
Read more in [Secure data](../../../solutions/observability/apps/application-data-security.md). | -| **How do APM agents and {{agent}} communicate?** | Ensure that any communication between APM agents and {{agent}} are both encrypted and authenticated.
Read more in [Secure communication with APM agents](../../../solutions/observability/apps/secure-communication-with-apm-agents.md). | -| **How do APM Server and the {{stack}} communicate?** | Use role-based access control to grant APM Server users access to secured resources. The roles that you set up depend on your organization’s security requirements and the minimum privileges required to use specific features.
Read more in [Secure communication with the {{stack}}](../../../solutions/observability/apps/secure-communication-with-elastic-stack.md). | -| **Who can use the Applications UI?** | Use role-based access control to grant users access to features of the Applications UI.
Read more in [Secure access to the Applications UI](../../../solutions/observability/apps/secure-access-to-applications-ui.md). | \ No newline at end of file +| **What kind of data is collected?** | Ensure that data doesn’t contain sensitive information like passwords, credit card numbers, health data, or other identifiable information.
Read more in [Secure data](/solutions/observability/apps/application-data-security.md). | +| **How do APM agents and {{agent}} communicate?** | Ensure that any communication between APM agents and {{agent}} are both encrypted and authenticated.
Read more in [Secure communication with APM agents](/solutions/observability/apps/secure-communication-with-apm-agents.md). | +| **How do APM Server and the {{stack}} communicate?** | Use role-based access control to grant APM Server users access to secured resources. The roles that you set up depend on your organization’s security requirements and the minimum privileges required to use specific features.
Read more in [Secure communication with the {{stack}}](/solutions/observability/apps/secure-communication-with-elastic-stack.md). | +| **Who can use the Applications UI?** | Use role-based access control to grant users access to features of the Applications UI.
Read more in [Secure access to the Applications UI](/solutions/observability/apps/secure-access-to-applications-ui.md). | \ No newline at end of file diff --git a/solutions/observability/apps/use-environment-variables-in-configuration.md b/solutions/observability/apps/use-environment-variables-in-configuration.md index 8e5ba8920f..a48b70b619 100644 --- a/solutions/observability/apps/use-environment-variables-in-configuration.md +++ b/solutions/observability/apps/use-environment-variables-in-configuration.md @@ -14,7 +14,6 @@ This documentation is only relevant for APM Server binary users. :::: - You can use environment variable references in the config file to set values that need to be configurable during deployment. To do this, use: `${VAR}` @@ -46,8 +45,6 @@ You can also specify environment variables when you override a config setting fr :::: - - ## Examples [_examples] Here are some examples of configurations that use environment variables and what each configuration looks like after replacement: @@ -61,7 +58,6 @@ Here are some examples of configurations that use environment variables and what | `name: ${NAME:?You need to set the NAME environment variable}` | no setting | None. Returns an error message that’s prepended with the custom text. | | `name: ${NAME:?You need to set the NAME environment variable}` | `export NAME=elastic` | `name: elastic` | - ## Specify complex objects in environment variables [_specify_complex_objects_in_environment_variables] You can specify complex objects, such as lists or dictionaries, in environment variables by using a JSON-like syntax. @@ -83,7 +79,6 @@ output.elasticsearch: When APM Server loads the config file, it resolves the environment variable and replaces it with the specified list before reading the `hosts` setting. - ### Opt-out of complex object parsing [_opt_out_of_complex_object_parsing] To opt-out of this complex object parsing behavior such that the environment variable is always parsed as a string, wrap the environment variable in double-quotes (`"`), and escape all double quotes within them. @@ -105,4 +100,3 @@ ES_PASSWORD='"foo,bar\"baz"' Do not use double-quotes (`"`) to wrap regular expressions. Otherwise, the backslash (`\`) will be interpreted as an escape character. :::: - diff --git a/solutions/observability/apps/use-internal-collection-to-send-monitoring-data.md b/solutions/observability/apps/use-internal-collection-to-send-monitoring-data.md index c95fda3d5c..f58731a699 100644 --- a/solutions/observability/apps/use-internal-collection-to-send-monitoring-data.md +++ b/solutions/observability/apps/use-internal-collection-to-send-monitoring-data.md @@ -8,7 +8,6 @@ applies_to: # Use internal collection to send monitoring data [apm-monitoring-internal-collection] - Use internal collectors to send {{beats}} monitoring data directly to your monitoring cluster. Or as an alternative to internal collection, use [Use {{metricbeat}} collection](use-metricbeat-to-send-monitoring-data.md). The benefit of using internal collection instead of {{metricbeat}} is that you have fewer pieces of software to install and maintain. 1. Create an API key or user that has appropriate authority to send system-level monitoring data to {{es}}. For example, you can use the built-in `apm_system` user or assign the built-in `apm_system` role to another user. For more information on the required privileges, see [Create a *monitoring* role](create-assign-feature-roles-to-apm-server-users.md#apm-privileges-to-publish-monitoring). For more information on how to use API keys, see [Grant access using API keys](grant-access-using-api-keys.md). @@ -25,7 +24,6 @@ Use internal collectors to send {{beats}} monitoring data directly to your monit 1. Specify one of `api_key` or `username`/`password`. - If you want to send monitoring events to an [{{ecloud}}](https://cloud.elastic.co/) monitoring cluster, you can use two simpler settings. When defined, these settings overwrite settings from other parts in the configuration. For example: ```yaml @@ -52,7 +50,6 @@ Use internal collectors to send {{beats}} monitoring data directly to your monit 2. This setting identifies the hosts and port numbers of {{es}} nodes that are part of the monitoring cluster. 3. Specify one of `api_key` or `username`/`password`. - If you want to use PKI authentication to send monitoring events to {{es}}, you must specify a different set of configuration options. For example: ```yaml @@ -70,8 +67,7 @@ Use internal collectors to send {{beats}} monitoring data directly to your monit You must specify the `username` as `""` explicitly so that the username from the client certificate (`CN`) is used. See [SSL/TLS output settings](ssltls-output-settings.md) for more information about SSL settings. 3. Start APM Server. -4. [View the monitoring data in {{kib}}](../../../deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md). - +4. [View the monitoring data in {{kib}}](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md). ## Settings for internal collection [apm-configuration-monitor] @@ -79,99 +75,80 @@ Use the following settings to configure internal collection when you are not usi You specify these settings in the X-Pack monitoring section of the `apm-server.yml` config file: - ### `monitoring.enabled` [_monitoring_enabled] The `monitoring.enabled` config is a boolean setting to enable or disable {{monitoring}}. If set to `true`, monitoring is enabled. The default value is `false`. - ### `monitoring.elasticsearch` [_monitoring_elasticsearch] The {{es}} instances that you want to ship your APM Server metrics to. This configuration option contains the following fields: - #### `api_key` [_api_key_3] The detail of the API key to be used to send monitoring information to {{es}}. See [Grant access using API keys](grant-access-using-api-keys.md) for more information. - #### `bulk_max_size` [_bulk_max_size_5] The maximum number of metrics to bulk in a single {{es}} bulk API index request. The default is `50`. For more information, see [{{es}}](configure-elasticsearch-output.md). - #### `backoff.init` [_backoff_init_5] The number of seconds to wait before trying to reconnect to {{es}} after a network error. After waiting `backoff.init` seconds, APM Server tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to `backoff.max`. After a successful connection, the backoff timer is reset. The default is `1s`. - #### `backoff.max` [_backoff_max_5] The maximum number of seconds to wait before attempting to connect to {{es}} after a network error. The default is `60s`. - #### `compression_level` [_compression_level_4] The gzip compression level. Setting this value to `0` disables compression. The compression level must be in the range of `1` (best speed) to `9` (best compression). The default value is `0`. Increasing the compression level reduces the network usage but increases the CPU usage. - #### `headers` [_headers_2] Custom HTTP headers to add to each request. For more information, see [{{es}}](configure-elasticsearch-output.md). - #### `hosts` [_hosts_4] The list of {{es}} nodes to connect to. Monitoring metrics are distributed to these nodes in round robin order. For more information, see [{{es}}](configure-elasticsearch-output.md). - #### `max_retries` [_max_retries_5] The number of times to retry sending the monitoring metrics after a failure. After the specified number of retries, the metrics are typically dropped. The default value is `3`. For more information, see [{{es}}](configure-elasticsearch-output.md). - #### `parameters` [_parameters_2] Dictionary of HTTP parameters to pass within the URL with index operations. - #### `password` [_password_4] The password that APM Server uses to authenticate with the {{es}} instances for shipping monitoring data. - #### `metrics.period` [_metrics_period] The time interval (in seconds) when metrics are sent to the {{es}} cluster. A new snapshot of APM Server metrics is generated and scheduled for publishing each period. The default value is 10 * time.Second. - #### `state.period` [_state_period] The time interval (in seconds) when state information are sent to the {{es}} cluster. A new snapshot of APM Server state is generated and scheduled for publishing each period. The default value is 60 * time.Second. - #### `protocol` [_protocol] The name of the protocol to use when connecting to the {{es}} cluster. The options are: `http` or `https`. The default is `http`. If you specify a URL for `hosts`, however, the value of protocol is overridden by the scheme you specify in the URL. - #### `proxy_url` [_proxy_url_4] The URL of the proxy to use when connecting to the {{es}} cluster. For more information, see [{{es}}](configure-elasticsearch-output.md). - #### `timeout` [_timeout_5] The HTTP request timeout in seconds for the {{es}} request. The default is `90`. - #### `ssl` [_ssl_5] Configuration options for Transport Layer Security (TLS) or Secure Sockets Layer (SSL) parameters like the certificate authority (CA) to use for HTTPS-based connections. If the `ssl` section is missing, the host CAs are used for HTTPS connections to {{es}}. For more information, see [SSL/TLS output settings](ssltls-output-settings.md). - #### `username` [_username_3] The user ID that APM Server uses to authenticate with the {{es}} instances for shipping monitoring data. diff --git a/solutions/observability/apps/use-metricbeat-to-send-monitoring-data.md b/solutions/observability/apps/use-metricbeat-to-send-monitoring-data.md index eceb54f4d4..c37d7ad109 100644 --- a/solutions/observability/apps/use-metricbeat-to-send-monitoring-data.md +++ b/solutions/observability/apps/use-metricbeat-to-send-monitoring-data.md @@ -8,7 +8,6 @@ applies_to: # Use Metricbeat to send monitoring data [apm-monitoring-metricbeat-collection] - In 7.3 and later, you can use {{metricbeat}} to collect data about APM Server and ship it to the monitoring cluster. The benefit of using {{metricbeat}} instead of internal collection is that the monitoring agent remains active even if the APM Server instance dies. To collect and ship monitoring data: @@ -16,7 +15,6 @@ To collect and ship monitoring data: 1. [Configure the shipper you want to monitor](#apm-configure-shipper) 2. [Install and configure {{metricbeat}} to collect monitoring data](#apm-configure-metricbeat) - ## Configure the shipper you want to monitor [apm-configure-shipper] 1. Enable the HTTP endpoint to allow external collection of monitoring data: @@ -61,7 +59,6 @@ To collect and ship monitoring data: 5. Start APM Server. - ## Install and configure {{metricbeat}} to collect monitoring data [apm-configure-metricbeat] 1. Install {{metricbeat}} on the same server as APM Server. To learn how, see [Get started with {{metricbeat}}](beats://reference/metricbeat/metricbeat-installation-configuration.md). If you already have {{metricbeat}} installed on the server, skip this step. @@ -105,7 +102,7 @@ To collect and ship monitoring data: If the Elastic {{security-features}} are enabled, you must also provide a user ID and password so that {{metricbeat}} can collect metrics successfully: - 1. Create a user on the {{es}} cluster that has the `remote_monitoring_collector` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). Alternatively, if it’s available in your environment, use the `remote_monitoring_user` [built-in user](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). + 1. Create a user on the {{es}} cluster that has the `remote_monitoring_collector` [built-in role](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). Alternatively, if it’s available in your environment, use the `remote_monitoring_user` [built-in user](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). 2. Add the `username` and `password` settings to the beat module configuration file. 4. Optional: Disable the system module in the {{metricbeat}}. @@ -122,7 +119,6 @@ To collect and ship monitoring data: In production environments, we strongly recommend using a separate cluster (referred to as the *monitoring cluster*) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster. :::: - For example, specify the {{es}} output information in the {{metricbeat}} configuration file (`metricbeat.yml`): ```yaml @@ -140,17 +136,15 @@ To collect and ship monitoring data: 1. In this example, the data is stored on a monitoring cluster with nodes `es-mon-1` and `es-mon-2`. 2. Specify one of `api_key` or `username`/`password`. - If you configured the monitoring cluster to use encrypted communications, you must access it via HTTPS. For example, use a `hosts` setting like `https://es-mon-1:9200`. ::::{important} The {{es}} {{monitor-features}} use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one ingest node. :::: - If the {{es}} {{security-features}} are enabled on the monitoring cluster, you must provide a valid user ID and password so that {{metricbeat}} can send metrics successfully: - 1. Create a user on the monitoring cluster that has the `remote_monitoring_agent` [built-in role](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). Alternatively, if it’s available in your environment, use the `remote_monitoring_user` [built-in user](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). + 1. Create a user on the monitoring cluster that has the `remote_monitoring_agent` [built-in role](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md). Alternatively, if it’s available in your environment, use the `remote_monitoring_user` [built-in user](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). ::::{tip} If you’re using {{ilm}}, the remote monitoring user requires additional privileges to create and read indices. For more information, see [Use feature roles](create-assign-feature-roles-to-apm-server-users.md). @@ -161,5 +155,5 @@ To collect and ship monitoring data: For more information about these configuration options, see [Configure the {{es}} output](beats://reference/metricbeat/elasticsearch-output.md). 6. [Start {{metricbeat}}](beats://reference/metricbeat/metricbeat-starting.md) to begin collecting monitoring data. -7. [View the monitoring data in {{kib}}](../../../deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md). +7. [View the monitoring data in {{kib}}](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md). diff --git a/solutions/observability/apps/use-opentelemetry-with-apm.md b/solutions/observability/apps/use-opentelemetry-with-apm.md index 9f06dc5179..6d0450fba9 100644 --- a/solutions/observability/apps/use-opentelemetry-with-apm.md +++ b/solutions/observability/apps/use-opentelemetry-with-apm.md @@ -10,22 +10,21 @@ applies_to: # Use OpenTelemetry with APM [apm-open-telemetry] +:::{include} _snippets/apm-server-vs-mis.md +::: ::::{note} For a complete overview of using OpenTelemetry with Elastic, explore [**Elastic Distributions of OpenTelemetry**](https://github.com/elastic/opentelemetry). - :::: - [OpenTelemetry](https://opentelemetry.io/docs/concepts/what-is-opentelemetry/) is a set of APIs, SDKs, tooling, and integrations that enable the capture and management of telemetry data from your services and applications. Elastic integrates with OpenTelemetry, allowing you to reuse your existing instrumentation to easily send observability data to the {{stack}}. There are several ways to integrate OpenTelemetry with the {{stack}}: -* [Elastic Distributions of OpenTelemetry language SDKs](../../../solutions/observability/apps/use-opentelemetry-with-apm.md#apm-otel-elastic-distros) -* [Upstream OpenTelemetry API/SDK + Elastic APM agent](../../../solutions/observability/apps/use-opentelemetry-with-apm.md#apm-otel-api-sdk-elastic-agent) -* [Upstream OpenTelemetry Collector and language SDKs](../../../solutions/observability/apps/use-opentelemetry-with-apm.md#apm-otel-upstream) -* [AWS Lambda collector exporter](../../../solutions/observability/apps/use-opentelemetry-with-apm.md#apm-otel-lambda) - +* [Elastic Distributions of OpenTelemetry language SDKs](/solutions/observability/apps/use-opentelemetry-with-apm.md#apm-otel-elastic-distros) +* [Upstream OpenTelemetry API/SDK + Elastic APM agent](/solutions/observability/apps/use-opentelemetry-with-apm.md#apm-otel-api-sdk-elastic-agent) +* [Upstream OpenTelemetry Collector and language SDKs](/solutions/observability/apps/use-opentelemetry-with-apm.md#apm-otel-upstream) +* [AWS Lambda collector exporter](/solutions/observability/apps/use-opentelemetry-with-apm.md#apm-otel-lambda) ## Elastic Distributions of OpenTelemetry language SDKs [apm-otel-elastic-distros] @@ -33,7 +32,6 @@ Elastic integrates with OpenTelemetry, allowing you to reuse your existing instr This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. :::: - Elastic offers several distributions of OpenTelemetry language SDKs. A *distribution* is a customized version of an upstream OpenTelemetry repository. Each Elastic Distribution of OpenTelemetry is a customized version of an [OpenTelemetry language SDK](https://opentelemetry.io/docs/languages/). :::{image} /solutions/images/observability-apm-otel-distro.png @@ -59,10 +57,9 @@ For more details about OpenTelemetry distributions in general, visit the [OpenTe :::: - ## Upstream OpenTelemetry API/SDK + Elastic APM agent [apm-otel-api-sdk-elastic-agent] -Use the OpenTelemetry API/SDKs with [Elastic APM agents](../../../solutions/observability/apps/fleet-managed-apm-server.md#_step_3_install_apm_agents) to translate OpenTelemetry API calls to Elastic APM API calls. +Use the OpenTelemetry API/SDKs with [Elastic APM agents](/solutions/observability/apps/fleet-managed-apm-server.md#_step_3_install_apm_agents) to translate OpenTelemetry API calls to Elastic APM API calls. :::{image} /solutions/images/observability-apm-otel-api-sdk-elastic-agent.png :alt: apm otel api sdk elastic agent @@ -80,12 +77,11 @@ Find more details about how to use an OpenTelemetry API or SDK with an Elastic A * [**APM Node.js agent →**](apm-agent-nodejs://reference/opentelemetry-bridge.md) * [**APM Python agent →**](apm-agent-python://reference/opentelemetry-api-bridge.md) - ## Upstream OpenTelemetry Collector and language SDKs [apm-otel-upstream] The {{stack}} natively supports the OpenTelemetry protocol (OTLP). This means trace data and metrics collected from your applications and infrastructure by an OpenTelemetry Collector or OpenTelemetry language SDK can be sent to the {{stack}}. -You can set up an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/), instrument your application with an [OpenTelemetry language SDK](https://opentelemetry.io/docs/languages/) that sends data to the collector, and use the collector to process and export the data to APM Server. +You can set up an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/), instrument your application with an [OpenTelemetry language SDK](https://opentelemetry.io/docs/languages/) that sends data to the collector, and use the collector to process and export the data to either {{apm-server-or-mis}}. :::{image} /solutions/images/observability-apm-otel-api-sdk-collector.png :alt: apm otel api sdk collector @@ -93,11 +89,10 @@ You can set up an [OpenTelemetry Collector](https://opentelemetry.io/docs/collec ::: ::::{note} -It’s also possible to send data directly to APM Server from an upstream OpenTelemetry SDK. You might do this during development or if you’re monitoring a small-scale application. Read more about when to use a collector in the [OpenTelemetry documentation](https://opentelemetry.io/docs/collector/#when-to-use-a-collector). +It’s also possible to send data directly to either {{apm-server-or-mis}} from an upstream OpenTelemetry SDK. You might do this during development or if you’re monitoring a small-scale application. Read more about when to use a collector in the [OpenTelemetry documentation](https://opentelemetry.io/docs/collector/#when-to-use-a-collector). :::: - This approach works well when you need to instrument a technology that Elastic doesn’t provide a solution for. For example, if you want to instrument C or C++ you could use the [OpenTelemetry C++ client](https://github.com/open-telemetry/opentelemetry-cpp). However, there are some limitations when using collectors and language SDKs built and maintained by OpenTelemetry, including: @@ -106,10 +101,9 @@ However, there are some limitations when using collectors and language SDKs buil * You won’t have access to Elastic enterprise APM features. * You may experience problems with performance efficiency. -For more on the limitations associated with using upstream OpenTelemetry tools, refer to [Limitations](../../../solutions/observability/apps/limitations.md). - -[**Get started with upstream OpenTelemetry Collectors and language SDKs →**](../../../solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md) +For more on the limitations associated with using upstream OpenTelemetry tools, refer to [Limitations](/solutions/observability/apps/limitations.md). +[**Get started with upstream OpenTelemetry Collectors and language SDKs →**](/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md) ## AWS Lambda collector exporter [apm-otel-lambda] diff --git a/solutions/observability/apps/use-select-metrics-emitted-directly-to-monitoring-cluster.md b/solutions/observability/apps/use-select-metrics-emitted-directly-to-monitoring-cluster.md index 0cb53f746b..e5992b80d6 100644 --- a/solutions/observability/apps/use-select-metrics-emitted-directly-to-monitoring-cluster.md +++ b/solutions/observability/apps/use-select-metrics-emitted-directly-to-monitoring-cluster.md @@ -8,7 +8,6 @@ applies_to: # Use the select metrics emitted directly to your monitoring cluster [apm-monitoring-local-collection] - In 8.11 and later, we emit a selected set of metrics directly to the monitoring cluster. The benefit of using local collection instead of internal collection is that the metrics are sent directly to your main monitoring index, making it easier to view shared data. ## The select metrics [apm-select-metrics] diff --git a/solutions/observability/apps/use-synthetics-cli.md b/solutions/observability/apps/use-synthetics-cli.md index 577a1c19b4..555537a845 100644 --- a/solutions/observability/apps/use-synthetics-cli.md +++ b/solutions/observability/apps/use-synthetics-cli.md @@ -10,8 +10,6 @@ applies_to: # Use the Synthetics CLI [synthetics-command-reference] - - ## `@elastic/synthetics` [elastic-synthetics-command] Elastic uses the [@elastic/synthetics](https://www.npmjs.com/package/@elastic/synthetics) Node.js library to run synthetic browser tests and report the test results. The library also provides a CLI to help you scaffold, develop/run tests locally, and push tests to {{kib}}. @@ -32,25 +30,22 @@ You will not need to use most command line flags. However, there are some you ma : RegExp pattern to match journey files in the current working directory. Defaults to `/*.journey.(ts|js)$/`, which matches files ending with `.journey.ts` or `.journey.js`. `--params ` -: JSON object that defines any variables your tests require. Read more in [Work with params and secrets](../../../solutions/observability/apps/work-with-params-secrets.md). - - Params passed will be merged with params defined in your [`synthetics.config.js` file](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-params). Params defined via the CLI take precedence. +: JSON object that defines any variables your tests require. Read more in [Work with params and secrets](/solutions/observability/apps/work-with-params-secrets.md). + Params passed will be merged with params defined in your [`synthetics.config.js` file](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-params). Params defined via the CLI take precedence. `--playwright-options ` -: JSON object to pass in custom Playwright options for the agent. For more details on relevant Playwright options, refer to the [the configuration docs](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-playwright-options). - - Options passed will be merged with Playwright options defined in your [`synthetics.config.js` file](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-playwright-options). Options defined via the CLI take precedence. +: JSON object to pass in custom Playwright options for the agent. For more details on relevant Playwright options, refer to the [the configuration docs](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-playwright-options). + Options passed will be merged with Playwright options defined in your [`synthetics.config.js` file](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-playwright-options). Options defined via the CLI take precedence. `--screenshots ` : Control whether or not to capture screenshots at the end of each step. Options include `'on'`, `'off'`, or `'only-on-failure'`. - This can also be set in the configuration file using [`monitor.screenshot`](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. - + This can also be set in the configuration file using [`monitor.screenshot`](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. `-c, --config ` -: Path to the configuration file. By default, test runner looks for a `synthetics.config.(js|ts)` file in the current directory. Synthetics configuration provides options to configure how your tests are run and pushed to Elastic. Allowed options are described in the [configuration file](../../../solutions/observability/apps/configure-synthetics-projects.md). +: Path to the configuration file. By default, test runner looks for a `synthetics.config.(js|ts)` file in the current directory. Synthetics configuration provides options to configure how your tests are run and pushed to Elastic. Allowed options are described in the [configuration file](/solutions/observability/apps/configure-synthetics-projects.md). `--reporter ` : One of `json`, `junit`, `buildkite-cli`, or `default`. Use the JUnit or Buildkite reporter to provide easily parsed output to CI systems. @@ -61,27 +56,23 @@ You will not need to use most command line flags. However, there are some you ma `--no-throttling` : Does not apply throttling. - Throttling can also be disabled in the configuration file using [`monitor.throttling`](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. - + Throttling can also be disabled in the configuration file using [`monitor.throttling`](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. ::::{note} Network throttling for browser based monitors is disabled. See this [documention](https://github.com/elastic/synthetics/blob/main/docs/throttling.md) for more details. :::: - `--no-headless` : Runs with the browser in headful mode. This is the same as setting [Playwright’s `headless` option](https://playwright.dev/docs/api/class-testoptions#test-options-headless) to `false` by running `--playwright-options '{"headless": false}'`. - ::::{note} Headful mode should only be used locally to see the browser and interact with DOM elements directly for testing purposes. Do not attempt to run in headful mode when running through Elastic’s global managed testing infrastructure or {{private-location}}s as this is not supported. :::: - `-h, --help` : Shows help for the `npx @elastic/synthetics` command. @@ -90,14 +81,11 @@ The `--pattern`, `--tags`, and `--match` flags for filtering are only supported :::: - ::::{note} For debugging synthetic tests locally, you can set an environment variable, `DEBUG=synthetics npx @elastic/synthetics`, to capture Synthetics agent logs. :::: - - ## `@elastic/synthetics init` [elastic-synthetics-init-command] Scaffold a new Synthetics project using Elastic Synthetics. @@ -108,8 +96,7 @@ This will create a template Node.js project that includes the synthetics agent, npx @elastic/synthetics init ``` -Read more about what’s included in a template Synthetics project in [Create a Synthetics project](../../../solutions/observability/apps/create-monitors-with-project-monitors.md). - +Read more about what’s included in a template Synthetics project in [Create a Synthetics project](/solutions/observability/apps/create-monitors-with-project-monitors.md). ## `@elastic/synthetics push` [elastic-synthetics-push-command] @@ -127,7 +114,6 @@ The `push` command includes interactive prompts to prevent you from accidentally :::: - ::::{note} If the journey contains external NPM packages other than the `@elastic/synthetics`, those packages will be bundled along with the journey code when the `push` command is invoked. However there are some limitations when using external packages: @@ -137,67 +123,59 @@ If the journey contains external NPM packages other than the `@elastic/synthetic :::: - `--auth ` -: API key used for [authentication](../../../deploy-manage/api-keys/elasticsearch-api-keys.md). You can also set the API key via the `SYNTHETICS_API_KEY` environment variable. - - If you are pushing to a [{{private-location}}](../../../solutions/observability/apps/create-monitors-in-synthetics-app.md), you must use an API key generated in 8.4 or higher. +: API key used for [authentication](/deploy-manage/api-keys/elasticsearch-api-keys.md). You can also set the API key via the `SYNTHETICS_API_KEY` environment variable. - On {{stack}}, you must be logged into {{kib}} as a user with the privileges described in [Writer role](../../../solutions/observability/apps/writer-role.md) to create an API key. + If you are pushing to a [{{private-location}}](/solutions/observability/apps/create-monitors-in-synthetics-app.md), you must use an API key generated in 8.4 or higher. - On {{obs-serverless}}, you must be logged in as a user with [Editor](../../../solutions/observability/apps/grant-users-access-to-secured-resources.md) access to create an API key. + On {{stack}}, you must be logged into {{kib}} as a user with the privileges described in [Writer role](/solutions/observability/apps/writer-role.md) to create an API key. + On {{obs-serverless}}, you must be logged in as a user with [Editor](/solutions/observability/apps/grant-users-access-to-secured-resources.md) access to create an API key. `--id ` : A unique id associated with your project. It will be used for logically grouping monitors. - If you used [`init` to create a project](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-init-command), this is the `` you specified. - - This can also be set in the configuration file using [`project.id`](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-project). The value defined via the CLI will take precedence. + If you used [`init` to create a project](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-init-command), this is the `` you specified. + This can also be set in the configuration file using [`project.id`](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-project). The value defined via the CLI will take precedence. `--url ` : The URL for the deployment or Observability Serverless project to which you want to upload the monitors. - This can also be set in the configuration file using [`project.url`](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-project). The value defined via the CLI will take precedence. + This can also be set in the configuration file using [`project.url`](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-project). The value defined via the CLI will take precedence. % Stateful only for --space `--space ` -: The identifier of the target [{{kib}} space](../../../deploy-manage/manage-spaces.md) for the pushed monitors. Spaces help you organize pushed monitors. Pushes to the "default" space if not specified. - - This can also be set in the configuration file using [`project.space`](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-project). The value defined via the CLI will take precedence. +: The identifier of the target [{{kib}} space](/deploy-manage/manage-spaces.md) for the pushed monitors. Spaces help you organize pushed monitors. Pushes to the "default" space if not specified. + This can also be set in the configuration file using [`project.space`](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-project). The value defined via the CLI will take precedence. `--schedule ` : The interval (in minutes) at which the monitor should run. - This can also be set in the configuration file using [`monitor.schedule`](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. - + This can also be set in the configuration file using [`monitor.schedule`](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. [`--locations Array`](https://github.com/elastic/synthetics/blob/v1.3.0/src/locations/public-locations.ts#L28-L37) : Where to deploy the monitor. Monitors can be deployed in multiple locations so that you can detect differences in availability and response times across those locations. - To list available locations, refer to [`@elastic/synthetics locations`](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command). - - This can also be set in the configuration file using [`monitor.locations` in the configuration file](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. + To list available locations, refer to [`@elastic/synthetics locations`](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command). + This can also be set in the configuration file using [`monitor.locations` in the configuration file](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. `--private-locations Array` -: The [{{private-location}}s](../../../solutions/observability/apps/monitor-resources-on-private-networks.md) to which the monitors will be deployed. These {{private-location}}s refer to locations hosted and managed by you, whereas `locations` are hosted by Elastic. You can specify a {{private-location}} using the location’s name. - - To list available {{private-location}}s, refer to [`@elastic/synthetics locations`](../../../solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command). +: The [{{private-location}}s](/solutions/observability/apps/monitor-resources-on-private-networks.md) to which the monitors will be deployed. These {{private-location}}s refer to locations hosted and managed by you, whereas `locations` are hosted by Elastic. You can specify a {{private-location}} using the location’s name. - This can also be set in the configuration file using [`monitor.privateLocations` in the configuration file](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. + To list available {{private-location}}s, refer to [`@elastic/synthetics locations`](/solutions/observability/apps/use-synthetics-cli.md#elastic-synthetics-locations-command). + This can also be set in the configuration file using [`monitor.privateLocations` in the configuration file](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. `--fields ` -: A list of key-value pairs that will be sent with each monitor event. The `fields` are appended to {{es}} documents as `labels`, and those labels are displayed in {{kib}} in the *Monitor details* panel in the [individual monitor’s *Overview* tab](../../../solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-individual-monitors-overview). +: A list of key-value pairs that will be sent with each monitor event. The `fields` are appended to {{es}} documents as `labels`, and those labels are displayed in {{kib}} in the *Monitor details* panel in the [individual monitor’s *Overview* tab](/solutions/observability/apps/analyze-data-from-synthetic-monitors.md#synthetics-analyze-individual-monitors-overview). Example: `--fields '{ "foo": bar", "team": "synthetics" }'` - This can also be set in the configuration file using [the `monitor.fields` option](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. - + This can also be set in the configuration file using [the `monitor.fields` option](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor). The value defined via the CLI will take precedence. `--yes` : The `push` command includes interactive prompts to prevent you from accidentally deleting or duplicating monitors. If running the CLI non-interactively, you can override these prompts using the `--yes` option. When the `--yes` option is passed to `push`: @@ -205,8 +183,6 @@ If the journey contains external NPM packages other than the `@elastic/synthetic * If you `push` a Synthetics project that used to contain one or more monitors but no longer contains any monitors, all monitors associated with the Synthetics project ID being pushed will be deleted. * If you `push` a Synthetics project that’s already been pushed using one Synthetics project ID and then try to `push` it using a *different* ID, it will create duplicates of all monitors in the Synthetics project. - - ## Tag monitors [tagging-and-filtering] Synthetics journeys can be tagged with one or more tags. Use tags to filter journeys when running tests locally or pushing them to Elastic. @@ -232,8 +208,7 @@ tags: - env:qa ``` -To apply tags to all browser and lightweight monitors, configure using [the `monitor.tags`](../../../solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor) field in the `synthetics.config.ts` file. - +To apply tags to all browser and lightweight monitors, configure using [the `monitor.tags`](/solutions/observability/apps/configure-synthetics-projects.md#synthetics-configuration-monitor) field in the `synthetics.config.ts` file. ### Filter monitors [_filtering_monitors] @@ -255,7 +230,6 @@ npx @elastic/synthetics push --config synthetics.qa.config.ts --tags env:qa npx @elastic/synthetics push --config synthetics.prod.config.ts --tags env:prod ``` - ## `@elastic/synthetics locations` [elastic-synthetics-locations-command] List all available locations for running synthetics monitors. @@ -272,13 +246,12 @@ To list both locations on Elastic’s global managed infrastructure and {{privat : The URL for the {{kib}} deployment or Observability Serverless project from which to fetch all available public and {{private-location}}s. `--auth ` -: API key used for [authentication](../../../deploy-manage/api-keys/elasticsearch-api-keys.md). +: API key used for [authentication](/deploy-manage/api-keys/elasticsearch-api-keys.md). ::::{note} -If an administrator has disabled Elastic managed locations for the role you are assigned and you do *not* include `--url` and `--auth`, all global locations managed by Elastic will be listed. However, you will not be able to push to these locations with your API key and will see an error: *You don’t have permission to use Elastic managed global locations*. For more details, refer to the [troubleshooting docs](../../../troubleshoot/observability/troubleshooting-synthetics.md#synthetics-troubleshooting-public-locations-disabled). +If an administrator has disabled Elastic managed locations for the role you are assigned and you do *not* include `--url` and `--auth`, all global locations managed by Elastic will be listed. However, you will not be able to push to these locations with your API key and will see an error: *You don’t have permission to use Elastic managed global locations*. For more details, refer to the [troubleshooting docs](/troubleshoot/observability/troubleshooting-synthetics.md#synthetics-troubleshooting-public-locations-disabled). :::: - ## `@elastic/synthetics totp ` [elastic-synthetics-totp-command] Generate a Time-based One-Time Password (TOTP) for multifactor authentication (MFA) in Synthetics. @@ -307,7 +280,6 @@ npx @elastic/synthetics totp --issuer --label