Skip to content

Add docs for advanced monitoring options #1361

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Oct 2, 2024
Merged

Add docs for advanced monitoring options #1361

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e3e0d30
Add docs for advanced monitoring options
kilfoyle Oct 1, 2024
284bb2f
Remove the 'override the default monitoring port' section
kilfoyle Oct 1, 2024
fe8cf38
Address Craig's comments
kilfoyle Oct 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 26 additions & 25 deletions docs/en/ingest-management/agent-policies.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ The following table illustrates the {fleet} user actions available to different
|{y}
|{n}

|<<change-policy-enable-agent-monitoring,Enable agent monitoring>>
|<<change-policy-enable-agent-monitoring,Configure agent monitoring>>
|{y}
|{n}

Expand All @@ -116,10 +116,6 @@ The following table illustrates the {fleet} user actions available to different
|{y}
|{n}

|<<agent-policy-http-monitoring>>
|{y}
|{n}

|<<agent-policy-log-level>>
|{y}
|{n}
Expand Down Expand Up @@ -310,19 +306,40 @@ Note that adding custom tags is not supported for a small set of inputs:

[discrete]
[[change-policy-enable-agent-monitoring]]
== Enable agent monitoring
== Configure agent monitoring

Use this setting to collect monitoring logs and metrics from {agent}. All monitoring data will be written to the specified **Default namespace**.
Use these settings to collect monitoring logs and metrics from {agent}. All monitoring data will be written to the specified **Default namespace**.

. In {fleet}, click **Agent policies**.
Select the name of the policy you want to edit.

. Click the **Settings** tab and scroll to **Enable agent monitorings**.
. Click the **Settings** tab and scroll to **Agent monitoring**.

. Select whether to collect agent logs, agent metrics, or both, from the {agents} that use the policy.

+
When this setting is enabled an {agent} integration is created automatically.

. Expand the **Advanced monitoring options** section to access <<advanced-agent-monitoring-settings,other monitoring settings>>.

. Save your changes for the updated monitoring settings to take effect.

[discrete]
[[advanced-agent-monitoring-settings]]
=== Advanced agent monitoring settings

**HTTP monitoring endpoint**::
Enabling this setting exposes a `/liveness` API endpoint that you can use to monitor {agent} health. By default, the endpoint returns a `200` OK status as long as {agent}'s internal main loop is responsive and can process configuration changes. It can be configured to also monitor the component states and return an error if anything is degraded or has failed. This endpoint can be used by Kubernetes to restart the container, for example.
+
When you enable this setting, you need to also provide the host URL and port where the endpoint can be accessed. Using the default `localhost` is recommended.
+
You can also enable profiling at `/debug/pprof` to control whether the {agent} exposes the `/debug/pprof/` endpoints with the monitoring endpoints. This is disabled by default. Data produced by these endpoints can be useful for debugging but present a security risk. It's recommended to leave this option disabled if the monitoring endpoint is accessible over a network.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can mention that the heap profiles available from /debug/pprof/ are included in diagnostics by default. CPU profiles are also included when the --cpu-profile option is included. On that note the CPU profile option is missing from https://www.elastic.co/guide/en/fleet/current/elastic-agent-cmd-options.html#elastic-agent-diagnostics-command.

❯ ./elastic-agent diagnostics --help
This command gathers diagnostics information from the Elastic Agent and writes it to a zip archive.

Usage:
  elastic-agent diagnostics [flags]

Flags:
  -p, --cpu-profile      wait to collect a CPU profile

The profiles exposed by /debug/pprof are documented in https://pkg.go.dev/net/http/pprof if you want to link to that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect. I've added a link to the pprof docs and also updated the Command Ref to show both the --cpu-profile option and the --p option as an alias for --cpu-profile.


**Diagnostics rate limiting**::
You can set a rate limit for the request diagnostics action handler. By default requests are limited to an interval of `1m` and a burst value of `1`. This setting does not affect diagnostics collected through the CLI.

**Diagnostics file upload**::
This setting configures retries for the file upload client. By default, a maximum of `10` retries are allowed with an initial duration of `1s` and a backoff duration of `1m`. The client may retry failed requests with exponential backoff.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may want to mention this only have an effect for Fleet managed agents and they apply to diagnostics requests from Fleet.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cmacknz I opened a separate issue to add these two options to the Configure monitoring for standalone Elastic Agents page, but now I'm thinking that the settings shouldn't be documented there since they affect only Fleet-managed agent. If you agree, I'll just close that issue.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, no reason to document these for standalone agents as they only apply to fleet managed agents.


[discrete]
[[change-policy-output]]
== Change the output of a policy
Expand Down Expand Up @@ -414,22 +431,6 @@ Select the name of the policy you want to edit.

. Set **Limit CPU usage** as needed. For example, to limit Go processes supervised by {agent} to two operating system threads each, set this value to `2`.

[discrete]
[[agent-policy-http-monitoring]]
== Override the default monitoring port

You can override the default port that {agent} uses to send monitoring data. It's useful to be able to adjust this setting if you have an application running on the machine on which the agent is deployed, and that is using the same port.

. In {fleet}, click **Agent policies**.
Select the name of the policy you want to edit.

. Click the **Settings** tab and scroll to **Advanced settings**.

//. Set **Agent HTTP monitoring** setting to enabled, and then specify a host and port for the monitoring data output.
. Specify a host and port for the monitoring data output.

//. Enable **buffer.enabled** if you'd like {agent} and {beats} to collect metrics into an in-memory buffer and expose these through a `/buffer` endpoint. This data can be useful for debugging or if the {agent} has issues communicating with {es}. Enabling this option may slightly increase process memory usage.

[discrete]
[[agent-policy-log-level]]
== Set the {agent} log level
Expand Down
2 changes: 2 additions & 0 deletions docs/en/ingest-management/fleet/monitor-elastic-agent.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,8 @@ monitoring settings for all agents enrolled in a specific agent policy:
. Under **Agent monitoring**, deselect (or select) one or both of these
settings: **Collect agent logs** and **Collect agent metrics**.

. Under **Advanced monitoring options** you can configure additional settings including an HTTP monitoring endpoint, diagnostics rate limiting, and diagnostics file upload limits. Refer to <<change-policy-enable-agent-monitoring,configure agent monitoring>> for details.

. Save your changes.

To turn off agent monitoring when creating a new agent policy:
Expand Down