Add list of agent OOB alert rules with descriptions #3608

karenzone · 2025-10-22T19:04:52Z

Related: #2760
Follow-up to: #3537
Fixes: #3815

github-actions · 2025-10-22T19:07:45Z

🔍 Preview links for changed docs

karenzone · 2025-10-22T20:32:10Z

reference/fleet/alert-templates.md

+| -------- | -------- |
+| [Elastic Agent] CPU usage spike|  Checks if {{agent}} or any of its processes were pegged at a high CPU for a specified window of time. This could signal a bug in an application and warrant further investigation.<br> - Condition: `system.process.cpu.total.time.ms` > 80% for 5 minutes<br>- Default: Enabled |
+| [Elastic Agent] Dropped events | Checks if percentage of events dropped to acked events from the pipeline are greater than or equal to 5%. Rows are distinct by agent id and component id. |
+| [Elastic Agent] Excessive memory usage|  Checks if {{agent}} or any of its processes have a high memory usage or memory usage that is trending higher. This could signal a memory leak in an application and warrant further investigation.<br>- Condition: Alert on `system.process.memory.rss.pct` > 80%<br>- Default: Enabled (perhaps the threshold should be higher if this is on by default) |


80%
- Default: Enabled (perhaps the threshold should be higher if this is on by default)

What did we decide?

I believe the threshold here is currently set to 50% - @MichelLosier is that correct?

Correct, 50% is the new threshold

karenzone · 2025-10-22T20:33:34Z

reference/fleet/alert-templates.md

+| [Elastic Agent] CPU usage spike|  Checks if {{agent}} or any of its processes were pegged at a high CPU for a specified window of time. This could signal a bug in an application and warrant further investigation.<br> - Condition: `system.process.cpu.total.time.ms` > 80% for 5 minutes<br>- Default: Enabled |
+| [Elastic Agent] Dropped events | Checks if percentage of events dropped to acked events from the pipeline are greater than or equal to 5%. Rows are distinct by agent id and component id. |
+| [Elastic Agent] Excessive memory usage|  Checks if {{agent}} or any of its processes have a high memory usage or memory usage that is trending higher. This could signal a memory leak in an application and warrant further investigation.<br>- Condition: Alert on `system.process.memory.rss.pct` > 80%<br>- Default: Enabled (perhaps the threshold should be higher if this is on by default) |
+| [Elastic Agent] Excessive restarts| Checks if excessive restarts on a host which require further investigation. Some of these restarts could have a business impact and getting an alert for them would allow us to act quickly to mitigate.<br>- Condition: Alert on (not sure) > 10 times in a 5 minute window<br>- Default: Enabled |


Alert on (not sure) > 10 times in a 5 minute window

What did we decide?

this is correct. currently set to greater than 10 restarts in the5 min window

reference/fleet/alert-templates.md

karenzone · 2025-10-22T20:39:52Z

@nimarezainia @MichelLosier I've added the alert rules and descriptions, and asked some questions about final decisions inline. Please take a look and let me know what you think. If there's a later "source of truth" for available alerts and descriptions, please let me know and I'll update the PR accordingly.

We still have some questions to answer (see inline comments," but I'm marking this as "Ready for review."

nimarezainia · 2025-10-22T23:42:27Z

@MichelLosier I don't see the "Agent Unhealthy" rule in staging? are we shipping the agent status changes as well?

benironside

I'm not very familiar with the subject matter but left some minor editorial suggestions for your consideration!

reference/fleet/alert-templates.md

Co-authored-by: Benjamin Ironside Goldstein <[email protected]>

karenzone · 2025-11-07T23:37:20Z

@nimarezainia @MichelLosier, let's wrap this one up and get it merged for our users.
A lot of information and implementation ideas were shared in slack, some of it forward-looking.

Please review this content with an eye for what's available to users now. We can update docs as new rules become available. Just open a docs issue!

nimarezainia · 2025-11-09T23:05:36Z

@MichelLosier could you please review these changes, we can always change the thresholds as we learn more. (I left a few comments for you above). thx

MichelLosier · 2025-11-10T14:48:28Z

reference/fleet/alert-templates.md

+| [Elastic Agent] Excessive restarts| Checks for excessive restarts on a host which require further investigation. Some restarts can have a business impact and getting alerts for them can enable timely mitigation.<br>- Condition: Alert on restarts > 10 restarts in a 5 minute window<br>- Default: Enabled |
+| [Elastic Agent] High pipeline queue | Checks if max of `beat.stats.libbeat.pipeline.queue.filled.pct` exceeds 90%. Rows are distinguished by agent ID and component ID. |
+| [Elastic Agent] Output errors | Checks if errors per minute from an agent component is greater than 5. Rows are distinguished by agent ID and component ID. |
+


We may want to add here the [Elastic Agent] Unhealthy status alerting rule. We can describe it as "Checks if an agent has transitioned to an 'unhealthy' status, which can indicate errors or degraded functionality of the agent.

#3608 (comment)

I checked staging and didn't see it, so I removed it.
Thanks for clarifying. I'll add it back.

MichelLosier

Looks great! Just the adjustment for the percentage threshold of the high memory rule, and adding a description of the Unhealthy status rule.

karenzone · 2025-11-11T17:58:07Z

@MichelLosier, please give it a look.

MichelLosier

Looks great!

karenzone · 2025-11-11T19:40:24Z

I've re-requested a review from ingest-docs as code-owner. Stay tuned!

colleenmcginnis

Some thoughts and questions below.

reference/fleet/alert-templates.md

karenzone · 2025-11-11T22:42:58Z

reference/fleet/alert-templates.md

+| [Elastic Agent] Excessive memory usage|  Checks if {{agent}} or any of its processes have a high memory usage or memory usage that is trending up. This could signal a memory leak in an application and warrant further investigation.<br>- Condition: Alert on `system.process.memory.rss.pct` more than 50%<br>- Default: Enabled |
+| [Elastic Agent] Excessive restarts| Checks for excessive restarts on a host. Some restarts can have a business impact, and getting alerts for them can enable timely mitigation.<br>- Condition: Alert on 11 or more restarts in a 5-minute window<br>- Default: Enabled |
+| [Elastic Agent] High pipeline queue | Checks percentage of pipeline queue. Rows are distinguished by agent ID and component ID. <br> - Condition: Alert on max of `beat.stats.libbeat.pipeline.queue.filled.pct` exceeding 90%  <br>- Default: Enabled|
+| [Elastic Agent] Output errors | Checks errors per minute from an agent component. Rows are distinguished by agent ID and component ID. <br> - Condition: Alert on 6 or more errors per minute  <br>- Default: Enabled|


Originally was "greater than 5." I changed it to "6 or more."
Please keep me honest here, @MichelLosier

That's still correct. The rule is defined with > 5, but if "6 or more" is a clearer read thats totally fine.

karenzone · 2025-11-12T20:05:12Z

Published content: https://www.elastic.co/docs/reference/fleet/alert-templates#ea-alert-rules

karenzone self-assigned this Oct 22, 2025

karenzone force-pushed the 2760-alert-assets-v2 branch from 71c44c1 to e6daf96 Compare October 22, 2025 20:19

Add list of OOB alert rules with descriptions

326a28c

karenzone force-pushed the 2760-alert-assets-v2 branch from e6daf96 to 326a28c Compare October 22, 2025 20:24

karenzone commented Oct 22, 2025

View reviewed changes

reference/fleet/alert-templates.md Outdated Show resolved Hide resolved

karenzone requested review from MichelLosier and nimarezainia October 22, 2025 20:37

karenzone marked this pull request as ready for review October 22, 2025 20:40

karenzone requested a review from a team as a code owner October 22, 2025 20:40

karenzone requested a review from nchaulet October 22, 2025 20:48

benironside reviewed Oct 25, 2025

View reviewed changes

karenzone mentioned this pull request Oct 28, 2025

[Fleet] Doc: Support alert rule template as asset type #2760

Closed

karenzone mentioned this pull request Nov 4, 2025

[Fleet] Doc: Expand Elastic Agent OOB rules content with rules and descriptions #3815

Closed

karenzone and others added 4 commits November 6, 2025 19:57

Incorporate review comments

c713d77

Merge branch 'main' into 2760-alert-assets-v2

d432b83

Update reference/fleet/alert-templates.md

eef7a94

Co-authored-by: Benjamin Ironside Goldstein <[email protected]>

Refine list of currently availabley alert rules

a06fa0f

nimarezainia approved these changes Nov 9, 2025

View reviewed changes

MichelLosier reviewed Nov 10, 2025

View reviewed changes

karenzone and others added 2 commits November 11, 2025 12:48

Re-add Unhealthy status to alert rules list

5dd62c4

Merge branch 'main' into 2760-alert-assets-v2

86cb0bf

MichelLosier approved these changes Nov 11, 2025

View reviewed changes

colleenmcginnis reviewed Nov 11, 2025

View reviewed changes

reference/fleet/alert-templates.md Outdated Show resolved Hide resolved

reference/fleet/alert-templates.md Outdated Show resolved Hide resolved

reference/fleet/alert-templates.md Outdated Show resolved Hide resolved

reference/fleet/alert-templates.md Outdated Show resolved Hide resolved

Incorporate more review comments

e42523f

karenzone commented Nov 11, 2025

View reviewed changes

karenzone requested a review from colleenmcginnis November 12, 2025 15:50

colleenmcginnis approved these changes Nov 12, 2025

View reviewed changes

Merge branch 'main' into 2760-alert-assets-v2

b6a1674

karenzone merged commit a15fff0 into elastic:main Nov 12, 2025
7 checks passed

karenzone deleted the 2760-alert-assets-v2 branch November 12, 2025 20:01

Add list of agent OOB alert rules with descriptions #3608

Add list of agent OOB alert rules with descriptions #3608

Uh oh!

Conversation

karenzone commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

karenzone Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nimarezainia Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

MichelLosier Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

karenzone Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

nimarezainia Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

karenzone commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nimarezainia commented Oct 22, 2025

Uh oh!

benironside left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

karenzone commented Nov 7, 2025

Uh oh!

nimarezainia commented Nov 9, 2025

Uh oh!

MichelLosier Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

karenzone Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

MichelLosier left a comment

Choose a reason for hiding this comment

Uh oh!

karenzone commented Nov 11, 2025

Uh oh!

MichelLosier left a comment

Choose a reason for hiding this comment

Uh oh!

karenzone commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

colleenmcginnis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

karenzone Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MichelLosier Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

karenzone commented Nov 12, 2025

Uh oh!

karenzone commented Oct 22, 2025 •

edited

Loading

github-actions bot commented Oct 22, 2025 •

edited

Loading

karenzone Oct 22, 2025 •

edited

Loading

karenzone commented Oct 22, 2025 •

edited

Loading

benironside left a comment •

edited

Loading

karenzone commented Nov 11, 2025 •

edited

Loading

karenzone Nov 11, 2025 •

edited

Loading