apps sc: Alert size over limit only on active indicies

[aarnq]

Warning

This is a public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request, nor
• business confidential information, such as customer names.

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• kind/adr

What does this PR do / why do we need this PR?

This changes the index over size limit alert to only consider indices with a positive document rate over the last hour, meaning it will eventually stop alerting for indices that are inactive such as after a successful rollover.

Information to reviewers

I haven't had the time to test it yet.

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
  • The change upgrades CRDs
  • The change updates the config and the schema
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts are not affected)
  • The metrics names did change (Grafana dashboards and Prometheus alerts were fixed)
• Logs checks:
  • The logs do not show any errors after the change
• Pod Security Policy checks:
  • Any changed pod is covered by Pod Security Admission
  • Any changed pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any pods to be blocked by Pod Security Admission or Policies
• Network Policy checks:
  • Any changed pod is covered by Network Policies
  • The change does not cause any dropped packets in the NetworkPolicy Dashboard
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Bug checks:
  • The bug fix is covered by regression tests

[aarnq]

@elastisys/goto-logging-stack Would anyone of you like to take over this idea to make the alerts only report indices that are actively written to?
I have not the time to properly test it.

[lunkan93]

Do you have time for this @salehsedghpour ? I might have time next week, but otherwise I wont have time until January, if not I can take it anyway.

[salehsedghpour]

Yeah sure, I'll test it, I might have time for it next week.

[salehsedghpour]

I've tested it and it works perfectly. I believe the use of elasticsearch_indices_docs_primary instead of elasticsearch_indices_docs_total would be more precise, as the primary shard should get the and validate the values and then forward to other replicas. What do you think @lunkan93?

[lunkan93]

I've tested it and it works perfectly. I believe the use of elasticsearch_indices_docs_primary instead of elasticsearch_indices_docs_total would be more precise, as the primary shard should get the and validate the values and then forward to other replicas. What do you think @lunkan93?

Nice! 👍

I agree that it makes more sense to use elasticsearch_indices_docs_primary 😄

[aarnq]

LGTM, cannot approve though as I opened it 😅