Add Auto-configure curator retention rule for indexPerNamespace

[chi-quita-a]

Warning

This is a public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request, nor
• business confidential information, such as customer names.

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• [kind/adr](set-me)

What does this PR do / why do we need this PR?

Automatically configures an OpenSearch Curator retention rule for all non-system indices when indexPerNamespace is enabled, removing a manual configuration step that is easy to miss.

When indexPerNamespace is enabled, logs are indexed by Kubernetes namespace instead of fixed patterns. This requires a retention rule to clean up non-system indices (^[^.].* pattern). Previously, users had to manually add this rule to their configuration in sc-config.yaml / common-config.yaml.

The Solution:

• Automatically adds the ^[^.].* pattern when opensearch.indexPerNamespace is true.

• Uses configurable defaults exposed in sc-config.yaml:

  • opensearch.curator.indexPerNamespaceDefaultSizeGB → default 5000

  • opensearch.curator.indexPerNamespaceDefaultAgeDays → default 30

• Preserves all existing user-configured retention rules (they are appended after the auto-generated non-system rule).

• Keeps behaviour unchanged when indexPerNamespace is `false.

• Fixes Automatically configure retention for OpenSearch with indexPerNamespace #2724

Information to reviewers

• The change is transparent for environments where indexPerNamespace is disabled.

• When indexPerNamespace is enabled, a previously required manual step is now automated but still configurable via sc-config.yaml.

How to test

Pre-condition: Ensure your environment is on a matching Welkin Apps version (no version mismatch between cluster/config/repository) before running these commands.

Test with indexPerNamespace disabled

Verify current setting (indexPerNamespace is false/not set):

yq '.opensearch.indexPerNamespace' ${CK8S_CONFIG_PATH}/common-config.yaml  

Apply changes

./bin/ck8s apply sc  

Inspect Curator ConfigMap

./bin/ck8s ops kubectl sc get configmap -n opensearch-system opensearch-curator -o yaml

Expected:

• Only the retention rules explicitly configured under opensearch.curator.retention are present.

• No ^[^.].* pattern is injected.

Test with indexPerNamespace enabled:

Enable indexPerNamespace

yq '.opensearch.indexPerNamespace = true' -i ${CK8S_CONFIG_PATH}/sc-config.yaml

Optionally override defaults (to verify configurability):

yq '.opensearch.curator.indexPerNamespaceDefaultSizeGB = 5000' -i ${CK8S_CONFIG_PATH}/sc-config.yaml
yq '.opensearch.curator.indexPerNamespaceDefaultAgeDays = 30'  -i ${CK8S_CONFIG_PATH}/sc-config.yaml

Apply changes

./bin/ck8s apply sc  

Inspect Curator ConfigMap

./bin/ck8s ops kubectl sc get configmap -n opensearch-system opensearch-curator -o yaml

Expected:

• A retention rule with pattern ^[^.].* appears first, using:

  • disk_space = indexPerNamespaceDefaultSizeGB (default 5000)

  • age filter = indexPerNamespaceDefaultAgeDays (default 30)

• All existing user-defined patterns (e.g. kubernetes-*, kubeaudit-*, other-*, authlog-*, security-auditlog-*) are still present and evaluated after the non-system rule.

Verify curator CronJob exists and is valid::

./bin/ck8s ops kubectl sc get cronjob -n opensearch-system opensearch-curator

Expected behavior:

• When indexPerNamespace is false: Only user-configured retention rules are applied
• When indexPerNamespace is true: The ^[^.].* rule is added first with defaults, then user-configured rules
• The curator role already has permissions to delete indices matching the non-system pattern configurer.yaml.gotmpl:134-142 .

Technical Details

Changes include:

• helmfile.d/values/opensearch/curator.yaml.gotmpl

  • Switch from static retention configuration to logic that derives retention rules from:

    • opensearch.curator.retention

    • opensearch.indexPerNamespace

    • opensearch.curator.indexPerNamespaceDefaultSizeGB

    • opensearch.curator.indexPerNamespaceDefaultAgeDays

• helmfile.d/charts/opensearch/curator/values.yaml

  • Extend Curator chart values to support the automatically generated non-system retention rule while preserving user-defined rules.

• config/schemas/config.yaml

  • Add schema entries under opensearch.curator:

    • indexPerNamespaceDefaultSizeGB (number, default 5000)

    • indexPerNamespaceDefaultAgeDays (number, default 30)

  • This ensures the new values are validated, documented, and configurable via sc-config.yaml.

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
  • The change updates CRDs
  • The change updates the config and the schema
• Documentation checks:
  • The public documentation required no updates
  • The public documentation required an update - [link to change](set-me)
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts required no updates)
  • The metrics names did change (Grafana dashboards and Prometheus alerts required an update)
• Logs checks:
  • The logs do not show any errors after the change
• PodSecurityPolicy checks:
  • Any changed Pod is covered by Kubernetes Pod Security Standards
  • Any changed Pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any Pods to be blocked by Pod Security Standards or Policies
• NetworkPolicy checks:
  • Any changed Pod is covered by Network Policies
  • The change does not cause any dropped packets in the NetworkPolicy Dashboard
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Bug checks:
  • The bug fix is covered by regression tests

[AlbinB97]

Really nice work Joy 👍

Added some comments that needs to be addressed.

[lunkan93]

Could you also add this pattern when indexPerNamespace is enabled?

- pattern: .orphaned*
  ageDays: xxx
  sizeGB: yyy

These indexes usually pop up when namespaces are removed, and since they start with . they don't get matched by the ^[^.].* pattern, so they remain indefinitely. It should have the same retention time as the other pattern.

[chi-quita-a]

Could you also add this pattern when indexPerNamespace is enabled?

- pattern: .orphaned*
  ageDays: xxx
  sizeGB: yyy

These indexes usually pop up when namespaces are removed, and since they start with . they don't get matched by the ^[^.].* pattern, so they remain indefinitely. It should have the same retention time as the other pattern.

Sure, I will make an implementation for that pattern.

[chi-quita-a]

Could you also add this pattern when indexPerNamespace is enabled?

- pattern: .orphaned*
  ageDays: xxx
  sizeGB: yyy

These indexes usually pop up when namespaces are removed, and since they start with . they don't get matched by the ^[^.].* pattern, so they remain indefinitely. It should have the same retention time as the other pattern.

retention:
  {{- if .Values.opensearch.indexPerNamespace }}
  - pattern: "^[^.].*"
    sizeGB: {{ .Values.opensearch.curator.indexPerNamespaceDefaultSizeGB }}
    ageDays: {{ .Values.opensearch.curator.indexPerNamespaceDefaultAgeDays }}
  - pattern: ".orphaned*"
    sizeGB: {{ .Values.opensearch.curator.indexPerNamespaceDefaultSizeGB }}
    ageDays: {{ .Values.opensearch.curator.indexPerNamespaceDefaultAgeDays }}
  {{- end }}

[Xartos]

LGTM, Great job 👍 Only nit, I'd change the kind from feature to improvement in the PR description

[shafi-elastisys]

Good work! LGTM 🚀

[AlbinB97]

LGTM, as Fredrik wrote, please change the kind of PR to improvement instead of feature before merging 👍