KubeClientCertificateExpiration alert false positives with cloud providers #6161

[romankucherov-cmyk]

Fixes #6161

What this PR does / why we need it

Prevents false positives in KubeClientCertificateExpiration alert when using cloud providers that renew certificates exactly 7 days before expiration.

Problem

Cloud providers like DigitalOcean automatically renew certificates 7 days before expiration, causing the alert to trigger during normal maintenance operations.

Solution

Change warning threshold from 7 days (604,800s) to 6 days 22 hours (601,200s) to create a 2-hour buffer.

Changes

• Single line change: 604800 → 601200 in alert rule
• Prevents false positives during cloud provider maintenance
• Maintains security monitoring for real certificate issues
• Critical alert (<24h) remains unchanged for emergencies

Testing

• Mathematically ensures cloud provider maintenance won't trigger warnings
• Still provides 6+ days warning for real certificate issues
• Backward compatible - no breaking changes

[jkroepke]

Please add a comment here

[jkroepke]

Please add more context/comments here.

[jkroepke]

The replacement doesn't hit. I can't see any visible notes in charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-system-apiserver.yaml.

And please bump the chart version, thanks!

[romankucherov-cmyk]

The replacement doesn't hit. I can't see any visible notes in charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-system-apiserver.yaml.

And please bump the chart version, thanks!

Added comments to fsSelector and kubeClientCertificateExpiration in both values.yaml and sync_prometheus_rules.py. Verified with helm template — these values are currently not used in any rendered templates, so the replacement doesn’t hit. Left them documented for future/compatibility reasons. Also bumped the chart version.

[jkroepke]

future/compatibility reasons

Ok, I would leave the PR open until the replacements hits.