-
Notifications
You must be signed in to change notification settings - Fork 492
Description
Component(s)
No response
What's wrong?
We've deployed Alloy-Mixin into our cluster and noticed that it reports ClusterConfigurationDrift. We use a mostly unmodified default configuration from k8s-monitoring and found the culprit to be about alloy-logs and alloy-singleton having different configuration hashes but tried to be combined.
Solution
One possible solution would be to include the app in the alert which would group only the right values together. Maybe it's even considered a bug in the k8s-monitoring deployment where the jobs should have different names.
Steps to reproduce
alloy/operations/alloy-mixin/alerts/clustering.libsonnet
Lines 81 to 84 in 35d1f1c
| if enableK8sCluster then ||| | |
| count without (sha256) ( | |
| max by (cluster, namespace, sha256, job, cluster_name) (alloy_config_hash and on(cluster, namespace, job, cluster_name) cluster_node_info) | |
| ) > 1 |
Here the metric is split up by cluster, namespace, job, cluster_name but alloy logs is not clustered and therefore both metrics have no cluster_name. Cluster, Namespace and Job are equal which results in the alert being fired. I'm unsure if the job label should be different but in the k8s-monitoring installation it's always set up to be integrations/alloy.
System information
No response
Software version
Grafana Allox Mixin v1.11.3
Configuration
Logs
{cluster="A", job="integrations/alloy", namespace="k8s-monitoring", sha256="7b8b51944ee4f949a012075b74ace00479747481c22f90d3bb555360dfa12955"}
1
{cluster="B", job="integrations/alloy", namespace="k8s-monitoring", sha256="70bce4407dc824c29512c016afd89f07b9c9abf3e268df05bb138ba9259ca6d5"}
1
{cluster="C", job="integrations/alloy", namespace="k8s-monitoring", sha256="2153cfe2c9267a8115e9bace94f4b612a84523766da1ce790e8f8ec761c2384d"}
1
{cluster="A", job="integrations/alloy", namespace="k8s-monitoring", sha256="938afc2bda065c4f0d132d807afeba6b6f87b669175a334c7aa18217c8ad1827"}
1
{cluster="B", job="integrations/alloy", namespace="k8s-monitoring", sha256="43cd6f37aaba8df6aff6c25596a690eb621eefc4eb8a03695d5a76f5c5eec1d7"}
1
{cluster="C", job="integrations/alloy", namespace="k8s-monitoring", sha256="34b34bd3f266705d6795312c03065c91b010715687c331b2555e771dcb6e5ff1"}
1
Tip
React with 👍 if this issue is important to you.