Skip to content

alloy mixin alerting for cluster drift broken #4945

@tobiaskohlbau

Description

@tobiaskohlbau

Component(s)

No response

What's wrong?

We've deployed Alloy-Mixin into our cluster and noticed that it reports ClusterConfigurationDrift. We use a mostly unmodified default configuration from k8s-monitoring and found the culprit to be about alloy-logs and alloy-singleton having different configuration hashes but tried to be combined.

Solution

One possible solution would be to include the app in the alert which would group only the right values together. Maybe it's even considered a bug in the k8s-monitoring deployment where the jobs should have different names.

Steps to reproduce

if enableK8sCluster then |||
count without (sha256) (
max by (cluster, namespace, sha256, job, cluster_name) (alloy_config_hash and on(cluster, namespace, job, cluster_name) cluster_node_info)
) > 1

Here the metric is split up by cluster, namespace, job, cluster_name but alloy logs is not clustered and therefore both metrics have no cluster_name. Cluster, Namespace and Job are equal which results in the alert being fired. I'm unsure if the job label should be different but in the k8s-monitoring installation it's always set up to be integrations/alloy.

System information

No response

Software version

Grafana Allox Mixin v1.11.3

Configuration


Logs

{cluster="A", job="integrations/alloy", namespace="k8s-monitoring", sha256="7b8b51944ee4f949a012075b74ace00479747481c22f90d3bb555360dfa12955"}
1

{cluster="B", job="integrations/alloy", namespace="k8s-monitoring", sha256="70bce4407dc824c29512c016afd89f07b9c9abf3e268df05bb138ba9259ca6d5"}
1

{cluster="C", job="integrations/alloy", namespace="k8s-monitoring", sha256="2153cfe2c9267a8115e9bace94f4b612a84523766da1ce790e8f8ec761c2384d"}
1

{cluster="A", job="integrations/alloy", namespace="k8s-monitoring", sha256="938afc2bda065c4f0d132d807afeba6b6f87b669175a334c7aa18217c8ad1827"}
1

{cluster="B", job="integrations/alloy", namespace="k8s-monitoring", sha256="43cd6f37aaba8df6aff6c25596a690eb621eefc4eb8a03695d5a76f5c5eec1d7"}
1

{cluster="C", job="integrations/alloy", namespace="k8s-monitoring", sha256="34b34bd3f266705d6795312c03065c91b010715687c331b2555e771dcb6e5ff1"}
1

Tip

React with 👍 if this issue is important to you.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions