fix: use rabbitmq length for RabbitMQNodeDown

[jackhodgkiss]

The RabbitMQNodeDown made the assumption that all deployments involve three controllers. However, this is not always the case as we do support deployments with a single controller or more than three controllers.

Before this would have caused false alerts in deployments with a single controller. Whilst also concealing alerts in deployments with more than three controllers.

[MoteHue]

LGTM now, thanks!

[jackhodgkiss]

This fails to template correctly.

  - alert: RabbitMQNodeDown
    expr: sum(rabbitmq_build_info{instance!=""}) < {{ groups['rabbitmq'] | length }}
    for: 30m
    labels:

[Alex-Welsh]

Kolla-Ansible uses copy, not template, for rules files [1], so they can either be hard-coded or templated by Kayobe.

Possible Kayobe groups are: all, ungrouped, seed, seed-hypervisor, container-image-builders, hypervisors, infra-vms, wazuh-manager, wazuh-agent, github-runners, github-writer, controllers, network, monitoring, storage, compute-vgpu, compute, overcloud, vgpu, iommu, mlnx, docker, docker-registry, ntp, baremetal-compute, mgmt-switches, ctl-switches, hs-switches, switches, ceph, mons, mgrs, osds, rgws, cis-hardening, redfish_exporter_targets, fix-hostname, tempest_runner, controllers_with_ironic_enabled_False

Short term I'd say we make a new variable in SKC and default it to the length of the controller group, and have a backlog task to make the prometheus rules files templatable in KA
[1] https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/prometheus/tasks/config.yml#L38

[seunghun1ee]

Good idea. Happy to +1 once it's in ready-to-review state

[Alex-Welsh]

Thanks Jack, I think this solution works well. Just need to update the release note