feat: Add join label functionality to alerts

[adberger]

This PR intends to get the work of #792 merged.
Fixes #791

[adberger]

Anyone could take a look at this?

[michel-numan]

Much needed update!

[skl]

lgtm for merging @povilasv

There's no change to the alert rules by default here, users need to explicity modify the xxx_join_labels config in order to enable this feature. PromQL looks good. So seems safe enough.

The only potential downside I can see, in a multi-cluster environemnt, for users that want to take advantage of this feature, the relevant KSM label metrics should be scraped for all clusters, otherwise (take pods for example) pods in clusters with missing kube_pod_labels will be dropped from the alerts because the join * on (...) kube_pod_labels will be empty for those clusters that do not scrape that metric for the affected pods.

This means if you have 3 clusters, and only one is configured to scrape kube_pod_labels, you'll never see KubePodCrashLooping alert for the other 2 clusters (they'll be silently dropped from the alert query).

Kinda convoluted scenario but maybe worth an additional comment in config.libsonnet @adberger.

[skl]

@adberger looks like a whitespace issue caused the lint check to fail

[adberger]

@adberger looks like a whitespace issue caused the lint check to fail

Will fix

[adberger]

@skl I've made the 2 changes, could you take another look?
Also do you prefer rebasing into 1 commit?

[skl]

Yeah, feel free to rebase/squash @adberger, the PRs on this repo seem to be normal merge commits so a single commit makes the history a little easier to follow 👍