[Enhancement]: OpenTelemetry based dashboards, recording/alert rules

[kristeey]

What's the general idea for the enhancement?

As OpenTelemetry (OTEL) collectors now have k8scluster, kubelet and hostmetrics receivers that fetches k8s system/api metrics that are more or less equivalent to the old-school (prometheus based) kube-state-metrics, kublet/cadvisor and node-exporter metrics, it would be nice to have new mixins that uses the OTEL based metrics. As of now I have not found any issues/discussions anywhere about this topic of providing community backed dashboards and recording/alert rules reflecting the new metrics.
This would give a more seamless experience and help with the adoption of OTEL as well as reducing configuration complexity.

There is not a one-2-one mapping of metric names between the otel based system/api metrics and the old-school metrics. For instance:
k8s.job.active_pods otel metric (or k8s_job_active_pods when rewritten to prometheus format) from the k8scluster receiver is basically the same as kube_job_status_active metric from the kube-state-metrics project.

There are also some metrics that do not map at all, and probably metrics that are completely new. Hence, I belive there is a need of new mixins.

Please provide any helpful snippets.

What parts of the codebase does the enhancement target?

Alerts, Rules, Dashboards

Anything else relevant to the enhancement that would help with the triage process?

No response

I agree to the following terms:

• I agree to follow this project's Code of Conduct.
• I have filled out all the required information above to the best of my ability.
• I have searched the issues of this repository and believe that this is not a duplicate.
• I have confirmed this proposal applies to the default branch of the repository, as of the latest commit at the time of submission.

[skl]

There was a similar topic recently from @Syndlex:

• join forces with dotdc/grafana-dashboards-kubernetes and provide #1067
• [kube-prometheus-stack] hello from kubernetes-mixin! 👋 prometheus-community/helm-charts#5735
• [enhancement] Opentelemetry exporter Support dotdc/grafana-dashboards-kubernetes#153

I'm open to integrating OTel metrics into the dashboards, though the recording and alert rules should probably be kept separate (or at least configurable, such that the default config excludes OTel metrics from rules).

There is a fair amount of work though, so a coordinated effort would help. For example:

1. Would we continue with the existing dashboards here or look to adapt and adopt the dotdc dashboards? The latter is a much more significant change for this repo, and would signify a major release (v2). It would have potentially much more impact on downstream projects, which rely on the existing dashboards (kube-prometheus-stack, cloud vendors etc).
2. Do users expect to see both OTel and Prometheus data coexisting within the same dashboards/panels/rules, or should they be kept explicitly separate?

It would be relatively low-friction to start creating OTel versions of the existing dashboards here, perhaps in a separate folder and default not included... arguably you could say including the dotc dashboards into that new folder is also significantly easier (rather than replacing the existing dashboards, users could have a choice of either/or/both).

Thinking aloud here... feedback welcome.

[kristeey]

I like the idea of separating it in different folders, as I think that most people would choose either fully otel native dashboards or the other way around based on how they gather their system/api metrics. If there was a complete set of otel based dashboards and rules, an otel user would probably don't bother with installing node-exporter and kube-state-metrics in favour of the native otel receivers, unless (as you mention) you are using downstream projects which rely on the old metrics in their own dashboards/rules (i.e mimir). One could argue that if you need both of the mixins (prom based and otel based) you could do so by installing them separately. I love the idea of joining forces with dotdc which provide dashboards that easily can be installed with tools like grafana operator etc.

[skl]

FYI I've posted in the #otel-helm channel of the CNCF Slack to see if they'd be open to contributions on the opentelemetry-kube-stack helm chart.

I'm getting the feeling that a project separate from this repo might be a cleaner approach for OTel support, and that would naturally fit within the OTel community.

[kristeey]

Amazing! I am a user of that helm chart (opentelemetry-helm-chart) so I'm eager to hear what direction this is going to!

[github-actions]

This issue has not had any activity in the past 30 days, so the
stale label has been added to it.

• The stale label will be removed if there is new activity
• The issue will be closed in 7 days if there is no new activity
• Add the keepalive label to exempt this issue from the stale check action

Thank you for your contributions!

[kristeey]

Is this just a loose end or has there been any development in this case?

[Syndlex]

Hey, I have working dashboards for otel (from the dotdc repo)
I currently don't have time to start developing those into jsonnet.
I can provide them here as a staring point if someone else is willing to start development.
Unfortunately I currently don't have a lot of time or resources to do it myself.