Kepler deployment using oss demo does not work

[dimanzt]

Dear all,
I have tried the instructions for deploying kepler on Kubernetes cluster (baremetal), but kepler exporter pod does not work.
Here is the outputs of the pod, any idea why this is not working for me?
Thanks,
Diman

$ kubectl logs -f -n monitoring kepler-exporter-lkfx5
I1010 18:17:14.128873 1 gpu_nvml.go:62] found 1 gpu devices
I1010 18:17:14.135447 1 gpu_nvml.go:73] GPU 0 NVIDIA A100X
I1010 18:17:14.135473 1 gpu.go:40] Using nvml to obtain gpu power
E1010 18:17:14.175815 1 utils.go:98] getCPUArch failure: no CPU power model found for architecture AMD Zen 3, 7nm
I1010 18:17:14.196609 1 qat.go:35] Failed to init qat-telemtry err: could not get qat status exit status 127
I1010 18:17:14.209427 1 exporter.go:157] Kepler running on version: b6fbec4
I1010 18:17:14.209454 1 config.go:274] using gCgroup ID in the BPF program: true
I1010 18:17:14.209508 1 config.go:276] kernel version: 5.15
I1010 18:17:14.209656 1 config.go:301] The Idle power will be exposed. Are you running on Baremetal or using single VM per node?
I1010 18:17:14.209672 1 exporter.go:169] LibbpfBuilt: false, BccBuilt: true
I1010 18:17:14.209682 1 exporter.go:188] EnabledBPFBatchDelete: true
I1010 18:17:14.209780 1 redfish.go:169] failed to get redfish credential file path
I1010 18:17:14.300609 1 exporter.go:203] Initializing the GPU collector
I1010 18:17:20.306256 1 watcher.go:66] Using in cluster k8s config
W1010 18:17:20.316794 1 reflector.go:424] github.com/sustainable-computing-io/kepler/pkg/kubernetes/watcher.go:122: failed to list : pods is forbidden: User "system:serviceaccount:monitoring:kepler-sa" cannot list resource "pods" in API group "" at the cluster scope
E1010 18:17:20.316905 1 reflector.go:140] github.com/sustainable-computing-io/kepler/pkg/kubernetes/watcher.go:122: Failed to watch : failed to list : pods is forbidden: User "system:serviceaccount:monitoring:kepler-sa" cannot list resource "pods" in API group "" at the cluster scope
W1010 18:17:21.603025 1 reflector.go:424] github.com/sustainable-computing-io/kepler/pkg/kubernetes/watcher.go:122: failed to list : pods is forbidden: User "system:serviceaccount:monitoring:kepler-sa" cannot list resource "pods" in API group "" at the cluster scope
E1010 18:17:21.603074 1 reflector.go:140] github.com/sustainable-computing-io/kepler/pkg/kubernetes/watcher.go:122: Failed to watch : failed to list : pods is forbidden: User "system:serviceaccount:monitoring:kepler-sa" cannot list resource "pods" in API group "" at the cluster scope
W1010 18:17:24.710974 1 reflector.go:424] github.com/sustainable-computing-io/kepler/pkg/kubernetes/watcher.go:122: failed to list : pods is forbidden: User "system:serviceaccount:monitoring:kepler-sa" cannot list resource "pods" in API group "" at the cluster scope
E1010 18:17:24.711029 1 reflector.go:140] github.com/sustainable-computing-io/kepler/pkg/kubernetes/watcher.go:122: Failed to watch : failed to list : pods is forbidden: User "system:serviceaccount:monitoring:kepler-sa" cannot list resource "pods" in API group "" at the cluster scope
W1010 18:17:30.040417 1 reflector.go:424] github.com/sustainable-computing-io/kepler/pkg/kubernetes/watcher.go:122: failed to list : pods is forbidden: User "system:serviceaccount:monitoring:kepler-sa" cannot list resource "pods" in API group "" at the cluster scope
E1010 18:17:30.040475 1 reflector.go:140] github.com/sustainable-computing-io/kepler/pkg/kubernetes/watcher.go:122: Failed to watch : failed to list : pods is forbidden: User "system:serviceaccount:monitoring:kepler-sa" cannot list resource "pods" in API group "" at the cluster scope
W1010 18:17:39.245634 1 reflector.go:424] github.com/sustainable-computing-io/kepler/pkg/kubernetes/watcher.go:122: failed to list : pods is forbidden: User "system:serviceaccount:monitoring:kepler-sa" cannot list resource "pods" in API group "" at the cluster scope
E1010 18:17:39.245686 1 reflector.go:140] github.com/sustainable-computing-io/kepler/pkg/kubernetes/watcher.go:122: Failed to watch : failed to list : pods is forbidden: User "system:serviceaccount:monitoring:kepler-sa" cannot list resource "pods" in API group "" at the cluster scope
W1010 18:17:57.484238 1 reflector.go:424] github.com/sustainable-computing-io/kepler/pkg/kubernetes/watcher.go:122: failed to list : pods is forbidden: User "system:serviceaccount:monitoring:kepler-sa" cannot list resource "pods" in API group "" at the cluster scope
E1010 18:17:57.484291 1 reflector.go:140] github.com/sustainable-computing-io/kepler/pkg/kubernetes/watcher.go:122: Failed to watch : failed to list : pods is forbidden: User "system:serviceaccount:monitoring:kepler-sa" cannot list resource "pods" in API group "" at the cluster scope
F1010 18:18:20.307094 1 watcher.go:129] watcher timed out waiting for caches to sync

$ kubectl describe pods -n monitoring kepler-exporter-lkfx5
Name: kepler-exporter-lkfx5
Namespace: monitoring
Priority: 0
Service Account: kepler-sa
Node: hsc-12.labs.hpecorp.net/10.93.244.72
Start Time: Mon, 09 Oct 2023 16:52:17 -0700
Labels: app.kubernetes.io/component=exporter
app.kubernetes.io/name=kepler-exporter
controller-revision-hash=7844d65b7b
pod-template-generation=1
Annotations:
Status: Running
IP: 10.93.244.72
IPs:
IP: 10.93.244.72
Controlled By: DaemonSet/kepler-exporter
Containers:
kepler-exporter:
Container ID: containerd://157a9e3b2a2d447ced45033c90e41e3bfd88c0ccc533530fafd9712e978b64f4
Image: quay.io/sustainable_computing_io/kepler:latest
Image ID: quay.io/sustainable_computing_io/kepler@sha256:1b86884317493fbd48a477a8d8f6e92d51ae368d0c0dea4470854eb55f95e231
Port: 9102/TCP
Host Port: 9102/TCP
Command:
/usr/bin/kepler
-address
0.0.0.0:9102
-enable-gpu
true
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Tue, 10 Oct 2023 11:17:13 -0700
Finished: Tue, 10 Oct 2023 11:18:20 -0700
Ready: False
Restart Count: 181
Environment:
NODE_NAME: (v1:spec.nodeName)
Mounts:
/lib/modules from lib-modules (rw)
/sys from tracing (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wdvmj (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType: Directory
tracing:
Type: HostPath (bare host directory volume)
Path: /sys
HostPathType: Directory
kube-api-access-wdvmj:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message

---

Normal Pulled 55m (x165 over 17h) kubelet (combined from similar events): Successfully pulled image "quay.io/sustainable_computing_io/kepler:latest" in 395.284502ms (395.306773ms including waiting)
Warning BackOff 31s (x4165 over 18h) kubelet Back-off restarting failed container kepler-exporter in pod kepler-exporter-lkfx5_monitoring(561b3fa4-f9a0-4258-a2fa-e1b4c52c2fca)

[rootfs]

The OSS demo is really old. Please refer to the doc and find way of installation that fit your environment
https://sustainable-computing.io/

[dimanzt]

Thanks for the reply,
I followed the general instruction as well and still can't see any metrics the kepler exporter on grafana

Here is my pods:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-flannel kube-flannel-ds-wchtf 1/1 Running 0 26h 10.93.226.123 sepehr-1
kube-system coredns-5dd5756b68-ln5fz 1/1 Running 0 26h 10.244.0.29 sepehr-1
kube-system coredns-5dd5756b68-mz8vz 1/1 Running 0 26h 10.244.0.28 sepehr-1
kube-system etcd-sepehr-1 1/1 Running 819 26h 10.93.226.123 sepehr-1
kube-system kube-apiserver-sepehr-1 1/1 Running 0 26h 10.93.226.123 sepehr-1
kube-system kube-controller-manager-sepehr-1 1/1 Running 0 26h 10.93.226.123 sepehr-1
kube-system kube-proxy-mrmd5 1/1 Running 0 26h 10.93.226.123 sepehr-1
kube-system kube-scheduler-sepehr-1 1/1 Running 832 26h 10.93.226.123 sepehr-1
monitoring alertmanager-main-0 2/2 Running 0 46m 10.244.0.36 sepehr-1
monitoring alertmanager-main-1 2/2 Running 0 46m 10.244.0.37 sepehr-1
monitoring alertmanager-main-2 2/2 Running 0 46m 10.244.0.38 sepehr-1
monitoring blackbox-exporter-d9597b5ff-gn2sx 3/3 Running 0 51m 10.244.0.30 sepehr-1
monitoring grafana-64c5895b4-rzh4b 1/1 Running 0 51m 10.244.0.31 sepehr-1
monitoring kube-state-metrics-674c5fc7f8-9kwhf 3/3 Running 0 51m 10.244.0.34 sepehr-1
monitoring node-exporter-pkh4x 2/2 Running 0 51m 10.93.226.123 sepehr-1
monitoring prometheus-adapter-7cc789bfcc-gq2wh 1/1 Running 0 51m 10.244.0.35 sepehr-1
monitoring prometheus-adapter-7cc789bfcc-vpb5p 1/1 Running 0 51m 10.244.0.32 sepehr-1
monitoring prometheus-k8s-0 2/2 Running 0 46m 10.244.0.39 sepehr-1
monitoring prometheus-k8s-1 2/2 Running 0 46m 10.244.0.40 sepehr-1
monitoring prometheus-operator-749b97889c-w8ngb 2/2 Running 0 51m 10.244.0.33 sepehr-1

and this is what I am seeing on the grafana dashboard.

I'm not sure how to see the power metrics without the grafana dashboard, the instructions doesn't talk about it.

[sthaha]

I am only guess based on limited information about the setup.
It seems to me that Prometheus may not be discovering kepler service. Can you ensure that prometheus has the required RBAC?
E.g. kepler-operator development env setup applies these files - https://github.com/sustainable-computing-io/kepler-operator/tree/v1alpha1/hack/monitoring/rbac

[marceloamaral]

@dimanzt could you please detail how did you install Kepler?

I cannot see the kepler namespace in you new deployment.

But, if you can see the kepler pod, can you share the kepler log?

[dimanzt]

Thanks @sthaha and @marceloamaral it seems like this demo was old and I used the kepler website to deploy kepler and now it works for me

[sthaha]

Good to know that this is resolved. Closing the discussion :)