Skip to content

Commit e0e4f14

Browse files
author
Noah Perks Sloan
committed
feat: enable customresource metrics by configuration
The main use case is just surfacing fields from custom resources without having to write a lot of code for each one.
1 parent 7e28ed2 commit e0e4f14

File tree

10 files changed

+1440
-2
lines changed

10 files changed

+1440
-2
lines changed

‎docs/README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,10 @@ sum(kube_pod_container_resource_requests{resource="memory"}) by (namespace, pod,
7878
* on (namespace, pod) group_left() (sum(kube_pod_status_phase{phase="Running"}) by (pod, namespace) == 1)
7979
```
8080

81+
## Metrics from Custom Resources
82+
83+
See [Custom Resource State Metrics](customresourcestate-metrics.md) for more information.
84+
8185
## CLI Arguments
8286

8387
Additionally, options for `kube-state-metrics` can be passed when executing as a CLI, or in a kubernetes / openshift environment. More information can be found here: [CLI Arguments](cli-arguments.md)

‎docs/customresourcestate-metrics.md

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
# Custom Resource State Metrics
2+
3+
This section describes how to add metrics based on the state of a custom resource without writing a custom resource
4+
registry and running your own build of KSM.
5+
6+
## Configuration
7+
8+
The `KSM_CUSTOM_RESOURCE_METRICS` environment variable can be used to configure the metrics for custom resources:
9+
10+
```yaml
11+
apiVersion: apps/v1
12+
kind: Deployment
13+
metadata:
14+
name: kube-state-metrics
15+
namespace: kube-system
16+
spec:
17+
template:
18+
spec:
19+
containers:
20+
- name: kube-state-metrics
21+
env:
22+
- name: KSM_CUSTOM_RESOURCE_METRICS
23+
value: |
24+
spec:
25+
resources:
26+
- groupVersionKind:
27+
group: myteam.io
28+
version: "v1"
29+
kind: Foo
30+
metrics:
31+
- name: active_count
32+
help: "Count of active Foo"
33+
...
34+
```
35+
36+
### Examples
37+
38+
The examples in this section will use the following custom resource:
39+
40+
```yaml
41+
kind: Foo
42+
apiVersion: myteam.io/vl
43+
metadata:
44+
annotations:
45+
bar: baz
46+
qux: quxx
47+
labels:
48+
foo: bar
49+
name: foo
50+
spec:
51+
order:
52+
- id: 1
53+
value: true
54+
- id: 3
55+
value: false
56+
replicas: 1
57+
status:
58+
active:
59+
type-a: 1
60+
type-b: 3
61+
conditions:
62+
- name: a
63+
value: 45
64+
- name: b
65+
value: 66
66+
sub:
67+
type-a:
68+
active: 1
69+
ready: 2
70+
type-b:
71+
active: 3
72+
ready: 4
73+
uptime: 43.21
74+
```
75+
76+
#### Single Values
77+
78+
```yaml
79+
kind: CustomResourceStateMetrics
80+
spec:
81+
resources:
82+
- groupVersionKind:
83+
group: myteam.io
84+
kind: "Foo"
85+
version: "v1"
86+
metrics:
87+
- name: "uptime"
88+
help: "Foo uptime"
89+
each:
90+
path: [status, uptime]
91+
```
92+
93+
Produces the metric:
94+
95+
```prometheus
96+
kube_myteam_io_v1_Foo_uptime 43.21
97+
```
98+
99+
#### Multiple Metrics/Kitchen Sink
100+
101+
```yaml
102+
kind: CustomResourceStateMetrics
103+
spec:
104+
resources:
105+
- groupVersionKind:
106+
group: myteam.io
107+
kind: "Foo"
108+
version: "v1"
109+
# labels can be added to all metrics from a resource
110+
commonLabels:
111+
crd_type: "foo"
112+
labelsFromPath:
113+
name: [metadata, name]
114+
metrics:
115+
- name: "ready_count"
116+
help: "Number Foo Bars ready"
117+
each:
118+
# targeting an object or array will produce a metric for each element
119+
# labelsFromPath and value are relative to this path
120+
path: [status, sub]
121+
122+
# if path targets an object, the object key will be used as label value
123+
labelFromKey: type
124+
# label values can be resolved specific to this path
125+
labelsFromPath:
126+
active: [active]
127+
# The actual field to use as metric value. Should be a number.
128+
value: [ready]
129+
commonLabels:
130+
custom_metric: "yes"
131+
labelsFromPath:
132+
# whole objects may be copied into labels by prefixing with "*"
133+
# *anything will be copied into labels, with the highest sorted * strings first
134+
"*": [metadata, labels]
135+
"**": [metadata, annotations]
136+
137+
# or specific fields may be copied. these fields will always override values from *s
138+
name: [metadata, name]
139+
foo: [metadata, labels, foo]
140+
```
141+
142+
Produces the following metrics:
143+
144+
```prometheus
145+
kube_myteam_io_v1_Foo_active_count{active="1",custom_metric="yes",foo="bar",name="foo",bar="baz",qux="quxx",type="type-a"} 1
146+
kube_myteam_io_v1_Foo_active_count{active="3",custom_metric="yes",foo="bar",name="foo",bar="baz",qux="quxx",type="type-b"} 3
147+
```
148+
149+
### Naming
150+
151+
The default metric names are prefixed to avoid collisions with other metrics.
152+
By default a namespace of `kube` and a subsystem based on your custom resource's GVK is used.
153+
You can override these with the namespace and subsystem fields.
154+
155+
```yaml
156+
kind: CustomResourceStateMetrics
157+
spec:
158+
resources:
159+
- groupVersionKind: ...
160+
namespace: myteam
161+
subsystem: foos
162+
metrics:
163+
- name: uptime
164+
...
165+
```
166+
167+
Produces:
168+
```prometheus
169+
myteam_foos_uptime 43.21
170+
```
171+
172+
To omit namespace and/or subsystem altogether, set them to `_`.
173+
174+
### Logging
175+
176+
If a metric path is registered but not found on a custom resource, an error will be logged. For some resources,
177+
this may produce a lot of noise. The error log [verbosity][vlog] for a metric or resource can be set with `errorLogV` on
178+
the resource or metric:
179+
180+
```yaml
181+
kind: CustomResourceStateMetrics
182+
spec:
183+
resources:
184+
- groupVersionKind: ...
185+
errorLogV: 0 # 0 = default for errors
186+
metrics:
187+
- name: uptime
188+
errorLogV: 10 # only log at high verbosity
189+
```
190+
191+
[vlog]: https://github.com/go-logr/logr#why-v-levels
192+
193+
### Path Syntax
194+
195+
Paths are specified as a list of strings. Each string is a path segment, resolved dynamically against the data of the custom resource.
196+
If any part of a path is missing, the result is nil.
197+
198+
Examples:
199+
200+
```yaml
201+
# simple path lookup
202+
[spec, replicas] # spec.replicas == 1
203+
204+
# indexing an array
205+
[spec, order, "0", value] # spec.order[0].value = true
206+
207+
# finding an element in a list by key=value
208+
[status, conditions, "[name=a]", value] # status.conditions[0].value = 45
209+
210+
# if the value to be matched is a number or boolean, the value is compared as a number or boolean
211+
[status, conditions, "[value=66]", name] # status.conditions[1].name = "b"
212+
```

‎go.mod

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ require (
44
github.com/brancz/gojsontoyaml v0.1.0
55
github.com/campoy/embedmd v1.0.0
66
github.com/dgryski/go-jump v0.0.0-20211018200510-ba001c3ffce0
7+
github.com/gobuffalo/flect v0.2.5
78
github.com/google/go-cmp v0.5.8
89
github.com/google/go-jsonnet v0.18.0
910
github.com/jsonnet-bundler/jsonnet-bundler v0.4.1-0.20200708074244-ada055a225fa
@@ -15,7 +16,9 @@ require (
1516
github.com/prometheus/exporter-toolkit v0.7.1
1617
github.com/robfig/cron/v3 v3.0.1
1718
github.com/spf13/pflag v1.0.5
19+
github.com/stretchr/testify v1.7.0
1820
golang.org/x/perf v0.0.0-20220411212318-84e58bfe0a7e
21+
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
1922
k8s.io/api v0.24.1
2023
k8s.io/apimachinery v0.24.1
2124
k8s.io/autoscaler/vertical-pod-autoscaler v0.10.0
@@ -43,6 +46,7 @@ require (
4346
github.com/evanphx/json-patch v4.12.0+incompatible // indirect
4447
github.com/fatih/color v1.10.0 // indirect
4548
github.com/form3tech-oss/jwt-go v3.2.3+incompatible // indirect
49+
github.com/fsnotify/fsnotify v1.5.1 // indirect
4650
github.com/ghodss/yaml v1.0.0 // indirect
4751
github.com/go-kit/log v0.2.1 // indirect
4852
github.com/go-logfmt/logfmt v0.5.1 // indirect
@@ -61,11 +65,13 @@ require (
6165
github.com/mailru/easyjson v0.7.6 // indirect
6266
github.com/mattn/go-colorable v0.1.8 // indirect
6367
github.com/mattn/go-isatty v0.0.12 // indirect
64-
github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
68+
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
6569
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
6670
github.com/modern-go/reflect2 v1.0.2 // indirect
6771
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
6872
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f // indirect
73+
github.com/onsi/ginkgo v1.16.5 // indirect
74+
github.com/onsi/gomega v1.17.0 // indirect
6975
github.com/pmezard/go-difflib v1.0.0 // indirect
7076
github.com/prometheus/procfs v0.7.3 // indirect
7177
golang.org/x/crypto v0.0.0-20220315160706-3147a52a75dd // indirect

‎main.go

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,13 @@ import (
2020
"context"
2121
"fmt"
2222
"os"
23+
"strings"
2324

2425
"github.com/prometheus/common/version"
26+
"gopkg.in/yaml.v3"
2527
"k8s.io/klog/v2"
28+
"k8s.io/kube-state-metrics/v2/pkg/customresource"
29+
"k8s.io/kube-state-metrics/v2/pkg/customresourcestate"
2630

2731
"k8s.io/kube-state-metrics/v2/pkg/app"
2832
"k8s.io/kube-state-metrics/v2/pkg/options"
@@ -46,8 +50,31 @@ func main() {
4650
os.Exit(0)
4751
}
4852

53+
var factories []customresource.RegistryFactory
54+
if config, set := resolveCustomResourceConfig(); set {
55+
crf, err := customresourcestate.FromConfig(config)
56+
if err != nil {
57+
klog.Fatal(err)
58+
}
59+
factories = append(factories, crf...)
60+
}
61+
4962
ctx := context.Background()
50-
if err := app.RunKubeStateMetrics(ctx, opts); err != nil {
63+
if err := app.RunKubeStateMetrics(ctx, opts, factories...); err != nil {
5164
klog.Fatalf("Failed to run kube-state-metrics: %v", err)
5265
}
5366
}
67+
68+
func resolveCustomResourceConfig() (customresourcestate.ConfigDecoder, bool) {
69+
if s, set := os.LookupEnv("KSM_CUSTOM_RESOURCE_METRICS"); set {
70+
return yaml.NewDecoder(strings.NewReader(s)), true
71+
}
72+
if file, set := os.LookupEnv("KSM_CUSTOM_RESOURCE_METRICS_FILE"); set {
73+
f, err := os.Open(file)
74+
if err != nil {
75+
klog.Fatal(err)
76+
}
77+
return yaml.NewDecoder(f), true
78+
}
79+
return nil, false
80+
}

0 commit comments

Comments
 (0)