Skip to content

Commit 1e78285

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #1710 from hack-edu/master
feat: enable customresource metrics by configuration
2 parents 0c6afa9 + e7725f9 commit 1e78285

File tree

13 files changed

+1545
-45
lines changed

13 files changed

+1545
-45
lines changed

‎docs/README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,10 @@ sum(kube_pod_container_resource_requests{resource="memory"}) by (namespace, pod,
7878
* on (namespace, pod) group_left() (sum(kube_pod_status_phase{phase="Running"}) by (pod, namespace) == 1)
7979
```
8080

81+
## Metrics from Custom Resources
82+
83+
See [Custom Resource State Metrics](customresourcestate-metrics.md) for experimental support for custom resources.
84+
8185
## CLI Arguments
8286

8387
Additionally, options for `kube-state-metrics` can be passed when executing as a CLI, or in a kubernetes / openshift environment. More information can be found here: [CLI Arguments](cli-arguments.md)

‎docs/cli-arguments.md

Lines changed: 38 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -25,40 +25,42 @@ spec:
2525
```txt
2626
$ kube-state-metrics -h
2727
Usage of ./kube-state-metrics:
28-
--add_dir_header If true, adds the file directory to the header of the log messages
29-
--alsologtostderr log to standard error as well as files
30-
--apiserver string The URL of the apiserver to use as a master
31-
--enable-gzip-encoding Gzip responses when requested by clients via 'Accept-Encoding: gzip' header.
32-
-h, --help Print Help text
33-
--host string Host to expose metrics on. (default "::")
34-
--kubeconfig string Absolute path to the kubeconfig file
35-
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
36-
--log_dir string If non-empty, write log files in this directory
37-
--log_file string If non-empty, use this log file
38-
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
39-
--logtostderr log to standard error instead of files (default true)
40-
--metric-allowlist string Comma-separated list of metrics to be exposed. This list comprises of exact metric names and/or regex patterns. The allowlist and denylist are mutually exclusive.
41-
--metric-annotations-allowlist string Comma-separated list of Kubernetes annotations keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional annotations provide a list of resource names in their plural form and Kubernetes annotation keys you would like to allow for them (Example: '=namespaces=[kubernetes.io/team,...],pods=[kubernetes.io/team],...)'. A single '*' can be provided per resource instead to allow any annotations, but that has severe performance implications (Example: '=pods=[*]').
42-
--metric-denylist string Comma-separated list of metrics not to be enabled. This list comprises of exact metric names and/or regex patterns. The allowlist and denylist are mutually exclusive.
43-
--metric-labels-allowlist string Comma-separated list of additional Kubernetes label keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional labels provide a list of resource names in their plural form and Kubernetes label keys you would like to allow for them (Example: '=namespaces=[k8s-label-1,k8s-label-n,...],pods=[app],...)'. A single '*' can be provided per resource instead to allow any labels, but that has severe performance implications (Example: '=pods=[*]').
44-
--metric-opt-in-list string Comma-separated list of metrics which are opt-in and not enabled by default. This is in addition to the metric allow- and denylists
45-
--namespaces string Comma-separated list of namespaces to be enabled. Defaults to ""
46-
--namespaces-denylist string Comma-separated list of namespaces not to be enabled. If namespaces and namespaces-denylist are both set, only namespaces that are excluded in namespaces-denylist will be used.
47-
--one_output If true, only write logs to their native severity level (vs also writing to each lower severity level)
48-
--pod string Name of the pod that contains the kube-state-metrics container. When set, it is expected that --pod and --pod-namespace are both set. Most likely this should be passed via the downward API. This is used for auto-detecting sharding. If set, this has preference over statically configured sharding. This is experimental, it may be removed without notice.
49-
--pod-namespace string Name of the namespace of the pod specified by --pod. When set, it is expected that --pod and --pod-namespace are both set. Most likely this should be passed via the downward API. This is used for auto-detecting sharding. If set, this has preference over statically configured sharding. This is experimental, it may be removed without notice.
50-
--port int Port to expose metrics on. (default 8080)
51-
--resources string Comma-separated list of Resources to be enabled. Defaults to "certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments"
52-
--shard int32 The instances shard nominal (zero indexed) within the total number of shards. (default 0)
53-
--skip_headers If true, avoid header prefixes in the log messages
54-
--skip_log_headers If true, avoid headers when opening log files
55-
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
56-
--telemetry-host string Host to expose kube-state-metrics self metrics on. (default "::")
57-
--telemetry-port int Port to expose kube-state-metrics self metrics on. (default 8081)
58-
--tls-config string Path to the TLS configuration file
59-
--total-shards int The total number of shards. Sharding is disabled when total shards is set to 1. (default 1)
60-
--use-apiserver-cache Sets resourceVersion=0 for ListWatch requests, using cached resources from the apiserver instead of an etcd quorum read.
61-
-v, --v Level number for the log level verbosity
62-
--version kube-state-metrics build version information
63-
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
28+
--add_dir_header If true, adds the file directory to the header of the log messages
29+
--alsologtostderr log to standard error as well as files
30+
--apiserver string The URL of the apiserver to use as a master
31+
--custom-resource-state-config string Inline Custom Resource State Metrics config YAML (experimental)
32+
--custom-resource-state-config-file string Path to a Custom Resource State Metrics config file (experimental)
33+
--enable-gzip-encoding Gzip responses when requested by clients via 'Accept-Encoding: gzip' header.
34+
-h, --help Print Help text
35+
--host string Host to expose metrics on. (default "::")
36+
--kubeconfig string Absolute path to the kubeconfig file
37+
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
38+
--log_dir string If non-empty, write log files in this directory
39+
--log_file string If non-empty, use this log file
40+
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
41+
--logtostderr log to standard error instead of files (default true)
42+
--metric-allowlist string Comma-separated list of metrics to be exposed. This list comprises of exact metric names and/or regex patterns. The allowlist and denylist are mutually exclusive.
43+
--metric-annotations-allowlist string Comma-separated list of Kubernetes annotations keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional annotations provide a list of resource names in their plural form and Kubernetes annotation keys you would like to allow for them (Example: '=namespaces=[kubernetes.io/team,...],pods=[kubernetes.io/team],...)'. A single '*' can be provided per resource instead to allow any annotations, but that has severe performance implications (Example: '=pods=[*]').
44+
--metric-denylist string Comma-separated list of metrics not to be enabled. This list comprises of exact metric names and/or regex patterns. The allowlist and denylist are mutually exclusive.
45+
--metric-labels-allowlist string Comma-separated list of additional Kubernetes label keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional labels provide a list of resource names in their plural form and Kubernetes label keys you would like to allow for them (Example: '=namespaces=[k8s-label-1,k8s-label-n,...],pods=[app],...)'. A single '*' can be provided per resource instead to allow any labels, but that has severe performance implications (Example: '=pods=[*]').
46+
--metric-opt-in-list string Comma-separated list of metrics which are opt-in and not enabled by default. This is in addition to the metric allow- and denylists
47+
--namespaces string Comma-separated list of namespaces to be enabled. Defaults to ""
48+
--namespaces-denylist string Comma-separated list of namespaces not to be enabled. If namespaces and namespaces-denylist are both set, only namespaces that are excluded in namespaces-denylist will be used.
49+
--one_output If true, only write logs to their native severity level (vs also writing to each lower severity level)
50+
--pod string Name of the pod that contains the kube-state-metrics container. When set, it is expected that --pod and --pod-namespace are both set. Most likely this should be passed via the downward API. This is used for auto-detecting sharding. If set, this has preference over statically configured sharding. This is experimental, it may be removed without notice.
51+
--pod-namespace string Name of the namespace of the pod specified by --pod. When set, it is expected that --pod and --pod-namespace are both set. Most likely this should be passed via the downward API. This is used for auto-detecting sharding. If set, this has preference over statically configured sharding. This is experimental, it may be removed without notice.
52+
--port int Port to expose metrics on. (default 8080)
53+
--resources string Comma-separated list of Resources to be enabled. Defaults to "certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments"
54+
--shard int32 The instances shard nominal (zero indexed) within the total number of shards. (default 0)
55+
--skip_headers If true, avoid header prefixes in the log messages
56+
--skip_log_headers If true, avoid headers when opening log files
57+
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
58+
--telemetry-host string Host to expose kube-state-metrics self metrics on. (default "::")
59+
--telemetry-port int Port to expose kube-state-metrics self metrics on. (default 8081)
60+
--tls-config string Path to the TLS configuration file
61+
--total-shards int The total number of shards. Sharding is disabled when total shards is set to 1. (default 1)
62+
--use-apiserver-cache Sets resourceVersion=0 for ListWatch requests, using cached resources from the apiserver instead of an etcd quorum read.
63+
-v, --v Level number for the log level verbosity
64+
--version kube-state-metrics build version information
65+
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
6466
```

‎docs/customresourcestate-metrics.md

Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
# Custom Resource State Metrics
2+
3+
This section describes how to add metrics based on the state of a custom resource without writing a custom resource
4+
registry and running your own build of KSM.
5+
6+
## Configuration
7+
8+
A YAML configuration file described below is required to define your custom resources and the fields to turn into metrics.
9+
10+
Two flags can be used:
11+
12+
* `--custom-resource-state-config "inline yaml (see example)"` or
13+
* `--custom-resource-state-config-file /path/to/config.yaml`
14+
15+
If both flags are provided, the inline configuration will take precedence.
16+
17+
```yaml
18+
apiVersion: apps/v1
19+
kind: Deployment
20+
metadata:
21+
name: kube-state-metrics
22+
namespace: kube-system
23+
spec:
24+
template:
25+
spec:
26+
containers:
27+
- name: kube-state-metrics
28+
args:
29+
- --custom-resource-state-config
30+
# in YAML files, | allows a multi-line string to be passed as a flag value
31+
# see https://yaml-multiline.info
32+
- |
33+
spec:
34+
resources:
35+
- groupVersionKind:
36+
group: myteam.io
37+
version: "v1"
38+
kind: Foo
39+
metrics:
40+
- name: active_count
41+
help: "Count of active Foo"
42+
...
43+
```
44+
45+
### Examples
46+
47+
The examples in this section will use the following custom resource:
48+
49+
```yaml
50+
kind: Foo
51+
apiVersion: myteam.io/vl
52+
metadata:
53+
annotations:
54+
bar: baz
55+
qux: quxx
56+
labels:
57+
foo: bar
58+
name: foo
59+
spec:
60+
order:
61+
- id: 1
62+
value: true
63+
- id: 3
64+
value: false
65+
replicas: 1
66+
status:
67+
active:
68+
type-a: 1
69+
type-b: 3
70+
conditions:
71+
- name: a
72+
value: 45
73+
- name: b
74+
value: 66
75+
sub:
76+
type-a:
77+
active: 1
78+
ready: 2
79+
type-b:
80+
active: 3
81+
ready: 4
82+
uptime: 43.21
83+
```
84+
85+
#### Single Values
86+
87+
The config:
88+
89+
```yaml
90+
kind: CustomResourceStateMetrics
91+
spec:
92+
resources:
93+
- groupVersionKind:
94+
group: myteam.io
95+
kind: "Foo"
96+
version: "v1"
97+
metrics:
98+
- name: "uptime"
99+
help: "Foo uptime"
100+
each:
101+
path: [status, uptime]
102+
```
103+
104+
Produces the metric:
105+
106+
```prometheus
107+
kube_myteam_io_v1_Foo_uptime 43.21
108+
```
109+
110+
#### Multiple Metrics/Kitchen Sink
111+
112+
```yaml
113+
kind: CustomResourceStateMetrics
114+
spec:
115+
resources:
116+
- groupVersionKind:
117+
group: myteam.io
118+
kind: "Foo"
119+
version: "v1"
120+
# labels can be added to all metrics from a resource
121+
commonLabels:
122+
crd_type: "foo"
123+
labelsFromPath:
124+
name: [metadata, name]
125+
metrics:
126+
- name: "ready_count"
127+
help: "Number Foo Bars ready"
128+
each:
129+
# targeting an object or array will produce a metric for each element
130+
# labelsFromPath and value are relative to this path
131+
path: [status, sub]
132+
133+
# if path targets an object, the object key will be used as label value
134+
labelFromKey: type
135+
# label values can be resolved specific to this path
136+
labelsFromPath:
137+
active: [active]
138+
# The actual field to use as metric value. Should be a number.
139+
value: [ready]
140+
commonLabels:
141+
custom_metric: "yes"
142+
labelsFromPath:
143+
# whole objects may be copied into labels by prefixing with "*"
144+
# *anything will be copied into labels, with the highest sorted * strings first
145+
"*": [metadata, labels]
146+
"**": [metadata, annotations]
147+
148+
# or specific fields may be copied. these fields will always override values from *s
149+
name: [metadata, name]
150+
foo: [metadata, labels, foo]
151+
```
152+
153+
Produces the following metrics:
154+
155+
```prometheus
156+
kube_myteam_io_v1_Foo_active_count{active="1",custom_metric="yes",foo="bar",name="foo",bar="baz",qux="quxx",type="type-a"} 1
157+
kube_myteam_io_v1_Foo_active_count{active="3",custom_metric="yes",foo="bar",name="foo",bar="baz",qux="quxx",type="type-b"} 3
158+
```
159+
160+
### Naming
161+
162+
The default metric names are prefixed to avoid collisions with other metrics.
163+
By default, a namespace of `kube` and a subsystem based on your custom resource's group+version+kind is used.
164+
You can override these with the namespace and subsystem fields.
165+
166+
```yaml
167+
kind: CustomResourceStateMetrics
168+
spec:
169+
resources:
170+
- groupVersionKind: ...
171+
namespace: myteam
172+
subsystem: foos
173+
metrics:
174+
- name: uptime
175+
...
176+
```
177+
178+
Produces:
179+
```prometheus
180+
myteam_foos_uptime 43.21
181+
```
182+
183+
To omit namespace and/or subsystem altogether, set them to `_`.
184+
185+
### Logging
186+
187+
If a metric path is registered but not found on a custom resource, an error will be logged. For some resources,
188+
this may produce a lot of noise. The error log [verbosity][vlog] for a metric or resource can be set with `errorLogV` on
189+
the resource or metric:
190+
191+
```yaml
192+
kind: CustomResourceStateMetrics
193+
spec:
194+
resources:
195+
- groupVersionKind: ...
196+
errorLogV: 0 # 0 = default for errors
197+
metrics:
198+
- name: uptime
199+
errorLogV: 10 # only log at high verbosity
200+
```
201+
202+
[vlog]: https://github.com/go-logr/logr#why-v-levels
203+
204+
### Path Syntax
205+
206+
Paths are specified as a list of strings. Each string is a path segment, resolved dynamically against the data of the custom resource.
207+
If any part of a path is missing, the result is nil.
208+
209+
Examples:
210+
211+
```yaml
212+
# simple path lookup
213+
[spec, replicas] # spec.replicas == 1
214+
215+
# indexing an array
216+
[spec, order, "0", value] # spec.order[0].value = true
217+
218+
# finding an element in a list by key=value
219+
[status, conditions, "[name=a]", value] # status.conditions[0].value = 45
220+
221+
# if the value to be matched is a number or boolean, the value is compared as a number or boolean
222+
[status, conditions, "[value=66]", name] # status.conditions[1].name = "b"
223+
```

0 commit comments

Comments
 (0)