v1.9.0-rc.1 / 2019-12-12

Note: the metric has changed in order to fix the bug. See PR #997 for further details.

[BUGFIX] Move resource version from metric label to metric number value #997

v1.9.0-rc.0 / 2019-12-04

• [CHANGE] Add tools as go modules #927
• [FEATURE] Add kube_hpa_spec_target_metric metric. #966
• [FEATURE] Add hpa stats for current utilization and average value. #961
• [FEATURE] Add kube_namespace_status_condition metric. #949
• [FEATURE] Add kube_persistentvolumeclaim_status_condition and kube_persistentvolumeclaim_status_phase metrics #952
• [FEATURE] Add kube_networkpolicy_* metrics #893
• [FEATURE] Add kube_volumeattachment_* metrics #946
• [FEATURE] add kube_mutatingwebhookconfiguration_*, kube_validatingwebhookconfiguration_* metrics #914
• [ENHANCEMENT] Add pod_cidr label to the kube_node_info metric. #963
• [ENHANCEMENT] Generate all manifests from jsonnet #908
• [ENHANCEMENT] Add kube-state-metrics alerting rules. #962
• [ENHANCEMENT] Add "Evicted" to the set of container terminated reasons. #968
• [BUGFIX] Add check for ReclaimPolicy, VolumeBindingMode storageclass metrics. #929
• [BUGFIX] Use single MultiListerWatcher for single store to fix missing metrics when watching multiple namespaces. #969
• [BUGFIX] Fix nil panics parsing HPA CPU current metrics. #993

v1.8.0 / 2019-10-01

After a testing period of 7 days, there were no additional bugs found or features introduced.

• [CHANGE] Pin go version to go mod artifact file #882
• [BUGFIX] Correct mapping of hpa condition status labels #890
• [FEATURE] Introduce sharding (and experimental auto-sharding) #613
• [FEATURE] Add kube_pod_status_unschedulable metric #835
• [CHANGE] Makefile: Remove tmpdir after container build #867
• [ENHANCEMENT] Add metrics about kube-state-metric's interaction with the Kubernetes API. #866
• [FEATURE] Add kube_node_role metric #877
• [ENHANCEMENT] Add support for Darwin OS in e2e test #879

v1.8.0-rc.1 / 2019-09-24

• [CHANGE] Pin go version to go mod artifact file #882
• [BUGFIX] Correct mapping of hpa condition status labels #890

v1.8.0-rc.0 / 2019-09-10

• [FEATURE] Introduce sharding (and experimental auto-sharding) #613
• [FEATURE] Add kube_pod_status_unschedulable metric #835
• [CHANGE] Makefile: Remove tmpdir after container build #867
• [ENHANCEMENT] Add metrics about kube-state-metric's interaction with the Kubernetes API. #866
• [FEATURE] Add kube_node_role metric #877
• [ENHANCEMENT] Add support for Darwin OS in e2e test #879

v1.7.2 / 2019-08-05

A security issue was discovered in the v1.7.0 and v1. 7.1 versions of kube-state-metrics [1]. The issue is of Medium severity level and upgrading to the latest release v1.7.2 [2] of kube-state-metrics is highly encouraged to fix this issue, as well as deleting the time-series data that could potentially disclose secret information.

Am I vulnerable?

If you are using the kube-state-metrics versions v1.7.0 or v1.7.1, you are running a vulnerable version. To find out which version you are running, you can verify the image tag of your kube-state-metrics deployment.

The following commands should give you the deployed image tag. (Please note that this may vary depending on which namespace kube-state-metrics is deployed in and the deployment name itself):

kubectl get deployment -n kube-system kube-state-metrics -o yaml | grep image:

How do I mitigate the vulnerability?

Update the image of kube-state-metrics to quay.io/coreos/kube-state-metrics:v1.7.2.

If you are unable to upgrade to the latest version of kube-state-metrics, you can filter out all of the annotation metrics by passing the following flag to kube-state-metrics:

--metric-blacklist="kube_.*_annotations"

Make sure to delete all the time series data from Prometheus as well, below is an example command. (Note that this will only work from Prometheus v2.1 onward. More details on time series data deletion can be found in the Prometheus docs [3])

# This command deletes all of the annotation metrics emitted by kube-state-metrics
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~"kube_.+_annotations"}'

This requires the Admin APIs to be enabled. Start Prometheus with the --web.enable-admin-api flag to do so.
Please remember that the delete API only marks the time-series data for deletion. The actual removal happens during the next compaction process. To trigger this, the clean tombstones API can be used:

curl -X POST http://localhost:9090/api/v1/admin/tsdb/clean_tombstones 

Vulnerability Details

An experimental feature was added to the v1.7.0 release that enabled annotations to be exposed as metrics. By default, the kube-state-metrics metrics only expose metadata about Secrets. However, a combination of the default kubectl behavior and this new feature can cause the entire secret content to end up in metric labels thus inadvertently exposing the secret content in metrics.

We are not aware of other annotations that disclose information in the same way, but as a precaution we have reverted the feature and will think more thoroughly about the implications should we ever introduce something like it again.

This feature has been reverted and released as the v1.7.2 release. If you are running the v1.7.0 or v1.7.1 release, please upgrade to the v1.7.2 release as soon as possible.

Thank you to Moritz S. for reporting this issue! Also thank you to Tariq Ibrahim, Frederic Branczyk and Lili Cosic for the coordination in making the fix and release.

Thank you for your understanding,
kube-state-metrics maintainers

[1] https://github.com/kubernetes/kube-state-metrics
[2] https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.7.2
[3] https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series

• [CHANGE] Revert "add kube_*_annotations metrics for all objects" #859
• [CHANGE] Remove kube_namespace_annotations metric #859

v1.7.1 / 2019-07-18

• [BUGFIX] Fix crash due to missing ingress http block. #831
• [BUGFIX] Add nil guard checks when querying VPA objects. #832

v1.7.0 / 2019-07-17

After a testing period of 7 days, there were no additional bugs found or features introduced.

[BUGFIX] Use the appsv1 apigroup when building the reflector for replicasets. #816
[CHANGE] Use appsv1 apigroup for ReplicaSet. #804
[CHANGE] Use distroless/static as base image to further reduce image size. #748
[CHANGE] Return standardized text for health endpoint. #767
[ENHANCEMENT] Add DeadlineExceeded pod termination reason. #740
[ENHANCEMENT] Add CreateContainerError and InvalidImageName waiting reasons. #763
[FEATURE] Add Pod init container metrics. #762
[FEATURE] Add Metrics About StorageClass. #777
[FEATURE] Allow black/white- listing of metrics by regex. #773
[FEATURE] Add metrics for VerticalPodAutoscaler objects (experimental and disabled by default). #791
[FEATURE] Add (experimental) kube_*_annotations metrics. #770

v1.7.0-rc.1 / 2019-07-10

[BUGFIX] Use the appsv1 apigroup when building the reflector for replicasets. #816

v1.7.0-rc.0 / 2019-07-10

• [CHANGE] Use appsv1 apigroup for ReplicaSet. #804
• [CHANGE] Use distroless/static as base image to further reduce image size. #748
• [CHANGE] Return standardized text for health endpoint. #767
• [ENHANCEMENT] Add DeadlineExceeded pod termination reason. #740
• [ENHANCEMENT] Add CreateContainerError and InvalidImageName waiting reasons. #763
• [FEATURE] Add Pod init container metrics. #762
• [FEATURE] Add Metrics About StorageClass. #777
• [FEATURE] Allow black/white- listing of metrics by regex. #773
• [FEATURE] Add metrics for VerticalPodAutoscaler objects (experimental and disabled by default). #791
• [FEATURE] Add (experimental) kube_*_annotations metrics. #770