how to bring metrics client to use k8s metrics APIs #2229

huanggze · 2021-06-20T06:30:58Z

huanggze
Jun 20, 2021

I'd like to bring metrics client into my project, so I can query against k8s custom metrics API. Do we have demo or doc about that? It seems the default Reconciler.Client doesn't provide a way to GET metrics.

Answered by camilamacedo86

Jul 28, 2022

Hi @huanggze,

The projects built with the tool are using controller runtime and some metrics are provided by default. See its documentation: https://book.kubebuilder.io/reference/metrics.html

The metrics are exported from the Manager, see:

kubebuilder/testdata/project-v3/main.go

Lines 70 to 71 in 26d12ab

     MetricsBindAddress: metricsAddr,  
   Port: 9443,

However, if for someone reason you are looking to check how via code implementation you can access the metrics endpoint and check it out you can see how we do it in our e2e tests, see:

See here we granting the permissions:

kubebuilder/test/e2e/v3/plugin_cluster_test.go

Lines 257 to 265 in 26d12ab

By("grantin…

View full answer

camilamacedo86 · 2022-07-28T08:30:14Z

camilamacedo86
Jul 28, 2022
Maintainer

Hi @huanggze,

The projects built with the tool are using controller runtime and some metrics are provided by default. See its documentation: https://book.kubebuilder.io/reference/metrics.html

The metrics are exported from the Manager, see:

kubebuilder/testdata/project-v3/main.go

Lines 70 to 71 in 26d12ab

    
           MetricsBindAddress:     metricsAddr, 
        
           Port:                   9443,

However, if for someone reason you are looking to check how via code implementation you can access the metrics endpoint and check it out you can see how we do it in our e2e tests, see:

See here we granting the permissions:

kubebuilder/test/e2e/v3/plugin_cluster_test.go

Lines 257 to 265 in 26d12ab

    
           By("granting permissions to access the metrics") 
        
           _, err = kbc.Kubectl.Command( 
        
           	"create", "clusterrolebinding", fmt.Sprintf("metrics-%s", kbc.TestSuffix), 
        
           	fmt.Sprintf("--clusterrole=e2e-%s-metrics-reader", kbc.TestSuffix), 
        
           	fmt.Sprintf("--serviceaccount=%s:%s", kbc.Kubectl.Namespace, kbc.Kubectl.ServiceAccount)) 
        
           ExpectWithOffset(1, err).NotTo(HaveOccurred()) 
        
           _ = curlMetrics(kbc)

See here we getting the metrics via curl:

kubebuilder/test/e2e/v3/plugin_cluster_test.go

Lines 371 to 416 in 26d12ab

    
           func curlMetrics(kbc *utils.TestContext) string { 
        
           	By("reading the metrics token") 
        
           	// Filter token query by service account in case more than one exists in a namespace. 
        
           	token, err := ServiceAccountToken(kbc) 
        
           	ExpectWithOffset(2, err).NotTo(HaveOccurred()) 
        
           	ExpectWithOffset(2, len(token)).To(BeNumerically(">", 0)) 
        
           	By("creating a curl pod") 
        
           	cmdOpts := []string{ 
        
           		"run", "curl", "--image=curlimages/curl:7.68.0", "--restart=OnFailure", "--", 
        
           		"curl", "-v", "-k", "-H", fmt.Sprintf(`Authorization: Bearer %s`, strings.TrimSpace(token)), 
        
           		fmt.Sprintf("https://e2e-%s-controller-manager-metrics-service.%s.svc:8443/metrics", 
        
           			kbc.TestSuffix, kbc.Kubectl.Namespace), 
        
           	} 
        
           	_, err = kbc.Kubectl.CommandInNamespace(cmdOpts...) 
        
           	ExpectWithOffset(2, err).NotTo(HaveOccurred()) 
        
           	By("validating that the curl pod is running as expected") 
        
           	verifyCurlUp := func() error { 
        
           		// Validate pod status 
        
           		status, err := kbc.Kubectl.Get( 
        
           			true, 
        
           			"pods", "curl", "-o", "jsonpath={.status.phase}") 
        
           		ExpectWithOffset(3, err).NotTo(HaveOccurred()) 
        
           		if status != "Completed" && status != "Succeeded" { 
        
           			return fmt.Errorf("curl pod in %s status", status) 
        
           		} 
        
           		return nil 
        
           	} 
        
           	EventuallyWithOffset(2, verifyCurlUp, 240*time.Second, time.Second).Should(Succeed()) 
        
           	By("validating that the metrics endpoint is serving as expected") 
        
           	var metricsOutput string 
        
           	getCurlLogs := func() string { 
        
           		metricsOutput, err = kbc.Kubectl.Logs("curl") 
        
           		ExpectWithOffset(3, err).NotTo(HaveOccurred()) 
        
           		return metricsOutput 
        
           	} 
        
           	EventuallyWithOffset(2, getCurlLogs, 10*time.Second, time.Second).Should(ContainSubstring("< HTTP/2 200")) 
        
           	By("cleaning up the curl pod") 
        
           	_, err = kbc.Kubectl.Delete(true, "pods/curl") 
        
           	ExpectWithOffset(3, err).NotTo(HaveOccurred()) 
        
           	return metricsOutput 
        
           }

Following are the metrics returned:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.96.251.224:8443...
* TCP_NODELAY set
* Connected to e2e-gdcf-controller-manager-metrics-service.e2e-gdcf-system.svc (10.96.251.224) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* error setting certificate verify locations, continuing anyway:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [15 bytes data]
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
{ [45 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [1780 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
} [8 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=e2e-gdcf-controller-manager-7f74cfc6dd-qgcm6@1656145994
*  start date: Jun 25 07:33:13 2022 GMT
*  expire date: Jun 25 07:33:13 2023 GMT
*  issuer: CN=e2e-gdcf-controller-manager-7f74cfc6dd-qgcm6-ca@1656145993
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55e3941bbb40)
} [5 bytes data]
> GET /metrics HTTP/2
> Host: e2e-gdcf-controller-manager-metrics-service.e2e-gdcf-system.svc:8443
> user-agent: curl/7.68.0-DEV
> accept: */*
> authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IldJWW1IN2hpVE1WNmtFalFKRGE5Y01RUVExZXNIQW0tV25UdXpzYk9zcmsifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjU2MTQ5NjE4LCJpYXQiOjE2NTYxNDYwMTgsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJlMmUtZ2RjZi1zeXN0ZW0iLCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoiZTJlLWdkY2YtY29udHJvbGxlci1tYW5hZ2VyIiwidWlkIjoiNWI5OTVkOWEtYzA3Yy00Mjg2LWFkMmQtNTI5ZWU2ODQ5NmQ2In19LCJuYmYiOjE2NTYxNDYwMTgsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDplMmUtZ2RjZi1zeXN0ZW06ZTJlLWdkY2YtY29udHJvbGxlci1tYW5hZ2VyIn0.Q4YFQSDeXW9dBgecfU8hNBvWIDNWG7AVqCj_9WiXyBGAruDzpEsVFJCBFhGdUo33jcSdh3-6rG83U_jTM2FnifXf_53M-K9-9eb2spkeZHc9YBlMN1opvXYykZU65x9VQU_lJh5LFKcfAf3GUeszM1sYEfaCHTZuZrl8qwLdBYFpHEP440Ii7hU_NjwtuuW9mE7fjAMwZxOiRzHmS-9bI6HwQM5NPAjCevEOE-qhGOJqhretFPKUM1a9hHrx3nc_AQUN0kgP0cpDFLdGUP4FwbI6Q0brzi_Tav9WvjAKMYUgpv189-JRrmIWb97zF98aJQ6E4Gih-wYaBkqiWvzYTA
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [130 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
} [5 bytes data]
< HTTP/2 200 
< content-type: text/plain; version=0.0.4; charset=utf-8
# HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
< date: Sat, 25 Jun 2022 08:33:39 GMT
# TYPE certwatcher_read_certificate_errors_total counter
< 
certwatcher_read_certificate_errors_total 0
{ [5 bytes data]
# HELP certwatcher_read_certificate_total Total number of certificate reads
# TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total 0
# HELP controller_runtime_active_workers Number of currently used workers per controller
# TYPE controller_runtime_active_workers gauge
controller_runtime_active_workers{controller="foogdcf"} 0
# HELP controller_runtime_max_concurrent_reconciles Maximum number of concurrent reconciles per controller
# TYPE controller_runtime_max_concurrent_reconciles gauge
controller_runtime_max_concurrent_reconciles{controller="foogdcf"} 1
# HELP controller_runtime_reconcile_errors_total Total number of reconciliation errors per controller
# TYPE controller_runtime_reconcile_errors_total counter
controller_runtime_reconcile_errors_total{controller="foogdcf"} 0
# HELP controller_runtime_reconcile_time_seconds Length of time per reconciliation per controller
# TYPE controller_runtime_reconcile_time_seconds histogram
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.005"} 5
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.01"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.025"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.05"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.1"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.15"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.2"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.25"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.3"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.35"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.4"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.45"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.5"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.6"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.7"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.8"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="0.9"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="1"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="1.25"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="1.5"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="1.75"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="2"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="2.5"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="3"} 6

100 24702    0 24702    0     0   893k      0 --:--:-- --:--:-- --:--:--  893k
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="3.5"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="4"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="4.5"} 6
* Connection #0 to host e2e-gdcf-controller-manager-metrics-service.e2e-gdcf-system.svc left intact
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="5"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="6"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="7"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="8"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="9"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="10"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="15"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="20"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="25"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="30"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="40"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="50"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="60"} 6
controller_runtime_reconcile_time_seconds_bucket{controller="foogdcf",le="+Inf"} 6
controller_runtime_reconcile_time_seconds_sum{controller="foogdcf"} 0.009288
controller_runtime_reconcile_time_seconds_count{controller="foogdcf"} 6
# HELP controller_runtime_reconcile_total Total number of reconciliations per controller
# TYPE controller_runtime_reconcile_total counter
controller_runtime_reconcile_total{controller="foogdcf",result="error"} 0
controller_runtime_reconcile_total{controller="foogdcf",result="requeue"} 1
controller_runtime_reconcile_total{controller="foogdcf",result="requeue_after"} 0
controller_runtime_reconcile_total{controller="foogdcf",result="success"} 5
# HELP go_gc_cycles_automatic_gc_cycles_total Count of completed GC cycles generated by the Go runtime.
# TYPE go_gc_cycles_automatic_gc_cycles_total counter
go_gc_cycles_automatic_gc_cycles_total 5
# HELP go_gc_cycles_forced_gc_cycles_total Count of completed GC cycles forced by the application.
# TYPE go_gc_cycles_forced_gc_cycles_total counter
go_gc_cycles_forced_gc_cycles_total 0
# HELP go_gc_cycles_total_gc_cycles_total Count of all completed GC cycles.
# TYPE go_gc_cycles_total_gc_cycles_total counter
go_gc_cycles_total_gc_cycles_total 5
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.0001536
go_gc_duration_seconds{quantile="0.25"} 0.0002285
go_gc_duration_seconds{quantile="0.5"} 0.0003148
go_gc_duration_seconds{quantile="0.75"} 0.0003314
go_gc_duration_seconds{quantile="1"} 0.0004237
go_gc_duration_seconds_sum 0.001452
go_gc_duration_seconds_count 5
# HELP go_gc_heap_allocs_by_size_bytes_total Distribution of heap allocations by approximate size. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks.
# TYPE go_gc_heap_allocs_by_size_bytes_total histogram
go_gc_heap_allocs_by_size_bytes_total_bucket{le="8.999999999999998"} 6491
go_gc_heap_allocs_by_size_bytes_total_bucket{le="24.999999999999996"} 26517
go_gc_heap_allocs_by_size_bytes_total_bucket{le="64.99999999999999"} 43784
go_gc_heap_allocs_by_size_bytes_total_bucket{le="144.99999999999997"} 53714
go_gc_heap_allocs_by_size_bytes_total_bucket{le="320.99999999999994"} 56966
go_gc_heap_allocs_by_size_bytes_total_bucket{le="704.9999999999999"} 58770
go_gc_heap_allocs_by_size_bytes_total_bucket{le="1536.9999999999998"} 59742
go_gc_heap_allocs_by_size_bytes_total_bucket{le="3200.9999999999995"} 60044
go_gc_heap_allocs_by_size_bytes_total_bucket{le="6528.999999999999"} 60168
go_gc_heap_allocs_by_size_bytes_total_bucket{le="13568.999999999998"} 60228
go_gc_heap_allocs_by_size_bytes_total_bucket{le="27264.999999999996"} 60259
go_gc_heap_allocs_by_size_bytes_total_bucket{le="+Inf"} 60282
go_gc_heap_allocs_by_size_bytes_total_sum 9.208104e+06
go_gc_heap_allocs_by_size_bytes_total_count 60282
# HELP go_gc_heap_allocs_bytes_total Cumulative sum of memory allocated to the heap by the application.
# TYPE go_gc_heap_allocs_bytes_total counter
go_gc_heap_allocs_bytes_total 9.208104e+06
# HELP go_gc_heap_allocs_objects_total Cumulative count of heap allocations triggered by the application. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks.
# TYPE go_gc_heap_allocs_objects_total counter
go_gc_heap_allocs_objects_total 60282
# HELP go_gc_heap_frees_by_size_bytes_total Distribution of freed heap allocations by approximate size. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks.
# TYPE go_gc_heap_frees_by_size_bytes_total histogram
go_gc_heap_frees_by_size_bytes_total_bucket{le="8.999999999999998"} 2671
go_gc_heap_frees_by_size_bytes_total_bucket{le="24.999999999999996"} 14812
go_gc_heap_frees_by_size_bytes_total_bucket{le="64.99999999999999"} 24875
go_gc_heap_frees_by_size_bytes_total_bucket{le="144.99999999999997"} 32316
go_gc_heap_frees_by_size_bytes_total_bucket{le="320.99999999999994"} 33974
go_gc_heap_frees_by_size_bytes_total_bucket{le="704.9999999999999"} 35202
go_gc_heap_frees_by_size_bytes_total_bucket{le="1536.9999999999998"} 35794
go_gc_heap_frees_by_size_bytes_total_bucket{le="3200.9999999999995"} 35901
go_gc_heap_frees_by_size_bytes_total_bucket{le="6528.999999999999"} 35969
go_gc_heap_frees_by_size_bytes_total_bucket{le="13568.999999999998"} 35999
go_gc_heap_frees_by_size_bytes_total_bucket{le="27264.999999999996"} 36012
go_gc_heap_frees_by_size_bytes_total_bucket{le="+Inf"} 36019
go_gc_heap_frees_by_size_bytes_total_sum 4.470992e+06
go_gc_heap_frees_by_size_bytes_total_count 36019
# HELP go_gc_heap_frees_bytes_total Cumulative sum of heap memory freed by the garbage collector.
# TYPE go_gc_heap_frees_bytes_total counter
go_gc_heap_frees_bytes_total 4.470992e+06
# HELP go_gc_heap_frees_objects_total Cumulative count of heap allocations whose storage was freed by the garbage collector. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks.
# TYPE go_gc_heap_frees_objects_total counter
go_gc_heap_frees_objects_total 36019
# HELP go_gc_heap_goal_bytes Heap size target for the end of the GC cycle.
# TYPE go_gc_heap_goal_bytes gauge
go_gc_heap_goal_bytes 8.74592e+06
# HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory.
# TYPE go_gc_heap_objects_objects gauge
go_gc_heap_objects_objects 24263
# HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size.
# TYPE go_gc_heap_tiny_allocs_objects_total counter
go_gc_heap_tiny_allocs_objects_total 3855
# HELP go_gc_pauses_seconds_total Distribution individual GC-related stop-the-world pause latencies.
# TYPE go_gc_pauses_seconds_total histogram
go_gc_pauses_seconds_total_bucket{le="-5e-324"} 0
go_gc_pauses_seconds_total_bucket{le="9.999999999999999e-10"} 0
go_gc_pauses_seconds_total_bucket{le="9.999999999999999e-09"} 0
go_gc_pauses_seconds_total_bucket{le="9.999999999999998e-08"} 0
go_gc_pauses_seconds_total_bucket{le="1.0239999999999999e-06"} 0
go_gc_pauses_seconds_total_bucket{le="1.0239999999999999e-05"} 0
go_gc_pauses_seconds_total_bucket{le="0.00010239999999999998"} 5
go_gc_pauses_seconds_total_bucket{le="0.0010485759999999998"} 10
go_gc_pauses_seconds_total_bucket{le="0.010485759999999998"} 10
go_gc_pauses_seconds_total_bucket{le="0.10485759999999998"} 10
go_gc_pauses_seconds_total_bucket{le="+Inf"} 10
go_gc_pauses_seconds_total_sum NaN
go_gc_pauses_seconds_total_count 10
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 50
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.18.3"} 1
# HELP go_memory_classes_heap_free_bytes Memory that is completely free and eligible to be returned to the underlying system, but has not been. This metric is the runtime's estimate of free address space that is backed by physical memory.
# TYPE go_memory_classes_heap_free_bytes gauge
go_memory_classes_heap_free_bytes 999424
# HELP go_memory_classes_heap_objects_bytes Memory occupied by live objects and dead objects that have not yet been marked free by the garbage collector.
# TYPE go_memory_classes_heap_objects_bytes gauge
go_memory_classes_heap_objects_bytes 4.737112e+06
# HELP go_memory_classes_heap_released_bytes Memory that is completely free and has been returned to the underlying system. This metric is the runtime's estimate of free address space that is still mapped into the process, but is not backed by physical memory.
# TYPE go_memory_classes_heap_released_bytes gauge
go_memory_classes_heap_released_bytes 4.071424e+06
# HELP go_memory_classes_heap_stacks_bytes Memory allocated from the heap that is reserved for stack space, whether or not it is currently in-use.
# TYPE go_memory_classes_heap_stacks_bytes gauge
go_memory_classes_heap_stacks_bytes 917504
# HELP go_memory_classes_heap_unused_bytes Memory that is reserved for heap objects but is not currently used to hold heap objects.
# TYPE go_memory_classes_heap_unused_bytes gauge
go_memory_classes_heap_unused_bytes 1.857448e+06
# HELP go_memory_classes_metadata_mcache_free_bytes Memory that is reserved for runtime mcache structures, but not in-use.
# TYPE go_memory_classes_metadata_mcache_free_bytes gauge
go_memory_classes_metadata_mcache_free_bytes 6000
# HELP go_memory_classes_metadata_mcache_inuse_bytes Memory that is occupied by runtime mcache structures that are currently being used.
# TYPE go_memory_classes_metadata_mcache_inuse_bytes gauge
go_memory_classes_metadata_mcache_inuse_bytes 9600
# HELP go_memory_classes_metadata_mspan_free_bytes Memory that is reserved for runtime mspan structures, but not in-use.
# TYPE go_memory_classes_metadata_mspan_free_bytes gauge
go_memory_classes_metadata_mspan_free_bytes 14824
# HELP go_memory_classes_metadata_mspan_inuse_bytes Memory that is occupied by runtime mspan structures that are currently being used.
# TYPE go_memory_classes_metadata_mspan_inuse_bytes gauge
go_memory_classes_metadata_mspan_inuse_bytes 115736
# HELP go_memory_classes_metadata_other_bytes Memory that is reserved for or used to hold runtime metadata.
# TYPE go_memory_classes_metadata_other_bytes gauge
go_memory_classes_metadata_other_bytes 5.209216e+06
# HELP go_memory_classes_os_stacks_bytes Stack memory allocated by the underlying operating system.
# TYPE go_memory_classes_os_stacks_bytes gauge
go_memory_classes_os_stacks_bytes 0
# HELP go_memory_classes_other_bytes Memory used by execution trace buffers, structures for debugging the runtime, finalizer and profiler specials, and more.
# TYPE go_memory_classes_other_bytes gauge
go_memory_classes_other_bytes 1.67076e+06
# HELP go_memory_classes_profiling_buckets_bytes Memory that is used by the stack trace hash map used for profiling.
# TYPE go_memory_classes_profiling_buckets_bytes gauge
go_memory_classes_profiling_buckets_bytes 5432
# HELP go_memory_classes_total_bytes All memory mapped by the Go runtime into the current process as read-write. Note that this does not include memory mapped by code called via cgo or via the syscall package. Sum of all metrics in /memory/classes.
# TYPE go_memory_classes_total_bytes gauge
go_memory_classes_total_bytes 1.961448e+07
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 4.737112e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 9.208104e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 5432
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 39874
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 5.209216e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 4.737112e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.070848e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 6.59456e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 24263
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 4.071424e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.1665408e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.6561460180810273e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 64137
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 9600
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 15600
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 115736
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 130560
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 8.74592e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.67076e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 917504
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 917504
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.961448e+07
# HELP go_sched_goroutines_goroutines Count of live goroutines.
# TYPE go_sched_goroutines_goroutines gauge
go_sched_goroutines_goroutines 50
# HELP go_sched_latencies_seconds Distribution of the time goroutines have spent in the scheduler in a runnable state before actually running.
# TYPE go_sched_latencies_seconds histogram
go_sched_latencies_seconds_bucket{le="-5e-324"} 0
go_sched_latencies_seconds_bucket{le="9.999999999999999e-10"} 100
go_sched_latencies_seconds_bucket{le="9.999999999999999e-09"} 100
go_sched_latencies_seconds_bucket{le="9.999999999999998e-08"} 100
go_sched_latencies_seconds_bucket{le="1.0239999999999999e-06"} 100
go_sched_latencies_seconds_bucket{le="1.0239999999999999e-05"} 242
go_sched_latencies_seconds_bucket{le="0.00010239999999999998"} 342
go_sched_latencies_seconds_bucket{le="0.0010485759999999998"} 376
go_sched_latencies_seconds_bucket{le="0.010485759999999998"} 383
go_sched_latencies_seconds_bucket{le="0.10485759999999998"} 384
go_sched_latencies_seconds_bucket{le="+Inf"} 384
go_sched_latencies_seconds_sum NaN
go_sched_latencies_seconds_count 384
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 12
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.21
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 11
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.1571968e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.65614599236e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.5425792e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host.
# TYPE rest_client_requests_total counter
rest_client_requests_total{code="200",host="10.96.0.1:443",method="GET"} 52
rest_client_requests_total{code="200",host="10.96.0.1:443",method="PUT"} 13
rest_client_requests_total{code="201",host="10.96.0.1:443",method="POST"} 3
rest_client_requests_total{code="404",host="10.96.0.1:443",method="GET"} 1
# HELP workqueue_adds_total Total number of adds handled by workqueue
# TYPE workqueue_adds_total counter
workqueue_adds_total{name="foogdcf"} 6
# HELP workqueue_depth Current depth of workqueue
# TYPE workqueue_depth gauge
workqueue_depth{name="foogdcf"} 0
# HELP workqueue_longest_running_processor_seconds How many seconds has the longest running processor for workqueue been running.
# TYPE workqueue_longest_running_processor_seconds gauge
workqueue_longest_running_processor_seconds{name="foogdcf"} 0
# HELP workqueue_queue_duration_seconds How long in seconds an item stays in workqueue before being requested
# TYPE workqueue_queue_duration_seconds histogram
workqueue_queue_duration_seconds_bucket{name="foogdcf",le="1e-08"} 0
workqueue_queue_duration_seconds_bucket{name="foogdcf",le="1e-07"} 0
workqueue_queue_duration_seconds_bucket{name="foogdcf",le="1e-06"} 0
workqueue_queue_duration_seconds_bucket{name="foogdcf",le="9.999999999999999e-06"} 0
workqueue_queue_duration_seconds_bucket{name="foogdcf",le="9.999999999999999e-05"} 4
workqueue_queue_duration_seconds_bucket{name="foogdcf",le="0.001"} 5
workqueue_queue_duration_seconds_bucket{name="foogdcf",le="0.01"} 6
workqueue_queue_duration_seconds_bucket{name="foogdcf",le="0.1"} 6
workqueue_queue_duration_seconds_bucket{name="foogdcf",le="1"} 6
workqueue_queue_duration_seconds_bucket{name="foogdcf",le="10"} 6
workqueue_queue_duration_seconds_bucket{name="foogdcf",le="+Inf"} 6
workqueue_queue_duration_seconds_sum{name="foogdcf"} 0.0016871
workqueue_queue_duration_seconds_count{name="foogdcf"} 6
# HELP workqueue_retries_total Total number of retries handled by workqueue
# TYPE workqueue_retries_total counter
workqueue_retries_total{name="foogdcf"} 1
# HELP workqueue_unfinished_work_seconds How many seconds of work has been done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.
# TYPE workqueue_unfinished_work_seconds gauge
workqueue_unfinished_work_seconds{name="foogdcf"} 0
# HELP workqueue_work_duration_seconds How long in seconds processing an item from workqueue takes.
# TYPE workqueue_work_duration_seconds histogram
workqueue_work_duration_seconds_bucket{name="foogdcf",le="1e-08"} 0
workqueue_work_duration_seconds_bucket{name="foogdcf",le="1e-07"} 0
workqueue_work_duration_seconds_bucket{name="foogdcf",le="1e-06"} 0
workqueue_work_duration_seconds_bucket{name="foogdcf",le="9.999999999999999e-06"} 0
workqueue_work_duration_seconds_bucket{name="foogdcf",le="9.999999999999999e-05"} 1
workqueue_work_duration_seconds_bucket{name="foogdcf",le="0.001"} 4
workqueue_work_duration_seconds_bucket{name="foogdcf",le="0.01"} 6
workqueue_work_duration_seconds_bucket{name="foogdcf",le="0.1"} 6
workqueue_work_duration_seconds_bucket{name="foogdcf",le="1"} 6
workqueue_work_duration_seconds_bucket{name="foogdcf",le="10"} 6
workqueue_work_duration_seconds_bucket{name="foogdcf",le="+Inf"} 6
workqueue_work_duration_seconds_sum{name="foogdcf"} 0.009579800000000001
workqueue_work_duration_seconds_count{name="foogdcf"} 6

0 replies

camilamacedo86 · 2023-09-16T05:44:36Z

camilamacedo86
Sep 16, 2023
Maintainer

Closing since it is answered.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

how to bring metrics client to use k8s metrics APIs #2229

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

how to bring metrics client to use k8s metrics APIs #2229

Uh oh!

Uh oh!

huanggze Jun 20, 2021

Replies: 2 comments

Uh oh!

camilamacedo86 Jul 28, 2022 Maintainer

Uh oh!

camilamacedo86 Sep 16, 2023 Maintainer

huanggze
Jun 20, 2021

camilamacedo86
Jul 28, 2022
Maintainer

camilamacedo86
Sep 16, 2023
Maintainer