feat: add operator build_info metrics and go runtime metrics #6044
Conversation
karmada-bot
added
the
kind/feature
karmada-bot
added
the
size/L
|
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## master #6044 +/- ##
==========================================
- Coverage 48.36% 47.97% -0.40%
==========================================
Files 666 675 +9
Lines 54831 55880 +1049
==========================================
+ Hits 26519 26806 +287
- Misses 26594 27328 +734
- Partials 1718 1746 +28
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Thanks @dongjiang1989, can you help to explain why these metrics are necessary for you? |
Thanks @RainbowMango
Core Metrics list: |
|
Thanks for the clarification. I will take a look and get back to you. |
|
By the way, are you building some services based on Karmada? |
Yes. Like: Model hub and Cloud platform. |
|
Is it possible to introduce your platform on the Karmada community meeting? |
Very willing to share it. |
|
That would be great! |
karmada-bot
added
size/M
@RainbowMango I don't have editing permissions. Maybe can share it at 2025-02-11. |
|
Echo from the notes from meeting notes:
Please join the mail group and all members from the mail group can edit this doc. |
There was a problem hiding this comment.
Given we already have some metrics related to gc, goroutine, like:
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.0146e-05
go_gc_duration_seconds{quantile="0.25"} 4.6354e-05
go_gc_duration_seconds{quantile="0.5"} 5.3922e-05
go_gc_duration_seconds{quantile="0.75"} 0.000103302
go_gc_duration_seconds{quantile="1"} 0.000227208
go_gc_duration_seconds_sum 0.001537905
go_gc_duration_seconds_count 18
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 43
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.22.9"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 6.101576e+06
I wound like to understand the list of metrics introduced or removed by this PR and the reason behind these changes.
Additionally, If this change is accepted, my intension is to extend these metrics to all other components, not just the karmada-operator.
PS： Here is the full list metrics grabbed from v1.12.2:
Click Me to Show the List
HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
TYPE certwatcher_read_certificate_errors_total counter
certwatcher_read_certificate_errors_total 0
HELP certwatcher_read_certificate_total Total number of certificate reads
TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total 0
HELP controller_runtime_active_workers Number of currently used workers per controller
TYPE controller_runtime_active_workers gauge
controller_runtime_active_workers{controller="karmada-operator-controller"} 0
HELP controller_runtime_max_concurrent_reconciles Maximum number of concurrent reconciles per controller
TYPE controller_runtime_max_concurrent_reconciles gauge
controller_runtime_max_concurrent_reconciles{controller="karmada-operator-controller"} 5
HELP controller_runtime_reconcile_errors_total Total number of reconciliation errors per controller
TYPE controller_runtime_reconcile_errors_total counter
controller_runtime_reconcile_errors_total{controller="karmada-operator-controller"} 0
HELP controller_runtime_reconcile_panics_total Total number of reconciliation panics per controller
TYPE controller_runtime_reconcile_panics_total counter
controller_runtime_reconcile_panics_total{controller="karmada-operator-controller"} 0
HELP controller_runtime_reconcile_time_seconds Length of time per reconciliation per controller
TYPE controller_runtime_reconcile_time_seconds histogram
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.005"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.01"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.025"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.05"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.1"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.15"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.2"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.25"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.3"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.35"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.4"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.45"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.5"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.6"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.7"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.8"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="0.9"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="1"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="1.25"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="1.5"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="1.75"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="2"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="2.5"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="3"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="3.5"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="4"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="4.5"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="5"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="6"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="7"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="8"} 0
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="9"} 1
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="10"} 1
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="15"} 1
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="20"} 1
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="25"} 1
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="30"} 1
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="40"} 1
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="50"} 1
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="60"} 1
controller_runtime_reconcile_time_seconds_bucket{controller="karmada-operator-controller",le="+Inf"} 1
controller_runtime_reconcile_time_seconds_sum{controller="karmada-operator-controller"} 8.374751902
controller_runtime_reconcile_time_seconds_count{controller="karmada-operator-controller"} 1
HELP controller_runtime_reconcile_total Total number of reconciliations per controller
TYPE controller_runtime_reconcile_total counter
controller_runtime_reconcile_total{controller="karmada-operator-controller",result="error"} 0
controller_runtime_reconcile_total{controller="karmada-operator-controller",result="requeue"} 0
controller_runtime_reconcile_total{controller="karmada-operator-controller",result="requeue_after"} 0
controller_runtime_reconcile_total{controller="karmada-operator-controller",result="success"} 1
HELP controller_runtime_terminal_reconcile_errors_total Total number of terminal reconciliation errors per controller
TYPE controller_runtime_terminal_reconcile_errors_total counter
controller_runtime_terminal_reconcile_errors_total{controller="karmada-operator-controller"} 0
HELP controller_runtime_webhook_panics_total Total number of webhook panics
TYPE controller_runtime_webhook_panics_total counter
controller_runtime_webhook_panics_total 0
HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.0146e-05
go_gc_duration_seconds{quantile="0.25"} 4.6354e-05
go_gc_duration_seconds{quantile="0.5"} 5.3922e-05
go_gc_duration_seconds{quantile="0.75"} 0.000103302
go_gc_duration_seconds{quantile="1"} 0.000227208
go_gc_duration_seconds_sum 0.001537905
go_gc_duration_seconds_count 18
HELP go_goroutines Number of goroutines that currently exist.
TYPE go_goroutines gauge
go_goroutines 43
HELP go_info Information about the Go environment.
TYPE go_info gauge
go_info{version="go1.22.9"} 1
HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 6.101576e+06
HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 5.3223024e+07
HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.47988e+06
HELP go_memstats_frees_total Total number of frees.
TYPE go_memstats_frees_total counter
go_memstats_frees_total 362491
HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.811048e+06
HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 6.101576e+06
HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.88128e+06
HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 9.0112e+06
HELP go_memstats_heap_objects Number of allocated objects.
TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 47775
HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 6.127616e+06
HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.589248e+07
HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7373615791665132e+09
HELP go_memstats_lookups_total Total number of pointer lookups.
TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
HELP go_memstats_mallocs_total Total number of mallocs.
TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 410266
HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 4800
HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 15600
HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 179840
HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 228480
HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 1.0777016e+07
HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 843752
HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 884736
HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 884736
HELP go_memstats_sys_bytes Number of bytes obtained from system.
TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 2.3155976e+07
HELP go_threads Number of OS threads created.
TYPE go_threads gauge
go_threads 10
HELP leader_election_master_status Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. 'name' is the string used to identify the lease. Please make sure to group by name.
TYPE leader_election_master_status gauge
leader_election_master_status{name="karmada-operator"} 1
HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 2.51
HELP process_max_fds Maximum number of open file descriptors.
TYPE process_max_fds gauge
process_max_fds 1.048576e+06
HELP process_open_fds Number of open file descriptors.
TYPE process_open_fds gauge
process_open_fds 11
HELP process_resident_memory_bytes Resident memory size in bytes.
TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 4.9225728e+07
HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
TYPE process_start_time_seconds gauge
process_start_time_seconds 1.73736126681e+09
HELP process_virtual_memory_bytes Virtual memory size in bytes.
TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.307947008e+09
HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
HELP rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host.
TYPE rest_client_requests_total counter
rest_client_requests_total{code="200",host="10.96.0.1:443",method="GET"} 35
rest_client_requests_total{code="200",host="10.96.0.1:443",method="PUT"} 226
rest_client_requests_total{code="200",host="172.18.0.7:32574",method="GET"} 13
rest_client_requests_total{code="200",host="172.18.0.7:32574",method="PATCH"} 2
rest_client_requests_total{code="200",host="172.18.0.7:32574",method="PUT"} 8
rest_client_requests_total{code="201",host="10.96.0.1:443",method="POST"} 2
rest_client_requests_total{code="201",host="172.18.0.7:32574",method="POST"} 4
rest_client_requests_total{code="409",host="10.96.0.1:443",method="POST"} 17
rest_client_requests_total{code="409",host="172.18.0.7:32574",method="POST"} 26
HELP workqueue_adds_total Total number of adds handled by workqueue
TYPE workqueue_adds_total counter
workqueue_adds_total{controller="karmada-operator-controller",name="karmada-operator-controller"} 1
HELP workqueue_depth Current depth of workqueue
TYPE workqueue_depth gauge
workqueue_depth{controller="karmada-operator-controller",name="karmada-operator-controller"} 0
HELP workqueue_longest_running_processor_seconds How many seconds has the longest running processor for workqueue been running.
TYPE workqueue_longest_running_processor_seconds gauge
workqueue_longest_running_processor_seconds{controller="karmada-operator-controller",name="karmada-operator-controller"} 0
HELP workqueue_queue_duration_seconds How long in seconds an item stays in workqueue before being requested
TYPE workqueue_queue_duration_seconds histogram
workqueue_queue_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="1e-08"} 0
workqueue_queue_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="1e-07"} 0
workqueue_queue_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="1e-06"} 0
workqueue_queue_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="9.999999999999999e-06"} 0
workqueue_queue_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="9.999999999999999e-05"} 1
workqueue_queue_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="0.001"} 1
workqueue_queue_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="0.01"} 1
workqueue_queue_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="0.1"} 1
workqueue_queue_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="1"} 1
workqueue_queue_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="10"} 1
workqueue_queue_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="100"} 1
workqueue_queue_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="1000"} 1
workqueue_queue_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="+Inf"} 1
workqueue_queue_duration_seconds_sum{controller="karmada-operator-controller",name="karmada-operator-controller"} 1.3487e-05
workqueue_queue_duration_seconds_count{controller="karmada-operator-controller",name="karmada-operator-controller"} 1
HELP workqueue_retries_total Total number of retries handled by workqueue
TYPE workqueue_retries_total counter
workqueue_retries_total{controller="karmada-operator-controller",name="karmada-operator-controller"} 0
HELP workqueue_unfinished_work_seconds How many seconds of work has been done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.
TYPE workqueue_unfinished_work_seconds gauge
workqueue_unfinished_work_seconds{controller="karmada-operator-controller",name="karmada-operator-controller"} 0
HELP workqueue_work_duration_seconds How long in seconds processing an item from workqueue takes.
TYPE workqueue_work_duration_seconds histogram
workqueue_work_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="1e-08"} 0
workqueue_work_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="1e-07"} 0
workqueue_work_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="1e-06"} 0
workqueue_work_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="9.999999999999999e-06"} 0
workqueue_work_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="9.999999999999999e-05"} 0
workqueue_work_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="0.001"} 0
workqueue_work_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="0.01"} 0
workqueue_work_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="0.1"} 0
workqueue_work_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="1"} 0
workqueue_work_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="10"} 1
workqueue_work_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="100"} 1
workqueue_work_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="1000"} 1
workqueue_work_duration_seconds_bucket{controller="karmada-operator-controller",name="karmada-operator-controller",le="+Inf"} 1
workqueue_work_duration_seconds_sum{controller="karmada-operator-controller",name="karmada-operator-controller"} 8.374788526
workqueue_work_duration_seconds_count{controller="karmada-operator-controller",name="karmada-operator-controller"} 1
Hold on, we can postpone this until we make the final decision. |
|
/cc @RainbowMango Remove |
|
I left my comments on #6044 (comment) after exploring some docs about the short commit id. |
RainbowMango
left a comment
RainbowMango
left a comment
There was a problem hiding this comment.
Suggested the name of the short commit ID.
I'll focus on the metrics after we the consensus about the name.
karmada-bot
added
size/L
RainbowMango
left a comment
RainbowMango
left a comment
There was a problem hiding this comment.
Thanks for the quick response, generally looks good to me for the short commit part except a few more nits.
I will run another look for the metrcis.
pkg/version/version.go
Outdated
| func NewCollector(program string) prometheus.Collector { | ||
| return prometheus.NewGaugeFunc( | ||
| prometheus.GaugeOpts{ | ||
| Namespace: program, |
There was a problem hiding this comment.
Maybe we can omit the component name?
Because the metric emitted from a component indicates it's build info.
There was a problem hiding this comment.
The metrcs specification defines different metrics names, and is not distinguished by different tags of the same metrics name.
ref: https://github.com/prometheus/client_golang/blob/main/prometheus/collectors/version/version.go#L28-L30
pkg/version/version.go
Outdated
| ), | ||
| ConstLabels: prometheus.Labels{ | ||
| "version": Get().GitVersion, | ||
| "revision": Get().GitAbbreviativeCommit, |
There was a problem hiding this comment.
| "revision": Get().GitAbbreviativeCommit, | |
| "commit": Get().GitCommit, | |
| "abbr-commit": Get().GitAbbreviativeCommit, |
There was a problem hiding this comment.
Hmmmm....🤔 @RainbowMango Probably best to stay consistent with prometheus's specification.
There was a problem hiding this comment.
I apologize for not explaining why I suggested those changes.
I think the most important aspect of naming is clarity, and I feel the terms version and reversion are ambiguous, which could lead to misunderstandings. For example, I can't tell their difference in the context of Prometheus. (Can you give an example of the build info metric from Prometheus?)
Regarding your point about maintaining consistency with Prometheus, I'd love to, but Karmada might emit both commit and abbreviate commit, we should differentiate them in terms of naming to avoid confusion.
There was a problem hiding this comment.
example:
prometheus_operator_build_info{branch="refs/tags/v0.67.1", goarch="amd64", goos="linux", goversion="go1.20.6", instance="xxx.xxx.xxx.xxx:10250", job="kube-prometheus-stack-operator", namespace="monitoring", pod="kube-prometheus-stack-operator-84fbdc5578-wxqll", revision="ce782eafa", tags="unknown", version="0.67.1"}Keep branch, revision, version consistent with prometheus's specification.
There was a problem hiding this comment.
An ideal metric in my mind would look something like this:
// NewBuildInfoCollector returns a collector that exports metrics about the current version
// information.
func NewBuildInfoCollector() prometheus.Collector {
return prometheus.NewGaugeFunc(
prometheus.GaugeOpts{
Name: "component_build_info",
Help: "Component build metadata exposed as labels with a constant value of 1.",
ConstLabels: prometheus.Labels{
"git_version": Get().GitVersion,
"git_commit": Get().GitCommit,
"git_short_commit": Get().GitShortCommit,
"git_tree_state": Get().GitTreeState,
"build_date": Get().BuildDate,
"go_version": Get().GoVersion,
"compiler": Get().Compiler,
"platform": Get().Platform,
},
},
func() float64 { return 1 },
)
}- I don't think the component name needed to be as
namespace, because the component emitting the metric inherentlyownsit. - All build metadata information is clearly expressed through standardized labels to avoid ambiguity.
- I'd love to follow the mature practice like you mentioned above, but I don't think that fit us, mostly because we want more clear label names.
| // Unregister default NewGoCollector | ||
| ctrlmetrics.Registry.Unregister(collectors.NewGoCollector()) | ||
|
|
||
| ctrlmetrics.Registry.MustRegister( | ||
| // Go Runtime metrics about debug.GCStats (base metrics) and | ||
| // runtime/metrics. | ||
| collectors.NewGoCollector( | ||
| collectors.WithGoCollectorRuntimeMetrics( | ||
| // go runtime gc metrics. (e.g. `go_gc_duration_seconds` | ||
| // means garbage collection cycle pause duration) | ||
| collectors.MetricsGC, | ||
| // go runtime scheduler metrics. (e.g. `go_sched_gomaxprocs_threads` | ||
| // means the current runtime.GOMAXPROCS setting) | ||
| collectors.MetricsScheduler, | ||
| // go runtime memory metrics. (e.g. `go_memstats_alloc_bytes` | ||
| // means number of bytes allocated and still in use) | ||
| collectors.MetricsMemory, | ||
| // go runtime sync lock metrics. (e.g. `go_sync_mutex_wait_total_seconds_total` | ||
| // means Approximate cumulative time goroutines have spent blocked on a sync.Mutex, sync.RWMutex, or runtime-internal lock) | ||
| collectors.GoRuntimeMetricsRule{Matcher: regexp.MustCompile(`^/sync/.*`)}, | ||
| ), | ||
| ), | ||
| ) | ||
| // `karmada_operator_build_info` metrics for operator version upgrade | ||
| ctrlmetrics.Registry.MustRegister( | ||
| version.NewCollector("karmada_operator"), | ||
| ) |
There was a problem hiding this comment.
Hi @dongjiang1989 The [email protected] included your PR(kubernetes-sigs/controller-runtime#3070). I remember you said(at the last community meeting) that we can do the same if we can bump controller-runtime, right?
There was a problem hiding this comment.
Yes. update controller-runtime to v0.19.6
dongjiang1989
force-pushed
the
add-operator-metrics
branch
from
f4b2c3e to
532537d
Compare
There was a problem hiding this comment.
PR Overview
This PR adds new metrics to expose build information and Go runtime metrics for the operator, supporting enhanced observability for Prometheus and pod HPA.
- Introduces a new field (GitAbbreviativeCommit) in version information with associated updates to metrics collection and tests.
- Registers the new build_info metrics in the operator's controller manager.
Reviewed Changes
| File | Description |
|---|---|
| pkg/version/version.go | Adds GitAbbreviativeCommit to Info struct and updates NewCollector. |
| operator/cmd/operator/app/operator.go | Registers the new build_info metrics for operator version upgrades. |
| pkg/version/base.go | Adds a gitAbbreviativeCommit variable to support the new metrics. |
| pkg/version/version_test.go | Updates the expected output to include the new GitAbbreviativeCommit field. |
Copilot reviewed 17 out of 17 changed files in this pull request and generated no comments.
Comments suppressed due to low confidence (2)
pkg/version/version.go:30
- [nitpick] The field name 'GitAbbreviativeCommit' might be less clear than alternatives such as 'GitShortCommit'. Consider renaming for clarity and to align with common versioning terminology.
+ GitAbbreviativeCommit string `json:"gitAbbreviativeCommit"`
pkg/version/base.go:29
- [nitpick] Consider renaming 'gitAbbreviativeCommit' to 'gitShortCommit' to improve clarity and consistency with conventional naming.
+ gitAbbreviativeCommit = "unknown" // short sha1 from git, output of $(git rev-parse --short HEAD)
operator/cmd/operator/.gitignore
Outdated
There was a problem hiding this comment.
I don't understand why we need this .gitignore file here. Was it committed by mistake?
pkg/version/base.go
Outdated
| gitVersion = "v0.0.0-master" | ||
| gitCommit = "unknown" // sha1 from git, output of $(git rev-parse HEAD) | ||
| gitTreeState = "unknown" // state of git tree, either "clean" or "dirty" | ||
| gitAbbreviativeCommit = "unknown" // short sha1 from git, output of $(git rev-parse --short HEAD) |
There was a problem hiding this comment.
The Copilot gives one comment regarding the field name gitAbbreviativeCommit as follows:
The field name 'GitAbbreviativeCommit' might be less clear than alternatives such as 'GitShortCommit'. Consider renaming for clarity and to align with common versioning terminology.
I'm sold by Copilot, and feel GitShortCommit might be more clear, and apologize for my back-and-forth.
Let me know how you think?
pkg/version/version.go
Outdated
| ), | ||
| ConstLabels: prometheus.Labels{ | ||
| "version": Get().GitVersion, | ||
| "revision": Get().GitAbbreviativeCommit, |
There was a problem hiding this comment.
An ideal metric in my mind would look something like this:
// NewBuildInfoCollector returns a collector that exports metrics about the current version
// information.
func NewBuildInfoCollector() prometheus.Collector {
return prometheus.NewGaugeFunc(
prometheus.GaugeOpts{
Name: "component_build_info",
Help: "Component build metadata exposed as labels with a constant value of 1.",
ConstLabels: prometheus.Labels{
"git_version": Get().GitVersion,
"git_commit": Get().GitCommit,
"git_short_commit": Get().GitShortCommit,
"git_tree_state": Get().GitTreeState,
"build_date": Get().BuildDate,
"go_version": Get().GoVersion,
"compiler": Get().Compiler,
"platform": Get().Platform,
},
},
func() float64 { return 1 },
)
}- I don't think the component name needed to be as
namespace, because the component emitting the metric inherentlyownsit. - All build metadata information is clearly expressed through standardized labels to avoid ambiguity.
- I'd love to follow the mature practice like you mentioned above, but I don't think that fit us, mostly because we want more clear label names.
|
Two more comments are:
|
dongjiang1989
force-pushed
the
add-operator-metrics
branch
2 times, most recently
from
7488879 to
3d1ba66
Compare
Thanks @RainbowMango Please re-check it |
pkg/metrics/version.go
Outdated
| func NewCollector(program string) prometheus.Collector { | ||
| return prometheus.NewGaugeFunc( | ||
| prometheus.GaugeOpts{ | ||
| Namespace: program, |
There was a problem hiding this comment.
@CharlesQQ Do you think using the component name as a prefix for the metrics is necessary?
There was a problem hiding this comment.
No, it is usually used in combination with metrics and prometheus; when prometheus aggregates data, users usually add specific labels related to components from different sources.
dongjiang1989
force-pushed
the
add-operator-metrics
branch
2 times, most recently
from
9bb6f2a to
9990b07
Compare
There was a problem hiding this comment.
PR Overview
This PR introduces additional metrics for the operator by adding a build_info collector that exposes versioning details and Go runtime information.
- Added a new build_info metric collector in pkg/metrics/version.go using Prometheus GaugeFunc.
- Updated the operator to register the new build_info metric.
- Introduced a new GitShortCommit field in version information and updated corresponding tests.
Reviewed Changes
| File | Description |
|---|---|
| pkg/metrics/version.go | Adds a build_info Prometheus collector for operator version metrics. |
| operator/cmd/operator/app/operator.go | Registers the new build_info metric collector in the operator runtime. |
| pkg/version/base.go, pkg/version/version.go, pkg/version/version_test.go | Introduces and tests the GitShortCommit field in version information. |
Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.
dongjiang1989
force-pushed
the
add-operator-metrics
branch
from
4799e06 to
f365da6
Compare
|
I tested this patch on my side and basically this patch introduced the build info metric as follows: In addition, along with the Click me to show the full list |
|
Generally looks good to me. (though, I still can not understand why it is necessary to have the component name |
Thanks @RainbowMango Some message: |
|
Thanks for the info. Compared to the naming rule, I'm more curious about how you use the metric, especially what issues might raise if component names are not used as prefixes for the metric. |
Observabity is very important for us as we want to be informed of events such as a control plane failing to reach a registered member cluster, amongst other things, via alerts/alarms. Currently, we're not using the metrics from the operator for anything, but maybe will make use of them at some point. |
Signed-off-by: dongjiang <[email protected]> Co-Authored-By: Copilot <[email protected]>
dongjiang1989
force-pushed
the
add-operator-metrics
branch
from
e92efbe to
276cb0d
Compare
RainbowMango
left a comment
RainbowMango
left a comment
There was a problem hiding this comment.
/lgtm
Now waiting for CI signals.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: RainbowMango The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
karmada-bot
added
the
approved
7 participants
What type of PR is this?
/kind feature
What this PR does / why we need it:
More metrics info for prometheus and pod hpa
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?: