Skip to content

feat: add cpu_usage_millicores and memory_usage_bytes in information_schema.cluster_info table. #7051

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

feat: add cpu_usage_millicores and memory_usage_bytes in information_schema.cluster_info table. #7051

wants to merge 10 commits into from

Conversation

zyy17
Copy link
Collaborator

@zyy17 zyy17 commented Oct 3, 2025 •
edited
Loading

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

Wait for GreptimeTeam/greptime-proto#284.

Add cpu_usage_millicores and memory_usage_bytes in information_schema.cluster_info table:

mysql> select peer_hostname, total_cpu_millicores, cpu_usage_millicores, total_memory_bytes, memory_usage_bytes from information_schema.cluster_info;
+-------------------------------------+----------------------+----------------------+--------------------+--------------------+
| peer_hostname                       | total_cpu_millicores | cpu_usage_millicores | total_memory_bytes | memory_usage_bytes |
+-------------------------------------+----------------------+----------------------+--------------------+--------------------+
| mycluster-meta-54bd44bd94-g8dzm     |                 1000 |                    0 |         1073741824 |           28368896 |
| mycluster-meta-54bd44bd94-pkrvx     |                 1000 |                    2 |         1073741824 |           31895552 |
| mycluster-datanode-0                |                 1000 |                    1 |         1073741824 |           34570240 |
| mycluster-datanode-1                |                 1000 |                    1 |         1073741824 |           38170624 |
| mycluster-datanode-2                |                 1000 |                    1 |         1073741824 |           37085184 |
| mycluster-frontend-6c5d4bcf78-m7jtx |                 1000 |                    1 |         1073741824 |           45465600 |
| mycluster-flownode-0                |                 1000 |                    1 |         1073741824 |           43786240 |
+-------------------------------------+----------------------+----------------------+--------------------+--------------------+
7 rows in set (0.00 sec)

With this information, we can easily calculate the total resource usage and may use it to make other decisions about whether to run the high-resource-intensive operation or not.

For now, it only works for a containerized environment.

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.
  • API changes are backward compatible.
  • Schema or data changes are backward compatible.

zyy17 and others added 4 commits October 3, 2025 00:09
@zyy17
refactor: add hostname in cluster_info table
354ebd4 
Signed-off-by: zyy17 <[email protected]>
@zyy17
chore: update information schema result
ccc0549 
Signed-off-by: zyy17 <[email protected]>
@evenyag @zyy17
feat: enable zstd for bulk memtable encoded parts (GreptimeTeam#7045)
4792afb 
feat: enable zstd in bulk memtable

Signed-off-by: evenyag <[email protected]>
@zyy17
refactor: add get_total_cpu_millicores() / get_total_cpu_cores() …
524a4bc 
…/ `get_total_memory_bytes()` / `get_total_memory_readable()` in common-stat

Signed-off-by: zyy17 <[email protected]>
@github-actions github-actions bot added size/M docs-not-required This change does not impact docs. breaking-change This pull request contains breaking changes. labels Oct 3, 2025
@zyy17 zyy17 force-pushed the feat/report-node-usage branch 3 times, most recently from fc11f81 to 9cb8633 Compare October 4, 2025 06:21
@zyy17
feat: add cpu_usage_millicores and memory_usage_bytes in `informa…
bfbebb8 
…tion_schema.cluster_info` table

Signed-off-by: zyy17 <[email protected]>
@zyy17 zyy17 force-pushed the feat/report-node-usage branch from 9cb8633 to bfbebb8 Compare October 4, 2025 06:35
@zyy17 zyy17 changed the title feat!: add cpu_usage_millicores and memory_usage_bytes in information_schema.cluster_info table. feat: add cpu_usage_millicores and memory_usage_bytes in information_schema.cluster_info table. Oct 17, 2025
@zyy17 zyy17 removed the breaking-change This pull request contains breaking changes. label Oct 17, 2025
@zyy17 zyy17 force-pushed the feat/report-node-usage branch from 8628422 to 472b85c Compare October 17, 2025 14:55
@zyy17 zyy17 marked this pull request as ready for review October 17, 2025 15:42
@zyy17 zyy17 requested review from WenyXu, evenyag and fengjiachun and removed request for a team and zhongzc October 17, 2025 15:42
@zyy17 zyy17 force-pushed the feat/report-node-usage branch from 472b85c to 5ab40a9 Compare October 17, 2025 15:57
fengjiachun
fengjiachun reviewed Oct 20, 2025
View reviewed changes
src/catalog/src/system_schema/information_schema/cluster_info.rs
(PEER_HOSTNAME, &Value::from(node_info.hostname.as_str())),
(VERSION, &Value::from(node_info.version.as_str())),
(GIT_COMMIT, &Value::from(node_info.git_commit.as_str())),
(PEER_HOSTNAME, &Value::from(node_info.hostname.as_str())),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a repeated assignment? See line 236.

src/common/stat/src/resource.rs
if let Some(current_cpu_usage_usecs) = current_cpu_usage_usecs {
let cpu_usage = calculate_cpu_usage(
current_cpu_usage_usecs,
last_cpu_usage_usecs.load(Ordering::Relaxed),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the first loop, last_cpu_usage_usecs is 0. calculate_cpu_usage(current, 0, interval) calculates the average CPU usage from boot to now, which is meaningless. The baseline should only be recorded on the first time, and calculations should start from the second time.

WenyXu
WenyXu reviewed Oct 20, 2025
View reviewed changes
src/datanode/src/heartbeat.rs
Comment on lines +257 to +258
cpu_usage_millicores: 0,
memory_usage_bytes: 0,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we store only non-zero values on metasrv, or let resource_stat cache the statistics for each window?

@fengjiachun fengjiachun requested review from WenyXu and fengjiachun and removed request for MichaelScofield, evenyag and waynexia October 21, 2025 03:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fengjiachun fengjiachun Awaiting requested review from fengjiachun

@WenyXu WenyXu Awaiting requested review from WenyXu

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

docs-not-required This change does not impact docs. size/M

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zyy17 @fengjiachun @WenyXu @evenyag