Need help with CPU metrics

[SviridoffA]

Hi everyone! Can anyone help me fix CPU utilization statistics of virtual machine? Has anyone run into this issue before?
I using:
ACS 4.19.2
Ubuntu 24.02 on all hosts
Hypervisor KVM

Now its looks like:

[DaanHoogland]

@JoaoJandre @GutoVeronezi , would you know what happened here?

@SviridoffA , it would seem like your vm got frozen after every cycle it performed, but I'm sure something else must be in play.

[SviridoffA]

@DaanHoogland Thanks for reply. All virtual machines are operating normally. Some of them have highly sensitive monitoring configured, which is not detecting any issues, and all of this monitoring CPU-related graphs look good. Btw, all other metrics in CloudStack, such as IOPS, network, and RAM, also appear to be normal.

[SviridoffA]

I’m not sure if it’s related, but on some KVM nodes, I’m also seeing many warnings like this (its cloudstack agent log):

2025-04-10 00:13:07,375 WARN  [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:136c3771) Couldn't retrieve free memory, returning -1.
2025-04-10 00:14:43,128 WARN  [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) (logid:a33facb5) Couldn't retrieve free memory, returning -1.
2025-04-10 00:14:45,120 WARN  [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:52dbd59b) Couldn't retrieve free memory, returning -1.
2025-04-10 00:15:23,906 WARN  [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:a6ff20ea) Couldn't retrieve free memory, returning -1.
2025-04-10 00:17:38,198 WARN  [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null) (logid:b678a5f3) Couldn't retrieve free memory, returning -1.
2025-04-10 00:17:39,902 WARN  [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:79ff14ca) Couldn't retrieve free memory, returning -1.
2025-04-10 00:17:54,669 INFO  [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-4:null) (logid:4a5df2b2) Fetching health check result for 169.254.3.179 and executing fresh
 checks: false

[weizhouapache]

the warning Couldn't retrieve free memory, returning -1. is a known issue.
the CPU usage should be fetched correctly

[winterhazel]

@SviridoffA can you share the response of the request that the UI makes to the listVirtualMachinesUsageHistory API when you access the metrics?

[SviridoffA]

I navigated to the virtual machine - metrics section, set the time range to 12 hours, and got data points on the graph, just like in the screenshot above. I copied the request moment from apilog.log, formatted it into JSON, and saved it to a file. I'm attaching it to the message. As far as I can see, cpuused is present in all blocks. I also ran the query through cmk multiple times in a row over a large time period, and I didn’t find any issues with cpuused.
response.json

[SviridoffA]

I also didn’t find any missing timestamps, but for some reason, they aren’t displayed on the graph.

[winterhazel]

@SviridoffA thanks for providing the response.

This issue seems to be related to the Management Server's Locale. I was able to reproduce it by changing mine to Italian. This makes the API return decimal values separated by a comma in the cpuused field, but the UI expects that the decimal is separated by a dot.

I'll prepare a patch to address it.

[SviridoffA]

Thanks, @winterhazel . It’s great that you were able to figure out this problem. It would be great to have this patch. I really need it, because I think it would be problematic for us to change the locale to EN for now.

[winterhazel]

PR addressing this: #10699.