IO Delay very high

[di5cord20]

My proxmox server has become extremely slow of late. I have noticed the IO Delay peak at >95%. I'm looking for some help to troubleshoot this problem and appreciate any advice I can get. Thanks in advance.

[tteck]

Drives going bad?

[di5cord20]

How could I confirm if it's the drives?

It's a fairly new mini pc (Beelink Mini PC, EQ12 Intel 12th Gen N100(up to 3.4GHz) 4C/4T, Mini Computer 16GB DDR5 RAM 500GB SSD, Desktop PC Triple Display USB-C&Dual HDMI 4K 60Hz/Dual 2.5Gbps/Wif6/BT5.2/Grey) which I purchased late September. It's the stock drive so likely not high quality.

I have been creating dozens of LXCs using your scripts, tearing them down, creating them again. I noticed the AgentDVR script was constantly recording and using lots of IO. I stopped that LXC and did notice some improvements... but to your point, the drive might be the problem.

[di5cord20]

Did a bit more reading ... M.2 PCIe x1 SSD Standard 500GB PCle SSD. NVMe up to 800MB/S, SATA3 up to 500MB/s.
Max 2TB

I'd still like to salvage the drive if it's "not bad" (still don't know how to diagnose). Alternatively, I've been thinking about converting this box for OPNsense (replacing a dated USG 3) and pickup new hardward for my proxmox. That would be more costly, especially if I still need to replace the drive.

[di5cord20]

Not sure if this helps with the diagnosis:

[di5cord20]

Just noticed I'm constantly getting "pve pvestatd[1265]: metrics send error 'InfluxDB2': 500 read timeout" messages in journalctl. I did install InfluxDB2 and Grafana but have since stopped those containers. I tried starting the InfluxDB2 container but the messages persist (makes me think it never worked).

Questions:

1. How do I fix this error? (at this point, I want to keep it simple and exclude InfluxDB2 altogether)
2. Could this error be causing my IO issues?

I also observed that some of my manually stopped containers would start and the shutdown automatically. I started by disabling "start at boot" but that didn't prevent the start/shutdown from happening. I read https://forum.proxmox.com/threads/containers-and-vms-auto-start-after-manual-shutdown.128139/ and saw you helped someone by reminding them that the Monitor-All script might be causing the problem. I confirmed that I never installed that script.

Any idea what could be starting/stopping my manually stopped containers? Any chance that could also be impacting my IO issue?

Thanks again for any help...

[di5cord20]

Apologies for replying to my own post. I remembered that I followed https://www.smarthomebeginner.com/proxmox-grafana-dashboard/ when configuring Proxmox/InfluxDB. Reading that article again, I confirmed that I do in fact get stats into InfluxDB from Proxmox. It leaves me wondering why I'm getting timeout messages (perhaps do to poor I/O ??).

Long story short, I disabled proxmox sending data to InfluxDB via the Datacenter --> Metric Server. The messages are now gone from my journal. I'll keep monitoring the I/O for improvements.

Still don't know what caused the Start/Shutdown of my manually stopped containers...

[di5cord20]

Update: as of this evening, I/O is having under 1%. I believe the issue was CPU overload and not a bad drive (time will tell). I had assigned the max # of cpus to some of my more power hungry LXCs (unmanic, immich, a few docker lxcs) and I believe when they were at full use it left little to no CPU for PVE, etc. which resulted in very poor performance. I have now updated all my containers to only be given 50% of available CPUs at most. I will monitor and report back if anything changes.