Restart of HAOS from Proxmox results in Supervisor integration not connecting

[drthanwho]

This is a weird one I've seen some people (including me) have in the greek HA discord server. We have all set automatic proxmox backups of the HAOS VM. Everytime the backup starts (stop mode), the VM is stopped and when it comes back up there is no connection with the Supervisor (hassio integration). Even though the Supervisor is indeed online.

Restarting HA will then fix it and Supervisor related things will show.

I've allso seen the same happen if I hit reboot on the VM from Proxmox.

Is anyone else seen the same behaviour? Any ideas why this is happening?

[tteck]

I will attempt to replicate the process and see if I can achieve the same result...

[tteck]

The initial test was executed as anticipated, and no issues were encountered.

INFO: starting new backup job: vzdump 112 --remove 0 --notes-template '{{guestname}}' --node server --storage backups --mode stop --compress zstd
INFO: Starting Backup of VM 112 (qemu)
INFO: Backup started at 2023-06-11 16:20:29
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: haos10.2
INFO: include disk 'scsi0' 'local-lvm:vm-112-disk-1' 32G
INFO: include disk 'efidisk0' 'local-lvm:vm-112-disk-0' 4M
INFO: stopping virtual guest
INFO: creating vzdump archive '/mnt/backups/dump/vzdump-qemu-112-2023_06_11-16_20_29.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '8ef97f58-6138-4e2d-b62b-c9a438540dbb'
INFO: resuming VM again after 15 seconds
INFO:   4% (1.3 GiB of 32.0 GiB) in 3s, read: 446.0 MiB/s, write: 165.8 MiB/s
INFO:   6% (2.2 GiB of 32.0 GiB) in 6s, read: 309.3 MiB/s, write: 155.2 MiB/s
INFO:  11% (3.7 GiB of 32.0 GiB) in 9s, read: 508.6 MiB/s, write: 58.9 MiB/s
INFO:  15% (5.0 GiB of 32.0 GiB) in 12s, read: 434.4 MiB/s, write: 251.8 MiB/s
INFO:  16% (5.4 GiB of 32.0 GiB) in 15s, read: 157.7 MiB/s, write: 155.3 MiB/s
INFO:  22% (7.3 GiB of 32.0 GiB) in 18s, read: 621.1 MiB/s, write: 173.1 MiB/s
INFO:  26% (8.6 GiB of 32.0 GiB) in 21s, read: 449.5 MiB/s, write: 165.8 MiB/s
INFO:  41% (13.3 GiB of 32.0 GiB) in 24s, read: 1.6 GiB/s, write: 158.5 MiB/s
INFO:  64% (20.5 GiB of 32.0 GiB) in 27s, read: 2.4 GiB/s, write: 2.7 KiB/s
INFO:  98% (31.6 GiB of 32.0 GiB) in 31s, read: 2.8 GiB/s, write: 7.0 KiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 32s, read: 440.8 MiB/s, write: 0 B/s
INFO: backup is sparse: 28.24 GiB (88%) total zero data
INFO: transferred 32.00 GiB in 32 seconds (1.0 GiB/s)
INFO: archive file size: 1.44GB
INFO: adding notes to backup
INFO: Finished Backup of VM 112 (00:00:47)
INFO: Backup finished at 2023-06-11 16:21:16
INFO: Backup job finished successfully
TASK OK

[tteck]

Which backing file system are you utilizing?

[drthanwho]

Usin GZIP and Stop mode. Regarding the backup itself it works just fine and backups are created. The issue is with HA. Realistically I should open a bug with them. But want to first see if this is common at all

[tteck]

Examples of backing file systems include ext4, NTFS, ZFS, and Btrfs, among others. I'm utilizing ext4.

[drthanwho]

Ohhhh oops. ext4 😄

[drthanwho]

This is what shows up in the logs

2023-06-11 23:39:14.217 WARNING (Recorder) [homeassistant.components.recorder.util] Ended unfinished session (id=1445 from 2023-06-11 15:00:58)
2023-06-11 23:39:14.799 WARNING (MainThread) [homeassistant.components.hassio] Not connected with the supervisor / system too busy!
2023-06-11 23:39:15.395 WARNING (MainThread) [homeassistant.components.hassio] Can't read Supervisor data: System is not ready with state: setup
2023-06-11 23:39:15.412 ERROR (MainThread) [homeassistant.components.hassio.addon_panel] Can't read panel info: System is not ready with state: setup
2023-06-11 23:39:15.471 ERROR (MainThread) [homeassistant.setup] Error during setup of component hassio
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 286, in _async_setup_component
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/hassio/__init__.py", line 639, in async_setup
    await issues.setup()
  File "/usr/src/homeassistant/homeassistant/components/hassio/issues.py", line 300, in setup
    await self.update()
  File "/usr/src/homeassistant/homeassistant/components/hassio/issues.py", line 308, in update
    data = await self._client.get_resolution_info()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/hassio/handler.py", line 52, in _wrapper
    raise HassioAPIError(data["message"])
homeassistant.components.hassio.handler.HassioAPIError: System is not ready with state: setup

which of course results in no Addons sicne there's no connection to SU

Just not sure if this is related to Core or an OS/SU issue

[tteck]

Strange indeed 🤷🏻‍♂️

[drthanwho]

Where would you even open up an issue under? 🤔

[tteck]

lol... That is an excellent question! If the issue cannot be readily reproduced, it may indeed be challenging to obtain assistance.

[drthanwho]

Happening to 3 people I know so far. Has to be somehow reproducible

[tteck]

It is likely that there is a common factor among the three instances, apart from HAOS, that may be causing a bottleneck. Could this potentially be attributed to a custom component, integration, or another element?

[tteck]

Can the issue be reproduced with a fresh VM of HAOS?
By starting with a clean VM, you can eliminate any potential lingering configuration or custom component, integration issues that may be causing the problem.