Reliable Database Backup (dont just download [corrupted] db file]

[scyto]

Is your feature request related to a problem? Please describe

There are several scenarios where database corruption can occur.
This may happen if someone does something unsupported (put db on cifs/nfs)

Can also happen in scenarios outside of this unsupported scenario (e.g. unanticipated power loss, hardware issues, etc)

Today portainer can keep running just fine even if the on-disk database is corrupted.

This means portainer can run for weeks or months with corrupted database and because backup is a copy of the on-disk database all the backups can be corrupted without this being obvious.

Describe the solution you'd like

Change backup from a file copy process to a process that backs up (exports?) the tables to a new fresh and VALIDATED GOOD database file.

Describe alternatives you've considered

Doing manual script to stop portainer, backup the file, start portainer, then validate the file with bbolt / bolt db command like tools.

Additional context

I had this issue, this is not theoretical, the file was not on cifs/smb/nfs (i was using virtiofs to surface the bind mounts from a host into a swarm VM). I had no working backup file due to only keeping a few (bonus ask, let me do automate backup to Azure or SMB too)

[Nick-Portainer]

Hi @scyto Thanks we are tracking this internally

internal: FR-764

[scyto]

@Nick-Portainer hi any update, just hit this again, it make doing any version bump scary, i don't understand why it only happens on version bumps, or rather why a working system that goes up and down and has one corruption in one table works flawlessly untill an upgrade, are you doing agressive DB upgrades at start time?

since i logged this i (a darn good product manager but zero coding experience) have learnt vibcoding, so i made this in about 30 mins tonight.... https://github.com/scyto/portainer-database-salvage