Backup: Going against the grain to avoid double data

[aakruno]

Hi,

We currently have around 20 TB worth of data in our LME setup and we are working towards having a decent backup solution for this. My VMWare guy is screaming at me to avoid doubling this data, so I have decided to look into a solution that might break our LME setup a bit, or at least will go against what is documented.

Normally, in a containerised setup like this, I would mount the data path in a container to point towards, for example, an NFS share (as we are talking backup). Before, on a completely different journey, I tested out moving /var/lib/containers/storage/ onto a separate partition, but ELK would not start as it had detected that its path had been changed and there were probably some configurations that need to be changed before it was all too happy, so I knew it wouldn't be as easy as such.

I circumvented this by just mounting the new partition to /var/lib/containers, not pretty, but supported and doable.

Now, I am thinking, if I do not find a way to actually define the lme-backups container storage volume, I can find its container data volume and mount the NFS share to that path. Again, not pretty, definitely not supported but sometimes you have to circumvent the software to make the hardware happy.

I have, however, seen this in the backup documentation: Note: If managing backup storage becomes a concern, the LME Team plans to release future documentation for advanced options.

And I am hoping that this discussion thread might provide some answers from the team in what they were thinking in that regard, if anything, so I can avoid breaking too much.

[cbaxley]

You can make a symlink of the storage directory to another place where you want to actually store things.
This way the scripts and configs all work, but you have it where you want it.
Shut everything down.
Copy the storage dir to where you want them.
Rename the storage dir (for safe keeping)
mv /var/lib/containers/storage /var/lib/containers/storage.old
Then make your symlink and try starting everything again.
ln -s /new/directory /var/lib/containers/storage

You might have to reboot the whole machine to get podman to see the new volumes.

[aakruno]

You are right, I had completely forgotten symlink and that would have been the best way to manage this.

However, I still need to point lme-backup somewhere else, i.e. a NFS share. From a cautionary search, it seems that podman doesn't like putting container storage anywhere else than the path set in /etc/containers/storage.conf and I can't seem to find a decent way around it, as container storage IDs will most likely change upon update.

I would prefer to have it automated, without having to have double the amount of space available, then move it over afterwards and then removing the snapshot.

In an ideal world, it would still exist inside of ELK as part of the snapshot as well.