ZFS Pool has been modied

[nasirq]

Background:
I am using ZFS as the storage on a server running VMs and also user files shared via SAMBA.

And the pools are backed up on another server using a continously running process that creates snapshots and transfers them to the backup over and over again, keeping the backup almost in sync to the primary storage.

And on the backup server, I create snapshots daily so we have the capability to restore data from a previous date if required.

Issue:
We ran out of space on the primary server in the pool that hosts the user's data, because there was an old snapshot that should have been deleted and the changes ate up all the space. Once I came to know that we have run out of space, I simply deleted that old snapshot and everything was back to normal except some MS Outlook .pst files that got corrupted.

However the backup process crashed with the error:

cannot receive incremental stream: destination backup/pool1-home has been modified since most recent snapshot

There is nothing on the backup server that accessed the data, that could have modifed it. I tried doing a rollback, but no luck.

The only solution was to add "-F" to the zfs receive command. But this now deletes the daily snapshots that we take, and now we do not have the capability to go back in time and restore any data if required.

I tried doing zfs diff between the recent snapshot from the backup process and the pool, which says:

Unable to determine path or stats for object 736326 in backup/pool1-home@bkp1: No such file or directory

I tried to look for the file with the inode 736326 on the primary server using the command "zdb -vvv pool1-home@bkp1 736326", and following the tree using the field "parent" got to the file, and created a copy of it, so if it is corrupt in the database a new file is created.

However the problem is not solved. I am unable to run the backup process without the "-F" parameter.

Please help.

[IvanVolosyuk]

Before doing the next backup set the destination read-only. Do forced backup once or rollback to the last received snapshot. After that you shouldn't need -F

…

On Fri, 7 Oct 2022, 23:56 nasirq, ***@***.***> wrote: Background: I am using ZFS as the storage on a server running VMs and also user files shared via SAMBA. And the pools are backed up on another server using a continously running process that creates snapshots and transfers them to the backup over and over again, keeping the backup almost in sync to the primary storage. And on the backup server, I create snapshots daily so we have the capability to restore data from a previous date if required. Issue: We ran out of space on the primary server in the pool that hosts the user's data, because there was an old snapshot that should have been deleted and the changes ate up all the space. Once I came to know that we have run out of space, I simply deleted that old snapshot and everything was back to normal except some MS Outlook .pst files that got corrupted. However the backup process crashed with the error: *cannot receive incremental stream: destination backup/pool1-home has been modified since most recent snapshot* There is nothing on the backup server that accessed the data, that could have modifed it. I tried doing a rollback, but no luck. The only solution was to add "*-F*" to the zfs receive command. But this now deletes the daily snapshots that we take, and now we do not have the capability to go back in time and restore any data if required. I tried doing zfs diff between the recent snapshot from the backup process and the pool, which says: *Unable to determine path or stats for object 736326 in ***@***.***: No such file or directory* I tried to look for the file with the inode 736326 on the primary server using the command "*zdb -vvv ***@***.*** 736326*", and following the tree using the field "parent" got to the file, and created a copy of it, so if it is corrupt in the database a new file is created. However the problem is not solved. I am unable to run the backup process without the "-F" parameter. Please help. — Reply to this email directly, view it on GitHub <#14000>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HPJDVI6YJBBVEGOK33WCAMW3ANCNFSM6AAAAAAQ7SGV24> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[nasirq]

It does not work. Once I set the destination readonly, I cannot do a backup (receive incremental snapshot). The error is:

cannot receive incremental stream: pool is read-only

[IvanVolosyuk]

You have to set the dataset readonly property not the pool readonly property, like: zfs set readonly=on backup/pool1-home

…

On Sun, Oct 9, 2022 at 8:18 PM nasirq ***@***.***> wrote: It does not work. Once I set the destination readonly, I cannot do a backup (receive incremental snapshot). The error is: cannot receive incremental stream: pool is read-only — Reply to this email directly, view it on GitHub <#14000 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HK5CON7DRVDK5J6C2TWCKEWXANCNFSM6AAAAAAQ7SGV24> . You are receiving this because you commented.Message ID: ***@***.***>

[nasirq]

It worked perfectly. Please educate me what was the issue and how this solved it.

[IvanVolosyuk]

I think it is good practice to set receive destinations readonly. This makes it impossible to do accidental changes to the datasets. One common type of accidental changes is just change in access time (atime), which can probably be prevented by zfs set atime=off .... ZFS may create mountpoints for child datasets as well which may changes the datasets as well. Some images viewers can create thumbnail files and all sorts of random changes by random programs... Marking the dataset readonly prevents all of them and the dataset will be exactly at the state of the last received snapshot. Incremental receives will apply cleanly in this case.