beets "Safer Move"

[slannez]

When importing music libraries over the network many wrong things might happen, and erros might not be caught up by the OS. Is there a plan to implement a "safe move" operation which would check the content of the original and destination files are exactly the same before deleting the source?

[sampsyo]

Hmm; that sounds pretty specialized! Are you imagining a scenario where the filesystem says to beets, "yep, no problem, just finished that copy!" but actually the destination file is corrupted? So beets needs to checksum both files to ensure that they match after every copy? It seems possible to implement in a plugin, FWIW—you might even be able to accomplish that with the hooks plugin. We don't currently have plans like that for the beets core, however.

[wisp3rwind]

In fact, I doubt that this can easily be implemented as a plugin — hooks are probably too coarse-grained to intercept files being overwritten.

But I'm with @sampsyo, working around unreliable network file systems seems far out of scope. Is there any precedent of other software doing so? Would that even be a reliable workaround? After all, reading back the new file might read from a local cache.

#3796 might be interesting to watch for you.

[catap]

@slannez I'm trying to improve network usage by #4060

As nice side effect it removes file only when destination one is renamed.

Anyway, I'm very wonder when is it possible to write file via network (via OS function) and has wrong result?

[RollingStar]

Might be relevant. I find Python's unsafe file handling baffling. https://moltenform.com/articles/avoiding-race-conditions-copying/

[slannez]

Accuracy of file manipulation (copy/move/change) over the network highly depends on the OS and file system drivers implementing the functions. Copying from NAS to another NAS (Eg mounted folders for example) might lead to silent data corruption if the copy is pure binary-to-binary without proper checksum sanity check at the end of the operation. I lost quite a large library of document in the past... silently... proper bytes being replaced by zero -> no file size changes, no way to record data because old data was replaced with zeroes... IMHO the proposed commit is going in the right direction. Maybe we could add a "--safe" option to beets that would also check the content of the two files is higly likely to be identical... Randomly picking bytes from the files (if FS driver allow for efficiently random access of the files), or getting some kind of free/cheap checksum?

…

------------------------------------------ Sébastien Lannez +33 6 42 40 49 98 ------------------------------------------

On Sun, Oct 24, 2021 at 8:50 PM RollingStar ***@***.***> wrote: Might be relevant. I find Python's unsafe file handling baffling. https://moltenform.com/articles/avoiding-race-conditions-copying/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4082 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4ZGJANOPYFU2H4TNAS4OTUIRIOTANCNFSM5FETKD7Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[catap]

@slannez I'm very wondering on which OS/Network/driver you may achieve corruption of file via cp if it returns 0?

[wisp3rwind]

To be honest, and as I've stated in #4082 (reply in thread) (in particular the second paragraph there), I don't think this "paranoid" mode should be implemented by beets, it's not our job to workaround buggy filesystems or unstable hardware. Where we should improve is atomicity of file operations (started in #4060, but not complete due to lacking at least the mediafile layer).