Plot corruption after copying over LAN and then from one HDD to another

[Qwinn1]

Hello. Hoping someone can help. This is a 100% reproducible issue.

I have 3 plotting machines. Last week I ran out of room on 2 of them and still had room on the 3rd, so I copied new plots over my 2.5Ghz LAN to the 3rd machine for a couple of days. Initially, everything seemed fine.

But then, after adding some more HDDs, I decided to switch over all my drives from ext4 to btrfs for the long haul. That of course involves copying everything to the empty disks, reformatting, and filling them back up. During this process, I discovered that after copying from one internal drive to another (regardless of filesystems) on that 3rd machine, every plot I had previously copied across the network began generating the following error in the logs as soon as the move to the new HDD was finished:

"2021-06-25T09:38:25.028 harvester flax.plotting.plot_tools: INFO Not checking subdirectory \XXXXXX\Plot28\plot-k32-2021-06-21-20-15-2760f36273c332ba3f889bb83f01968bc9a9e8f77ee5116e39e89782d07565ef.plot, subdirectories not added by default"

Here are some relevant details:

1. There are no errors in the log generated by these plots UNTIL they are moved from one drive to another on the 3rd server. They are "found" and seem to work fine until they are moved.
2. Files that were copied across the network had permissions "-rw-rw-r" on the originating server but landed on the destination server as "-rwxr--r--".
3. Fixing these permissions prior to the copy from one HDD to another does NOT fix the issue. They will still generate the above errors upon completion of the move from one HDD to another. Fixing permissions after the copy does nothing either.

If it helps, this appears to be the relevant bit of code from plot_tools.py:

def _get_filenames(directory: Path) -> List[Path]:
    try:
        if not directory.exists():
            log.warning(f"Directory: {directory} does not exist.")
            return []
    except OSError as e:
        log.warning(f"Error checking if directory {directory} exists: {e}")
        return []
    all_files: List[Path] = []
    try:
        for child in directory.iterdir():
            if not child.is_dir():
                # If it is a file ending in .plot, add it - work around MacOS ._ files
                if child.suffix == ".plot" and not child.name.startswith("._"):
                    all_files.append(child)
            else:
                log.debug(f"Not checking subdirectory {child}, subdirectories not added by default")
    except Exception as e:
        log.warning(f"Error reading directory {directory} {e}")
    return all_files

Any help or idea what would be causing this? My LAN is operating on SMB2/SMB3 network protocol. The 3 plotters in question are running Ubuntu 20.04, but the farmer generating these errors in the log is farming via network shares on Windows 10. Hoping I just missed adding something to smb.conf or something.

[Qwinn1]

Additional info:

I get these errors on an INFO level log. I note this issue #6058:

#6058

...which suggests that I should only have seen this in DEBUG level log, which seems... odd.

[Qwinn1]

Some additional information based on further experimentation:

I set "map archive = no", "map system = no", "map hidden = no", and "inherit permissions = yes" in smb.conf of both servers and restarted the smbd services on both. This has stopped the permissions from getting screwed up in the network transfer. But this has not stopped the problem.

Also: Initially, I made the sloppy error of not renaming the plots I copied from ".plot" to something else for the copy across the network, since the client seems to be able to deal with not renaming them when copying from one HDD to another internally (it'll throw some errors, but the final plot in its final destination will get recognized and used without issue once the move is complete, and the plot can be moved yet again without issue). I tested by copying a plot across the network named something else so that the harvester never read the plot in an incomplete state. When I renamed the plot to .plot in its destination directory, the harvester immediately "found" it and all seemed to work fine. I figured I'd tackled the problem, but just in case I then moved that plot to another HDD... and bam. "Not searching subdirectory" happened again.

I don't necessarily need a way to fix the plots that have been corrupted in this way - if I have to wipe them and replot, not a big deal. What I can't deal with is being unable to copy plots across the network without having them effectively burned onto the drive they are on such that if it is ever relocated anywhere else it becomes unusable. This means if one of my plotters run out of room, I have to shut down and physically move drives around if I want to continue plotting on that server. THAT is a serious pain in the neck.

Halp! Please!

[loppefaaret]

is the "file descriptor" indicating that the entry for the ".plot" item in the filesystem is a directory, or a file ?
it might seem to indicate that this item on the filesystem, is flagged as being a directory ?
if you do a ls -l does the permission list the .plot as a "-rwxr--r--" or "drwxr--r--" ?
and finally, i see the error message is referencing flax.plotting.plot_tools and not chia.plotting.plot_tools
should you perhaps go talk to the flax people ?

[Qwinn1]

Thank you for the reply!

"if you do a ls -l does the permission list the .plot as a "-rwxr--r--" or "drwxr--r--" ?"

-rwxr--r--. Weird, eh? Although as I said, even after I got the permissions fixed in the transfer, the problem persists. The executable bit being set in my earlier attempts doesn't appear to have anything to do with it. If there's some other way to distinguish a file as having the directory bit set apart from the permissions beginning with a "d" when doing ls -l, I'm not aware of it.

"and finally, i see the error message is referencing flax.plotting.plot_tools and not chia.plotting.plot_tools should you perhaps go talk to the flax people ?"

Flax is a very useful tool for diagnosing issues with my chia setup because I can experiment and shut it down and restart it without interrupting my chia farming. But the code is identical in pretty much every way, and I see the exact same issues in my chia log as I do in my flax log. In fact, the problem began before I ever installed flax.

[Qwinn1]

Actually... looking at the code snippet I posted earlier... I'm not all that familiar with python, but:

            if not child.is_dir():
                # If it is a file ending in .plot, add it - work around MacOS ._ files
                if child.suffix == ".plot" and not child.name.startswith("._"):
                    all_files.append(child)
            else:
                log.debug(f"Not checking subdirectory {child}, subdirectories not added by default")

I do know a lot of other languages, and I am curious, what syntax makes that "else:" the else condition for "if not child.is_dir():" rather than "if child.suffix == ".plot" and not child.name.startswith("._"):"? The only thing I could see that would is the indentation, and this would be the first language I've played with where indentation actually modifies the context of code.

So, for a second assume my reading right and the else that is printing that error is not attached to "if not child.is_dir()" but rather the "if child.suffix == ".plot" and not child.name.startswith("._")". Still don't know why a file that had been copied over the network would fail that check either.

And the mystery persists as to why I see this error in my INFO logs (both chia and flax) when it clearly says "log.debug"...

[Qwinn1]

So I investigated Python a bit and it turns out, it IS the indentation that makes the else match the correct if. Cool. And glad I was able to correctly diagnose what would be required for that code to act the way it appeared it was meant to.

So, yeah, that only leaves the possibility that somehow those plots, after the network copy and then HDD transfer, are returning child.is_dir() = true. Yet when you ls -l, they don't show the directory bit set. And it only happens if you network copy AND then transfer to another disk internally - if you just network copy, or just move internally, the plots are found and work fine. I am still completely stumped on how that can be. Does anyone have any ideas? Can anyone else replicate it? Shouldn't take that long to test...

[BrandtH22]

Since this issue has been open for a number of years without additional comments, we will be closing this ticket but if we have closed this ticket in error do not hesitate to reach out to us again with any followup questions, comments, or if the issue persists after an update.

Please note that upgrading major versions of chia generally requires the config.yaml to be recreated using these steps:
1. Stop chia
2. Rename the config or move it to a different directory
3. In CLI run chia init
4. Copy any custom settings from the old config to the new (plot drive directories, pool info, trusted node info, etc) - be careful here as yaml is very picky about formatting including those leading spaces
5. Start Chia

The best place to reach our support team is on Discord (https://discord.gg/chia) or by reopening this ticket.