File corruption when transferring files from PyBoard V1 to Windows 11 PC using File Explorer

[etherbrat]

This is a testing of the waters (a bit lengthy for which I apologise).

PyBoard V1.0 - I have very recent firmware installed (latest before the introduction of ROMfs). I have definitely had file corruption resulting in lost time, but I can’t ID the cause. I’m using Thonny IDE for development of Python files on the PyBoard SD card.

The application has no exit statement, and includes interrupt code, so can only be stopped by using Thonny’s stop button. The application itself does not read or write files.

When I try to back up (copy, using Windows 11 File Explorer) my Python files from the SD card to the Windows PC, and especially when the file name on the PC already exists, resulting in the “Skip or Replace” prompt, both the source and the destination file are corrupted in the same way - usually both comprise an almost-complete snippet of a previous version of the file( (I kid you not!). I try to make sure that my application is not running at the time, but I may slip up occasionally, as there is no output to the REPL pane to indicate that anything is running.

My solution at the moment is to copy and paste source code (text, using an editor) from the PyBoard files to new files on the PC.

I am aware of multiple reports of file copy problems on Windows 11.

It strikes me as odd that Micropython syncs the file system after a restart but not immediately after the “Go” button is pressed in Thonny, after which the source code is saved, and the code is then run.

So the questions are: Is this a bug? Is this kind of corruption likely to happen in certain conditions? What’s a bullet-proof process for copying files from the SD card to the PC? Exclude the MicroPython utility program unless it’s fundamentally necessary for some reason; I prefer GUI tools.

[Josverl]

The problem is that no two computers can reliably write to the same physical filesystem at the same time without corrupting it.
And that is exactly what you are doing, and suffering from.
The problem is not Windows , nor MicroPython , nor Unix ( which would see the seem problems) but your setup and workflow.

Bulletproof: Avoid writing and reading from two places at one.
Best solution during development :

• edit your files on your PC
• Use mpremote mount <sourcefolder> , will allow your MCU to see a \remote folder with the files in the on your PC
• its a bit slower for read/write - but guaranteed no corruptions
• Add git init and you have a full and mature version management system

If you can eliminate all writing on the MCU, you will have less problems , but you will need frequent MCU restarts so make the MCU 'see' recent updates from your host.

[etherbrat]

Really? I thought I was at pains to make clear that no two computers are writing to the same physical file system at the same time: Thonny writes the files on the SD card; Windows reads the files when copying them. Unless you are asserting that Thonny does not close the file between saves, which would be a bug. How does "mounting" a remote directory (whatever that means for MicroPython) change any of this? I think you are shooting the messenger.

[Josverl]

Is there any MicroPython code running on your mcu that opens files?

Computer 1

Tonny is running on windows, so it's not tho NY , but windows writing

Computer 2:
Pyboard

[etherbrat]

As I said in my original post: No

[Josverl]

As I said in my original post: No
Indeed , I overlooked that.

Still the root cause remains:

Filesystems like FAT (commonly used in embedded systems) are not designed for concurrent access by multiple systems. > And using CDC+MSC is doing exactly that.

As such there is a large body off issues and reports that have been tracked down to the setup you are using.
It is convenient, but fragile , rather than convenient and bulletproof what you are asking for.

a few examples :

• https://github.com/orgs/micropython/discussions/16012
• https://github.com/orgs/micropython/discussions/14452#discussioncomment-9364538
• https://github.com/orgs/micropython/discussions/14315
• ESP32 S2/S3 USB MSC support? #8426
• Memory corrupted after micropython firmware programming Arduino OPTA #17020
  and the FAQ on this:
• https://github.com/peterhinch/micropython-samples/blob/master/README.md#15-a-usb-pitfall

If you prefer the convienence , and and want to improve reliability:
CircuitPython has implemented a automatic read-only switch that prevents corruption, by automatically limiting functionality.

[etherbrat]

Well ok. As your message came in, I was looking at a Python file that I had copied and pasted from the Thonny code pane to an editor on the PC. I closed Thonny and the editor. The PC file (examined with an Windows editor) contains the right text, but the file on the PyBoard SD card (examined with a Windows editor) is now corrupted (truncated). Sheesh. That tells me that the MicroPython firmware does not properly emulate an SD card as a stand-alone entity. No SD card worth the money would do what it does. I don't understand how mpremote can improve on this, unless it papers over some pretty big cracks, but I'll give it a try.

[robert-hh]

As said, it's a problem of concurrent writes by two entities and of caching. The PC software assumes to be the only entity having control over the SD card. So it can cache an write as needed for best performance. With MicroPython and SD card emulation there are two entities writing, and in addition Thonny as known source of surprises.
Using mpremote the situation is different, since it's only code on the MicroPython board accessing MicroPython file systems, and only PC software accessing files on the PC. That applies to simple file operations like mpremote cp but also to mpremote mount. But when using mpremote, DISBABLE USB MSC mode at the MicroPyhton board.

[scruss]

Looks like it's built for a command line junkie. After mounting the local directory, can I kill off the REPL but leave mpremote running so that Thonny can connect?

When you have MSC turned off, you can use Thonny's Files pane (View menu → Files) to access files on the Pyboard. This will avoid file corruption.

[etherbrat]

I’ll try it. Sorry for this dumb question, but what’s the recommended way of running my program from the REPL?

[scruss]

If you're using Thonny, using Thonny's built-in REPL is usually okay. If you notice output problems, though, mpremote in a terminal windowshould give you a clearer picture of what's happening

[etherbrat]

Ok. mpremote mount . fs ls run
What can I say? Yuk!! I mean to go forward, not backward, in features and convenience.

[scruss]

Use Thonny or mpremote. They're unlikely to work well together.

Thonny does unspecified things to REPL output. While it may be convenient to use, it can (inadvertently) hide problems for users. There are lots of threads here where the real problem was Thonny hiding/masking error output.

You might not like how mpremote looks, but at least it's honest with what it tells you. mpremote on its own should open the REPL. mpremote run file.py runs the local file.py.