Support writable overlay mounts on squash mounts

[dux119-cmd]

Operating systems like DOS, Amiga, macOS, and others expect to change the filesystem, and this is frequently used by games to save settings, write save game files, (among other uses).

Because squashfs is a read-only mount, this limits its use to games that don't modify their filesystems. Figuring this out for DOS games can be trial and error, and worst case the user has to forfeit the tidy benefits of using squash.

This PR lets emulators indicate that they need write access to the ROM area, in which case a writable overlayfs mount will be placed on top of the squashfs mount (if squash is used), with the writes going into the usual /userdata/saves/<platform>/<game> area.

The overlay feature is a follow-on context after squash:

• It handles squash's directory or single-file and linked output
• It logs and cleans up the virtual area on context close
• It preserves the writeable area if changes were made, but otherwise cleans it up if it's empty.

Sample log statements:

DEBUG (overlayfs.py:43):_mount mounted '/var/run/overlays/biomenace-bare.pc'
      with components 'lowerdir=/var/run/squashfs/biomenace-bare.pc,
                       upperdir=/userdata/saves/dos/biomenace-bare.pc/upper,
                       workdir=/userdata/saves/dos/biomenace-bare.pc/work'
    
DEBUG (overlayfs.py:85):mount_overlayfs cleaning up
      '/var/run/overlays/biomenace-bare.pc'
    
DEBUG (overlayfs.py:90):mount_overlayfs keeping populated save directory
      '/userdata/saves/dos/biomenace-bare.pc'

I've tested all four types of changes:

• Modifying an existing file in the squash base
• Deleting an existing file in the squash base
• Adding or changing a new file that doesn't exist in the squash base
• Deleting a previously created file not in the squash base

Using squashfs for PC games has a nice side-effect of keeping the underlying game in a known pristine state as any risky changes (patches, cracks, cheats and so on that a user might employ) can be wiped clean by deleting the game's save area.

As of this PR I've only toggled it on for the DOSBox Staging generator, but could be easily be added to others if someone has working ROMs for those platforms (Amiga, MacOS, etc..)

---

Testing:

1. Unzip biomenace-bare.pc.squashfs.zip into your /userdata/roms/dos/ area
2. Launch it with main and observe that C:\> echo hello > world creates an access denied error.
3. Launch it with this branch and you'll see world is created. This new file will persist across re-launches (and reboots) of the emulator and physical machine.
4. If you C:\> del world and then close the emulator you'll see that the /userdata/saves/dos/biomenace-bare.pc folder has been removed because there are no longer any changes versus the underlying squash.
5. Making changes to existing files (for example, del BMENACE3.EXE, or changing the content of a file) will persist (and you can dig into the saves area to see how they're represented).

[dmanlfc]

we are awaiting a major change which will affect this PR, just FYI

[dux119-cmd]

Thanks for the heads up @dmanlfc.

[dmanlfc]

currently PR merges are on hold. keep an eye on Master for the transition. the wip from Bryan is here as an fyi - master...bryanforbes:batocera.linux:feature/improve-es-features

[Darknior]

@dux119-cmd I’m discovering that it’s possible to use SquashFS with DOS? Why not — if you do it, it could actually be interesting. For compressing games, maybe? I don’t use DOSBox Staging.

In any case, with PURE you can zip the games, so I assume it more or less amounts to the same thing.

On the Batocera side, we added several tricks to handle Wine game saves properly, so they’re no longer lost in the bottles stored under system/ — that setup wasn’t great. Now we can declare the save path directly in the command file and redirect it to saves/, like you do, using symbolic links.

BUT… I have to admit that your approach of using a writable overlay and keeping only the modified files in saves/ is even better.

That really deserves to be added to WINE, if you can 🙂

[dux119-cmd]

currently PR merges are on hold. keep an eye on Master for the transition. the wip from Bryan is here as an fyi - master...bryanforbes:batocera.linux:feature/improve-es-features

Thanks for pointing me to this branch, @dmanlfc.

Moving the emulator properties out to YAML records looks great! I'll add the "writes-to-rom" property there, and drop it from the generator base and derived classes.

Then I can get this property from the EmulatorInfo object when we're mounting the squashfs and overlayfs, instead of having to jump through the hoops to setup a temporary generator.

[dux119-cmd]

@dux119-cmd I’m discovering that it’s possible to use SquashFS with DOS? Why not — if you do it, it could actually be interesting. For compressing games, maybe? I don’t use DOSBox Staging.

Yes; squashfs is really nice because it dedupes as well, so it's the perfect container for holding lots of variations of a single game (say, multiple languages, multiple versions, maybe with or without copy-protection), and then using a DOS choice menu to select the variant to launch.

It's even better than zip, because squashfs supports zstdandard (level 22, 1 MB chunks), and there are kernel settings that decompress chunks in parallel. So it runs circles around lzma and zip in terms of read latency.

In any case, with PURE you can zip the games, so I assume it more or less amounts to the same thing.

Yup; this is very nice and works well, too. We'll be able to enable squash for it, too.

On the Batocera side, we added several tricks to handle Wine game saves properly, so they’re no longer lost in the bottles stored under system/ — that setup wasn’t great. Now we can declare the save path directly in the command file and redirect it to saves/, like you do, using symbolic links.

BUT… I have to admit that your approach of using a writable overlay and keeping only the modified files in saves/ is even better.

That really deserves to be added to WINE, if you can 🙂

Hopefully :-)

I just need to expand my testing environment and add an x86-64 machine to add wine into the mix.

[Darknior]

Yes; squashfs is really nice because it dedupes as well, so it's the perfect container for holding lots of variations of a single game (say, multiple languages, multiple versions, maybe with or without copy-protection), and then using a DOS choice menu to select the variant to launch.

"Ohhhh, excellent idea! I hadn't thought of that :) I also understand the goal of your PR better now.

Just so you know, we've already considered this scenario ourselves—it happens on all Batocera emulators. However, we didn't go with squashfs for practical reasons regarding game updates, like Amiga LHA files which are updated quite often (new translations, new hacks, etc.). Because of that, we concluded that the best approach would be to simply modify how EmulationStation (ES) displays things. But the issue is that the code is in C++, not Python, and no one on our team is proficient in it :(

By modifying ES, the simple and effective idea would be to manage games by name and folder, similar to how Emby or Jellyfin handle movies. Those platforms are a goldmine for media management ideas since they've had a huge community for years. In Jellyfin, you create a folder and give it the name of the MOVIE (the game, in our case). For example: Megaman 2 (USA) And inside, you name all versions of your game the same way, using "space dash space" to specify the difference. For example: Megaman 2 (USA).zip Megaman 2 (USA) - [Trad Fre 1.0].zip Megaman 2 (USA) - [Hack New Skin ...].zip Megaman 2 (USA) - SuperFX Hack ...

The goal would be for ES to display only the game name as it does now—without flags or RetroAchievements icons—but with a small downward triangle. This would indicate that selecting it "unfolds" the list to show all the "children" versions, allowing the user to choose which one to launch.

This method allows for a specific description, media, and core for each version; it's more work, but it's cleaner and more explicit. The alternative would be a method like yours, where info and media are set for the folder and applied to all children. It’s easier to manage, but I like it less. Since ES can already hide or show folders based on whether they contain one or more games, we would just need to add a rule: "if the name of the child games starts with the name of the parent folder."

It's a bit of a digression, but we think the idea is solid. Even if we don't re-compress games that are already in ZIP format, it makes set management much lighter and super flexible! :)"

I just need to expand my testing environment and add an x86-64 machine to add wine into the mix.

For sure, thanks a lot :)
WINDOWS (WINE) is really the place were we use squashfs in Bato and the read only is a pain ...

[dux119-cmd]

@Darknior - fully agree that a standardized "child games" or "variants" squashfs content layout is the right approach, especially with the simplicity that you've described ("child games start with the name of the parent folder.", because it's simple enough for anyone to hold in their head (or script/automate), so the various archivist groups can create definitive preservation "packs" for specific games or entire platforms (and distribute them as single .squashfs files).

[dux119-cmd]

@bryanforbes , @dmanlfc: ok; rebased against main.

Similar to hud and bezel bools, I added a common shared custom feature to DOSBox Staging's YAML.

As for getting the bool during emulator launch, the PR sticks to using the generator function, which continues to parallel how other code is getting the mouse, hud, and bezel bools for a given emulator.

[dux119-cmd]

(reason for the the force-push is I'm just rebasing against latest main as new commits land)

[dmanlfc]

@bryanforbes are you ok with this now?

[aderumier]

mugen could be a great target to support writable squashfs mount. (the games files are generally readonly, but we need to rewrite config at the root of the game for resolution && other seting).

[bryanforbes]

@dmanlfc I'll review it again today