Podman --volume and modifying the lower dir of an Overlay mount

[PhrozenByte]

Podman's --volume option supports the O flag to create an overlay mount. It's even possible to manually specify the upper and work dirs. However, I stumbled over a notice in the docs regarding the lower dir:

podman/docs/source/markdown/options/volume.md

Lines 136 to 137 in 6b8bc6f

	Do not modify the source directory mounted into the <<container|pod>> with an overlay mount,
	it can cause unexpected failures. Only modify the directory after the container finishes running.

I did some Git blaming and this notice exists from the start.

What "unexpected failures" does modifying the lower dir of an overlay mount cause?

I use (better: used, I'm currently reviving a few things) the "original" unionfs-fuse to create multi-layered union mounts for complex dev projects for more than 10 years now. I never had any issues with modifying a lower dir while unionfs is mounted. I never switched to overlayfs (neither native, nor fuse-overlayfs), simply because the old setup worked, but since I switched to containers on production a long time ago, it was about time to do the same for my dev setup. Consequently, I'm investigating alternatives for my dev setup now. I'm curious what issues might arise when modifying the lower dir of a Podman --volume Overlay mount?

[Luap99]

I think this is kernel warning mostly, I don't know what bad things could happen maybe @giuseppe knows more about it

https://docs.kernel.org/filesystems/overlayfs.html#changes-to-underlying-filesystems

Changes to the underlying filesystems while part of a mounted overlay filesystem are not allowed. If the underlying filesystem is changed, the behavior of the overlay is undefined, though it will not result in a crash or deadlock.

[giuseppe]

with "unexpected failures" we only refer to the kernel warning that @Luap99 pointed out.

If you make sure the "metacopy", "index", "xino" and "redirect_dir" features are not used for the overlay mount, then offline modifications are ok; but with a running container, you are basically changing the data overlay works on under its feet. Some operations are not atomic (and that is ok if the lower dir is not modified as it should be), so depending on the timing you might get some weird results.

[PhrozenByte]

Thank you very much! This helps a lot. I didn't even think about this being a kernel limitation at first, because there's no such limitation with unionfs-fuse. I did some more research and the issue seems to be with caching: The kernel simply makes the assumption that the lower dir is immutable (which is weird IMHO, this is unique for a dir-based mount, only block-based filesystems make that assumption and more importantly enforce it) and thus just doesn't check whether e.g. a new file appears.

Does this limitation apply to fuse-overlayfs as well?

Unfortunately unmounting (i.e. stopping the container) is no option. In theory I could setup things similar to how deploying to a remote server works, but this also requires rather complex path rewriting and more critical path merging (unfortunately; that's a technical debt of an old framework we chose 14 years ago). Unfortunately my IDE doesn't support that. As a last resort I could write a sync script with inotify hooks...

It looks like that the reason this being no issue with unionfs-fuse is that it uses way less caching. When performing e.g. a dir listing it just checks all dirs, all the time. This makes it very slow in comparison (i.e. one filesystem operation becomes two, three, four, …), but I don't care much, because everything is on a NVMe and we're talking about a local dev setup…

So, I tested it and it indeed seems like that bind-mounting a unionfs-fuse mountdir into the container works. Are there any limitations I should be aware of?

[giuseppe]

Does this limitation apply to fuse-overlayfs as well?

yes in the sense that there is some caching happening and that is of course invalid if the underlying layer is modified.

I am not sure about unionfs-fuse not having these problems, probably you've not hit it yet, because if a file operation becomes two or more, then it seems to me there is the chance of a race condition and you'll hit the same problem.

[PhrozenByte]

Of course, race conditions are still possible, just as with any other concurrent I/O. For example, unionfs-fuse uses the FUSE page cache. My question rather was whether fuse-overlayfs implements additional caching that might cause issues not related to race conditions.

I did some testing and the answer is: It does. fuse-overlayfs doesn't just end up doing weird things like native overlayfs, it just completely breaks.

Let's create a simple test setup for unionfs-fuse vs. native overlayfs vs. fuse-overlayfs:

test:~ $ mkdir lower upper work union
test:~ $ mount … # see below
test:~ $ echo foo > lower/file1
test:~ $ cat union/file1
foo
test:~ $ echo bar > union/file1
test:~ $ ls lower upper union
lower:
file1

union:
file1

upper:
file1
test:~ $ cat lower/file1
foo
test:~ $ cat upper/file1
bar
test:~ $ cat union/file1
bar
test:~ $ mv lower/file1 lower/file2
test:~ $ …

First of all, let's try unionfs-fuse for a baseline:

test:~ $ unionfs -o cow upper=rw:lower=ro union
test:~ $ …
test:~ $ mv lower/file1 lower/file2
test:~ $ ls lower upper union
lower:
file2

union:
file1  file2

upper:
file1
test:~ $ cat lower/file2
foo
test:~ $ cat upper/file1
bar
test:~ $ cat union/file1
bar
test:~ $ cat union/file2
foo

Yes, it's a little weird that moving the file in the lower dir makes two files appear in the union dir after we've modified the file there, but this kinda makes sense: It behaves as if there was one file in the lower and another file in the upper dir from the beginning.

Let's see how native overlayfs performs:

test:~ $ sudo mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work union
test:~ $ …
test:~ $ mv lower/file1 lower/file2
test:~ $ ls lower upper union
lower:
file2

union:
file1  file2

upper:
file1
test:~ $ cat lower/file2
foo
test:~ $ cat upper/file1
bar
test:~ $ cat union/file1
bar
test:~ $ cat union/file2
bar

Again two files appear, but union/file2 returns bar? That doesn't seem right... I mean, from a continuity perspective this behaviour also makes kinda sense (we modified the file earlier, so it makes kinda sense that it still returns that content)... It's just that by remounting the filesystem the contents suddenly change.

So, it behaves differently, maybe a little unexpected, but at least doesn't break. What about fuse-overlayfs?

test:~ $ fuse-overlayfs -o lowerdir=lower,upperdir=upper,workdir=work union
test:~ $ echo foo > lower/file1
test:~ $ cat union/file1
cat: union/file1: No such file or directory

Okay, that's quite a sudden failure... 😄

Here are some more tests with fuse-overlayfs: (click to expand)

test:~ $ fuse-overlayfs -o lowerdir=lower,upperdir=upper,workdir=work union
test:~ $ echo foo > lower/file1
test:~ $ cat union/file1
cat: union/file1: No such file or directory
test:~ $ ls lower upper union
lower:
file1

union:

upper:
test:~ $ echo bar > union/file1
test:~ $ ls lower upper union
lower:
file1

union:
file1

upper:
file1
test:~ $ cat lower/file1
foo
test:~ $ cat upper/file1
bar
test:~ $ cat union/file1
bar
test:~ $ mv lower/file1 lower/file2
test:~ $ cat union/file2
cat: union/file2: No such file or directory
test:~ $ ls lower upper union
lower:
file2

union:
file1

upper:
file1
test:~ $ rm upper/file1
test:~ $ ls lower upper union
lower:
file2

union:
file1

upper:

test:~ $ cat union/file1
cat: union/file1: No such file or directory
test:~ $ cat union/file2
cat: union/file2: No such file or directory

I'm not sure whether fuse-overlayfs wants to replicate the kernel's undefined behaviour, but my question is definitely answered: Neither native overlayfs nor fuse-overlayfs work.

So, right now it looks like the best solution is to keep on using unionfs-fuse. As I said earlier, I already tested whether the unionfs-fuse mount can be bind mounted into the container and this seems to work. Are there any limitations with bind-mounting a FUSE filesystem like unionfs-fuse into a container I should be aware of?

Thanks again for your help! ❤️

[giuseppe]

No, there should not be any difference. You need the user owning the mount mapped in the container though, so be careful if you specify different mappings