Docker containers using vfs when PVE uses ZFS

[lordratner]

Hi,

I just found out about this from troubleshooting a Nextcloud install. It looks like when docker is installed in an LXC on Proxmox with ZFS, the Docker Storage Driver (view with docker info) defaults to vfs.

According to docker, "The vfs storage driver is intended for testing purposes, and for situations where no copy-on-write filesystem can be used. Performance of this storage driver is poor, and is not generally recommended for production use."
https://docs.docker.com/storage/storagedriver/select-storage-driver/

Over on reddit someone was helping me with Nextcloud and he worked through the vfs issue, but it's a rather involved process, which made me think of TTeck! Here's what he had to do, and the issue was setting up the filesystem in a way that allowed for the use of the overlay2 storage driver and maintaining the ability to take snapshots and migrate.

https://www.reddit.com/r/Proxmox/comments/zahqfa/comment/iyo3303/

Your scripted docker LXC, but do you think it would be worthwhile to set it up on the preferred (by docker) filesystem? There is also the zfs storage drive in docker, though it's newer and less documented.

[tteck]

At one time, I had any Docker install with ZFS use fuse-overlayfs
Would that be better?

[lordratner]

Apparently not. According to docker: "fuse-overlayfs is preferred only for running Rootless Docker on a host that does not provide support for rootless overlay2. On Ubuntu and Debian 10, the fuse-overlayfs driver does not need to be used, and overlay2 works even in rootless mode. Refer to the rootless mode documentation for details."

And apparently there is some issue with the fuse-overlayfs causing entire PVE nodes to crash. And because it's a userspace driver, it can slow down dramatically in a virtual environment.

Maybe the zfs storage driver is the right answer? But I have no idea how it would be implemented.

[tteck]

I'll take a look... no promises

https://docs.docker.com/storage/storagedriver/zfs-driver/

[ThellraAK]

@tteck I think this is related as I'm also using zfs

It is 'using' 1.3GB but is actually ~14GB on disk. I don't think it's just weird reporting either, it's reporting that much disk used in proxmox as well.

df -h
Filesystem Size Used Avail Use% Mounted on
proxzfspool/subvol-6001-disk-0 40G 16G 25G 39% /

docker system prune -a
...
Total reclaimed space: 1.344GB

/bookstack# df -h
Filesystem Size Used Avail Use% Mounted on
proxzfspool/subvol-6001-disk-0 40G 1.4G 39G 4% /

EDIT: Switching it over to a 'directory' storage backing on ext4 made the disk usage behave as expected.

[TheRealGramdalf]

Just a quick update @tteck, this should now be solved as of proxmox 8.1. overlay2 is now supported with ZFS as the backing filesystem (as of OpenZFS 2.2.0)

One thing I noticed was that there seems to be a check (in install.func iirc?) for whether the backing storage driver is ZFS - but it only applies to the regular docker LXC, not the docker-alpine LXC. In either case, it can now be dropped for proxmox versions 8.1+.

It would probably also be a good idea to implement a check for whether the ZFS pool has the feature enabled - I can provide more info if you need a direction to look in

[tteck]

I vote to drop fuse-overlayfs all together.

[tteck]

Dropped fuse-overlayfs as an option for a Docker storage driver.