Support for tmpdir feature?

[IvanVolosyuk]

I read about planned support for tmpdir in bcachefs and it sound like a very cool feature. Would be nice to have something like this in ZFS. Basically make certain directories behave as temporary files storage - no ZIL, no metadata stored, spill data to disk on memory pressure, similar to tmpfs. Right know I use tmpfs for /var/tmp in gentoo for all compilations which competes for memory pages with ZFS ARC. Having this feature will make it very attractive to just use 'normal' ZFS directory for this with gain ARC efficiency.

https://bcachefs.org/Roadmap/#tmpdir_support

The downside I can see is additional complexity, ZFS already have a lot of code path for various special case operations. I very interested in other people and devs opinion about this.

[snajpa]

[IvanVolosyuk]

You missed encryption support which seem to finally stabilized now, but I see you point. Stability is first priority, but eventually? I guess you can link here open issues / PRs for tmpfile which can be considered blockers.

[snajpa]

I just hope the design will properly account for multitenant setups... rest of my comments is not really relevant anyway

[robn]

It's not really clear what's planned from that short description, but I can't really think of anything significant OpenZFS can do better in this area.

Linux tmpfs will already grow and shrink according to its contents, so it's only going to compete with the ARC in the sense that it is holding things in memory that other things can't use, but that would be the same for an ARC-backed tmpfs - it wouldn't be able to evict those buffers.

Its also likely to be less efficient, because all of ZFS' data structures are designed assuming they need to be read from and written to disk blocks. If you're storing things only in memory, there's much better data structures available.

And without any storage behind them, there's really not much of the OpenZFS "specialities" that make any sense. Scrubbing and error correction don't seem to be useful. Snapshots and rollbacks could work but it still all needs to be held in memory. Maybe receiving a snapshot in a network boot kind of situation? I dunno, a squashfs or even a cpio stream is gonna be smaller and easier, I think?

And if you really want it, you can build a pool out of tmpfs-backed files or block devices right now.

I'd genuinely like to hear real use cases; I couldn't (quickly) find anything about what prompted the bcachefs team to put this on the roadmap. Any pointers?

[robn]

Thinking on it more on my walk (glorious late winter day here in Melbourne), I realised I missed a subtlety.

I'd got stuck on "in memory" as "in physical memory", but of course tmpfs is backed by virtual memory, that is, parts of it can be evicted to a swap device. So it can grow larger than physical memory, which makes it more interesting wrt to OpenZFS.

So it's worth noting that ARC buffer memory currently requires physical memory (actually "slab" memory) - it can't currently be swapped out. That would be a huge undertaking (see #10255) but might not be necessary for such a feature anyway.

So I think we have many of the tools already to create a kind of "volatile" dataset that is zeroed on mount.

Obviously, we can no-op fsync() and friends with sync=disabled. So that's nice and easy right now.

Zero-on-mount can be mimicked right now by creating a "zero" snapshot at dataset creation, then doing a rollback before mount. There's other ways too, like cloning an empty snapshot into place before mount. There's been talk of having a magical "zero state" implicit snapshot for all datasets in other contexts, and I think it would be generally useful anyway for very little cost, but regardless, it can be scripted easily right now.

Thay would get you something that looks like a tmpdir, but it still actually storing to the main pool, so it does compete for IO and storage space, but more importantly, is on the wrong schedule: we want it written out at cache evict, not at cache entry.

We do have a "spill cache to disk" mechanism though: L2ARC. So that might mean we have most of the mechanism needed for something like this, we just have to wire it up differently. I don't know the real complexity of that though, I'm just thinking concept at this stage. Probably it impacts management of the ARC lists; maybe we should be less eager to evict a "tmpdir" buf if it isn't backed by main pool storage.

Anyway, I'll keep pondering. I'm still interested in specific use cases. For now though, back to my lovely sunny day!

[IvanVolosyuk]

Yes, this is pretty much what I thought: tmpfs + swap replacement.

One specific use case I have in mind is compiling Gentoo world. I can use zfs + tmpfs + swap for that. There is a larger amount of space allocated in tmpfs and parallel compilations can use large amount of ram as well. It makes ARC, tmpfs and gcc compete for memory, which works as intended. On memory pressure: compilation programs and tmpfs are evicted into swap, while ARC is just shrinking (if there is no L2ARC).

The tmpfs is actually discouraged for Gentoo compilation and it is hard to measure the required size in advance:
https://wiki.gentoo.org/wiki/Portage_TMPDIR_on_tmpfs

The doc rightfully says that modern filesystems should perform reasonably well, but in reality there is no hint to the filesystem to mark the writes unimportant and the data volatile. There is an OpenZFS per dataset property which can be useful here: sync=disabled, but it is still required from OpenZFS to include all the data into TXGs and save corresponding metadata. What if we can delay this and don't included temporary data into TXGs unless we experience memory pressure?