ZFS running on S3 object storage via ZeroFS

[Barre]

Hi Everyone,

I wanted to share something unexpected that came out of a filesystem project I've been working on.

I built ZeroFS, an NBD + NFS server that makes S3 storage behave like a real filesystem using an LSM-tree backend. While testing it, I got curious and tried creating a ZFS pool on top of it... and it actually worked!

So now we have ZFS running on S3 object storage, complete with snapshots, compression, and all the ZFS features we know and love. The demo is here: https://asciinema.org/a/kiI01buq9wA2HbUKW8klqYTVs

ZeroFS handles the heavy lifting of making S3 look like block storage to ZFS (through NBD), with caching and batching to deal with S3's latency.

This enables pretty fun use-cases such as Geo-Distributed ZFS :)

https://github.com/Barre/zerofs?tab=readme-ov-file#geo-distributed-storage-with-zfs

The ZeroFS project is at https://github.com/Barre/zerofs if anyone's curious about the underlying implementation.

Bonus: ZFS ends up being a pretty compelling end-to-end test in the CI! https://github.com/Barre/ZeroFS/actions/runs/16341082754/job/46163622940#step:12:49

[Barre]

Here is Postgres benchmarks I just run of Postgres on top of ZFS, on top of S3 through ZeroFS

pgbench (16.9 (Ubuntu 16.9-0ubuntu0.24.04.1))
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 50
number of threads: 15
maximum number of tries: 1
number of transactions per client: 100000
number of transactions actually processed: 5000000/5000000
number of failed transactions: 0 (0.000%)
latency average = 0.943 ms
initial connection time = 48.043 ms
tps = 53041.006947 (without initial connection time)

Read only

postgres@ubuntu-16gb-fsn1-1:/root$ pgbench -c 50 -j 15 -t 100000 -S example
pgbench (16.9 (Ubuntu 16.9-0ubuntu0.24.04.1))
starting vacuum...end.
transaction type: <builtin: select only>
scaling factor: 50
query mode: simple
number of clients: 50
number of threads: 15
maximum number of tries: 1
number of transactions per client: 100000
number of transactions actually processed: 5000000/5000000
number of failed transactions: 0 (0.000%)
latency average = 0.121 ms
initial connection time = 53.358 ms
tps = 413436.248089 (without initial connection time)