Using SaunaFS for VM storage (RAW images vs secondary storage)

[yaroslav-gwit]

Hi everyone 👋

I’ve recently come across SaunaFS and I’m really impressed by the direction this project is taking! I’m currently evaluating it as part of a virtualization stack, and wanted to ask a few questions before diving deeper.

My main goal is to determine whether SaunaFS could serve as:

• Primary VM storage - storing .raw disk images directly on the filesystem.
• Secondary/addon VM storage - hosting shared datasets like documents, videos, or large blob files accessed by multiple VMs or containers.

I’d love to hear your thoughts on whether either approach is viable and what potential performance or reliability trade-offs I should be aware of.

A few more specific questions:

HA and failover:

• I’m planning to implement my own failover/orchestration daemon based on Etcd to manage Master–Shadow transitions.
• Could you outline the manual step-by-step process for promoting a Shadow to Master safely?
• What should I watch out for during the transition (e.g., version mismatches, incomplete metadata replication, etc.)?
• After a full cluster reboot, is there a way to check or compare the metadata version between Masters and Shadows, so that the correct node becomes active automatically?

---

Documentation source:

As far as I understand, SaunaFS is a fork of LizardFS.
→ Is it safe to assume that when documentation is missing, I can refer to the LizardFS docs for general guidance (at least until SaunaFS diverges more significantly)?

---

Replication vs Erasure Coding:

• I’m trying to decide which storage mode would be better suited for RAW VM images.
• EC writes typically require more computation, but I’ve seen in your docs that EC can actually outperform replication for regular filesets.
• Does the same hold true for large sequential I/O workloads like VM disk images?

---

FreeBSD compatibility:

• Do you currently test or support SaunaFS on FreeBSD?
• I run a few clusters based on FreeBSD (mostly for virtualization and ZFS), so I’m curious if the existing codebase can be compiled or adapted without major issues.

---

Prometheus metrics:
Are there any plans (or existing implementations) for Prometheus-compatible metrics in SaunaFS?
It would be great to expose basic health and performance stats - like chunkserver space usage, replication status, and master/shadow state - for integration into Grafana dashboards.

Thank you!

Thanks a lot for your time and for the work you’re putting into SaunaFS 🙌
I’m really excited to test it out and will happily share some benchmarks once I have a few test nodes running.

[uristdwarf]

Hi @yaroslav-gwit!

Prepare for a lot of text:

I’m planning to implement my own failover/orchestration daemon based on Etcd to manage Master–Shadow transitions.
Could you outline the manual step-by-step process for promoting a Shadow to Master safely?

Interesting. I'm not sure how you would use etcd or how it would differ from uraft, but I would be very interested in the results.

The manual way is:

1. Shut down the current master (if it isn't already)
2. Edit the configuration file on shadow and change the personality to master
3. Reload the configuration.

There may be other things, like saunafs-admin promote-shadow, but that requires the ha-cluster-managed option.

This doesn't include things like IP/host re-assignment etc. so clients need to be either restarted and pointed to the new host, or the IP/host needs to be changed so it points to the new master. You will have to figure that out on your own (The uraft source code has some of this stuff if you are interested).

What should I watch out for during the transition (e.g., version mismatches, incomplete metadata replication, etc.)?

I don't know if you are talking about software versions, or metadata versions, so I'll assume both for clarity:

Regarding software version mismatches: We try to avoid any major compatibility issues in the minor version (the y in vx.y.z), so versions should work with each other in the major version. The only exception currently is that a older shadow cannot connect to a newer master.

But generally the tested upgrade procedure is 0) metaloggers 1) shadows, 2) master, 3) chunkservers and 4) clients

Regarding the metadata version, this is indeed the most important value you need to check. You should promote the metadata server with the latest version.

Again, you may want to consult the uraft source code for the LizardFS attempt at this.

incomplete metadata replication

This is potentially the biggest pain right now for us: Currently, the master can respond to the client with success before the metadata is replicated to other shadows/metaloggers. The time window size where this happens can vary, but it can happen, for example, that a file is created, the master crashes, a shadow is promoted, and you can't write to the file because it doesn't exist. It's a potential race condition.

We are thinking about perhaps adding a option to master to make sure it synchronizes with N amount of shadows or all of them before responding to client. But that isn't currently implemented and may impact performance.

After a full cluster reboot, is there a way to check or compare the metadata version between Masters and Shadows, so that the correct node becomes active automatically?

You can use saunafs-admin metadataserver-status <master host> <master client port (9421)> to get the metadata version from running shadows/master. Alternatively, you can use sfsmetarestore -g or read the changelog file for the latest metadata version.

Is it safe to assume that when documentation is missing, I can refer to the LizardFS docs for general guidance (at least until SaunaFS diverges more significantly)?

Mostly yes (commands have different names and there are a few new options added, but we've left untouched most other things), but please mention it in our Slack or issues if you find anything missing.

I’m trying to decide which storage mode would be better suited for RAW VM images.

Depends. Do you have multiple VM's reading the same image in different locations (different buildings, cities etc.)? Then I'd consider replication. Otherwise, then EC is the best.

EC writes typically require more computation, but I’ve seen in your docs that EC can actually outperform replication for regular filesets.

EC is better for parallelized IO to multiple disks/servers. This is mostly because the clients are doing the computation, but the Intel ISAL library also heavily helps in this regards. Even on a normal consumer laptop, it's usually very fast. So it should not be a issue at all with computation when writing. Reading data with missing chunks might be slower though.

Of course, this also depends heavily on your infrastructure and your file size. Fewer drives/chunkservers also means it's less effective.

Does the same hold true for large sequential I/O workloads like VM disk images?

On large files? Yes.

FreeBSD compatibility:

Not officially supported. But I'm not against fixing things on it, especially if someone else does it :^).

In all seriousness, we don't compile and test for it, but we may address issues related to it, and also accept PR's to fix the aforementioned issues. So if there is interest from the community to support it, we will accept and try to help in running it on BSD.

Regarding current compatibility: I don't know. All I know is that on LizardFS it probably was working, and maybe it will still work on SaunaFS. You'll need to figure out the dependencies/libraries etc. all by yourself. Also, maybe you need to edit the code with #ifdefs. You can message on Slack if you need help with that.

Prometheus metrics

There is already an experimental option on master to enable Prometheus metrics (for master), but it's experimental and for good reason: There might be UB with the forking in the background.

I've been working on it for the past year, but that has been a major blocker and we are trying to get rid of the forking somehow.

But I've also been thinking working around that problem. Currently we are trying use a C++ prometheus library, which embeds a small web server into master. But I think it may be easier and more performant to work with the current network protocol interface and create a prometheus exporter on top of it. We recently introduced a API to access our monitoring (see here), and I'm thinking about extending that with a prometheus exporter as well, using the same techniques.

I don't know when I will be working on it, but hopefully very soon.

[yaroslav-gwit]

Hi!

First of all, thank you so much for taking the time to write such a detailed reply - I really appreciate it ❤️
It definitely wasn’t a small question, so a long and thoughtful answer like this is exactly what I was hoping for. You’ve cleared up a lot of things for me, especially around the upgrade order, failover flow, and metadata version handling - that helps a ton.

A few follow-ups and thoughts on my side:

Etcd & failover orchestration:
In our setup, the cluster nodes are already registered and managed through Etcd, so the idea is to extend that with a few extra key values and a small daemon that monitors changes and automatically performs the failover sequence.
We’ll be using a floating IP approach to make the transition transparent for clients.
Easier said than done, of course 😅 - but that’s the general outline.

FreeBSD support:
That’s great to hear! We currently use KVM/QEMU under Linux, but we’re planning to introduce FreeBSD’s bhyve support later down the line. Knowing that SaunaFS could potentially be made to work there is encouraging - if I run into build issues, I'll try to document them and share fixes back upstream.

Prometheus metrics:
Your explanation about the Prometheus work made perfect sense - and I can see why the forking issue is a blocker.
In my experience, when kernel-level or in-process exporters become tricky, a good fallback is to expose a lightweight Unix socket or HTTP endpoint at the "core" layer of the software, and then develop a separate Prometheus exporter binary around it.

If your new API is reasonably stable, it should be pretty straightforward to build such an external exporter as a first step - and that would also keep it decoupled from the master process until the internal metrics endpoint is mature.

Ubuntu repositories:
I also noticed that the documentation mentions the Ubuntu repositories aren’t publicly available and can be accessed on request. Could you clarify what the current process looks like for that? Are they temporarily private while being stabilized, or is access limited for another reason?

Thanks again for the thoughtful response - and for being so open to community feedback and contributions.

SaunaFS is shaping up to be a really promising project, and I’m excited to test it in a few cluster scenarios soon. I’ll happily share back some results once I have early benchmarks or FreeBSD build notes.

[uristdwarf]

Ubuntu repositories:
I also noticed that the documentation mentions the Ubuntu repositories aren’t publicly available and can be accessed on request. Could you clarify what the current process looks like for that? Are they temporarily private while being stabilized, or is access limited for another reason?

We consider the binary packages and repositories to be part of the paid support we provide. We actually want to encourage people to create their own packages (for example, NixOS users have made a package, and there's a package pending in Debian FTP by yours truly).

However, we currently provide packages free of charge as long as you are willing to fill out a form. It should be linked on the docs.saunafs.com, but apparently it isn't currently. I'll see that it's fixed soon.

[uristdwarf]

Here's the link to the form: https://saunafs.com/repository-access

[uristdwarf]

P.S: The scripts we use for building the packages for Ubuntu 22.04 and 24.04 are also available on https://github.com/leil-io/debian-package

[yaroslav-gwit]

This is really good, thank you. I'll have a look at the DEB build scripts, as it would be nice to confirm if any given release "works" well on our own VM cluster, before rolling it out to all users at once (we've got the auto-patching turned on) 😸