[Question] the file storage wal

[milksteven]

The AutoMQ support File storage, primarily NFS protocol-based file storage services, is also widely used in big data and other business scenarios, such as AWS EFS and AWS FSx series. There are some fundamental problems with NFS which makes it hard to build a reliable data system on top of it. It's mostly due to it being networked, i o syscalls failed. File coherency is also a big problem. You might persist a file but it's not immediately readable, or you can list a file and you cannot read it. Why can AutoMQ implement NFS-based WAL on FSxN? I have doubts about this.

[daniel-y]

Honestly, I’m not entirely sure I fully understand your question. Could you provide more details? I’ll attempt to answer based on my assumptions:

1. Our NFS WAL implementation uses Direct IO, with synchronous IO Write and Mount options, eliminating any buffered cache at any level.
2. Managed NFS servers like FSxN or EFS provide strong consistency guarantees. For example: https://docs.aws.amazon.com/efs/latest/ug/features.html#consistency
3. We use NFS solely for WAL, leveraging some stateful capabilities of NFS 4.1 to ensure the correctness of our implementation.

Additionally, NFS WAL is not a feature in the current open-source repository. Feel free to contact us for further discussion: https://www.automq.com/contact

[daniel-y]

Let's close this issue now, let me know if you want to discuss this topic deeper.