Merge "Document the new auto re-export and fsid mode features" into main

Author: benjamin-maynard

deployment/README.md

@@ -211,12 +211,40 @@ These mount options are for the proxy to the source server.
 | DISABLED_NFS_VERSIONS | The versions of NFS that should be disabled in `nfs-kernel-server`. Explicitly disabling unwanted NFS versions prevents clients from accidentally auto-negotiating an undesired NFS version. Specify multiple versions to disable with a comma separated list. Acceptable values are `3`, `4`, `4.0`, `4.1`, `4.2`. NFS Version 2 is always diabled. | False    | `4.0,4.1,4.2` |
 | NUM_NFS_THREADS       | The number of NFS Threads to use for KNFSD.                                                                                                                                                                                                                                                                                                          | False    | `512`         |
 
+**NOTE:** When using NFS v4, it is recommended that you use NFS v4.1 or greater. NFS v4.1 has many improvements to fix limitations of the NFS v4.0 protocol.
+
 ### Export Options
 
-| Variable       | Description                                                                       | Required | Default |
-| -------------- | --------------------------------------------------------------------------------- | -------- | ------- |
-| NOHIDE         | When `true`, adds the `nohide` option to all the exports.                         | False    | `true`  |
-| EXPORT_OPTIONS | Any custom NFS exports options. These options will be applied to all NFS exports. | False    | `""`    |
+| Variable       | Description                                                                                                                                | Required | Default |
+| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | -------- | ------- |
+| NOHIDE         | When `true`, adds the `nohide` option to all the exports. Overridden by AUTO_REEXPORT                                                      | False    | `true`  |
+| AUTO_REEXPORT  | When `true` enables the `crossmnt` option on all exports and automatically re-exports any nested mounts that were not explicitly exported. | False    | `false` |
+| EXPORT_OPTIONS | Any custom NFS exports options. These options will be applied to all NFS exports.                                                          | False    | `""`    |
+
+Use of `AUTO_REEXPORT` requires that `FSID_MODE` is `local` or `external`. `external` is recommended. See [Auto Re-export](./auto-re-export.md) for more detail.
+
+### FSID database options
+
+| Variable             | Description | Required | Default |
+| FSID_MODE            | How to assign FSIDs (File System Identifiers) to each export. The options are `static`, `local`, or `external`. | False | `"external"` |
+| FSID_DATABASE_DEPLOY | Set to `false` to prevent automatically creating a Cloud SQL instance when `FSID_MODE` is set to `external`.    | False | `true`       |
+| FSID_DATABASE_CONFIG | Allows Overriding the default FSID database configuration when `FSID_MODE` is set to `external`.                | False | `""`         |
+
+The recommended `FSID_MODE` is to always use `external`. For more details on `FSID_MODE` and `FSID_DATABASE_CONFIG` see [Filesystem Identifiers](./fsids.md).
+
+Older knfsd versions that do not support `FSID_MODE` always used `static` to assign FSIDs. If you're upgrading from an earlier knfsd version it's advised you switch to `external` when possible, though you can use `static` to keep the existing behaviour.
+
+The `FSID_MODE` option supports:
+
+* `static` - Each export is explicitly allocated an incrementing FSID number on start-up. This requires all the exports to be known at start-up and is not compatible with `AUTO_REEXPORT=true`. `static` is only recommended if using an explicit `EXPORT_MAP`.
+
+* `local` - Each export is automatically allocated an FSID number by the `mountd` using using the standard NFS `fsidd` service. This uses a local sqlite database to store FSID mappings. This is not recommended for production and should only be used for single instance proxy clusters.
+
+  If multiple proxy instances in a cluster allocate a different FSID to the same export then I/O errors or data corruption may occur if a client changes instance.
+
+  Also, even with a single proxy instance, if the instance is rebooted
+
+* `external` - Each export is automatically allocated an FSID number by the `mountd` using using the `knfsd-fsidd` service. This uses a Cloud SQL Postgres instance to store the FSID mappings. This ensures that all the instances in cluster allocate the same FSID to each export.
 
 ### Autoscaling Configuration
 

deployment/auto-re-export.md

@@ -0,0 +1,82 @@
+# Auto Re-export
+
+With older kernel versions all re-exported filesystems had to be explicitly re-exported. This limitation existed because each re-export required an explicit FSID.
+
+As of kernel 6.1 a new auto re-export feature was added; made possible by the addition of the new `fsidd` service that can automatically assign an FSID to an export when required.
+
+Auto re-export is useful in the following conditions:
+
+* The source server has nested mounts on separate filesystems (FSIDs).
+
+  This will depend upon the implementation of the source NFS server. For example, NetApp Junctions are presented as separate filesystems and operate similar to Linux NFS' `nohide`/`crossmnt` option. Other NFS servers such as Isilon OneFS present nested volumes as if they were a single filesystem.
+
+* The source server uses `crossmnt`/`nohide` (or similar) and you already use it with the clients. For example, when using NetApp Junctions.
+
+* The source server has a large number of nested mounts (over 1,000) making it impractical or impossible to explicitly re-export every mount.
+
+## Example
+
+Assume the source server is exporting the following nested mounts:
+
+```text
+/dev/sda1 on /files
+/dev/sda2 on /files/archive
+```
+
+With auto re-export enabled the proxy server only needs to explicitly export `/files`. When a client tries to access `/files/archive` the proxy will detect the change in file system, automatically re-export `/files/archive` using the FSID service to allocate a unique FSID to the export.
+
+In NFS v4, this style of nested mount is directly supported by the NFS protocol. In NFS v3 this style of re-export relies on the `crossmnt` or `nohide` flags. There are some caveats that come with `crossmnt` or `nohide`, see `man exports.5` for more details.
+
+## NFS protocol versions
+
+### NFS version 3
+
+NFS version 3 does not directly support nested volumes. Officially clients have to use the mount protocol to explicitly mount nested volumes.
+
+Some NFS servers, including the Linux Kernel NFS server (knfsd), support a non-standard option that allows clients to see the nested volume without first mounting the volume. The Linux NFS client then detects the change in FSID and automatically mounts the nested volume.
+
+On Linux this option is `nohide`/`crossmnt`. This can cause issues with NFS clients that do not support this, see `man exports.5` for details.
+
+If the source server does not support an equivalent of `nohide` then auto re-export might not work. In this case the knfsd proxy will only see an empty directory instead of the nested volume.
+
+For NFS v3 all root volumes still need to be explicitly re-exported. If you have a large number of root volumes consider mounting using NFS v4 instead.
+
+### NFS version 4
+
+The NFS v4 protocol was updated to directly support nested volumes. Under the NFS v4 protocol all exported volumes are considered to be nested volumes of a pseudo-root volume.
+
+Because NFS v4 directly supports nested volumes the `nohide`/`crossmnt` issues of NFS v3 do not apply. However, the knfsd proxy still has to assign an FSID for each nested volume; either explicitly or by using auto re-export.
+
+If you connect to the source server using NFS v4 you can re-export the pseudo-root path `/` with auto re-export enabled to automatically re-export every export.
+
+**NOTE:** When using NFS v4, it is recommended that you use NFS v4.1 or greater. NFS v4.1 has many improvements to fix limitations of the NFS v4.0 protocol.
+
+## Configuration
+
+Use of an FSID service to automatically allocate FSIDs for exports is required when using auto re-export. An `FSID_MODE="external"` is recommended so that all the knfsd proxy instances in the cluster use the same FSID number for each export.
+
+```terraform
+module "nfs_proxy" {
+
+    source = "github.com/GoogleCloudPlatform/knfsd-cache-utils//deployment/terraform-module-knfsd?ref=v1.0.0-beta4"
+
+    # Include your standard KNFSD configuration, this example only shows the
+    # configuration values specific to the auto re-export feature.
+
+    # Explicitly re-export only the root volumes from the source server.
+    # Not using auto-discovery such as EXPORT_HOST_AUTO_DETECT to avoid
+    # explicitly re-exporting nested volumes.
+    EXPORT_MAP                    = "10.0.5.5;/remoteexport;/remoteexport"
+
+    # Enable the auto re-export feature
+    AUTO_REEXPORT                 = true
+
+    # Store FSID mappings in an external database so that all knfsd proxy
+    # instances in the cluster allocate the same FSID to each export.
+    FSID_MODE                     = "external"
+
+    # If possible, use an NFS v4 protocol version for the proxy clients as
+    # the NFS v4 protocol has direct support nested volumes.
+    DISABLED_NFS_VERSIONS         = "3,4.0,4.2"
+}
+```