Merge pull request ceph#50212 from rishabh-d-dave/fs-swap-subcmd

cephfs: add command "ceph fs swap"

Reviewed-by: Venky Shankar <[email protected]>
Reviewed-by: Patrick Donnelly <[email protected]>

Author: rishabh-d-dave

PendingReleaseNotes

@@ -50,6 +50,13 @@
   recommend that users with versioned buckets, especially those that existed
   on prior releases, use these new tools to check whether their buckets are
   affected and to clean them up accordingly.
+* CephFS: Two FS names can now be swapped, optionally along with their IDs,
+  using "ceph fs swap" command. The function of this API is to facilitate
+  file system swaps for disaster recovery. In particular, it avoids situations
+  where a named file system is temporarily missing which would prompt a higher
+  level storage operator (like Rook) to recreate the missing file system.
+  See https://docs.ceph.com/en/latest/cephfs/administration/#file-systems
+  docs for more information.
 
 >=18.0.0
 

doc/cephfs/administration.rst

@@ -92,6 +92,46 @@ The CephX IDs authorized to the old file system name need to be reauthorized
 to the new name. Any on-going operations of the clients using these IDs may be
 disrupted. Mirroring is expected to be disabled on the file system.
 
+::
+
+    fs swap <fs1-name> <fs1_id> <fs2-name> <fs2_id> [--swap-fscids=yes|no] [--yes-i-really-mean-it]
+
+Swaps names of two Ceph file sytems and updates the application tags on all
+pools of both FSs accordingly. Certain tools that track FSCIDs of the file
+systems, besides the FS names, might get confused due to this operation. For
+this reason, mandatory option ``--swap-fscids`` has been provided that must be
+used to indicate whether or not FSCIDs must be swapped.
+
+.. note:: FSCID stands for "File System Cluster ID".
+
+Before the swap, mirroring should be disabled on both the CephFSs
+(because the cephfs-mirror daemon uses the fscid internally and changing it
+while the daemon is running could result in undefined behaviour), both the
+CephFSs should be offline and the file system flag ``refuse_client_sessions``
+must be set for both the CephFS.
+
+The function of this API is to facilitate disaster recovery where a new file
+system reconstructed from the previous one is ready to take over for the
+possibly damaged file system. Instead of two ``fs rename`` operations, the
+operator can use a swap so there is no FSMap epoch where the primary (or
+production) named file system does not exist. This is important when Ceph is
+monitored by automatic storage operators like (Rook) which try to reconcile
+the storage system continuously. That operator may attempt to recreate the
+file system as soon as it is seen to not exist.
+
+After the swap, CephX credentials may need to be reauthorized if the existing
+mounts should "follow" the old file system to its new name. Generally, for
+disaster recovery, its desirable for the existing mounts to continue using
+the same file system name. Any active file system mounts for either CephFSs
+must remount. Existing unflushed operations will be lost. When it is judged
+that one of the swapped file systems is ready for clients, run::
+
+    ceph fs set <fs> joinable true
+    ceph fs set <fs> refuse_client_sessions false
+
+Keep in mind that one of the swapped file systems may be left offline for
+future analysis if doing a disaster recovery swap.
+
 
 Settings
 --------

doc/man/8/ceph.rst

@@ -23,7 +23,7 @@ Synopsis
 
 | **ceph** **df** *{detail}*
 
-| **ceph** **fs** [ *add_data_pool* \| *authorize* \| *dump* \| *feature ls* \| *flag set* \| *get* \| *ls* \| *lsflags* \| *new* \| *rename* \| *reset* \| *required_client_features add* \| *required_client_features rm* \| *rm* \| *rm_data_pool* \| *set*] ...
+| **ceph** **fs** [ *add_data_pool* \| *authorize* \| *dump* \| *feature ls* \| *flag set* \| *get* \| *ls* \| *lsflags* \| *new* \| *rename* \| *reset* \| *required_client_features add* \| *required_client_features rm* \| *rm* \| *rm_data_pool* \| *set* \| *swap* ] ...
 
 | **ceph** **fsid**
 
@@ -474,6 +474,15 @@ Usage::
 
     ceph fs set <fs-name> <fs-setting> <value>
 
+Subcommand ``swap`` swaps the names of two Ceph file system and updates
+application tags on the pool of the file systems accordingly. Optionally,
+FSIDs of the filesystems can also be swapped along with names by passing
+``--swap-fscids``.
+
+Usage::
+
+    ceph fs swap <fs1-name> <fs1-id> <fs2-name> <fs2-id> [--swap-fscids] {--yes-i-really-meant-it}
+
 fsid
 ----
 

qa/suites/fs/functional/tasks/admin.yaml

@@ -10,3 +10,4 @@ tasks:
       fail_on_skip: false
       modules:
         - tasks.cephfs.test_admin
+        - tasks.cephfs.admin.test_fs_swap

qa/tasks/ceph_test_case.py

@@ -76,8 +76,8 @@ def _verify(self, proc, exp_retval=None, exp_errmsgs=None):
 
         proc_stderr = proc.stderr.getvalue().lower()
         msg = ('didn\'t find any of the expected string in stderr.\n'
-               f'expected string: {exp_errmsgs}\n'
-               f'received error message: {proc_stderr}\n'
+               f'expected string -\n{exp_errmsgs}\n'
+               f'received error message -\n{proc_stderr}\n'
                'note: received error message is converted to lowercase')
         for e in exp_errmsgs:
             if e in proc_stderr:
@@ -105,6 +105,8 @@ def negtest_ceph_cmd(self, args, retval=None, errmsgs=None, **kwargs):
         # execution is needed to not halt on command failure because we are
         # conducting negative testing
         kwargs['check_status'] = False
+        # log stdout since it may contain something useful when command fails
+        kwargs['stdout'] = StringIO()
         # stderr is needed to check for expected error messages.
         kwargs['stderr'] = StringIO()