Merge pull request ceph#54620 from rishabh-d-dave/mgr-vol-clone-stats

mgr/vol: show progress and stats for the subvolume snapshot clones

Reviewed-by: Venky Shankar <[email protected]>

Author: vshankar

PendingReleaseNotes

@@ -265,6 +265,15 @@ CephFS: Disallow delegating preallocated inode ranges to clients. Config
   exposed via the new `--snap-id` option for `rbd clone` command.
 * RBD: The output of `rbd snap ls --all` command now includes the original
   type for trashed snapshots.
+* CephFS: "ceph fs clone status" command will now print statistics about clone
+  progress in terms of how much data has been cloned (in both percentage as
+  well as bytes) and how many files have been cloned.
+* CephFS: "ceph status" command will now print a progress bar when cloning is
+  ongoing. If clone jobs are more than the cloner threads, it will print one
+  more progress bar that shows total amount of progress made by both ongoing
+  as well as pending clones. Both progress are accompanied by messages that
+  show number of clone jobs in the respective categories and the amount of
+  progress made by each of them.
 
 >=18.0.0
 

doc/cephfs/fs-volumes.rst

@@ -758,16 +758,40 @@ Here is an example of an ``in-progress`` clone:
 ::
 
     {
-        "status": {
-            "state": "in-progress",
-            "source": {
-                "volume": "cephfs",
-                "subvolume": "subvol1",
-                "snapshot": "snap1"
-            }
+      "status": {
+        "state": "in-progress",
+        "source": {
+          "volume": "cephfs",
+          "subvolume": "subvol1",
+          "snapshot": "snap1"
+        },
+        "progress_report": {
+          "percentage cloned": "12.24%",
+          "amount cloned": "376M/3.0G",
+          "files cloned": "4/6"
         }
+      }
     }
 
+A progress report is also printed in the output when clone is ``in-progress``.
+Here the progress is reported only for the specific clone. For collective
+progress made by all ongoing clones, a progress bar is printed at the bottom
+in ouput of ``ceph status`` command::
+
+  progress:
+    3 ongoing clones - average progress is 47.569% (10s)
+      [=============...............] (remaining: 11s)
+
+If the number of clone jobs are more than cloner threads, two progress bars
+are printed, one for ongoing clones (same as above) and other for all
+(ongoing+pending) clones::
+
+  progress:
+    4 ongoing clones - average progress is 27.669% (15s)
+      [=======.....................] (remaining: 41s)
+    Total 5 clones - average progress is 41.667% (3s)
+      [===========.................] (remaining: 4s)
+
 .. note:: The ``failure`` section will be shown only if the clone's state is ``failed`` or ``cancelled``
 
 Here is an example of a ``failed`` clone:

qa/suites/fs/volumes/tasks/volumes/test/clone-progress.yaml

@@ -0,0 +1,5 @@
+tasks:
+  - cephfs_test_runner:
+      fail_on_skip: false
+      modules:
+        - tasks.cephfs.test_volumes.TestCloneProgressReporter

qa/tasks/cephfs/mount.py

@@ -775,6 +775,10 @@ def run_shell(self, args, **kwargs):
 
         return self.client_remote.run(args=args, **kwargs)
 
+    def get_shell_stdout(self, args, timeout=300, **kwargs):
+        return self.run_shell(args=args, timeout=timeout, **kwargs).stdout.\
+            getvalue().strip()
+
     def run_shell_payload(self, payload, wait=True, timeout=900, **kwargs):
         kwargs.setdefault('cwd', self.mountpoint)
         kwargs.setdefault('omit_sudo', False)