Merge PR ceph#54396 into main

* refs/pull/54396/head:
	PendingReleaseNote: add note for pause_purging and pause_cloninig
	doc/cephfs: add note for config option pause_purging and pause_cloning
	qa/cephfs: add tests for mgr/vol config pause_cloning
	qa/cephfs: extend wait for trash empty
	qa/cephfs: add tests for config option pause_purging
	qa/cephfs: don't strip any whitespace for get_shell_stdout
	mgr/vol: add pause/resume mechanism for async jobs

Reviewed-by: Venky Shankar <[email protected]>

Author: vshankar

PendingReleaseNotes

@@ -48,6 +48,13 @@
   This can be necessary in cases where we wish to prevent the manager from being
   flooded by module commands when Ceph services are down or degraded.
 
+* CephFS: It is now possible to pause the threads that asynchronously purge
+  the deleted subvolumes by using the config option
+  "mgr/volumes/pause_purging".
+* CephFS: It is now possible to pause the threads that asynchronously clone
+  the subvolume snapshots by using the config option
+  "mgr/volumes/pause_cloning".
+
 * CephFS: Modifying the setting "max_mds" when a cluster is
   unhealthy now requires users to pass the confirmation flag
   (--yes-i-really-mean-it). This has been added as a precaution to tell the

doc/cephfs/fs-volumes.rst

@@ -959,6 +959,32 @@ Configure the maximum number of concurrent clone operations. The default is 4:
 
    ceph config set mgr mgr/volumes/max_concurrent_clones <value>
 
+Pause the threads that asynchronously purge trashed subvolumes. This option is
+useful during cluster recovery scenarios:
+
+.. prompt:: bash #
+
+    ceph config set mgr/volumes/pause_purging true
+
+To resume purging threads:
+
+.. prompt:: bash #
+
+    ceph config set mgr/volumes/pause_purging false
+
+Pause the threads that asynchronously clone subvolume snapshots. This option is
+useful during cluster recovery scenarios:
+
+.. prompt:: bash #
+
+    ceph config set mgr/volumes/pause_cloning true
+
+To resume cloning threads:
+
+.. prompt:: bash #
+
+    ceph config set mgr/volumes/pause_cloning false
+
 Configure the ``snapshot_clone_no_wait`` option:
 
 The ``snapshot_clone_no_wait`` config option is used to reject clone-creation

doc/cephfs/troubleshooting.rst

@@ -134,6 +134,31 @@ things to do:
   be changed in such circumstances, run the command to change ``max_mds`` with
   the confirmation flag (``--yes-i-really-mean-it``)
 
+* **Turn off async purge threads** The volumes plugin spawns threads for
+  asynchronously purging trashed/deleted subvolumes. To help troubleshooting or
+  recovery effort, these purge threads can be disabled using:
+
+.. code:: bash
+
+    ceph config set mgr mgr/volumes/pause_purging true
+
+  To resume purging run::
+
+    ceph config set mgr mgr/volumes/pause_purging false
+
+* **Turn off async cloner threads** The volumes plugin spawns threads for
+  asynchronously cloning subvolume snapshots. To help troubleshooting or
+  recovery effort, these cloner threads can be disabled using:
+
+.. code:: bash
+
+    ceph config set mgr mgr/volumes/pause_cloning true
+
+  To resume cloning run::
+
+    ceph config set mgr mgr/volumes/pause_cloning false
+
+
 
 Expediting MDS journal trim
 ===========================

qa/suites/fs/volumes/tasks/volumes/test/async.yaml

@@ -0,0 +1,6 @@
+tasks:
+  - cephfs_test_runner:
+      fail_on_skip: false
+      modules:
+        - tasks.cephfs.test_volumes.TestPausePurging
+        - tasks.cephfs.test_volumes.TestPauseCloning

qa/tasks/cephfs/mount.py

@@ -785,7 +785,7 @@ def run_shell(self, args, **kwargs):
 
     def get_shell_stdout(self, args, timeout=300, **kwargs):
         return self.run_shell(args=args, timeout=timeout, **kwargs).stdout.\
-            getvalue().strip()
+            getvalue()
 
     def run_shell_payload(self, payload, wait=True, timeout=900, **kwargs):
         kwargs.setdefault('cwd', self.mountpoint)