qa/suites/rbd/nvmeof: add multi-subsystem setup and thrash test

1. qa/tasks/nvmeof.py:
    1.1. create multiple rbd images for all subsystems
    1.2. add NvmeofThrasher and ThrashTest
2. qa/tasks/mon_thrash.py: add 'switch_thrashers' option
3. nvmeof_setup_subsystem.sh: create multiple subsystems and enable HA
4. Restructure qa/suites/rbd/nvmeof: Create two sub-suites
   - "basic" (nvmeof_initiator job)
   - "thrash" (new: nvmeof_mon_thrash and nvmeof_thrash jobs)

Signed-off-by: Vallari Agrawal <[email protected]>

Author: VallariAg

qa/suites/rbd/nvmeof/.qa qa/suites/nvmeof/.qaqa/suites/rbd/nvmeof/.qa renamed to qa/suites/nvmeof/.qa

@@ -0,0 +1,15 @@
+use_shaman: True
+tasks:
+- install:
+    extra_packages:
+        - nvme-cli
+- cephadm: 
+    watchdog_setup:
+- cephadm.shell:
+    host.a:
+    # get state before nvmeof deployment
+    - ceph orch status
+    - ceph orch ps
+    - ceph orch host ls
+    - ceph orch device ls
+    - ceph osd lspools

qa/suites/rbd/nvmeof/% qa/suites/nvmeof/basic/%qa/suites/rbd/nvmeof/% renamed to qa/suites/nvmeof/basic/%

@@ -8,10 +8,18 @@ roles:
   - ceph.nvmeof.nvmeof.a
 - - host.b
   - mon.b
+  - mon.c
   - osd.2
   - osd.3
   - osd.4
   - client.1
   - ceph.nvmeof.nvmeof.b
 - - client.2
 - - client.3
+
+overrides:
+  ceph:
+    conf:
+      mon:
+        # cephadm can take up to 5 minutes to bring up remaining mons
+        mon down mkfs grace: 300

qa/suites/rbd/nvmeof/base/.qa qa/suites/nvmeof/basic/.qaqa/suites/rbd/nvmeof/base/.qa renamed to qa/suites/nvmeof/basic/.qa

@@ -0,0 +1 @@
+../.qa/