DAOS-9576 chk: aggregated patch for cat_recovery (#13718)

* DAOS-15084 chk: aggregated patch for cat_recovery

DAOS employed various fault tolerance mechanisms to cope with regular temporary
or permanent hardware failures, such as Raft engine used for pool/container metadata,
EC/Replica used for user data, etc. These mechanisms ensure the system survives
from most regular failures, and automatic self- healing mechanisms are in place to
bring back redundancy once tolerable failure happens. However, several factors can
challenge this design:

- Users can control (and lower) the data protection on a per-container basis.
- The system is facing some unexpected events causing more failures than it is
   designed to tolerate.
- The self-healing mechanism may fail in some specific conditions (e.g., ENOSPC
   on the surviving nodes).
- Hardware bugs (e.g., broken flush, firmware issue, data corruption, ...) are causing
   massive corruption.
- Software bugs (e.g., corner cases, overflow, ...).
- Human errors.

The DAOS catastrophic recovery feature is introduced to address the failure cases
above. While it is unreasonable to assume that all cases can be covered, this feature
covers the most likely ones. The first goal is to detect corruptions and distributed
consistency issues and then offer a remediation path whenever possible. 
Remediation options can range from a transparent, automatic fix to a manual repair
or deletion of a pool or container. If catastrophic recovery fails, then the system will
ultimately have to be reformatted. Another aspect to take into account is that the
check and repair should complete in a reasonable amount of time and the framework
should provide estimates on how long it is expected to take for each pool and allow
the administrator to prioritize some pools over others.

This patch allows offline check & repair (when possible) of a DAOS system.

Signed-off-by: Fan Yong <[email protected]>
Signed-off-by: Dalton Bohning <[email protected]>
Signed-off-by: Kris Jacque <[email protected]>
Co-authored-by: Dalton Bohning <[email protected]>

Author: Nasf-Fan

Jenkinsfile

@@ -108,7 +108,8 @@ void fixup_rpmlintrc() {
                     '/usr/bin/hello_drpc',
                     '/usr/bin/daos_firmware',
                     '/usr/bin/daos_admin',
-                    '/usr/bin/daos_server']
+                    '/usr/bin/daos_server',
+                    '/usr/bin/ddb']
 
     String content = readFile(file: 'utils/rpms/daos.rpmlintrc') + '\n\n' +
                      '# https://daosio.atlassian.net/browse/DAOS-11534\n'

ci/codespell.ignores

@@ -34,3 +34,5 @@ expres
 signalling
 laf
 cacl
+chk
+falloc

debian/changelog

@@ -1,3 +1,9 @@
+daos (2.5.101-4) unstable; urgency=medium
+  [ Fan Yong ]
+  * NOOP change to keep in parity with RPM version
+
+ -- Fan Yong <[email protected]>  Fri, 05 Apr 2024 09:30:00 +0900
+
 daos (2.5.101-3) unstable; urgency=medium
   [ Ashley M. Pittman ]
   * Updated pydaos install process

debian/daos-server-tests.install

@@ -8,6 +8,7 @@ usr/bin/smd_ut
 usr/bin/bio_ut
 usr/bin/vea_ut
 usr/bin/vos_tests
+usr/bin/ddb_tests
 usr/bin/vea_stress
 usr/bin/vos_perf
 usr/bin/obj_ctl

debian/daos-server.install

@@ -8,7 +8,10 @@ usr/bin/daos_server_helper
 usr/bin/daos_server
 usr/bin/daos_engine
 usr/bin/daos_metrics
+usr/bin/ddb
+usr/lib64/daos_srv/libchk.so
 usr/lib64/daos_srv/libcont.so
+usr/lib64/daos_srv/libddb.so
 usr/lib64/daos_srv/libdtx.so
 usr/lib64/daos_srv/libmgmt.so
 usr/lib64/daos_srv/libobj.so

src/SConscript

@@ -106,6 +106,7 @@ def scons():
 
     # Build each DAOS component
     SConscript('rsvc/SConscript')
+    SConscript('chk/SConscript')
     SConscript('mgmt/SConscript')
     SConscript('pool/SConscript')
     SConscript('container/SConscript')
@@ -128,6 +129,9 @@ def scons():
     # Build utilities
     SConscript('utils/SConscript')
 
+    # Build ddb
+    SConscript('ddb/SConscript')
+
     # Build the control plane components
     SConscript('control/SConscript')
 

src/bio/bio_context.c

@@ -9,8 +9,6 @@
 #include "bio_internal.h"
 #include "bio_wal.h"
 
-#define BIO_BLOB_HDR_MAGIC	(0xb0b51ed5)
-
 struct blob_cp_arg {
 	spdk_blob_id		 bca_id;
 	struct spdk_blob	*bca_blob;

src/bio/bio_internal.h

@@ -19,6 +19,7 @@
 
 #include "smd.pb-c.h"
 
+#define BIO_BLOB_HDR_MAGIC	(0xb0b51ed5)
 #define BIO_DMA_PAGE_SHIFT	12	/* 4K */
 #define BIO_DMA_PAGE_SZ		(1UL << BIO_DMA_PAGE_SHIFT)
 #define BIO_XS_CNT_MAX		BIO_MAX_VOS_TGT_CNT /* Max VOS xstreams per blobstore */

src/cart/crt_corpc.c

@@ -57,8 +57,8 @@ crt_corpc_info_init(struct crt_rpc_priv *rpc_priv,
 		rpc_priv->crp_flags |= CRT_RPC_FLAG_COLL;
 		if (co_info->co_grp_priv->gp_primary)
 			rpc_priv->crp_flags |= CRT_RPC_FLAG_PRIMARY_GRP;
-		if (flags & CRT_RPC_FLAG_FILTER_INVERT)
-			rpc_priv->crp_flags |= CRT_RPC_FLAG_FILTER_INVERT;
+		rpc_priv->crp_flags |= flags & (CRT_RPC_FLAG_FILTER_INVERT |
+						CRT_RPC_FLAG_CO_FAILOUT);
 
 		co_hdr->coh_grpid = grp_priv->gp_pub.cg_grpid;
 		co_hdr->coh_filter_ranks = co_info->co_filter_ranks;
@@ -906,6 +906,11 @@ crt_corpc_req_hdlr(struct crt_rpc_priv *rpc_priv)
 	}
 
 forward_done:
+	if (rc != 0 && rpc_priv->crp_flags & CRT_RPC_FLAG_CO_FAILOUT) {
+		crt_corpc_complete(rpc_priv);
+		goto out;
+	}
+
 	/* NOOP bcast (no child and root excluded) */
 	if (co_info->co_child_num == 0 && co_info->co_root_excluded)
 		crt_corpc_complete(rpc_priv);

src/chk/SConscript

@@ -0,0 +1,31 @@
+# pylint: disable=consider-using-f-string
+# pylint: disable-next=wrong-spelling-in-comment
+"""Build check library"""
+
+
+def scons():
+    """Execute build"""
+    Import('env', 'prereqs')
+
+    env.AppendUnique(LIBPATH=[Dir('.')])
+
+    denv = env.Clone()
+
+    # common
+    prereqs.require(denv, 'argobots', 'protobufc')
+    chk_pb = denv.SharedObject(['chk.pb-c.c'])
+    Export('chk_pb')
+
+    if not prereqs.server_requested():
+        return
+
+    # chk
+    chk = denv.d_library('chk',
+                         [chk_pb, 'chk_srv.c', 'chk_common.c', 'chk_vos.c',
+                          'chk_rpc.c', 'chk_upcall.c', 'chk_iv.c', 'chk_leader.c',
+                          'chk_engine.c'], install_off="../..")
+    denv.Install('$PREFIX/lib64/daos_srv', chk)
+
+
+if __name__ == "SCons.Script":
+    scons()