DAOS-17127 cart: Fix random rank calculation (#16014) (#16111)

knard38 · web-flow · commit 07c8d571cdd3 · 2025-03-19T13:16:01.000-05:00
Fix the code of the  DAOS lib calculating the target rank to send RPC.
Update the DAOS user documentation to properly handle system
extension and rank exclusion.

Signed-off-by: Cedric Koch-Hofer &lt;cedric.koch-hofer@hpe.com&gt;
diff --git a/docs/admin/administration.md b/docs/admin/administration.md
@@ -899,6 +899,31 @@ and then restarted to rejoin the system. An failed engine might also be excluded
 from the pools it hosted, please check the pool operation section on how to
 reintegrate an excluded engine.
 
+After one or more DAOS engines being excluded, the DAOS agent cache needs to be
+refreshed.  For detailed information, please refer to the [1][System Deployment
+documentation].  Before refreshing the DAOS Agent cache, it should be checked
+that the exclusion information has been spread to the Management Service leader.
+This could be done using the `dump-attachinfo` sub-command of the `daos_agent`
+executable:
+
+```bash
+daos_agent -o /tmp/daos_agent-tmp.yml dump-attachinfo
+```
+
+This usage of the `daos_agent` command needs a minimal DAOS agent configuration
+file `/tmp/daos_agent-tmp.yml` such as:
+
+```yaml
+name: daos_server
+access_points:
+- sertver-1
+port: 10001
+transport_config:
+  allow_insecure: true
+log_file: /tmp/daos_agent-tmp.log
+```
+
+
 ### Shutdown
 
 When up and running, the entire system can be shutdown.
@@ -1041,7 +1066,7 @@ dmg as follows:
 $ dmg storage format -l ${new_storage_node}
 ```
 
-new_storage_node should be replaced with the hostname or the IP address of the
+`new_storage_node` should be replaced with the hostname or the IP address of the
 new storage node (comma separated list or range of hosts for multiple nodes)
 to be added.
 
@@ -1054,6 +1079,11 @@ the system (this can be checked with `dmg system query -v`).
     said, existing pools won't be automatically extended to use the new servers.
     Please see the pool operation section for how to extend the pool membership.
 
+After extending the system, the cache of the `daos_agent` service of the client
+nodes needs to be refreshed.  For detailed information, please refer to the
+[1][System Deployment documentation].
+
+
 ## Software Upgrade
 
 The DAOS v2.0 wire protocol and persistent layout is not compatible with
@@ -1102,3 +1132,5 @@ Examples:
   * daos_server 2.4.0 is only compatible with daos_engine 2.4.0
   * daos_agent 2.6.0 is compatible with daos_server 2.4.0 (2.5 is a development version)
   * dmg 2.4.1 is compatible with daos_server 2.4.0
+
+[1]: <deployment.md#refresh-agent-cache>(Refresh DAOS Agent Cache)
diff --git a/docs/admin/deployment.md b/docs/admin/deployment.md
@@ -1360,9 +1360,6 @@ engines:
 <end>
 ```
 
-There are a few optional providers that are not built by default. For detailed
-information, please refer to the [DAOS build documentation][6].
-
 !!! note
     DAOS Control Servers will need to be restarted on all hosts after updates to the server
     configuration file.
@@ -1372,9 +1369,6 @@ information, please refer to the [DAOS build documentation][6].
 
     This will be the host which bootstraps the DAOS management service (MS).
 
->The support of the optional providers is not guarantee and can be removed
->without further notification.
-
 ### Network Configuration
 
 #### Network Scan
@@ -1731,42 +1725,56 @@ Once the service file is installed and `systemctl daemon-reload` has been run to
 reload the configuration, the `daos_agent` can be started through systemd
 as shown above.
 
-#### Disable Agent Cache (Optional)
+#### Refresh Agent Cache
 
-In certain circumstances (e.g. for DAOS development or system evaluation), it
-may be desirable to disable the DAOS Agent's caching mechanism in order to avoid
-stale system information being retained across reformats of a system. The DAOS
-Agent normally caches a map of rank-to-fabric URI lookups as well as client network
-configuration data in order to reduce the number of management RPCs required to
-start an application. When this information becomes stale, the Agent must be
-restarted in order to repopulate the cache with new information.
-Alternatively, the caching mechanism may be disabled, with the tradeoff that
-each application launch will invoke management RPCs in order to obtain system
-connection information.
+In certain circumstances (e.g. [system extension][6] with new servers), it may
+be needed to refresh the DAOS Agent cache.  The DAOS Agent normally caches a map
+of rank-to-fabric URI lookups as well as client network configuration data in
+order to reduce the number of management RPCs required to start an application.
 
-To disable the DAOS Agent caching mechanism, set the following environment
-variable before starting the `daos_agent` process:
+After a new server has been added to the system, or an existing server has been
+permanently removed from the system, the administrator should ensure that the
+Agent is not serving stale system information to new clients. There are three
+options to achieve this goal:
 
-`DAOS_AGENT_DISABLE_CACHE=true`
+1. Send the `SIGUSR2` signal to the `daos_agent` process to force a refresh on
+   demand. This could be done with running the following command
 
-If running from systemd, add the following to the `daos_agent` service file in
-the `[Service]` section before reloading systemd and restarting the
-`daos_agent` service:
+```bash
+pkill -x -SIGUSR2 daos_agent
+```
 
-`Environment=DAOS_AGENT_DISABLE_CACHE=true`
+2. Add a cache expiration value, defined in minutes, to the Agent configuration
+   file `daos_agent.yml` in order to cause a cache refresh when the data is
+   older than the defined value.
 
+```yaml
+cache_expiration: 30
+```
 
-[^1]: https://github.com/intel/ipmctl
+3. Disable the caching mechanism completely, with the tradeoff that each
+   application launch will invoke management RPCs in order to obtain system
+   connection information.  To disable the DAOS Agent caching mechanism, set the
+   following environment variable before starting the `daos_agent` process:
 
-[^2]: https://github.com/daos-stack/daos/tree/master/utils/config
+```bash
+DAOS_AGENT_DISABLE_CACHE=true
+```
 
-[^3]: [https://www.open-mpi.org/faq/?category=running\#mpirun-hostfile](https://www.open-mpi.org/faq/?category=running#mpirun-hostfile)
+   If running from systemd, add the following line to the `daos_agent` service
+   file in the `[Service]` section before reloading systemd and restarting the
+   `daos_agent` service:
 
-[^4]: https://github.com/daos-stack/daos/tree/master/src/control/README.md
+```ini
+Environment=DAOS_AGENT_DISABLE_CACHE=true
+```
 
-[^5]: https://github.com/pmem/ndctl/issues/130
+   It is also possible to disable the `daos_agent` cache with adding the
+   following entry into the `daos_agent.yml` configuration file:
 
-[6]: <../dev/development.md#building-optional-components> (Building DAOS for Development)
+```yaml
+disable_caching: true
+```
 
 ## Multi-user DFuse setup
 
@@ -1781,3 +1789,16 @@ fuse and must be enabled by root before any user can use it.  To allow this then
 uncomment a line in `/etc/fuse.conf` to enable the `user_allow_other` setting.  The daos-client rpm
 does not do this automatically. An administrator must set this option on all nodes on which they
 want to provide a persistent multi-user dfuse service.
+
+
+[^1]: https://github.com/intel/ipmctl
+
+[^2]: https://github.com/daos-stack/daos/tree/master/utils/config
+
+[^3]: [https://www.open-mpi.org/faq/?category=running\#mpirun-hostfile](https://www.open-mpi.org/faq/?category=running#mpirun-hostfile)
+
+[^4]: https://github.com/daos-stack/daos/tree/master/src/control/README.md
+
+[^5]: https://github.com/pmem/ndctl/issues/130
+
+[6]: <administration.md#system-extension>(Extension of the DAOS system)
diff --git a/src/client/api/event.c b/src/client/api/event.c
@@ -1,5 +1,7 @@
 /**
  * (C) Copyright 2016-2024 Intel Corporation.
+ * (C) Copyright 2025 Hewlett Packard Enterprise Development LP
+ * (C) Copyright 2025 Google LLC
  *
  * SPDX-License-Identifier: BSD-2-Clause-Patent
  */
@@ -117,7 +119,7 @@ daos_eq_lib_reset_after_fork(void)
 	eq_ref            = 0;
 	ev_thpriv_is_init = false;
 	crt_info          = daos_crt_init_opt_get(false, 1);
-	rc                = dc_mgmt_net_cfg(NULL, crt_info);
+	rc                = dc_mgmt_net_cfg_init(NULL, crt_info);
 	if (rc == 0)
 		rc = daos_eq_lib_init(crt_info);
 	D_FREE(crt_info->cio_provider);
diff --git a/src/client/api/init.c b/src/client/api/init.c
@@ -212,7 +212,7 @@ daos_init(void)
 	 * get CaRT configuration (see mgmtModule.handleGetAttachInfo for the
 	 * handling of NULL system names)
 	 */
-	rc = dc_mgmt_net_cfg(NULL, crt_info);
+	rc = dc_mgmt_net_cfg_init(NULL, crt_info);
 	if (rc != 0)
 		D_GOTO(out_attach, rc);
 
@@ -223,7 +223,7 @@ daos_init(void)
 	D_FREE(crt_info->cio_domain);
 	if (rc != 0) {
 		D_ERROR("failed to initialize eq_lib: "DF_RC"\n", DP_RC(rc));
-		D_GOTO(out_attach, rc);
+		D_GOTO(out_net_cfg, rc);
 	}
 
 	/**
@@ -290,6 +290,8 @@ daos_init(void)
 	pl_fini();
 out_eq:
 	daos_eq_lib_fini();
+out_net_cfg:
+	dc_mgmt_net_cfg_fini();
 out_attach:
 	dc_mgmt_drop_attach_info();
 out_job:
@@ -350,6 +352,7 @@ daos_fini(void)
 	if (rc != 0)
 		D_ERROR("failed to disconnect some resources may leak, " DF_RC "\n", DP_RC(rc));
 
+	dc_mgmt_net_cfg_fini();
 	dc_tm_fini();
 	dc_mgmt_drop_attach_info();
 	dc_agent_fini();
diff --git a/src/client/api/rpc.c b/src/client/api/rpc.c
@@ -1,5 +1,6 @@
 /**
  * (C) Copyright 2016-2024 Intel Corporation.
+ * (C) Copyright 2025 Hewlett Packard Enterprise Development LP
  *
  * SPDX-License-Identifier: BSD-2-Clause-Patent
  */
@@ -96,7 +97,7 @@ daos_rpc_send_wait(crt_rpc_t *rpc)
 }
 
 struct rpc_proto {
-	int            nr_ranks;
+	int            rank_idx;
 	crt_endpoint_t ep;
 	int            version;
 	int            rc;
@@ -114,7 +115,17 @@ query_cb(struct crt_proto_query_cb_info *cb_info)
 	int			 rc;
 
 	if (daos_rpc_retryable_rc(cb_info->pq_rc)) {
-		rproto->ep.ep_rank = (rproto->ep.ep_rank + 1) % rproto->nr_ranks;
+		int      nr_ranks;
+		d_rank_t rank;
+
+		/** select next rank to issue the retry proto query rpc to */
+		nr_ranks = dc_mgmt_net_get_num_srv_ranks();
+		D_ASSERT(nr_ranks > 0);
+		rproto->rank_idx = (rproto->rank_idx + 1) % nr_ranks;
+		rank             = dc_mgmt_net_get_srv_rank(rproto->rank_idx);
+		D_ASSERT(rank != CRT_NO_RANK);
+		rproto->ep.ep_rank = rank;
+
 		rproto->timeout += 3;
 		rc = crt_proto_query_with_ctx(&rproto->ep, rproto->base_opc, rproto->ver_array,
 					      rproto->array_size, rproto->timeout, query_cb, rproto,
@@ -139,6 +150,8 @@ daos_rpc_proto_query(crt_opcode_t base_opc, uint32_t *ver_array, int count, int
 	crt_context_t		 ctx = daos_get_crt_ctx();
 	int                      rc;
 	int			 i;
+	int                      nr_ranks;
+	d_rank_t                 rank;
 
 	rc = dc_mgmt_sys_attach(NULL, &sys);
 	if (rc != 0) {
@@ -151,8 +164,16 @@ daos_rpc_proto_query(crt_opcode_t base_opc, uint32_t *ver_array, int count, int
 		D_GOTO(out_detach, rc = -DER_NOMEM);
 
 	/** select a random rank to issue the proto query rpc to */
-	rproto->nr_ranks   = dc_mgmt_net_get_num_srv_ranks();
-	rproto->ep.ep_rank = d_rand() % rproto->nr_ranks;
+	nr_ranks = dc_mgmt_net_get_num_srv_ranks();
+	if (nr_ranks == 0) {
+		D_ERROR("failed to select an attached ranks: no attached ranks");
+		D_GOTO(out_free, -DER_NONEXIST);
+	}
+	rproto->rank_idx = d_rand() % nr_ranks;
+	rank             = dc_mgmt_net_get_srv_rank(rproto->rank_idx);
+	D_ASSERT(rank != CRT_NO_RANK);
+	rproto->ep.ep_rank = rank;
+
 	rproto->ep.ep_tag  = 0;
 	rproto->ver_array = ver_array;
 	rproto->array_size = count;
diff --git a/src/include/daos/mgmt.h b/src/include/daos/mgmt.h
@@ -1,5 +1,6 @@
 /**
  * (C) Copyright 2016-2024 Intel Corporation.
+ * (C) Copyright 2025 Hewlett Packard Enterprise Development LP
  *
  * SPDX-License-Identifier: BSD-2-Clause-Patent
  */
@@ -59,7 +60,10 @@ ssize_t dc_mgmt_sys_encode(struct dc_mgmt_sys *sys, void *buf, size_t cap);
 ssize_t dc_mgmt_sys_decode(void *buf, size_t len, struct dc_mgmt_sys **sysp);
 
 int
-     dc_mgmt_net_cfg(const char *name, crt_init_options_t *crt_info);
+dc_mgmt_net_cfg_init(const char *name, crt_init_options_t *crt_info);
+void
+    dc_mgmt_net_cfg_fini(void);
+
 int dc_mgmt_net_cfg_check(const char *name);
 int dc_mgmt_get_pool_svc_ranks(struct dc_mgmt_sys *sys, const uuid_t puuid,
 			       d_rank_list_t **svcranksp);
@@ -71,6 +75,8 @@ int dc_mgmt_notify_pool_connect(struct dc_pool *pool);
 int dc_mgmt_notify_pool_disconnect(struct dc_pool *pool);
 int dc_mgmt_notify_exit(void);
 int dc_mgmt_net_get_num_srv_ranks(void);
+d_rank_t
+     dc_mgmt_net_get_srv_rank(int idx);
 int dc_mgmt_get_sys_info(const char *sys, struct daos_sys_info **info);
 void dc_mgmt_put_sys_info(struct daos_sys_info *info);
 int dc_get_attach_info(const char *name, bool all_ranks, struct dc_mgmt_sys_info *info,
diff --git a/src/mgmt/cli_mgmt.c b/src/mgmt/cli_mgmt.c
