rdma: fix uninitialized value on fi_cq_readerr call

In commit 35264d2, we changed the error processing code so that
`fi_cq_err_entry` was left uninitialized (instead of zero-initialized)
on call to `fi_cq_readerr`. However, per the Libfabric spec, the
`err_data_size` field of `fi_cq_err_entry` needs to be set to zero (or a
valid buffer provided):

> If err_data_size is 0 on input, or the fabric was opened with release
> < 1.5, then any buffer referenced by err_data will be ignored on input.

ref: https://ofiwg.github.io/libfabric/v2.1.0/man/fi_cq.3.html

Fix by zeroing out the whole struct, as we did before. Add explanatory
comments to this line in both protocols.

Signed-off-by: Eric Raut <[email protected]>

Author: rauteric

src/nccl_ofi_rdma.cpp

@@ -2020,7 +2020,12 @@ static int ofi_process_cq_rail(nccl_net_ofi_rdma_ep_t *ep, nccl_net_ofi_ep_rail_
 			if (OFI_UNLIKELY(ret != 0))
 				goto exit;
 		} else if (OFI_UNLIKELY(rc == -FI_EAVAIL)) {
-			struct fi_cq_err_entry err_entry;
+			/*
+			 * On call to fi_cq_readerr, Libfabric requires some members of
+			 * err_entry to be zero-initialized or point to valid data.  For
+			 * simplicity, just zero out the whole struct.
+			 */
+			struct fi_cq_err_entry err_entry = { };
 
 			ret = fi_cq_readerr(rail->cq, &err_entry, 0);
 			if (OFI_UNLIKELY(ret == -FI_EAGAIN)) {

src/nccl_ofi_sendrecv.cpp

@@ -295,6 +295,11 @@ static int sendrecv_cq_process(struct fid_cq *cq)
 {
 	ssize_t rc = 0;
 	int ret = 0;
+	/*
+	 * On call to fi_cq_readerr, Libfabric requires some members of
+	 * err_entry to be zero-initialized or point to valid data.  For
+	 * simplicity, just zero out the whole struct.
+	 */
 	struct fi_cq_err_entry err_buffer = {};
 	struct fi_cq_tagged_entry cqe_tagged_buffers[cq_read_count];