sendrecv: fix random system crashes during multi-threaded comm abort scenario

The `err_buffer` needs to be zero initialized before passed to libfabric `fi_cq_readerr`.
Earlir when the buffer was allocated outside the while() loop, we missed resetting it
to zero before invoking `fi_cq_readerr` everytime inside the while loop. and this was
causing random memory corruptions.

The fix is either (1) allocate the `err_buffer` outside and zero init every time before
calling `fi_cq_readerr` or (2) move the allocation+zero init to inside the while loop.

This commit implements the option(2): moved the `err_buffer` allocation+zero init to
inside the while loop.

Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>

Author: snadampal

src/nccl_ofi_sendrecv.cpp

@@ -277,12 +277,6 @@ static int sendrecv_cq_process(struct fid_cq *cq)
 {
 	ssize_t rc = 0;
 	int ret = 0;
-	/*
-	 * On call to fi_cq_readerr, Libfabric requires some members of
-	 * err_entry to be zero-initialized or point to valid data.  For
-	 * simplicity, just zero out the whole struct.
-	 */
-	struct fi_cq_err_entry err_buffer = {};
 	struct fi_cq_tagged_entry cqe_tagged_buffers[cq_read_count];
 
 	while (true) {
@@ -296,7 +290,12 @@ static int sendrecv_cq_process(struct fid_cq *cq)
 		}
 		else if (OFI_UNLIKELY(rc == -FI_EAVAIL)) {
 			nccl_net_ofi_context_t *ctx;
-
+			/*
+			 * On call to fi_cq_readerr, Libfabric requires some members of
+			 * err_entry to be zero-initialized or point to valid data.  For
+			 * simplicity, just zero out the whole struct.
+			 */
+			struct fi_cq_err_entry err_buffer = {};
 			rc = fi_cq_readerr(cq, &err_buffer, 0);
 
 			if (OFI_UNLIKELY(rc == -FI_EAGAIN)) {