Skip to content

[Bugfix]Fix 'invalid argument' error when setting QP state to INIT vi… #722

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[Bugfix]Fix 'invalid argument' error when setting QP state to INIT vi… #722

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 5 additions & 10 deletions mooncake-transfer-engine/src/transport/rdma_transport/rdma_endpoint.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -355,17 +355,12 @@ int RdmaEndPoint::doSetupConnection(int qp_index, const std::string &peer_gid,
return ERR_INVALID_ARGUMENT;
auto &qp = qp_list_[qp_index];

// Any state -> RESET
// Any state -> RESET (removed)
// Modifying the QP to RESET is not necessary here, as it is already in the
// RESET state. This avoids the 'invalid argument' issue on some RDMA
// devices, such as the Intel E810.
Comment on lines +358 to +361
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Any state -> RESET (removed)
// Modifying the QP to RESET is not necessary here, as it is already in the
// RESET state. This avoids the 'invalid argument' issue on some RDMA
// devices, such as the Intel E810.
// 1. Query QP State
ibv_qp_attr cur_attr;
ibv_qp_init_attr init_attr;
int cur_state = IBV_QPS_ERR;
if (ibv_query_qp(qp, &cur_attr, IBV_QP_STATE, &init_attr) == 0) {
cur_state = cur_attr.qp_state;
} else {
PLOG(WARNING) << "[Handshake] ibv_query_qp failed, proceed to RESET anyway";
}
// 2. If current state is not reset
if (cur_state != IBV_QPS_RESET) {
ibv_qp_attr attr{};
attr.qp_state = IBV_QPS_RESET;
int ret = ibv_modify_qp(qp, &attr, IBV_QP_STATE);
if (ret) {
std::string message = "Failed to modify QP to RESET";
PLOG(ERROR) << "[Handshake] " << message;
if (reply_msg) *reply_msg = message + ": " + strerror(errno);
return ERR_ENDPOINT;
}
}

ibv_qp_attr attr;
memset(&attr, 0, sizeof(attr));
attr.qp_state = IBV_QPS_RESET;
int ret = ibv_modify_qp(qp, &attr, IBV_QP_STATE);
if (ret) {
std::string message = "Failed to modify QP to RESET";
PLOG(ERROR) << "[Handshake] " << message;
if (reply_msg) *reply_msg = message + ": " + strerror(errno);
return ERR_ENDPOINT;
}
int ret;

// RESET -> INIT
memset(&attr, 0, sizeof(attr));
Expand Down
Loading