Skip to content

ggml-rpc: chunk send()/recv() to avoid EINVAL for very large tensors over RPC (macOS & others) #15188

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 13, 2025
Merged

ggml-rpc: chunk send()/recv() to avoid EINVAL for very large tensors over RPC (macOS & others) #15188

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0e7aa4e
ggml-rpc: chunk send()/recv() to avoid EINVAL for very large tensors …
Aug 8, 2025
829d6b6
ggml-rpc: rename RPC_IO_CHUNK->MAX_CHUNK_SIZE, use std::min() for cap…
Tak-RS Aug 11, 2025
b44560e
rpc: drop n==0 special case in send_data(); retry in loop per review
Tak-RS Aug 12, 2025
7c7c3d9
rpc: remove trailing whitespace in send_data()
Tak-RS Aug 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions ggml/src/ggml-rpc/ggml-rpc.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@

namespace fs = std::filesystem;

static constexpr size_t RPC_IO_CHUNK = 1024ull * 1024ull * 1024ull; // 1 GiB
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename to MAX_CHUNK_SIZE


#ifdef _WIN32
typedef SOCKET sockfd_t;
using ssize_t = __int64;
Expand Down Expand Up @@ -323,27 +325,43 @@ static std::shared_ptr<socket_t> create_server_socket(const char * host, int por
static bool send_data(sockfd_t sockfd, const void * data, size_t size) {
size_t bytes_sent = 0;
while (bytes_sent < size) {
ssize_t n = send(sockfd, (const char *)data + bytes_sent, size - bytes_sent, 0);
size_t size_to_send = size - bytes_sent;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

size_t size_to_send = std::max(size - bytes_sent, MAX_CHUNK_SIZE);

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean std::min, not std::max, sorry

if (size_to_send > RPC_IO_CHUNK) size_to_send = RPC_IO_CHUNK;
ssize_t n = send(sockfd, (const char *)data + bytes_sent, size_to_send, 0);
if (n < 0) {
#ifndef _WIN32
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replace with GGML_LOG_ERROR

perror("send");
#else
fprintf(stderr, "send failed (bytes_sent=%zu, size_to_send=%zu)\n", bytes_sent, size_to_send);
#endif
return false;
}
bytes_sent += n;
}
return true;
}


static bool recv_data(sockfd_t sockfd, void * data, size_t size) {
size_t bytes_recv = 0;
while (bytes_recv < size) {
ssize_t n = recv(sockfd, (char *)data + bytes_recv, size - bytes_recv, 0);
size_t size_to_recv = size - bytes_recv;
if (size_to_recv > RPC_IO_CHUNK) size_to_recv = RPC_IO_CHUNK;
ssize_t n = recv(sockfd, (char *)data + bytes_recv, size_to_recv, 0);
if (n <= 0) {
#ifndef _WIN32
perror("recv");
#else
fprintf(stderr, "recv failed (bytes_recv=%zu, size_to_recv=%zu)\n", bytes_recv, size_to_recv);
#endif
return false;
}
bytes_recv += n;
}
return true;
}


static bool send_msg(sockfd_t sockfd, const void * msg, size_t msg_size) {
if (!send_data(sockfd, &msg_size, sizeof(msg_size))) {
return false;
Expand Down