Skip to content

ggml-rpc: chunk send()/recv() to avoid EINVAL for very large tensors over RPC (macOS & others) #15188

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 13, 2025
Merged

ggml-rpc: chunk send()/recv() to avoid EINVAL for very large tensors over RPC (macOS & others) #15188

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0e7aa4e
ggml-rpc: chunk send()/recv() to avoid EINVAL for very large tensors …
Aug 8, 2025
829d6b6
ggml-rpc: rename RPC_IO_CHUNK->MAX_CHUNK_SIZE, use std::min() for cap…
Tak-RS Aug 11, 2025
b44560e
rpc: drop n==0 special case in send_data(); retry in loop per review
Tak-RS Aug 12, 2025
7c7c3d9
rpc: remove trailing whitespace in send_data()
Tak-RS Aug 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 18 additions & 5 deletions ggml/src/ggml-rpc/ggml-rpc.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,12 @@
#include <cstring>
#include <fstream>
#include <filesystem>
#include <algorithm>

namespace fs = std::filesystem;

static constexpr size_t MAX_CHUNK_SIZE = 1024ull * 1024ull * 1024ull; // 1 GiB

#ifdef _WIN32
typedef SOCKET sockfd_t;
using ssize_t = __int64;
Expand Down Expand Up @@ -323,23 +326,33 @@ static std::shared_ptr<socket_t> create_server_socket(const char * host, int por
static bool send_data(sockfd_t sockfd, const void * data, size_t size) {
size_t bytes_sent = 0;
while (bytes_sent < size) {
ssize_t n = send(sockfd, (const char *)data + bytes_sent, size - bytes_sent, 0);
size_t size_to_send = std::min(size - bytes_sent, MAX_CHUNK_SIZE);
ssize_t n = send(sockfd, (const char *)data + bytes_sent, size_to_send, 0);
if (n < 0) {
GGML_LOG_ERROR("send failed (bytes_sent=%zu, size_to_send=%zu)\n",
bytes_sent, size_to_send);
return false;
}
bytes_sent += n;
bytes_sent += (size_t)n;
}
return true;
}

static bool recv_data(sockfd_t sockfd, void * data, size_t size) {
size_t bytes_recv = 0;
while (bytes_recv < size) {
ssize_t n = recv(sockfd, (char *)data + bytes_recv, size - bytes_recv, 0);
if (n <= 0) {
size_t size_to_recv = std::min(size - bytes_recv, MAX_CHUNK_SIZE);
ssize_t n = recv(sockfd, (char *)data + bytes_recv, size_to_recv, 0);
if (n < 0) {
GGML_LOG_ERROR("recv failed (bytes_recv=%zu, size_to_recv=%zu)\n",
bytes_recv, size_to_recv);
return false;
}
if (n == 0) {
GGML_LOG_ERROR("recv returned 0 (peer closed?)\n");
return false;
}
bytes_recv += n;
bytes_recv += (size_t)n;
}
return true;
}
Expand Down
Loading