PIP 36: Max Message Size

PIP 36: Max Message Size

• Status: Proposed
• Author: Yong Zhang
• Discusstion Thread:
• Issue:

Motivation

Currently MaxMessageSize is hardcoded in Pulsar and it can’t be modified in server configuration. So there is no way when user want to modify the limit to transfer larger size message. Hence we add a MaxMessageSize config in broker.conf to solve this problem. Because broker server will decide how much message size will be received so client need know how much message client can be sent.

Hence we propose adding a new flag max_message_size in protocol to tell a client what is the max message size that it can use once it connected.

Changes

Wire Protocol

We propose to introduce an optional field called max_message_size in CommandConnected .

message CommandConnected {
	required string server_version = 1;
	optional int32 protocol_version = 2 [default = 0];

	// used for telling clients what is the max message size it can use
	optional int64 max_message_size = 3 [default = 5 * 1024 * 1024];
}

On server side, we need send max_message_size to clients when broker complete connect:

// complete the connect and sent newConnected command
private void completeConnect(int clientProtoVersion, String clientVersion, long maxMessageSize) {
    ctx.writeAndFlush(Commands.newConnected(clientProtoVersion, maxMessageSize));
    state = State.Connected;
    remoteEndpointProtocolVersion = clientProtoVersion;
    if (isNotBlank(clientVersion) && !clientVersion.contains(" ") /* ignore default version: pulsar client */) {
        this.clientVersion = clientVersion.intern();
    }
}

On client side, client should set max message size in configuration and it should be smaller than server support.